๋ฐ˜์‘ํ˜•

๐Ÿ Python & library 49

[Optuna] ๋”ฅ๋Ÿฌ๋‹ ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ์ตœ์ ํ™”ํ•˜๊ธฐ

Optuna๋Š” ํŒŒ์ด์ฌ ๊ธฐ๋ฐ˜์˜ ํ•˜์ดํผํŒŒ๋ผ๋ฏธํ„ฐ ์ตœ์ ํ™” (hyperparameter optimization) ํ”„๋ ˆ์ž„์›Œํฌ๋กœ, ์‹ฌํ”Œํ•˜๊ณ  ์œ ์—ฐํ•œ API๋ฅผ ์ œ๊ณตํ•œ๋‹ค. ๋ณธ ๊ธ€์—์„œ๋Š” Optuna์˜ ์ฃผ์š” ๊ธฐ๋Šฅ๊ณผ ์‚ฌ์šฉ๋ฐฉ๋ฒ•์„ ๊ฐ„๋‹จํžˆ ์†Œ๊ฐœํ•˜๊ณ ์ž ํ•œ๋‹ค. ๊ณต์‹ Docs: https://optuna.readthedocs.io/en/stable/index.html Optuna: A hyperparameter optimization framework — Optuna 3.4.0 documentation © Copyright 2018, Optuna Contributors. Revision 4ea580fc. optuna.readthedocs.io Basic concepts Optuna๋Š” study์™€ trial์„ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ •์˜ํ•œ๋‹ค. Stu..

[PyTorch] Autograd ์ž‘๋™๋ฐฉ์‹ ์•Œ์•„๋ณด๊ธฐ

์œ„ ๋™์˜์ƒ์—์„œ PyTorch Autograd๋ฅผ ์ดํ•ดํ•˜๊ธฐ ์‰ฝ๊ฒŒ ์„ค๋ช…ํ•ด์ฃผ๊ณ  ์žˆ๋‹ค. ๋‹ค์Œ์€ ์œ„ ๋™์˜์ƒ์„ ๊ฐ„๋‹จํžˆ ์ •๋ฆฌํ•œ ๊ธ€์ด๋‹ค. 1. torch.Tensor ๊ฐ tensor์€ ๋‹ค์Œ์˜ attr์„ ๊ฐ–๋Š”๋‹ค data: tensor์˜ ๊ฐ’ grad: tensor์˜ gradient ๊ฐ’. is_leaf์ธ ๊ฒฝ์šฐ์—๋งŒ gradient๊ฐ€ ์ž๋™์œผ๋กœ ์ €์žฅ๋œ๋‹ค. grad_fn: gradient function. ํ•ด๋‹น tensor๊ฐ€ ์–ด๋–ค ์—ฐ์‚ฐ์„ ํ†ตํ•ด forward๋˜์—ˆ๋Š”์ง€์— ๋”ฐ๋ผ ๊ฒฐ์ •๋œ๋‹ค. ex) a * b = c ์ธ ๊ฒฝ์šฐ c์˜ grad_fn์€ MulBackward์ด๋‹ค. is_leaf์ธ ๊ฒฝ์šฐ None is_leaf: (backward ๊ธฐ์ค€) ๊ฐ€์žฅ ๋งˆ์ง€๋ง‰ tensor์ธ์ง€ requires_grad: ๊ณ„์‚ฐ ๊ทธ๋ž˜ํ”„์˜ ์ผ๋ถ€๋กœ ๋“ค์–ด๊ฐˆ ๊ฒƒ์ธ์ง€ 2. gr..

HuggingFace Space ๋งŒ๋“ค๊ธฐ

HuggingFace Space๋Š” ๊ฐ„ํŽธํ•˜๊ณ  ์‰ฝ๊ฒŒ ML ๋ฐ๋ชจ ์•ฑ์„ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ๊ณต๊ฐ„์ด๋‹ค. Spaces - Hugging Face Spaces Discover amazing ML apps made by the community! huggingface.co Space๋“ค์˜ ์˜ˆ์‹œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™๋‹ค. 1: text-to-image Rich Text To Image - a Hugging Face Space by songweig huggingface.co 2: chat-with-GPT4 Chat-with-GPT4 - a Hugging Face Space by ysharma huggingface.co ๋ณธ ๊ธ€์—์„œ๋Š” ์ด๋Ÿฌํ•œ HuggingFace Space๋ฅผ ๋งŒ๋“œ๋Š” ๋ฒ•์— ๋Œ€ํ•ด ๊ฐ„๋žตํžˆ ์†Œ๊ฐœํ•˜๊ณ ์ž ํ•ฉ๋‹ˆ๋‹ค. ๋ณด๋‹ค ์ž์„ธํ•œ ์„ค๋ช…์€ ๊ณต์‹..

PyTorch 2.0์—์„œ ๋‹ฌ๋ผ์ง€๋Š” ์  - torch.compile

PyTorch 2.0 Overview pytorch.org PyTorch 2.0์€ 22๋…„ 12์›” PyTorch Conference์—์„œ ๋ฐœํ‘œ๋˜์—ˆ๊ณ , 23๋…„ 3์›” ์ •์‹ ๋ฆด๋ฆฌ์ฆˆ ๋˜์—ˆ๋‹ค. ์ด์ „์˜ PyTorch 1.x ๋ฒ„์ „๋“ค๋ณด๋‹ค ๋น ๋ฅด๊ณ , Pythonicํ•˜๊ณ  Dynamicํ•˜๋‹ค๊ณ  ํ•œ๋‹ค. ์–ด๋–ค ์ ๋“ค์ด ๋‹ฌ๋ผ์กŒ์„์ง€ ํ•œ๋ฒˆ ์•Œ์•„๋ด…์‹œ๋‹ค. torch.compile torch.compile์€ PyTorch 2.0์˜ ๋ฉ”์ธ API์ด๋‹ค. ๋ชจ๋ธ์„ ๋ฏธ๋ฆฌ ์ปดํŒŒ์ผํ•˜์—ฌ ์†๋„๋ฅผ ๋†’์ด๋Š” ๊ธฐ์ˆ ์ด๋‹ค. torch.compile์€ TorchDynamo, AOTAutograd, PrimTorch, TorchInductor ๋„ค ๊ฐ€์ง€์˜ ์ƒˆ๋กœ์šด ๊ธฐ์ˆ ์„ ๊ธฐ๋ฐ˜์œผ๋กœ ๋งŒ๋“ค์–ด์กŒ๋‹ค. ๊ฐ ๊ธฐ์ˆ ์— ๋Œ€ํ•œ ์ž์„ธํ•œ ์„ค๋ช…์€ ์—ฌ๊ธฐ์—์„œ ์ฐพ์•„๋ณผ ์ˆ˜ ์žˆ๋‹ค. ์‚ฌ์šฉ๋ฒ• torch.compile์€..

[PyTorch Lightning] checkpoint ์ €์žฅํ•˜๊ธฐ

๊ธฐ๋ณธํŽธ - ์ž๋™ ์ €์žฅ Saving and loading checkpoints (basic) — PyTorch Lightning 1.9.0 documentation Shortcuts pytorch-lightning.readthedocs.io PyTorch Lightning์˜ Trainer์„ ์ด์šฉํ•ด ํ•™์Šต์„ ์ง„ํ–‰ํ•˜๋ฉด, ์ž๋™์œผ๋กœ ๊ฐ€์žฅ ๋งˆ์ง€๋ง‰ training epoch์˜ checkpoint๋ฅผ ์ €์žฅํ•ด์ค€๋‹ค. trainer = Trainer() ๋งŒ์•ฝ checkpoint๊ฐ€ ์ €์žฅ๋˜๋Š” ์œ„์น˜๋ฅผ ๋ฐ”๊พธ๊ณ  ์‹ถ๋‹ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ง€์ •ํ•ด์ค„ ์ˆ˜ ์žˆ๋‹ค. trainer = Trainer(default_root_dir='path/to/') ํ˜น์€ ๋ณ„๋„๋กœ checkpoint ์ €์žฅ์„ ํ•˜์ง€ ์•Š์œผ๋ ค๋ฉด ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ง€์ •ํ•˜๋ฉด ๋œ๋‹ค. trainer = Tr..

[PyTorch Lightning] ๋กœ๊ทธ ๊ธฐ๋ก, Tensorboard๋กœ Loggingํ•˜๊ธฐ

Logging — PyTorch Lightning 1.8.6 documentation Shortcuts pytorch-lightning.readthedocs.io ๋กœ๊ทธ๋ฅผ ๊ธฐ๋กํ•˜๋Š” ๋ฐฉ๋ฒ•์€ Lightening Module์—์„œ self.log()๋‚˜ self.log_dict()๋ฅผ ์ด์šฉํ•˜๋ฉด ๋œ๋‹ค. def training_step(self, batch, batch_idx): self.log_dict({'acc': acc, 'recall': recall}) self.log('acc', acc) logging (log(), log_dict() ๋ชจ๋‘ ๋™์ผํ•˜๊ฒŒ ์ ์šฉ) ์˜ ์ค‘์š”ํ•œ ์ธ์ž๋Š” on_step๊ณผ on_epoch์ด๋‹ค. on_step: ํ˜„์žฌ step์— logging on_epoch: ๋กœ๊ทธ๋ฅผ ์ถ•์ ํ•˜์—ฌ epoch ๋งˆ์ง€๋ง‰์— ..

[PyTorch] tensor.detach()์˜ ๊ธฐ๋Šฅ๊ณผ ์˜ˆ์‹œ ์ฝ”๋“œ

PyTorch tensor์— ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” detach() method๋Š” gradient์˜ ์ „ํŒŒ๋ฅผ ๋ฉˆ์ถ”๋Š” ์—ญํ• ์„ ํ•œ๋‹ค. https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html torch.Tensor.detach — PyTorch 1.13 documentation Shortcuts pytorch.org ๋‹ค์Œ ์˜ˆ์‹œ๋ฅผ ํ†ตํ•ด ์‰ฝ๊ฒŒ ์ดํ•ดํ•  ์ˆ˜ ์žˆ๋‹ค. import torch import torch.nn as nn class TestModel(nn.Module): def __init__(self): super().__init__() self.layer1 = nn.Linear(10, 10) self.layer2 = nn.Linear(10, 10) def..

[PyTorch] nn.Embedding ์ดˆ๊ธฐํ™”ํ•˜๊ธฐ (initialization)

PyTorch์˜ nn.Embedding layer์„ ์ดˆ๊ธฐํ™”ํ•˜๋Š” ๋ฐฉ๋ฒ•์—๋Š” ๋‘ ๊ฐ€์ง€๊ฐ€ ์žˆ๋‹ค. embedding = nn.Embedding(num_embeddings, embedding_dim) 1. torch.tensor์˜ ๋‚ด์žฅ method ์ด์šฉํ•˜๊ธฐ embedding.weight.data.uniform_(-1, 1) torch.tensor์€ uniform_ ๋“ฑ์˜ ๋‚ด์žฅ method๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ์–ด ์ด๋ฅผ ํ†ตํ•ด ๊ฐ’์„ ์ดˆ๊ธฐํ™”ํ•  ์ˆ˜ ์žˆ๋‹ค. 2. torch.nn.init ์ด์šฉํ•˜๊ธฐ nn.init.uniform_(embedding.weight, -1.0, 1.0) torch.nn.init์˜ method๋“ค์„ ์ด์šฉํ•  ์ˆ˜๋„ ์žˆ๋‹ค. ์ด ๋ฐฉ๋ฒ•์„ ์ด์šฉํ•˜๋ฉด uniform_ ์ด์™ธ์—๋„ xavier_uniform_ ๋“ฑ ๋ณด๋‹ค ๋‹ค์–‘ํ•œ initi..

[Tensorboard] ์ฐจํŠธ ์ƒ‰๊น” ๋ณ€๊ฒฝํ•˜๊ธฐ

Tensorboard์—์„œ๋Š” ๊ฐ ์ฐจํŠธ์— ์ž๋™์œผ๋กœ ์ƒ‰๊น”์„ ํ• ๋‹นํ•œ๋‹ค. ๊ทธ๋Ÿฐ๋ฐ ๊ฐ€๋” ์ด ์ƒ‰๊น”์ด ์„œ๋กœ ๊ฒน์ณ์„œ ๋ถˆํŽธํ•  ๋•Œ๊ฐ€ ์žˆ๋‹ค.. ์•ˆํƒ€๊น๊ฒŒ๋„ Tensorboard์—์„œ ์ฐจํŠธ ์ƒ‰๊น” ๋ณ€๊ฒฝ์€ ์ง€์›ํ•˜์ง€ ์•Š๊ณ  ์žˆ๋‹ค๊ณ  ํ•œ๋‹ค. ๋Œ€์‹  ์™„์„ฑ๋œ ์ฐจํŠธ๋ฅผ ๊ทธ๋ฆผํŒŒ์ผ๋กœ ์ €์žฅํ•˜๊ฑฐ๋‚˜ ํ•  ๋•Œ ์ƒ‰๊น”์„ ๋ฐ”๊พธ๋Š” ๋ฐฉ๋ฒ•์„ ์†Œ๊ฐœํ•˜๊ณ ์ž ํ•œ๋‹ค. 1. ๋‹ค์šด๋กœ๋“œ ๋ฒ„ํŠผ ํ™œ์„ฑํ™” Tensorboard ํ™”๋ฉด์˜ ์ขŒ์ธก์ƒ๋‹จ์— Show data downloads links๋ฅผ ๋ˆŒ๋Ÿฌ ๋ฐ์ดํ„ฐ ๋‹ค์šด๋กœ๋“œ ๋ฒ„ํŠผ์„ ํ™œ์„ฑํ™”ํ•œ๋‹ค. 2. ๋ฐ์ดํ„ฐ ๋‹ค์šด๋กœ๋“œ ์‚ฌ์šฉํ•  ์ฐจํŠธ์˜ ์ขŒ์ธก ํ•˜๋‹จ ๋‹ค์šด๋กœ๋“œ ๋ฒ„ํŠผ์„ ๋ˆŒ๋Ÿฌ .svg ํŒŒ์ผ์„ ๋‹ค์šด๋กœ๋“œํ•œ๋‹ค. ๋‹ค์šด๋กœ๋“œ ๋ฐ›์€ ํŒŒ์ผ์€ ๋ธŒ๋ผ์šฐ์ €์—์„œ ์—ด ์ˆ˜ ์žˆ๋‹ค. 3. ์ƒ‰๊น” ์ˆ˜์ • .svg ํŒŒ์ผ์„ text editor๋กœ ์—ด์–ด ๊ฐ’์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ๋‹ค. ๋ณด๊ธฐ ํž˜๋“ค๊ฒŒ ๋˜์–ด์žˆ๊ธด ํ•œ๋ฐ.. ๊ฐ ..

[Tensorflow] pretrained BERT๋ฅผ ์ด์šฉํ•œ text classification

Tensorflow tutorial-Classify text with BERT๋ฅผ ๋ฒˆ์—ญ ๋ฐ ์ •๋ฆฌํ•œ ๊ธ€. BERT๋ฅผ ์ด์šฉํ•œ ์˜ˆ์ œ๊ฐ€ ๋Œ€๋ถ€๋ถ„ Huggingface๋ฅผ ์ด์šฉํ•œ ๊ฒƒ์ธ๋ฐ, BERT๋ฅผ ๊ณต๋ถ€ํ•˜๊ธฐ์—๋Š” Huggingface๋ฅผ ์“ฐ์ง€ ์•Š๊ณ  Tensorflow๋‚˜ PyTorch๋ฅผ ์ด์šฉํ•œ ์ฝ”๋“œ๊ฐ€ ๋” ๋‚˜์„ ๊ฑฐ๋ผ๊ณ  ์ƒ๊ฐํ•ด ์ฐพ๋‹ค๊ฐ€ ๋ฐœ๊ฒฌํ–ˆ๋‹ค. ์›๊ธ€ ๋งํฌ: (ํ•œ๊ตญ์–ด ๋ฒˆ์—ญ์ด ์ž˜ ์•ˆ๋˜์–ด ์žˆ์Œ) BERT๋กœ ํ…์ŠคํŠธ ๋ถ„๋ฅ˜ | Text | TensorFlow ์ด ํŽ˜์ด์ง€๋Š” Cloud Translation API๋ฅผ ํ†ตํ•ด ๋ฒˆ์—ญ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. Switch to English BERT๋กœ ํ…์ŠคํŠธ ๋ถ„๋ฅ˜ ์ปฌ๋ ‰์…˜์„ ์‚ฌ์šฉํ•ด ์ •๋ฆฌํ•˜๊ธฐ ๋‚ด ํ™˜๊ฒฝ์„ค์ •์„ ๊ธฐ์ค€์œผ๋กœ ์ฝ˜ํ…์ธ ๋ฅผ ์ €์žฅํ•˜๊ณ  ๋ถ„๋ฅ˜ํ•˜์„ธ์š”. ์ด ํŠœํ† ๋ฆฌ์–ผ์—๋Š” ์ผ๋ฐ˜ www.tensorflow.org Setup prep..

๋ฐ˜์‘ํ˜•