๋ฐ˜์‘ํ˜•

๐Ÿ Python & library/PyTorch 18

[PyTorch] Autograd ์ž‘๋™๋ฐฉ์‹ ์•Œ์•„๋ณด๊ธฐ

์œ„ ๋™์˜์ƒ์—์„œ PyTorch Autograd๋ฅผ ์ดํ•ดํ•˜๊ธฐ ์‰ฝ๊ฒŒ ์„ค๋ช…ํ•ด์ฃผ๊ณ  ์žˆ๋‹ค. ๋‹ค์Œ์€ ์œ„ ๋™์˜์ƒ์„ ๊ฐ„๋‹จํžˆ ์ •๋ฆฌํ•œ ๊ธ€์ด๋‹ค. 1. torch.Tensor ๊ฐ tensor์€ ๋‹ค์Œ์˜ attr์„ ๊ฐ–๋Š”๋‹ค data: tensor์˜ ๊ฐ’ grad: tensor์˜ gradient ๊ฐ’. is_leaf์ธ ๊ฒฝ์šฐ์—๋งŒ gradient๊ฐ€ ์ž๋™์œผ๋กœ ์ €์žฅ๋œ๋‹ค. grad_fn: gradient function. ํ•ด๋‹น tensor๊ฐ€ ์–ด๋–ค ์—ฐ์‚ฐ์„ ํ†ตํ•ด forward๋˜์—ˆ๋Š”์ง€์— ๋”ฐ๋ผ ๊ฒฐ์ •๋œ๋‹ค. ex) a * b = c ์ธ ๊ฒฝ์šฐ c์˜ grad_fn์€ MulBackward์ด๋‹ค. is_leaf์ธ ๊ฒฝ์šฐ None is_leaf: (backward ๊ธฐ์ค€) ๊ฐ€์žฅ ๋งˆ์ง€๋ง‰ tensor์ธ์ง€ requires_grad: ๊ณ„์‚ฐ ๊ทธ๋ž˜ํ”„์˜ ์ผ๋ถ€๋กœ ๋“ค์–ด๊ฐˆ ๊ฒƒ์ธ์ง€ 2. gr..

PyTorch 2.0์—์„œ ๋‹ฌ๋ผ์ง€๋Š” ์  - torch.compile

PyTorch 2.0 Overview pytorch.org PyTorch 2.0์€ 22๋…„ 12์›” PyTorch Conference์—์„œ ๋ฐœํ‘œ๋˜์—ˆ๊ณ , 23๋…„ 3์›” ์ •์‹ ๋ฆด๋ฆฌ์ฆˆ ๋˜์—ˆ๋‹ค. ์ด์ „์˜ PyTorch 1.x ๋ฒ„์ „๋“ค๋ณด๋‹ค ๋น ๋ฅด๊ณ , Pythonicํ•˜๊ณ  Dynamicํ•˜๋‹ค๊ณ  ํ•œ๋‹ค. ์–ด๋–ค ์ ๋“ค์ด ๋‹ฌ๋ผ์กŒ์„์ง€ ํ•œ๋ฒˆ ์•Œ์•„๋ด…์‹œ๋‹ค. torch.compile torch.compile์€ PyTorch 2.0์˜ ๋ฉ”์ธ API์ด๋‹ค. ๋ชจ๋ธ์„ ๋ฏธ๋ฆฌ ์ปดํŒŒ์ผํ•˜์—ฌ ์†๋„๋ฅผ ๋†’์ด๋Š” ๊ธฐ์ˆ ์ด๋‹ค. torch.compile์€ TorchDynamo, AOTAutograd, PrimTorch, TorchInductor ๋„ค ๊ฐ€์ง€์˜ ์ƒˆ๋กœ์šด ๊ธฐ์ˆ ์„ ๊ธฐ๋ฐ˜์œผ๋กœ ๋งŒ๋“ค์–ด์กŒ๋‹ค. ๊ฐ ๊ธฐ์ˆ ์— ๋Œ€ํ•œ ์ž์„ธํ•œ ์„ค๋ช…์€ ์—ฌ๊ธฐ์—์„œ ์ฐพ์•„๋ณผ ์ˆ˜ ์žˆ๋‹ค. ์‚ฌ์šฉ๋ฒ• torch.compile์€..

[PyTorch] tensor.detach()์˜ ๊ธฐ๋Šฅ๊ณผ ์˜ˆ์‹œ ์ฝ”๋“œ

PyTorch tensor์— ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” detach() method๋Š” gradient์˜ ์ „ํŒŒ๋ฅผ ๋ฉˆ์ถ”๋Š” ์—ญํ• ์„ ํ•œ๋‹ค. https://pytorch.org/docs/stable/generated/torch.Tensor.detach.html torch.Tensor.detach — PyTorch 1.13 documentation Shortcuts pytorch.org ๋‹ค์Œ ์˜ˆ์‹œ๋ฅผ ํ†ตํ•ด ์‰ฝ๊ฒŒ ์ดํ•ดํ•  ์ˆ˜ ์žˆ๋‹ค. import torch import torch.nn as nn class TestModel(nn.Module): def __init__(self): super().__init__() self.layer1 = nn.Linear(10, 10) self.layer2 = nn.Linear(10, 10) def..

[PyTorch] nn.Embedding ์ดˆ๊ธฐํ™”ํ•˜๊ธฐ (initialization)

PyTorch์˜ nn.Embedding layer์„ ์ดˆ๊ธฐํ™”ํ•˜๋Š” ๋ฐฉ๋ฒ•์—๋Š” ๋‘ ๊ฐ€์ง€๊ฐ€ ์žˆ๋‹ค. embedding = nn.Embedding(num_embeddings, embedding_dim) 1. torch.tensor์˜ ๋‚ด์žฅ method ์ด์šฉํ•˜๊ธฐ embedding.weight.data.uniform_(-1, 1) torch.tensor์€ uniform_ ๋“ฑ์˜ ๋‚ด์žฅ method๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ์–ด ์ด๋ฅผ ํ†ตํ•ด ๊ฐ’์„ ์ดˆ๊ธฐํ™”ํ•  ์ˆ˜ ์žˆ๋‹ค. 2. torch.nn.init ์ด์šฉํ•˜๊ธฐ nn.init.uniform_(embedding.weight, -1.0, 1.0) torch.nn.init์˜ method๋“ค์„ ์ด์šฉํ•  ์ˆ˜๋„ ์žˆ๋‹ค. ์ด ๋ฐฉ๋ฒ•์„ ์ด์šฉํ•˜๋ฉด uniform_ ์ด์™ธ์—๋„ xavier_uniform_ ๋“ฑ ๋ณด๋‹ค ๋‹ค์–‘ํ•œ initi..

Numpy & PyTorch๋กœ 2D fourier transform, inverse fourier transformํ•˜๊ธฐ

์‚ฌ์šฉํ•  ์ด๋ฏธ์ง€ Load image import cv2 import matplotlib.pyplot as plt img = cv2.imread('dog.jpg', cv2.IMREAD_GRAYSCALE) img = cv2.resize(img, (256, 256)) print(img.shape) plt.imshow(img, cmap='gray') plt.colorbar() (256, 256) FFT and IFFT using numpy import numpy as np fft_img = np.fft.fftshift(np.fft.fft2(img, norm='ortho')) print(fft_img.shape, fft_img.dtype) plt.imshow(np.abs(fft_img), cmap='gray') p..

[PyTorch] make_grid๋กœ ์—ฌ๋Ÿฌ ๊ฐœ์˜ ์ด๋ฏธ์ง€ ํ•œ๋ฒˆ์— plotํ•˜๊ธฐ

Official Docs: https://pytorch.org/vision/main/generated/torchvision.utils.make_grid.html make_grid — Torchvision main documentation Shortcuts pytorch.org ์—ฌ๋Ÿฌ๊ฐœ์˜ ์ด๋ฏธ์ง€๋ฅผ ํ•ฉ์ณ์„œ ํ•˜๋‚˜์˜ grid๋กœ ๋งŒ๋“ค์–ด์ฃผ๋Š” torchvision.utils.make_grid ํ•จ์ˆ˜๋ฅผ ์†Œ๊ฐœํ•ด ๋ณด๊ณ ์ž ํ•œ๋‹ค. tensor (Tensor or list) : grid๋ฅผ ๋งŒ๋“ค ์ด๋ฏธ์ง€๋“ค. 4D mini-batch Tensor (B x C x H x W) ํ˜น์€ ๋™์ผํ•œ ํฌ๊ธฐ์˜ image๋ฅผ ๋‹ด๊ณ  ์žˆ๋Š” list๋ฅผ ์ค„ ์ˆ˜ ์žˆ๋‹ค. nrow (int, optional) : grid์˜ ํ–‰ ๊ฐฏ์ˆ˜๋ฅผ ์ง€์ •ํ•ด์ค„ ์ˆ˜ ์žˆ๋‹ค. ์—ด ๊ฐฏ์ˆ˜๋Š” ์ž..

[PyTorch] model weight ๊ฐ’ ์กฐ์ •ํ•˜๊ธฐ / weight normalization

PyTorch model์˜ weight๊ฐ’์„ ๋ฐ”๋กœ ๋ฐ”๊พธ๋ ค๊ณ  ํ•˜๋ฉด FloatTensor์„ parameter๋กœ ์‚ฌ์šฉํ•  ์ˆ˜ ์—†๋‹ค๋Š” ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ•œ๋‹ค. import torch.nn as nn model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 4, bias=False)) model[2].weight = model[2].weight/2. >> cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected) ํ•™์Šต ๋„์ค‘ ์‚ฌ์šฉํ•˜๋Š” weight normalization์€ nn.utils.weight_norm์„ ์‚ฌ์šฉํ•˜๋ฉด ๋˜๋Š” ๊ฒƒ ๊ฐ™์ง€๋งŒ, ๋ฐ”๋กœ weight ๊ฐ’์— ..

[PyTorch] nn.Conv์˜ padding๊ณผ padding_mode

PyTorch ์—์„œ ์ œ๊ณตํ•˜๋Š” convolution ํ•จ์ˆ˜์— ์„ค์ • ๊ฐ€๋Šฅํ•œ parameter ์ค‘padding๊ณผ padding_mode๋ผ๋Š” ๊ฒƒ์ด ์žˆ๋‹ค. padding์˜ ๊ฒฝ์šฐ padding์˜ ํฌ๊ธฐ๋ฅผ ์ง€์ •ํ•  ์ˆ˜ ์žˆ๋Š” parameter์ธ๋ฐ (int ํ˜น์€ tuple), PyTorch 1.9.0๋ถ€ํ„ฐ string์œผ๋กœ ์ง€์ •ํ•  ์ˆ˜ ์žˆ๋Š” ์˜ต์…˜์ด ์ถ”๊ฐ€๋˜์—ˆ๋‹ค. ์ด๋Š” Tensorflow์—์„œ๋Š” ์›๋ž˜ ์žˆ๋˜ ์˜ต์…˜์ธ๋ฐ, padding์˜ ํฌ๊ธฐ๋ฅผ ์ง์ ‘ ์ง€์ •ํ•˜๋Š” ๋Œ€์‹  same ํ˜น์€ valid ์˜ต์…˜์„ ์ฃผ๋ฉด input size์— ๋งž๊ฒŒ ์ž๋™์œผ๋กœ padding ํฌ๊ธฐ๊ฐ€ ์„ค์ •๋œ๋‹ค. valid๋Š” padding์„ ๋”ฐ๋กœ ์ฃผ์ง€ ์•Š๊ณ  input image ์ž์ฒด๋งŒ์„ ์ด์šฉํ•ด convolution ์—ฐ์‚ฐ์„ ์ˆ˜ํ–‰ํ•œ๋‹ค. same์€ output size๊ฐ€ input size์™€..

[PyTorch] Weight clipping

์ด์ „ ๊ธ€์—์„œ PyTorch๋ฅผ ์ด์šฉํ•œ gradient clipping์„ ์†Œ๊ฐœํ–ˆ๋Š”๋ฐ, gradient๊ฐ€ ์•„๋‹ˆ๋ผ weight ๊ฐ’ ์ž์ฒด๋ฅผ ์ผ์ • ๋ฒ”์œ„ ์•ˆ์œผ๋กœ ์ œํ•œํ•ด์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ๋‹ค. ์ด ๊ฒฝ์šฐ Weight clipping์„ ์ˆ˜ํ–‰ํ•ด์ฃผ๋Š” class๋ฅผ ์ •์˜ํ•˜์—ฌ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋‹ค. ์ถœ์ฒ˜: https://discuss.pytorch.org/t/set-constraints-on-parameters-or-layers/23620 Set constraints on parameters or layers Hi, are there any ways in Pytorch to set the range of parameters or values in each layer? For example, is it able to constrain th..

[PyTorch] Enable anomaly detection (torch.autograd.detect_anomaly() / torch.autograd.set_detect_anomaly(True))

๋”ฅ๋Ÿฌ๋‹ ๋ชจ๋ธ์„ ํ•™์Šต์‹œํ‚ค๋˜ ์ค‘ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์—๋Ÿฌ๊ฐ€ ๋ฐœ์ƒํ–ˆ๋‹ค. RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDAComplexFloatType [1, 256, 232]], which is output 0 of SubBackward0, is at version 10; expected version 9 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). ..

๋ฐ˜์‘ํ˜•