What is the PyTorch implementation of a common optimizer
This article will explain in detail how the PyTorch implementation of the common optimizer is, and the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.
Here we mainly talk about the implementation of different common optimizer code and make a simple comparison on a small data set.
Among them, SGD and SGDM, as well as Adam is the optimizer that comes with pytorch, while RAdam is a recently proposed optimizer that is more powerful than Adam, but in general, real bosses are still using SGDM as optimizer.
Import the necessary libraries:
Import torchimport torch.nn as nnimport torch.nn.functional as Fimport torch.optim as optimimport matplotlib.pyplot as pltimport torch.utils.data as Datafrom torch.optim.optimizer import Optimizerimport math
Main program part:
LR = 0.01BATCH_SIZE = 32EPOCH = 12
# fake datasetx = torch.unsqueeze (torch.linspace (- 1,1,300), dim=1) y = x.pow (2) + 0.1 * torch.normal (torch.zeros (* x.size ()
Torch_dataset = Data.TensorDataset (x, y) loader = Data.DataLoader (dataset=torch_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2)
Class Net (nn.Module): def _ init__ (self): super (Net, self). _ _ init__ () self.hidden = nn.Linear (1,20) self.prediction = nn.Linear (20,1)
Def forward (self, x): X = F.relu (self.hidden (x)) x = self.prediction (x) return x
Def main () net_SGD = Net () net_Momentum = Net () net_Adam = Net () net_RAdam = Net () nets = [net_SGD, net_Momentum, net_Adam, net_RAdam] opt_SGD = optim.SGD (net_SGD.parameters (), lr=LR) opt_Momentum = optim.SGD (net_Momentum.parameters (), lr=LR, momentum=0.9) opt_Adam = optim.Adam (net_Adam.parameters (), lr=LR, betas= ) opt_RAdam = RAdam (net_RAdam.parameters (), lr=LR,weight_decay=0) optimizers = [opt_SGD, opt_Momentum, opt_Adam, opt_RAdam] loss_func = nn.MSELoss () losses_his = [[], [] # training for epoch in range (EPOCH): print ('EPOCH:', epoch) for step, (batch_x) Batch_y) in enumerate (loader): Backx = batch_x Backy = batch_y for net, opt, l_his in zip (nets, optimizers, losses_his): out = net (BFLX) loss = loss_func (out) Bachely) opt.zero_grad () loss.backward () opt.step () l_his.append (loss.item ()) labels = ['SGD',' Momentum', 'Adam','RAdam'] for I, l_his in enumerate (losses_his): plt.plot (l_his Label=labels [I]) plt.legend (loc='best') plt.xlabel ('Steps') plt.ylabel (' Loss') plt.ylim ((0,0.2)) plt.show ()
If _ _ name__ = ='_ _ main__': main ()
The following figure is a comparison of the optimizer:
It can be seen that the effect of Adam is very good. Then SGDM second, SGDM is often used by bosses, so although the effect of SGDM does not seem to be as good as Adam, it is still recommended to try the effect of SGDM in the project.
On the common optimizer PyTorch implementation is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.
