Pytorch模型不学习身份功能？

如何解决Pytorch模型不学习身份功能？

我用pytorch编写了一些模型，即使经过很多时期后仍然无法学习任何东西。为了调试该问题，我制作了一个简单的模型来对输入的标识函数进行建模。困难在于，即使训练了5万个历元，该模型也不会学到任何东西，

import torch
import torch.nn as nn

torch.manual_seed(1)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.input = nn.Linear(2,4)
        self.hidden = nn.Linear(4,4)
        self.output = nn.Linear(4,2)
        self.relu = nn.ReLU()
        self.softmax = nn.softmax(dim=1)
        self.dropout = nn.Dropout(0.5)
    def forward(self,x):
        x = self.input(x)
        x = self.dropout(x)
        x = self.relu(x)
        x = self.hidden(x)
        x = self.dropout(x)
        x = self.relu(x)
        x = self.output(x)
        x = self.softmax(x)
        return x


X = torch.tensor([[1,0],[1,[0,1],1]],dtype=torch.float)

net = Net()

criterion = nn.CrossEntropyLoss()

opt = torch.optim.Adam(net.parameters(),lr=0.001)


for i in range(100000):
    opt.zero_grad()
    y = net(X)
    loss = criterion(y,torch.argmax(X,dim=1))
    loss.backward()
    if i%500 ==0:
        print("Epoch: ",i)
        print(torch.argmax(y,dim=1).detach().numpy().tolist())
        print("Loss: ",loss.item())
        print()

输出

Epoch:  52500
[0,1,0]
Loss:  0.6554909944534302

Epoch:  53000
[0,0]
Loss:  0.7004914283752441

Epoch:  53500
[0,0]
Loss:  0.7156486511230469

Epoch:  54000
[0,0]
Loss:  0.7171240448951721

Epoch:  54500
[0,0]
Loss:  0.691678524017334

Epoch:  55000
[0,0]
Loss:  0.7301554679870605

Epoch:  55500
[0,0]
Loss:  0.728650689125061

我的实现有什么问题？

解决方法

有一些错误：

1. 缺少optimizer.step() ：

optimizer.step()基于反向传播的梯度和其他累积动量以及所有这些来更新参数。

1. softmax损失CrossEntropy的使用：

Pytorch CrossEntropyLoss准则将nn.LogSoftmax()和nn.NLLLoss()合并为一个类。即它应用softmax然后取负对数。因此，在您采用 softmax（softmax（output））的情况下。解决方案是在训练时仅使用linear输出层，并使用softmax层进行预测。

1. 小型网络的辍学价值高：

这会导致拟合不足。

这是更正的代码：

import torch
import torch.nn as nn

torch.manual_seed(1)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.input = nn.Linear(2,4)
        self.hidden = nn.Linear(4,4)
        self.output = nn.Linear(4,2)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        # self.dropout = nn.Dropout(0.0)
    def forward(self,x):
        x = self.input(x)
        # x = self.dropout(x)
        x = self.relu(x)
        x = self.hidden(x)
        # x = self.dropout(x)
        x = self.relu(x)
        x = self.output(x)
        # x = self.softmax(x)
        return x

    def predict(self,x):
        with torch.no_grad():
            out = self.forward(x)
        return self.softmax(out)


X = torch.tensor([[1,0],[1,[0,1],1]],dtype=torch.float)

net = Net()

criterion = nn.CrossEntropyLoss()

opt = torch.optim.Adam(net.parameters(),lr=0.001)


for i in range(100000):
    opt.zero_grad()
    y = net(X)
    loss = criterion(y,torch.argmax(X,dim=1))
    loss.backward()
    # This was missing before
    opt.step()
    if i%500 ==0:
        print("Epoch: ",i)
        pred = net.predict(X)
        print(f'prediction: {torch.argmax(pred,dim=1).detach().numpy().tolist()},actual: {torch.argmax(X,dim=1)}')
        print("Loss: ",loss.item())

输出：

Epoch:  0
prediction: [0,actual: tensor([0,1,1])
Loss:  0.7042869329452515
Epoch:  500
prediction: [0,1])
Loss:  0.1166711300611496
Epoch:  1000
prediction: [0,1])
Loss:  0.05215628445148468
Epoch:  1500
prediction: [0,1])
Loss:  0.02993333339691162
Epoch:  2000
prediction: [0,1])
Loss:  0.01916157826781273
Epoch:  2500
prediction: [0,1])
Loss:  0.01306679006665945
Epoch:  3000
prediction: [0,1])
Loss:  0.009280549362301826
.
.
.