如何解决为什么pytorch中的正则化和暂存代码不匹配?pytorch中用于正则化的公式是什么? case1类型1正则化: case2类型2正则化:
我一直在尝试在PyTorch中的二进制分类模型上进行L2正则化,但是当我将PyTorch的结果与暂存代码匹配时,它不匹配, pytorch代码:
select * from insert_values('table1','table2','table3');
临时代码:
class LogisticRegression(nn.Module):
def __init__(self,n_input_features):
super(LogisticRegression,self).__init__()
self.linear=nn.Linear(4,1)
self.linear.weight.data.fill_(0.0)
self.linear.bias.data.fill_(0.0)
def forward(self,x):
y_predicted=torch.sigmoid(self.linear(x))
return y_predicted
model=LogisticRegression(4)
criterion=nn.bceloss()
optimizer=torch.optim.SGD(model.parameters(),lr=0.05,weight_decay=0.1)
dataset=Data()
train_data=DataLoader(dataset=dataset,batch_size=1096,shuffle=False)
num_epochs=1000
for epoch in range(num_epochs):
for x,y in train_data:
y_pred=model(x)
loss=criterion(y_pred,y)
loss.backward()
optimizer.step()
optimizer.zero_grad()
def sigmoid(z):
s = 1/(1+ np.exp(-z))
return s
def yinfer(X,beta):
return sigmoid(beta[0] + np.dot(X,beta[1:]))
def cost(X,Y,beta,lam):
sum = 0
sum1 = 0
n = len(beta)
m = len(Y)
for i in range(m):
sum = sum + Y[i]*(np.log( yinfer(X[i],beta)))+ (1 -Y[i])*np.log(1-yinfer(X[i],beta))
for i in range(0,n):
sum1 = sum1 + beta[i]**2
return (-sum + (lam/2) * sum1)/(1.0*m)
def pred(X,beta):
if ( yinfer(X,beta) > 0.5):
ypred = 1
else :
ypred = 0
return ypred
有人可以告诉我为什么会这样吗? L2值= 0.1
解决方法
很好的问题。我仔细阅读了 PyTorch 文档,并找到了答案。答案非常棘手。基本上,有两种方式来计算正规化。 (有关夏季的内容,请跳到最后一部分)。
PyTorch 使用第一类型(正则化因子不除以批处理大小)。
下面是一个示例代码,演示了这一点:
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import torch.optim as optim
class model(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(1,1)
self.linear.weight.data.fill_(1.0)
self.linear.bias.data.fill_(1.0)
def forward(self,x):
return self.linear(x)
model = model()
optimizer = optim.SGD(model.parameters(),lr=0.1,weight_decay=1.0)
input = torch.tensor([[2],[4]],dtype=torch.float32)
target = torch.tensor([[7],[11]],dtype=torch.float32)
optimizer.zero_grad()
pred = model(input)
loss = F.mse_loss(pred,target)
print(f'input: {input[0].data,input[1].data}')
print(f'prediction: {pred[0].data,pred[1].data}')
print(f'target: {target[0].data,target[1].data}')
print(f'\nMSEloss: {loss.item()}\n')
loss.backward()
print('Before updation:')
print('--------------------------------------------------------------------------')
print(f'weight [data,gradient]: {model.linear.weight.data,model.linear.weight.grad}')
print(f'bias [data,gradient]: {model.linear.bias.data,model.linear.bias.grad}')
print('--------------------------------------------------------------------------')
optimizer.step()
print('After updation:')
print('--------------------------------------------------------------------------')
print(f'weight [data]: {model.linear.weight.data}')
print(f'bias [data]: {model.linear.bias.data}')
print('--------------------------------------------------------------------------')
输出的:
input: (tensor([2.]),tensor([4.]))
prediction: (tensor([3.]),tensor([5.]))
target: (tensor([7.]),tensor([11.]))
MSEloss: 26.0
Before updation:
--------------------------------------------------------------------------
weight [data,gradient]: (tensor([[1.]]),tensor([[-32.]]))
bias [data,gradient]: (tensor([1.]),tensor([-10.]))
--------------------------------------------------------------------------
After updation:
--------------------------------------------------------------------------
weight [data]: tensor([[4.1000]])
bias [data]: tensor([1.9000])
--------------------------------------------------------------------------
这里 m =批次大小= 2,lr = alpha = 0.1,lambda = weight_decay = 1 。
现在考虑张量 weight ,该张量具有值= 1和grad = -32
case1(类型1正则化):
weight = weight - lr(grad + weight_decay.weight)
weight = 1 - 0.1(-32 + 1(1))
weight = 4.1
case2(类型2正则化):
weight = weight - lr(grad + (weight_decay/batch size).weight)
weight = 1 - 0.1(-32 + (1/2)(1))
weight = 4.15
从输出中,我们可以看到更新后的 weight = 4.1000 。得出结论, PyTorch 使用 type1 正则化。
最后,在您的代码中,您将遵循 type2 正则化。因此,只需将最后几行更改为此:
# for k in range(0,len(beta)):
# temp_beta[k] = temp_beta[k] + lam * beta[k] #regularization here
temp_beta= temp_beta / (1.0*n)
beta = beta - alpha*(temp_beta + lam * beta)
PyTorch的损失函数不不包含正则化术语(在 optimizers 内部实现),因此也删除了正则化您的自定义 cost 函数中的字词。
总结:
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。