我正在玩ANN,这是Udactity DeepLearning课程的一部分.
我成功建立和训练网络,并对所有权重和偏差引入了L2正则化.现在我正在尝试隐藏层的退出,以改善泛化.我想知道,将L2正则化引入同一层的隐藏层和辍学是否有意义？如果是这样,怎么办？
在辍学期间,我们关闭隐藏层的一半激活,并将其余神经元输出的数量翻倍.在使用L2时,我们计算所有隐藏权重的L2范数.但是如果我们使用dropout,我不知道如何计算L2.我们关闭一些激活,现在我们不应该从L2计算中删除“不用”的权重？任何关于此事的参考将是有用的,我还没有找到任何信息.
为了防止你有兴趣,我的代码为ANN与L2正则化在下面：
#for NeuralNetwork model code is below #We will use SGD for training to save our time. Code is from Assignment 2 #beta is the new parameter - controls level of regularization. Default is 0.01 #but feel free to play with it #notice,we introduce L2 for both biases and weights of all layers beta = 0.01 #building tensorflow graph graph = tf.Graph() with graph.as_default(): # Input data. For the training data,we use a placeholder that will be fed # at run time with a training minibatch. tf_train_dataset = tf.placeholder(tf.float32,shape=(batch_size,image_size * image_size)) tf_train_labels = tf.placeholder(tf.float32,num_labels)) tf_valid_dataset = tf.constant(valid_dataset) tf_test_dataset = tf.constant(test_dataset) #Now let's build our new hidden layer #that's how many hidden neurons we want num_hidden_neurons = 1024 #its weights hidden_weights = tf.Variable( tf.truncated_normal([image_size * image_size,num_hidden_neurons])) hidden_biases = tf.Variable(tf.zeros([num_hidden_neurons])) #Now the layer itself. It multiplies data by weights,adds biases #and takes ReLU over result hidden_layer = tf.nn.relu(tf.matmul(tf_train_dataset,hidden_weights) + hidden_biases) #time to go for output linear layer #out weights connect hidden neurons to output labels #biases are added to output labels out_weights = tf.Variable( tf.truncated_normal([num_hidden_neurons,num_labels])) out_biases = tf.Variable(tf.zeros([num_labels])) #compute output out_layer = tf.matmul(hidden_layer,out_weights) + out_biases #our real output is a softmax of prior result #and we also compute its cross-entropy to get our loss #Notice - we introduce our L2 here loss = (tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( out_layer,tf_train_labels) + beta*tf.nn.l2_loss(hidden_weights) + beta*tf.nn.l2_loss(hidden_biases) + beta*tf.nn.l2_loss(out_weights) + beta*tf.nn.l2_loss(out_biases))) #Now we just minimize this loss to actually train the network optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss) #nice,Now let's calculate the predictions on each dataset for evaluating the #performance so far # Predictions for the training,validation,and test data. train_prediction = tf.nn.softmax(out_layer) valid_relu = tf.nn.relu( tf.matmul(tf_valid_dataset,hidden_weights) + hidden_biases) valid_prediction = tf.nn.softmax( tf.matmul(valid_relu,out_weights) + out_biases) test_relu = tf.nn.relu( tf.matmul( tf_test_dataset,hidden_weights) + hidden_biases) test_prediction = tf.nn.softmax(tf.matmul(test_relu,out_weights) + out_biases) #Now is the actual training on the ANN we built #we will run it for some number of steps and evaluate the progress after #every 500 steps #number of steps we will train our ANN num_steps = 3001 #actual training with tf.Session(graph=graph) as session: tf.initialize_all_variables().run() print("Initialized") for step in range(num_steps): # Pick an offset within the training data,which has been randomized. # Note: we Could use better randomization across epochs. offset = (step * batch_size) % (train_labels.shape[0] - batch_size) # Generate a minibatch. batch_data = train_dataset[offset:(offset + batch_size),:] batch_labels = train_labels[offset:(offset + batch_size),:] # Prepare a dictionary telling the session where to Feed the minibatch. # The key of the dictionary is the placeholder node of the graph to be fed,# and the value is the numpy array to Feed to it. Feed_dict = {tf_train_dataset : batch_data,tf_train_labels : batch_labels} _,l,predictions = session.run( [optimizer,loss,train_prediction],Feed_dict=Feed_dict) if (step % 500 == 0): print("Minibatch loss at step %d: %f" % (step,l)) print("Minibatch accuracy: %.1f%%" % accuracy(predictions,batch_labels)) print("Validation accuracy: %.1f%%" % accuracy( valid_prediction.eval(),valid_labels)) print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(),test_labels))
好的,经过一些额外的努力,我设法解决它,并将L2和辍学引入我的网络,代码如下.在同一个网络中,我没有辍学(L2已经到位)略有改善.我仍然不确定是否真的很值得介绍他们两个,L2和辍学的努力,但至少它的作品,并略微提高了结果.
#ANN with introduced dropout #This time we still use the L2 but restrict training dataset #to be extremely small #get just first 500 of examples,so that our ANN can memorize whole dataset train_dataset_2 = train_dataset[:500,:] train_labels_2 = train_labels[:500] #batch size for SGD and beta parameter for L2 loss batch_size = 128 beta = 0.001 #that's how many hidden neurons we want num_hidden_neurons = 1024 #building tensorflow graph graph = tf.Graph() with graph.as_default(): # Input data. For the training data,num_labels)) tf_valid_dataset = tf.constant(valid_dataset) tf_test_dataset = tf.constant(test_dataset) #Now let's build our new hidden layer #its weights hidden_weights = tf.Variable( tf.truncated_normal([image_size * image_size,hidden_weights) + hidden_biases) #add dropout on hidden layer #we pick up the probabylity of switching off the activation #and perform the switch off of the activations keep_prob = tf.placeholder("float") hidden_layer_drop = tf.nn.dropout(hidden_layer,keep_prob) #time to go for output linear layer #out weights connect hidden neurons to output labels #biases are added to output labels out_weights = tf.Variable( tf.truncated_normal([num_hidden_neurons,num_labels])) out_biases = tf.Variable(tf.zeros([num_labels])) #compute output #notice that upon training we use the switched off activations #i.e. the variaction of hidden_layer with the dropout active out_layer = tf.matmul(hidden_layer_drop,which has been randomized. # Note: we Could use better randomization across epochs. offset = (step * batch_size) % (train_labels_2.shape[0] - batch_size) # Generate a minibatch. batch_data = train_dataset_2[offset:(offset + batch_size),:] batch_labels = train_labels_2[offset:(offset + batch_size),tf_train_labels : batch_labels,keep_prob : 0.5} _,test_labels))
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
