如何解决Keras中的自定义损失功能不起作用
我正在使用Keras实现PPO算法,但是在Keras中遇到了自定义损失函数的一些问题。
Tensorflow版本:2.3.0
Keras版本:2.4.3
def ppo_loss(old_prediction,advantage,reward,value):
def loss(y_true,y_pred):
# print(type(y_true),type(y_pred),type(old_prediction),type(advantage),type(reward),type(value))
newpolicy_probs = y_pred
ratio = K.exp(K.log(newpolicy_probs + 1e-10) - K.log(old_prediction + 1e-10))
clip_ratio = K.clip(ratio,min_value=1 - EPSILON,max_value=1 + EPSILON)
surrogate1 = ratio * advantage
surrogate2 = clip_ratio * advantage
actor_loss = -K.mean(K.minimum(surrogate1,surrogate2))
critic_loss = K.mean(K.square(reward - value))
entropy_loss = K.mean(-(newpolicy_probs * K.log(newpolicy_probs + 1e-10)))
total_loss = CRITIC_disCOUNT * critic_loss + actor_loss - BETA * entropy_loss
return total_loss
return loss
这是actor_model:
def actor_model(input_dims,output_dims):
state = Input(shape=(input_dims,),name='state_input')
old_prediction = Input(shape=(output_dims,name='old_prediction_input')
advantage = Input(shape=(1,name='advantage_input')
reward = Input(shape=(1,name='reward_input')
value = Input(shape=(1,name='value_input')
x = Dense(HIDDEN_UNITS,activation='tanh',name='fc1')(state)
x = Dense(HIDDEN_UNITS,activation='tanh')(x)
policy = Dense(output_dims,name='policy')(x)
actor_network = Model(inputs=[state,old_prediction,value],outputs=[policy])
actor_network.compile(optimizer=Adam(lr=LEARNING_RATE),loss=ppo_loss(
old_prediction=old_prediction,advantage=advantage,reward=reward,value=value),# run_eagerly=True
)
# actor_network.summary()
return actor_network
我收到此错误:
Traceback (most recent call last):
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py",line 59,in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle,device_name,op_name,TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example,the following function will fail:
@tf.function
def has_init_scope():
my_constant = tf.constant(1.)
with tf.init_scope():
added = my_constant * 2
The graph tensor has name: old_prediction_input:0
During handling of the above exception,another exception occurred:
Traceback (most recent call last):
File "train.py",line 426,in <module>
agent.train()
File "train.py",line 370,in train
actor_loss = self.actor.fit(
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py",line 108,in _method_wrapper
return method(self,*args,**kwargs)
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py",line 1098,in fit
tmp_logs = train_function(iterator)
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py",line 780,in __call__
result = self._call(*args,**kwds)
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py",line 840,in _call
return self._stateless_fn(*args,**kwds)
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 2829,in __call__
return graph_function._filtered_call(args,kwargs) # pylint: disable=protected-access
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 1843,in _filtered_call
return self._call_flat(
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 1923,in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 545,in call
outputs = execute.execute(
File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py",line 72,in quick_execute
raise core._SymbolicException(
tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors,but found [<tf.Tensor 'old_prediction_input:0' shape=(None,2) dtype=float32>,<tf.Tensor 'advantage_input:0' shape=(None,1) dtype=float32>,<tf.Tensor 'reward_input:0' shape=(None,<tf.Tensor 'value_input:0' shape=(None,1) dtype=float32>]
因此,在互联网上搜索后,我找到了一种解决方案,即将run_eagerly=True
添加到model.compile()方法中,如下所示:
actor_model.compile(...,run_eagerly=True)
但是添加了这个之后,我从actor_model.history['loss']
得到了0损失值
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。