如何解决此实例的状态已从外部设置为无服用毒丸
我们在 kubernetes 上运行气流。虽然气流运行 DAG 没有任何问题,但我在气流日志中看到以下奇怪的错误消息。
但是,当我查看运行 DAG 的 kubernetes pod 日志时,它没有任何问题,并且 DAG 中的所有任务都已成功完成。我在气流日志中收到以下错误消息的任何原因?此外,所有日志都没有打印在气流日志中,尽管所有内容都在 kubernetes pod 日志中打印而没有任何错误消息。
气流日志正在打印消息,直到 python 代码中的一个步骤花费超过 1 小时......这就是气流日志停止写入文件的时间。不确定这是否与运行超过特定时间时的任务超时有关
如果您对此有任何想法,请告诉我。谢谢
[2021-04-08 23:36:33,929] - 2021-04-08 23:36:33,929 INFO - b'orchestrator - PrevIoUs append blob deleted\n'
[2021-04-08 23:36:33,930] - 2021-04-08 23:36:33,930 INFO - b'orchestrator - Starting data recipe using recipes.recipe-journal-lines-sap\n'
[2021-04-09 01:50:58,705] - 2021-04-09 01:50:58,705 WARNING - State of this instance has been externally set to None. Taking the poison pill.
[2021-04-09 01:50:58,710] - Sending Signals.SIGTERM to GPID 12851
[2021-04-09 01:50:58,710] - Received SIGTERM. Terminating subprocesses.
[2021-04-09 01:50:59,399] - Pod Launching Failed: Task received SIGTERM signal
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py",line 251,in execute
get_logs=self.get_logs)
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",line 117,in run_pod
return self._monitor_pod(pod,get_logs)
File "/usr/local/lib/python3.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py",line 124,in _monitor_pod
for line in logs:
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py",line 808,in __iter__
for chunk in self.stream(decode_content=True):
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py",line 572,in stream
for line in self.read_chunked(amt,decode_content=decode_content):
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py",line 764,in read_chunked
self._update_chunk_length()
File "/usr/local/lib/python3.7/site-packages/urllib3/response.py",line 694,in _update_chunk_length
line = self._fp.fp.readline()
File "/usr/local/lib/python3.7/socket.py",line 589,in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.7/ssl.py",line 1071,in recv_into
return self.read(nbytes,buffer)
File "/usr/local/lib/python3.7/ssl.py",line 929,in read
return self._sslobj.read(len,buffer)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py",line 943,in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。