如何解决多个 DAG 一次在 Airflow 2.0 上无意运行
我第一次用 postgresql 安装了 Airflow。 但是，在网络上激活 DAG 后，大约会同时创建 16 个 DAG，如下图所示。 enter image description here
作为测试，我制作了一个 dag，每秒一次将当前日期输入数据库 10 秒。如果有意，它应该每秒生成一行，但上面的问题每秒会生成多行。 enter image description here
以下是 DAG 代码。
import pendulum
from airflow.models import DAG
from airflow.utils.dates import days_ago
from airflow.operators.bash import BashOperator
from datetime import datetime,timedelta
kst = pendulum.timezone("Asia/SEOul")
args = {'owner': 'jovyan','start_date': datetime(2021,5,3,16,20,10,tzinfo=kst),'run_as_user':'bskim'}
#args = {'owner': 'jovyan','start_date': days_ago(n=1)}
dag = DAG(dag_id='talend_first_dag',default_args=args,schedule_interval='0/1 * * * *')
talend_command = "/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS_run.sh "
t1 = BashOperator(task_id='talend_first',bash_command=talend_command,dag=dag)
t1
下面是结果“ps-ef |grep 气流”。
(venv) [bskim@swproduct airflow_home]$ ps -ef |grep airflow
bskim 108569 103922 2 17:23 pts/0 00:00:30 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow webserver
bskim 108607 108569 0 17:23 pts/0 00:00:01 gunicorn: master [airflow-webserver]
bskim 109264 103922 2 17:28 pts/0 00:00:25 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109266 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109274 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109275 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109276 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109277 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109278 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109280 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109281 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109282 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109283 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109285 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109286 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109287 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109288 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109290 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109291 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109292 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109293 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109294 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109296 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109297 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109298 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109300 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109301 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109302 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109303 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109304 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109306 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109307 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109308 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109310 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109311 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109312 109264 0 17:28 pts/0 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow scheduler
bskim 109313 109264 0 17:28 pts/0 00:00:02 airflow scheduler -- DagFileProcessorManager
bskim 110825 108607 1 17:36 pts/0 00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim 110921 108607 1 17:36 pts/0 00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim 111012 108607 1 17:37 pts/0 00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim 111109 108607 1 17:37 pts/0 00:00:05 [ready] gunicorn: worker [airflow-webserver]
bskim 112652 109310 0 17:41 pts/0 00:00:00 airflow task supervisor: ['airflow','tasks','run','talend_idx_2_dag','talend_idx_2','2021-05-03T09:26:00+00:00','--local','--pool','default_pool','--subdir','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 112729 109312 0 17:41 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:27:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 112754 112652 1 17:41 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:26:00+00:00 --job-id 448 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpepxvlj4v --error-file /tmp/tmpupd5hmkg
bskim 112781 112729 1 17:41 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:27:00+00:00 --job-id 449 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp2kswizo1 --error-file /tmp/tmpiifxr0x7
bskim 112981 109276 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','talend_first_dag','talend_first','2021-05-03T10:43:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113052 109277 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:44:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113055 112981 2 17:42 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:43:00+00:00 --job-id 450 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmppkeav48n --error-file /tmp/tmp920t73mg
bskim 113065 108607 3 17:42 pts/0 00:00:02 gunicorn: worker [airflow-webserver]
bskim 113090 109274 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:45:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113121 113052 2 17:42 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:44:00+00:00 --job-id 451 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmphhj4r3oh --error-file /tmp/tmpa8ad955w
bskim 113162 109275 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:46:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113168 113090 2 17:42 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:45:00+00:00 --job-id 452 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp_vhz10e0 --error-file /tmp/tmpsgh_rus8
bskim 113199 109278 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:47:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113200 113162 2 17:42 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:46:00+00:00 --job-id 453 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpy3RSSjwn --error-file /tmp/tmp39hnh9o1
bskim 113286 109280 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:28:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113287 109285 0 17:42 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:48:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113288 113199 3 17:42 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:47:00+00:00 --job-id 454 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpo1_l0dct --error-file /tmp/tmpe0ynvaaa
bskim 113299 109281 0 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:29:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113300 109283 0 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:49:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113301 113286 3 17:43 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:28:00+00:00 --job-id 456 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpblx5vzdt --error-file /tmp/tmpbzp5s9st
bskim 113303 113287 3 17:43 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:48:00+00:00 --job-id 455 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpawabzixj --error-file /tmp/tmpr4rau1_t
bskim 113331 109286 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:30:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113332 109282 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:50:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113334 113300 3 17:43 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:49:00+00:00 --job-id 457 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmppn54vxqi --error-file /tmp/tmpc36akon7
bskim 113336 113299 3 17:43 ? 00:00:01 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:29:00+00:00 --job-id 458 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpffxckf4j --error-file /tmp/tmpqnx2vdom
bskim 113347 113332 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:50:00+00:00 --job-id 459 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmphexft182 --error-file /tmp/tmpq0_wfubu
bskim 113348 113331 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:30:00+00:00 --job-id 460 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpxp7m7131 --error-file /tmp/tmp7qpmc7kp
bskim 113351 109288 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:31:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113352 109287 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:51:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113366 113360 6 17:43 ? 00:00:01 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim 113376 109290 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:32:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113381 113352 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:51:00+00:00 --job-id 461 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmpg2lyaeig --error-file /tmp/tmpvpcv2k_d
bskim 113382 113351 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:31:00+00:00 --job-id 462 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp62kc8itw --error-file /tmp/tmpvhebo3kj
bskim 113401 113398 6 17:43 ? 00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim 113449 109293 1 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:33:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113450 109291 2 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:52:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113462 113376 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:32:00+00:00 --job-id 463 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpj3908y63 --error-file /tmp/tmp7e2vqj77
bskim 113468 113464 6 17:43 ? 00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim 113483 109292 2 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:34:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113484 109300 2 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:53:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113488 113449 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:33:00+00:00 --job-id 464 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmp2d7zszd2 --error-file /tmp/tmpxhrfjto_
bskim 113490 113450 3 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:52:00+00:00 --job-id 465 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp05cgdqd1 --error-file /tmp/tmpv8x5_4lc
bskim 113497 113491 5 17:43 ? 00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim 113513 109294 3 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T09:35:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py']
bskim 113514 109296 4 17:43 pts/0 00:00:00 airflow task supervisor: ['airflow','2021-05-03T10:54:00+00:00','/home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py']
bskim 113515 113483 4 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_idx_2_dag talend_idx_2 2021-05-03T09:34:00+00:00 --job-id 466 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_idx_2.py --cfg-path /tmp/tmpmmkwhfic --error-file /tmp/tmpi5wzbio8
bskim 113518 113484 4 17:43 ? 00:00:00 /home/bskim/dev/airflow/workspace/venv/bin/python /home/bskim/dev/airflow/workspace/venv/bin/airflow tasks run talend_first_dag talend_first 2021-05-03T10:53:00+00:00 --job-id 467 --pool default_pool --raw --subdir /home/bskim/dev/airflow/workspace/airflow_home/dags/talend_first.py --cfg-path /tmp/tmp1b9oaxu0 --error-file /tmp/tmpyxqlwzvs
bskim 113532 113529 4 17:43 ? 00:00:00 java -Dtalend.component.manager.m2.repository=/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib -xms256M -Xmx1024M -cp .:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/routines.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-slf4j-impl-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-api-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/log4j-core-2.12.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/postgresql-42.2.9.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/crypto-utils.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/slf4j-api-1.7.25.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/../lib/dom4j-2.1.1.jar:/home/bskim/dev/temp/AIRFLOW_TEST_INSERT_DUMMY_ROWS/AIRFLOW_TEST_INSERT_DUMMY_ROWS/airflow_test_insert_dummy_rows_0_1.jar: test.airflow_test_insert_dummy_rows_0_1.AIRFLOW_TEST_INSERT_DUMMY_ROWS --context=Dev
bskim 113535 103922 0 17:43 pts/0 00:00:00 grep --color=auto airflow
执行器是LocalExecutor，即使设置为SequentialExecutor，结果也是一样的。
我因为这个问题浪费了两天时间。我需要你的帮助！
解决方法
Airflow 的 catchup_by_default 设置有一个默认的 True 值，因此如果您没有更改默认值，如果您不希望它们从给定的 
回填运行，则必须在 DAG 中指定 catchup=False {1}}。
start_date另一种可能的解决方案（假设您需要 dag = DAG(dag_id='talend_first_dag',default_args=args,schedule_interval='0/1 * * * *',catchup=False)
）是不要在过去设置 catchup。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
