如何解决使用气流模块的 PythonVirtualenvOperator 无法使用 AttributeError 执行：模块“airflow”没有属性“utils”
我在虚拟环境中部署了 Airflow，如果我尝试通过导入 Airflow 模块（例如获取变量）来执行 PythonVirtualenvOperator，它会给我 AttributeError。猜猜我不完全了解 Airflow 如何执行 VirtualenvOperator，因此如何克服它，所以任何建议和见解将不胜感激
我的测试 DAG 代码
from airflow import DAG
from airflow.operators.python import PythonVirtualenvOperator
from airflow.utils.dates import days_ago
args = {
'owner': 'airflow',}
dag = DAG(
dag_id='example_python_operator_shieet',default_args=args,schedule_interval=None,start_date=days_ago(2),tags=['example'],)
def callable_virtualenv():
from time import sleep
from colorama import Back,Fore,Style
from airflow.models import Variable
print(Fore.RED + 'some red text')
print(Back.GREEN + 'and with a green background')
print(Style.DIM + 'and in dim text')
print(Style.RESET_ALL)
for _ in range(10):
print(Style.DIM + 'Please wait...',flush=True)
sleep(10)
print('Finished')
virtualenv_task = PythonVirtualenvOperator(
task_id="virtualenv_python",python_callable=callable_virtualenv,requirements=["colorama==0.4.0",'apache-airflow==2.0.1'],system_site_packages=False,dag=dag,)
运行，当 PythonVirtualenvOperator 使用气流模块时，它为我提供了相同的日志
*** Reading local file: /opt1/app_data/airflow/logs/weather_update/updating_forecasts/2021-04-19T12:27:09.086267+00:00/1.log
[2021-04-19 15:27:10,221] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:1042} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,234] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-04-19 15:27:10,234] {taskinstance.py:1044} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,241] {taskinstance.py:1063} INFO - Executing <Task(PythonVirtualenvOperator): updating_forecasts> on 2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,244] {standard_task_runner.py:52} INFO - Started process 24052 to run task
[2021-04-19 15:27:10,248] {standard_task_runner.py:76} INFO - Running: ['airflow','tasks','run','weather_update','updating_forecasts','2021-04-19T12:27:09.086267+00:00','--job-id','73','--pool','default_pool','--raw','--subdir','DAGS_FOLDER/weather_dag.py','--cfg-path','/tmp/tmp2j8ueojr','--error-file','/tmp/tmpvh9s78fv']
[2021-04-19 15:27:10,248] {standard_task_runner.py:77} INFO - Job 73: Subtask updating_forecasts
[2021-04-19 15:27:10,289] {logging_mixin.py:104} INFO - Running <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [running]> on host sk-bi-08.erkapharm.ru
[2021-04-19 15:27:10,330] {taskinstance.py:1257} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=burzilov.ea@erkapharm.com
AIRFLOW_CTX_DAG_OWNER=burzilov.ea
AIRFLOW_CTX_DAG_ID=weather_update
AIRFLOW_CTX_TASK_ID=updating_forecasts
AIRFLOW_CTX_EXECUTION_DATE=2021-04-19T12:27:09.086267+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,331] {process_utils.py:135} INFO - Executing cmd: virtualenv /tmp/venvxuo1gayl
[2021-04-19 15:27:10,336] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:10,556] {process_utils.py:141} INFO - created virtual environment Cpython3.7.3.final.0-64 in 134ms
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - creator Cpython3Posix(dest=/tmp/venvxuo1gayl,clear=False,no_vcs_ignore=False,global=False)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - seeder FromAppData(download=False,pip=bundle,setuptools=bundle,wheel=bundle,via=copy,app_data_dir=/home/RKF/burzilov.ea/.local/share/virtualenv)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - added seed packages: pip==21.0.1,setuptools==54.2.0,wheel==0.36.2
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2021-04-19 15:27:10,574] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/pip install pandas numpy pyodbc sqlalchemy apache-airflow==2.0.1
[2021-04-19 15:27:10,579] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:11,603] {process_utils.py:141} INFO - Collecting pandas
[2021-04-19 15:27:11,644] {process_utils.py:141} INFO - Using cached pandas-1.2.4-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
[2021-04-19 15:27:12,048] {process_utils.py:141} INFO - Collecting numpy
[2021-04-19 15:27:12,112] {process_utils.py:141} INFO - Using cached numpy-1.20.2-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Collecting pyodbc
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Using cached pyodbc-4.0.30-cp37-cp37m-linux_x86_64.whl
[2021-04-19 15:27:12,519] {process_utils.py:141} INFO - Collecting sqlalchemy
[2021-04-19 15:27:12,530] {process_utils.py:141} INFO - Using cached sqlAlchemy-1.4.9-cp37-cp37m-manylinux2014_x86_64.whl (1.5 MB)
[2021-04-19 15:27:12,634] {process_utils.py:141} INFO - Collecting apache-airflow==2.0.1
[2021-04-19 15:27:12,699] {process_utils.py:141} INFO - Using cached apache_airflow-2.0.1-py3-none-any.whl (4.5 MB)
[2021-04-19 15:27:14,536] {process_utils.py:141} INFO - Collecting sqlalchemy-jsonfield~=1.0
[2021-04-19 15:27:14,538] {process_utils.py:141} INFO - Using cached sqlAlchemy_JSONField-1.0.0-py3-none-any.whl (10 kB)
[2021-04-19 15:27:14,603] {process_utils.py:141} INFO - Collecting attrs<21.0,>=20.0
[2021-04-19 15:27:14,606] {process_utils.py:141} INFO - Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Collecting python-slugify<5.0,>=3.0.0
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Using cached python_slugify-4.0.1-py2.py3-none-any.whl
[2021-04-19 15:27:14,716] {process_utils.py:141} INFO - Collecting itsdangerous>=1.1.0
[2021-04-19 15:27:14,718] {process_utils.py:141} INFO - Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
[2021-04-19 15:27:14,799] {process_utils.py:141} INFO - Collecting graphviz>=0.12
[2021-04-19 15:27:14,801] {process_utils.py:141} INFO - Using cached graphviz-0.16-py2.py3-none-any.whl (19 kB)
[2021-04-19 15:27:14,895] {process_utils.py:141} INFO - Collecting markdown<4.0,>=2.5.2
[2021-04-19 15:27:14,897] {process_utils.py:141} INFO - Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
[2021-04-19 15:27:14,970] {process_utils.py:141} INFO - Collecting flask<2.0,>=1.1.0
[2021-04-19 15:27:14,973] {process_utils.py:141} INFO - Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
[2021-04-19 15:27:15,104] {process_utils.py:141} INFO - Collecting connexion[flask,swagger-ui]<3,>=2.6.0
[2021-04-19 15:27:15,107] {process_utils.py:141} INFO - Using cached connexion-2.7.0-py2.py3-none-any.whl (77 kB)
[2021-04-19 15:27:15,199] {process_utils.py:141} INFO - Collecting requests>=2.20.0
[2021-04-19 15:27:15,202] {process_utils.py:141} INFO - Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
[2021-04-19 15:27:15,286] {process_utils.py:141} INFO - Collecting markupsafe<2.0,>=1.1.1
[2021-04-19 15:27:15,288] {process_utils.py:141} INFO - Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (33 kB)
[2021-04-19 15:27:15,366] {process_utils.py:141} INFO - Collecting python-dateutil<3,>=2.3
[2021-04-19 15:27:15,370] {process_utils.py:141} INFO - Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
[2021-04-19 15:27:15,494] {process_utils.py:141} INFO - Collecting rich==9.2.0
[2021-04-19 15:27:15,498] {process_utils.py:141} INFO - Using cached rich-9.2.0-py3-none-any.whl (164 kB)
[2021-04-19 15:27:15,555] {process_utils.py:141} INFO - Collecting marshmallow-oneofschema>=2.0.1
[2021-04-19 15:27:15,556] {process_utils.py:141} INFO - Using cached marshmallow_oneofschema-2.1.0-py2.py3-none-any.whl (5.7 kB)
[2021-04-19 15:27:15,614] {process_utils.py:141} INFO - Collecting typing-extensions>=3.7.4
[2021-04-19 15:27:15,616] {process_utils.py:141} INFO - Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
[2021-04-19 15:27:15,682] {process_utils.py:141} INFO - Collecting flask-caching<2.0.0,>=1.5.0
[2021-04-19 15:27:15,684] {process_utils.py:141} INFO - Using cached Flask_Caching-1.10.1-py3-none-any.whl (34 kB)
[2021-04-19 15:27:15,744] {process_utils.py:141} INFO - Collecting tabulate<0.9,>=0.7.5
[2021-04-19 15:27:15,745] {process_utils.py:141} INFO - Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
[2021-04-19 15:27:15,813] {process_utils.py:141} INFO - Collecting jinja2<2.12.0,>=2.10.1
[2021-04-19 15:27:15,817] {process_utils.py:141} INFO - Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
[2021-04-19 15:27:15,872] {process_utils.py:141} INFO - Collecting apache-airflow-providers-http
[2021-04-19 15:27:15,874] {process_utils.py:141} INFO - Using cached apache_airflow_providers_http-1.1.1-py3-none-any.whl (20 kB)
[2021-04-19 15:27:16,201] {process_utils.py:141} INFO - Collecting cryptography>=0.9.3
[2021-04-19 15:27:16,208] {process_utils.py:141} INFO - Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
[2021-04-19 15:27:16,314] {process_utils.py:141} INFO - Collecting croniter<0.4,>=0.3.17
[2021-04-19 15:27:16,316] {process_utils.py:141} INFO - Using cached croniter-0.3.37-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:16,398] {process_utils.py:141} INFO - Collecting alembic<2.0,>=1.2
[2021-04-19 15:27:16,400] {process_utils.py:141} INFO - Using cached alembic-1.5.8-py2.py3-none-any.whl (159 kB)
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Collecting python-nvd3~=0.15.0
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Using cached python_nvd3-0.15.0-py3-none-any.whl
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Collecting unicodecsv>=0.14.1
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Using cached unicodecsv-0.14.1-py3-none-any.whl
[2021-04-19 15:27:16,568] {process_utils.py:141} INFO - Collecting setproctitle<2,>=1.1.8
[2021-04-19 15:27:16,570] {process_utils.py:141} INFO - Using cached setproctitle-1.2.2-cp37-cp37m-manylinux1_x86_64.whl (36 kB)
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Collecting flask-login<0.5,>=0.3
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Using cached Flask_Login-0.4.1-py2.py3-none-any.whl
[2021-04-19 15:27:16,682] {process_utils.py:141} INFO - Collecting apache-airflow-providers-imap
[2021-04-19 15:27:16,684] {process_utils.py:141} INFO - Using cached apache_airflow_providers_imap-1.0.1-py3-none-any.whl (15 kB)
[2021-04-19 15:27:16,768] {process_utils.py:141} INFO - Collecting colorlog>=4.0.2
[2021-04-19 15:27:16,770] {process_utils.py:141} INFO - Using cached colorlog-5.0.1-py2.py3-none-any.whl (10 kB)
[2021-04-19 15:27:16,874] {process_utils.py:141} INFO - Collecting pygments<3.0,>=2.0.1
[2021-04-19 15:27:16,878] {process_utils.py:141} INFO - Using cached Pygments-2.8.1-py3-none-any.whl (983 kB)
[2021-04-19 15:27:16,976] {process_utils.py:141} INFO - Collecting argcomplete~=1.10
[2021-04-19 15:27:16,978] {process_utils.py:141} INFO - Using cached argcomplete-1.12.2-py2.py3-none-any.whl (38 kB)
[2021-04-19 15:27:17,055] {process_utils.py:141} INFO - Collecting importlib-resources~=1.4
[2021-04-19 15:27:17,057] {process_utils.py:141} INFO - Using cached importlib_resources-1.5.0-py2.py3-none-any.whl (21 kB)
[2021-04-19 15:27:17,113] {process_utils.py:141} INFO - Collecting apache-airflow-providers-sqlite
[2021-04-19 15:27:17,114] {process_utils.py:141} INFO - Using cached apache_airflow_providers_sqlite-1.0.2-py3-none-any.whl (14 kB)
[2021-04-19 15:27:17,185] {process_utils.py:141} INFO - Collecting cattrs~=1.1
[2021-04-19 15:27:17,187] {process_utils.py:141} INFO - Using cached cattrs-1.5.0-py3-none-any.whl (19 kB)
[2021-04-19 15:27:17,274] {process_utils.py:141} INFO - Collecting pyjwt<2
[2021-04-19 15:27:17,275] {process_utils.py:141} INFO - Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
[2021-04-19 15:27:17,397] {process_utils.py:141} INFO - Collecting lazy-object-proxy
[2021-04-19 15:27:17,398] {process_utils.py:141} INFO - Using cached lazy_object_proxy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (55 kB)
[2021-04-19 15:27:17,457] {process_utils.py:141} INFO - Collecting flask-wtf<0.15,>=0.14.3
[2021-04-19 15:27:17,459] {process_utils.py:141} INFO - Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:17,603] {process_utils.py:141} INFO - Collecting flask-appbuilder~=3.1.1
[2021-04-19 15:27:17,632] {process_utils.py:141} INFO - Using cached Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
[2021-04-19 15:27:17,696] {process_utils.py:141} INFO - Collecting apache-airflow-providers-ftp
[2021-04-19 15:27:17,698] {process_utils.py:141} INFO - Using cached apache_airflow_providers_ftp-1.0.1-py3-none-any.whl (14 kB)
*some other apache-airflow dependencies installation logs*
[2021-04-19 15:27:23,603] {process_utils.py:141} INFO - Collecting certifi>=2017.4.17
[2021-04-19 15:27:23,607] {process_utils.py:141} INFO - Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
[2021-04-19 15:27:24,201] {process_utils.py:141} INFO - Collecting isodate
[2021-04-19 15:27:24,203] {process_utils.py:141} INFO - Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
[2021-04-19 15:27:26,537] {process_utils.py:141} INFO - Installing collected packages: zipp,six,pyrsistent,importlib-Metadata,attrs,markupsafe,jsonschema,isodate,werkzeug,urllib3,PyYAML,pytz,openapi-schema-validator,jinja2,itsdangerous,idna,greenlet,defusedxml,click,chardet,certifi,WTForms,text-unidecode,sqlalchemy,requests,python3-openid,pyjwt,pycparser,openapi-spec-validator,marshmallow,inflection,flask,dnspython,clickclick,Babel,apispec,typing-extensions,swagger-ui-bundle,sqlalchemy-utils,pytzdata,python-slugify,python-editor,python-dateutil,pygments,prison,numpy,natsort,marshmallow-sqlalchemy,marshmallow-enum,Mako,lockfile,flask-wtf,Flask-sqlAlchemy,Flask-OpenID,flask-login,Flask-JWT-Extended,flask-babel,email-validator,docutils,connexion,commonmark,colorama,cffi,unicodecsv,termcolor,tenacity,tabulate,sqlalchemy-jsonfield,setproctitle,rich,python-nvd3,python-daemon,psutil,pendulum,pandas,marshmallow-oneofschema,markdown,lazy-object-proxy,iso8601,importlib-resources,gunicorn,graphviz,flask-caching,flask-appbuilder,dill,cryptography,croniter,colorlog,cattrs,cached-property,argcomplete,apache-airflow-providers-sqlite,apache-airflow-providers-imap,apache-airflow-providers-http,apache-airflow-providers-ftp,alembic,pyodbc,apache-airflow
[2021-04-19 15:27:35,766] {process_utils.py:141} INFO - Successfully installed Babel-2.9.0 flask-babel-1.0.0 Flask-JWT-Extended-3.25.1 Flask-OpenID-1.2.5 Flask-sqlAlchemy-2.5.1 Mako-1.1.4 PyYAML-5.4.1 WTForms-2.3.3 alembic-1.5.8 apache-airflow-2.0.1 apache-airflow-providers-ftp-1.0.1 apache-airflow-providers-http-1.1.1 apache-airflow-providers-imap-1.0.1 apache-airflow-providers-sqlite-1.0.2 apispec-3.3.2 argcomplete-1.12.2 attrs-20.3.0 cached-property-1.5.2 cattrs-1.5.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 click-7.1.2 clickclick-20.10.2 colorama-0.4.4 colorlog-5.0.1 commonmark-0.9.1 connexion-2.7.0 croniter-0.3.37 cryptography-3.4.7 defusedxml-0.7.1 dill-0.3.3 dnspython-2.1.0 docutils-0.17.1 email-validator-1.1.2 flask-1.1.2 flask-appbuilder-3.1.1 flask-caching-1.10.1 flask-login-0.4.1 flask-wtf-0.14.3 graphviz-0.16 greenlet-1.0.0 gunicorn-19.10.0 idna-2.10 importlib-Metadata-1.7.0 importlib-resources-1.5.0 inflection-0.5.1 iso8601-0.1.14 isodate-0.6.0 itsdangerous-1.1.0 jinja2-2.11.3 jsonschema-3.2.0 lazy-object-proxy-1.6.0 lockfile-0.12.2 markdown-3.3.4 markupsafe-1.1.1 marshmallow-3.11.1 marshmallow-enum-1.5.1 marshmallow-oneofschema-2.1.0 marshmallow-sqlalchemy-0.23.1 natsort-7.1.1 numpy-1.20.2 openapi-schema-validator-0.1.5 openapi-spec-validator-0.3.0 pandas-1.2.4 pendulum-2.1.2 prison-0.1.3 psutil-5.8.0 pycparser-2.20 pygments-2.8.1 pyjwt-1.7.1 pyodbc-4.0.30 pyrsistent-0.17.3 python-daemon-2.3.0 python-dateutil-2.8.1 python-editor-1.0.4 python-nvd3-0.15.0 python-slugify-4.0.1 python3-openid-3.2.0 pytz-2021.1 pytzdata-2020.1 requests-2.25.1 rich-9.2.0 setproctitle-1.2.2 six-1.15.0 sqlalchemy-1.4.9 sqlalchemy-jsonfield-1.0.0 sqlalchemy-utils-0.37.0 swagger-ui-bundle-0.0.8 tabulate-0.8.9 tenacity-6.2.0 termcolor-1.1.0 text-unidecode-1.3 typing-extensions-3.7.4.3 unicodecsv-0.14.1 urllib3-1.26.4 werkzeug-1.0.1 zipp-3.4.1
[2021-04-19 15:27:36,306] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/python /tmp/venvxuo1gayl/script.py /tmp/venvxuo1gayl/script.in /tmp/venvxuo1gayl/script.out /tmp/venvxuo1gayl/string_args.txt
[2021-04-19 15:27:36,312] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:36,875] {process_utils.py:141} INFO - [[34m2021-04-19 15:27:36,875[0m] {[34mlogging_config.py:[0m59} ERROR[0m - Unable to load the config,contains a configuration error.[0m
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py",line 93,in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found,n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - During handling of the above exception,another exception occurred:
[2021-04-19 15:27:36,line 542,in configure
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - formatters[name])
[2021-04-19 15:27:36,line 672,in configure_formatter
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - c = _resolve(cname)
[2021-04-19 15:27:36,line 96,877] {process_utils.py:141} INFO - The above exception was the direct cause of the following exception:
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py",line 132,in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - res = weather_update(*arg_dict["args"],**arg_dict["kwargs"])
[2021-04-19 15:27:36,line 40,in weather_update
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - from airflow.models import Variable
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/__init__.py",line 46,877] {process_utils.py:141} INFO - settings.initialize()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/settings.py",line 434,in initialize
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - LOGGING_CLASS_PATH = configure_logging()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py",line 62,in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - raise e
[2021-04-19 15:27:36,line 57,877] {process_utils.py:141} INFO - dictConfig(logging_config)
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py",line 799,in dictConfig
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfigClass(config).configure()
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py",line 545,878] {process_utils.py:141} INFO - 'formatter %r' % name) from e
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - ValueError: Unable to configure formatter 'airflow_coloured'
[2021-04-19 15:27:37,210] {taskinstance.py:1455} ERROR - Command '['/tmp/venvxuo1gayl/bin/python','/tmp/venvxuo1gayl/script.py','/tmp/venvxuo1gayl/script.in','/tmp/venvxuo1gayl/script.out','/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py",line 1112,in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context,task)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py",line 1285,in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context,task_copy)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py",line 1315,in _execute_task
result = task_copy.execute(context=context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py",line 493,in execute
super().execute(context=serializable_context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py",line 117,in execute
return_value = self.execute_callable()
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py",line 531,in execute_callable
string_args_filename,File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/utils/process_utils.py",line 145,in execute_in_subprocess
raise subprocess.CalledProcessError(exit_code,cmd)
subprocess.CalledProcessError: Command '['/tmp/venvxuo1gayl/bin/python','/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
[2021-04-19 15:27:37,212] {taskinstance.py:1503} INFO - Marking task as Failed. dag_id=weather_update,task_id=updating_forecasts,execution_date=20210419T122709,start_date=20210419T122710,end_date=20210419T122737
[2021-04-19 15:27:37,227] {configuration.py:356} WARNING - section/key [smtp/smtp_user] not found in config
解决方法
您似乎混淆了 PythonVirtualenvOperator 和 PythonOperator 的用例。
如果您只是想在任务（在您的情况下为 callable_virtualenv()）中运行可调用的 Python，您可以使用 PythonOperator。在这种情况下，您是在虚拟环境、系统范围内还是使用 Docker 安装 Airflow 都没有关系。
您的代码中发生的情况如下：PythonVirtualenvOperator 创建另一个虚拟环境（与您运行 Airflow 的环境完全无关），将 Airflow 安装到其中，并尝试导入 Variable .但是未配置另一个 Airflow 安装，这就是您收到这些异常的原因。您可以将第二个 Airflow 安装的 AIRFLOW_HOME 环境变量设置为与第一个 Airflow 安装使用的目录相同的目录，这实际上应该可以工作，但对我来说似乎有点矫枉过正。
因此，您可以将 colorama 安装到安装 Airflow 的相同环境中，并将 PythonVirtualenvOperator 替换为 PythonOperator。
顺便说一句，可调用对象中的那些 print() 将被重定向到日志文件中，而不是打印到终端，因此将 colorama 与它们一起使用可能没有多大意义。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。