在 kubernetes operator 中访问 GOOGLE_APPLICATION_CREDENTIALS

如何解决在 kubernetes operator 中访问 GOOGLE_APPLICATION_CREDENTIALS

我正在向气流 dag 添加一个任务，如下所示：

examples_task = KubernetesPodoperator(
    task_id='examples_generation',dag=dag,namespace='test',image='test_amazon_image',name='pipe-labelled-examples-generation-tf-record-operator',env={
        'GOOGLE_APPLICATION_CREDENTIALS': Variable.get('google_cloud_credentials')
    },arguments=[
        "--assets_path",Variable.get('assets_path'),"--folder_source",Variable.get('folder_source'),"--folder_destination",Variable.get('folder_destination'),"--gcs_folder_destination",Variable.get('gcs_folder_destination'),"--aws_region",Variable.get('aws_region'),"--s3_endpoint",Variable.get('s3_endpoint')
    ],get_logs=True)

我以为我可以将服务帐户 json 文件粘贴为变量并调用它，但这不起作用并且气流/谷歌文档不清楚。你是怎么做到的？

解决方法

将 json 移植到参数中的解决方案

examples_task = KubernetesPodOperator(
    task_id='examples_generation',dag=dag,namespace='test',image='test_amazon_image',name='pipe-labelled-examples-generation-tf-record-operator',arguments=[
        "--folder_source",Variable.get('folder_source'),"--folder_destination",Variable.get('folder_destination'),"--gcs_folder_destination",Variable.get('gcs_folder_destination'),"--aws_region",Variable.get('aws_region'),"--s3_endpoint",Variable.get('s3_endpoint')
        "--gcs_credentials",Variable.get('google_cloud_credentials')

    ],get_logs=True)

然后在cli集合中

import json
from google.cloud import storage
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_info(json.loads(gcs_credentials))
client = storage.Client(project='project_id',credentials=credentials)