如何解决AWS Batch:使用云形成将 efs 卷挂载到容器中
我使用 Cloud Formation 创建了这一切。
现在我想向此作业定义添加一个 EFS 卷(名称:EFS-000,文件系统 ID:fs-9999999)和一个 MountPoint。
我读过
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/tutorial-efs-volumes.html
- https://aws.amazon.com/es/premiumsupport/knowledge-center/batch-mount-efs
在第一个链接中,我们有一个任务定义示例(AWS ECS 而不是 AWS Batch 概念)
viewmodel Scope
将正确的代码添加到我的 Cloud Formation 配方中似乎很容易(我选择了 yaml 语法)。在我的 ContainerDeFinition 中,我添加了...
{
"containerDeFinitions": [
{
"memory": 128,"portMappings": [
{
"hostPort": 80,"containerPort": 80,"protocol": "tcp"
}
],"essential": true,"mountPoints": [
{
"containerPath": "/usr/share/Nginx/html","sourceVolume": "efs-html"
}
],"name": "Nginx","image": "Nginx"
}
],"volumes": [
{
"name": "efs-html","efsVolumeConfiguration": {
"fileSystemId": "fs-1324abcd","transitEncryption": "ENABLED"
}
}
],"family": "efs-tutorial"
}
但是当我运行 Cloud Formation 配方时,我得到....
Volumes:
- Name: SRV
EfsVolumeConfiguration:
FileSystemId: fs-9999999
TransitEncryption: ENABLED
The following resource(s) Failed to update: [ContentJob1].
如果 EfsVolumeConfiguration 不是有效的属性...
如何使用 Cloud Formation 将 EFS 卷添加到 AWS Batch 作业定义?
解决方法
在 CloudFormation 支持 EFS 之前,我使用了这个解决方案。不过还是有用的。
我创建了一个启动模板
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: !Join ['',[!Ref ServiceName,'-EC2-',LaunchTemplate]]
LaunchTemplateData:
UserData:
Fn::Base64: !Sub |
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"
runcmd:
- mkdir /mnt/efs-misc
- mkdir /mnt/efs-jobs
- echo "${EfsJobs}:/ /mnt/efs-jobs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=300,retrans=2,noresvport 0 0" >> /etc/fstab
- echo "${EfsMisc}:/ /mnt/efs-misc nfs4 nfsvers=4.1,noresvport 0 0" >> /etc/fstab
- mount -a
--==MYBOUNDARY==--
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096
然后我创建了一个使用这个模板的计算环境。
BatchCompute:
Type: 'AWS::Batch::ComputeEnvironment'
DependsOn: LaunchTemplate
Properties:
ComputeEnvironmentName: !Join ['','-EC2']]
ComputeResources:
AllocationStrategy: SPOT_CAPACITY_OPTIMIZED
DesiredvCpus: 0
Ec2KeyPair: !Join ['-',[MyComputeEnv,!Ref EnvironmentLow]]
InstanceRole: !GetAtt BatchInstanceProfile.Arn
InstanceTypes:
- c4.large
- c5.large
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: $Latest
MaxvCpus: 256
MinvCpus: 0
DesiredvCpus: 0
SecurityGroupIds:
- !Ref DefaultSecurityGroup
Subnets: !Split [',',!Join [',[!Ref SubnetA,!Ref SubnetB,!Ref SubnetC ]]]
Type: SPOT
SpotIamFleetRole: SpotFleetTagServiceRole
ServiceRole: !Ref BatchServiceRole
State: ENABLED
Type: Managed
最后,我使用了 AWS::Batch::JobDefinition 的 ContainerProperties/[Volumes|MountPoints] 来添加我的 EFS 卷。
ContentJob0:
Type: 'AWS::Batch::JobDefinition'
Properties:
Type: Container
ContainerProperties:
Command:
- ls
Environment:
- Name: AAA
Value: AAA
ExecutionRoleArn: !GetAtt BatchJobRole.Arn
Image: !Join ['',[!Ref 'AWS::AccountId','.dkr.ecr.',!Ref 'AWS::Region','.amazonaws.com/',!Ref Image ]]
LogConfiguration:
LogDriver: awslogs
JobRoleArn: !Ref BatchContainerRole
Memory: 2048
Vcpus: 2
Volumes:
- Name: MISC
Host:
SourcePath: '/mnt/efs-0'
- Name: JOBS
Host:
SourcePath: '/mnt/efs-1'
MountPoints:
- SourceVolume: MISC
ContainerPath: '/mnt/efs-0'
- SourceVolume: JOBS
ContainerPath: '/mnt/efs-1'
JobDefinitionName: !Join ['-',['MyJob',!Ref ServiceName,'EC2','0']]
PlatformCapabilities:
- EC2
PropagateTags: true
RetryStrategy:
Attempts: 3
Timeout:
AttemptDurationSeconds: 300
由于我需要在 LaunchTemplate 中设置 SPOT 实例的价格和其他东西,这对我来说是一个很好的解决方案。
InstanceMarketOptions:
MarketType: spot
SpotOptions:
MaxPrice: 0.096
,
AWS 批处理 added EFS Volume support on April 1,2021。
用户文档位于 https://docs.aws.amazon.com/batch/latest/userguide/efs-volumes.html
博客文章位于 https://aws.amazon.com/blogs/hpc/introducing-support-for-per-job-amazon-efs-volumes-in-aws-batch/
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。