最近的一个项目,需要实现一个工作任务流(task pipeline),基于之前CICD的经验,jenkins pipeline和drone的pipeline进入候选。
drone是基于go的cicd解决方案,github上有1.6万+star,本文简单对比了其和jenkins的区别,重点介绍了drone的pipeline原理,并简单分析了代码。
jenkins 与 drone
对比项 | jenkins | drone |
---|---|---|
pipeline定义 | 编写jenkinsfile | 编写流程yml |
运行方式 | 在一个pod里运行 | 每一步骤起对应的container,通过挂载volume实现数据共享 |
运行环境 | 物理机或者容器环境,包括K8S | docker容器环境 |
开发语言 | java | golang |
drone pipeline好处是相对更轻量级,yml定义也相对简洁清晰,按照功能来划分容器,可以方便的实现task的复用,而jenkins则是完全打包到一个镜像,会造成单个镜像体积过大,比如jenkins的单个镜像超过2G。
drone的pipeline,是基于https://github.com/cncd/pipeline 实现的,这里简单分析下其原理。
编译和执行 drone pipeline
先安装:
go get -u github.com/cncd/pipeline go install github.com/cncd/pipeline/pipec
然后测试
cd $GOPATH/github.com/cncd/pipeline/samples/sample_1 # ll total 28 drwxr-xr-x 2 root root 4096 Jan 22 11:44 ./ drwxr-xr-x 13 root root 4096 Jan 22 11:02 ../ -rw-r--r-- 1 root root 549 Jan 22 11:02 .env -rw-r--r-- 1 root root 6804 Jan 22 16:30 pipeline.json -rw-r--r-- 1 root root 229 Jan 22 11:02 pipeline.yml -rw-r--r-- 1 root root 138 Jan 22 11:02 README.md
先来查看pipeline.yml
定义
workspace: base: /go path: src/github.com/drone/envsubst clone: git: image: plugins/git depth: 50 pipeline: build: image: golang:1.7 commands: - go get -t ./... - go build - go test -v
上面的yml定义了:
- 工作目录workspace
- 初始化工作,git clone仓库,仓库地址在.env里定义
- 然后是定义pipeline,
- pipeline下面是step数组,这里只有一个build
- 使用golang:1.7镜像
- 构建命令在commands数组里定义
通过pipec compile
compile配置文件:
# pipec compile Successfully compiled pipeline.yml to pipeline.json
查看编译后的pipeline.json
{ "pipeline": [ { "name": "pipeline_clone_0","alias": "git","steps": [ { "name": "pipeline_clone_0","image": "plugins/git:latest","working_dir": "/go/src/github.com/drone/envsubst","environment": { "CI": "drone","CI_BUILD_CREATED": "1486119586","CI_BUILD_EVENT": "push","CI_BUILD_NUMBER": "6","CI_BUILD_STARTED": "1486119585","CI_COMMIT_AUTHOR": "bradrydzewski","CI_COMMIT_AUTHOR_NAME": "bradrydzewski","CI_COMMIT_BRANCH": "master","CI_COMMIT_MESSAGE": "added a few more test cases for escaping behavior","CI_COMMIT_REF": "refs/heads/master","CI_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6","CI_REMOTE_URL": "https://github.com/drone/envsubst.git","CI_REPO": "drone/envsubst","CI_REPO_LINK": "https://github.com/drone/envsubst","CI_REPO_NAME": "drone/envsubst","CI_REPO_REMOTE": "https://github.com/drone/envsubst.git","CI_SYstem": "pipec","CI_SYstem_ARCH": "linux/amd64","CI_SYstem_LINK": "https://github.com/cncd/pipec","CI_SYstem_NAME": "pipec","CI_WORKSPACE": "/go/src/github.com/drone/envsubst","DRONE": "true","DRONE_ARCH": "linux/amd64","DRONE_BRANCH": "master","DRONE_BUILD_CREATED": "1486119586","DRONE_BUILD_EVENT": "push","DRONE_BUILD_LINK": "https://github.com/cncd/pipec/drone/envsubst/6","DRONE_BUILD_NUMBER": "6","DRONE_BUILD_STARTED": "1486119585","DRONE_COMMIT": "d0876d3176965f9552a611cbd56e24a9264355e6","DRONE_COMMIT_AUTHOR": "bradrydzewski","DRONE_COMMIT_BRANCH": "master","DRONE_COMMIT_MESSAGE": "added a few more test cases for escaping behavior","DRONE_COMMIT_REF": "refs/heads/master","DRONE_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6","DRONE_JOB_STARTED": "1486119585","DRONE_REMOTE_URL": "https://github.com/drone/envsubst.git","DRONE_REPO": "drone/envsubst","DRONE_REPO_LINK": "https://github.com/drone/envsubst","DRONE_REPO_NAME": "envsubst","DRONE_REPO_OWNER": "drone","DRONE_REPO_SCM": "git","DRONE_WORKSPACE": "/go/src/github.com/drone/envsubst","PLUGIN_DEPTH": "50" },"volumes": [ "pipeline_default:/go" ],"networks": [ { "name": "pipeline_default","aliases": [ "git" ] } ],"on_success": true,"auth_config": {} } ] },{ "name": "pipeline_stage_0","alias": "build","steps": [ { "name": "pipeline_step_0","image": "golang:1.7","CI_SCRIPT": "CmlmIFsgLW4gIiRDSV9ORVRSQ19NQUNISU5FIiBdOyB0aGVuCmNhdCA8PEVPRiA+ICRIT01FLy5uZXRyYwptYWNoaW5lICRDSV9ORVRSQ19NQUNISU5FCmxvZ2luICRDSV9ORVRSQ19VU0VSTkFNRQpwYXNzd29yZCAkQ0lfTkVUUkNfUEFTU1dPUkQKRU9GCmNobW9kIDA2MDAgJEhPTUUvLm5ldHJjCmZpCnVuc2V0IENJX05FVFJDX1VTRVJOQU1FCnVuc2V0IENJX05FVFJDX1BBU1NXT1JECnVuc2V0IENJX1NDUklQVAp1bnNldCBEUk9ORV9ORVRSQ19VU0VSTkFNRQp1bnNldCBEUk9ORV9ORVRSQ19QQVNTV09SRAoKZWNobyArICJnbyBnZXQgLXQgLi8uLi4iCmdvIGdldCAtdCAuLy4uLgoKZWNobyArICJnbyBidWlsZCIKZ28gYnVpbGQKCmVjaG8gKyAiZ28gdGVzdCAtdiIKZ28gdGVzdCAtdgoK","HOME": "/root","SHELL": "/bin/sh" },"entrypoint": [ "/bin/sh","-c" ],"command": [ "echo $CI_SCRIPT | base64 -d | /bin/sh -e" ],"aliases": [ "build" ] } ],"auth_config": {} } ] } ],"networks": [ { "name": "pipeline_default","driver": "bridge" } ],"volumes": [ { "name": "pipeline_default","driver": "local" } ],"secrets": null }
简单分析结构:
- pipeline 定义了执行的stage,每个stage有一个或者多个step
- networks、volumes、secrets 分别定义网络、存储和secrets
- 通过network,实现container互通
- 通过volumes实现数据共享
最后执行,通过pipec exec
:
# pipec exec proc "pipeline_clone_0" started + git init Initialized empty Git repository in /go/src/github.com/drone/envsubst/.git/ + git remote add origin https://github.com/drone/envsubst.git + git fetch --no-tags --depth=50 origin +refs/heads/master: From https://github.com/drone/envsubst * branch master -> FETCH_HEAD * [new branch] master -> origin/master + git reset --hard -q d0876d3176965f9552a611cbd56e24a9264355e6 + git submodule update --init --recursive proc "pipeline_clone_0" exited with status 0 proc "pipeline_step_0" started + go get -t ./... + go build + go test -v === RUN TestExpand --- PASS: TestExpand (0.00s) === RUN TestFuzz --- PASS: TestFuzz (0.01s) === RUN Test_len --- PASS: Test_len (0.00s) === RUN Test_lower --- PASS: Test_lower (0.00s) === RUN Test_lowerFirst --- PASS: Test_lowerFirst (0.00s) === RUN Test_upper --- PASS: Test_upper (0.00s) === RUN Test_upperFirst --- PASS: Test_upperFirst (0.00s) === RUN Test_default --- PASS: Test_default (0.00s) PASS ok github.com/drone/envsubst 0.009s proc "pipeline_step_0" exited with status 0
pipeline 原理分析
编译过程
可以形象的理解为 .env+pipeline.yml --> pipeline.json
编译过程不复杂,主要是解析pipeline.yml为Config
:
Config struct { Cache libcompose.Stringorslice Platform string Branches Constraint Workspace Workspace Clone Containers Pipeline Containers Services Containers Networks Networks Volumes Volumes Labels libcompose.SliceorMap }
然后转换为json对应的config:
Config struct { Stages []*Stage `json:"pipeline"` // pipeline stages Networks []*Network `json:"networks"` // network deFinitions Volumes []*Volume `json:"volumes"` // volume deFinitions Secrets []*Secret `json:"secrets"` // secret deFinitions }
该部分主要代码在pipeline/frontend里
执行过程
我们主要关注执行过程,主要代码在pipeline/backend里。
config,err := pipeline.Parse(reader) if err != nil { return err }
然后创建执行环境,目前的代码仅docker可用,k8s是空代码。
var engine backend.Engine if c.Bool("kubernetes") { engine = kubernetes.New( c.String("kubernetes-namepsace"),c.String("kubernetes-endpoint"),c.String("kubernetes-token"),) } else { engine,err = docker.NewEnv() if err != nil { return err } }
接着开始执行
ctx,cancel := context.WithTimeout(context.Background(),c.Duration("timeout")) defer cancel() ctx = interrupt.WithContext(ctx) return pipeline.New(config,pipeline.WithContext(ctx),pipeline.WithLogger(defaultLogger),pipeline.WithTracer(defaultTracer),pipeline.WithEngine(engine),).Run()
其中pipeline.NEW创建了Runtime
对象;
type Runtime struct { err error // 错误信息 spec *backend.Config // 配置信息 engine backend.Engine // docker engine started int64 // 开始时间 ctx context.Context tracer Tracer logger Logger }
其中Engine,操作容器的interface,目前仅docker可用。
// Engine defines a container orchestration backend and is used // to create and manage container resources. type Engine interface { // Setup the pipeline environment. Setup(context.Context,*Config) error // Start the pipeline step. Exec(context.Context,*Step) error // Kill the pipeline step. Kill(context.Context,*Step) error // Wait for the pipeline step to complete and returns // the completion results. Wait(context.Context,*Step) (*State,error) // Tail the pipeline step logs. Tail(context.Context,*Step) (io.ReadCloser,error) // Destroy the pipeline environment. Destroy(context.Context,*Config) error }
关注Run:
// Run starts the runtime and waits for it to complete. func (r *Runtime) Run() error { // 延迟函数,用于销毁docker env defer func() { r.engine.Destroy(r.ctx,r.spec) }() // 初始化docker engine r.started = time.Now().Unix() if err := r.engine.Setup(r.ctx,r.spec); err != nil { return err } // 依次运行stage for _,stage := range r.spec.Stages { select { case <-r.ctx.Done(): return ErrCancel // 执行 case err := <-r.execAll(stage.Steps): if err != nil { r.err = err } } } return r.err }
重点在于使用errgroup.Group通过协程方式运行step:
// 执行所有steps func (r *Runtime) execAll(procs []*backend.Step) <-chan error { var g errgroup.Group done := make(chan error) // 遍历执行step for _,proc := range procs { // 协程 exec proc := proc g.Go(func() error { return r.exec(proc) }) } go func() { done <- g.Wait() close(done) }() return done } // 执行单个step func (r *Runtime) exec(proc *backend.Step) error { switch { case r.err != nil && proc.OnFailure == false: return nil case r.err == nil && proc.OnSuccess == false: return nil } // trace日志 if r.tracer != nil { state := new(State) state.Pipeline.Time = r.started state.Pipeline.Error = r.err state.Pipeline.Step = proc state.Process = new(backend.State) // empty if err := r.tracer.Trace(state); err == ErrSkip { return nil } else if err != nil { return err } } // docker engine执行 if err := r.engine.Exec(r.ctx,proc); err != nil { return err } // 记录日志信息 if r.logger != nil { rc,err := r.engine.Tail(r.ctx,proc) if err != nil { return err } go func() { r.logger.Log(proc,multipart.New(rc)) rc.Close() }() } if proc.Detached { return nil } // 等待docker engine执行完成 wait,err := r.engine.Wait(r.ctx,proc) if err != nil { return err } if r.tracer != nil { state := new(State) state.Pipeline.Time = r.started state.Pipeline.Error = r.err state.Pipeline.Step = proc state.Process = wait if err := r.tracer.Trace(state); err != nil { return err } } if wait.OOMKilled { return &OomError{ Name: proc.Name,Code: wait.ExitCode,} } else if wait.ExitCode != 0 { return &ExitError{ Name: proc.Name,} } return nil }
作者:Jadepeng
出处:jqpeng的技术记事本--http://www.cnblogs.com/xiaoqi 您的支持是对博主最大的鼓励,感谢您的认真阅读。 本文版权归作者所有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则保留追究法律责任的权利。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。