从在 docker 容器内运行的 jenkins 连接到 Docker 守护进程

如何解决从在 docker 容器内运行的 jenkins 连接到 Docker 守护进程

我有一个 jenkins 在一个 docker 容器内运行，一个 docker 在不同的 docker 容器内运行。

我已将本地机器的 /var/run/docker.sock 文件映射到 docker 容器，并且能够在 docker 容器内执行 docker 命令。 docker 容器和 jenkins 容器都在同一个网络上。但是在从 jenkins 连接到 docker 容器时，连接被拒绝。我已为 /var/run/docker.sock 文件授予 666 权限，但仍无法在两者之间进行连接。两个容器可以互相ping通。

解决方法

TL;DR

您可以通过 tcp 或通过在容器之间共享 docker 套接字连接到 Docker in Docker 环境。

---

此示例包含使用 docker 编排的 docker-compose 中的所有内容。

.
├── docker-compose.yaml
├── Dockerfile
├── etc
│   └── nginx
│       └── conf.d
│           └── default.conf
└── plugins.txt

docker-compose.yaml 在 jenkins 和 nginx 服务后面设置了 docker:20.10.5-dind。

tcp

version: '3.7'
services:
  nginx:
    image: 'nginx:1.19'
    container_name: 'nginx'
    restart: 'always'
    depends_on:
    - 'jenkins'
    ports:
    - '80:80'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - './etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf'
  jenkins:
    build:
      context: '.'
    container_name: 'jenkins'
    restart: 'always'
    expose:
    - '50000'
    - '8080'
    environment:
    - 'DOCKER_HOST=tcp://docker:2376'
    - 'DOCKER_CERT_PATH=/certs/client'
    - 'DOCKER_TLS_VERIFY=1'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - 'certs:/certs:ro'
  docker:
    image: 'docker:20.10.5-dind'
    container_name: 'docker'
    privileged: true
    volumes:
    - 'certs:/certs'
volumes:
  jenkins:
  certs:

注意：docker 客户端证书在 docker 和 jenkins 容器之间共享，并且环境设置在 jenkins 容器中连接到 docker 服务。

nginx config is slightly modified from the doc：

upstream jenkins {
  keepalive 32;
  server jenkins:8080 max_fails=3;
}
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}
server {
  listen *:80;
  listen [::]:80;
  server_name _;

  charset utf-8;
  ignore_invalid_headers off;

  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
      root   /usr/share/nginx/html;
  }
  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }
  location /userContent {
    root /var/jenkins_home/;
    if (!-f $request_filename){
      rewrite (.*) /$1 last;
      break;
    }
    sendfile on;
  }
  location / {
    sendfile off;
    proxy_pass         http://jenkins;
    proxy_redirect     default;
    proxy_http_version 1.1;

    proxy_set_header   Connection        $connection_upgrade;
    proxy_set_header   Upgrade           $http_upgrade;

    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_max_temp_file_size 0;

    client_max_body_size       10m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;
    proxy_buffering            off;
    proxy_request_buffering    off;
    proxy_set_header Connection "";
  }
}

jenkins 服务是使用 docker 客户端和默认建议的 jenkins 插件以及 Docker 和 Docker Pipeline 插件预烘焙的自定义构建图像:

FROM docker:20.10.5-dind as docker
FROM jenkins/jenkins:alpine
USER root
COPY --from=docker /usr/local/bin/docker /usr/local/bin/docker
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt
USER jenkins

github:1.33.1
pipeline-model-api:1.8.4
scm-api:2.6.4
mailer:1.32.1
workflow-support:3.8
font-awesome-api:5.15.2-2
pipeline-milestone-step:1.3.2
git:4.6.0
plain-credentials:1.7
resource-disposer:0.15
jackson2-api:2.12.1
jquery3-api:3.5.1-3
gradle:1.36
credentials:2.3.15
docker-workflow:1.26
workflow-scm-step:2.12
display-url-api:2.3.4
bootstrap4-api:4.6.0-2
antisamy-markup-formatter:2.1
command-launcher:1.5
pipeline-stage-tags-metadata:1.8.4
snakeyaml-api:1.27.0
pipeline-stage-view:2.19
script-security:1.76
okhttp-api:3.14.9
pipeline-stage-step:2.5
workflow-step-api:2.23
timestamper:1.11.8
pipeline-github-lib:1.0
token-macro:2.13
pam-auth:1.6
workflow-cps-global-lib:2.18
ws-cleanup:0.39
pipeline-model-definition:1.8.4
workflow-aggregator:2.6
jsch:0.1.55.2
matrix-auth:2.6.5
ssh-credentials:1.18.1
ant:1.11
jjwt-api:0.11.2-9.c8b45b8bb173
momentjs:1.1.1
trilead-api:1.0.13
durable-task:1.35
workflow-job:2.40
git-server:1.9
ssh-slaves:1.31.5
plugin-util-api:2.0.0
git-client:3.6.0
lockable-resources:2.10
checks-api:1.5.0
pipeline-input-step:2.12
cloudbees-folder:6.15
pipeline-build-step:2.13
popper-api:1.16.1-2
pipeline-graph-analysis:1.10
matrix-project:1.18
workflow-api:2.41
github-branch-source:2.9.7
workflow-basic-steps:2.23
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-multibranch:2.22
workflow-cps:2.90
ldap:1.26
build-timeout:1.20
echarts-api:5.0.1-1
pipeline-model-extensions:1.8.4
structs:1.22
junit:1.48
docker-java-api:3.1.5.2
docker-plugin:1.2.2
workflow-durable-task-step:2.38
credentials-binding:1.24
jdk-tool:1.5
bouncycastle-api:2.20
docker-commons:1.17
github-api:1.123
authentication-tokens:1.4
email-ext:2.82
branch-api:2.6.2
pipeline-rest-api:2.19
ace-editor:1.1
handlebars:1.1.1

初始 jenkins 设置后，创建 X.509 客户端证书服务器凭据，然后使用 docker 使用 tcp 服务配置 Docker Cloud。

注意：您可以使用以下命令获取用于创建 X.509 客户端证书服务器凭据的客户端证书、客户端密钥和服务器 CA 证书：

docker exec docker cat /certs/client/key.pem
docker exec docker cat /certs/client/cert.pem
docker exec docker cat /certs/server/ca.pem

插座

version: '3.7'
services:
  nginx:
    image: 'nginx:1.19'
    container_name: 'nginx'
    restart: 'always'
    depends_on:
    - 'jenkins'
    ports:
    - '80:80'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - './etc/nginx/conf.d/default.conf:/etc/nginx/conf.d/default.conf'
  jenkins:
    build:
      context: '.'
    container_name: 'jenkins'
    restart: 'always'
    expose:
    - '50000'
    - '8080'
    volumes:
    - 'jenkins:/var/jenkins_home'
    - 'socket:/var/run'
  docker:
    image: 'docker:20.10.5-dind'
    container_name: 'docker'
    privileged: true
    volumes:
    - 'socket:/var/run'
volumes:
  jenkins:
  socket:

注意：docker 套接字在 docker 卷中的 jenkins 和 socket 容器之间共享。

默认情况下，docker 套接字归 root:root 所有，jenkins 用户无法连接到共享套接字，您可以将套接字组所有权更改为共享套接字的 GID jenkins 用户：docker exec docker chown 0:1000 /var/run/docker.sock。

在初始 jenkins 设置后，使用共享 docker 套接字使用 unix 服务配置 Docker Cloud。