基于 Kubernetes 的 Jenkins 服務也可以去 Docker 了
作者：陳少文
在 1.20 版本之后, Kubernetes 社區(qū)放棄了對 Docker 的支持, 而后又有其他社區(qū)接手, 隱約給 Docker 蒙上了一層陰影。在這樣的背景下, 我們開始考慮非 Docker 環(huán)境下, 如何進行 CICD 實踐。
本文轉載自微信公眾號「問其」，作者陳少文 。轉載本文請聯(lián)系問其公眾號。
1. 去 Docker 給 CICD 帶來新的挑戰(zhàn)
在 CICD 場景下, 我們經(jīng)常需要在流水線中構建和推送鏡像。
在之前的文檔 《在 Kubernetes 上動態(tài)創(chuàng)建 Jenkins Slave》[1] 中, 我描述了通過掛載 /var/run/docker.sock 文件, 允許在 Docker 驅動的 Kubernetes 集群中構建和推送鏡像。在文檔 《如何在 Docker 中使用 Docker》[2]中, 我又進行了更加詳細地闡述, 其原理是共享主機 Docker Daemon。
在 1.20 版本之后, Kubernetes 社區(qū)放棄了對 Docker 的支持, 而后又有其他社區(qū)接手, 隱約給 Docker 蒙上了一層陰影。在這樣的背景下, 我們開始考慮非 Docker 環(huán)境下, 如何進行 CICD 實踐。
非 Docker 環(huán)境意味著之前掛載 /var/run/docker.sock 的方式失效了, 我們需要尋找新的解決方案。
2. 測試集群環(huán)境
2.1 Kubernetes - 1.17.9
執(zhí)行如下命令, 查看 Kubernetes 版本:
- kubectl version
- Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:18:16Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
- Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:10:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
2.2 Containerd - 1.4.3
執(zhí)行如下命令, 查看 containerd 版本:
- containerd --version
- containerd github.com/containerd/containerd v1.4.3 269548fa27e0089a8b8278fc4fc781d7f65a939b
3. 鏡像管理工具 Podman
由于 Containerd 不支持 Docker API, 常見的 docker build、docker push 等命令在 Containerd 環(huán)境下無法使用。因此, 需要一種不依賴于 Docker, 針對 OCI 標準的鏡像構建和推送工具。
3.1 Podman 簡介
Podman 是一個實現(xiàn) OCI 標準的容器和鏡像管理工具, 同時也是 Daemonless, 不需要守護進程, 也支持非特權用戶使用。Podman 提供了類似 Docker CLI 的功能, 大部分情況下可以執(zhí)行 alias docker=podman 使用 Podman 替換 Docker , 而不會有任何問題。
3.2 Podman 安裝
- 安裝 Podman 命令行工具
安裝方法可以參考 Podman 的安裝指引[3]。這里以 CentOS 7 為例:
- curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_7/devel:kubic:libcontainers:stable.repo
- yum -y install podman
- 查看 Podman 版本
- podman --version
- podman version 3.0.1
- 查看命令參數(shù)
這里為了方便查閱, 貼出完整的幫助文檔。
- podman --help
- manage pods and images
- Usage:
- podman [flags]
- podman [command]
- Available Commands:
- attach Attach to a running container
- build Build an image using instructions from Containerfiles
- commit Create new image based on the changed container
- container Manage Containers
- cp Copy files/folders between a container and the local filesystem
- create Create but do not start a container
- diff Inspect changes on container's file systems
- events Show podman events
- exec Run a process in a running container
- export Export container's filesystem contents as a tar archive
- generate Generated structured data
- healthcheck Manage Healthcheck
- help Help about any command
- history Show history of a specified image
- image Manage images
- images List images in local storage
- import Import a tarball to create a filesystem image
- info Display podman system information
- init Initialize one or more containers
- inspect Display the configuration of a container or image
- kill Kill one or more running containers with a specific signal
- load Load an image from container archive
- login Login to a container registry
- logout Logout of a container registry
- logs Fetch the logs of a container
- mount Mount a working container's root filesystem
- network Manage Networks
- pause Pause all the processes in one or more containers
- play Play a pod
- pod Manage pods
- port List port mappings or a specific mapping for the container
- ps List containers
- pull Pull an image from a registry
- push Push an image to a specified destination
- restart Restart one or more containers
- rm Remove one or more containers
- rmi Removes one or more images from local storage
- run Run a command in a new container
- save Save image to an archive
- search Search registry for image
- start Start one or more containers
- stats Display a live stream of container resource usage statistics
- stop Stop one or more containers
- system Manage podman
- tag Add an additional name to a local image
- top Display the running processes of a container
- umount Unmounts working container's root filesystem
- unpause Unpause the processes in one or more containers
- unshare Run a command in a modified user namespace
- varlink Run varlink interface
- version Display the Podman Version Information
- volume Manage volumes
- wait Block on one or more containers
- Flags:
- --cgroup-manager string Cgroup manager to use (cgroupfs or systemd) (default "systemd")
- --cni-config-dir string Path of the configuration directory for CNI networks
- --config string Path of a libpod config file detailing container server configuration options
- --conmon string Path of the conmon binary
- --cpu-profile string Path for the cpu profiling results
- --events-backend string Events backend to use
- --help Help for podman
- --hooks-dir strings Set the OCI hooks directory path (may be set multiple times)
- --log-level string Log messages above specified level: debug, info, warn, error, fatal or panic (default "error")
- --namespace string Set the libpod namespace, used to create separate views of the containers and pods on the system
- --network-cmd-path string Path to the command for configuring the network
- --root string Path to the root directory in which data, including images, is stored
- --runroot string Path to the 'run directory' where all state information is stored
- --runtime string Path to the OCI-compatible binary used to run containers, default is /usr/bin/runc
- --storage-driver string Select which storage driver is used to manage storage of images and containers (default is overlay)
- --storage-opt stringArray Used to pass an option to the storage driver
- --syslog Output logging information to syslog as well as the console
- --tmpdir string Path to the tmp directory
- --trace Enable opentracing output
- -v, --version Version of podman
- Use "podman [command] --help" for more information about a command.
Podman 在覆蓋 Docker 命令的同時，增加了對 Pod 操作的支持。
3.3 主機上測試編譯并推送鏡像
在使用上可以直接將 docker 命令替換為 podman 即可。
- 編譯鏡像
- echo -e 'FROM busybox\nRUN echo "hello world"' | podman build -t docker.io/shaowenchen/myimage:latest -
- STEP 1: FROM busybox
- Getting image source signatures
- Copying blob 5c4213be9af9 done
- Copying config 491198851f done
- Writing manifest to image destination
- Storing signatures
- STEP 2: RUN echo "hello world"
- hello world
- STEP 3: COMMIT
- 4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
- 4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
- 查看編譯的鏡像
- podman images |grep shaowenchen
- docker.io/shaowenchen/myimage latest 4c8794086d9d 4 minutes ago 1.46 MB
- 登錄 DockerHub
- podman login docker.io -u shaowenchen
- Password:
- Login Succeeded!
- 推送鏡像
- podman push docker.io/shaowenchen/myimage:latest
- Getting image source signatures
- Copying blob 2893437c336c done
- Copying blob 84009204da3f done
- Copying config 4c8794086d done
- Writing manifest to image destination
- Storing signatures
4. Jenkns 中使用 Podman 構建鏡像
4.1 關鍵配置
- 使用 hostPath 將 /var/lib/containers 掛載到主機上
也可以使用 PVC，但是 PVC 可能需要加參數(shù)，見下文。
否則會有如下報錯:
- Error: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
- privileged 特權模式
否則會有如下報錯:
- Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
- Podman 參數(shù) --cgroup-manager=cgroupfs
在使用 PVC 作為存儲目錄時, 需要考慮這項配置。內核通過 Cgroup Driver 隔離一組資源, 可選的參數(shù)有 cgroupfs 和 systemd, 需要與集群環(huán)境保持一致, 因為他們共用一個內核。我的測試環(huán)境使用的是 cgroupfs 。
否則會有如下報錯:
- unable to write system event: "write unixgram @0011c->/run/systemd/journal/socket: sendmsg: no such file or directory
- Podman 參數(shù) --events-backend=file
這項配置通常不會 Block 執(zhí)行流程，如果你想保持日志更加干凈，可以添加。
否則會有如下報錯:
- unable to write system event: "write unixgram @0011c->/run/systemd/journal/socket: sendmsg: no such file or directory
4.2 示例一: 在 Jenkinsfile 中顯式使用 yaml 模板
這里將容器 /var/lib/containers 掛載到主機 /var/lib/containers 目錄，也可以掛載到主機 /tmp 目錄，并沒有強制要求。主機目錄只是提供一個存放數(shù)據(jù)的地方。
- pipeline {
- agent {
- kubernetes {
- yaml """
- apiVersion: v1
- kind: Pod
- spec:
- containers:
- - name: centos
- image: centos:7
- command:
- - cat
- tty: true
- securityContext:
- privileged: true
- volumeMounts:
- - name: storage
- mountPath: /var/lib/containers
- volumes:
- - name: storage
- hostPath:
- path: /var/lib/containers
- """
- }}
- stages {
- stage('Hello') {
- steps {
- container('centos') {
- sh '''
- curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/devel:kubic:libcontainers:stable.repo
- yum -y install podman
- echo -e 'FROM busybox\nRUN echo "hello world"' | podman --events-backend=file build -t docker.io/shaowenchen1/myimage:latest -
- podman --events-backend=file images |grep shaowenchen1
- '''
- }
- }
- }
- }
- }
Jenkins 的執(zhí)行日志:
- ···
- Dependency Updated:
- systemd.x86_64 0:219-78.el7_9.3 systemd-libs.x86_64 0:219-78.el7_9.3
- Complete!
- + podman --events-backend=file build -t docker.io/shaowenchen1/myimage:latest -
- + echo -e 'FROM busybox
- RUN echo "hello world"'
- STEP 1: FROM busybox
- STEP 2: RUN echo "hello world"
- --> Using cache 4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
- STEP 3: COMMIT docker.io/shaowenchen1/myimage:latest
- --> 4c8794086d9
- 4c8794086d9de80f71d182457b6d2cb18b9d61975b98bcd4cb167bdcabae5b2c
- + podman --events-backend=file images
- + grep shaowenchen1
- docker.io/shaowenchen1/myimage latest 4c8794086d9d 19 hours ago 1.46 MB
4.3 示例二: 使用 PVC 掛載 /var/lib/containers 目錄
在使用 PVC 存儲 Podman 數(shù)據(jù)時，需要提前準備好集群的存儲。
- 查看集群是否有默認的 StorageClass
- kubectl get sc
- NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
- openebs-device openebs.io/local Delete WaitForFirstConsumer false 19d
- openebs-hostpath (default) openebs.io/local Delete WaitForFirstConsumer false 19d
- openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 19d
- openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 19d
- 為 Podman 創(chuàng)建一個 PVC
這里的 namespace 需要與 Jenkins 中動態(tài) Agent 所在的 namespace 保持一致。
- cat <<EOF | kubectl apply -f -
- apiVersion: v1
- kind: PersistentVolumeClaim
- metadata:
- name: storage
- namespace: default
- spec:
- accessModes:
- - ReadWriteOnce
- resources:
- requests:
- storage: 30Gi
- EOF
- 查看創(chuàng)建的 PVC
- kubectl -n default get pvc
- NAME STATUS VOLUME CAPACITY ACCESS MODES
- storage Pending openebs-hostpath 11s
由于使用的是 WaitForFirstConsumer 模式，需要等到有 Pod 使用 PVC 時，才會綁定 PV。
- 創(chuàng)建 Jenkins 流水線執(zhí)行
- pipeline {
- agent {
- kubernetes {
- yaml """
- apiVersion: v1
- kind: Pod
- spec:
- containers:
- - name: centos
- image: centos:7
- command:
- - cat
- tty: true
- securityContext:
- privileged: true
- volumeMounts:
- - name: storage
- mountPath: /var/lib/containers
- volumes:
- - name: storage
- persistentVolumeClaim:
- claimName: storage
- """
- }}
- stages {
- stage('Hello') {
- steps {
- container('centos') {
- sh '''
- curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/devel:kubic:libcontainers:stable.repo
- yum -y install podman
- echo -e 'FROM busybox\nRUN echo "hello world"' | podman --events-backend=file build -t docker.io/shaowenchen2/myimage:latest -
- podman --events-backend=file images |grep shaowenchen2
- '''
- }
- }
- }
- }
- }
Jenkins 的執(zhí)行日志:
- ···
- Dependency Updated:
- systemd.x86_64 0:219-78.el7_9.3 systemd-libs.x86_64 0:219-78.el7_9.3
- Complete!
- + echo -e 'FROM busybox
- RUN echo "hello world"'
- + podman --events-backend=file build -t docker.io/shaowenchen2/myimage:latest -
- STEP 1: FROM busybox
- STEP 2: RUN echo "hello world"
- --> Using cache f4676f5b5e47a78970f2d97f4a5b77423f381e9742faae06d8c1a2d93bdb27c2
- STEP 3: COMMIT docker.io/shaowenchen2/myimage:latest
- --> f4676f5b5e4
- f4676f5b5e47a78970f2d97f4a5b77423f381e9742faae06d8c1a2d93bdb27c2
- + podman --events-backend=file images
- + grep shaowenchen2
- docker.io/shaowenchen2/myimage latest f4676f5b5e47 2 hours ago 1.46 MB
5. 總結
本文主要提供了一種在非 Docker 驅動的 Kubernetes 集群中，進行 CICD 鏡像構建的思路，使用 Podman 替換 Docker 。選擇 Podman 的原因是, 其使用方式更貼近 Docker，而 Buildah 需要用戶修改鏡像編譯指令，因為 Buildah 使用的是 buildah bud。
在生產(chǎn)實踐過程中，我們需要將 Podman 命令打包到 CI Agent 的基礎鏡像中。通過 alias docker=podman , 對基于 Docker 命令的流水線進行替換。
下面簡單總結一下，使用 Podman 的要點:
- 支持緩存。通過掛載 /var/lib/containers 目錄，可以緩存鏡像，并且可以根據(jù)業(yè)務劃分到不同目錄。
- 與 Docker 無縫替換。如果有 hook 的地方，可以用戶無感知地切換。
- 更加通用。針對 OCI 標準實現(xiàn)，不依賴具體組件。
- 特權模式。容器中運行 Podman 需要特權模式。容器套娃很難擺脫的運行模式。
6. 參考
https://github.com/kubernetes-sigs/cri-tools
http://docs.podman.io/en/latest/
參考資料
[1]《在 Kubernetes 上動態(tài)創(chuàng)建 Jenkins Slave》: https://www.chenshaowen.com/blog/creating-jenkins-slave-dynamically-on-kubernetes.html
[2]《如何在 Docker 中使用 Docker》: https://www.chenshaowen.com/blog/how-to-use-docker-in-docker.html
[3]Podman 的安裝指引: https://podman.io/getting-started/installation
責任編輯：武曉燕
來源：
問其
