基于K8S的StatefulSet部署MySQL集群
環(huán)境:
# 采用NFS存儲(chǔ)卷的方式 持久化存儲(chǔ)mysql數(shù)據(jù)目錄
需求:
展示如何使用 StatefulSet 控制器運(yùn)行一個(gè)有狀態(tài)的應(yīng)用程序。此例是多副本的 MySQL 數(shù)據(jù)庫(kù)。 示例應(yīng)用的拓?fù)浣Y(jié)構(gòu)有一個(gè)主服務(wù)器和多個(gè)副本,使用異步的基于行(Row-Based) 的數(shù)據(jù)復(fù)制。

- 搭建一個(gè)“主從復(fù)制”(Maser-Slave Replication)的 MySQL 集群
- 存在一個(gè)主節(jié)點(diǎn)【master】,有多個(gè)從節(jié)點(diǎn)【slave】
- 從節(jié)點(diǎn)可以水平拓展
- 所有的寫(xiě)操作,只能在主節(jié)點(diǎn)上執(zhí)行
- 讀操作可以在所有節(jié)點(diǎn)上執(zhí)行
一、部署NFS服務(wù)器
- #服務(wù)器安裝nfs服務(wù),提供nfs存儲(chǔ)功能
- 1、安裝nfs-utils
- yum install nfs-utils (centos)
- 或者 apt-get install nfs-kernel-server (ubuntu)
- 2、啟動(dòng)服務(wù)
- systemctl enable nfs-server
- systemctl start nfs-server
- 3、創(chuàng)建共享目錄完成共享配置
- mkdir /home/nfs #創(chuàng)建共享目錄
- 4、編輯共享配置
- vim /etc/exports
- #語(yǔ)法格式: 共享文件路徑 客戶(hù)機(jī)地址(權(quán)限) #這里的客戶(hù)機(jī)地址可以是IP,網(wǎng)段,域名,也可以是任意*
- /home/nfs *(rw,async,no_root_squash)
- #服務(wù)器安裝nfs服務(wù),提供nfs存儲(chǔ)功能
- 1、安裝nfs-utils
- yum install nfs-utils (centos)
- 或者 apt-get install nfs-kernel-server (ubuntu)
- 2、啟動(dòng)服務(wù)
- systemctl enable nfs-server
- systemctl start nfs-server
- 3、創(chuàng)建共享目錄完成共享配置
- mkdir /home/nfs #創(chuàng)建共享目錄
- 4、編輯共享配置
- vim /etc/exports
- #語(yǔ)法格式: 共享文件路徑 客戶(hù)機(jī)地址(權(quán)限) #這里的客戶(hù)機(jī)地址可以是IP,網(wǎng)段,域名,也可以是任意*
- /home/nfs *(rw,async,no_root_squash)
- 服務(wù)自檢命令
- exportfs -arv
- 5、重啟服務(wù)
- systemctl restart nfs-server
- 6、本機(jī)查看nfs 共享目錄
- #showmount -e 服務(wù)器IP地址 (如果提示命令不存在,則需要yum install showmount)
- showmount -e 127.0.0.1
- /home/nfs *
- 7、客戶(hù)端模擬掛載[所有k8s的節(jié)點(diǎn)都需要安裝客戶(hù)端]
- [root@master-1 ~]# yum install nfs-utils (centos)
- 或者 apt-get install nfs-common (ubuntu)
- [root@master-1 ~]# mkdir /test
- [root@master-1 ~]# mount -t nfs 172.16.201.209:/home/nfs /test
- #取消掛載
- [root@master-1 ~]# umount /test
二、配置PV 動(dòng)態(tài)供給(NFS StorageClass),創(chuàng)建pvc
部署NFS實(shí)現(xiàn)自動(dòng)創(chuàng)建PV插件: 一共設(shè)計(jì)到4個(gè)yaml 文件 ,官方的文檔有詳細(xì)的說(shuō)明
https://github.com/kubernetes-incubator/external-storage


- root@k8s-master1:~ # mkdir /root/pvc
- root@k8s-master1:~ # cd /root/pvc
創(chuàng)建rbac.yaml 文件
- root@k8s-master1:pvc # cat rbac.yaml
- kind: ServiceAccount
- apiVersion: v1
- metadata:
- name: nfs-client-provisioner
- ---
- kind: ClusterRole
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: nfs-client-provisioner-runner
- rules:
- - apiGroups: [""]
- resources: ["persistentvolumes"]
- verbs: ["get", "list", "watch", "create", "delete"]
- - apiGroups: [""]
- resources: ["persistentvolumeclaims"]
- verbs: ["get", "list", "watch", "update"]
- - apiGroups: ["storage.k8s.io"]
- resources: ["storageclasses"]
- verbs: ["get", "list", "watch"]
- - apiGroups: [""]
- resources: ["events"]
- verbs: ["create", "update", "patch"]
- ---
- kind: ClusterRoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: run-nfs-client-provisioner
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- namespace: default
- roleRef:
- kind: ClusterRole
- name: nfs-client-provisioner-runner
- apiGroup: rbac.authorization.k8s.io
- ---
- kind: Role
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- rules:
- - apiGroups: [""]
- resources: ["endpoints"]
- verbs: ["get", "list", "watch", "create", "update", "patch"]
- ---
- kind: RoleBinding
- apiVersion: rbac.authorization.k8s.io/v1
- metadata:
- name: leader-locking-nfs-client-provisioner
- subjects:
- - kind: ServiceAccount
- name: nfs-client-provisioner
- # replace with namespace where provisioner is deployed
- namespace: default
- roleRef:
- kind: Role
- name: leader-locking-nfs-client-provisioner
- apiGroup: rbac.authorization.k8s.io
創(chuàng)建deployment.yaml 文件
官方默認(rèn)的鏡像地址,國(guó)內(nèi)可能無(wú)法下載,可以使用 image:
fxkjnj/nfs-client-provisioner:latest
#定義NFS 服務(wù)器的地址,共享目錄名稱(chēng)
- root@k8s-master1:pvc # cat deployment.yaml
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: nfs-client-provisioner
- ---
- kind: Deployment
- apiVersion: apps/v1
- metadata:
- name: nfs-client-provisioner
- spec:
- replicas: 1
- strategy:
- type: Recreate
- selector:
- matchLabels:
- app: nfs-client-provisioner
- template:
- metadata:
- labels:
- app: nfs-client-provisioner
- spec:
- serviceAccountName: nfs-client-provisioner
- containers:
- - name: nfs-client-provisioner
- image: fxkjnj/nfs-client-provisioner:latest
- volumeMounts:
- - name: nfs-client-root
- mountPath: /persistentvolumes
- env:
- - name: PROVISIONER_NAME
- value: fuseim.pri/ifs
- - name: NFS_SERVER
- value: 172.16.201.209
- - name: NFS_PATH
- value: /home/nfs
- volumes:
- - name: nfs-client-root
- nfs:
- server: 172.16.201.209
- path: /home/nfs
創(chuàng)建class.yaml
- root@k8s-master1:pvc # cat class.yaml
- apiVersion: storage.k8s.io/v1
- kind: StorageClass
- metadata:
- name: managed-nfs-storage
- provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
- parameters:
- archiveOnDelete: "true"
部署
- root@k8s-master1:pvc # kubectl apply -f .
- #查看存儲(chǔ)卷
- root@k8s-master1:pvc # kubectl get sc
- NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
- managed-nfs-storage fuseim.pri/ifs Delete Immediate false 25h
三、編寫(xiě)mysql 相關(guān)yaml文件
MySQL 示例部署包含一個(gè) ConfigMap、兩個(gè) Service 與一個(gè) StatefulSet。
ConfigMap:
vim mysql-configmap.yaml
- apiVersion: v1
- kind: ConfigMap
- metadata:
- name: mysql
- labels:
- app: mysql
- data:
- master.cnf: |
- # Apply this config only on the master.
- [mysqld]
- log-bin
- slave.cnf: |
- # Apply this config only on slaves.
- [mysqld]
- super-read-only
說(shuō)明:
在這里,我們定義了 master.cnf 和 slave.cnf 兩個(gè) MySQL 的配置文件
- master.cnf 開(kāi)啟了log-bin,可以使用二進(jìn)制日志文件的方式進(jìn)行主從復(fù)制.
- slave.cnf 開(kāi)啟了 super-read-only ,表示從節(jié)點(diǎn)只接受主節(jié)點(diǎn)的數(shù)據(jù)同步的所有寫(xiě)的操作,拒絕其他的寫(xiě)入操作,對(duì)于用戶(hù)來(lái)說(shuō)就是只讀的
- master.cnf 和 slave.cnf 已配置文件的形式掛載到容器的目錄中
Service:
vim mysql-services.yaml
- # Headless service for stable DNS entries of StatefulSet members.
- apiVersion: v1
- kind: Service
- metadata:
- name: mysql
- labels:
- app: mysql
- spec:
- ports:
- - name: mysql
- port: 3306
- clusterIP: None
- selector:
- app: mysql
- ---
- # Client service for connecting to any MySQL instance for reads.
- # For writes, you must instead connect to the master: mysql-0.mysql.
- apiVersion: v1
- kind: Service
- metadata:
- name: mysql-read
- labels:
- app: mysql
- spec:
- ports:
- - name: mysql
- port: 3306
- selector:
- app: mysql
說(shuō)明:
clusterIP: None,使用無(wú)頭服務(wù) Headless Service(相比普通Service只是將spec.clusterIP定義為None,也就是沒(méi)有clusterIP,直接使用endport 來(lái)通信)來(lái)維護(hù)Pod網(wǎng)絡(luò)身份,會(huì)為每個(gè)Pod分配一個(gè)數(shù)字編號(hào)并且按照編號(hào)順序部署。還需要在StatefulSet添加serviceName: “mysql”字段指定StatefulSet控制器
另外statefulset控制器網(wǎng)絡(luò)標(biāo)識(shí),體現(xiàn)在主機(jī)名和Pod A記錄:
• 主機(jī)名:<statefulset名稱(chēng)>-<編號(hào)>
例如: mysql-0
• Pod DNS A記錄:<statefulset名稱(chēng)-編號(hào)>.<service-name> .<namespace>.svc.cluster.local (POD 之間通過(guò)DNS A 記錄互相通信)
例如:
mysql-0.mysql.default.svc.cluster.local
StatefulSet:
vim mysql-statefulset.yaml
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: mysql
- spec:
- selector:
- matchLabels:
- app: mysql
- serviceName: mysql
- replicas: 3
- template:
- metadata:
- labels:
- app: mysql
- spec:
- initContainers:
- - name: init-mysql
- image: mysql:5.7
- command:
- - bash
- - "-c"
- - |
- set -ex
- # Generate mysql server-id from pod ordinal index.
- [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
- ordinal=${BASH_REMATCH[1]}
- echo [mysqld] > /mnt/conf.d/server-id.cnf
- # Add an offset to avoid reserved server-id=0 value.
- echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
- # Copy appropriate conf.d files from config-map to emptyDir.
- if [[ $ordinal -eq 0 ]]; then
- cp /mnt/config-map/master.cnf /mnt/conf.d/
- else
- cp /mnt/config-map/slave.cnf /mnt/conf.d/
- fi
- volumeMounts:
- - name: conf
- mountPath: /mnt/conf.d
- - name: config-map
- mountPath: /mnt/config-map
- - name: clone-mysql
- image: fxkjnj/xtrabackup:1.0
- command:
- - bash
- - "-c"
- - |
- set -ex
- # Skip the clone if data already exists.
- [[ -d /var/lib/mysql/mysql ]] && exit 0
- # Skip the clone on master (ordinal index 0).
- [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
- ordinal=${BASH_REMATCH[1]}
- [[ $ordinal -eq 0 ]] && exit 0
- # Clone data from previous peer.
- ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
- # Prepare the backup.
- xtrabackup --prepare --target-dir=/var/lib/mysql
- volumeMounts:
- - name: data
- mountPath: /var/lib/mysql
- subPath: mysql
- - name: conf
- mountPath: /etc/mysql/conf.d
- containers:
- - name: mysql
- image: mysql:5.7
- env:
- - name: MYSQL_ALLOW_EMPTY_PASSWORD
- value: "1"
- ports:
- - name: mysql
- containerPort: 3306
- volumeMounts:
- - name: data
- mountPath: /var/lib/mysql
- subPath: mysql
- - name: conf
- mountPath: /etc/mysql/conf.d
- resources:
- requests:
- cpu: 500m
- memory: 1Gi
- livenessProbe:
- exec:
- command: ["mysqladmin", "ping"]
- initialDelaySeconds: 30
- periodSeconds: 10
- timeoutSeconds: 5
- readinessProbe:
- exec:
- # Check we can execute queries over TCP (skip-networking is off).
- command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
- initialDelaySeconds: 5
- periodSeconds: 2
- timeoutSeconds: 1
- - name: xtrabackup
- image: fxkjnj/xtrabackup:1.0
- ports:
- - name: xtrabackup
- containerPort: 3307
- command:
- - bash
- - "-c"
- - |
- set -ex
- cd /var/lib/mysql
- # Determine binlog position of cloned data, if any.
- if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
- # XtraBackup already generated a partial "CHANGE MASTER TO" query
- # because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
- cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
- # Ignore xtrabackup_binlog_info in this case (it's useless).
- rm -f xtrabackup_slave_info xtrabackup_binlog_info
- elif [[ -f xtrabackup_binlog_info ]]; then
- # We're cloning directly from master. Parse binlog position.
- [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
- rm -f xtrabackup_binlog_info xtrabackup_slave_info
- echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
- MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
- fi
- # Check if we need to complete a clone by starting replication.
- if [[ -f change_master_to.sql.in ]]; then
- echo "Waiting for mysqld to be ready (accepting connections)"
- until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
- echo "Initializing replication from clone position"
- mysql -h 127.0.0.1 \
- -e "$(<change_master_to.sql.in), \
- MASTER_HOST='mysql-0.mysql', \
- MASTER_USER='root', \
- MASTER_PASSWORD='', \
- MASTER_CONNECT_RETRY=10; \
- START SLAVE;" || exit 1
- # In case of container restart, attempt this at-most-once.
- mv change_master_to.sql.in change_master_to.sql.orig
- fi
- # Start a server to send backups when requested by peers.
- exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
- "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
- volumeMounts:
- - name: data
- mountPath: /var/lib/mysql
- subPath: mysql
- - name: conf
- mountPath: /etc/mysql/conf.d
- resources:
- requests:
- cpu: 100m
- memory: 100Mi
- volumes:
- - name: conf
- emptyDir: {}
- - name: config-map
- configMap:
- name: mysql
- volumeClaimTemplates:
- - metadata:
- name: data
- spec:
- storageClassName: "managed-nfs-storage"
- accessModes: ["ReadWriteOnce"]
- resources:
- requests:
- storage: 0.5Gi
說(shuō)明:
- 使用xtrbackup 工具進(jìn)行容器初始化數(shù)據(jù)的備份,https://www.toutiao.com/i6999565563710292484
- 使用linux 自帶的ncat 工具進(jìn)行容器初始化數(shù)據(jù)的拷貝[使用ncat指令,遠(yuǎn)程地從前一個(gè)節(jié)點(diǎn)拷貝數(shù)據(jù)到本地] https://www.cnblogs.com/chengd/p/7565280.html
- 使用mysql的binlog 主從復(fù)制 來(lái)保證主從之間的數(shù)據(jù)一致
- 利用pod的主機(jī)名的序號(hào)來(lái)判斷當(dāng)前節(jié)點(diǎn)為主節(jié)點(diǎn)還是從節(jié)點(diǎn),再根據(jù)對(duì)于節(jié)點(diǎn)拷貝不同的配置文件到指定位置
- 使用mysqladmin的ping 作為數(shù)據(jù)庫(kù)的健康檢測(cè)方式
- 使用nfs存儲(chǔ)的 PV 動(dòng)態(tài)供給(StorageClass),持久化mysql的數(shù)據(jù)文件
四、部署并測(cè)試
- root@k8s-master1:~/kubernetes/mysql# ll
- total 24
- drwxr-xr-x 2 root root 4096 Nov 3 16:42 ./
- drwxr-xr-x 8 root root 4096 Nov 3 13:33 ../
- -rw-r--r-- 1 root root 278 Nov 2 22:15 mysql-configmap.yaml
- -rw-r--r-- 1 root root 556 Nov 2 22:08 mysql-services.yaml
- -rw-r--r-- 1 root root 5917 Nov 3 14:22 mysql-statefulset.yaml
- root@k8s-master1:~/kubernetes/mysql# kubectl apply -f .
- configmap/mysql create
- service/mysql create
- service/mysql-read create
- statefulset.apps/mysql create
- #動(dòng)態(tài)追蹤查看Pod的狀態(tài):
- root@k8s-master1:~/kubernetes/mysql# kubectl get pods -l app=mysql --watch
- NAME READY STATUS RESTARTS AGE
- mysql-0 2/2 Running 0 3h12m
- mysql-1 2/2 Running 0 3h11m
- mysql-2 2/2 Running 0 3h10m
可以看到,StatefulSet 啟動(dòng)成功后,將會(huì)有三個(gè) Pod 運(yùn)行。
接下來(lái),我們可以嘗試向這個(gè) MySQL 集群發(fā)起請(qǐng)求,執(zhí)行一些 SQL 操作來(lái)驗(yàn)證它是否正常:
- kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\
- mysql -h mysql-0.mysql <<EOF
- CREATE DATABASE test;
- CREATE TABLE test.messages (message VARCHAR(250));
- INSERT INTO test.messages VALUES ('hello');
- EOF
如上所示,我們通過(guò)啟動(dòng)一個(gè)容器,使用 MySQL client 執(zhí)行了創(chuàng)建數(shù)據(jù)庫(kù)和表、以及插入數(shù)據(jù)的操作。需要注意的是,我們連接的 MySQL 的地址必須是 mysql-0.mysql(即:Master 節(jié)點(diǎn)的 DNS A 記錄, 因?yàn)镻OD 之間通過(guò)DNS A 記錄互相通信)只有 Master 節(jié)點(diǎn)才能處理寫(xiě)操作。
而通過(guò)連接 mysql-read 這個(gè) Service,我們就可以用 SQL 進(jìn)行讀操作,如下所示:
- kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never --\
- mysql -h mysql-read -e "SELECT * FROM test.messages"
- #你應(yīng)該獲得如下輸出:
- Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false
- +---------+
- | message |
- +---------+
- | hello |
- +---------+
- pod "mysql-client" deleted
或者:
- root@k8s-master1:~/kubernetes/mysql# kubectl run -it --rm --image=mysql:5.7 --restart=Never mysql-client -- mysql -h mysql-read
- If you don't see a command prompt, try pressing enter.
- Welcome to the MySQL monitor. Commands end with ; or \g.
- Your MySQL connection id is 7251
- Server version: 5.7.36 MySQL Community Server (GPL)
- Copyright (c) 2000, 2021, Oracle and/or its affiliates.
- Oracle is a registered trademark of Oracle Corporation and/or its
- affiliates. Other names may be trademarks of their respective
- owners.
- Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
- mysql> SELECT * FROM test.messages;
- +---------+
- | message |
- +---------+
- | hello |
- +---------+
- 1 row in set (0.00 sec)
- mysql>