Kubernetes 集群備份工具 velero 使用指南
作者：whale_life
Velero 對(duì)存儲(chǔ)的支持較好，可以支持很多種存儲(chǔ)資源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等。
??想了解更多關(guān)于開源的內(nèi)容，請(qǐng)?jiān)L問：??
簡(jiǎn)介
Velero 是一款可以安全的備份、恢復(fù)和遷移 Kubernetes 集群資源和持久卷等資源的備份恢復(fù)軟件。
Velero 實(shí)現(xiàn)的 kubernetes 資源備份能力，可以輕松實(shí)現(xiàn) Kubernetes 集群的數(shù)據(jù)備份和恢復(fù)、復(fù)制 kubernetes 集群資源到其他 kubernetes 集群或者快速?gòu)?fù)制生產(chǎn)環(huán)境到測(cè)試環(huán)境等功能，這種備份就類似于把資源的 yaml 文件進(jìn)行整體備份，從而保障資源的完整性。
Velero 對(duì)存儲(chǔ)的支持較好，可以支持很多種存儲(chǔ)資源，比如 AWS S3、Azure Blob、Google Cloud Storage、Alibaba Cloud OSS、Swift、MinIO 等等.
流程圖
注意：本文檔采用 MinIO 作為后端存儲(chǔ)。
Docker 部署 MinIO
因?yàn)槲覀円ＷC存儲(chǔ)和集群隔離，這樣能保證在集群出問題的時(shí)候，不影響備份，所以就將 MinIO 部署在其他服務(wù)器上，與 Kubernetes 環(huán)境隔離。避免因意外情況宕機(jī)的時(shí)候影響更多。
mkdir -pv /data/{minio,config}
docker pull minio/minio
docker pull minio/mc:latest
docker run -d -p 9000:9000 -p 9090:9090 --name=minio --restart=always -e "MINIO_ROOT_USER=admin" -e "MINIO_ROOT_PASSWORD=admin123456" -v /data/minio:/data -v /data/config:/root/.minio minio/minio server /data --console-address ":9090" --address ":9000"
# 對(duì)minio 授權(quán)
docker run -it --entrypoint=/bin/sh -v /data/config:/root/.minio minio/mc
mc config host add velero http://127.0.0.1:9000 admin admin123456 --api S3v4
mc mb -p velero/velero
注意，mc 使用的是 api 端口，并不是 console 界面端口，所以 mc 在授權(quán)的時(shí)候需要使用 API 端口。
然后瀏覽器通過配置的賬號(hào)和 password 即可登錄：
安裝 velero 在 k8s master 上
下載 velero：
https://github.com/vmware-tanzu/velero/releases/download/v1.9.0/velero-v1.9.0-linux-amd64.tar.gz。
1、創(chuàng)建密鑰并安裝velero
cat > credentials-velero <<EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = admin123456
EOF
# 安裝velero，配置自動(dòng)補(bǔ)全
cp velero /usr/local/bin/
echo 'source <(velero completion bash)' >> /etc/profile && source /etc/profile
2、K8s集群安裝velero
# 需要提前下載鏡像 velero/velero-plugin-for-aws:v1.2.1 和velero/velero:v1.9.0
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.2.1 \
--bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.0.2.196:9000
安裝時(shí)產(chǎn)生的日志。
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ? Use 'kubectl logs deployment/velero -n velero' to view the status.
驗(yàn)證是否安裝完畢。
[root@master ~]# kubectl get pod -n velero
NAME READY STATUS RESTARTS AGE
velero-cf79784ff-zdfp9 1/1 Running 0 47s
[root@master ~]# kubectl api-versions | grep velero
velero.io/v1
3、velero 命令
velero create backup NAME [flags]
# 剔除 namespace
--exclude-namespaces stringArray namespaces to exclude from the backup
# 剔除資源類型
--exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
# 包含集群資源類型
--include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup
# 包含 namespace
--include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *)
# 包含 namespace 資源類型
--include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
# 給這個(gè)備份加上標(biāo)簽
--labels mapStringString labels to apply to the backup
-o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
# 對(duì)指定標(biāo)簽的資源進(jìn)行備份
-l, --selector labelSelector only back up resources matching this label selector (default <none>)
# 對(duì) PV 創(chuàng)建快照
--snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup
# 指定備份的位置
--storage-location string location in which to store the backup
# 備份數(shù)據(jù)多久刪掉
--ttl duration how long before the backup can be garbage collected (default 720h0m0s)
# 指定快照的位置，也就是哪一個(gè)公有云驅(qū)動(dòng)
--volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored
4、測(cè)試
velero 在安裝包中有一個(gè)測(cè)試demo，我們可以利用測(cè)試demo來進(jìn)行測(cè)試驗(yàn)證。
[root@master ~]# cd velero-v1.9.0-linux-amd64/examples/nginx-app/
[root@master nginx-app]# ls
base.yaml README.md with-pv.yaml
# 注意修改 with-pv 中的 storageclass
storageClassName: longhorn
# 修改 svc 類型為 ClusterIP
type: ClusterIP
[root@master nginx-app]# kubectl apply -f with-pv.yaml
namespace/nginx-example created
persistentvolumeclaim/nginx-logs created
deployment.apps/nginx-deployment created
service/my-nginx created
[root@master nginx-app]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 79s
[root@master nginx-app]# kubectl get all -n nginx-example
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-6779884c68-4nv2s 2/2 Running 0 2m16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-nginx ClusterIP 10.108.189.178 <none> 80/TCP 2m16s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-deployment 1/1 1 1 2m16s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-deployment-6779884c68 1 1 1 2m16s
5、備份
[root@master nginx-app]# velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
[root@master nginx-app]# velero backup describe nginx-backup
Name: nginx-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.23.5
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=23
Phase: Failed (run `velero backup logs nginx-backup` for more information)
Errors: 0
Warnings: 0
Namespaces:
Included: nginx-example
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2022-06-30 17:44:49 +0800 CST
Completed: 2022-06-30 17:44:49 +0800 CST
Expiration: 2022-07-30 17:44:49 +0800 CST
Velero-Native Snapshots: <none included>
[root@master nginx-app]# velero backup logs nginx-backup
An error occurred: gzip: invalid header
登錄 minio 查看，備份記錄。
6、恢復(fù)測(cè)試
刪除 nginx-app：
[root@master velero-v1.9.0-linux-amd64]# kubectl delete -f examples/nginx-app/base.yaml恢復(fù) nginx：
[root@master velero-v1.9.0-linux-amd64]# velero restore create --from-backup nginx-backup --wait
Restore request "nginx-backup-20220630192521" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe nginx-backup-20220630192521` and `velero restore logs nginx-backup-20220630192521`.
[root@master velero-v1.9.0-linux-amd64]# kubectl get pods -n nginx-example
NAME READY STATUS RESTARTS AGE
nginx-deployment-6779884c68-4nv2s 2/2 Running 0 27s
恢復(fù)會(huì)新建一個(gè) restores。
目錄注意：??velero restore??? 恢復(fù)不會(huì)覆蓋??已有的資源???，只恢復(fù)當(dāng)前集群中??不存在的資源??。已有的資源不會(huì)回滾到之前的版本，如需要回滾，需在restore之前提前刪除現(xiàn)有的資源。
備份所有命名空間：
velero backup logs k8s-all
7、定時(shí)備份
比較推薦做定時(shí)全量備份，這個(gè)能保證我們是一直全量最新備份的狀態(tài)。
# 每天12點(diǎn)備份一次
velero create schedule k8s-all --schedule="0 0 * * *"
查看備份：
velero get backup #備份查看
velero get schedule #查看定時(shí)備份
velero get restore #查看已有的恢復(fù)
velero get plugins #查看插件
責(zé)任編輯：jianghua
來源：
51CTO開源基礎(chǔ)軟件社區(qū)
