k8s部署高可用配置中心apollo-手動(dòng)驗(yàn)證成功
前言
在前一篇文章中簡(jiǎn)單地介紹了《5分鐘通過(guò)docker快速部署并使用apollo配置中心》，用戶可以快速地了解到配置中心apollo的基本使用，但是在生產(chǎn)環(huán)境中我們往往需要高可用的部署配置中心，這樣我們就得有k8s來(lái)進(jìn)行部署。在apollo的官方文檔中有關(guān)于k8s部署的文章(https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes)，但是在執(zhí)行的過(guò)程中是會(huì)遇到一定的坑的。包括實(shí)際部署中遇到的，鏡像的制作、portal服務(wù)的多實(shí)例支持、ingress的創(chuàng)建都沒(méi)有說(shuō)到，本人在部署的過(guò)程中已經(jīng)全部解決以上問(wèn)題，希望可以幫到?jīng)]有部署過(guò)的用戶。
部署成功頁(yè)面
下面是部署完成后，訪問(wèn)apollo的登錄頁(yè)面
登錄頁(yè)面
輸入用戶名密碼：apollo/admin，部署環(huán)境完成portal頁(yè)面圖
部署完成apollo后頁(yè)面
k8s的dashboard部署頁(yè)面，本文部署了dev, fat, pro三個(gè)環(huán)境。
部署環(huán)境完成k8s后dashboard圖
部署過(guò)程
本文在部署的時(shí)候使用了當(dāng)前最新的apollo版本為：1.7.1，所以下面的所有構(gòu)建也是基于當(dāng)前版本的。
一、構(gòu)建鏡像
首先從git上下載源碼，可以從github下載：https://github.com/ctripcorp/apollo;也可以從gitee下載:https://gitee.com/nobodyiam/apollo，國(guó)內(nèi)的會(huì)快一點(diǎn)。然后進(jìn)入到目錄
- /scripts/apollo-on-kubernetes
去構(gòu)建鏡像。
1、 直接使用編譯的的包進(jìn)行安裝，獲取 apollo 壓縮包
可以直接從官網(wǎng)下載，因?yàn)間ithub實(shí)在是太慢了。建議直接從我的百度云下載。
A、下載比較慢，直接用我百度云
- 鏈接：https://pan.baidu.com/s/1eLL2ocYE1uzXcvzO2Y3dNg
- 提取碼：nfvm
B、從 https://github.com/ctripcorp/apollo/releases 下載預(yù)先打好的 java 包
- (1)進(jìn)入scripts/apollo-on-kubernetes/
- 執(zhí)行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-portal-1.7.1-github.zip
- (2)進(jìn)入scripts/apollo-on-kubernetes/
- 執(zhí)行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-adminservice-1.7.1-github.zip
- (3)進(jìn)入scripts/apollo-on-kubernetes/
- 執(zhí)行 wget https://github.com/ctripcorp/apollo/releases/download/v1.7.1/apollo-configservice-1.7.1-github.zip
2、解壓壓縮包, 獲取程序 jar 包
不要忘記重命名，把版本號(hào)去掉。
- 解壓 apollo-portal-1.7.1-github.zip
- 獲取 apollo-portal-1.7.1.jar, 重命名為 apollo-portal.jar, 放到 scripts/apollo-on-kubernetes/apollo-portal-server
- 解壓 apollo-adminservice-1.7.1-github.zip
- 獲取 apollo-adminservice-1.7.1.jar, 重命名為 apollo-adminservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-admin-server
- 解壓 apollo-configservice-1.7.1-github.zip
- 獲取 apollo-configservice-1.7.1.jar, 重命名為 apollo-configservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-config-server
3、構(gòu)建鏡像
注意：因?yàn)樵S多地方都要同時(shí)改，在構(gòu)建的時(shí)候要確定命名空間，我使用zizai。
要構(gòu)建如下的鏡像：alpine-bash-3.8-image，apollo-config-server，apollo-admin-server和apollo-portal-server，對(duì)應(yīng)的鏡像文件，在對(duì)應(yīng)的目錄下：
構(gòu)建鏡像要去到對(duì)應(yīng)的Dockerfile同級(jí)目錄下去執(zhí)行。
例如，去到scripts/apollo-on-kubernetes/apollo-config-server下執(zhí)行：
- docker build -t apollo-config-server:v1.7.1 .
注意，總共要構(gòu)建4個(gè)鏡像。整體的思路是：先構(gòu)建鏡像，然后打tag，再推到倉(cāng)庫(kù)里去。
在對(duì)應(yīng)目錄下，總結(jié)整體腳本如下：
- alpine-bash-3.8-image的鏡像：
- docker build -t alpine-bash:3.8 .
- docker tag alpine-bash:3.8 hub.thinkinpower.net/zizai/alpine-bash:3.8
- docker push hub.thinkinpower.net/zizai/alpine-bash:3.8
- apollo對(duì)應(yīng)的鏡像：
- docker build -t apollo-config-server:v1.7.1 .
- docker tag apollo-config-server:v1.7.1 hub.xx.net/zizai/apollo-config-server:v1.7.1
- docker push hub.xx.net/zizai/apollo-config-server:v1.7.1
- docker build -t apollo-admin-server:v1.7.1 .
- docker tag apollo-admin-server:v1.7.1 hub.xx.net/zizai/apollo-admin-server:v1.7.1
- docker push hub.xx.net/zizai/apollo-admin-server:v1.7.1
- docker build -t apollo-portal-server:v1.7.1 .
- docker tag apollo-portal-server:v1.7.1 hub.thinkinpower.net/zizai/apollo-portal-server:v1.7.1
- docker push hub.thinkinpower.net/zizai/apollo-portal-server:v1.7.1
二、部署apollo到kubernetes
1、創(chuàng)建數(shù)據(jù)庫(kù)腳本
說(shuō)明一下：
- 在實(shí)際的生產(chǎn)環(huán)境使用中，通過(guò)分布式存儲(chǔ)來(lái)實(shí)現(xiàn)的磁盤(pán)在mysql這種IO密集性應(yīng)用中，性能問(wèn)題會(huì)顯得非常突出。所以在實(shí)際應(yīng)用中，一般不會(huì)把mysql這種應(yīng)用直接放入kubernetes中管理，而是使用專(zhuān)用的服務(wù)器來(lái)獨(dú)立部署。而像web這種無(wú)狀態(tài)應(yīng)用依然會(huì)運(yùn)行在kubernetes當(dāng)中，這個(gè)時(shí)候web服務(wù)器要連接kubernetes管理之外的數(shù)據(jù)庫(kù)，有兩種方式：一是直接連接數(shù)據(jù)庫(kù)所在物理服務(wù)器IP，另一種方式就是借助kubernetes的Endpoints直接將外部服務(wù)器映射為kubernetes內(nèi)部的一個(gè)服務(wù)。
我們使用外面的mysql作為數(shù)據(jù)庫(kù)，不會(huì)將mysql部署到k8s里。
執(zhí)行目錄scripts/apollo-on-kubernetes/db下的腳本。Apollo服務(wù)端共需要兩個(gè)數(shù)據(jù)庫(kù)：ApolloPortalDB和ApolloConfigDB。每一個(gè)配置的config一個(gè)數(shù)據(jù)庫(kù)腳本，portal一個(gè)數(shù)據(jù)庫(kù)腳本。數(shù)據(jù)庫(kù)腳本見(jiàn)：https://github.com/ctripcorp/apollo/tree/master/scripts/apollo-on-kubernetes/db，在git里已經(jīng)有。如果apollo 開(kāi)啟了 4 個(gè)環(huán)境, 即 dev、test-alpha、test-beta、prod, 在MySQL 中導(dǎo)入 scripts/apollo-on-kubernetes/db 下的文件。
2、部署k8s的yaml文件
官網(wǎng)的yaml可以下載修改就可以了，因?yàn)槲矣米约旱膫}(cāng)庫(kù)的鏡像，并且多次測(cè)試，我主要有如下的修改：
(1)配置文件要?jiǎng)h除掉安全提示：
- securityContext:
- privileged: true
(2)添加倉(cāng)庫(kù)的密鑰：
- imagePullSecrets:
- - name: registry-harbor
(3)下面修改為每次都拉鏡像：Always
- imagePullPolicy: Always
(4)添加mysql的配置信息
我只用3個(gè)環(huán)境，需要修改的文件如圖：
因?yàn)樾薷牡帽容^多，我將在下面列出每一個(gè)文件。我只拿開(kāi)發(fā)環(huán)境apollo-env-dev的作為一個(gè)示例，其它的只是對(duì)應(yīng)修改就可以了。在執(zhí)行的時(shí)候，建議大家從下面的(3)、(2)、(1)的順序執(zhí)行下面的文件。
(1)、service-apollo-admin-server-dev.yaml
- ---
- # configmap for apollo-admin-server-dev
- kind: ConfigMap
- apiVersion: v1
- metadata:
- namespace: zizai
- name: configmap-apollo-admin-server-dev
- data:
- application-github.properties: |
- spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.zizai:3306/DevApolloConfigDB?characterEncoding=utf8
- spring.datasource.username = admin
- spring.datasource.password = mysql-admin
- eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-admin-server-dev
- labels:
- app: service-apollo-admin-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8090
- targetPort: 8090
- selector:
- app: pod-apollo-admin-server-dev
- type: ClusterIP
- sessionAffinity: ClientIP
- ---
- kind: Deployment
- apiVersion: apps/v1
- metadata:
- namespace: zizai
- name: deployment-apollo-admin-server-dev
- labels:
- app: deployment-apollo-admin-server-dev
- spec:
- replicas: 3
- selector:
- matchLabels:
- app: pod-apollo-admin-server-dev
- strategy:
- rollingUpdate:
- maxSurge: 1
- maxUnavailable: 1
- type: RollingUpdate
- template:
- metadata:
- labels:
- app: pod-apollo-admin-server-dev
- spec:
- imagePullSecrets: # dokcer倉(cāng)庫(kù)密碼，不需要的可以去掉
- - name: registry-harbor
- affinity:
- podAntiAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - weight: 100
- podAffinityTerm:
- labelSelector:
- matchExpressions:
- - key: app
- operator: In
- values:
- - pod-apollo-admin-server-dev
- topologyKey: kubernetes.io/hostname
- volumes:
- - name: volume-configmap-apollo-admin-server-dev
- configMap:
- name: configmap-apollo-admin-server-dev
- items:
- - key: application-github.properties
- path: application-github.properties
- initContainers:
- - image: hub.thinkinpower.net/zizai/alpine-bash:3.8
- imagePullPolicy: Always
- name: check-service-apollo-config-server-dev
- command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.zizai:8080"]
- containers:
- - image: hub.thinkinpower.net/zizai/apollo-admin-server:v1.7.1
- imagePullPolicy: Always
- name: container-apollo-admin-server-dev
- ports:
- - protocol: TCP
- containerPort: 8090
- volumeMounts:
- - name: volume-configmap-apollo-admin-server-dev
- mountPath: /apollo-admin-server/config/application-github.properties
- subPath: application-github.properties
- env:
- - name: APOLLO_ADMIN_SERVICE_NAME
- value: "service-apollo-admin-server-dev.zizai"
- readinessProbe:
- tcpSocket:
- port: 8090
- initialDelaySeconds: 10
- periodSeconds: 5
- livenessProbe:
- tcpSocket:
- port: 8090
- initialDelaySeconds: 120
- periodSeconds: 10
- dnsPolicy: ClusterFirst
- restartPolicy: Always
(2)、service-apollo-config-server-dev.yaml
- ---
- # configmap for apollo-config-server-dev
- kind: ConfigMap
- apiVersion: v1
- metadata:
- namespace: zizai
- name: configmap-apollo-config-server-dev
- data:
- application-github.properties: |
- spring.datasource.url = jdbc:mysql://service-mysql-for-apollo-dev-env.zizai:3306/DevApolloConfigDB?characterEncoding=utf8
- spring.datasource.username = admin
- spring.datasource.password = mysql-admin
- eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-meta-server-dev
- labels:
- app: service-apollo-meta-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8080
- targetPort: 8080
- selector:
- app: pod-apollo-config-server-dev
- type: ClusterIP
- clusterIP: None
- sessionAffinity: ClientIP
- ---
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-apollo-config-server-dev
- labels:
- app: service-apollo-config-server-dev
- spec:
- ports:
- - protocol: TCP
- port: 8080
- targetPort: 8080
- nodePort: 30002
- selector:
- app: pod-apollo-config-server-dev
- type: NodePort
- sessionAffinity: ClientIP
- ---
- kind: StatefulSet
- apiVersion: apps/v1
- metadata:
- namespace: zizai
- name: statefulset-apollo-config-server-dev
- labels:
- app: statefulset-apollo-config-server-dev
- spec:
- serviceName: service-apollo-meta-server-dev
- replicas: 3
- selector:
- matchLabels:
- app: pod-apollo-config-server-dev
- updateStrategy:
- type: RollingUpdate
- template:
- metadata:
- labels:
- app: pod-apollo-config-server-dev
- spec:
- imagePullSecrets: # dokcer倉(cāng)庫(kù)密碼，不需要的可以去掉
- - name: registry-harbor
- affinity:
- podAntiAffinity:
- preferredDuringSchedulingIgnoredDuringExecution:
- - weight: 100
- podAffinityTerm:
- labelSelector:
- matchExpressions:
- - key: app
- operator: In
- values:
- - pod-apollo-config-server-dev
- topologyKey: kubernetes.io/hostname
- volumes:
- - name: volume-configmap-apollo-config-server-dev
- configMap:
- name: configmap-apollo-config-server-dev
- items:
- - key: application-github.properties
- path: application-github.properties
- containers:
- - image: hub.thinkinpower.net/zizai/apollo-config-server:v1.7.1
- imagePullPolicy: Always
- name: container-apollo-config-server-dev
- ports:
- - protocol: TCP
- containerPort: 8080
- volumeMounts:
- - name: volume-configmap-apollo-config-server-dev
- mountPath: /apollo-config-server/config/application-github.properties
- subPath: application-github.properties
- env:
- - name: APOLLO_CONFIG_SERVICE_NAME
- value: "service-apollo-config-server-dev.zizai"
- readinessProbe:
- tcpSocket:
- port: 8080
- initialDelaySeconds: 10
- periodSeconds: 5
- livenessProbe:
- tcpSocket:
- port: 8080
- initialDelaySeconds: 120
- periodSeconds: 10
- dnsPolicy: ClusterFirst
- restartPolicy: Always
(3)、service-mysql-for-apollo-dev-env.yaml
- ---
- # 為外部 mysql 服務(wù)設(shè)置 service
- kind: Service
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-mysql-for-apollo-dev-env
- labels:
- app: service-mysql-for-apollo-dev-env
- spec:
- ports:
- - protocol: TCP
- port: 3306
- targetPort: 3306
- type: ClusterIP
- sessionAffinity: None
- ---
- kind: Endpoints
- apiVersion: v1
- metadata:
- namespace: zizai
- name: service-mysql-for-apollo-dev-env
- subsets:
- - addresses:
- - ip: 10.29.254.48
- ports:
- - protocol: TCP
- port: 3306
3、添加Ingress
官網(wǎng)給的示例是用k8s的NodePort來(lái)訪問(wèn)，但是在實(shí)際中，我們用會(huì)用Ingress來(lái)訪問(wèn)Portal。
注意：因?yàn)槲覀冊(cè)诓渴餻ortal的時(shí)候是多實(shí)例的，所以Ingress要添加保持會(huì)話，要不頁(yè)面會(huì)登錄不了，進(jìn)入不了portal頁(yè)面。具體為：
- metadata:
- annotations:
- nginx.ingress.kubernetes.io/affinity: "cookie" # 解決會(huì)話保持
- nginx.ingress.kubernetes.io/session-cookie-name: "route"
- nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
- nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
ingress的代碼示例為如下：
- apiVersion: extensions/v1beta1
- kind: Ingress
- metadata:
- name: zizai-apollo-portal
- namespace: zizai
- annotations:
- nginx.ingress.kubernetes.io/affinity: "cookie" # 解決會(huì)話保持
- nginx.ingress.kubernetes.io/session-cookie-name: "route"
- nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
- nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
- spec:
- rules:
- - host: zizai-apollo-portal.test.thinkinpower.net
- http:
- paths:
- - path: /
- backend:
- serviceName: service-apollo-portal-server
- servicePort: 8070
4、配置nginx
添加nginx訪問(wèn)到Ingress里：
nginx配置文件：zizai-apollo-portal.test.thinkinpower.net.conf
- server {
- listen 80;
- server_name zizai-apollo-portal.test.thinkinpower.net;
- access_log /data/logs/nginx/zizai-apollo-portal.test.thinkinpower.net.access.log main;
- error_log /data/logs/nginx/zizai-apollo-portal.test.thinkinpower.net.error.log;
- root /data/webapps/zizai-apollo-portal.test.thinkinpower.net/test/static;
- index index.html index.htm;
- client_max_body_size 50m;
- location / {
- proxy_set_header Host $http_host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- proxy_pass http://kubernetes; # 指向集群的
- }
- }
這樣就可以根據(jù)域名：http://zizai-apollo-portal.test.thinkinpower.net 訪問(wèn)portal了。
(1)創(chuàng)建的部署：
(2)創(chuàng)建的有部署副本：
(3)創(chuàng)建的service:
(4)創(chuàng)建的ingress:
(5)創(chuàng)建的配置字典：
三、簡(jiǎn)單使用
本文章將只有簡(jiǎn)單的使用，后面會(huì)有文章介紹詳細(xì)的使用，需要的可以在本文留言。
1、創(chuàng)建項(xiàng)目
2、選擇一個(gè)環(huán)境添加變量timeout
3、如果是在添加環(huán)境的過(guò)程中，刷新頁(yè)面會(huì)有“添加補(bǔ)缺環(huán)境”的提示
添加補(bǔ)缺環(huán)境
就這樣實(shí)現(xiàn)了k8s部署高可能的apollo。有建議的可以在評(píng)論區(qū)留言。謝謝!
