聊聊Kubernetes Without Kube-Proxy
本文轉(zhuǎn)載自微信公眾號(hào)「運(yùn)維開(kāi)發(fā)故事」,作者夕陽(yáng)西下。轉(zhuǎn)載本文請(qǐng)聯(lián)系運(yùn)維開(kāi)發(fā)故事公眾號(hào)。
引言
最近聽(tīng)好多朋友說(shuō),cilium很強(qiáng),勢(shì)必將成為主流。因其使用了ebpf,性能好,而且支持網(wǎng)絡(luò)策略。于是,決定花點(diǎn)時(shí)間學(xué)習(xí)一下。在通過(guò)官網(wǎng)文檔學(xué)習(xí)過(guò)程中,發(fā)現(xiàn)使用cilium作為CNI,居然可以不用安裝kube-proxy了。這讓我想起來(lái),之前在面試中被問(wèn)到的一個(gè)問(wèn)題,面試官問(wèn)我:kube-proxy是否可以不用安裝,是否有其他替代品。這下不就有答案了嘛。
順便吐槽一下,看官方文檔學(xué)習(xí),是真的有點(diǎn)難(畢竟全英文);不過(guò)還是建議大家看官方文檔學(xué)習(xí),不要翻譯成中文哦。那么接下來(lái),就由我來(lái)實(shí)操一下。
環(huán)境說(shuō)明
序號(hào) | 事項(xiàng) | 說(shuō)明 |
---|---|---|
1 | kubernetes version | v1.21.3 |
2 | cilium version | v1.10.3 |
3 | kubernetes安裝方式 | kubeadm |
4 | cilium組網(wǎng)模式 | vxlan |
5 | os | ubuntu 18.04 |
6 | kubernetes集群規(guī)模 | 1master、2node |
正文
在master上初始化集群,并通過(guò)添加--skip-phases=addon/kube-proxy參數(shù)忽略kube-proxy的安裝
- kubeadm init --apiserver-advertise-address=10.211.55.50 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.21.3 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all --skip-phases=addon/kube-proxy
在兩個(gè)node上執(zhí)行kubeadm join,加入集群
- kubeadm join 10.211.55.50:6443 --token ouez6j.02ms269v8i4psl7p --discovery-token-ca-cert-hash sha256:5fdafe0fe1adb3b60cd7bc33f033f028279a94a3944816424cc7f5bb498f6868
使用helm(v3)來(lái)安裝cilium。先添加cilium庫(kù)
- helm repo add cilium https://helm.cilium.io/
使用如下命令安裝cilium,添加kubeProxyReplacement=strict參數(shù)
- helm install cilium cilium/cilium --version 1.10.3 --namespace kube-system --set kubeProxyReplacement=strict --set k8sServiceHost=10.211.55.50 --set k8sServicePort=6443
檢查cilium安裝結(jié)果
- # 查看cilium agent,以daemonset方式部署在每個(gè)node節(jié)點(diǎn)上
- root@cilium1:/# kubectl -n kube-system get pods -l k8s-app=cilium
- NAME READY STATUS RESTARTS AGE
- cilium-8gwg2 1/1 Running 0 8m4s
- cilium-t9ffc 1/1 Running 0 8m39s
- cilium-x42r6 1/1 Running 0 8m16s
- # 查看cilum operator
- root@cilium1:~# kubectl get po -A -o wide |grep cilium-operator
- kube-system cilium-operator-5df88875-867hd 1/1 Running 5 41h 172.16.88.47 cilium3 <none> <none>
- kube-system cilium-operator-5df88875-9kx8c 1/1 Running 5 41h 172.16.88.253 cilium2 <none> <none>
檢查是否有kube-proxy組件??梢园l(fā)現(xiàn)并沒(méi)有該組件
- root@cilium1:/# kubectl get po -n kube-system
- NAME READY STATUS RESTARTS AGE
- cilium-8gwg2 1/1 Running 0 10m
- cilium-operator-5df88875-867hd 1/1 Running 5 27h
- cilium-operator-5df88875-9kx8c 1/1 Running 5 27h
- cilium-t9ffc 1/1 Running 0 11m
- cilium-x42r6 1/1 Running 0 10m
- coredns-59d64cd4d4-hbwg4 1/1 Running 1 27h
- coredns-59d64cd4d4-l2pmt 1/1 Running 1 27h
- etcd-cilium1 1/1 Running 2 27h
- kube-apiserver-cilium1 1/1 Running 2 27h
- kube-controller-manager-cilium1 1/1 Running 2 27h
- kube-scheduler-cilium1 1/1 Running 2 27h
檢查cilium狀態(tài),確保安裝正確
- root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium status
- Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
- KVStore: Ok Disabled
- Kubernetes: Ok 1.21 (v1.21.3) [linux/amd64]
- Kubernetes APIs: ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
- KubeProxyReplacement: Strict [eth0 10.211.55.50 (Direct Routing)]
- Cilium: Ok 1.10.3 (v1.10.3-4145278)
- NodeMonitor: Listening for events on 8 CPUs with 64x4096 of shared memory
- Cilium health daemon: Ok
- IPAM: IPv4: 2/254 allocated from 10.0.0.0/24,
- BandwidthManager: Disabled
- Host Routing: Legacy
- Masquerading: BPF [eth0] 10.0.0.0/24 [IPv4: Enabled, IPv6: Disabled]
- Controller Status: 20/20 healthy
- Proxy Status: OK, ip 10.0.0.41, 0 redirects active on ports 10000-20000
- Hubble: Ok Current/Max Flows: 817/4095 (19.95%), Flows/s: 0.95 Metrics: Disabled
- Encryption: Disabled
- Cluster health: 3/3 reachable (2021-08-07T15:29:05Z)
部署nginx來(lái)測(cè)試一下網(wǎng)絡(luò)聯(lián)通性
- # nginx deployment yaml文件
- cat deployment-nginx.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
- name: nginx
- spec:
- selector:
- matchLabels:
- run: nginx
- replicas: 4
- template:
- metadata:
- labels:
- run: nginx
- spec:
- containers:
- - name: nginx
- image: nginx
- ports:
- - containerPort: 80
- # 創(chuàng)建nginx deployment
- kubectl create -f deployment-nginx.yaml
- # 查看部署結(jié)果
- root@cilium1:/# kubectl get po -o wide
- NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
- nginx-649c4b9857-8f2v5 1/1 Running 1 26h 10.0.2.212 cilium2 <none> <none>
- nginx-649c4b9857-mhsxs 1/1 Running 1 26h 10.0.1.23 cilium3 <none> <none>
- nginx-649c4b9857-qw2jj 1/1 Running 1 26h 10.0.2.69 cilium2 <none> <none>
- nginx-649c4b9857-vj9w2 1/1 Running 1 26h 10.0.1.126 cilium3
創(chuàng)建一個(gè)nodeport service來(lái)驗(yàn)證service的可訪問(wèn)
- # 創(chuàng)建service
- kubectl expose deployment nginx --type=NodePort --port=80
- # 查看service
- root@cilium1:/# kubectl get svc nginx
- NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
- nginx NodePort 10.97.209.103 <none> 80:31126/TCP 26h
驗(yàn)證nodeport、cluster可訪問(wèn)
- # 通過(guò)nodeport
- root@cilium1:/# curl 127.0.0.1:31126
- <!DOCTYPE html>
- <html>
- <head>
- <title>Welcome to nginx!</title>
- # 通過(guò)service:port
- root@cilium1:/# curl 10.97.209.103
- <!DOCTYPE html>
- <html>
- <head>
- <title>Welcome to nginx!</title>
- # 檢查iptables 發(fā)現(xiàn)為空
- root@cilium1:/# iptables-save | grep KUBE-SVC
- root@cilium1:/#
- # 檢查ciliun service
- root@cilium1:/# kubectl exec -n kube-system cilium-t9ffc -- cilium service list
- Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
- ID Frontend Service Type Backend
- 1 10.96.0.1:443 ClusterIP 1 => 172.16.88.57:6443
- 2 10.96.0.10:9153 ClusterIP 1 => 10.0.2.229:9153
- 2 => 10.0.2.80:9153
- 3 10.96.0.10:53 ClusterIP 1 => 10.0.2.229:53
- 2 => 10.0.2.80:53
- 4 10.97.209.103:80 ClusterIP 1 => 10.0.2.69:80
- 2 => 10.0.1.23:80
- 3 => 10.0.1.126:80
- 4 => 10.0.2.212:80
- 5 172.16.88.57:31126 NodePort 1 => 10.0.2.69:80
- 2 => 10.0.1.23:80
- 3 => 10.0.1.126:80
- 4 => 10.0.2.212:80
- 6 0.0.0.0:31126 NodePort 1 => 10.0.2.69:80
- 2 => 10.0.1.23:80
- 3 => 10.0.1.126:80
- 4 => 10.0.2.212:80
從上面的安裝和測(cè)試結(jié)果來(lái),雖然我們沒(méi)有安裝k8s的kube-proxy組件,但是集群依然正常。說(shuō)明kube-proxy組件確實(shí)是可以被替代的。
總結(jié)
以上雖然完成了kubernetes without kube-proxy的搭建和測(cè)試工作,但還是有很多事情沒(méi)說(shuō)明。比如使用cilium的系統(tǒng)要求、cilium是什么、有幾種組網(wǎng)模式、網(wǎng)絡(luò)策略。不過(guò)請(qǐng)不要著急,期待我后續(xù)的文章。
參考
https://docs.cilium.io/en/v1.10/gettingstarted/kubeproxy-free/#kubernetes-without-kube-proxy
https://kubernetes.io/docs/concepts/cluster-administration/addons/
https://helm.sh/docs/intro/install/