Kubernetes in kubernetes 搭建高可用環境
4 likes1,125 views
Kubernetes in Kubernetes: 搭建高可用環境 discusses running Kubernetes itself on Kubernetes to provide a self-hosted Kubernetes cluster with high availability. It describes how Bootkube can be used to bootstrap an initial control plane and deploy a full self-hosted control plane. It also discusses options for exposing services externally like NodePort, load balancers, Ingress, and using Keepalived for high availability virtual IPs. The document emphasizes that self-hosting Kubernetes makes operations easier and supports disaster recovery for the control plane.
![Copyright 2017 ITRI 工業技術研究院
Self-hosted Control Plane
16
[root@centos7 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kube-apiserver-kkxq9 1/1 Running 0 1d
kube-controller-manager-2953862963-t7m1q 1/1 Running 0 1d
kube-controller-manager-2953862963-wlsjp 1/1 Running 0 1d
kube-dns-2431531914-gqnnd 3/3 Running 0 1d
kube-flannel-wnk1j 2/2 Running 0 1d
kube-flannel-xcsx2 2/2 Running 0 1d
kube-flannel-xrksj 2/2 Running 1 1d
kube-proxy-04x11 1/1 Running 0 1d
kube-proxy-11n6t 1/1 Running 0 1d
kube-proxy-1zlgz 1/1 Running 0 1d
kube-scheduler-1873817829-4c7mm 1/1 Running 1 1d
kube-scheduler-1873817829-pmp0n 1/1 Running 0 1d
pod-checkpointer-11q7g 1/1 Running 0 1d
pod-checkpointer-11q7g-10.201.3.6 1/1 Running 0 1d](https://image.slidesharecdn.com/kubernetesinkubernetes-170922033835/85/Kubernetes-in-kubernetes-16-320.jpg)
![Copyright 2017 ITRI 工業技術研究院
Self-hosted Control Plane (cont.)
17
[root@centos7 ~]# kubectl get deployment -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-controller-manager 2 2 2 2 2d
kube-dns 1 1 1 1 2d
kube-scheduler 2 2 2 2 2d
[root@centos7 ~]# kubectl get svc -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 2d
[root@centos7 ~]# kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 2d
kube-flannel 3 3 3 3 3 <none> 2d
kube-proxy 3 3 3 3 3 <none> 2d
pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 2d
[root@centos7 ~]# kubectl get secret -n kube-system
NAME TYPE DATA AGE
kube-apiserver Opaque 7 2d
kube-controller-manager Opaque 2 2d](https://image.slidesharecdn.com/kubernetesinkubernetes-170922033835/85/Kubernetes-in-kubernetes-17-320.jpg)
Kubernetes in kubernetes 搭建高可用環境
- 1. Kubernetes in Kubernetes： 搭建高可用環境 資訊與通訊研究所 蔣是文 Mac Chiang 交通大學資工所 鄭偉聖 Sam Zheng
- 2. Copyright 2017 ITRI 工業技術研究院 Agenda • Cluster Management Issues • Self-hosted Kubernetes • Service with High Availability • Summary 2
- 3. Copyright 2017 ITRI 工業技術研究院 Microservices Challenges 3 • A lots of microservices/components • Zero downtime deployment • Incremental roll out of features faster • Improves the ability scale efficiently
- 4. Copyright 2017 ITRI 工業技術研究院 Kubernetes Cluster 4 • Scheduling • Deployment • Healing • Discovery/Load balancing • Scaling But how about operating Kubernetes?
- 5. Copyright 2017 ITRI 工業技術研究院 Kubernetes Operation Tasks 5 • Deployment • Upgrade / rollback • Scaling • Monitoring • Healing • Security • … • A lot of manual/semi-manual work • No standard way to approach all the problems Problems! http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example
- 6. Copyright 2017 ITRI 工業技術研究院 What’s Self-hosted Kubernetes? 6 • Running all required and optional components of a Kubernetes cluster on top of Kubernetes itself • Kubernetes manages own core components • Core component deployed as native API objects https://www.youtube.com/watch?v=EbNxGK9MwN4
- 7. Copyright 2017 ITRI 工業技術研究院 Why Self-hosted Kubernetes? 7 • Small Dependencies ▪ Reduce the number of components required (Kubelet and Docker) • Deployment consistency ▪ Reduce the number of moving parts relying on the host OS • Introspection ▪ Can be debugged and inspected by users using existing Kubernetes APIs • Cluster Upgrades ▪ Upgrade the components via Kubernetes APIs • Easier Highly-Available Configurations ▪ Easier to scale up and monitor an HA environment without complex external tooling https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md
- 8. Copyright 2017 ITRI 工業技術研究院 Launching a Self-hosted Cluster 8 • Need an initial control plane to bootstrap a self-hosted cluster • Bootkube ▪ Provides a temporary control plane to run a full blown self-hosted control plane ▪ Run only on very first node, then not needed again https://www.youtube.com/watch?v=EbNxGK9MwN4
- 9. Copyright 2017 ITRI 工業技術研究院 Kubernetes Architecture 9 Master node etcd Scheduler Controller Manager API Server Worker node Kubelet Kube-proxy dockerPod Pod Worker node Kubelet Kube-proxy dockerPod Pod
- 10. Copyright 2017 ITRI 工業技術研究院 Spectrum of Self-hosted Ways 10 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md • 0-4 self-hosted cluster • kubelet: daemon set • API server: pod / service • 1-4 self-hosted cluster • system installed Kubelet • 2-4 self-hosted cluster • everything except etcd self-hosted
- 11. Copyright 2017 ITRI 工業技術研究院 How bootkube works? 11 etcd Bootkube API Server Scheduler Controller Manager Kubelet
- 12. Copyright 2017 ITRI 工業技術研究院 How bootkube works? (cont.) 12 etcd Bootkube API Server Scheduler Controller Manager Kubelet Create • Deployment • Daemonset • Service • Secret kubectl
- 13. Copyright 2017 ITRI 工業技術研究院 How bootkube works? (cont.) 13 etcd Bootkube API Server Scheduler Controller Manager Kubelet Pods API Server Scheduler Controller Manager create
- 14. Copyright 2017 ITRI 工業技術研究院 How bootkube works? (cont.) 14 etcd Bootkube API Server Scheduler Controller Manager Kubelet Pods API Server Scheduler Controller Manager create
- 15. Copyright 2017 ITRI 工業技術研究院 Pods How bootkube works? (cont.) 15 etcd Kubelet API Server Scheduler Controller Manager
- 16. Copyright 2017 ITRI 工業技術研究院 Self-hosted Control Plane 16 [root@centos7 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE kube-apiserver-kkxq9 1/1 Running 0 1d kube-controller-manager-2953862963-t7m1q 1/1 Running 0 1d kube-controller-manager-2953862963-wlsjp 1/1 Running 0 1d kube-dns-2431531914-gqnnd 3/3 Running 0 1d kube-flannel-wnk1j 2/2 Running 0 1d kube-flannel-xcsx2 2/2 Running 0 1d kube-flannel-xrksj 2/2 Running 1 1d kube-proxy-04x11 1/1 Running 0 1d kube-proxy-11n6t 1/1 Running 0 1d kube-proxy-1zlgz 1/1 Running 0 1d kube-scheduler-1873817829-4c7mm 1/1 Running 1 1d kube-scheduler-1873817829-pmp0n 1/1 Running 0 1d pod-checkpointer-11q7g 1/1 Running 0 1d pod-checkpointer-11q7g-10.201.3.6 1/1 Running 0 1d
- 17. Copyright 2017 ITRI 工業技術研究院 Self-hosted Control Plane (cont.) 17 [root@centos7 ~]# kubectl get deployment -n kube-system NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kube-controller-manager 2 2 2 2 2d kube-dns 1 1 1 1 2d kube-scheduler 2 2 2 2 2d [root@centos7 ~]# kubectl get svc -n kube-system NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 2d [root@centos7 ~]# kubectl get ds -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 2d kube-flannel 3 3 3 3 3 <none> 2d kube-proxy 3 3 3 3 3 <none> 2d pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 2d [root@centos7 ~]# kubectl get secret -n kube-system NAME TYPE DATA AGE kube-apiserver Opaque 7 2d kube-controller-manager Opaque 2 2d
- 18. Copyright 2017 ITRI 工業技術研究院 Disaster Recovery 18 • Node failure in HA deployments (Kubernetes) • Partial loss of control plane components (Kubernetes) • Power cycling the entire control plane (Kubernetes) • Permanent loss of control plane (External tool) http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example
- 19. Copyright 2015 ITRI 工業技術研究院 Service with High Availability 19
- 20. Copyright 2017 ITRI 工業技術研究院 Kubernetes Networking 20
- 21. Copyright 2017 ITRI 工業技術研究院 21
- 22. Copyright 2017 ITRI 工業技術研究院 22
- 23. Copyright 2017 ITRI 工業技術研究院 How do we to export the service IP to public network on bare metal? • In kubernetes we have some existed solution NodePort CloudProvider Load Balancer Ingress • But it is enough? 23
- 24. Copyright 2017 ITRI 工業技術研究院 NodePort 24 kubeProxy kubeProxy kubeProxy NodePort NodePort NodePort Pod Pod Pod
- 25. Copyright 2017 ITRI 工業技術研究院 NodePort 25 kubeProxy kubeProxy kubeProxy NodePort NodePort NodePort Pod Pod Pod
- 26. Copyright 2017 ITRI 工業技術研究院 Load Balancer • Cloud Provider e.g. AWS , GCP, OpenStack • Load Balancer is created by Cloud Provider, and provide the external IP to for service • But it is only for Cloud Provider, the bare metal cannot do this 26
- 27. Copyright 2017 ITRI 工業技術研究院 Load Balancer 27 kubeProxy kubeProxy kubeProxy NodePort NodePort NodePort Cloud LoadBalancer Pod Pod Pod
- 28. Copyright 2017 ITRI 工業技術研究院 28
- 29. Copyright 2017 ITRI 工業技術研究院 Ingress 29 • HTTP Load Balancing • SSL Termination • Content-base Routing • Not fully for external network
- 30. Copyright 2017 ITRI 工業技術研究院 Ingress 30 Ingress nginx.com echoheaders.com ServiceA ServiceB 10.0.0.1:80
- 31. Copyright 2017 ITRI 工業技術研究院 Ingress 31 Ingress Controller Ingress Resource Load Balancer watches configure
- 32. Copyright 2017 ITRI 工業技術研究院 Ingress 32 kubeProxy kubeProxy kubeProxy Pod PodPodPod PodPod Momo.foo.com Jojo.foo.com yoyo.foo.com hostnetwork
- 33. Copyright 2017 ITRI 工業技術研究院 Ingress 33 kubeProxy kubeProxy kubeProxy Pod PodPodPod PodPod DNS
- 34. Copyright 2017 ITRI 工業技術研究院 34
- 35. Copyright 2017 ITRI 工業技術研究院 Keep-Alived VIP • Real High-Availability • Virtual IP Address • IP to Service • Configmap • DaemonSet 35 Ref : - https://github.com/kubernetes/contrib/tree/master/keepalived-vip
- 36. Copyright 2017 ITRI 工業技術研究院 Keep-Alived VIP 36 kubeProxy kubeProxy kubeProxy Pod PodPod Keepalived Keepalived Keepalived 140.113.1.1 140.113.1.2 140.113.1.3 Vip: 140.113.1.50
- 37. Copyright 2017 ITRI 工業技術研究院 Keep-Alived VIP 37 kubeProxy kubeProxy kubeProxy PodPodPod Keepalived Keepalived Keepalived 140.113.1.1 140.113.1.2 140.113.1.3 Vip: 140.113.1.50 Pod
- 38. Copyright 2017 ITRI 工業技術研究院 Keep-Alived VIP 38 kubeProxy kubeProxy kubeProxy PodPodPod Keepalived Keepalived Keepalived 140.113.1.1 140.113.1.2 140.113.1.3 Vip: 140.113.1.50
- 39. Copyright 2017 ITRI 工業技術研究院 39
- 40. Copyright 2017 ITRI 工業技術研究院 Keep-Alived VIP + Ingress 40 kubeProxy kubeProxy kubeProxy PodPod Pod Keepalived Keepalived Keepalived 140.113.1.1 140.113.1.2 140.113.1.3 Vip: 140.113.1.50 Momo.foo.com Jojo.foo.com
- 41. Copyright 2017 ITRI 工業技術研究院 Summary • Self-hosted K8S ▪ Make K8S operations more easier ▪ Bootkube is still a incubator project ▪ Support disaster recovery • Service with High Availability ▪ Using DNS to provide your service – Ingress ▪ More vip – keepalived-VIP ▪ Using cloud to build your kubernetes – cloud Load Balancer ▪ Test – NodePort 41