Kubernetes Scheduler deep dive
- 5. Kubernetes master node worker node worker node pod API server
- 6. Kubernetes master node worker node worker node API serverpod etcd
- 7. Kubernetes master node worker node worker node API serverpod etcd
- 8. Kubernetes master node worker node worker node API serverpod etcd scheduler
- 9. Kubernetes master node worker node worker node API serverpod etcd scheduler
- 10. Kubernetes master node worker node worker node API serverpod etcd scheduler
- 11. Kubernetes master node worker node worker node API serverpod etcd scheduler
- 12. Kubernetes master node worker node worker node API server etcd scheduler pod kubectl or api call
- 14. Kubernetes master node worker node worker node API serverpod etcd scheduler apiVersion: v1 kind: Pod metadata: name: nginx spec: nodeName:
- 15. Kubernetes master node worker node worker node API serverpod etcd scheduler apiVersion: v1 kind: Pod metadata: name: nginx spec: nodeName:worker-1
- 17. Scheduler #1 - watch kube-apiserverkube-scheduler ?watch=true podSpec.NodeName=’’
- 18. Scheduler #2 - Pred. filter kube-apiserverkube-scheduler ?watch=true predicate podSpec.NodeName=’’
- 19. Scheduler #3 - Priority kube-apiserverkube-scheduler ?watch=true predicate priority podSpec.NodeName=’’
- 20. Scheduler #4 - Select Node kube-apiserverkube-scheduler ?watch=true predicate priority Round-Robin podSpec.NodeName=’’
- 21. Scheduler #5 - bind kube-apiserverkube-scheduler ?watch=true predicate priority Round-Robin podSpec.NodeName=’’ bind
- 22. 그렇다면 어떻게 Scheduler가 pod 정보를 어떻게 받는지 에 대해 파헤쳐 보죠!
- 23. master node worker node worker node API server scheduler $ curl -H “Authorization: Bearer $TOKEN” $APISERVER/api/v1/pods?watch=true ?watch=true
- 24. master node worker node worker node API server scheduler $ curl -H “Authorization: Bearer $TOKEN” $APISERVER/api/v1/pods?watch=true ?watch=true
- 25. Scheduler
- 26. Scheduler
- 27. Scheduler
- 28. 이 부분을 좀 더 자세하게 살펴봅시다
- 29. Scheduler
- 30. Scheduler
- 33. Scheduler
- 34. Scheduler
- 35. Scheduler
- 36. Scheduler
- 37. Scheduler
- 40. Scheduler kube-scheduler schedulerOne SchedulingQueue pod Schedule PVC validation findNode (filter)
- 44. 이 부분을 좀 더더 자세하게 살펴봅시다 with schedulingQueue
- 45. Scheduler
- 46. Scheduler
- 50. kubernetes API server Scheduler Controller manager etcd Declarative configuration yaml or json Describes Desired state
- 51. kubernetes control loop Desired State Current State watch action
- 55. Informer Scheduler ReplicaSetController kube-apiserver controller informer Add, Update, Delete notification hook serialization cache deserialization
- 57. scheduler algorithm 좀 더더더 자세하게 살펴봅시다
- 58. predicate
- 59. kube-scheduler filtering #default predicate
- 61. kube-scheduler filtering #check node node kubelet node.kubernetes.io/unschedulable
- 65. scheduler 복잡해요! 성능은?? 좀 더더더더자세하게 살펴봅시다
- 66. findNodeThatFit predicates on all the nodes - CPU intensive - independent of each other Concurrently on 16 Go-routines
- 67. performance predicates on all the nodes - CPU intensive - independent of each other Concurrently on 16 Go-routines if feasible nodes is enough, cancel()
- 68. performance
- 69. numNodesToFind = predicate func를 멈출 최소 feasible node count - minimum 100 - if percentageOfNodesToScore - 100 < numNodesToFind < (총 노드 수) * percentageOfNodesToScore / 100 - if !percentageOfNodesToScore - 5 < percentageOfNodesToScore = (50 - (총 노드수)/125) - 100 < numNodesToFind < (총 노드 수) * percentageOfNodesToScore / 100 kube-scheduler #feasible node min count
- 73. priority kube-scheduler priority C node D node A node B node E node F node C node 2score ,1weight D node 1score ,1weight A node 1score ,1weight B node 2score ,1weight E node 2score ,2weight F node 4score ,1weight
- 74. priority kube-scheduler priority C node D node A node B node E node F node C node 2score ,1weight D node 1score ,1weight A node 1score ,1weight B node 2score ,1weight E node 2score ,2weight F node 4score ,1weight C node total: 2 D node total 1 A node total: 1 B node total: 2 E node total: 4 F node total: 4
- 75. priority kube-scheduler priority C node D node A node B node E node F node C node 2score ,1weight D node 1score ,1weight A node 1score ,1weight B node 2score ,1weight E node 2score ,2weight F node 4score ,1weight C node total: 2 D node total 1 A node total: 1 B node total: 2 E node total: 4 F node total: 4
- 78. scheduler internal cache 좀 더더더더자세하게 살펴봅시다
- 79. kube-scheduler filtering #assume scheduler apiserver etcd node (kubelet) memory 4G (requested 3G) CPUs 2 (requested 1) A pod request memory 1G CPUs 1
- 80. kube-scheduler filtering #assume scheduler apiserver etcd node (kubelet) memory 4G (requested 3G) CPUs 2 (requested 1) A pod request memory 1G CPUs 1
- 81. kube-scheduler filtering #assume scheduler apiserver etcd node (kubelet) memory 4G (requested 3G) CPUs 2 (requested 1) A pod request memory 1G CPUs 1 B pod request memory 1G CPUs 1
- 82. kube-scheduler filtering #assume scheduler apiserver etcd node (kubelet) memory 4G (requested 4G) CPUs 2 (requested 2) A pod B pod request memory 1G CPUs 1
- 83. kube-scheduler filtering #assume scheduler apiserver etcd node (kubelet) memory 4G (requested 3G) CPUs 2 (requested 1) A pod request memory 1G CPUs 1 B pod request memory 1G CPUs 1 schedulerCache assume!!
- 84. kube-scheduler filtering Total requested resource = running pods request + assumed pod request
- 85. 이제 더더더더더 보지말고 정리를 한번 해보죠.
- 87. API server scheduler controller-manager etcd kubelet kube-proxy docker kubelet kube-proxy docker pod container PodSpec.NodeName=’{{ worker node }}’ Desired State Current State watch action
- 93. 곧 끝나가요.., 마지막 하나만(extension) 더 ...
- 94. extend kubernetes scheduler - change upstream source code - multiple scheduler ( default && custom) - scheduler extender - scheduler framework
- 95. change upstream source code
- 96. multiple scheduler master node worker node worker node API server kubectl or api call pod Custom scheduler Default scheduler spec: schedulerName: CustomScheduler
- 97. scheduler extender master node worker node worker node API server priority kubectl or api call pod Scheduler Extender Default scheduler predicate bind
- 98. scheduler extender master node worker node worker node API server priority kubectl or api call pod Scheduler Extender Default scheduler predicate bind /api/custom.metrics.k8s.io
- 99. scheduler extender 출처: 알리바바 클라우드(https://www.alibabacloud.com/blog/gpu-sharing-scheduler-extender-now-supports-fine-grained-kubernetes-clusters_594926)
- 100. scheduler extender #problem - extension points are limited (predicate, priority, ..) - latency (data format, webhook) - hard to aborting scheduling - can’t use scheduler’s cache
- 101. scheduler framework
- 102. Thank u for ur attention. 김동진(dj.kim) kakao bank corp.