Extending Kubernetes with Operators
- 1. DXP Cloud Extending Kubernetes with Operators Iliyan Peychev Director of Research Development Innovations) Liferay Cloud
- 3. Source - https://kubernetes.io/docs/concepts/extend-kubernetes/ 1. Kubectl plugins: extend the kubectl binary, affect the local user's environment 2. API Access Extensions: allow/block authenticating requests 3. Custom Resources: extensions of the Kubernetes API 4. Scheduler Extensions: assigns pods to nodes. The default scheduler can be replaced, or multiple schedulers can run simultaneously 5. Controllers: custom resource API + control loop is called the Operator pattern 6. Network Plugins: allow for different implementations of pod networking 7. Storage Plugins: add support for new types of storage Kubernetes Extension Points We will focus here
- 5. "Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop" Kubernetes Operator Pattern Source https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
- 7. How Operators can be useful? Deploying Applications on demand An Operator can deploy user's applications on demand. For example, it can check for new releases of your application and deploy (release) automatically on given environment. It could also rollback releases in case of failure. Performing application upgrades If an application has to be upgraded, the Operator can do most of the underlying work like: ● Update app version ● Apply new configuration ● Perform DB backup ● Apply DB schema changes Perform backup/restore An Operator perform backup/restore of the DB of given application ● The backup could be done automatically on given schedule. ● Restore could be done automatically, if some conditions are met Chaos engineering Why could it not check the resiliency of our system, and the readiness of our SRE Team? An Operator can cause disruptions in a given environment, for parts of it, or as a whole
- 8. Where to get operators?
- 11. Example: deploy an application stack on demand
- 18. Creating an Operator in Go
- 19. --- Status Conditions Permissions Configurations --- Webhooks Default and admission ones --- Reconcile loop Maintain the state of the applications close to the desired one --- Create CRD Describe your application stack. Each application should have its own CR Creating an Operator in Go Step 1 Use Kubebuilder or Operator SDK to create the skeleton of the operator Step 2 Define the Custom Resources (CR create Custom Resources Definition yaml files. They describe your applications. Step 3 Implement the reconciliation loop - take some actions when a CR arrives Step 4 Implement admission webhooks, status conditions, define permissions, configurations.
- 20. Lets flesh out our example
- 21. Step 1 define Custom Resources
- 22. Step 1 define Custom Resources There is no need to write the CRDs by hand Kubebuilder/Operator SDK will generate them for you from your Go structures
- 23. Generate CRD for the Release Manifest from your Go structures
- 24. Generate CRD for the Release Manifest from your Go structures
- 25. Define CRD for each application ● CRD for the API ● CRD for the Console ● CRD for the Admin etc. CRD for the Release Manifest - done!
- 26. Define CRD for each application e.g. for Console
- 27. For each CR, there is a controller skeleton generated Step 2 write the reconciliation logic The developer should write the reconciliation logic
- 28. This normally means fetching the CR from Kubernetes, via the Client, then Create other CR, or Kubernetes native objects (Deployment, PVC, StatefulSet) Step 2 write the reconciliation logic
- 29. Update the status conditions of the CR Create a finalizer (if needed), and execute it when the CR is deleted Step 2 write the reconciliation logic
- 30. Fetch the CR of the release Resolve the dependencies For each app being released, create its own CR Then its own controller will be invoked Each app controller will create Kubrnetes native objects
- 31. Normally there is some kind of template, and the native Kubernetes object(s) are being created based on it Step 2 write the reconciliation logic
- 32. Step 2 template
- 33. Step 2 prepare and render the template
- 34. Step 2 create/update native k8s objects
- 35. This is not a must, but a good practice Step 3 write webhooks, status conditions, configurations Webhooks allow/reject CR, or set default values Status conditions - represent the current status
- 36. Step 3 Webhook
- 37. Step 3 Status conditions Note: this is an external example - a Cert Manager Pod
- 38. Implement finalizers Step 3 Finalizers When additional actions need to be performed on deleting the custom resource
- 39. Operator's own configuration file can be a Kind too Step 3 Custom configuration
- 40. Step 3 Custom configuration
- 41. Step 3 Custom configuration
- 43. Write tests Write tests Write tests Write tests Write tests
- 44. Writing integration tests Operator SDK and Kubebuilder come with a framework to create integration tests A test environment is being created for you on the scaffolding step Use it to create integration tests for both the controllers and webhooks
- 47. Thank you!