A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel
3 likes3,950 views
The document discusses the use of Kubernetes, Jsonnet, and Bazel at Databricks, highlighting the challenges of managing multiple Kubernetes clusters and deployments. It emphasizes the benefits of using Jsonnet to simplify configuration management and improve deployment processes, as well as Bazel for customizable builds. The presentation concludes with an invitation for potential hires to join the team.
A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel
- 1. A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel Miles Yucht SF Kubernetes Meetup July 26, 2017 1
- 2. Who am I? Miles • CS @ Princeton • SWE on Growth team @ Databricks since 2015 • "Lead Developer" of kubecfg 2
- 3. Agenda • Kubernetes @ Databricks • Why Jsonnet • Combining Bazel and Kubernetes • Q & A 3
- 4. Kubernetes 4
- 5. Kubernetes at Databricks ● Multiple Kubernetes clusters, hundreds of deployments ● 6+ terabytes of memory, hundreds of cores ● Many customizations for security ● Everything new goes onto Kubernetes ● All managed by a 7-person team 5
- 6. Kubernetes at Databricks We're hiring! Come talk to Priya or me after the talks! 6
- 7. Combinatorial Configurations 6 clusters, 3 environments, 4 modes of deployment ● Lots of parameters ● Extremely verbose ○ >75k lines of checked-in YAML templates ● YAML is not modular ● Hard to manage, manipulate, and correctly refactor templates 7
- 8. Kubernetes development lifecycle ● Simple to test & develop in Kubernetes ● Keep dev, test, staging, prod environments similar ● Critical for testing kubernetes/cloud features ○ E.g. replication, PROXY protocol 8
- 9. Many-step Deployment 9 Build Docker Image Tag Docker Image Push Docker ImageEdit Template Apply Template
- 10. kubecfg! ● Python wrapper around kubectl ○ Still have all your favorite kubectl commands ● Preprocesses inputs and passes the result to kubectl ● Augments kubectl ○ Easy to add new commands to kubecfg ● Automates several common workflows ○ Simpler for developers ○ Fewer chances to make mistakes 10
- 11. Jsonnet 11
- 12. Jsonnet ● DSL for defining JSON data ● Break up monolithic configurations into composable parts ● >75k lines of YAML → ~20k lines of Jsonnet ● To learn about how we use Jsonnet: ○ https://databricks.com/blog/2017/06/26/declarative-infrastructure-js onnet-templating-language.html 12
- 13. Jsonnet → JSON When running kubecfg apply -f webapp.jsonnet: ● jsonnet webapp.jsonnet > /tmp/aaa.json ● kubectl apply -f /tmp/aaa.json 13
- 14. Original kubecfg architecture 14 Jsonnet Template Kubernetes spec Kubectl
- 15. Integrating build and deploy Kubernetes template ⇕ Tagged image 15
- 16. Integrating build and deploy Kubernetes template ⇕ How to produce the artifact that it depends on 16
- 17. Integrating build and deploy 1. What do you need to do to prepare the artifact? → Ask your build tool. 2. How do you refer to the artifact? → Use a tagged docker image. 17
- 18. Bazel 18
- 19. Bazel ● Open-source, language-agnostic build tool ● Very customizable/configurable ● Ported our build from SBT to Bazel mid-2016 ● To learn about how we use Bazel: ○ Blog post coming soon! 19
- 21. Kubectl Templates { image: "registry.databricks.com/webapp:latest", ... } Kubernetes template ⇕ Tagged image 21
- 23. Kubecfg Templates { target: "//docker/webapp", ... } Kubernetes template ⇕ Bazel target 23
- 24. Integrating build and deploy Build: ● bazel build //docker/webapp:webapp Tag & Push (new): ● bazel run //docker/webapp:webapp_push -t <tag> Deploy (new): ● $(cat //docker/webapp:webapp_repository):<tag> 24
- 25. Tagging images For clean checkouts: ● <GIT_SHA>-<COMMIT_TIMESTAMP> For dirty checkouts: ● <GIT_SHA>-<COMMIT_TIMESTAMP>-DIRTY-<DIFF_SHA> 25
- 26. Current kubecfg architecture 26 Jsonnet Template “Kubernetes spec” with Bazel targets Kubectl Kubernetes spec with images Build docker image Tag and push image git ref-log
- 27. Conclusion ● kubecfg makes deployment a two-step operation 1. git checkout my-branch 2. kubecfg apply -f the-template.jsonnet ● Speeds up development cycle ● Helps with tracking down errors 27
- 28. Future of kubecfg ● We plan on open sourcing this library ● Development in Bazel makes this easier ○ https://github.com/bazelbuild/rules_docker 28
- 29. Thank you We're hiring! Talk to me or Priya if you're interested in joining our team. 29