Editor's Notes
- #2: Yes, I’ve gone for hipster hand drawn slides I bought an Apple Pencil so I’ve gotta justify it somehow
- #3: Introduction Mention Apptio
- #4: Don’t worry, I’m not going to try wrap this prez. I’m going to tell you a story about how we got here This is a story about how we discovered we need configuration mgmt for our Kubernetes clusters
- #5: Our old deployment system consists of a bunch of java wrappers around some Bash scripts which whet some artifacts It’s slow. We wanted to try something else, and Kubernetes was a thing
- #6: We built the clusters the way we know how to build stuff - with puppet We used kubeadm and consul for load balancing EKS was a twinkle in Amazon’s eye (and it’s still shit) Using cloud-init didn’t sound like much fun Everyone saying configuration management isn’t needed hasn’t had their Infoseek team run a Nessus scan
- #7: We realized that clusters aren’t very usable out of the box So we added a bunch of stuff and called it AKP A better name than EKS fwiw We generally used helm charts
- #8: Helm is amazing, it makes installing default components simple and easy. However, this is fine unless you need to make a change to a helm chart. Pull requests can be slow Helm is insecure
- #9: we built another cluster. And another. And we kept having to install the same stuff, but with slightly different values Eventually one day, our devs came to us and asked why our dns wasn’t updating correctly. Turns out, someone had put the wrong configuration in the ingress controller
- #10: We realized we had built snowflakes. There was no repeatability and worse, we had no configurability as we scaled out.
- #11: We needed to install these Kubernetes components in a repeatable way we went back to what we knew, puppet Used the puppet helm module to install charts Puppet gave us configurability with hiera
- #12: Puppet isn’t cluster aware at all It ran on the master nodes, all 3 of them We regularly ran into race conditions where puppet would do strange things It’s just not designed for this higher level abstraction
- #13: These are familiar problems But we couldn’t understand how nobody else was having this problem Is everyone happy with running snowflakes? Does everyone have one cluster? I did what anyone would do - twitter thread - jk - I wrote a blogpost
- #14: We tried a few different things Ksonnet - we liked this but it was complicated, and seemed focused towards app deployment We then decided to try templating values.yaml and realized we were just insane Kapitan - more yaml templating, but this ninja2 Ansible - say no more
- #15: Notice the trend here. Why as an industry have we made it acceptable to use templating languages for configuration? When did this happen? Who’s responsible? Kubernetes is quite happy to accept JSON and computers are good at generating it
- #16: After some frustration, one of my smarter colleagues decided he was going to write something
- #17: He named it kr8 It was initially a set of bash scripts We rewrote it in Go a while ago, but some of the bash scripts still remain
- #18: Can take helm charts, pure jsonnet, yaml or json and manipulate it Creates deployable manifests for each cluster, which you can read and understand and debug Deplorable with kubectl!
- #19: A component is something you want on some or all clusters They contain a parameters definition, and a taskfile (go-task) and some jsonnet That jsonnet depends on your component source (helm - patches)
- #20: Clusters is a hierarchical directory Named clusters contain a cluster.jsonnet You can also have a params.jsonnet which is inherited for clusters below it So if you have multiple clusters in the hierarchy, it’s easy to supply values for clusters globally
- #21: So let’s see this in action
- #22: Our industry is changing Kubernetes and cloud providers are here Terraform, pulumi are tools in this space There might be a better way of doing this This might be the best way, but maybe not? The people in this room have solved this problem for multiple abstraction layers That abstraction layer has changed. We don’t just need to configure operating systems anymore, we need to use the concepts we’ve learned and push them into the new layers Developers don’t get it, I’ve seen this in the wild
- #23: There might be a better way of doing this This might be the best way, but maybe not? If someone wants to solve this problem for us, please do If you can write decent Go, we would love some PRs