Kubernetes: Increasing velocity without sacrificing quality
Download as PPTX, PDF•0 likes•271 views
The document discusses how CloudHealth Technologies utilizes Kubernetes to enhance developer velocity while maintaining quality. It outlines their challenges, development environments, testing strategies, and deployment processes, emphasizing the importance of consistency and collaboration. Additionally, it highlights the use of namespaces for individual developers and the effectiveness of canary deployments to minimize the risk of bad releases.
Kubernetes: Increasing velocity without sacrificing quality
- 1. Using Kubernetes to increase developer velocity (without sacrificing quality) Adam Schepis Architect @ CloudHealth Technologies
- 2. About Me ● At CloudHealth I build things in the cloud that help our customers to confidently build things in the cloud. ● I love working on distributed systems with high scalability requirements. ● I have met Spiderman. @aschepis
- 5. Our Challenges Innovation in the Cloud ● AWS - 100+ feature announcements in 2018 ● Azure - 13 announcements (chunkier) ● GCP - Next '18 in July (more than 100 announcements last year)
- 6. ● Started our Kubernetes journey in early 2017 ● Running a number of production workloads ● Kubernetes is a key component of platform overhaul in 2018 CloudHealth and Kubernetes
- 7. 7 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Our Stack Helm ● Service Lifecycle ● Values templates ● Trivial rollback ● Canary deployments were a bit tricky (would love suggestions) Linkerd ● Service discovery ● Circuit breakers ● Metrics ● Distributed tracing (via Zipkin) Romana ● CNI ● Cloud Native ● Works well with AWS ● Not a full mesh network
- 8. 8 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Primary Clusters Development ● ~50 Nodes ● Namespace per developer ● Devs given free reign within their namespace. ● Collaboration via linkerd Test/Staging ● ~20 Nodes ● Stable version of each service ● Namespace per- pull request ● Integration tests w/ new code + stable services Production ● ~50 Nodes ● Namespace per- service group ● Tight restrictions on access ● Distributed tracing
- 9. 9 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Development Environments ● ch CLI tool ● Light wrapper around setup, dev, and service lifecycle ● simplifies and accelerates dev workflow ● commands ○ ch init ○ ch new service <foo> ○ ch build ○ ch deploy ○ ch run (build + deploy) ○ ch supervisor Consistency drives both velocity and quality
- 10. 10 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Development Environments ● Service endpoints are always in helm chart values ○ Injected as environment variables ○ can use any namespace https://auth/graphql for my namespace or https://auth.david/graphql for David's namespace ● Test locally by using linkerd endpoint as http_proxy to reach remote services in desired namespace ● Namespace for each developer ● Collaborate without deploying the world. Collaborating in a shared dev cluster
- 11. 11 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. Our Build Pipeline Pull Request Human Gate Staging PR namespace Prod Canary Prod Svc Group - Unit - Integration - Pact (Contracts) - Javadoc - RDoc - etc. Published to S3.
- 12. 12 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. When things go wrong ● Each PR has its own namespace to deploy service into ● Integration tests operate against stable versions of services it depends on ● When failure happens dev can: ○ Access resources in namespace through linkerd with customer header ○ Shell into a pod to check it out ○ Look at logs ○ Exercise failing service manually ○ Access UI of failing service (if one exists) Failed Builds
- 13. 13 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. When things go wrong ● Human gates between canary and full prod deploy ● Canaries can be validated by ○ verifying that it is serving production requests ○ looking at error reporting service ○ using linkerd headers to ensure a request against canary ○ tailing logs ○ looking at performance and application metrics compared to current production code ● Can temporarily scale down to 0. ● Rollback with helm is fast and trivial. Bad Canaries
- 14. 14 © 2018 CLOUDHEALTH® TECHNOLOGIES INC. ● Consistency in tooling == velocity and quality ● Shared dev cluster ○ collaboration, shared understanding ● Namespace/deploy per PR ○ Velocity - faster to diagnose and fix test failures ○ Quality - easier to reach root cause via live debugging ● Canary Builds ○ Quickly detect bad deploys without heavy impact to customers ○ Confidence in deploys post-canary In Summary
- 15. Thank you! Questions? Adam Schepis @aschepis
Editor's Notes
- #3: I'm adam architect at cloudhealth What gets me excited in the morning is building systems (often distributed) with high scalability requirements
- #4: We have grown (a lot!) 30-260 in 3 years eng 10 -> 70 code "grew organically" with us More devs + big, complex platform + tribal knowledge = a drag on velocity
- #5: Market has matured QUALITY! Our customers aren't early adopters any more No tolerance for product or data quality issues
- #6: Innovation in the Cloud VELOCITY! 100+ announcements in 6 weeks of 2018 Azure - 13 very chunk announcements Hybrid Cloud/Datacenters enterprise customers ask for this cloud + datacenter will exist for the foreseeable future in large enterprises International growth Alibaba supporting many currencies
- #7: Decided on k8s in early 2017 Evaluated ECS, Mesos, Docker Swarm We run production workloads for background analytics and batch jobs for serving data in mainline customer requests in application Kubernetes is one of the backbones of our platform overhaul strategy in 2018
- #8: We use helm (wrapped in some light custom tooling) for managing service lifecycles It has worked very well canary deployments were a bit painful i would love to talk to people who have done canaries via helm or use helm and do canary deploys Linkerd for our service mesh daemonset in k8s teams don't have to worry about deploying sidecars Platform team doesn't have to run around explaining why they should) we get distributed tracing via zipkin telemeter Romana CNI Originally used weave but had some issues as cluster approached 50 nodes this may have been our inexperience Romana has been beneficial for us since we are on AWS and it intelligently manages route tables for us, avoiding limitation imposed by AWS Like pretty much everyone else we also use a whole bunch of other technologies for building, delivering, and monitoring services.
- #9: Dev Cluster shared by engineering team each engineer has a namespace and they can deploy
- #10: Golang built for macOS, linux light wrapper enough to make faster, not so heavy that you can't see under the covers. ch init set up dev env setup dev tools (kubectl, helm, ...) minikube (not by default anymore) Self-provision access to development/staging cluster through google auth
- #11: Adding service http_proxy No native support on Node. 😢
- #13: Reasons for failures Unit test failure integration test failure performance regressions contract validation failures What can a dev do Because the failing build still lives in a namespace a dev can inspect the running service, perform tests, etc