Cloud Native Apps with GitOps
7 likes1,886 views
GitOps provides a way for organizations to adopt continuous delivery practices using cloud native technologies like Kubernetes and Docker. It advocates storing all infrastructure configurations and code in a Git repository to enable automated deployment and management of applications. Weaveworks adopted GitOps and was able to recover from a complete system failure in just 45 minutes by having their entire system defined as code. GitOps follows the principles of continuous delivery and observability by integrating deployment pipelines, monitoring, and security policies to drive faster, more reliable software releases.
Cloud Native Apps with GitOps
- 1. More, Better, Faster! Cloud Native Apps with GitOps Alexis Richardson CEO, Weaveworks & TOC Chair, CNCF September 27, 2017
- 2. WTF I want to show you how Weaveworks learned to apply technology to make our business go much faster, using cloud native and continuous delivery
- 3. WTF GitOps – empowering developers to do operations
- 4. How did we get here
- 6. 1994
- 7. 1995
- 8. 1996
- 9. 2003
- 10. 2004
- 11. 2005
- 12. (1999-) 2008
- 13. Netflix wanted to improve streaming service for all, fast Must Read!https://www.slideshare.net/AmazonWebServices/dmg206
- 14. 2009-2010 – cloud native appears as a set of requirements https://www.slideshare.net/AmazonWebServices/dmg206 Must Read!
- 15. Investors pile in because: velocity equals big bucks
- 16. We are all developers now ● If you can merge a pull request on GitHub then you are a developer ● The industry has learnt how to connect GitHub to running applications via CICD pipelines, enabling Git to drive Ops… GitOps
- 17. Frontend (Nginx) The Interwebs RDS Instance Frontend (Nginx) Frontend (Nginx) Authenticatin g Proxy WeaverangCollection Service AWS DynamoDB Query Service WeaverangControl Service AWS SQS WeaverangPipe Service AWS S3 Consul Cluster NATS Cluster Static Content Users Service Core Services Scope Services AWS DynamoDB AWS S3 WeaverangDistribution Service WeaverangIngester Service Consul Cluster Cortex ServicesMonitoring & Management Services Grafana Prometheus Weave Scope WeaverangNode Exporter WeaverangScope Probes Kubediff One time, when we blew up Weave Cloud…
- 18. Before After
- 19. 45 minutes from complete and total wipeout back to happy
- 20. “I want to go faster” Then Adopt continuous delivery & align business and tech to work as one team Tech teams are empowered to act quickly upon business needs, so they must be multi-skilled and “own the system, own the changes”. more operations roles in “dev” teams And Use the right Cloud Native technologies Automation means getting fewer errors and scaling safely Git, CICD, containers, orchestration GitOps = Cloud Native + Continuous Delivery
- 22. Fundamental Theorem of DevOps What can be described, can be automated and accelerated
- 23. But most businesses release infrequently
- 24. DevOps while ubiquitously known has not yet delivered magic velocity & continuous everything
- 25. 1. New cloud Native apps and tools is forcing change 2. Accelerate all the things 3. Automation phase shift – adapting to many releases per day Summer is coming
- 26. Cloud Native
- 27. “Curate and promote tools for cloud native” Microservices Containerised apps Dynamically orchestrated
- 28. Industry has bought in
- 29. Kubernetes is a platform for cloud native apps “orchestration” Means that it runs containerised apps the way Linux runs processes Powerful but ”low level” – will have many simplifying tools in future Important: It is declarative automated infrastructure Kubernetes
- 30. We can store Kubernetes config in Git and validate it
- 32. Weave Kubernetes gRPC Docker Fluentd OpenTracing Prometheus *** CUNNING SUBLIMINAL ADVERT ***
- 33. GitOps
- 34. • We use declarative infrastructure ie. Kubernetes, Docker, Terraform, & more • Our entire system including code, config, monitoring rules, dashboards, is described in GitHub with full audit trail • We can roll our major or minor changes as pull requests, and automatically check for diffs if system diverges from the desired “source of truth” in Git How did Weaveworks rebuild our systems in 45 mins?
- 35. • Config is code • Code must be version controlled • Config must be version controlled too GitOps follows the Logic of DevOps
- 36. GitOps follows the Logic of DevOps • Config is code • Code must be version controlled • Config must be version controlled too • What can be described can be automated • Describe everything: code, config, monitoring & policy; and then keep it in version control
- 37. GitOps • Git as a source of truth for desired state of whole system • Compare desired with actual state to fire diff alerts • Make ops changes by pull request
- 38. What this gets us • Any developer can use GitHub • Anyone can join team and ship a new app or make changes easily • All changes can be triggered, stored, audited and validated in Git And we didn’t have to do anything very new or clever
- 39. The future is joined up • DevOps is evolving to accommodate the potential of cloud native tools to get more joined up CICD and release automation at a much higher quality • GitOps shows us how to join up workflows and action oriented dashboards in ways that make sense for developers doing more ops
- 40. GitOps Deep Dive
- 41. GitOps journey • Day 0 – push first app on first cluster & validate that it works • Day 1 – add CICD updates & rollbacks via Git PRs • Day 2 – observing and controlling a production system • Day 3 – scale up – eg. better service routing (mesh) & security policy
- 42. The three pillars of GitOps CompletePipeline Observability Security
- 43. Pipelines – ABCDE pattern Deployment App Dev Build (CI) Containers Any Cluster Any Cloud Execution Push app to cloud
- 44. Pipelines – ABCDE pattern Deployment App Dev Build (CI) Containers Any Cluster Any Cloud Execution Push app to cloud
- 45. GitOps - do CD right • Config is code & everything is config (‘declarative infra’) • Code (& config!) must be version controlled • CD tools that do not record changes in version control are harmful
- 46. Continuous Delivery/Deployment The GitOps Pipeline – automate releases, sync with Git Image Repo OrchestratorDeploy Synchronizer Config change Manual deployment Git Code change Git Update Hint Continuous Integration Deploy Automator CI Pipeline
- 47. Takeaways • Pushing apps & changes is the fundamental operation • GitOps needs complete pipelines that join up CI, CD and Release Automation in one flow • The right tools must be used – they coordinate between Git, CI, and the services running in the cluster, enabling sophisticated deployment policies
- 48. Observability
- 49. GitOps & Observability • If a change is released and no- one is around to see it, then did it really work?
- 50. Read the whole thing – https://twitter.com/mipsytipsy/status/911711540008628224
- 51. Observability – understanding whole system wellness • In GitOps we want to get developers comfortable with operational concepts like monitoring, tracing, and incident handling • Like doctors, we must be able to validate health as well as diagnose problems, using a common language and a coherent set of tools
- 52. Observability – understanding whole system wellness monitor log trace visualize troubleshoot debugdiffs alerts validate Incident management & response
- 53. BigCos pioneer integrated solutions to spur faster action
- 54. Observability Asking hard questions of the system is not the same as setting up a load of alerts
- 55. Diffs & auto sync are really great Bake in metrics end to end and full stack from the start For alerts, use RED metrics focus on services You can’t avoid some instrumentation – but that’s ok since all in Git Visuals in Git – grafanalib Policy & Rules in Git (traffic, incident management) Automate (autogenerate) per-service screens & keep in Git Some lessons we learnt running Weave Cloud
- 56. Digression
- 57. • Observability is a way to verify that our system is in the desired state as specified in Git eg. diffs & alerts & more • An observable system is one that can be controlled, via a feedback control loop that drives continuous improvement A bit of theory
- 62. The GitOps Pipeline is really driving a CONTROL LOOP… GitOps loop Deployment App Dev Build (CI) Containers Any Cluster Any Cloud Execution Release ObserveOperate
- 63. Runtime System Monitoring Tracing Logging… Control System Git Diff People Release Observe Operate
- 64. System Actual State Observable State Understanding Desired State (in Git) Observe Orient Decide Act Release Diff ROODA Loop
- 65. Fundamental Theorem of GitOps What can be described and observed can be automated and controlled and accelerated
- 66. Takeaways ● Observability is fundamental to automation and understanding ● It is holistic and encompasses any question you could ask about the difference between desired and observed state ● You must bake it in from start, using monitoring, tracing, diff tools …
- 68. Recap… • Day 0 – push first app on first cluster & validate that it works • Day 1 – add CICD updates & rollbacks via Git PRs • Day 2 – observing and controlling a production system • Day 3 – scale up – eg. better service routing (mesh) & security policy
- 69. Who sees what Who talks to whom Matters more as you scale Based on rules Routing, Firewalls, ACLs, Rollouts Declarative? Store them in Git Security
- 70. ● By using diffs, we can immediately and automatically enforce convergence to a correct (desired) system state ● SOX: Git repos control which developers touch the system, which via GitOps CICD tooling can be mapped directly into running clusters ● Secrets? Security: some examples we have seen @ Weaveworks
- 71. Parting Thoughts
- 72. ● A much easier way to deliver and manage better apps, faster ● Works anywhere! ● Much more resilient – 45 mins to recover from total system wipeout What we got
- 73. ● Git is a source of truth *for everything* in cloud native era ● GitOps ROODA loop improves velocity & collaboration ● Focus on the 3 pillars: pipelines, observability & security central This is leading us to new insights, new tools, new dashboards today Key takeaways
- 75. Why GitOps The need for speed! Business expects tech to be super responsive consolidation of dev & ops skills in the most agile teams Automation: a phase shift is coming If we want to go from 1 release per MONTH to 1 or more release per DAY then we need to automate the complete lifecycle New app types will accelerate change DevOps and cloud adoption have arrived. New application types are emerging. Many use tools like Kubernetes & Docker which support “everything as code” and practices that deliver a complete automated & accelerated lifecycle
- 76. Where ops happens Powered by Kubernetes