Evolving Services Into A Cloud Native World
Download as PPTX, PDF•0 likes•91 views
The document discusses how Workday evolved its services into the cloud-native world using Kubernetes and custom controllers. It highlights the necessity for advanced orchestration in some stateful services and the benefits of adopting Kubernetes features, such as control loops for managing deployments. Additionally, it outlines the importance of good design in system reliability and development efficiency.
Evolving Services Into A Cloud Native World
- 1. Evolving Services into the Cloud Native World @IainHull#CloudNativeLondon
- 4. ERP @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 5. Apps Tech @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 6. Workday ProductionCustomer Workday’s Architecture XML XML Internet Mobile and Browser Data Services UIS OMS Analytics P S Persistence Services OOperationsDEPDeployment IS Integration Services User Interface Services Object Management Services (OMS) @IainHull
- 7. DEP Deployment Internet MobileandBrowser Data Services OMS ANA Analytics Persistence Services Operations IS Integration Services User Interface Services Object Management Services (OMS) Workday Way Automation Drive @IainHullWorkday’s Architecture @IainHull
- 8. Workday’s Architecture User Interface Business Data Transaction Service Big Data API Gateway Monitoring Console Workday Way Automation Drive @IainHull
- 9. OMS Object Management Services (OMS) Workday Way Automation Drive Workday’s Architecture DEP Deployment @IainHull
- 10. Workday Way Automation Drive Object Management Services Documents Transaction Service Business Data Workday Way Automation Drive @IainHull
- 11. Workday Way Automation Drive Object Management Services Transaction Service Business Data Documents Reporting Workday Way Automation Drive @IainHull
- 12. Workday Way Automation Drive Object Management Services Transaction Service Business Data Documents Cache Reporting Workday Way Automation Drive @IainHull
- 13. Workday Way Automation Drive Object Management Services Transaction Service Business Data Documents Cache Reporting Payroll Jobs Workday Way Automation Drive OMS Unit of Isolation @IainHull
- 14. Object Management Services @IainHull
- 15. Public Cloud @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 20. Structure of a Controller @IainHull
- 21. Structure of a Controller Configuration Desired state • External Environment Database • Custom Resource Definition in Kubernetes API @IainHull
- 22. Structure of a Controller Configuration Lessons • Configuration in a Git repo • GitOps to Kubernetes CRD @IainHull
- 23. Structure of a Controller Standalone service • Monitors the current state • Issues commands to converges to desired state Control Loop @IainHull
- 24. Structure of a Controller Lessons • Hosted in Kubernetes (own namespace) • Event driven is easier Control Loop @IainHull
- 25. Structure of a Controller Lessons • Distributed system – converge after failures • Beware of resonance during recovery Control Loop @IainHull
- 26. Structure of a Controller Sense the current state • Service discovery • Kubernetes resources Sensors @IainHull
- 27. Structure of a Controller Lessons • Only sensing Kubernetes objects • Use command tracking id Sensors @IainHull
- 28. Structure of a Controller ☸︎ Actuator Commands into Kubernetes objects • Integrated – less moving pieces • External – single point of control/security/audit @IainHull
- 29. Structure of a Controller ☸︎ Actuator Commands into Kubernetes objects • Run kubectl as a sidecar • Can also sense Kubernetes objects @IainHull
- 30. Structure of a Controller ☸︎ Actuator Lessons: • Idempotent (control loop names objects) • Be careful of resonance @IainHull
- 31. Structure of a Controller ☸︎ Actuator Lessons: • Integrated for single controller • External for multiple controllers @IainHull
- 32. Structure of a Controller ☸︎ Templates Templates • Raw JSON/YAML • Helm @IainHull
- 33. Structure of a Controller ☸︎ Templates Lessons: • Helm is the future (caveat) • Helm install the templates @IainHull
- 34. Structure of a Controller ☸︎ Templates Lessons: • Raw JSON/YAML – version the container • Helm – version the chart @IainHull
- 36. Manage stateful services Advanced orchestration External dynamic configuration Why Follow? @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 37. Long time to boot Long running jobs Long sessions Not all state is stored on disk @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 38. Kubernetes supports modern apps Custom controllers support your apps @IainHullFrom The Hudsucker Proxy by the Coen Brothers
- 39. ? ??? ? @IainHullFrom The Hudsucker Proxy by the Coen Brothers Evolving Services into the Cloud Native World @IainHull#CloudNativeLondon
- 40. Fin
- 41. Evolving Services into the Cloud Native World How Workday manage stateful services with a custom controller on Kubernetes? Kubernetes and declarative infrastructure greatly simplify the way we deploy and manage software. Most services can be orchestrated with the control loops supplied by Kubernetes (deployments, stateful sets or jobs). Some stateful services in Workday require more advanced orchestration, and re-architecting them is not an easy option. In this talk you will discover why some of our services require extra orchestration, and how we evolved an existing service into a control loop on top of Kubernetes. The control loop organises multiple services into groups these are dynamically created, deleted and scaled. It also orchestrates blue/green deployments of each group. Now we can adopt more kubernetes features and retire some of our old scheduling code. Finally you will learn the process we follow to evaluate and design our own control loops and when you might find them useful. Bio Iain is a principal software engineer at Workday using Kubernetes and Scala to deliver their next generation elastic grid. His twin passions are large scale distributed computing and applying clean code to complex problems. He is interested in good design and how this can improve system reliability and reduce friction during development. He loves sharing his experiences as he learns and builds new systems. He regularly speaks at local meetups in Dublin and has presented at conferences including GotoConf, Scala Days, Functional Kats and Lambda World.
- 42. Feedback • EPR Critical for the business • Apps consider a stack • Scale OMS with out changing Apps • Before Automated deploy: • - Introduce a controller: Sample Kubernetes is a stateless app (many of our apps don't) • - System to write controllers - not just kubernetes • Failure conditions: Things can fail • Trim start • Why do you need templates - helm • Go and Scala - owner of skuber is in workday • Cluster separate from k8s • No Propriatory Language
Editor's Notes
- #2: How Workday use a custom controller to manage stateful services on Kubernetes
- #3: Principal Engineer at Workday – I build systems that orchestrate workday’s internal services Today I’m going to tell you a little about Workday’s business, how that influences our architecture As this will help understand motivations for the controller and what lessons we have learned
- #4: Workday is a SaaS Finance and Human Capital Management company We run in the cloud, but what do we do …
- #5: ERP … Enterprise resource planning We are the system of record for our customers – how many employees – how much are they paid – how much did we spend last month – what is their revenue year to date We provide a single consistent view of their business - are their source of truth ERP isn’t just one application it’s a suite of different Applications The business domain of each application can vastly differ but technically they are very similar Domain Model / Transactions / User interfaces / Analytics “We had to ensure the architecture was flexible enough to accommodate technology changes, the growth of our customer base, and regulatory changes, all without disrupting our users. ” …. To achieve this ….
- #6: Workday engineering is split in two Apps implement the business logic of each application – Technology provide the platform that makes this easy Apps deliver value to our customers – Technology deliver the apps Apps build new functionality without worrying about the technology – Technology scales to our next 5x customer without requiring apps changes This separation frees us to introduce technologies like kubernetes when it makes sense without affecting the business
- #7: We serve UI and data traffic over the internet with something called Object Management Services
- #8: Today Workdays architecture is similar but as we have grown and acquired companies it has become federated
- #10: In this talk I am going to concentrate on the deployment and object management services These account for more than 75% of our hardware costs
- #11: The heart of Workday is a Proprietary In-memory graph database Append only, so all application data is audited Metadata driven driven – which apps use to – model graph data / define transactions / user interfaces and analytics Each ERP application is implemented in metadata Mysql is Business Data Store -> it stores the graphs commit log -> operational characteristics only -> backup/restore/DR
- #12: Added Reporting services to scale read-only requests.
- #13: Next we added a caching to distribute changes faster than through mysql Now we don’t need to hold the full graph in memory (this enables parts of the object graph to be garbage collected and quickly retrieved later)
- #14: Finally move more processing out of the transaction service Now we support distributed writes This cluster is the graph database and transaction processor – this is where our applications run We completely changed the architecture of the Object Management System to scale to massive companies Each cluster manages a handful of tenants - It is a unit of isolation … as we add customers …
- #15: We add cluster. Now we manage hundreds of these clusters in each datacenter, account for 75% of our harware costs We deploy new software of to our customers every week, all production clusters patched in less than 2 hours every Friday night. We use machine learning to predict seasonal load changes between customers and efficiently resize these clusters during the patch
- #16: Now we are expanding Public Cloud – allows us to offer services in new locations Most importantly provides an opportunity to reimagine how we deliver, orchestrate and manage our service Kubernetes provides the right abstraction, and the tools to abolish downtime during the patch As we perfect these techniques with Kubernetes in the more agile public cloud, we are bringing them and kubernetes back to our datacenters
- #17: Kubernetes is great at managing modern 12 factor applications It provides controllers that can monitor, upgrade and scale: Stateless services with deployments Stateful services that require static ips or disk with stateful sets Or monitor one off tasks with Jobs Most services in Workday follow this pattern, but sometimes you need more specialist orchestration…
- #18: Deploy
- #19: Load Indicator
- #20: Green Blue Upgrade That’s how we upgrade a clustered inmemory graph database with any customer downtime
- #24: Go, python, java, Scala
- #26: message flow consistency
- #29: Kubectl sidecar Simple rest api, also many clients (go, java, scala – skuber)
- #30: Kubectl sidecar
- #33: Raw – stored in config maps
- #34: Helm is where you want to be Running Helm in a service is troublesome It works great on the command line or scripted with Jenkins However it is not currently designed to be run concurrently be a shared service We are big fans or Helm and have researched three different ways to integrate it. At the moment if you plan Helm templates to use consider using an open source Helm operator bitnami-labs/helm-crd
- #35: Raw – stored in config maps
- #36: The open source - operator framework is great It registers for Kubernetes events and uses these to drive the control loop We have researched it and really like it Most of our services are written in Scala and use Skuber to integration with Kubernetes The author is currently adding an operator framework to this also
- #37: Why would you follow us and write your own controller Kubernetes handles most of your needs but … Blue/green deploys - Mediated rolling upgrade Teach kubernetes some of your business concepts
- #39: Custom controllers enable you to add support for you existing applications You decide when/how/if to rearchitect Everyone wants their services to be like cattle, but some cattle make burgers other are more like kobe beef – they deliver lots of value to your business Custom controllers are an option and they enable everyone to realise all the benefits of kubernetes no matter what type of cattle you have