Making the Shift Left - Bringing Ops to Dev before bringing applications to production (DigitalXchange 2023)
Download as PPTX, PDF0 likes41 views
The document discusses bringing operations considerations into the development process earlier, referred to as "shifting left." It advocates designing applications with operations in mind from the beginning. This includes understanding operational objectives, constraints, and service level agreements. Application telemetry and monitoring are also important to incorporate from the start. The document provides examples of how to implement operational practices like deployments, health checks, and incident response processes in a shifted left model where development and operations work more closely together.
Making the Shift Left - Bringing Ops to Dev before bringing applications to production (DigitalXchange 2023)
- 1. Classificatie: vertrouwelijk Making the Shift Left â Bringing Ops to Dev before bringing applications to production
- 2. Classificatie: vertrouwelijk Making the Shift Left Bringing Ops to Dev before bringing applications to production DigitalXchange â 3rd June 2023 Lucas Jellema, CTO & Architect Conclusion Application Platform
- 3. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 Architect on many integration initiatives with dozens of large and small organizations in The Netherlands Oracle ACE Director, Java Rockstar, author of two books on integration frequent speaker on conferences & active blogger Lucas Jellema Cloud Solution Architect & CTO [email protected] | technology.amis.nl | @lucasjellema | lucas-jellema 3
- 4. Classificatie: vertrouwelijk âą 25 years Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 4
- 5. Classificatie: vertrouwelijk âą Light Bulb moment Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 5
- 6. Classificatie: vertrouwelijk âą Production! Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 6
- 7. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 7 business process Application Application Application Platform Platform Platform Platform Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure
- 8. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 8 business process Application Application Application Platform Platform Platform Platform Infrastructure Infrastructure Infrastructure Infrastructure Infrastructure
- 9. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 9
- 10. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 10 âą you make it run after you have built it âą you keep it running
- 11. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 11 âą you make it run after you have built it âą you keep it running
- 12. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 12 âą you make it run after you have built it âą you keep it running
- 13. Classificatie: vertrouwelijk DevOps Cycle Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 13 https://medium.com/t%C3%BCrk-telekom-bulut-teknolojileri/devops-lifecycle-continuous-integration-and-development-e7851a9c059d
- 14. Classificatie: vertrouwelijk Shift Left â Operations & Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 14
- 15. Classificatie: vertrouwelijk Shift Left â Operations and Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 15
- 16. Classificatie: vertrouwelijk Shift Left â Operations and Development without hand-over Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 16 Operations Application Platform & Infra Development
- 17. Classificatie: vertrouwelijk What is Ops? âą Make Up, Make Perform (as needed) âą Handle Exceptions âą Ensure Safe & Secure âą Watch | Control | Reduce Costs âą Optimize for Sustainable/Green run âą Grease the wheels & Clean the floors âą clean, prune, small technical maintenance, odd little jobs âą (prepare for) Disaster Recovery âą Report on Day to Day operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 17
- 18. Classificatie: vertrouwelijk Objectives & Constraints for Operations âą Service Level Agreement (contract) âą Business Requirements âą Principles, Guidelines, Rules, Constraints âą architecture, âą security, âą regulatory âą Professional responsibility, ethical Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 18
- 19. Classificatie: vertrouwelijk What to Measure and what to Aim for? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 19
- 20. Classificatie: vertrouwelijk What level of care? âą Application + Platform + Infra need to allow for the required level of care âą observe status (& trend ïł) âą compare with goals âą work within constraints âą decide act upon action âą have protocols to follow and instruments to take action âą This requires early preparation Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 20
- 21. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 21
- 22. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 22 Basware invoices record, approve/deny daily approved invoices batch MS Business Central - Finance integration ⏠⏠⏠âŹ
- 23. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 23
- 24. Classificatie: vertrouwelijk Common thread Requirements Development Build, Test, Q&A Release & Deploy Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 25
- 25. Classificatie: vertrouwelijk Common thread Requirements Development Build, Test, Q&A Release & Deploy Operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 26
- 26. Classificatie: vertrouwelijk Shift Left Requirements Development Build, Test, Q&A Release & Deploy Operations Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 27 Ops by Design
- 27. Classificatie: vertrouwelijk Required for successful operations âą Definition of what is success âą What are non-functional constraints and conditions? âą Security, availability, performance, costs, CO2 âą KPIs and targets? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 28 Whenever the monitoring indicated that a service was down â the entire portal was stopped and restarted One particularly unstable service determined whether today was the userâs birthday. It triggered multiple restarts per day âŠ
- 28. Classificatie: vertrouwelijk Application âFingerprintâ âą Describe for each Application: âą business value and owner/stakeholders âą priority / criticality âą business process âą trigger (when/why) âą result (& verification) âą source and destination systems âą data structures & filtering | mapping âą non happy flows âą quality controls âą security âą operational targets Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 29 Foundation for Solution Design and Test plan, basis for Operational mechanisms & processes
- 29. Classificatie: vertrouwelijk Dev => Ops Dev Intake / Ops Acceptance Deploy / Rollout / Activate Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 30 Observe Interpret Act Learn & Improve Routine Management Certificate, credential, user, endpoints; Backup, Archive, Purge, Patching, Cost Allocation Support Desk Fire drills / Chao(p)s Testing Report & Improve Recovery, Fail Over, Hot Fix, Rollback of Rollout, Scale Out, ⊠Improve instrumentation, scalability, recovery, non happy flow handling, configurability Ops
- 30. Classificatie: vertrouwelijk Intake: Check on Operability of Application âą Fingerprint âą Definition of Success âą QA-ed âą Operability of Application Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 31 Dev Intake / Ops Acceptance Photo by Priscilla Du Preez on Unsplash
- 31. Classificatie: vertrouwelijk Ops Acceptance Test â confirm that operations can be done âą Simulate realistic production runtime scenarios âą Areas of interest: âą detection | analysis | instruction | facilities | action | result Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 32
- 32. Classificatie: vertrouwelijk Deployment | Roll Out | Activation âą Automated âą Configurable environment specific dependencies at Application, Platform & Infra âą Verifiable through post-rollout Smoke Tests âą Activate for subset of workload: Blue/Green, Canary âą Scenario to Rollback Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 33 Dev Intake / Ops Acceptance Deploy / Rollout / Activate
- 33. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 34 API
- 34. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 35 âą Saturday Midnight âą The loss of service was experienced on Monday morning Catalog Service API Products Service Products Service ?
- 35. Classificatie: vertrouwelijk Beyond Smoke Test: Health Checks & In-Production Testing âą Periodically check availability of owned and called endpoints âą In Production Testing: dummy business objects âbusiness side-effect freeâ but otherwise very real Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 36 API ping No side effect â test all the way accessibility API create API validate Process business objects that are known to be dummies â not sent to external systems
- 36. Classificatie: vertrouwelijk Instrument & Emit - Observe â Interpret - Act Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 37 application application Platform & Infra telemetry Collect process & interpret Alert Report Visualize Analyze Act AIOps
- 37. Classificatie: vertrouwelijk Telemetry â M(E)LT âą The need to know âą what happens âą what is the status âą Dimensions âą real time and after-the-fact âą fine grained and coarse grained âą business & functional, security, cost, CO2 and technical âą Types of Telemetry: MELT âą Events, Profiles & Exceptions Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 38
- 38. Classificatie: vertrouwelijk The Invisible Route Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 39
- 39. Classificatie: vertrouwelijk Tracing provides CCTV-like insight: when, where, what, why, who, how long âą Track individual session | flow âą Powerful aggregation across traces âą group by component, status, origin, trace context attributes Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 41
- 40. Classificatie: vertrouwelijk âą Log event (typically when something happens): logs can be associated with a job | request | transaction: âą Logs provide drill down details for trace-span âą Consolidated logs â from across application landscape â are powerful analysis tool âą timestamp is crucial to sort and correlate platform & infra logs Logs â fine grained reports Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 42
- 41. Classificatie: vertrouwelijk Metrics â periodic mini status reports Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 43 platform & infra metrics âą Timestamped measurements âą Sometimes pushed, often polled/scraped
- 42. Classificatie: vertrouwelijk OpenTelemetry âą Cross industry standard for collection of Telemetry âą Virtually all tools for monitoring, analyzing, visualizing telemetry support OpenTelemetry as source Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 44
- 43. Classificatie: vertrouwelijk Events â telemetry context for analyzing and predicting âą Black Friday âą Horizontal Scale Out (double capacity for Service X) âą Rollout/activation of new version of Service Y âą Start of batch job Z âą Snowstorm in the Köln area âą Upgrade of Library P âą New certificate for Component Q âą Purged 2 TB of data from Time Series Database R âą Outage in Azure Data Center (Frankfurt region) âą Dortmund .. Bayern ⊠Borussia .. nein, MĂŒnchen meister âą New engineer joined the DevOps team âą Recovery test on database C âą Ticket logged regarding poor performance âą Expiry of Certificate K Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 45
- 44. Classificatie: vertrouwelijk Profiling âą Dynamic analysis at runtime âą Periodic system snapshots (sampling) âą Metrics âą CPU & Memory usage per process & child process | thread âą Application Call tree âą Duration & frequency of function calls âą Detailed bottleneck analysis Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 46
- 45. Classificatie: vertrouwelijk Collect, Correlate, Interpret, Alert âą Create the overall picture âą Detect âą Find whatâs mssng â unexpected unevents âą Automate Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 47
- 46. Classificatie: vertrouwelijk Ops Obs is a veritable BI challenge Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 48 platform infra Integration WebApp Function Job API Service Traces, Log entries, Metrics, Context Events Instrumentation & Configuration determines telemetry Telemetry Lake
- 47. Classificatie: vertrouwelijk The Ideal Dashboard ⊠Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 49
- 48. Classificatie: vertrouwelijk The Ideal Dashboard ⊠Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 50 Dashboard Past & Future Now
- 49. Classificatie: vertrouwelijk The Ideal Dashboard ⊠Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 51 Dashboard - Now Past & Future Now Alerts Notifications Actions Heartbeat â Proof of Health
- 50. Classificatie: vertrouwelijk The Ideal Dashboard ⊠Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 52 Dashboard â Past & Future Past & Future Now Trend & Pattern Charts Predictions Simulations / What-If explorations
- 51. Classificatie: vertrouwelijk Act upon Alert âą Need to know what to do when and how to do it then âą how important is it? what is priority? Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 53 ?
- 52. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 54
- 53. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 55
- 54. Classificatie: vertrouwelijk 1202 Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 56
- 55. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 57
- 56. Classificatie: vertrouwelijk Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 58
- 57. Classificatie: vertrouwelijk Means & Mandate to act â Application & Platform/Infra âą Inspect âą Reconfigure âą Scale âą Restart âą Resubmit âą Apply Patch âą (Re)schedule job âą Perform recovery âą Provision & Update Infra âą Acquire (temporary) elevated privileges Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 59 Observe Interpret Act Learn & Improve Routine Management Fire drills / Chao(p)s Testing Report & Improve
- 58. Classificatie: vertrouwelijk Operations Firedrills âą Testing Resilience of âą systems & automated procedures âą processes âą people Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 60
- 59. Classificatie: vertrouwelijk Improvements over time âą Observe and extrapolate âą Explore What-If scenarios âą Automate âą Calibrate âą Refine schedules âą Fine-tune Tools and Procedures âą Extend configurability âą Automate (AI), dynamic, predictive configuration âą Refine instrumentation âą Improve telemetry processing âą Decommission Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 61
- 60. Classificatie: vertrouwelijk Summary âą Operations for successful business functionality âą Define success âą Design & Build Observability & Operability Test Operations âą Shift Left â âOps by Designâ âą Not DevOops but DevOps Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 62 Application Platform
- 61. Classificatie: vertrouwelijk Thank you for your attention I hope this was useful Making the Shift Left - Bringing Ops to Dev - June 3rd 2023 [email protected] | technology.amis.nl | @lucasjellema | lucas-jellema 63
Editor's Notes
- #14: https://medium.com/t%C3%BCrk-telekom-bulut-teknolojileri/devops-lifecycle-continuous-integration-and-development-e7851a9c059d
- #22: angry suppliers their bills have not been paid. Why Not? They have been approved (in the invoice management app) - but not forwarded to the financial system where payment is done
- #24: the web portal was never the fastest application in the world but usually performed at a stable if barely acceptable level several times during the month its performance collapsed and it became unusable. 20-30 seconds wait times and time outs. it was unclear why that happened â at those specific moments
- #33: Common scenarios: Loss of network Crash of server (container) (mid-job) Peak traffic load Incorrect message payload Inaccessible endpoint Invalid credentials Power failure (forced restart)
- #40: https://www.google.com/url?sa=i&url=https%3A%2F%2Fblue-sea-697d.quartiers047.workers.dev%3A443%2Fhttps%2Fwww.atlasobscura.com%2Farticles%2Fantwerp-invisible-route-map-surveillance-cameras&psig=AOvVaw1abfitkkT8VjPbxt27ywgm&ust=1669532135034000&source=images&cd=vfe&ved=0CBEQjhxqFwoTCLD4iICiy_sCFQAAAAAdAAAAABAE
- #42: https://opentelemetry.io/docs/concepts/signals/traces/#span-links
- #55: https://www.slideserve.com/ayala/se-3910-real-time-systems
- #60: expired certificate Product Owner cannot be reached Wiki is down Storage inaccessible Critical vulnerability in library DDOS attack detected Ransomware alert Data Corruption (human error) failing DNS server Azure availability zone is down team tech lead falls ill P1 bug in recent production release Edge connectivity lost Power outage
- #61: expired certificate Product Owner cannot be reached Wiki is down Storage inaccessible Critical vulnerability in library DDOS attack detected Ransomware alert Data Corruption (human error) failing DNS server Azure availability zone is down team tech lead falls ill P1 bug in recent production release Edge connectivity lost Power outage