Measuring DevOps Performance
- 1. Measuring Performance: See the Science of DevOps Measurement in Action
- 2. Your Hosts Dr. Nicole Forsgren @nicolefv Robert Reeves @robertreeves
- 3. The Highlights • Measuring Performance • Most important things to remember • Common mistakes • An example of a good measure: Software Delivery Performance • Where are you? • Maturity Models • Broken • Where they work • Learning theory • You can help
- 5. Common Mistakes • Outcomes > outputs • Global or team > Local or individual
- 7. Common Mistakes • Some common examples: • Lines of code • Velocity • Utilization
- 8. Common Mistakes: Lines of Code • More is better? • Bloated software • Higher maintenance costs • Higher cost of change • Less is better? • Cryptic code that no one can read • Ideal: solve business problems with most efficient code
- 9. Common Mistakes: Velocity • Agile: problems are broken down into stories, which are assigned “points” of estimated effort to complete • At end of sprint, total points signed off by customer is recorded = velocity • Velocity is a capacity planning tool. NOT a productivity tool. • Why doesn’t this work for productivity? • Velocity is a relative measure, not absolute. So: bad for comparing teams • Gaming by inflating estimates • Focus on team completion at the expense of collaboration (a global goal)
- 10. Common Mistakes: Utilization • Utilization is only good up to a point • Higher utilization is better? • High utilization doesn’t allow slack for unplanned work • Queue theory: as utilization approaches 100%, lead times approach infinity • Once you hit higher and higher levels of utilization (a poor goal of productivity), teams will take longer and longer to get work done
- 12. Measuring Software Delivery Performance • Focus on both Outcomes and Global measures: - Deploy frequency (when business demands) - Lead Time for Changes - Mean Time to Recover (MTTR) - Change Fail Rate
- 13. We see More throughput More stability In tandem. Without the tradeoffs that some suggest are necessary.
- 14. High Performing DevOps Teams More agile More frequent Code deployments 46x That’s the difference between multiple times per day and once a week or less. Faster lead time from commit to deploy 440x That’s the difference between less than an hour and more than a week.
- 15. High Performing DevOps Teams More reliable Faster mean time to recover from downtime 96x That means high performers recover in less than an hour instead of several days As likely that changes will fail 1/5x That means high performers changes fail 0-15% of the time, compared to 31-45% of the time.
- 17. Maturity Models (generally bad news)
- 18. Maturity models are for CHUMPS
- 19. Maturity models are for CHUMPS
- 22. You Can Help!
- 23. Your Role in this • Start by measuring a few things • Focus on outcomes and global measures • Think about what things you control that can drive improvements in those measures – both with tech and with not tech • Share your stories! Leverage community
- 24. Questions?
- 25. Thank You