Data Driven DevOps
- 1. #DataDrivenDevOps ”Without data you are just another person with an opinion” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes W. Edwards Deming
- 2. shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 3. The speakers @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes @jbaruch @LeonStigter
- 5. Poll time! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work?
- 6. Dunning-Kruger Effect a.k.a. ”optimism” People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes “
- 7. Second-system effect a.k.a. “self-confidence” The tendency of small, elegant, and successful systems, to be succeeded by over- engineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes “
- 8. The consequences @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 9. disillusion @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Software I like Software I know really well
- 10. overconfidence @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Every software is 80% ready 80% of the time Every developer, ever “
- 11. Hard to give up @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 12. @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 13. How departments measure @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering
- 14. How departments measure @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering
- 15. @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 16. @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 17. @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 18. Engineering metrics in pre-devops era @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Dev • Velocity • Architecture • Compliance QA • Incidents • Defects • MTTR • Code coverage • Test suite stability Ops • SLA • Cost env setup • Average customer cost • Tools stability
- 19. DEVOPS! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes ?
- 20. Velocity to agile is like ? Todevops… @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes ?
- 21. Velocity to agile is like profit Todevops… @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Profit
- 22. Very much alike! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗
- 23. Very much alike! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗
- 24. Very much alike! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗
- 25. Poll time! Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn't look right?
- 26. What’s the problem? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Velocity
- 27. True for both! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Metric Easily understandable Unity actionable Profit ✓ ✓ ✗ Velocity ✓ ✓ ✗
- 28. Too coarse! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Profit
- 29. Too coarse! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes DevOps
- 30. This is devops* @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Ops QADev Common goals, tools, culture Deep specialization *Unless you’re netfliX **You’re not
- 31. This is devops @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Ops QADev
- 32. Same for metrics! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Ops QADev profitSpecialized metrics
- 33. Same for metrics! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Ops QADev
- 34. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 35. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 36. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 37. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 38. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 39. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 40. How do metrics collaborate? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes dev ops qa dev • Time to build • Release pipelines stability • Artifact replication topology • False reopen count • QA coverage ops • Avg cost of customer • Smoke tests quality qa • Test suite stability • Incoming defect velocity • Test execution time • Time to create new QA lab influenced affected
- 41. Cooking with metrics @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 43. Why? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 44. ROI – is this activity is the right thing to do?
- 46. trust
- 47. sampling @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 48. Complete sample
- 49. @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 51. How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 53. Meet scrum team “avengers”
- 54. anecdotes from the daily scrum stand-ups @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” “
- 55. Let’s gather some data! 0 2 4 6 8 10 week 1 week 2 week 3 week 4 week 5 week 6 week 7 week 8 week 9 week 10 week 11 week 12 week 13 week 14 Environment Utilization Utilized Available @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 56. That doesn’t look good! But why? 0 5 10 15 20 25 Raj Jane Christina Helen Lee Baruch John Scott Nataly Ruppert Josh Patrick Michael 1 2 3 6 7 10 13 13 14 15 16 18 25 Days the environment stays checked out Days checked out @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 57. Continuous improvement 0 2 4 6 8 10 12 14 16 week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23 Environment Utilization Utilized Available @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 58. anecdotes from the daily scrum stand-ups @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” “
- 59. Let’s gather some data!
- 60. Return on investment @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 61. anecdotes from the daily scrum stand-ups @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” “
- 62. Let’s gather some data! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 63. Trust and predictability 0:00 0:40 1:20 2:00 2:40 3:20 4:00 4:40 5:20 6:00 6:40 7:20 8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40 15:20 16:00 16:40 17:20 18:00 18:40 19:20 20:00 20:40 21:20 22:00 22:40 23:20 GC time GC time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 64. Dos and Don’ts
- 65. Measuring the right thing
- 66. LoC LOL, anyone? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 69. Common vocabulary is important ”medium” In russia”medium” In brazil
- 70. Metrics generate incentives, be aware! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
- 71. Tweak as you go
- 72. Elevate! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Raw Data Product Specific Dashboards Integrated Dashboards
- 73. How departments measure @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Sales Marketing Support HR Finance Anecdotes from daily standup Engineering
- 74. Turning up the heat
- 75. Step into data-driven life @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Removes blame game Builds accountability and trust Creates common base for discussion Enables DevOps
- 76. Q&A and twitter ads @jbaruch @LeonStigter #DevOpsDaysBmore #DataDrivenDevOps http://jfrog.com/shownotes