Listen to Your Machines: DevOps Analytics for Better Feedback Loops
4 likes1,747 views
The document discusses effective DevOps practices, emphasizing the importance of collaboration between development and operations teams, and the role of data in improving performance. It highlights various metrics that matter for DevOps teams, such as code quality, deployment speed, and business impact, while promoting the use of shared data for real-time insights and continuous improvement. Additionally, it outlines the benefits of using tools like Splunk to centralize data and enhance visibility across the entire DevOps lifecycle.
Listen to Your Machines: DevOps Analytics for Better Feedback Loops
- 1. Copyright © 2016 Splunk Inc. Listen to Your Machines: DevOps Analytics for Better Feedback Loops Andi Mann, Chief Technology Advocate Splunk @andimann
- 2. ‘Known good practices’ for collecting, correlating, and analyzing DevOps data 3
- 3. Effective DevOps Practices ● Improve collaboration and sharing between dev and ops ● Build trust and accountability between teams ● Break down barriers and improve handoffs between silos ● Establish trust and transparency between Dev. and Ops. ● Streamline flow of code from idea to cash ● Creating feedback loops at every stage ● Focus on impact on business goals and customer experience 4
- 4. FROM EVERY TOOL, EVERY PROCESS, EVERY COMPONENT, ON-PREM OR OFF One Constant - Machine Data 5
- 5. Common Data Fabric 6 API SDKs UI Server, Storage. N/W Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Other Tools Ticketing/Help Desk Custom Applications Visibility Across the Whole Ops Environment API Services
- 6. Common Data Fabric 7 API SDKs UI Other Tools Escalation/ Collaboration Visibility Across the Whole Dev Lifecycle Plan Code Build Test/QA Stage Release Config Monitor
- 7. Important DevOps data and metrics for different DevOps teams 8
- 8. Computing UK’s ‘Metrics that Matter’ Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016 9
- 9. More DevOps Metrics that Might Matter 10 Culture e.g. • Retention • Work hours • Callouts Process e.g. • Idea-to-cash • MTTR • Deliver time Quality e.g. • Tests passed • Tests failed • Best/worst Systems e.g. • Throughput • Uptime • Build times Activity e.g. • Commits • Tests run • Releases Impact e.g. • Signups • Checkouts • Revenue
- 10. Specific Metrics For Each Stakeholder 11 BizOpsStageBuildSecQADevPMOBiz • time to deliver • idea to cash • ROI • process times • team efficiency • unplanned work • code volume • commit volume • release speed • test volume • code coverage • exception counts • access attempts • remediation time • code quality • build speed • failure rates • manual exceptions • performance • latency • scalability • response time • uptime/availability • resource usage • revenue • signups • satisfaction
- 11. Shared Metrics for Multiple Stakeholders 12 BizOpsStageBuildSecQADevPMOBiz • time to deliver • scalability • ROI • time to deliver • team efficiency • ROI • team efficiency • scalability • release speed • remediation time • code quality • performance • remediation time • code quality • manual exceptions • scalability • performance • manual exceptions • release speed • code quality • scalability • remediation time • performance • scalability • performance • release speed • ROI
- 12. What About Just Dev and Ops Velocity time to deploy time to build ‘idea to customer’ story throughout build failure rate Quality number of defects downtime per release code coverage response time impacts build failures Business impact customer satisfaction application usage user signup/cancel transaction failures sales volumes Human impact employee satisfaction team productivity staff retention work hours ‘work from home’ days Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016 13
- 13. How different orgs and teams benefit from using DevOps data in multiple feedback loops 14
- 14. State of DevOps 2016: ‘Metrics that Matter’ 15 Source: 2016 State of DevOps Report, DevOps Research and Assessment
- 15. Objective data enables continuous improvement Defect Information Capacity Planning Quality Standards Enhancement Requests Integration Requirements Acceptance Metrics Service Levels and KPIs Application Development Test and Acceptance Production BuildCodePlan Test/QA Stage Release Config Monitor Infrastructure Dependencies 16
- 16. Shared Data Helps Find and Fix Issues Faster Real-time dashboards show error rate in production and impact of pushing new builds 17 Developers can search and visualize web logs, Java logs—without production access Alerts notify developers as soon as a problem arises 17
- 17. Shared Data Increases App Delivery Velocity 18 DevOps Teams iterate with continuous insights Product Managers identify new opportunities Code continuously delivered to market Auditors have visibility Customers are happy
- 18. Shared Data Improves Code Quality 19 Code Quality Scans Static Security Scans White BoxDevelopers check in code Automated Acceptance Tests Dynamic Security Scans Black Box “Chaos Monkey” Tests Test Fail: Return Test Fail: Return Production QA Prod Pattern QA Pattern Library Test Pass: Promote Test Pass: Promote to Production Pattern library used for test and QA
- 19. Shared Data Aligns DevOps With Business Impact 20
- 20. Some Real-World ‘Metrics That Matter’ 21 “Developers can focus on innovation and not on building monitoring tools.” “Web Ops can measure performance of releases in pre-prod, prod and in QA.” “Gather all data, and it starts looking like one big system, instead of a bazillion teeny ones that hate each other.” “We measure customer sentiment on Google Play in real time and can correlate it with code releases and app performance.”
- 21. Enable Improved DevOps Agility 22 Key Customer Benefits -Robert Gonsalves, Web Operations “It’s like we were working without peripheral vision before and now we have it.” • Increased success rate of deployments • Ability to detect issues before they affect broad production • Monitoring deployment process several times per day
- 22. Deliver Better Code Quality 23 Key Customer Benefits -Principal Engineer, Apollo Group “Developers are now able to look for errors and troubleshoot issues five to 10 times faster by having all their event data centralized in Splunk.” • Provide full visibility into QA sanity and load testing before production • Exceed SLA thresholds with full visibility and benchmark key infrastructure metrics and errors • Easily troubleshoot if tests do not contain the expected results
- 23. Enable Data-Driven Continuous Delivery 24 -Alison Perkins, Senior Systems Engineer “Dump all the logs into Splunk, and it starts looking like one big system, instead of a bazillion teeny ones that hate each other.” Key Customer Benefits • Quickly validate and troubleshoot code pushes to production • Ensure that new code does not negatively impact performance or user experience • Reduced one application’s error rate by 2 orders of magnitude in a matter of weeks
- 24. Enable Closer Business-IT Alignment Allows DevOps to ensure quality of releases & avoid negative impact on service performance. Analyze which new website features are being adopted, and how, by end users. Insight fed back into the development cycle to improve customer engagement. 25
- 25. Data-oriented service delivery visualization to help leaders see value in opaque DevOps processes 26
- 26. Data From Dev and Ops Tools 27
- 27. Data From Provisioning and Config 28
- 28. Data from Release Servers 29
- 29. Data from Infrastructure Systems 30
- 30. Data from Database Servers 31
- 31. Data From Automation Servers 32
- 32. Data From DevOps Systems 33
- 33. Advanced analytics to enable data-driven DevOps decisions 34
- 34. DevOps VSM In a Glass Table View – all good 35
- 35. Glass Table view – threshold exceeded 36
- 36. Deep Dive – Build Service Status 37
- 37. Deep Dive – Build Server Error Message 38
- 38. Machine-learning & predictive analytics in a DevOps-driven service delivery culture 39
- 39. Apply Advanced Algorithms to Your Data 40
- 40. Track and Predict Anomalies 41
- 41. Summary 42
- 42. FROM EVERY TOOL, EVERY PROCESS, EVERY COMPONENT, ON-PREM OR OFF One Constant - Machine Data 43
- 43. Common Data Fabric 44 API SDKs UI Server, Storage. N/W Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Other Tools Ticketing/Help Desk Custom Applications Visibility Across the Whole Ops Environment API Services
- 44. Common Data Fabric 45 API SDKs UI Other Tools Escalation/ Collaboration Visibility Across the Whole Dev Lifecycle Plan Code Build Test/QA Stage Release Config Monitor
- 45. Code Repository Automation Systems Application Monitoring CI/Build Servers Project & Issue Tracking Dev/Test/ Staging Servers InfrastructureCloud 46 Explore Visualize Customize ShareAnalyze Common Data Fabric API SDKs UI
- 46. INCREASE APP DELIVERY VELOCITY IMPROVE CODE QUALITY INCREASE BUSINESS IMPACT Improve the Impact of Application Delivery 47
- 47. For more, please visit Splunk.com • Splunk for DevOps - www.splunk.com/DevOps • Splunk DevOps Ecosystem Apps: splunkbase.splunk.com • Splunk blogs: blogs.splunk.com • Splunk community: www.splunk.com/community • DevOps demo available
- 48. Sources/Additional Reading ● splunk.com/DevOps - Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc. ● dev.splunk.com – Resources for developing with or on ther Splunk platform, incl. SDKs, API Docs, guides, etc. ● blogs.splunk.com – Check the ‘DevOps’ and ‘Ansible’ tags for specifics, including how to deploy Spunk w/ Ansible ● splunkbase.splunk.com – Splunk add-ons and applications incl. Ansible Tower App for Splunk and 1000+ more ● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016 ● 2016 State of DevOps Report, DevOps Research and Assessment ● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis ● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford ● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay ● Metrics that Matter, Mark Michaelis, IntelliTect ● DevOps and the Cost of Downtime: Fortune 1000, IDC ● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014 ● The Seven Habits Of Highly Effective DevOps, Forrester Research, Inc., October 2, 2014 49
- 49. Thank You