DevOps Cardiff - Monitoring Automation for DevOps
2 likes994 views
This document discusses monitoring for online services and microservices. It notes that companies are moving to public/private clouds and microservices while also adopting DevOps practices. Traditional monitoring stacks are discussed, as well as how Netflix and Expedia monitor usage trends rather than individual alerts. Data from 60 companies is presented on how monitoring needs change as servers increase. The key is allowing development teams to own monitoring for the services they create and support. Dataloop aims to provide simple, self-service metric collection and dashboards so monitoring is decoupled from release cycles and more data-driven.
DevOps Cardiff - Monitoring Automation for DevOps
- 1. www.dataloop.io | @dataloopio | [email protected] Monitoring for Online Services
- 2. Obligatory about me slide.. • Born a SysAdmin • Always worked in software companies • Head of Ops a bunch of times • Helped launch quite a few SaaS products • Founded Dataloop.IO end of 2013 • Worked solely on monitoring ever since
- 3. Context of talk Companies running online services at scale who are are on public / private cloud, moving to micro-services and ‘doing’ devops
- 4. The World As We Know It Application MySQL Database OpsView (Nagios) Logstash ElasticSearch Kibana AppDynamicsPingdom GraphiteCollectD PagerDuty Amazon AWS Alfresco JVM SOLR Transformations Browser Google Analytics Custom Scripts Reporting System SQL DB’s Mixpanel GoSquared Geckoboard
- 5. Netflix’s “Monitoring System” “We aggregate the call volume from all of our offices and when that changes significantly we investigate. It then might be a case of letting Microsoft know that Xbox is down”
- 6. Expedia’s “Monitoring System” “We have a very consistent shape to our bookings graph. If that changes by more than a few % throughout the day we know something is up even if nothing else is alerting”
- 7. Our Sample
- 9. How Your Tools Change with No. of Servers
- 10. Your Typical Monitoring Stack Is my site up or down? (External) What happened? (Logs) How is my application performing? (APM) What’s my app actually doing? (Custom Metrics) Is everything working as expected? (Service) Dashing (Custom Dashboards)
- 11. Your Automation Today • Tied to releases • Often a bit of a mess • Ops centric • High friction • Bus factor
- 13. Attempts to improve things • Educate, educate, educate! (and then encourage devs to learn cm) • Automation and ‘just drop the script here’ in the repo • AppDynamics and New Relic • Sensu, Check_MK and custom portals / wiki pages etc
- 14. Why?
- 15. Making Software Is Comparable to Manufacturing
- 16. Books!
- 17. How Software Is Developed
- 18. The Solution Technology only gets you so far. To be really successful you need to devolve monitoring down to the teams that own their part of the service.
- 19. The key to good monitoring
- 20. Why You Need An Operational Historian http://en.wikipedia.org/wiki/Operational_historian
- 21. Developers Must Own Your Monitoring • Devs outnumber Ops 7 – 1 in micro-services • The best person to write the check script • Responsible for the service not just code • Production data makes everyone cleverer • Bridges the gap between Dev + Ops • Data driven decisions!
- 22. Our focus at Dataloop Simple setup by Ops team (agent install and tags) Self service metric collection via ‘create in the browser’ script editor Decoupled script deployment from cm via drag and drop Real-time dashboards that anyone can create Simple alerting rules that teams can manage themselves
- 24. Operational Intelligence != Analytics
- 26. Red Stops The floor
- 27. Public Status
- 31. Q&A