ELK Wrestling (Leeds DevOps)
3 likes•1,769 views
The document discusses the implementation of an ELK stack for logging and monitoring metrics at a hospitality company, highlighting challenges with off-the-shelf solutions like Splunk. It details the architecture used, the processes for logging various data types, and the importance of resilience and scalability in their setup. The author shares lessons learned from mistakes made during the implementation, emphasizing the value of data accessibility for decision-making.
ELK Wrestling (Leeds DevOps)
- 1. { title: ‘ELK Wrestling’, author: ‘Steve Elliott’, company: ‘LateRooms.com’, type: ‘DevOpsLeeds, @timestamp: ‘2014-10-13T18:30Z’ }
- 2. Featuring Live Demo! Please tweet! Include: “leedsdevops”
- 3. Home growing a metrics culture Needed visibility of live issues Had trialled off the shelf before (Splunk) Hadn’t gained traction Wanted the data still
- 4. Options... Tried Splunk ...Bit pricey, pay for HW and volume of data indexed Looked at cloud based options, were also expensive
- 5. It started with Badger...
- 6. Logging and Monitoring Project Locate and implement the tools we needed Started with Cube for metrics (wouldn’t recommend) Moved onto Logging
- 9. What can we log? Pretty much anything with a timestamp Error log Web logs Proxy logs Releases? Tweets?
- 10. Logstash ELK
- 11. High level architectural design Web servers Queue Dashboards Elasticsearch Rest of Badger
- 12. Real time search and analytics database
- 13. Who’s using it? Certain other hotel website... ...Clever people
- 14. Working with Elasticsearch ● RESTful API ● JSON ● Many libraries to deal with it (new on ElasticLinq for C#)
- 16. Clustering Excellent distributed features Easy to use Node Self discovery Different Node Types (Data, Master, Search, Client) “Live” SSD “Archive” HDD
- 17. More in depth architecture IIS Logs Errors WMI Collector (e.g. Live Server) Queue Forwarder Cube (/TSDB) Search Analytics Rabbit MQ Filter & Forward
- 18. Logstash Inputs Filters Outputs e.g.HTTP logs, UDP, error logs, tweets. e.g. UDP, elasticsearch, graphite, IRC (e.g. Filter, grok, lookup IP, magic…)
- 19. Why the Queue? ● Resiliancy ● Single source of data for everyone ● Logstash used to recommend RabbitMQ, now they recommend Redis ● We still use RabbitMQ, works for us
- 20. Kibana ● Easy to build dashboards ● Gateway drug to ElasticSearch queries ● Examples!
- 24. But...
- 25. Demo
- 26. Mistake: Dashboard Fatigue Too many dashboards to watch! Need to do more on alerting
- 27. Mistake: Using elasticsearch as a TSDB Lots of graphs just cared about top level values, should use a TSDB (such as graphite) instead Elasticsearch use case for more in-depth data analysis
- 28. Mistake: Trying to keep too much data ● Nodes going out of memory or disk space is bad ● Long GC can cause nodes to drop ● Can lead to split brain ● More shards = more memory ● usage, watch your scaling
- 29. Scaling Hit two bottlenecks - Ingestion (solved with SSDs) - Search (solved by scaling horizontally) 1.4.0 brings stability improvements, should handle oom better
- 30. Other Mistakes Should have automated sooner (Good chef/puppet support) Should have used “normal” logstash more More node More awesome??
- 31. What went right? ● Free and easy access to Data ● Doesn’t need to be on elasticsearch, but the tooling makes it easy ● Give people access and they’ll seek out the data to drive decisions - start the feedback loop ● Dev/Test instance
- 32. ELK in the wild Data Driven QA Data Driven...Managering
- 33. But wait, theres more! Curator, Kibana 4 (Woo - aggregations), alerting, linking logs together… Too much to cover here!
- 34. Thanks for Listening! More: elasticsearch.org, logstash.net Blog: www.tegud.net Twitter: @tegud Github: www.github.com/tegud Come say hi!