Logging, Metrics, and APM: The Operations Trifecta
1 like2,804 views
The document discusses the integration and benefits of logs, metrics, and application performance monitoring (APM) within a unified operational data stack. It highlights features such as unified dashboards, alerting, and machine learning for anomaly detection, while detailing the evolution and capabilities of the Elastic Stack for these purposes. Additionally, it outlines practical implementations, data sources, and future roadmaps for enhancing operational data analysis.
![Metrics vs Logs
64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events](https://image.slidesharecdn.com/logsmetricsapm-santa-190723161257/85/Logging-Metrics-and-APM-The-Operations-Trifecta-9-320.jpg)
![Metrics vs Logs
64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352
64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
For each event, print out what happened.
Logs are chronological records of events
07/Mar/2017 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA
07/Mar/2017 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB
07/Mar/2017 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC
Every x minutes, measure the CPU load and print it out, and annotate with meta-data.
Metrics are periodic measurements of numeric KPIs](https://image.slidesharecdn.com/logsmetricsapm-santa-190723161257/85/Logging-Metrics-and-APM-The-Operations-Trifecta-15-320.jpg)
Logging, Metrics, and APM: The Operations Trifecta
- 1. !1 Tanya Bragin Sept 2018 Logging, Metrics, and APM: The Operations Trifecta
- 3. !3 Benefits of Logs + Metrics + APM in one stack
- 4. !4 Unified Dashboards Same UI for KPI summaries and root cause analysis
- 5. !5 Unified Alerting Trigger off any operational data to provide unified SLA monitoring
- 6. !6 Unified Machine Learning Correlate multiple data sources for more intelligent anomaly detection
- 7. !7 Operational gains Single technology for operational data saves on administrative costs
- 8. !8 Elastic Stack for logs
- 9. Metrics vs Logs 64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291 64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352 64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253 For each event, print out what happened. Logs are chronological records of events
- 10. Making logging more turnkey with ‘modules’ • Turnkey experience for specific data types • Data to dashboard in just one step • Automated parsing and enrichment • Default dashboards, alerts, ML jobs
- 11. Logging modules System • Linux / MacOS • Windows Events Containers • Docker • Kubernetes Databases • MySQL • PostgreSQL Queues • Kafka • Redis Web servers • Apache • Nginx Audit data • Filesystem • System calls WINLOGBEATFILEBEATAUDITBEAT Infrastructure Applications
- 12. !12 Ad-hoc log search and visualization Kibana Discover, Visualize, Dashboard
- 13. !13 Hot/Warm architectures in EC / ECE • One click hot-warm deployments • Shipped in EC in Aug 2018 • ECE support coming!
- 14. !14 Elastic Stack for metrics
- 15. Metrics vs Logs 64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291 64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352 64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253 For each event, print out what happened. Logs are chronological records of events 07/Mar/2017 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA 07/Mar/2017 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB 07/Mar/2017 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC Every x minutes, measure the CPU load and print it out, and annotate with meta-data. Metrics are periodic measurements of numeric KPIs
- 16. !16 Evolution of Elasticsearch into Metrics Store
- 17. Elasticsearch for search and numerical analytics Inverted Index for full-text search Columnar store for structured data BKD Trees for numerical operations Rollups
- 18. • Elasticsearch primarily used for application search • Lucene data structure: Inverted index Elasticsearch beginnings Circa 2010
- 19. • Elasticsearch 1.0 evolves to support a columnar store (built on top of Lucene “doc values”) • Structured string and numerical data can be stored there for fast retrieval and summarization / analytics Elasticsearch evolving to support analytics ~ 2010 to 2014 https://www.elastic.co/blog/elasticsearch-as-a-column-store
- 20. • Elasticsearch 5.0 adds more data structures for efficient storing and querying numbers (BKD Trees) • These structures become the default storage for numerical and geospatial data in Elasticsearch Elasticsearch storage efficiencies 2016 https://www.elastic.co/blog/searching-numb3rs-in-5.0 1-Dimension 2-Dimensions
- 21. • Elasticsearch 6.0 improves Lucene sparse values storage efficiency (41.5% in Metricbeat index size) Elasticsearch storage efficiencies 2017 https://www.elastic.co/blog/minimize-index-storage-size-elasticsearch-6-0
- 22. Rollup support for long-term retentions https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space Added in Elasticsearch 6.3
- 23. !23 DEMO
- 24. !24 Elastic Stack as a Metrics Solution
- 25. Metrics modules System • Linux • MacOS • Windows • Perfmon Infrastructure Cloud • AWS • GCP • Azure • DigitalOcean • Alibaba Containers • Docker • Kubernetes Virtualization • vSphere PACKETBEATMETRICBEAT Network • Netflow • Packets • TLS Envelope Storage • Ceph LOGSTASHHEARTBEAT
- 26. Applications Datastores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike • Graphite Web servers • Apache • Nginx Other • HAProxy • Zookeeper Queues • Kafka • Redis • RabbitMQ Caches • Memcached Uptime • Heartbeat Custom apps • JMX/Jolokia • PHP-FPM • Golang Metrics modules PACKETBEATMETRICBEAT LOGSTASHHEARTBEAT
- 27. Roadmap: New operational data sources New Beats, Logstash inputs and modules Default actions for existing modules Agentless Shippers • Cloud Monitoring (Azure, Amazon, GCP, …) • Security Analytics (Bro, Suricata, Sysmon,…) • Machine Learning jobs for Docker/Kubernetes • Default alerts for top 5 modules • Deploy as functions • Ship data without needing to tent to infrastructure
- 28. • Correlate data from different sources • Ability to re-use analysis content • Ability to re-use Elastic-provided content Correlation between logs, metrics, and APM Benefits • Version 0.1 published: github.com/elastic/ecs • Working with internal groups to validate • Community feedback welcome! Status Elastic Common Schema
- 29. Visualizing time series data Time Series Visual Builder
- 30. Visualizing time series data Annotations
- 31. !31 Elastic Stack for APM
- 32. What is APM? Example 08:32:10 Request "/api/checkout" 08.32:11 Response "/api/checkout 500 ERROR"
- 33. What is APM? Example 08:32:10 Request "/api/products/top" 08.32:17 Response "/api/products/top 200 OK" 7 seconds - zZzzZZz
- 34. How does APM work? Data processor apm-server Data storage elasticsearch Browser Agent Web server Agent Web server Agent Web server Agent UI kibana Browser Agent Browser Agent
- 35. • Focuses on search experience on top of APM data • ‘Just another index’ in Elastic Stack Elastic APM APM adds end-user experience and application-level monitoring to the stack Language support ● Python ● Node.js ● Ruby (Beta) ● RUM (Beta) ● Java (Beta) ● Go (Beta)
- 36. Curated UI for APM Combine custom workflow with freedom of search
- 37. Roadmap: Distributed Tracing Trace and map across multiple services • See the end-to-end view and navigate to individual transactions • Based on the notion of a end-to- end Trace ID across services • Investigating compatibility with OpenTracing API and aligning with W3C trace context spec
- 38. Single transaction Distributed tracing Transaction Span Span Span HTTP request Response
- 39. Distributed tracing example Distributed tracing Trace A Transaction 1 Span Span Span Transaction 2 Span Transaction 3 Span Span
- 40. APM is another index in Elasticsearch Need another visualization? Build a dashboard, no need to wait for your vendor
- 41. !41 DEMO
- 42. !42
- 43. What now? Try it yourself!
- 44. !44 Come to Speaker AMA! Questions?