Combining Logs, Metrics, and Traces for Unified Observability
0 likes•745 views
The document discusses the evolution and benefits of a unified observability approach utilizing the Elastic Stack for logs, metrics, and APM, addressing complexities in cloud-native environments. It emphasizes the importance of a single interface for monitoring various data sources, enabling intelligent anomaly detection and improved collaboration between development and operations teams. The pricing model is aligned with business value, promoting flexibility and future-proofing for diverse workloads.
Combining Logs, Metrics, and Traces for Unified Observability
- 1. Cyrille Le Clerc Director of Product Management, Observability June 2020 Combining Logs, Metrics, and Traces for Unified Observability
- 3. Higher resource utilization increases monitoring complexity • Orchestration/Hypervisor • Dynamic/ephemeral jobs • You can no longer "point" to where that job lives Shift to cloud-native yields maintainable code, with costs • Traditional licensing models don't scale as well as your applications • Hurdles with autoscaling Monitoring Complexity Hardware & software trends are evolving in tandem Evolving Architectures ~↑ Monitoring Complexity
- 5. Development Team Ops: Log Monitoring Uptime Response Time Uptime Tool Ops: Infra Monitoring Web Logs App Logs Database Logs Container Logs Log Tool Ops: Service Monitoring Real User Monitoring Txn Perf Monitoring Distributed Tracing APM Tool Container Metrics Host Metrics Database Metics Network Metrics Storage Metrics Metrics Tool Status Quo: Siloed Collection of Tools
- 6. APM Data Uptime DataMetrics DataLog Data Elastic Approach to Observability Uptime Response Time Correctness Certificate Validation Web Logs App Logs Database Logs Container Logs Real User Monitoring Txn Perf Monitoring Distributed Tracing Dependency Mapping Host/Container Metrics Database Metics Network Metrics Storage Metrics Dev & Ops Teams Elastic Common Schema
- 7. Unified User Interface Same UI for KPI dashboards and root-cause analysis
- 8. • Ship data from anywhere -- and correlate across your data sources • The data is yours -- no API rate limiting, no data black boxes • Cloud native scale -- no constraints on dimensions and cardinalities Unified Data Layer with Common Schema Open data keeps your data out of silos
- 9. Correlate multiple data sources for more intelligent anomaly detection Unified Machine Learning and Alerting
- 10. Pricing aligned with business value Unified Licensing Model PER AGENT $$$$ PER HOST $$$$ PER INGEST $$$$ PER MONITOR $$$$ PER ADD-ON $$$$ • Intuitive Single, unified pricing model. No add-ons. • Cloud native No problem using with container workloads and serverless. • Future proof You pay for capacity and are not locked into a specific use case.
- 11. Elastic Stack for logs
- 12. Adopt an Open Approach to Centralized Logging Turnkey data ingestion, intuitive search UI
- 13. Make Logs Actionable with Machine Learning
- 14. Meet Audit Requirements with Log Lifecycle Management You’re in control of how your data is tiered Hot. Warm. Cold. Frozen. Index Lifecycle Management Policy based data management that optimize your cluster behind the scene Cold storage with online search Specialized indices for efficient long-term storage Log archival and rehydratation Robust snapshot management via API and UI
- 15. Elastic Stack for metrics
- 16. Evolution of Elastic Stack to a Metrics Store BKD trees Data structures optimized for numerical time series analysis. Columnar storage Structured data storage, resulting in compact storage and faster analytics Rollups Aggregate older data into bigger time buckets Aggregations framework Analytics features to slice and dice data along various dimensions 2012 2016 2014 2018 2019 2020 Prometheus support Support for ingesting data from Prometheus exporters and server Improved support for histograms Dedicated histogram data type in Elasticsearch
- 17. Turnkey Data Onboarding 100s of data sources at your fingertips Now 100+ integrations and growing! Recently added AWS LAmbda AWS Virtual Private Cloud Amazon Aurora DB Azure Database Metrics Azure Container Metrics Google Cloud Platform Pub/Sub Istio ...
- 18. Turn Metrics into Intelligence Flexible time-series analytics and data visualization
- 19. Elastic as an Infrastructure Metrics Solution
- 20. Elastic Stack for APM
- 21. 21 Elastic APM Elastic joins forces with Opbeat A next-generation APM solution designed for developers 2017 2018 Distributed tracing Distributed tracing with W3C Trace Context standard 2020 2019 Elastic APM GA Native Elastic integration, Agents for Python, Node.js, Ruby, Javascript; Real User Monitoring ● Java ● Go ● .NET ● Node.js ● Javascript ● RUM Language Support ● Python ● Ruby ● Go ● PHP (in dev) • Turnkey agents • Auto-instrument for common frameworks • OpenTracing & OpenTelemetry to avoid lock-in
- 22. 22 Evolution of Elastic Stack to Open Source APM Elastic joins forces with Opbeat A next-generation APM solution designed for developers 2017 2018 Distributed tracing Distributed tracing with W3C Trace Context standard 2020 2019 Service Maps, annotations Full featured UI to navigate APM data Elastic APM GA Native Elastic integration, Agents for Python, Node.js, Ruby, Javascript; Real User Monitoring
- 23. Distributed Tracing End-to-end transaction tracking
- 24. Reduce MTTR with a Unified UI Navigate traces, metrics, and logs in one UI for faster issue resolution
- 25. 25 Elastic Stack for uptime
- 28. 28 Demo
- 29. Demo Demo app: Google’s microservices demo “Online Boutique”
- 30. Demo App Architecture Microservices Architecture
- 32. Observe Anomaly Detection with Machine Learning
- 33. Detect Alerts
- 36. Investigate Single Pane of Glass Events on the system
- 37. Investigate Single Pane of Glass Contextual Links ● Trace logs ● Host logs & Metrics ● Custom links ○ Other monitoring tools ○ Support / dev tools
- 38. Investigate Single Pane of Glass
- 39. Investigate Single Pane of Glass
- 40. Investigate Single Pane of Glass
- 41. Investigate Single Pane of Glass
- 42. Investigate Hich Cardinalities for Better Search
- 43. Investigate Hich Cardinalities for Better Search
- 44. 44 Demo
- 45. Demo Key Takeaways • Unified Observability ○ Single tool ○ Investigate deep in any dimension ○ All data types: logs, metrics, APM, and synthetics ○ All layers: application and infrastructure ○ High cardinalities for powerful search
- 46. 46 Demo
- 47. What now? Try it yourself!
- 48. While you observe, why not protect? Elastic SIEM & Endpoint
- 49. Questions?