Why you should use Elastic for infrastructure metrics
0 likes471 views
The document presents the advantages of using Elastic for infrastructure metrics, emphasizing its ability to manage the complexities of modern dynamic ecosystems. It outlines core features such as centralized ingest, flexible alerting, and intelligent monitoring that adapts to evolving architectures. Additionally, it discusses the need for actionable insights through dashboards, visualizations, and automated machine learning for anomaly detection.
Why you should use Elastic for infrastructure metrics
- 1. 1 Why you should use Elastic for Infrastructure Metrics Dimitri Mazmanov Principal Product Manager Observability Carlos Pérez-Aradros Tech Lead Observability
- 2. 2 This presentation and the accompanying oral presentation contain forward-looking statements, including statements concerning plans for future offerings; the expected strength, performance or benefits of our offerings; and our future operations and expected performance. These forward-looking statements are subject to the safe harbor provisions under the Private Securities Litigation Reform Act of 1995. Our expectations and beliefs in light of currently available information regarding these matters may not materialize. Actual outcomes and results may differ materially from those contemplated by these forward-looking statements due to uncertainties, risks, and changes in circumstances, including, but not limited to those related to: the impact of the COVID-19 pandemic on our business and our customers and partners; our ability to continue to deliver and improve our offerings and successfully develop new offerings, including security-related product offerings and SaaS offerings; customer acceptance and purchase of our existing offerings and new offerings, including the expansion and adoption of our SaaS offerings; our ability to realize value from investments in the business, including R&D investments; our ability to maintain and expand our user and customer base; our international expansion strategy; our ability to successfully execute our go-to-market strategy and expand in our existing markets and into new markets, and our ability to forecast customer retention and expansion; and general market, political, economic and business conditions. Additional risks and uncertainties that could cause actual outcomes and results to differ materially are included in our filings with the Securities and Exchange Commission (the “SEC”), including our Annual Report on Form 10-K for the most recent fiscal year, our quarterly report on Form 10-Q for the most recent fiscal quarter, and any subsequent reports filed with the SEC. SEC filings are available on the Investor Relations section of Elastic’s website at ir.elastic.co and the SEC’s website at www.sec.gov. Any features or functions of services or products referenced in this presentation, or in any presentations, press releases or public statements, which are not currently available or not currently available as a general availability release, may not be delivered on time or at all. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. Customers who purchase our products and services should make the purchase decisions based upon services and product features and functions that are currently available. All statements are made only as of the date of the presentation, and Elastic assumes no obligation to, and does not currently intend to, update any forward-looking statements or statements relating to features or functions of services or products, except as required by law. Forward-Looking Statements
- 3. 3 Evolving Architectures ~↑ Monitoring Complexity Hardware & software trends are evolving in tandem Higher resource utilization increases monitoring complexity • Orchestration/Hypervisor • Dynamic/ephemeral jobs • You can no longer "point" to where that job lives Shift to cloud-native yields maintainable code, with costs • Traditional licensing models don't scale as well as your applications • Hurdles with autoscaling Monitoring Complexity
- 4. 4 Applications VMs/Containers Other DBs, Services & Middleware Orchestration InfrastructureAPM Metrics Logs Uptime Uptime APM Metrics APM Logs APM APM Metrics Logs Uptime Metrics Logs Uptime APM
- 5. 5 • Support the full stack • Easily ingest from new sources • Monitor dynamic ecosystems • Ability to interact with your data – Aggregations and visualizations – Different views based on who is looking • Rich and flexible alerting • Long term, reliable storage • Bonus points for full Observability Needs from a monitoring solution Core features and functionally
- 7. 7
- 8. 8 Instructions right in Kibana Growing list of integrations ● Download and install Metricbeat ● Edit the configuration for destination ● Enable and configure the module ● Start the beats ● Explore!
- 9. 9 ● Deploy Elastic Agent ● Choose the integration type ● Register and configure the data source ● Specify the data you want to collect ● Explore! Elastic Fleet Centralized ingest and configuration
- 10. 10 Use your existing shippers Core features and functionality Your App Prometheus Exporter Your App Prometheus Exporter Metricbeat + Elasticsearch Prometheus Server Metricbeat + Elasticsearch Azure Monitor
- 11. 11 Autodiscover Automatically monitor new containers ● Perfect for dynamic ecosystems ● Automatically picks up new instances ● Works with K8s, Docker, AWS, etc. ● Hints based auto-discovery for K8s ● Full context backed by Elastic Common Schema
- 12. 12 Elastic for time series
- 13. Storing Metrics in Elasticsearch ● Metrics stored as numeric fields ○ Depending on expected values: float, double, integer... ● Dimensions/labels normally stored as keyword ● Several metrics per document ○ more efficient ○ one doc per combination of dimensions (time series) { "@timestamp": "2018-09-27T10:08:38", "system": { "cpu": { "nice": 8, "user": 2, }, “load”: 1.2, }, "host": "frontend01.bigorg.dev", "zone": “europe-west”, ... } Data model
- 14. Storing Metrics in Elasticsearch { "@timestamp": "2018-09-27T10:08:38", "system": { "cpu": { "nice": 8, "user": 2, }, “load”: 1.2, }, "host": "frontend01.bigorg.dev", "zone": “europe-west”, ... } Correlation
- 15. 15 Elastic Common Schema Established, predictable fields
- 16. ● Several types for numbers double, integer, float depending on size needs… ● Distributed Histograms (7.6 ● IPs query by IP/subnet ● Geo Map your metrics ● Dates Rich typing and filtering Much more than single type numbers and string labels
- 17. Powerful aggregations • Common metric aggs (sum, avg, count, min, max…) • With more choices on top! – Mutate data / calculate metrics at query time with scripting – Grouping is not limited to labels: Geo proximity, filters, ranges
- 18. Index lifecycle management Reduce storage costs as data ages 1 2 3 1 2 3 Hot Nodes Cold Nodes Warm Nodes 1
- 19. Rollups Reduce storage costs as data ages
- 20. Distributed by design • Horizontally scalable • Cross cluster search • Cross cluster replication Easy to scale
- 21. 21 Powerful data store Beyond Time Series ● Inverted index + columnar store ● Optimized numeric field types (BKD ● Powerful aggregations framework ● Fast response even for high-cardinality queries ● ILM & Data Rollups ● With all of the benefits of the Elastic Stack
- 22. 22 Making metrics actionable with Elastic
- 23. 23 Dashboards & Visualizations Out-of-the-box visibility ● Ship with most integrations ● Mix and match for your needs ● Leverage Kibana drilldowns for custom navigation paths ● Of course, dedicated Metrics and Logs apps
- 24. 24 Metrics App Birds-eye view or drill down
- 25. 25 Integrated Alerting Automatically detect and alert ● Many types of alerts ● Prefiltering based on context ● Multiple facets per alert ○ CPU and Memory ○ Network TX and RX ● Automatically split alerts on chosen field (per container/pod/host) ● Deviations in logging rates
- 26. 26 Machine Learning Automatically detect and alert ● Automate anomaly detection at scale and across disparate data sources ● Find patterns in your logs ● Automatically call out anomalies and outliers
- 27. 27 Full Observability Unified data, UI and alerting
- 28. 28 Thank You!