OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes
0 likes155 views
The document discusses Grafana Labs' open source observability tools including Prometheus, Grafana, Loki, and Tempo. It provides an overview of each tool's capabilities and Grafana Labs' contributions to them. It also describes how Loki provides highly scalable and cost-effective log aggregation through its storage model and query processing.
![sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m]))
Range vector aggregations
● rate() / bytes_rate()
● count_over_time() / bytes_over_time() /
absent_over_time()
● sum_over_time() / avg_over_time() /
min_over_time() / max_over_time () *
Instant vector aggregations
● sum() / min() / max() / avg()
● stddev() / stdvar()
● count()
● topk() / bottomk()
* Unwrapped range aggregations
LogQL: Metric queries](https://image.slidesharecdn.com/osmc-emilasiemes-grafana-221124093328-73fce72e/85/OSMC-2022-The-Power-of-Metrics-Logs-Traces-with-Open-Source-by-Emil-Andreas-Siemes-29-320.jpg)
OSMC 2022 | The Power of Metrics, Logs & Traces with Open Source by Emil-Andreas Siemes
- 1. The Power of Metrics, Logs & Traces with Open Source
- 2. ● Loves running ● Baking sourdough bread https: //www.linkedin.com/in/emil-andreas-siemes-a793926/ Emil Siemes Principal Solutions Engineer
- 3. 1000+ Employees across 40+ countries (50+ in Germany) 1M+ Instances across LGTM Cloud and OSS 13M+ Users across OSS and Cloud Free tier Founded in 2014
- 4. We help our customers Build and Operate resilient systems, to provide reliable services to their customers.
- 5. Investing in the core of observability
- 6. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
- 7. Prometheus Largest contributor - employs 44% of Prometheus maintainers Open source is at the of what we do GRAFANA Vizualize, Correlate & Alert on Multi-sourced Data Grafana LOKI Highly Scalable Log Aggregation Grafana MIMIR Scalable & Performant backend for all time-series metrics Graphite Leading contributor on Graphite Grafana TEMPO Highly Scalable & Cost Effective Tracing OpenTelemetry Large contributor - employs maintainters; first class support in Grafana K6 Modern load testing built for developers Grafana On-Call Oncall management system with simple workflow integrated with your telemetry
- 8. Deployment Flexibility Open-source | Enterprise | Grafana Cloud Big tent Flexibility and openness without limits to integrate 3rd party data Observability by open source A modern, complete and highly modulable platform fully built on OSS A Unique Philosophy : Regain Control Over Your Observability Strategy
- 9. LGTM: A prometheus inspired O11y stack Tempo Backend compatible with OpenTelemtry, inexpensive at scale, 100% sampling is possible Loki Low TCO, Highly scalable and easy to operate logging system Mimir Highly scalable, highly available multi-tenant, durable storage drop-in replacement for Prometheus
- 10. Correlation by design to reduce MTTR Exemplars Labels & service discovery {service="foo", region="eu-west", instance="node-123"} Auto-generated metrics Logs for trace TraceID in logs Logs to metrics extraction S3
- 13. An open source web SDK for frontend application observability
- 15. Frontend Application Observability in Grafana Cloud Fully managed offering for capturing, visualizing, and analyzing frontend application telemetry data within Grafana Cloud, powered by Grafana Faro PRIVATE BETA available today for Grafana Cloud customers
- 16. Getting started with Grafana Faro is super simple Install the Grafana Faro Web SDK: Import it into your app: npm install @grafana/faro-web-sdk import { initializeFaro } from '@grafana/faro-web-sdk' initializeFaro({ url: '<http://my-endpoint>', apiKey: '<api_key>', app: { name: 'my-app' }, });
- 18. Loki
- 19. Out-of-order ingestion Simplified deployment Promtail Kafka support Regex performance improvements Parallelized binary ops in queries Improved tail latencies with request hedging Promtail Cloudflare, Docker, and Graylog support Query multiple tenants at once Faster instant queries Filter & delete certain log lines July 2022 April 2022 Nov. 2021 Loki v2.4.0 Loki v2.5.0 Loki v2.6.0 Loki v2.7.0 Coming soon
- 21. Node #1 Node #2 Loki Promtail Promtail Grafana logcli AlertManager REST API Object Storage Overview Node #n Promtail
- 22. @160014567432 Timestamp Value 22346 Labels/Selectors key-value pairs {app=”nginx”, env=”dev”} Metric name http_request The Prometheus storage model
- 23. 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Content log line Labels/Selectors key-value pairs indexed unindexed GET /about 1034 Debug “page not found” Object Storage Object Storage Aka “the index” Aka “chunks” The Loki storage model
- 24. ● Log any and all formats ● Smaller indexes ● Cheaper to run ● Fast queries ● Cut and slice your logs in dynamic ways Grafana Loki Query time processing VS ● Decide on log formats ● Larger indexes ● More expensive to run ● Faster queries ● Restricted to format chosen at ingestion time Traditional solutions Ingest time processing The better tradeoff
- 25. 10TB 200MB Log Data Index Think of it more like a table of contents than an index. Loki does not index the contents of the logs. Instead, entries are grouped into streams and indexed with Prometheus-style labels. Efficient log storage
- 26. 1PB 80TB Raw Logs Label match Brute force search - heavily parallelized Time frame 1TB 120GB+/s Fast queries Think of it as massively distributed grep.
- 27. Per Second A day A month Active tenants 700MB 43TB 1.25PB 4500 170Cores 1TB RAM 27 Large cluster in Grafana Cloud
- 28. {cluster=”us-central1”, job=~”dev/loki-.*”} |= “trace_id=7ca877dbda” |~ "SeriesStore.*" Label matchers ● = contains string. ● != does not contain string. ● =~ matches regular expression. ● !~ does not match regular expression. Filter expressions ● |= contains string. ● != does not contain string. ● |~ matches regular expression. ● !~ does not match regular expression. LogQL: Log queries Parser expressions Formatter expressions
- 29. sum by (host) (rate({job="mysql"} |= "error" != "timeout" | json | duration > 10s [1m])) Range vector aggregations ● rate() / bytes_rate() ● count_over_time() / bytes_over_time() / absent_over_time() ● sum_over_time() / avg_over_time() / min_over_time() / max_over_time () * Instant vector aggregations ● sum() / min() / max() / avg() ● stddev() / stdvar() ● count() ● topk() / bottomk() * Unwrapped range aggregations LogQL: Metric queries
- 30. Grafana Agent Promtail ● Targets discovery for Kubernetes, Syslog, files and more ● Automatically attach labels to your log lines ● Advanced pipeline mechanism for parsing, transforming and filtering your logs ● Build and expose custom metrics from your logs data But Loki is open. logstash Lambda Loki makes log collection easy
- 31. Single Binary - Testing - Small installations without HA or horizontal scaling needs SSD or Microservices - Horizontal scalability - Separate Read/Write paths - Large installations Grafana Cloud ● Batteries included: ○ Multi-tenant ○ Automatic scaling and upgrades ○ RBAC ○ Per tenant QoS Enterprise Logs ● Self hosted ● Supported by Grafana Labs ● Security out of the box ● Part of the GES stack 31 How to run Loki
- 32. Thank you!