Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry | InfluxDays NA 2021
0 likes279 views
The document discusses observability with InfluxDB/IOx and OpenTelemetry. It provides definitions and descriptions of InfluxDB/TSM and InfluxDB/IOx storage engines, highlighting their strengths. It also defines OpenTelemetry as a standard for observability signals including metrics, logs and traces. The document outlines how OpenTelemetry aims to provide a common instrumentation standard to unify the observability stack, and describes how InfluxData supports and integrates with OpenTelemetry.
Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry | InfluxDays NA 2021
- 1. Observability with InfluxDB/IOx & OpenTelemetry Jacob Marble, InfluxData
- 2. © 2021 InfluxData Inc. All Rights Reserved. About Me
- 3. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB/TSM: Definition ● Columnar Series-oriented storage engine ○ Like a log-structured merge-tree ■ Key is roughly concat(tags, field key, timestamp) ■ Value is field value ○ “Column per series” ○ Leverages mmap() syscall ● Series index ○ Index to data by concat(tags, field key) ● Introduced in 2015 ○ Superseded engine based on BoltDB ● Implemented in Golang
- 4. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB/TSM: Good The TSM storage engine is good at: ● Storage capacity: hundreds of GBs ● Storage cardinality: millions of series ● Query limits: selective, analytic ● Strong use cases: IoT, metrics, events*
- 5. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB/IOx: Definition ● Distributed columnar storage engine ○ Parquet ■ Columns are typed data columns ■ Time is a column type ■ Tags, fields: semantic property of columns ○ Arrow, Flight ■ In-memory, network ○ Federated storage, write, query ■ Collection, storage, query ● No series index ○ Partition keys ○ Dictionary encoding ● Under development since 2020 ● Implemented in Rust
- 6. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. InfluxDB/IOx: Better The IOx storage engine is good at: ● Storage capacity: unlimited ● Storage cardinality: unlimited ● Query limits: network protocols ● Strong use cases: IoT, metrics, events, logs, traces, ...
- 7. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: Definition All observability signals in one standard. ✅ Tracing ✅ Logs ✅ Metrics The project has wide industry support. ● Top-tier cloud vendors and APM SaaS are committed. ● SDKs for ~10 languages. ● Second most active CNCF project in 2020. ○ Most active was Kubernetes.
- 8. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Open Source Observability: Today metrics logs traces instrumentation Prometheus printf() Jaeger collection Prometheus Expo Logfmt, JSON Thrift or gRPC delivery Pull via HTTP Push via stdout Push via UDP pipeline Telegraf Fluentd Jaeger storage InfluxDB MongoDB Elasticsearch query Grafana Graylog Jaeger
- 9. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. Open Source Observability: Today application application application application application application application application Telegraf Fluentd Jaeger InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger SDK SDK Prom. Expo Logfmt Jaeger Thrift SDK
- 10. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: Tomorrow metrics logs traces instrumentation OpenTelemetry SDK collection OTLP delivery Push via gRPC pipeline OpenTelemetry Collector storage ? query
- 11. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: Tomorrow application application application application application application application application OTel SDK ? OTLP OTel Collector
- 12. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry SDK: Versatile application application application application application application application application OTel SDK Telegraf Fluentd Jaeger Prom. Expo Logfmt Jaeger Thrift InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger
- 13. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Versatile OTel Collector InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger application application application application application application application application SDK SDK SDK Prom. Expo Logfmt Jaeger Thrift
- 14. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Migration: Incremental InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger application application application application application application application application OTel SDK OTLP OTel Collector
- 15. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry SDK: Span ID Everywhere application application application application application application application application span := newSpan(parent) defer span.Finish() log.Warn("error...") OTel SDK span ID: 3849 log record spanID: 3849 OTel Collector
- 16. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Temporal Aggregation OpenTelemetry Collector - bar = avg(15m) metric: foo gauge 10s metric: bar gauge 15m
- 17. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Spatial Aggregation metric: foo labels: k8s_node metric: bar labels: ∅ OpenTelemetry Collector - bar = sum()
- 18. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Traces→Metrics span: foo duration: 456ms metric: bar histogram OpenTelemetry Collector - if (span = foo) then bar = hist(duration)
- 19. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Tail Sampling span: foo duration: 456ms trace sampled OpenTelemetry Collector - if (span = foo && duration > 100ms) then sample trace
- 20. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: Vendor Neutral application application application application application application application application OTel SDK ? OTel Collector OTLP
- 21. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: SaaS APM application application application application application application application application OTel Collector OTel SDK $aa$ Very Fancy UI OTLP OTLP
- 22. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. OpenTelemetry: Open Source End-to-End application application application application application application application application OTel SDK Something People Want Object Storage InfluxDB/IOx OTel Collector OTLP OTLP
- 23. © 2021 InfluxData Inc. All Rights Reserved. © 2021 InfluxData Inc. All Rights Reserved. InfluxData 💜 OpenTelemetry: Status ● InfluxData common schema for observability ○ github.com/influxdata/influxdb-observability ● OTel Collector sends Line Protocol ● OTel Collector receives Line Protocol ● Telegraf sends OTLP ● Telegraf receives OTLP ● Jaeger UI queries IOx (prototype)
- 24. © 2021 InfluxData Inc. All Rights Reserved. Questions
- 25. © 2021 InfluxData Inc. All Rights Reserved. Thank you Jacob Marble, InfluxData [email protected]