Unified Observability - Alkin Tezuysal - FOSSASIA Summit March 2025 .pdf
0 likes120 views
As modern infrastructures become more complex, traditional monitoring tools often struggle to provide a complete understanding of system health. This presentation will explore how ClickHouse, a high-performance columnar database, enables unified observability by consolidating metrics, logs, traces, and eBPF data into a scalable platform. Attendees will learn how fragmented telemetry systems can hinder root-cause analysis and operational efficiency and how ClickHouse overcomes these challenges by offering seamless integration with tools like OpenTelemetry, Grafana, Prometheus, and more. We will also compare observability solutions, such as Coroot, qryn, and custom stacks with Jaeger and Loki, to demonstrate how ClickHouse simplifies data management, enhances real-time analysis, and supports large-scale telemetry use cases. This session will provide practical insights into implementing ClickHouse for observability, enabling engineers to focus on system insights while minimizing infrastructure complexity.
More Related Content
Similar to Unified Observability - Alkin Tezuysal - FOSSASIA Summit March 2025 .pdf (20)
Unified Observability - Alkin Tezuysal - FOSSASIA Summit March 2025 .pdf
- 1. @2025 Altinity, Inc. Unified Observability Alkin Tezuysal - FOSSASIA Summit March 2025 Leveraging ClickHouse as a Comprehensive Telemetry Database
- 2. @2025 Altinity, Inc. Let’s get connected! Alkin Tezuysal - Director of Services @AltinityDB ● Linkedin : https://www.linkedin.com/in/askdba/ Open Source Database Evangelist ● Previously ChistaDATA, PlanetScale, Percona and Pythian as Senior Technical Manager, SRE, DBA ● Earlier in life Enterprise DBA , Informix, Oracle, DB2 , SQL Server ● Recent Recognitions: ○ Most Influential in Database Community 2022 - The Redgate 100 ○ MySQL Cookbook, 4th Edition 2022 - O'Reilly Media, Inc. ○ MySQL Rockstar 2023 - Oracle (MySQL Community) ○ Database Design and Modeling with PostgreSQL and MySQL 2024 - <Packt> @ask_dba @2025 Altinity, Inc. Born to Sail, Forced to Work!
- 3. @2025 Altinity, Inc. Sailing Trivia What is the purpose of a “boom” on a sailboat?
- 4. @2025 Altinity, Inc. AGENDA ● Observibility Basics ● Introduction to ClickHouse ● Open Telemetry aka OTel features ● Monitoring tools vs DIY observibility
- 5. @2025 Altinity, Inc. Challenges with Disparate Telemetry Systems Separate tools for metrics, logs, and traces.
- 6. @2025 Altinity, Inc. —Grafana State of Observability Report “Minimum no. of o11y tools deployed by a typical organization ” 6
- 7. @2025 Altinity, Inc. Metric s Exporters - Where’s the problem? OpenTelemetry Jaeger Scrapers - What’s the problem? Fluentd Logstash Vector Monitoring - Is there a problem? Grafana Kibana PMM Traces Logs
- 8. @2025 Altinity, Inc. The OpenTelemetry Project Observability = Visibility + Understanding
- 9. @2025 Altinity, Inc. What’s stored? ● Metrics, traces, logs, profiles, events ● Resource metadata ● Graphs & topologies ● Snapshots & deltas ● Configuration
- 10. @2025 Altinity, Inc. Observibility Pipeline
- 11. @2025 Altinity, Inc. Storage? Challenge Solution is ClickHouse
- 12. @2025 Altinity, Inc. Introducing ClickHouse Massively scalable Can scale both horizontally and vertically Fast execution Columnar vectorized execution SQL Compatible Support for ANSI SQL
- 13. @2025 Altinity, Inc. Capabilities of column store is not limited to Full-text search Efficient compression Real-time analytics Relational Petabyte-scale
- 14. @2025 Altinity, Inc. ClickHouse for Observability Open Telemetry De-facto standard for traces, metrics, logs ClickHouse Fast, reliable and scalable column store.
- 15. @2025 Altinity, Inc. How tables written in ClickHouse? Index Columns Index Columns Index Columns Part Part Rewritten, Bigger Part Update and delete also rewrite parts
- 16. @2025 Altinity, Inc. MergeTree family Fully merged part Unmerged, freshly inserted part Query efficiency
- 17. @2025 Altinity, Inc. How does this help? Fast writes Time-friendly Easy cleanup Cost-effective
- 18. @2025 Altinity, Inc. Data Transformation & Management • Materialized Views • TTL • Tiered storage
- 19. @2025 Altinity, Inc. Integrations • Grafana Datasource Plugin • Jaeger w/ ClickHouse backend • cLoki • Kafka table engine
- 20. @2025 Altinity, Inc. Integrations via OpenTelemetry
- 21. @2025 Altinity, Inc. More Benefits • Excellent compression, even with variable schemas • Practically unlimited cardinality • Horizontally scalable ingestion & querying
- 22. @2025 Altinity, Inc. Challenges • SQL is not PromQL* • Overly complex for small data volumes* • Not a turn-key solution
- 23. @2025 Altinity, Inc. Finally "The OpenTelemetry project does not include any kind of database or backend UI."
- 24. @2025 Altinity, Inc. Complete observability solution
- 26. @2025 Altinity, Inc. Coroot Batteries-included, no-code observability
- 30. @2025 Altinity, Inc. qryn "Querying" — LogQL, PromQL, and TempoQL for OpenTelemetry sources, with ClickHouse storage
- 32. @2025 Altinity, Inc. SOMETIMES, COMPARISONS ARE GOOD coroot qryn • eBPF-based Node-Agent • OTLP ingestion via Collector Gateway • Uses (mostly) standard OpenTelemetry Exporter schema + new schema for profiles • Uses on its own collector exporter / collector distribution • Exposes Tempo, Loki, OTLP, and Prometheus APIs • Projects into compatible formats using Materialized Views
- 33. @2025 Altinity, Inc. Schema Considerations • ZSTD Compression • Delta encoding • Bloom filter indexes for maps (resources) and logs • MergeTree, partitioned on time • 7-day TTL
- 34. @2025 Altinity, Inc. OpenTelemetry Collector Exporter for ClickHouse • Maps for metadata • Efficient full-body text-search • Materialized View for span durations
- 35. @2025 Altinity, Inc. qryn • Fingerprints for unique time series • Indexed labels (via Materialized Views) • Allows for efficient updates (ReplacingMergeTree) • Null Engine for raw ingest
- 36. @2025 Altinity, Inc. Scaling for Production High-availability considerations & architecture
- 37. @2025 Altinity, Inc. Managing Multiple Collectors
- 38. @2025 Altinity, Inc. The Altinity Operator • PVC management • Rolling upgrades • Built-in monitoring
- 39. @2025 Altinity, Inc. Alerting & Other Considerations Set up alerts in Grafana based on query thresholds. Integrate with notification channels like email, Slack, or PagerDuty.
- 40. @2025 Altinity, Inc. Conclusion - Why Unified Observability Storage? • Simplified management • Simplified scaling • Cost management • Standardization and normalization of metadata • Post-hoc dependency mapping • Cross-signal correlation
- 41. @2025 Altinity, Inc. Thank You Q&A