OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable Advanced Performance Management- Satbir Chahal
0 likes111 views
The document discusses how OpsVerse migrated their Jaeger distributed tracing storage from Cassandra to ClickHouse for improved performance monitoring. Jaeger is an open source distributed tracing system that was originally designed to use Elasticsearch or Cassandra for storage. While Cassandra worked well for basic functionality, it lacked capabilities for advanced analytics. ClickHouse supports richer query functions and better handles large datasets. The document outlines the steps OpsVerse took to implement the ClickHouse storage plugin for Jaeger and deploy ClickHouse on Kubernetes using the ClickHouse Operator. This migration enabled more insightful performance monitoring and analytics.
![®
®
Sleep Easy at Night
● Enable Backups
○ https://github.com/AlexAkulov/clickhouse-backup as a K8s CronJob
○ [In Progress] ClickHouseBackup as a CR:
https://github.com/Altinity/clickhouse-operator/issues/862
● Enable metrics and alerts:
○ /metrics endpoint serves by default (have Prometheus scrape it)
OSACon 2022
“Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse](https://image.slidesharecdn.com/os63a21-221121203635-1bf11406/85/OSA-Con-2022-Switching-Jaeger-Distributed-Tracing-to-ClickHouse-to-Enable-Advanced-Performance-Management-Satbir-Chahal-11-320.jpg)
OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable Advanced Performance Management- Satbir Chahal
- 1. ® ® OSACon 2022 Switching Jaeger Distributed Tracing Storage to ClickHouse for Advanced Performance Monitoring (APM) Satbir Chahal, OpsVerse Open Source Analytics Conference 2022
- 2. ® ® Purpose Why and how OpsVerse migrated Jaeger storage to ClickHouse OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 3. ® ® What is Jaeger? ● A distributed tracing system open-sourced by Uber in 2015, now part of CNCF ● Instrumented apps achieve visibility of a request’s traversal of the system by propagating trace context between services ● Each of these operations (e.g., svc-to-svc request) is known as a span - a collection of spans compose a trace OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse Image courtesy of jaegertracing.io
- 4. ® ® Jaeger Architecture ● Jaeger has native support for Elasticsearch and Cassandra as storage ● Later, gRPC storage plugin support was added (more on this soon) ● Leaned toward Cassandra due to operational overhead of maintaining ES clusters (resources and manual TTLs) ● Cassandra was okay… until we wanted to do more OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse * Image courtesy of jaegertracing.io ** In our case, jaeger-clients are replaced by OpenTelemetry; an OpenTelemetry Collector is between the Application and the Jaeger Collector.
- 5. ® ® Cassandra Pros ● Scalable with near-zero maintenance ● Fast writes - Kafka consumer group lags were also near-zero ● General advantages (not pertaining to traces per se): ○ Fault tolerance ○ Replication OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse Cons ● No JOINs ○ Services index one table ○ Operations index in one table ● Bulk querying load can be expensive ● Minimal aggregate functions So advanced analytics on the data will require a custom process
- 6. ® ® Why ClickHouse (Data) ● Great data compression and batch ingestion ● Some existing familiarity: ○ Our team was exposed to ClickHouse when we ran another open source tool using it as a storage backend (PostHog) ○ Near SQL compatibility (for querying) ● Rich set of functions: ○ Aggregate (topk, stdev, etc) ○ JSON, String, Array, Maps, etc ● Its weaknesses are tolerable for distributed tracing: ○ Data consistency isn’t guaranteed ○ Delete/updates are slow (batch operations) OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 7. ® ® Why ClickHouse (Kubernetes) Some key pre-requisites met: ● Jaeger adds support for gRPC storage plugins ○ Paves way for community to support (virtually) any database ○ Plugin implements the storage gRPC protobuf interfaces ● Community releases https://github.com/jaegertracing/jaeg er-clickhouse/ (mid 2021) OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse So… we need to run a ClickHouse instance (to connect plugin): ● The Kubernetes Operator for ClickHouse (by Altinity) grows in popularity https://github.com/Altinity/clickhouse-operator ● Plenty of community content (blogs, videos, webinars, meetups) allow team to run and PoC quickly ● Becomes the preferred method of running ClickHouse on Kubernetes
- 8. ® ® The Steps ● Swap out Bitnami Cassandra helm chart for the K8s ClickHouse Operator (CHOP) ● Add a ClickHouseInstallation CR template (for CHOP to reconcile) to stack deployment charts ● Make sure the ClickHouse gRPC plugin for Jaeger binary makes it onto the container images that connect to the database: ○ Jaeger Ingester ○ Jaeger Querier ● Update Jaeger helm chart to specify gRPC storage rather than Cassandra OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 9. ® ® Some Gotchas (and Best Practices) ● Get familiar with ClickHouseInstallation Custom Resource spec: ○ It will create a LoadBalancer Service (possibly public) by default (update CR spec templates.serviceTemplates) ○ No “readonly” user by default (update CR spec configuration.users) ○ Similar for default user (limit network CIDRs), retention periods, disk space and resource request/limits OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 10. ® ® Some Gotchas (and Best Practices) … cont’d ● Resource management conflicts with GitOps tools (e.g., ArgoCD) ○ ArgoCD may think it is managing a CHOP-managed resource (like a PVC) ○ This can lead to unintended Terminating state of resource ○ Update ClickHouseInstallation template metadata annotations with: ■ argocd.argoproj.io/compare-options: IgnoreExtraneous ■ argocd.argoproj.io/sync-options: Prune=false ○ Also, can set CHOP config to exclude the label propagation your specific tool depends on OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 11. ® ® Sleep Easy at Night ● Enable Backups ○ https://github.com/AlexAkulov/clickhouse-backup as a K8s CronJob ○ [In Progress] ClickHouseBackup as a CR: https://github.com/Altinity/clickhouse-operator/issues/862 ● Enable metrics and alerts: ○ /metrics endpoint serves by default (have Prometheus scrape it) OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 12. ® ® Begin Querying ● Add the ClickHouse instance as a data source to your favorite data visualization tool (Grafana, Apache Superset, etc) ● Rich functions and parsers allow for some query magic ● Use Materialized Columns (or Views) to create additional columns from existing data - for even faster querying OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 13. ® ® OpsVerse APM Using Jaeger+ClickHouse OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse
- 14. ® ® Q&A OSACon 2022 “Switching Jaeger Storage to ClickHouse to Enable APM Insights” Satbir Chahal, OpsVerse Thank you! @Satbir Chahal | @OpsVerse Community Slack tiny.cc/slck https://opsverse.io