Cortex: Prometheus as a Service, One Year On
4 likes1,623 views
The document discusses the evolution of Cortex, a Prometheus as a service, highlighting its multi-tenancy, scaling, and performance challenges over the past year. Key issues addressed include DynamoDB write throughput, query performance, and costs associated with storage. Future plans involve direct chunk writes, improved indexing, and enhanced storage solutions for better efficiency.
Cortex: Prometheus as a Service, One Year On
- 1. Cortex: Prometheus as a Service, One Year On Tom Wilkie, PromCon 2017 [email protected] https://github.com/weaveworks/cortex
- 4. Cortex: Prometheus as a Service • Natively multi tenant; isolate different customers in the same services. • Different story around scaling & HA • “Virtually infinite” retention and durability • Opportunities for performance enhancements Cortex YourYourYourYour Your Jobs Alertmanager Grafana Prometheus HA
- 5. Frontend Ditributor DynamoDB Memcache Consul Ingester Write requests Read requests Control requests Prometheus Your Jobs S3 Cortex Architecture
- 7. Problem #1: DynamoDB Write Throughput
- 9. Frontend Ditributor DynamoDB Memcache Consul Ingester Write requests Read requests Control requests Prometheus Your Jobs S3Table Manager Cortex Architecture
- 10. Problem #2: DynamoDB Write Throughput, again
- 11. Original schema: • Hash Key: <user ID>:<hour>:<metric name> • Range Key: <label name>:<label value>:<chunk ID> New schema: • Hash Key: <user ID>:<day>:<metric name>:<label name> • Range Key: <chunk ID>:<chunk end time> https://github.com/weaveworks/cortex/pull/262
- 12. Problem #3: Queries of Death
- 13. Frontend Ditributor Querier Table Manager DynamoDB Memcache Consul Ingester Write requests Read requests Control requests Prometheus Your Jobs S3 Cortex Architecture
- 14. Problem #3: Recording rules and alerts
- 15. Frontend Ditributor Querier Table Manager DynamoDB Memcache Consul Ingester Write requests Read requests Control requests Prometheus Your Jobs S3 Ruler Cortex Architecture
- 16. Problem #4: Long tail
- 19. Problem #5: Cost
- 20. S3 DynamoDB IOP Cost ($/IOP) 5x10-6 2x10-7 Storage Cost ($/GB/Month) 0.023 0.250 https://github.com/weaveworks/cortex/issues/141
- 21. 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0.055 0.06 0.005 0.01 0.015 0.02 0.025 Object size (GB) Cost ($) S3 DynamoDB
- 22. Frontend Ditributor Querier Table Manager DynamoDB Memcache Consul RulerIngester Write requests Read requests Control requests Prometheus Your Jobs Cortex Architecture
- 23. Problem #6: DynamoDB, again
- 24. Frontend Ditributor Querier Table Manager BigTable Memcache Consul RulerIngester Write requests Read requests Control requests Prometheus Your Jobs Cortex Architecture
- 25. DynamoDB BigTable 99th Percentile Write Latency (ms) 70-100 50-150 99th Percentile Read Latency (ms) 100-2500 ~250 LOC ~2000 ~400 DynamoDB numbers courtesy of Weaveworks
- 26. Closing thoughts
- 27. 1. DynamoDB Write Throughput 2. DynamoDB Write Throughput, again 3. Recording rules and alerts 4. Long tail 5. Cost 6. DynamoDB, again
- 28. Running for >12months • Availability: querier unavailable for <12hrs ~99.9% • Durability: lost <2 days of data >99.5% • 99th percentile write performance ~60ms • 99th percentile query performance <200ms
- 29. Future • Direct chunk writes from Prometheus to Cortex Chunk Store • Separate ingester index for better load balancing • Use prometheus/tsdb for the ingesters • Etcd & gossip for ring storage • Chunks in Google Cloud Storage
- 30. One more thing…
- 31. I left Weaveworks at the begging of June to focus on Prometheus & Cortex development. Since then I’ve teamed up with David to develop some ideas around Prometheus, logging, and tracing. We’re available for Prometheus hosting, consulting, training and support. email: [email protected]
- 32. Metrics
- 33. Logs
- 34. Traces