Dip into prometheus
0 likes•115 views
The document discusses cloud monitoring and metrics, highlighting the importance of application metrics over just system metrics, particularly in Kubernetes environments. It evaluates various options for monitoring solutions including vendor-specific tools, dedicated SaaS products, and self-hosted solutions, emphasizing cost considerations and integration capabilities. The presentation also introduces Prometheus as a prominent time series database suitable for metric collection and analysis.
Dip into prometheus
- 1. Dip Into P8s
- 2. hello! I am Zaar Hai Staff Cloud Architect at DoiT International linkedin.com/in/zaar 2
- 4. Cloud Out of the box ✘ VM CPU/Network/Disk stats… ✘ But not memory ➢ requires vendor-specific agents 4
- 5. Cloud Out of the box ✘ VM CPU/Network/Disk stats… ✘ But not memory ➢ requires vendor-specific agents ✘ Gets much better with GKE ➢ Memory usage for pods IF your YAMLs behave 5
- 6. Cloud Out of the box ✘ VM CPU/Network/Disk stats… ✘ But not memory ➢ requires vendor-specific agents ✘ Gets much better with GKE ➢ Memory usage for pods IF your YAMLs behave ✘ All of the above are metrics… But about our app? 6
- 7. App Metrics It’s all about app metrics 7
- 8. Why metics at all? 8
- 9. When logs are not enough ✘ Too detailed to see the big picture ✘ Hard to see KPIs / trends 9
- 10. Logs to metrics ✘ Log-based metrics in StackDriver ➢ Fragile configuration ➢ Disjoint from the app ➢ Vendor specific ✘ Smart log parsers, e.g. Coralogix Parsed 100 docs in 0.31 seconds Parsed 100 docs in 0.38 seconds Parsed 100 docs in 0.34 seconds ✘ An option when retrofitting monitoring on the existing app 10
- 11. So what are metrics? 11
- 12. Metrics are 12 Just a tuple Numeric value Time stamp Metric name
- 13. Metric examples Process A: 2020-07-28T02:32:06Z http_requests_total 1239 2020-07-28T02:32:07Z http_requests_total 1245 Process B: 2020-07-28T02:32:06Z http_requests_total 1185 2020-07-28T02:32:07Z http_requests_total 1185 Now we can aggregate across! 13
- 14. Metric dimensions Process A: 2020-07-28T02:32:06Z http_requests_total 1239 Process B: 2020-07-28T02:32:06Z http_requests_total 1185 14
- 15. Metric dimensions Process A: 2020-07-28T02:32:06Z http_requests_total{code=200} 1227 2020-07-28T02:32:06Z http_requests_total{code=404} 12 Process B: 2020-07-28T02:32:06Z http_requests_total{code=200} 1177 2020-07-28T02:32:06Z http_requests_total{code=404} 8 15
- 16. Metric dimensions 2020-07-28T02:32:06Z http_requests_total{code=200, process=A} 1227 http_requests_total{code=404, process=A} 12 http_requests_total{code=200, process=B} 1177 http_requests_total{code=404, process=B} 8 16
- 17. Metric dimensions 2020-07-28T02:32:06Z http_requests_total{code=200, process=A, path=/foo} 1107 http_requests_total{code=404, process=A, path=/foo} 12 http_requests_total{code=404, process=A, path=/bar} 120 http_requests_total{code=200, process=B, path=/foo} 1005 http_requests_total{code=404, process=B, path=/foo} 8 http_requests_total{code=200, process=B, path=/bar} 172 17
- 18. Now we can graph that! 18
- 20. 1,000Metrics Per MicroService 3,504,000,000Samples per month 80,000Samples per minute 20 x20
- 21. 1,000Metrics Per MicroService 3,504,000,000Samples per month 80,000Samples per minute 21 x20 Just for your app
- 22. Wait, there is more! ✘ 10-20k metrics per average K8s node ✘ That’s 1,200,000/minute for 15 node cluster ➢ Assuming 15s collection interval ✘ Or 52,560,000,000 samples a month! ✘ And that’s just for average sized app 22
- 24. “No matter what you chose, stay within 24 A Single Pane of Glass
- 25. Option 1 Stick with the Vendor 25
- 26. Enhance the existing ✘ Both AWS and GCP give you so much for free already ✘ Just add your app metrics ➢ And they support that! ✘ No need to ship system metrics, e.g. K8s - they are alredy there 26
- 27. But it’s costly ✘ GCP StackDriver ➢ $84/month per 1k metrics ➢ Price drops after 300k metrics 27
- 28. But it’s costly ✘ GCP StackDriver ➢ $84/month per 1k metrics ➢ Price drops after 300k metrics ✘ AWS CloudWatch ➢ $300/month per 1k metrics ➢ $100/month after first 10k, $50 after the first 240k 28
- 29. But it’s costly ✘ GCP StackDriver ➢ $84/month per 1k metrics ➢ Price drops after 300k metrics ✘ AWS CloudWatch ➢ $300/month per 1k metrics ➢ $100/month after first 10k, $50 after the first 240k ✘ Logs are expensive too, btw ➢ GCP SD: $0.50/GB ➢ AWS CW: $0.60/GB + charge of $0.0057 per scanned GB for queries 29
- 30. But it’s costly ✘ 20 μSvc app with 1k metrics and 1GB logs/day per μSvc: ➢ 1*20*30 = 600GB/month ➢ 20k metrics/month ✘ Will cost you: ➢ GCP: $300 for logs + $1760 for metrics ➢ AWS: $360 for logs + $4000 for metrics ✘ It’s only half a story! ➢ With containers metrics are short lived 30
- 31. Further considerations ✘ Vendor specific APIs to ship ➢ Challenging for multi-cloud ➢ Gets better with K8s ✘ Limited to 1 minute resolution 31
- 33. There are many out there ✘ DataDog, Sysdig, NewRelic, Splunk, SumoLogic, Grafana (hosted) ✘ Once you see the pricing, GCP/AWS $-figures make sense :) ✘ Lot’s of added features though: ➢ AI-assisted anomaly detection, etc. ✘ Multi-cloud! 33
- 34. There are many out there ✘ DataDog, Sysdig, NewRelic, Splunk, SumoLogic, Grafana (hosted) ✘ Once you see the pricing, GCP/AWS $-figures make sense :) ✘ Lot’s of added features though: ➢ AI-assisted anomaly detection, etc. ✘ Multi-cloud! ✘ But now you need to ship all your system metrics ➢ Can become expensive quickly 34
- 35. Option 3 Host your own 35
- 36. Simpler than it may sound Instrument Collect & Store Display & Alert 36
- 37. Grafana to Display (and Alert) De-facto dashboarding software for DevOps and beyond 37
- 38. Grafana Multiple Data sources 38 GCP Pub/Sub AWS SES Email Parser
- 39. Grafana Multiple Data sources 39 GCP Pub/Sub AWS SES Email Parser One Grafana Dashboard StackDriver CloudW atch Prometheus
- 40. Hybrid SaaS As a hybrid SaaS, or “Option 2.5” you can: ✘ Setup hosted Graphana on Graphana Labs ✘ Connect it to CloudWatch, StackDriver, etc. ✘ Ship your app-only metrics to Graphana Labs ➢ At $16/month per 1k metrics ✘ Still limited for 1 minute resolution for CloudWatch/Stackdriver 40
- 42. Where to? ✘ We have 3 billion app / 50 billion system metric samples per month ✘ Storage size per sample matters here 42
- 43. Where to? ✘ We have 3 billion app / 50 billion system metric samples per month ✘ Storage size per sample matters here ✘ MySQL ➢ ~50 bytes per sample (including indexing, etc) ➢ 2.3TB for 50b samples ✘ ElasticSearch ➢ ~20 bytes per sample ➢ 930GB for 50b samples 43
- 44. General purpose DBs are expensive ✘ MySQL ➢ $230 for 1 month retention ➢ $1380 for 3 month retention ✘ ElasticSearch ➢ $93 for 1 month retention ➢ $550 for 3 month retention ✘ That’s just for storage! For one app! 44
- 45. But metrics data is unique ✘ Immutable (no updates) ✘ Write once ✘ Lots of metrics do not change often ✘ And this is why Time Series Databases were born! 45
- 47. Prometheus at glance ✘ Not a first TSDB, but became a golden standard ✘ 1-2 bytes per sample ➢ $30-$60 storage cost for 3 month retention as in the previous example ✘ Can process 1 million samples per minute on your laptop 47
- 48. Not just TSDB ✘ Prometheus discovers: ➢ Your GCE VM ➢ Your GKE pods ✘ Prometheus pulls metrics from targets ✘ Prometheus stores metrics and allows you to query them OR ✘ Federates them further to a central storage 48
- 49. Collection at glance 49 GKE Cluster POD POD POD P8S
- 50. Collection at glance 50 GKE Cluster POD POD POD P8S VM VM VM P8S
- 51. Collection at glance 51 GKE Cluster POD POD POD P8S VM VM VM P8S P8S Thanos VictoriaMetrics etc. Grafana
- 53. Python example 53 import time from flask import Flask from prometheus_client import start_http_server, Summary app = Flask(__name__) REQUEST_TIME = Summary("request_processing_seconds", "Time spent processing request") @app.route("/") @REQUEST_TIME.time() def hello_world(): return "Hello, World!n" if __name__ == "__main__": start_http_server(8081) app.run(port=8080) Dedicated port!
- 54. Python example - in action! 54 $ python app.py & $ curl localhost:8080 $ curl localhost:8080 $ curl localhost:8081 # HELP python_gc_objects_collected_total Objects collected during gc # TYPE python_gc_objects_collected_total counter python_gc_objects_collected_total{generation="0"} 247.0 python_gc_objects_collected_total{generation="1"} 151.0 python_gc_objects_collected_total{generation="2"} 0.0 # HELP python_gc_objects_uncollectable_total Uncollectable object found during GC # TYPE python_gc_objects_uncollectable_total counter python_gc_objects_uncollectable_total{generation="0"} 0.0 python_gc_objects_uncollectable_total{generation="1"} 0.0 python_gc_objects_uncollectable_total{generation="2"} 0.0 # HELP python_gc_collections_total Number of times this generation was collected # TYPE python_gc_collections_total counter python_gc_collections_total{generation="0"} 60.0 python_gc_collections_total{generation="1"} 5.0 python_gc_collections_total{generation="2"} 0.0 # HELP python_info Python platform information # TYPE python_info gauge python_info{implementation="CPython",major="3",minor="8",patchlevel="3",version="3.8.3"} 1.0 # HELP process_virtual_memory_bytes Virtual memory size in bytes. # TYPE process_virtual_memory_bytes gauge process_virtual_memory_bytes 2.34852352e+08 # HELP process_resident_memory_bytes Resident memory size in bytes. # TYPE process_resident_memory_bytes gauge process_resident_memory_bytes 2.6411008e+07 # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds. # TYPE process_cpu_seconds_total counter process_cpu_seconds_total 0.29000000000000004 # HELP process_open_fds Number of open file descriptors. # TYPE process_open_fds gauge process_open_fds 7.0 # HELP process_max_fds Maximum number of open file descriptors. # TYPE process_max_fds gauge process_max_fds 1024.0 # HELP request_processing_seconds Time spent processing request # TYPE request_processing_seconds summary request_processing_seconds_count 2.0 request_processing_seconds_sum 1.3547949492931366e-05 # HELP request_processing_seconds_created Time spent processing request # TYPE request_processing_seconds_created gauge request_processing_seconds_created 1.5959190974287152e+09
- 56. More to come in Part II 56