![Writing alerts: a real-world example
- alert: NodeCPUStuckInIOWait
annotations:
message: '{{ $labels.instance }} spent more than half its CPU time in
IOWait in the last 5 minutes'
doc: "This alert fires if CPU time in IOWait mode calculated on a 5
minutes window for a given instance was more than 50% in the last 15
minutes."
expr: |
rate(node_cpu_seconds_total{mode="iowait"}[5m]) > 0.5
for: 15m
labels:
severity: warning](https://image.slidesharecdn.com/valentinaprometheus-200227093724/85/Prometheus-basics-23-320.jpg)
Prometheus - basics
- 1. Cloud-Native Monitoring With Prometheus Valentina Contenti Kubernets Solutions Engineer [email protected]
- 2. What is Prometheus? Community Driven Open-source Monitoring and Alerting stack, ships a time series database, an alerting entity and a number of integration tools to expose metrics. Made for dynamic cloud environments.
- 3. What is Prometheus made for? • Instrumentation for applications and systems • Metrics collection and storage • Querying, alerting, dashboarding
- 4. What is Prometheus not made for? • Logging or tracing • Automatic anomaly detection • Scalable or durable storage
- 5. A brief but distinguished history Started in 2012 as a SoundCloud internal project Second project to join CNCF after Kubernetes Prometheus v1.0.0 released in 2016 Prometheus v2.0.0 released in 2017
- 6. Core features • Simplicity + efficiency • Dimensional data model • Powerful query language - PromQL • Service discovery integration
- 7. Architecture
- 8. Simplicity + efficiency: • Local storage, no clustering • 1 million+ samples/s • Millions of series • 1-2 bytes per sample • HA by running two • Go: static binary
- 9. Data model https://prometheus.io/docs/concepts/data_model/ • Prometheus stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. • Every time series is uniquely identified by its metric name and optional key-value pairs called labels. • The metric name specifies the general feature of a system that is measured
- 10. Time series with labels node_cpu_seconds_total{cpu="0",instance="demo.robustperception.io:9100",job="nod e",mode="idle"} 14838327.84 metric name labels • Flexible • No hierarchy • Explicit dimensions
- 11. Time series with labels • Time series: tuple {time, value}
- 12. PromQL - simple but powerful
- 14. Better, persistent graphs: Grafana https://grafana.com/
- 15. Pulling metrics How do you expose those metrics? • Exporters • Language specific client libraries • Service Discovery
- 17. Exporters/Clientlibs The community has contributed exporters and clientlibs for pretty much everything https://prometheus.io/docs/instrumenting/exporters/ https://prometheus.io/docs/instrumenting/clientlibs/
- 18. Dynamic Environments: new challenges! • Dynamic VMs • Cluster schedulers • Microservices • …many services, dynamic hosts, and ports
- 19. Service Discovery https://prometheus.io/docs/prometheus/latest/configuration/configuration/ Prometheus has built-in support for: • VM providers (AWS, Azure, Google, ...) • Cluster managers (Kubernetes, Marathon, …) • Generic mechanisms (DNS, Consul, Zookeeper, custom, ...)
- 20. Service Discovery https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus- kubernetes.yml Let’s use Kubernetes as an example
- 21. Alerting https://prometheus.io/docs/practices/alerting/ You can write your own alerting rules. (of course, the community has examples/projects)
- 22. Writing alerts: a simple example - alert: <alert_name> annotations: message: '{{ $labels.<some_label> }} <summary>' doc: "<description>" expr: | <condition> for: 15m labels: severity: warning
- 23. Writing alerts: a real-world example - alert: NodeCPUStuckInIOWait annotations: message: '{{ $labels.instance }} spent more than half its CPU time in IOWait in the last 5 minutes' doc: "This alert fires if CPU time in IOWait mode calculated on a 5 minutes window for a given instance was more than 50% in the last 15 minutes." expr: | rate(node_cpu_seconds_total{mode="iowait"}[5m]) > 0.5 for: 15m labels: severity: warning
- 24. Alert Dispatching The alertmanager dispatches alerts to the right channel according to their severity
- 25. Alertmanger https://prometheus.io/docs/alerting/configuration/ You can configure your dispatching service just the way you like it. (again, of course, there are examples)
- 26. To recap: • Simplicity + efficiency • Dimensional data model • Powerful query language • Service discovery integration
- 27. Try it at home! http://demo.robustperception.io:9090/consoles/index.html