Monitoring Kubernetes with Prometheus
3 likes•2,359 views
The document discusses monitoring Kubernetes using Prometheus, focusing on the benefits of detailed insights and alerting mechanisms. It outlines the architecture of microservices within Kubernetes, including configurations for deployments and scraping metrics. Additionally, it covers the installation process and practical examples for monitoring services and infrastructure with Prometheus and Grafana.
![# HELP etcd_store_writes_total Total number of writes seen …
# TYPE etcd_store_writes_total counter
etcd_store_writes_total{action="compareAndDelete"} 2
etcd_store_writes_total{action="compareAndSwap"} 4016
etcd_store_writes_total{action="create"} 218
etcd_store_writes_total{action="set"} 5
count by(job)(up == 0) / count by(job)(up)
rate(etcd_store_writes_total{action="set"}[1m]))
sum without(action)(rate(etcd_store_writes_total[1m]))
Examples](https://image.slidesharecdn.com/containerdaysnyc-monitoringkuberneteswithprometheus-161124043612/85/Monitoring-Kubernetes-with-Prometheus-6-320.jpg)
![scrape_configs:
- job_name: etcd
static_configs:
- targets: ["172.17.4.51:2379"]
- job_name: kube-components
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- # Custom filtering and label mapping
Configuration](https://image.slidesharecdn.com/containerdaysnyc-monitoringkuberneteswithprometheus-161124043612/85/Monitoring-Kubernetes-with-Prometheus-8-320.jpg)
![# continued
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_service_name]
regex: "kube-(.*)-prometheus-discovery"
- action: keep
source_labels: [__meta_kubernetes_endpoint_port_name]
regex: "prometheus"
- action: replace
source_labels: [__meta_kubernetes_service_name]
target_label: job
regex: "(kube-.*)-prometheus-discovery"
Configuration](https://image.slidesharecdn.com/containerdaysnyc-monitoringkuberneteswithprometheus-161124043612/85/Monitoring-Kubernetes-with-Prometheus-9-320.jpg)
Monitoring Kubernetes with Prometheus
- 1. Monitoring Kubernetes with Prometheus Tobias Schmidt - ContainerDays NYC November 4, 2016 github.com/grobie - @dagrobie
- 3. Prometheus Monitoring system and time series database
- 4. ● Microservice architecture ○ Many more services than in traditional host-based monitoring ○ Short lifecycles ○ Heterogeneous workloads ● Insight ○ Detailed (instance/endpoint/version/... drilldown) and aggregated (across a service) of everything (hardware to service) ○ Trends (act before something becomes a problem) ● Alerting ○ Symptom vs. Cause ○ Grouping, flexible silencing Motivation
- 5. Overview
- 6. # HELP etcd_store_writes_total Total number of writes seen … # TYPE etcd_store_writes_total counter etcd_store_writes_total{action="compareAndDelete"} 2 etcd_store_writes_total{action="compareAndSwap"} 4016 etcd_store_writes_total{action="create"} 218 etcd_store_writes_total{action="set"} 5 count by(job)(up == 0) / count by(job)(up) rate(etcd_store_writes_total{action="set"}[1m])) sum without(action)(rate(etcd_store_writes_total[1m])) Examples
- 7. # prometheus.yaml prometheus.io/docs/operating/configuration/ global: # Settings applying to all jobs scrape_configs: # Define different scrape jobs Rules_files: # Load files specifying rules to pre-calculate expressions # as well as alerts. Configuration
- 8. scrape_configs: - job_name: etcd static_configs: - targets: ["172.17.4.51:2379"] - job_name: kube-components kubernetes_sd_configs: - role: endpoints relabel_configs: - # Custom filtering and label mapping Configuration
- 9. # continued relabel_configs: - action: keep source_labels: [__meta_kubernetes_service_name] regex: "kube-(.*)-prometheus-discovery" - action: keep source_labels: [__meta_kubernetes_endpoint_port_name] regex: "prometheus" - action: replace source_labels: [__meta_kubernetes_service_name] target_label: job regex: "(kube-.*)-prometheus-discovery" Configuration
- 11. ● Pod ○ Group of one or more containers, share context and namespaces ○ Co-located and co-scheduled (allows for side-cars) ● Service ○ Logical set of Pods, stable access points ● Deployment ○ Declaration of the desired state (what to run, how to get there) ● Daemon / Pet / Replica sets ○ Definition of groups of pods (each node / stateful / stateless) ● ConfigMap ○ Configuration data / files (can be mounted in containers) Domain objects
- 12. Workshop Monitoring Kubernetes with Prometheus
- 13. git checkout 1.setup kubectl get nodes
- 14. Running Prometheus in Kubernetes Installation and configuration
- 15. ● What we need ○ Pod specification defining how to run Prometheus ○ Load and manage configuration ○ Service specification to access Prometheus on stable IP ● Options ○ Write own pod+service+petset+... manifests ○ Kubernetes Helm chart in the making https://github.com/kubernetes/charts/pull/151 ○ CoreOS wrote an Operator managing Prometheus and its configuration: https://github.com/coreos/kube-prometheus https://coreos.com/blog/the-prometheus-operator.html Running Prometheus inside of Kubernetes
- 16. Prometheus Operator © CoreOS https://coreos.com/blog/the-prometheus-operator.html
- 18. Monitoring Kubernetes infrastructure Configuration and discovery
- 22. Monitoring services in Kubernetes Configuration and discovery
- 24. Practical examples Queries and dashboards
- 27. My Philosophy on Alerting Rob Ewaschuk https://docs.google.com/docu ment/u/1/d/199PqyG3UsyXlwi eHaqbGiWVa8eMWi8zzAn0Yfc Apr8Q/preview Further reading
- 28. Thank you Tobias Schmidt - ContainerDays NYC November 4, 2016 github.com/grobie - @dagrobie