![Sample Configurations
{
"endpoint" : "http://localhost:8000/nginx_status",
"metrics_config" : [
{ "name" : "activeConnections",
"metric_type" : "gauge",
"units" : "number of active connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Active connections: ([0-9]+)"
},
{ "name" : "reading",
"metric_type" : "gauge",
"units" : "number of reading connections",
"data_type" : "int",
"polling_frequency" : 10,
"regex" : "Reading: ([0-9]+) .*"
}
]
}
Get all prometheus metrics:
{
"endpoint" : "http://localhost:9100/metrics",
}
Get selected prometheus metrics:
{
"endpoint" : "http://localhost:8000/metrics",
"metrics_config" : [
{ "scheduler_binding_latency",
"scheduler_e2e_scheduling_latency",
"scheduling_algorithm_latency"
}
]
}](https://image.slidesharecdn.com/nativecontainermonitoring-150820002027-lva1-app6891/85/Native-container-monitoring-17-320.jpg)
Native container monitoring
- 1. Container native Monitoring Rohit Jnagal Anushree Narasimha
- 3. Overview â Monitoring for containers â Monitoring in a distributed system â cAdvisor Application Metrics â In cAdvisor â Plumbing through a cluster Future Work â Heapster/Kubedash â Templates Outline
- 4. Monitoring in three parts Collection â collecting data and metrics â making it discoverable â plumbing it to higher-level systems Processing â Ingesting, aggregation â Analytics Managing â Actions based on signals â Alerts. Pagers!
- 5. Monitoring in three parts Collection â collecting data and metrics â making it discoverable â plumbing it to higher-level systems Processing â Ingesting, aggregation â Analytics Managing â Actions based on signals â Alerts. Pagers!
- 6. Collection in servers/VMs â Node agent â Knows the binary to monitor â Monitoring logic plugged into agent Moving to containers â Same node agent - understands multiple applications. â Applications can push data to agent. â Sidecars â Applications publish data. Pulled off by off- host agents.
- 7. Near-instant data â schedulers in cluster management tools â load balancers â Alerting systems Slower feedback loops â Autoscaling / Autoupdaters â CPI2 Historical Data â Offline analysis. consumers
- 8. Monitoring in distributed environment ⢠Hosts are invisible ⢠Containers can hop around ⢠Monitoring needs to track and correlate multiple containers
- 9. In Action!
- 10. cAdvisor Analyzes resource usage and performance characteristics of running containers Native Docker support, work with LXC and any other container format Knows containers deeply and monitors their performance github.com/google/cadvisor
- 12. API Node and container spec http://host:8080/api/v2.0/machine http://host:8080/api/v2.0/spec/redis?type=docker Hierarchical Container stats: http://host:8080/api/v2.0/stats/nginx?type=docker http://host:8080/api/v2.0/stats?recursive=true Others: /summary, /events, /storage, /attributes, /ps
- 14. Application Metrics Every container is packaged with its monitoring data. Monitoring moves with the container. Use container composability to stack up metrics from all layers. Use metadata in image or runtime (Docker labels!) to configure monitoring
- 15. Configuring containers FROM redis ADD redis_config.json /var/cadvisor/redis_config.json LABEL io.cadvisor.metric.redis="/var/cadvisor/redis_config.json" In cAdvisor read Labels âio.cadvisor.metric.*â read /rootfs/proc/<pid>/root/<config path>
- 16. Monitoring Configuration Holds metadata about metrics â Endpoint (Location to collect metrics from) â Name of metric â Type (Counter, Gauge, ...) â Data Type (int, float) â Units (kbps, seconds, count) â Polling Frequency â Regexps (Regular expressions to be used to collect a metric)
- 17. Sample Configurations { "endpoint" : "http://localhost:8000/nginx_status", "metrics_config" : [ { "name" : "activeConnections", "metric_type" : "gauge", "units" : "number of active connections", "data_type" : "int", "polling_frequency" : 10, "regex" : "Active connections: ([0-9]+)" }, { "name" : "reading", "metric_type" : "gauge", "units" : "number of reading connections", "data_type" : "int", "polling_frequency" : 10, "regex" : "Reading: ([0-9]+) .*" } ] } Get all prometheus metrics: { "endpoint" : "http://localhost:9100/metrics", } Get selected prometheus metrics: { "endpoint" : "http://localhost:8000/metrics", "metrics_config" : [ { "scheduler_binding_latency", "scheduler_e2e_scheduling_latency", "scheduling_algorithm_latency" } ] }
- 18. App Metrics in action
- 19. Endpoint for custom metrics: http://localhost:8080/api/v2.0/appmetrics/containerName Application metrics being collected can be discovered from the spec: http://localhost:8080/api/v2.0/spec/containerName Regular stats api also reports application metrics: http://localhost:8080/api/v2.0/stats/containerName API
- 20. Kubernetes Open-source Container orchestration from Google Declarative pattern for managing containers Physical hosts abstracted out as resources Inspired and informed by Borg Kubernetes UI powered by cAdvisor github.com/kubernetes/kubernetes
- 21. App Metrics for Kubernetes All kubernetes components run in containers All system services on node runs in containers All kubernetes nodes run cAdvisor (built into kubelet) All kubernetes components expose prometheus metrics App-metrics for kubernetes components!
- 24. Templates Add templates for applications that have stable stats API LABEL io.cadvisor.metric.type=redis Infer monitoring information â Lookup ports through docker inspection â hit known endpoints (e.g.: /metrics for prometheus) â overrides through config
- 25. Tags Adding tags to specific metrics Convey metric intent to processors eg. Autoscalers
- 26. Ongoing work Endpoints Storage drivers Standard config syntax Automagic Plumbing through heapster/kubedash/kubernetes/...
- 27. Thank you! Rohit Jnagal jnagal@google Anushree Narasimha anushree.bnp@gmail cAdvisor github.com/google/cadvisor kubernetes kubernetes.io irc #google-containers