The top 5 Kubernetes metrics to monitor
0 likes•1,014 views
This document discusses metrics to monitor in a Kubernetes environment across 5 layers: 1. Application metrics like request rates, errors, and durations 2. Service metrics like database connections and service-specific metrics 3. Kubernetes deployment metrics like available replicas and rolling update status 4. Kubernetes internal metrics like node status and resource availability 5. Host/node metrics like CPU, memory, and disk usage Monitoring all 5 layers provides visibility into the health of applications, services, Kubernetes clusters and underlying infrastructure.
![@bencerillo #sysdigCCWFS
[db].net.connections < 2 conn/s
2. Services metrics
● Many of them
● You know them
● Not with metadata](https://image.slidesharecdn.com/thetop5kubernetesmetricstomonitor-ccwfs-170926232425/85/The-top-5-Kubernetes-metrics-to-monitor-10-320.jpg)
The top 5 Kubernetes metrics to monitor
- 1. Jorge Salamero, Sysdig Evangelist
- 2. Jorge Salamero, Sysdig Evangelist metrics to Monitor 1322 15:22:23.584870025 1 nginx (13057) < sendfile res=37492 offset=37492 1323 15:22:23.584905419 1 nginx (13057) > write fd=4(<f>/var/log/nginx/access.log) s 1324 15:22:23.584957752 1 nginx (13057) < write res=100 data=127.0.0.1 - - [01/Sep/2 "-" "Wget/ 1325 15:22:23.584961686 1 nginx (13057) > close fd=12(<f>/var/www/html/index.html) 1326 15:22:23.584965334 1 nginx (13057) < close res=0 1327 15:22:23.584981869 1 nginx (13057) > setsockopt 1328 15:22:23.585028420 1 nginx (13057) < setsockopt 1329 15:22:23.585038763 1 nginx (13057) > recvfrom fd=11(<4t>127.0.0.1:55800->127.0. 1330 15:22:23.585044661 1 nginx (13057) < recvfrom res=-11(EAGAIN) data= tuple=NULL 1331 15:22:23.585051805 1 nginx (13057) > epoll_wait maxevents=512
- 3. @bencerillo #sysdigCCWFS Monitoring is for SysAdmins or DevOps APM is for developers only
- 4. @bencerillo #sysdigCCWFS Developers run and monitor their services Monitoring is for SysAdmins or DevOps APM is for developers only
- 5. @bencerillo #sysdigCCWFS New infrastructure interdependent layers Services / containers not coupled with nodes Kubernetes metadata for multidimensional metrics How Kubernetes changes monitoring
- 6. @bencerillo #sysdigCCWFS 1. Application 2. Services 3. Kubernetes deployment 4. Kubernetes internals 5. Host / node The 5 layers to monitor in Kubernetes
- 7. @bencerillo #sysdigCCWFS Is your application running properly? RED Rates Errors Duration USE Utilization Saturation Errors + KPI metrics (statsd, JMX, Prometheus)
- 8. @bencerillo #sysdigCCWFS net.http.request.count < 10/s net.http.request.errors > 2/s net.http.request.time > 500ms 1. Application metrics
- 9. @bencerillo #sysdigCCWFS Middlewares NodeJS / JVM / Golang memory, heap, threads, etc Backend MySQL, MongoDB, Redis, Cassandra, etc Are my services working properly?
- 10. @bencerillo #sysdigCCWFS [db].net.connections < 2 conn/s 2. Services metrics ● Many of them ● You know them ● Not with metadata
- 11. @bencerillo #sysdigCCWFS Is Kubernetes doing its job? ● Enough pods/containers running for each app? ● Are pods available? Liveness and readiness probes ● Are deployments stuck on a rolling update? ● Scheduled :: Running :: Available :: Desired :: MaxUnavailable
- 12. @bencerillo #sysdigCCWFS The holy check: kubernetes.deployment.replicas.available < kubernetes.deployment.replicas.desired 3. Kubernetes metrics
- 13. @bencerillo #sysdigCCWFS Are Kubernetes internals healthy? ● Internal services: kubelet, API server, dockerd, etcd, etc ● Are nodes ready? or unschedulable? ● Available resources? ● Events: CrashLoopBackOff, Image pull errors, OOM evictions, etc
- 14. @bencerillo #sysdigCCWFS kubernetes.node.unschedulable > 0 4. Kubernetes internals
- 15. @bencerillo #sysdigCCWFS Are boxes or Cloud infrastructure working? ● Are the nodes up and with resources available? ● CPU, memory, disk, network fs.used.percent > 90% kubernetes.node.outOfDisk
- 16. @bencerillo #sysdigCCWFS 1. Application 2. Services 3. Kubernetes deployment 4. Kubernetes internals 5. Host / node The 5 layers to monitor in Kubernetes