Monitoring and Log Management for
6 likes14,586 views
This document discusses centralized logging and monitoring for Docker Swarm and Kubernetes orchestration platforms. It covers collecting container logs and metrics through agents, automatically tagging data with metadata, and visualizing logs and metrics alongside events through centralized log management and monitoring systems. An example monitoring setup is described for a Swarm cluster of 3000+ nodes running 60,000 containers.
![Fix it – run syslog server first!
docker run -d -p 514:514 factorish/syslog -t tcp
docker run –logging-driver=syslog … nginx
curl localhost:9004
docker logs syslog
==> syslog listening on tcp
<30>Nov 17 18:23:43
nginx#nginx1#afebdfff0eed[1710]: 172.17.0.1 - -
[17/Nov/2016:18:23:43 +0000] "GET / HTTP/1.1" 200
612 "-" "curl/7.49.1" "-"](https://image.slidesharecdn.com/monitoringandlogmanagementfordockerswarmandkubernetes-161207105316/85/Monitoring-and-Log-Management-for-16-320.jpg)
Monitoring and Log Management for
- 1. Monitoring and Log Management for Docker Swarm and Kubernetes Stefan Thies Sematext Group, Inc.
- 2. Sematext & I Logsene SPM log s metric s Docker Agent #nodejs
- 3. Agenda •What is • Centralized Log Management + Performance Monitoring • Kubernetes / Swarm •Container Logs •Container Metrics •Example: #Swarm3k Monitoring •Summary
- 5. Centralized Monitoring Expose Metrics Collect Metrics Ship Metrics Store Metrics Aggregate Metrics Visualize Metrics • Correlation with Logs Anomaly Detection Alerting Server + App / Container Configuration Monitoring Agents Time Series Database Dashboard Tools, Alerting Tools, ChatOps Tools
- 7. Orchestration Container POD Node Node 1 POD 1 Namespace ns1 Kibana Elasticsearch POD 2 Namespace ns2 Redis Services (proxy) Replication Controllers DaemonSets 3 HorizontalPod Autoscaler
- 8. Kubernetes Dashboard / Heapster •Current status •Shows basic resource usage for workloads (Pod) •Simple logs view •Heapster is required for autoscaling features
- 9. Orchestration Container Stacks Nodes Node 1 ELK (compose, app bundle) Kibana 1 Elasticsearch 1 Redis (service) redis1 3 Node 2 ELK Elasticsearch 2 Elasticsearch 3
- 10. Kubernetes != Swarm •Common base is Docker • Docker Logs & Metrics • Docker API
- 11. Container Logs
- 12. Docker Logging Drivers Docker json-file (default) Files journald (CoreOS) System journal Syslog TCP UDP Fluentd TCP $plunk TCP Gelf Centralized Log Management Local Log Shipper
- 13. Docker logs Containers (should) log to stdout/stderr !!! docker logs container_id docker logs container_name Docker API Docker client Container logs
- 14. Fun with Docker logging drivers $ docker run --log-driver=syslog --log-opt syslog-address=udp://$HOSTNAME:514 --log-opt tag=„{{.ImageName}}#{{.Name}}#{{.ID}}" -p 9003:80 –name nginx1 -d nginx $ docker logs nginx 1 "logs" command is supported only for "json-file" and "journald" logging drivers (got: syslog) Add Context!
- 15. More fun with TCP logging drivers! docker run --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:514 --log-opt tag="{{.ImageName}}#{{.Name}}#{{.ID}}" -p 9004:80 -d nginx docker: Error response from daemon: Failed to initialize logging driver: dial tcp 127.0.0.1:514: getsockopt: connection refused.
- 16. Fix it – run syslog server first! docker run -d -p 514:514 factorish/syslog -t tcp docker run –logging-driver=syslog … nginx curl localhost:9004 docker logs syslog ==> syslog listening on tcp <30>Nov 17 18:23:43 nginx#nginx1#afebdfff0eed[1710]: 172.17.0.1 - - [17/Nov/2016:18:23:43 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.49.1" "-"
- 17. Is UDP better?
- 18. Alternatives? Docker Log files json-file or journald API Agent Remote Log Storage Disk Buffer Docker API provides the most complete information! Reliable networks and backend services? Better buffer & retransmit in case of failure! Attach metadata to logs/metrics or route data to different servers or indices? “docker logs” works & logs are stored on local disk! Centralize search, analytics, alerts, access permissions Parse logs
- 19. Automatic tagging of logs, metrics, events • Automatic tagging of log / metrics with • Docker • Container Name / ID • Image Name / ID • Labels / Environment • Hostname / IP • Kubernetes • Namespace, Pod Name , UID • Swarm • Swarm Service Name , ID, Compose Project, Container # (scale) • Single collector for logs, metrics, events, metadata • Base for correlation and visualisation
- 21. Collection
- 22. Metric collection via Docker API
- 23. Smart monitoring agent - all in one Docker API Agent Remote Storage Disk Buffer Docker API provides Labels, Metrics, Logs, Events … Reliable networks and backend services? Better buffer & retransmit in case of failure! Auto-tagging using container labels. Discovery of services Centralize logs, metrics, analytics, alerts, access permissions Metrics, Logs, Events
- 24. Integrate application monitoring in the stack -Custom images - add/remove app with all req. options -Start monitoring, reading config from etcd App Config to expose metrics App Monitor Configured for App Container Service Discovery etcd consul
- 25. Auto Discovery via Docker API and Labels? App Container config to expose metrics App MonitorDocker Monitor run discovery Docker Automatic run
- 27. Node Storage •Good kids clean up their rooms. Good Docker ops clean up their disks by removing unused containers & images.
- 28. Number of containers per host •Verify deployment strategies
- 29. CPU quota per container
- 30. Container memory and OOM counter
- 31. Docker Events
- 33. Limit container resources for your apps! •Set CPU quotas cpu-quota=6000 •Limit Memory and configure App in container to the same limits! m 512mb •Disable Swap: memory-swap=0 •To limit a Docker container from eating all your disk IO use e.g. device-write-bps /dev/sda:1mb
- 34. Automatic Deployment of monitoring agents • One command to run a service on each node joining the cluster • Kubernetes: • DaemonSet creates a pod per node kubectl create -f sematext-agent.yml • Swarm: • Global Service docker service create –mode global ...
- 36. Swarm3k Requirements •Monitoring • Host metrics • Container metrics • Docker Events • Task Monitoring •Collect Container Logs: Task Errors only •3000+ Nodes (actual: 4.7k) •150.000 (actual: 60k) •Duration 8 hours – 28 GB data collected •Public/shared Dashboard for the community
- 37. Pre-flight test with 500 nodes •60.000 containers deployed in less than 5 minutes!
- 38. Swarm3k in one picture
- 39. Limits in visualisation Missing Labels to group hosts or containers
- 40. Summary •Setup of Monitoring & Logging is complex in dynamic environments •Kubernetes != Swarm (yet). Common base: Docker Containers •Smart Agents to collect, analyze, aggregate metrics, events and logs • Auto discovery of containers for data collection • Use metadata tag metrics & logs as base for correlation and visualization • Integrate monitoring in application stacks for app specific metrics • Auto Discovery of services and automatic configuration for application level monitoring
- 41. We are engineers! We develop DevOps tools! We are DevOps people! We do fun stuff ;) join us http://sematext.com/jobs is hiring!
- 42. Thank you for listening! Get in touch! Stefan Thies [email protected] @seti321 http://sematext.com @sematext join us http://sematext.com/jobs Come talk to us at the booth