Monitoring, Logging and Tracing on Kubernetes
20 likes6,709 views
The document discusses monitoring, logging and tracing tools for Kubernetes including Heapster, Grafana, Fluentd, Elastic Stack, Jolokia and OpenTracing. It provides examples of deploying Heapster with InfluxDB and Grafana for metrics collection, Fluentd to ingest container logs into Elasticsearch, and using Jolokia and OpenTracing for remote access to JMX metrics and distributed tracing functionality.
Monitoring, Logging and Tracing on Kubernetes
- 1. #Perform2017 with Heapster, Grafana, Fluentd, Elastic Stack, Jolokia and OpenTracing Martin Etmajer @metmajer Dynatrace, Technology Lead Monitoring, Logging and Tracing on Kubernetes
- 3. Wolfgang Amadeus Mozart 1756 – 1791
- 4. The „Jay Z“ of classical music
- 5. Mike Grabner
- 7. Schnitzel
- 8. #Perform2017 Monitoring with Kubernetes Dashboard, Heapster, influxdb & Grafana
- 10. $ git clone https://github.com/kubernetes/dashboard.git $ kubectl create –f src/deploy/kubernetes-dashboard.yaml Example: Deploy Kubernetes Dashboard likely to be present in your kube-system already
- 14. We metrics!♥
- 16. collects resources usage, etc.
- 17. collects resources usage, etc. exposed via REST
- 18. Heapster runs in a Pod
- 20. Example: Heapster (standalone) source Kubernetes REST client Kubernetes Dashboard via built-in client
- 22. Example: Deploy Heapster (standalone) $ git clone https://github.com/kubernetes/heapster.git $ kubectl create –f heapster/deploy/kube-config/standalone/
- 25. Example: Heapster (with influxdb and Grafana) source Kubernetes sinks influxdb grafana query
- 27. Example: Deploy Heapster (with influxdb and Grafana) $ git clone https://github.com/kubernetes/heapster.git $ kubectl create –f heapster/deploy/kube-config/influxdb/
- 29. #Perform2017 Logging with Fluentd and Elastic Stack
- 30. continuous stream of log data for each container
- 31. unified logging
- 34. Example: Fluentd (for container logs to elasticsearch) watch container logs elasticsearch kibana queryingest via configuration
- 36. Example: Deploy Fluentd (for container logs to elasticsearch) $ git clone https://github.com/kubernetes/kubernetes.git $ kubectl create –f kubernetes/cluster/addons/fluentd-elasticsearch/
- 41. remote access to JMX MBeans
- 42. remote access to JMX MBeans exposed via REST
- 43. Java Management Extensions (JMX) to manage and monitor applications Source: https://en.wikipedia.org/wiki/Java_Management_Extensions Dynamic MBean MXBean Standard MBean Instrumentation Level instrument resources MBeanServerAgent Level agent for registered MBeans Connectors Adaptors Remote Management Level provides access to MBeanServerRMI, HTTP/SOAP HTML, SNMP
- 44. Example: Jolokia (get list of accessible MBeans) $ curl http://localhost:8080/jolokia/list includes memory, threading, gc, etc.
- 46. Example: Jolokia (read memory usage data) $ curl http://localhost:8080/jolokia/read/java.lang:type=Memory name of MBean
- 48. Example: Jolokia (read threading data) $ curl http://localhost:8080/jolokia/read/java.lang:type=Threading name of MBean
- 51. a formally specified standard API for reporting with various API implementations
- 53. design goals
- 54. Why distributed (end-to-end) tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls
- 55. Why distributed (end-to-end) tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls What does the architecture look like?What does the architecture look like?
- 56. Why distributed (end-to-end) tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls Where is most of the time spent?
- 57. Why distributed (end-to-end) tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls How are services invoked?
- 58. Why distributed (end-to-end) tracing? Source: https://research.google.com/pubs/pub36356.html initiator services remote calls Which services aren’t invoked?
- 64. Thank you.
Editor's Notes
- #10: get an overview of the applications running on the Kubernetes cluster create, modify and delete Kubernetes resources troubleshooting (logs) https://kubernetes.io/docs/user-guide/ui/
- #20: https://github.com/kubernetes/heapster/blob/master/docs/source-configuration.md https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md https://github.com/kubernetes/heapster/blob/master/docs/model.md