Elasticsearch Monitoring in Openshift
1 like411 views
This document summarizes a presentation about monitoring Elasticsearch in OpenShift. It discusses the challenges of running Elasticsearch in OpenShift due to limited resources, and how Prometheus and Grafana are used to monitor Elasticsearch metrics. It also describes the Elasticsearch Operator which manages Elasticsearch clusters through custom resource definitions. Skilled personnel are needed to maintain the cluster through upgrades and troubleshooting.
Elasticsearch Monitoring in Openshift
- 1. ELASTICSEARCH MONITORING IN OPENSHIFT OpenShift Cluster Logging Component Lukáš Vlček SW Engineer, Red Hat DevConf 2019, Brno
- 2. DevConf 2019, Brno 2 AGENDA We have about 20 minutes ● OpenShift Cluster Logging ● Challenges of running Elasticsearch in OpenShift ● Prometheus & Grafana ● Elasticsearch Operator ● Questions
- 4. DevConf 2019, Brno 4 OPENSHIFT CLUSTER LOGGING A container log message flow on a node Kubernetes master cri-o Collector Files on disk send off the node kubelet kubectl logs <pod> /var/log/containers/*.log Container Container Container stdout stderr
- 5. DevConf 2019, Brno 5 OPENSHIFT CLUSTER LOGGING Per Project (Namespace) isolation Pod Container Pod Container Container Namespace Pod Container Container Namespace
- 6. DevConf 2019, Brno 6 OPENSHIFT CLUSTER LOGGING Openshift-logging Project Openshift-logging Namespace Elasticsearch Node Elasticsearch Node Elasticsearch Node Collector Collector Collector ES service Kibana Curator
- 8. DevConf 2019, Brno 8 THE DATA MODEL OPERATIONAL & PERFORMANCE TUNING SKILLED PEOPLE WHAT ARE THE CHALLENGES Scaling Elasticsearch with limited resources is hard
- 9. DevConf 2019, Brno DATA MODEL
- 10. DevConf 2019, Brno 10 How we store logs into individual indices. ● One index per namespace ● One index per day CH#1 - THE DATA MODEL How the data is physically stored is important Pod Container Pod Container Container Namespace Pod Container Container Namespace Today Today Today - 1 Today - 1 Today - 2 Today - 2 Today - 3 Today - 3
- 11. DevConf 2019, Brno 11 Scaling issues: ● Number of indices/shards ● Number of index fields Reduce number of indices/shards: ● Rollover API ● Commercial (requires license) ● Spread the load to more clusters CH#1 - THE DATA MODEL How the data is physically stored is important Way too many shards! Index mapping explosion
- 12. DevConf 2019, Brno PERFORMANCE & OPERATIONAL TUNING
- 13. DevConf 2019, Brno 13 Physical and functional layout: ● Fast local storage ● Dedicated node roles ● Hot-Warm Monitoring and Alerting: ● Commercial (requires license) ● DIY/3rd party solutions CH#2 - OPS & PERF TUNING Topology of Elasticsearch cluster and underlying HW is important
- 14. DevConf 2019, Brno SKILLED PEOPLE
- 15. DevConf 2019, Brno 15 There is always someone who takes care of your cluster and typically performs the following tasks: ● Maintenance ● Upgrades ● Troubleshooting CH#3 - SKILLED PEOPLE Someone gets woke up when it happens
- 16. PROMETHEUS
- 17. DevConf 2019, Brno 17 100% open-source and community driven ● Prometheus is systems monitoring and alerting toolkit ● Pull-based metrics gathering system, simple text format for metrics exposition ● PromQL: powerful query language ● CNCF member project PROMETHEUS https://prometheus.io/
- 18. DevConf 2019, Brno 18 Elasticsearch native plugin ● Adding new HTTP endpoint ● Seamless integration with security existing model (SearchGuard) ● “Low level” Java ● Simple integration testing ● Upgrade require node restart ● https://github.com/vvanholl/elastic search-prometheus-exporter ELASTICSEARCH ➔ PROMETHEUS Elasticsearch does not support metric exposition in Prometheus format Prometheus exporter ● Requires additional container(s) ● More configuration (security) ● “High level” REST client ● Complex integration testing ● Independent upgrades
- 19. GRAFANA
- 20. DevConf 2019, Brno 20 Dashboard to visualize Elasticsearch metrics stored in Prometheus. ● Cluster ● Shrads ● OS & JVM ● Indexing ● Searching ● Caches ● Throttling ● Indices (WIP) ● ...
- 21. DevConf 2019, Brno 21 ELASTICSEARCH-MIXIN Magic of Kubernetes-mixin and Jsonnet templates Bundle of: ● Prometheus recording rules, alerting rules ● Grafana dashboard Build on top of Jsonnet and Kubernetes-mixin. Currently at https://github.com/lukas-vlcek/elasticsearch-mixin. This location is likely to change in the future (community mixins).
- 23. DevConf 2019, Brno 23 ELASTICSEARCH OPERATOR Operator pattern for Cluster Logging and Elasticsearch Elasticsearch (Cluster) Kibana openshift-logging Namespace Fluentd openshift-operators Namespace elasticsearch-operator ClusterLogging CR cluster-logging-operator Elasticsearch CR
- 24. DevConf 2019, Brno 24 OPENSHIFT MONITORING INTEGRATION Prometheus and Grafana artifacts get loaded openshift-logging Namespace openshift-operators Namespace openshift-monitoring Namespace Elasticsearch-mixin artifacts Prometheus Grafana Alert Manager
- 25. DevConf 2019, Brno 25 WHAT THE FUTURE HOLDS Cluster Logging Operator and Elasticsearch Operator are opening new horizons Operator pattern makes deployment and maintenance of complex components simpler ● Where the logs are routed ● Diversified Elasticsearch Topologies ● Modify cluster based on OpenShift Platform Metrics
- 26. TIME FOR YOUR QUESTIONS!
- 28. TRADEMARKS and ATTRIBUTIONS: Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Prometheus is trademark of The Linux Foundation registered in the United Stated and/or other countries. Red Hat and OpenShift are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the U.S. and other countries. Grafana is trademarks of Coding Instinct. Used photographs are public domain sourced from https://www.flickr.com/photos/newcastlelibraries/albums