Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
8 likes2,453 views
The document outlines strategies for building a resilient log aggregation pipeline using Elasticsearch and Kafka, focusing on efficient log shipping, buffering, and the architecture of data storage and analysis. It emphasizes using smaller, size-based indices for better performance and discusses a hot-cold architecture for optimized resource allocation. Additionally, it highlights the importance of selecting appropriate shippers and offers recommendations for configurations to enhance throughput and availability.
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
- 1. Building Resilient Log Aggregation Pipeline Using Elasticsearch and Kafka Rafał Kuć @ Sematext Group, Inc.
- 3. Next 30 minutes… Log shipping - buffers - protocols - parsing Central buffering - Kafka - Redis Storage & Analysis - Elasticsearch - Kibana - Grafana
- 4. Log shipping architecture File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 5. Focus: Elasticsearch File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 13. Daily indices are a good start 2016.11.18 2016.11.19 2016.11.22 2016.11.23. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches
- 14. Daily indices are a good start 2016.11.18 2016.11.19 2016.11.22 2016.11.23. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches We delete whole indices
- 21. Slice using size Predictable searching and indexing performance Better indices balancing Fewer shards Easier handling of spiky loads Less costs because of better hardware utilization
- 22. Proper Elasticsearch configuration Keep index.refresh_interval at maximum possible value 1 sec -> 100%, 5 sec -> 125%, 30 sec -> 175% You can loosen up merges - possible because of heavy aggregation use - segments_per_tier -> higher - max_merge_at_once-> higher - max_merged_segment -> lower All prefixed with index.merge.policy } higher indexing throughput
- 25. Hot – cold architecture ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold
- 26. Hot – cold architecture logs_2016.11.22 ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold curl -XPUT localhost:9200/logs_2016.11.22 -d '{ "settings" : { "index.routing.allocation.exclude.tag" : "cold", "index.routing.allocation.include.tag" : "hot" } }'
- 27. Hot – cold architecture logs_2016.11.22 ES hot ES cold ES cold indexing
- 28. Hot – cold architecture logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing
- 29. Hot – cold architecture logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing move index after day ends curl -XPUT localhost:9200/logs_2016.11.22/_settings -d '{ "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag” : "cold" }'
- 30. Hot – cold architecture logs_2016.11.23 logs_2016.11.22 ES hot ES cold ES cold indexing
- 31. Hot – cold architecture logs_2016.11.23 logs_2016.11.24 logs_2016.11.22 ES hot ES cold ES cold indexing
- 32. Hot – cold architecture logs_2016.11.23 logs_2016.11.24 logs_2016.11.22 ES hot ES cold ES cold indexing move index after day ends
- 33. Hot – cold architecture logs_2016.11.24 logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing
- 35. Hot – cold architecture summary ES cold Optimize costs – different hardware for different tier Performance – use case optimized hardware Isolation – long running searches don’t affect indexing
- 42. Focus: Centralized Buffer File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 45. Kafka & topics security_logs access_logs app1_logs app2_logs Kafka stores data in topics written on disk
- 50. Things to remember when using Kafka Scales by adding more partitions not threads The more IOPS the better Keep the # of consumers equal to # of partitions Replicas used for HA and FT only Offsets stored per consumer – multiple destinations easily possible
- 51. Focus: Shipper File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 53. Buffers performance & availability batches & threads when central buffer is gone
- 54. Buffer types Disk || memory || combined hybrid approach On source || centralized App Buffer App Buffer file or local log shipper easy scaling – fewer moving parts often with the use of lightweight shipper App App Kafka / Redis / Logstash / etc… one place for all changes extra features made easy (like TTL) ES ES
- 56. Protocols UDP – fast, cool for the application, not reliable TCP – reliable (almost) application gets ACK when written to buffer Application level ACKs may be needed HTTP RELP Beats Kafka Logstash, rsyslog, Fluentd Logstash, rsyslog Logstash, Filebeat Logstash, rsyslog, Filebeat, Fluentd
- 57. Choosing the shipper application rsyslog Elasticsearch http socket memory & disk assisted queues
- 59. What about OS? Say NO to swap Set the right disk scheduler CFQ for spinning disks deadline for SSD Use proper mount options for ext4 noatime nodirtime data=writeback, nobarier For bare metal check CPU governor disable transparent huge pages /proc/sys/vm/nr_hugepages=0