Log Management Systems
Download as ODP, PDF1 like584 views
This document compares message and metric management solutions like Fluentd and Logstash. It discusses how these solutions can collect, store, visualize, and alert on log and metric data from heterogeneous environments. While commercial solutions like Splunk are very expensive, open source solutions like Fluentd, Logstash, Elasticsearch, and Kibana provide similar functionality through various "bricks" or components at no cost. The document analyzes key differences between Fluentd and Logstash, such as their configuration, buffering approaches, high availability features, and plugin ecosystems.
Log Management Systems
- 1. Log Management Systems A comparison of message and metric management solutions Presenter: Mehdi Hamidi ( @eXtrem0us )
- 2. Introduction
- 4. What is LOG? ●Combination of Time, Tag and Message ●Indicates State of Applications (?) ●Human and Machine Readable Messages (?)
- 7. Level of logs in syslog standard ●From Debug to Panic ●rsyslog, syslog, syslog-ng ●/var/log/syslog /var/log/rsyslog
- 8. Importance of logs ●Companies and Businesses ●Even Personal Use! (Twitter, Sensors,... )
- 9. LogAnalyzer: a simple solution :)
- 10. LogAnalyzer: a simple solution :)
- 11. Importance of Logging Systems WHAT Actually We NEED? Collect Messages Metrics ●Store ●Visualize ●Alert
- 12. Importance of Logging Systems Heterogeneous Environment Write our own script for each type of log (?) Not in an enterprise environment with lots of devices and services! ●Technical Fragility and dependency to Individuals ●Strong Dependency to knowledge about underlying process
- 15. Commercial Solutions Splunk (500M/Day is Free, then: 5,000,000 $) ●Nagios Everything is restricted to Nagios Concept No separation between metrics and messages No stylish diagrams (in free solution) Problems in cloud infrastructure No realtime monitoring No manipulating messages (1,995 $ for commercial solution) ●Online Services
- 16. Good logging system Specifications ●Have a common interface ●Decouple data sources from data outputs Prevent mentioned dependencies No effect of adding new data source/output Reliability Persistent Buffering ●Extensibility High Availability Load Balancing Robustness
- 17. Lots of OpenSource Bricks (OSB!) Logging Systems: ● Fluentd ● LogStash ● GrayLog ● Logalice ● Rsyslog ● Scribe Message Stores: ● ElasticSeach ● Hadoop ● MongoDB ● File ● RDBMS ● Redis ● ... Visualization (Dashboards): ● Kibana ● Grafana ● Gaylog-WebUI ● PacketBeat ● Chronograph ● ... Metric Stores: ● InfluxDB ● Prometheus ● Graphite ● ... Alerting: ● Kapacitor ● Skyline ● Oculus ● Cabot
- 18. Log Nature Semistructured or Unstructured Generated Massively More Written and less Read (That's why we use NoSQL)
- 19. Popular Stacks (metrics): TICK Stack
- 20. Popular Stacks (Messages) : ELK Stack
- 24. Overview Fluentd: Written in Cruby Used in Google Cloud Platform and Kubernetes Maintained by Tresure Data ●Logstash: Written in Jruby Used in ELK Stack Maintained by Elastic Co. ➢Both use their own RubyGems Repo ➢Out of the box nature, less dependencies
- 25. Configuration Fluentd: Each Input is tagged Logs are routed by tags Logstash: All inputs are Gathered and Scattered Conditional Outputs, No tags
- 26. Configuration
- 27. Transport and Buffering Fluentd: built-in LogStash: bundled Redis version 5.3: persistent buffering
- 28. Full Buffer or Output Exception occurrence Fluentd: Exception: streaming Block input plugin: batch Drop oldest chunk: monitoring LogStash: Retry Discard Dead Letter Queuing
- 29. High Availability and Load balancing
- 30. High Availability and Load balancing
- 31. High Availability and Load balancing
- 32. High Availability and Load balancing
- 33. Memory Fluentd: 40 M Logstash: 120 M (in big clusters matter)
- 34. Forwarders ●Fluentd: Fluentbit (Written in C) Fluentd-Forwarder (Written in Go) (all in one) ●LogStash: Filebeat Metricbeat Packetbeat Winlogbeat (beat family: separated component for each purpose)
- 35. Community and Support Fluentd: Poor Japanese Blogs Google Group Logstash: Rich Documents Blogs IRC Meetups and Certs
- 36. Plugins Fluentd Plugins Verified Input/Output 554 44 Filter 90 8 Parser 30 2 Formatter 6 0 Obsolete 8 0 Plugins are maintained more by other people.
- 37. Plugins All Plugins are in a Single GitHub Repo. LogStash Plugins Input 52 Filter 46 Output 55
- 38. Questions?