Fluentd and Docker - running fluentd within a docker container
4 likes1,474 views
Fluentd is a data collection tool for unified logging that allows for extensible and reliable data collection. It uses a simple core with plugins to provide buffering, high availability, load balancing, and streaming data transfer based on JSON. Fluentd can collect log data from various sources and output to different destinations in a flexible way using its plugin architecture and configuration files. It is widely used in production for tasks like log aggregation, filtering, and forwarding.
Fluentd and Docker - running fluentd within a docker container
- 1. FLUENTD: UNIFIED LOGGING LAYER John Hammink October 14, 2015 Cask Big Data Application Meetup
- 2. About Me • A recovering software & QA engineer turned digital artist once interested in fractals; • now into data visualization based on large datasets rendered directly to GPU (RGL, various Python GL libraries, etc.) • github: jammink2; twitter: rijksband
- 3. Tweet NOW! “At @caskdata learning how to collect more event data using #Fluentd”
- 4. WHAT’S FLUENTD? An extensible & reliable data collection tool simple core + plugins buffering, HA (failover), load balancing, etc. like syslogd
- 5. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
- 7. ✓ duplicated code for error handling... ✓ messy code for retrying mechnism... Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system bash scripts ruby scripts rsync log file bash python scripts custom loggger cron other custom scripts...
- 9. Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system filter / buffer / route
- 10. extensible
- 11. CORE PLUGINS • Divide & Conquer • Buffering & Retries • Error Handling • Message Routing • Parallelism • Read Data • Parse Data • Buffer Data • Write Data • Format Data Common Concerns Use Case Specific
- 12. architecture
- 13. INTERNAL ARCHITECTURE “input-ish” “output-ish” Input Parser Buffer Output FormatterFilter
- 14. Internal Architecture (Simplified) Input Buffer Output Plugin Plugin Plugin 2012-02-04 01:33:51 myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } time tag record
- 15. Architecture :: Input plugins Input HTTP+JSON (in_http) File tail (in_tail) Syslog (in_syslog) ... Plugin ✓ Receive logs ✓ Or pull logs from data sources ✓ in non-blocking manner
- 16. Architecture :: Output plugins Plugin ✓ Write or send event logs Output File (out_file) Amazon S3 (out_s3) MongoDB (out_mongo) ...
- 17. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety Buffer Memory (buf_memory) File (buf_file)
- 18. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety chunk chunk chunk output Input
- 21. DIVIDE & CONQUER & RETRY error retry error retry retry retry
- 22. reliable process
- 23. THIS?
- 24. OR THIS?
- 25. M X N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer/filter/route
- 26. use cases
- 28. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>
- 31. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match *.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
- 32. FLUENTD ON KUBERNETES (NOV 2015)
- 33. FLUENTD LOGGING DRIVER (APR 2015)
- 34. Tweet Again! “Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver”
- 35. DEMO: FLUENTD + DOCKER
- 36. THANK YOU! AND TREASURE DATA IS HIRING! WWW.TREASUREDATA.COMC/CAREERS