Logging with Elasticsearch, Logstash & Kibana
92 likes•52,519 views
This document discusses logging with the ELK stack (Elasticsearch, Logstash, Kibana). It provides an overview of each component, how they work together, and demos their use. Elasticsearch is for search and indexing, Logstash centralizes and parses logs, and Kibana provides visualization. Tools like Curator help manage time-series data in Elasticsearch. The speaker demonstrates collecting syslog data with Logstash and viewing it in Kibana. The ELK stack provides centralized logging and makes queries like "check errors from yesterday between times" much easier.
![Curator
1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10!
2 2014-04-21T17:57:19.419 INFO main:333 Job starting...!
3 2014-04-21T17:57:19.420 INFO _new_conn:180 Starting new HTTP connection (1): localhost!
4 2014-04-21T17:57:19.422 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.002s]!
5 2014-04-21T17:57:19.423 INFO main:359 Deleting indices older than 10 days...!
6 2014-04-21T17:57:19.430 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?
expand_wildcards=closed [status:200 request:0.007s]!
7 2014-04-21T17:57:19.433 INFO find_expired_indices:209 logstash-2014.04.21 is 10 days, 0:00:00 above the cutoff.!
8 2014-04-21T17:57:19.433 INFO index_loop:309 DELETE index operations completed.!
9 2014-04-21T17:57:19.433 INFO main:364 Closing indices older than 7 days...!
10 2014-04-21T17:57:19.434 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?
expand_wildcards=closed [status:200 request:0.001s]!
11 2014-04-21T17:57:19.435 INFO find_expired_indices:209 logstash-2014.04.21 is 7 days, 0:00:00 above the cutoff.!
12 2014-04-21T17:57:19.435 INFO index_loop:309 CLOSE index operations completed.!
13 2014-04-21T17:57:19.435 INFO main:369 Disabling bloom filter on indices older than 2 days...!
14 2014-04-21T17:57:19.437 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings?
expand_wildcards=closed [status:200 request:0.002s]!
15 2014-04-21T17:57:19.438 INFO find_expired_indices:209 logstash-2014.04.21 is 2 days, 0:00:00 above the cutoff.!
16 2014-04-21T17:57:19.438 INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.!
17 2014-04-21T17:57:19.438 INFO main:379 Done in 0:00:00.020348.!](https://image.slidesharecdn.com/2014-04-22loggingwithlogstashbastianwidmercampusbern-140423103207-phpapp02/85/Logging-with-Elasticsearch-Logstash-Kibana-46-320.jpg)
Logging with Elasticsearch, Logstash & Kibana
- 1. Bastian Widmer / @dasrecht Logging with Elasticsearch, Logstash & Kibana
- 2. But why?
- 3. „Can you check the errors from yesterday between 15.02 and 15.07“
- 8. Who are you? Bastian Widmer @dasrecht / bastianwidmer.ch Switzerland Development and Operations Engineer
- 9. Agenda 1 Introduction 2 3 4 5 Architecture ELK Stack Tools! Demo
- 12. Elasticsearch
- 13. Elasticsearch • Java • Search and Index • Distributed — Copies & Shards • Clustering • API — JSON / RESTful • Apache Lucene
- 14. Elasticsearch • Index like a Database • Replica Copies for Fault Tolerance • Shard Lucene Instance which indexes the Data see : http://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html
- 15. Elasticsearch
- 16. Logstash
- 17. Logstash • Multiple Input / Multiple Output • Centralize Logs • Collect • Parse • Store / Forward
- 18. Logstash
- 19. The life of an event • Input • Filters • Output • Codecs
- 20. Logstash • JRuby* • >1.4.0 - FlatJAR Release is gone • Instead of running „java -jar logstash.jar“ — „bin/logstash“ • Contrib Plugins • Daily Indices ! * see https://gist.github.com/jordansissel/978956
- 21. Input • File • Syslog • Redis • logstash-forwarder (former Lumberjack)
- 22. Filters • Grok • Mutate • Drop • Clone • GeoIP (!!!)
- 23. Outputs • Elasticsearch • File • Graphite • StatsD
- 24. Logstash 1 input {! 2 stdin { }! 3 }! 4 ! 5 output {! 6 stdout {! 7 codec => rubydebug! 8 }! 9 }! !
- 25. Logstash 1 vagrant@precise64$ ./logstash agent -f 1_simpleconfig.cfg! 2 very important log message!! 3 {! 4 "message" => "very important log message!",! 5 "@version" => "1",! 6 "@timestamp" => "2014-04-21T16:18:02.952Z",! 7 "host" => "precise64"! 8 }
- 26. Logstash 1 input {! 2 stdin { }! 3 }! 4 output {! 5 elasticsearch{! 6 host => "127.0.0.1"! 7 }! 8 stdout {! 9 codec => rubydebug! 10 }! 11 }
- 27. Logstash 1 input {! 2 file {! 3 path => "/var/log/syslog"! 4 start_position => beginning! 5 }! 6 }! 7 ! 8 output {! 9 stdout {! 10 codec => rubydebug! 11 }! 12 elasticsearch{! 13 host => "127.0.0.1"! 14 }! 15 }
- 28. Logstash Errno::EBADF: Bad file descriptor - Bad file descriptor
- 29. Kibana
- 32. Architecture
- 36. Architecture Shipper Shipper Shipper Broker Indexer Search and Storage Syslog Logstash Elasticsearch
- 38. Architecture Shipper Shipper Shipper BrokerBroker Indexer Search and Storage Logstash Redis Logstash Elasticsearch
- 39. Tools! (because anyone needs a bit help)
- 41. • Close • Remove • Backup • Restore elasticsearch-index-mgmt http://s.nrdy.ch/eee
- 42. But then…
- 43. Curator • Time Series Indices? THIS IS THE TOOL! • Close Indexes • Delete (by space or time) • Disable Bloom Filter • Optimize / ForceMerge • https://github.com/elasticsearch/curator
- 44. Curator • Time Series Indexes? THIS IS THE TOOL! • Close Indexes • Remove Indexes • Remove by Space Usage • Disable Bloom Filter • https://github.com/elasticsearch/curator Curator Perfect for Time Series Indexes
- 45. Curator • Close indices older than 14 days, delete indices older than 30 days curator --host my-elasticsearch -d 30 -c 14 • Disable bloom filter for indices older than 2 days, close indices older than 14 days, delete indices older than 30 days: curator --host my-elasticsearch -b 2 -c 14 -d 30
- 46. Curator 1 root@precise64:/home/vagrant# curator -c 7 -b 2 -d 10! 2 2014-04-21T17:57:19.419 INFO main:333 Job starting...! 3 2014-04-21T17:57:19.420 INFO _new_conn:180 Starting new HTTP connection (1): localhost! 4 2014-04-21T17:57:19.422 INFO log_request_success:49 GET http://localhost:9200/ [status:200 request:0.002s]! 5 2014-04-21T17:57:19.423 INFO main:359 Deleting indices older than 10 days...! 6 2014-04-21T17:57:19.430 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.007s]! 7 2014-04-21T17:57:19.433 INFO find_expired_indices:209 logstash-2014.04.21 is 10 days, 0:00:00 above the cutoff.! 8 2014-04-21T17:57:19.433 INFO index_loop:309 DELETE index operations completed.! 9 2014-04-21T17:57:19.433 INFO main:364 Closing indices older than 7 days...! 10 2014-04-21T17:57:19.434 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.001s]! 11 2014-04-21T17:57:19.435 INFO find_expired_indices:209 logstash-2014.04.21 is 7 days, 0:00:00 above the cutoff.! 12 2014-04-21T17:57:19.435 INFO index_loop:309 CLOSE index operations completed.! 13 2014-04-21T17:57:19.435 INFO main:369 Disabling bloom filter on indices older than 2 days...! 14 2014-04-21T17:57:19.437 INFO log_request_success:49 GET http://localhost:9200/logstash-*/_settings? expand_wildcards=closed [status:200 request:0.002s]! 15 2014-04-21T17:57:19.438 INFO find_expired_indices:209 logstash-2014.04.21 is 2 days, 0:00:00 above the cutoff.! 16 2014-04-21T17:57:19.438 INFO index_loop:309 DISABLE BLOOM FILTER FOR index operations completed.! 17 2014-04-21T17:57:19.438 INFO main:379 Done in 0:00:00.020348.!
- 52. Take Home • Centralized Logging saves time • Is fun with the ELK Stack • Gives you Graphs to Interpret • „can you check the errors from yesterday between 15.02 and 15.07“ get’s A LOT easier • Start here tomorrow: http://logstash.net/docs/ 1.4.0/tutorials/getting-started-with-logstash
- 53. Thank you for having me here! Slides : http://s.nrdy.ch/campus-logging
- 55. Images Used • Elk : https://www.flickr.com/photos/ucumari/353839518/ • Paper Stash : https://www.flickr.com/photos/shehan365/8394630603/ • Architecture : https://www.flickr.com/photos/dasrecht/6743411525/