Using Logstash, elasticsearch & kibana

Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23

So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg

Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif

http://brotality.com/wp-content/uploads/2012/12/madness.jpg

I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time

Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
● JVM
● logstash.jar

Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elasticsearch
Mongodb
Redis
File
...
Plugins
Not here yet? JRuby to the rescue

ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● Full text search
● YAML config file / JSON format!!
● Document oriented JSON
Getting started: Logstash JAR includes it / download and set cluster.name
This is where it will be worth to spend some time tuning

Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / graph and more

Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana

After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels

Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude => ["*.gz"]
start_position => "beginning"
format => "json_event"
}
}
output {
tcp {
host => "0.0.0.0"
mode => "server"
port => 1666
}
}
Logstash-server.conf
input {
tcp {
type => "httpd"
host => "127.0.0.1"
mode => "client"
port => "1666"
}
tcp {
type => "app"
host => "127.0.0.1"
mode => "client"
port => "2666"
}
}
output {
elasticsearch {
cluster => "logstash"
}
}

Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@message", "(.+)" ]
drop => true
add_tag => [ "no_blank_lines" ]
}
# make the multilines be treated like a single line
multiline {
type => "tomcat"
pattern => "^dddd"
negate => true
what => "previous"
}

Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
match => [ "@message", ".+Exception: .+" ]
drop => false
add_tag => [ "java_exception" ]
}
# get the log level, operation id, module and timestamp as separated fields
grok {
type => "tomcat"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."
add_tag => [ "groked" ]
}
# fix the timestamp
date {
type => "tomcat"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]
add_tag => [ "timestamp_fix" ]
}
}

Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs using brokers or other methods beyond
files and syslog
● Logging in JSON format is a nice to have
● Share the log visualization
● Use the brokers Luke
● If you develop internalize this
http://www.masterzen.fr/2013/01/13/the-10-
commandments-of-logging/

Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns
● http://grokdebug.herokuapp.com/
● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)
●
http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
● http://kibana.org/
●
https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/
●
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/

Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is better to stay away
Have a nice day
All the images, videos and stuff are property of their respective owners, look at the cat
and don't sue me
http://stuffpoint.com/cats/image/41633/cute-cat-picture/

Using Logstash, elasticsearch & kibana

More Related Content

What's hot (20)

Similar to Using Logstash, elasticsearch & kibana (20)

More from Alejandro E Brito Monedero (14)

Recently uploaded (20)

Using Logstash, elasticsearch & kibana