Elk with Openstack
Download as PPTX, PDF2 likes1,748 views
Log analysis challenges include searching logs across multiple services and servers. The ELK stack provides a solution with Logstash to centralize log collection, Elasticsearch for storage and search, and Kibana for visualization. Logstash uses input, filter, and output plugins to collect, parse, and forward logs. Example configurations show using stdin and filters to parse OpenStack logs before outputting to Elasticsearch and Kibana for analysis and dashboards.
![Example – 2 (Basic filter on syslog with stdin input and rubydebug output)
input {
stdin {}
}
filter{
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname}
%{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
output{
stdout { codec => rubydebug }
}](https://image.slidesharecdn.com/elk-slideshare-160629132420/85/Elk-with-Openstack-13-320.jpg)
![Example – 3 (Openstack filters with file input and multiple output)
input {
file {
path => ['/var/log/nova/nova-api-metadata.log']
tags => ['nova', 'oslofmt', 'novametaapi']
type => "nova"
}
…
}
filter{
…
else if "novaapi" in [tags] {
if [module] == "nova.osapi_compute.wsgi.server" {
mutate {
gsub => ['logmessage',""",""]
}
grok {
match => { "logmessage" => "[req-%{NOTSPACE:requestid} %{NOTSPACE:user_id} %{NOTSPACE:tenant}] %{NOTSPACE:requesterip} %{NOTSPACE:method}
%{NOTSPACE:url_path} %{NOTSPACE:http_ver} status: %{NUMBER:response} len: %{NUMBER:bytes} time: %{NUMBER:seconds}" }
add_field => ["api", "nova"]
add_tag => ["apimetrics"]
}
…
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
Ref: https://github.com/bingoarunprasath/logstash-openstack-filters](https://image.slidesharecdn.com/elk-slideshare-160629132420/85/Elk-with-Openstack-14-320.jpg)
Elk with Openstack
- 1. ELK with Openstack Arun prasath S June 16, 2016
- 2. Challenges in log analysis • Multiple services • Multiple servers behind load balancers • Searching the logs (cat, tail, sed, grep, awk) • Finding logs in particular time in multiple servers • Finding fields (Instance ID, name, IP address) in multiple servers and correlating them • Log analysis , summary, visualization
- 3. ELK user operation demo • Performing a normal search • Filtering based on time, fields • Viewing document data • Viewing field data statistics • Visualize data • Dashboards
- 5. Broker • Temporary buffer between logstash agents and central server • Enhance performance by providing caching buffer for log events • Adds resiliency • Incase the indexing fails, the events are held in queue instead of getting lost
- 6. Logstash • Runs on JVM • Multiple Input / Multiple output • Centralize logs • Collect • Parse • Store / Forward • 3 components – Input, Filter and Output
- 7. Logstash conf - /etc/logstash/conf.d input { } filter{ } output{ }
- 8. Logstash - Input • Input plugin enables a specific source of events to be read by Logstash. • Some examples of input • Beats • File • Stdin • Eventlog • More here
- 9. Logstash - Filter • A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event. • Some examples are • Csv • Date • Grok • Json • More here
- 10. Logstash - Output • An output plugin sends event data to a particular destination. • Some examples are • Csv • redis • elasticsearch • File • Jira, Nagios, pagerduty • stdout • More here
- 11. Logstash - codec • A codec plugin changes the data representation of an event • Some examples are • Collectd - Reads events from the collectd binary protocol using UDP • Graphite - Reads graphite formatted lines • Json - Reads JSON formatted content, creating one event per element in a JSON array • Plain - Reads plaintext with no delimiting between events • rubydebug - Applies the Ruby Awesome Print library to Logstash events • More here
- 12. Example - 1 (stdin input / rubydebug output) input { stdin {} } filter{ } output{ stdout { codec => rubydebug } }
- 13. Example – 2 (Basic filter on syslog with stdin input and rubydebug output) input { stdin {} } filter{ grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } output{ stdout { codec => rubydebug } }
- 14. Example – 3 (Openstack filters with file input and multiple output) input { file { path => ['/var/log/nova/nova-api-metadata.log'] tags => ['nova', 'oslofmt', 'novametaapi'] type => "nova" } … } filter{ … else if "novaapi" in [tags] { if [module] == "nova.osapi_compute.wsgi.server" { mutate { gsub => ['logmessage',""",""] } grok { match => { "logmessage" => "[req-%{NOTSPACE:requestid} %{NOTSPACE:user_id} %{NOTSPACE:tenant}] %{NOTSPACE:requesterip} %{NOTSPACE:method} %{NOTSPACE:url_path} %{NOTSPACE:http_ver} status: %{NUMBER:response} len: %{NUMBER:bytes} time: %{NUMBER:seconds}" } add_field => ["api", "nova"] add_tag => ["apimetrics"] } … } output { elasticsearch { hosts => ["localhost:9200"] } stdout { codec => rubydebug } } Ref: https://github.com/bingoarunprasath/logstash-openstack-filters
- 15. Elasticsearch • Searching and storing of logs • Built on Apache lucene (https://lucene.apache.org/core/) • Massively distributed • High availability • Developer friendly , RESTful API
- 16. Kibana • Dashboard • Provides various options to search data • Creates bar charts, pie charts and various other data visualizations. • Can create custom dashboard and add the saved visualizations. • Simple data export
- 17. Installation notes • Install Java • Install elasticsearch and Kibana • Install nginx for reverse proxy and basic AUTH • Install logstash, generate SSH certificates • Configure one output to elasticsearch • Load Kibana dashboard • Setup filebeat / logstash in agent machines and output to logstash • Ansible role - https://galaxy.ansible.com/bingoarunprasath/elk/ • Filters - https://github.com/bingoarunprasath/logstash-openstack- filters