![7
Filebeat
filebeat:
prospectors:
paths:
/var/log/*.log
input_type: log
paths:
/var/log/apache2/*
input_type: log
output:
elasticsearch:
hosts: ["localhost:9200"]
Sample confiuration :
{
"@timestamp": "20160106T00:00:00Z",
"type": "log",
"message": "<line from file>"
}
Sample published event:](https://image.slidesharecdn.com/logaggregation-170107114801/85/Log-aggregation-and-analysis-9-320.jpg)
![8
Metricbeat
metricbeat.modules:
module: redis
metricsets:
info
hosts: ["127.0.0.1:6379"]
output.elasticsearch:
hosts: ["localhost:9200"]
Sample configuration:
{
"@timestamp": "20160106T00:00:00Z",
"type": "metricsets",
"redis": {
...
}
}
Sample published event:](https://image.slidesharecdn.com/logaggregation-170107114801/85/Log-aggregation-and-analysis-10-320.jpg)
![10
Logstash – contd.
GeoIP Lookup /
Data Mutation /
Structuring data
input {
beats {
port => 5044
}
}
filter{
if[type] == "nginxlogs" {
grok {
match => {"message" => "%{PATTERN}"}
}
geoip {
source => "remote_addr"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash%{+YYYY.MM.dd}"
}
}
Sample confiuration :](https://image.slidesharecdn.com/logaggregation-170107114801/85/Log-aggregation-and-analysis-12-320.jpg)
Log aggregation and analysis
- 1. Log Aggregation and Analysis Using Elastic Stack
- 3. 1 Logging - What ? ● Human readable and machine parseable format ● Record now, analyze later ● Various sources of events – Webservers – User activity on a website – Application logs – Node metrics – Other operational data – Mobile / IoT devices
- 4. 2 Logging - Why ? ● Record keeping ● Operational Insights – Monitor and optimize resource consumption / utilization – Early detection – find out before it goes wrong ● Incident forensics – Where did it go wrong ? – Consistency related bugs – Failing system is better than an incosistent system ● Answer Questions in (near)real-time – Functional metrics – How many users logged in within last hour?, Which location is most active right now?, What's the average response time for X page?
- 5. 3 Available tools Splunk Graylog Elastic stack License Paid Free Freemium Setup complexity Easy Medium Medium Hosting On premise / Hosted On premise / Hosted On-premise / Hosted Capabilities Ingestion / Storage / Analytics / Alerts Ingestion / Storage / Analytics / Alerts Ingestion / Storage / Analytics / Alerts Scalable ? Yes Yes Yes Architecture Monolith Monolith Divided into different components – Each can be used separately And many more - Grafana, Logsearch, MS Azure Log Analytics, Loggly
- 6. 4 Elastic Stack Explore and visualize your data. Search, Dashboards and many more. Lightweight data shippers Parse, Enrich & Transport Data Store, search, and analyze your data.
- 7. 5 Step 1 - Gathering logs Lightweight Data Shippers Beats is the platform for single-purpose data shippers. They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch. ● Using log appenders / handlers ● Beats
- 8. 6 Beats (by Dre Elastic) ● Filebeat – Reads from file – Non-intrusive ● Metricbeat – Collects metrics from systems and services – Modules available for Apache, nginx, Docker, Kafka, PostgreSQL and more ● Packetbeat – Lightweight network packet analyzer – Modules available for HTTP, DNS, AMQP and more ● Winlogbeat – Collects windows event logs ● Add your own – Dozens of community developed beats available – Extensible architecture – Easy to create on our own – Written in Go
- 11. 9 Step 2 - Processing logs using Logstash ● Ingest-process-output pipeline ● Ingest Data of All Shapes, Sizes, and Sources – Beats, log4j, redis, tcp/udp, HTTP ● Process – Transform unstructured data to structured data using grok filter – Filter out unnecessary data – Mutate data (calculate fields, add extra context, get geo co-ordinates from IP address, etc) ● Stash it away – Data stores (elasticsearch, files, mongoDB, redis), other services (email, pagerduty, redmine, irc, jira), brokers (kafka, rabbitMQ) and many more ● Scalable, Durable
- 12. 10 Logstash – contd. GeoIP Lookup / Data Mutation / Structuring data input { beats { port => 5044 } } filter{ if[type] == "nginxlogs" { grok { match => {"message" => "%{PATTERN}"} } geoip { source => "remote_addr" } } } output { elasticsearch { hosts => ["localhost:9200"] index => "logstash%{+YYYY.MM.dd}" } } Sample confiuration :
- 13. 11 Step 3 - Storing logs in Elasticsearch ● Distributed RESTful search and analytics engine (JSON/HTTP) ● Fast – get your answers instantly ● Scalable – Run on your laptop or hundreds of servers ● Resilient and Highly Available – Clustering, Failure detection ● Full text search, Aggregation, Geo filtering (within x mile radius), Suggestions (show more like this), Fuzzy search, Scripting
- 14. 12 Step 4 - Explore and Visualize using Kibana ● Works seamlessly with Elasticsearch ● Easy yet powerful search interface ● Supports histograms, line graphs, pie charts and many more ● Visualize geospatial data ● Extensible – Create your own visualization ● Create and share dashboards
- 15. 13 Elastic Stack in Action
- 16. 14 Demo 1 – Parsing nginx logs Nginx Access Logs Filebeat conf Filebeat Read file(s) push to Logstash Logstash Parse logs GeoIP lookup User agent parsing Push to Elasticsearch Elasticsearch Index and store Kibana Search and Analyze
- 17. 15 Demo 2 – Logs from a Django application Elasticsearch Index and store Kibana Search and Analyze Logstash Collect logs Push to Elasticsearch Django App Logstash handler
- 18. 16 Demo 3 – Capture and monitor node metrics Metricbeat conf Metricbeat Read metrics from nodes Elasticsearch Index and store Kibana Search and Analyze
- 19. 17 Q & A
- 20. 18 Thank You Ahmedabad Java Meetup Group https://www.meetup.com/Ahmedabad-Java-Meetup-Group/ @JavaMeetup AhmedabadJavaMeetup Dhaval Mehta [email protected] @mehtadhaval07