Log analysis using Logstash,ElasticSearch and Kibana
Download as PPTX, PDF44 likes21,365 views
This document provides an overview of Logstash, Elasticsearch, and Kibana for log analysis. It discusses how logging is used for troubleshooting, security, and monitoring. It then introduces Logstash as an open-source log collection and parsing tool. Elasticsearch is described as a search and analytics engine that indexes log data from Logstash. Kibana provides a web interface for visualizing and searching logs stored in Elasticsearch. The document concludes with discussing demo, installation, scaling, and deployment considerations for these log analysis tools.
Log analysis using Logstash,ElasticSearch and Kibana
- 1. Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar
- 2. • Logging • Pains of Log Management • Introducing Logstash • Elasticsearch • Kibana • Demo • Installing Logstash, Elasticsearch Kibana • Questions Agenda
- 3. • Why do we need Logging ? – Troubleshoot Issues – Security • Analyze logs to detect patterns • Detect Malware Activity - Intrusion Detection, Denial of Service • Unauthorized Resource Usage – Monitoring • Monitor Resource Usage • Developers and Logging – Logging Aids in Development ? – Forget about Production !!!!! Logging
- 4. • “Capture-it-all” Approach • What to Log? Everything • DevOps Movement • Logs are archived for years • Big Data • Application Usage Statistics Logging
- 5. • Searching the logs – Command line, cat, tail, sed, grep, awk – Regular Expressions • Multiple Servers behind the load balancer • Multi-Tier Architecture – Web Application – Service Layer – Correlation between various components in a System • Geographically distributed – Timestamps Log management
- 6. • Centralize all the Logs – Too much information to go through – Increasingly hard to correlate the contextual Data • Add Searching and Indexing Technology – grep – Custom logging frameworks , custom integration of logging, searching technologies • Monitor the Logs Log management
- 7. • Logstash to the Rescue –Integration Framework • Log Collection • Centralization • Parsing • Storage and Search Logstash
- 8. • JRuby – Run on Java Virtual Machine (JVM) – Simple Message Based Architecture – Single Agent that can be configured for multiple things – OPEN SOURCE • Four Components – Shipper – Broker and Indexer – Search and Storage – Web Interface Logstash
- 9. Architecture Image courtesy of Logstashbook
- 10. Architecture - Broker • Acts as Temp Buffer between Logstash Agents and the Central server – Enhance Performance by providing caching buffer for log events – Adds Resiliency • Incase the Indexing fails, the events are held in a queue instead of getting lost • AMQP,0MQ, Redis
- 11. • Indexing and Searching Tool – Built on Lucene • Search and Index data available Restfully as JSON over HTTP • Comes bundled with Logstash – embedded • Text indexing Search Engine – Searches on the Index rather than on the content • Creates Indexes of the incoming content – Uses Apache Lucene to create Indexes • ElasticSearch can have a schema – Fields on which Indexes are created ElasticSearch
- 12. • Indexes are stored in Lucene Instances called “Shards” • ElasticSearch can have multiple nodes • Two Types of Shards – Primary – Replica • Replicas of Primary Shards – Protect the data – Make Searches Faster ElasticSearch
- 13. • Wouldn’t it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service • Kibana provides a Simple but Powerful web Interface – Customizable Dashboards – Search the log events • Support Lucene Query Syntax – Creation of tables, graphs and sophisticated visualizations Kibana
- 14. Kibana
- 15. Kibana
- 16. Demo
- 17. • Send Alerts – Emails – Instant Messaging – Other Monitoring System • Collect and Deliver Metrics to metric engine Alerts / Monitoring Support
- 18. • Small VMs with limited memory • Outsourced managed servers • Java not installed • Alternatives – Syslog • Rsyslog • Syslogd • Syslog-NG – Logstash Forwarder (Lumber Jack) Shipping Logs with Logstash Agent
- 19. • Scale each component as needed • Can be built into using chef and puppet scripts Scaling / Deployment
Editor's Notes
- #4: DevOps -- the kind of guys who have both a developer and an operator hat making sure that custom developed applications are running smoothly