From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity NYC 2015)
6 likes14,317 views
The document presents a comprehensive overview of log analysis using Elasticsearch and tools like Logstash and Kibana, provided by Sematext. It includes details on their offerings such as training sessions and technical resources for implementation. Additionally, it highlights various configurations, commands, and performance considerations for managing log data effectively.
![Analysis
(Macintosh; Intel Mac OSX; en)
["Macintosh", "Intel", "Mac", "OSX", "en"]
["macintosh", "intel", "mac", "osx", "en"]
standard tokenizer
lowercase token filter](https://image.slidesharecdn.com/velocityny2015-151013200943-lva1-app6891/85/From-Zero-to-Production-Hero-Log-Analysis-with-Elasticsearch-from-Velocity-NYC-2015-9-320.jpg)
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity NYC 2015)
- 1. From zero to production hero: Log analysis with Elasticsearch Rafał Kuć Radu Gheorghe
- 3. Our Company → Sematext HQ: NYC + Globally Distributed Team Search & Big Data Consulting Production Support for Solr & Elasticsearch Training for Solr & Elasticsearch (online and onsite) Training in NYC next week! Oct 19 & 20
- 4. Our Company → Sematext
- 5. Agenda Kibana Elasticsearch essentials, tuning and scaling Logstash rsyslog Logstash + rsyslog Commands & Configs: https://github.com/sematext/velocity
- 8. Lucene Essentials GET 1,3,5 PUT 2,4 {"verb": "GET"} 1)GETdocument stored indexed
- 9. Analysis (Macintosh; Intel Mac OSX; en) ["Macintosh", "Intel", "Mac", "OSX", "en"] ["macintosh", "intel", "mac", "osx", "en"] standard tokenizer lowercase token filter
- 10. Field data GET 1,2 PUT 2,3
- 11. Field data GET 1,2 PUT 2,3
- 12. Field data GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT
- 13. Field data GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT expensive
- 14. Field data GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT expensive heap
- 15. Field data GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT expensive heap http://bio-img.s3.amazonaws.com/bds/formhdr-cvr-5-memory-killing-foods-v2.png
- 16. DocValues GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT at index time; on disk https://www.lorextechnology.com/images/products/HDD250GB/900x600/security-certified-HDD250GB-L1.png
- 17. DocValues GET 1,2 PUT 2,3 1) GET 2) GET,PUT 3) PUT no uninverting! at index time; on disk https://www.lorextechnology.com/images/products/HDD250GB/900x600/security-certified-HDD250GB-L1.png OS caches instead of heap
- 18. Logstash /var/log/apache.log GET /index.html grok { "verb": "GET", "path": "/index.html" } - w $numberOfWorkers workers => 2 filter output input Elasticsearch
- 19. rsyslog /var/log/apache.log GET /index.html mmnormalize { "verb": "GET", "path": "/index.html" } queue.workerThreads queue.dequeueBatchSize omelasticsearch imfile input module Elasticsearch main queue (RAM+Disk) queue.type queue.size ...
- 20. mmnormalize parse tree sys tem log d -ng => scales very well with # of rules (performance depends more on log length)
- 21. rsyslog + Redis via Kafka rsyslog Apache Kafka Logstash Elasticsearch file input mmnormalize omkafka + JSON template Kafka input + JSON codec Elasticsearch output
- 22. Free eBooks @ sematext.com We are hiring too http://sematext.com/about/jobs.html
- 24. Thank you! Rafał Kuć @kucrafal [email protected] Radu Gheorghe @radu0gheorghe [email protected] Sematext @sematext http://sematext.com