"How about no grep and zabbix?". ELK based alerts and metrics.
5 likes10,089 views
This document provides an overview of the ELK (Elasticsearch, Logstash, Kibana) stack for collecting, analyzing, and visualizing log and metrics data. It describes the components of the ELK stack and how they work together, including how Logstash can be used to transform raw log data into structured JSON documents for indexing in Elasticsearch. The document also discusses how Kibana can be used to visualize and explore the data in Elasticsearch, and how the ELK stack can be used for advanced capabilities like custom metrics, alerts, and monitoring through tools like Elastalert and Kibana dashboards.
![Logstash transformation example
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800]
"GET /xampp/status.php HTTP/1.1" 200 3891
"http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-9-320.jpg)
![Logstash transformation example
{
"message" : "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php
HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"",
"@timestamp" : "2013-12-11T08:01:45.000Z",
"@version" : "1",
"host" : "cadenza",
"clientip" : "127.0.0.1",
"path" : "/xampp/status.php",
"response" : "200",
// ...
}](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-10-320.jpg)
![Elasticsearch query DSL
message:exception
log_level:(ERROR OR INFO)
stack_trace:”Out of memory”
_exists_:body
date:[now-1h TO now]
age:(>=10 AND <20)
wildcards (*, ?)
regex search
fuzzy search (Levenshtein)
proximity search
boosting weights](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-14-320.jpg)
![Elasticsearch query DSL
Even more power with JSON query DSL:
{
"query_string" : {
"fields" : [“message", “stack_trace"],
"query" : “exception AND fatal"
}
}](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-15-320.jpg)
![Kibana
Converts these:
{
"message" : "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php
HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"",
"@timestamp" : "2013-12-11T08:01:45.000Z",
"@version" : "1",
"host" : "cadenza",
"clientip" : "127.0.0.1",
"path" : "/xampp/status.php",
"response" : "200",
// ...
}](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-17-320.jpg)
![Log custom fields (JVM)
import net.logstash.logback.marker.Markers._
val fields = Map[String, Any](
"str_field" -> "str_value",
"num_field" -> 666
)
logger.info(appendEntries(fields), "log message");](https://image.slidesharecdn.com/howaboutnogrepandzabbixadapted-151229151814/85/How-about-no-grep-and-zabbix-ELK-based-alerts-and-metrics-31-320.jpg)
"How about no grep and zabbix?". ELK based alerts and metrics.
- 1. How about no grep and zabbix? or ELK based metrics and alerts. Vladimir Pavkin QIWI Conf #3
- 3. Plan ELK stack components ELK setups Basic usage (grep) Advanced usage (alerts, metrics)
- 4. Da heck is ELK? Elasticsearch Logstash Kibana
- 5. Logstash Inputs 1+ Filters 0+ Outputs 1+ • raw data • various formats • enriched events • standard formats
- 6. Logstash inputs file, syslog, log4j http, tcp, udp rabbitmq twitter, rss …
- 7. Logstash filters grok geoip drop, clone, mutate, split …
- 9. Logstash transformation example 127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
- 10. Logstash transformation example { "message" : "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"", "@timestamp" : "2013-12-11T08:01:45.000Z", "@version" : "1", "host" : "cadenza", "clientip" : "127.0.0.1", "path" : "/xampp/status.php", "response" : "200", // ... }
- 11. Elasticsearch Distributed data storage, optimized for text search. Is managed/queried through HTTP REST API. Based on open-source Apache Lucene project.
- 12. Elasticsearch vocab Document - data unit used for indexing and searching. Contains fields. (JSON) Field - a part of a Document, has a name and a value. Term - the unit of search. A word we look for in text
- 13. Elasticsearch inverted index Documents: 1 => How about no grep 2 => How about no zabbix Index: how => <1>, <2> about => <1>, <2> zabbix => <2> ...
- 14. Elasticsearch query DSL message:exception log_level:(ERROR OR INFO) stack_trace:”Out of memory” _exists_:body date:[now-1h TO now] age:(>=10 AND <20) wildcards (*, ?) regex search fuzzy search (Levenshtein) proximity search boosting weights
- 15. Elasticsearch query DSL Even more power with JSON query DSL: { "query_string" : { "fields" : [“message", “stack_trace"], "query" : “exception AND fatal" } }
- 16. Elasticsearch clustering Just easy as hell
- 17. Kibana Converts these: { "message" : "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"", "@timestamp" : "2013-12-11T08:01:45.000Z", "@version" : "1", "host" : "cadenza", "clientip" : "127.0.0.1", "path" : "/xampp/status.php", "response" : "200", // ... }
- 18. Kibana To this:
- 19. Kibana timestamp field Not strings/objects Events!
- 20. ELK Setups
- 21. Simple setup (startup mode) App1 App2 App3 logback logback logback VPN Single instance Logstash Elastic search TCP/UDP port http:9200 Kibana http:5601
- 24. 1. Search that whole cluster! Events from all your apps are in one place! Rocking useful and flexible timeline Powerful, text oriented search engine
- 25. 2. Default logback fields Quickly filter events by: Host name or IP Log level, logger name Search only in stack trace.
- 26. 3. Customize and save searches Save frequently used search Show only specific fields (e.g. message and host name)
- 27. Showtime
- 28. Advanced ELK. Metrics and alerts.
- 29. 1. Custom fields (search/filter) Payment event: { “user” : “9112223344”, “amount” : 500.00, “provider” : 464 }
- 30. Log custom fields (JVM) <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <destination>127.0.0.1:4560</destination> </appender> <root level="DEBUG"> <!--...--> <appender-ref ref="LOGSTASH" /> </root> libraryDependencies += "net.logstash.logback" % "logstash-logback-encoder" % "4.5.1"
- 31. Log custom fields (JVM) import net.logstash.logback.marker.Markers._ val fields = Map[String, Any]( "str_field" -> "str_value", "num_field" -> 666 ) logger.info(appendEntries(fields), "log message");
- 32. 2. Elastalert Configured with a set of “rules” Performs periodical queries to ES Triggers alerts for matched rules
- 33. 2. Elastalert rule types Frequency Spike Flatline Cardinality Blacklist, Whitelist Arbitrary query # rules/500_frequency.yaml es_host: localhost es_port: 9200 name: 500 code frequency alert type: frequency index: logstash-* num_events: 50 timeframe: minutes: 1 alert: - "email" email: - “[email protected]"
- 34. 2. Elastalert alert types Email Slack Jira Custom shell command
- 35. 2. Elastalert with custom fields Any alert you can imagine can be implemented on custom fields: Average payment drops below 300 RUB Processing response latency spike User A buys something on Herbalife
- 36. 3. Monitoring and Metrics Realtime Lots of aggregations Beautiful visualizations Super power with custom fields
- 39. Showtime
- 40. 3. Why so useful? Tech monitoring (UI and flexibility) Business metrics (approx.) Alerts (both tech and business) Instant market/system reaction
- 41. Thanks !