SlideShare a Scribd company logo

OSMC 2014: Using elasticsearch, logstash & kibana in system administration | Alexander Reelsen

NETWAYS
NETWAYS

The document provides an introduction to the ELK Stack, consisting of Elasticsearch, Logstash, and Kibana, which are used for data management, analysis, and visualization. It highlights the importance of data-driven decision making, addresses challenges related to big data, and presents case studies demonstrating the stack's application in real-time analytics. Key components and features of the ELK Stack are detailed, including Logstash's data collection capabilities, Elasticsearch's search functionalities, and Kibana's visualization tools.

Software
1 of 68
Downloaded 94 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
Introduction into the ELK stack Alexander Reelsen alexander.reelsen@elasticsearch.com @spinscale Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Agenda • Introduction • The ELK stack • Samples, samples, samples • Summary Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
About Elasticsearch • Founded 2012 in Amsterdam • Funded by Benchmark, Index Ventures and NEA Ventures • Distributed company Offices in Los Altos, Amsterdam, London, Berlin, Phoenix • Offering support subscriptions & trainings • We’re hiring Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
About me • Joined early 2013 • Interested in all things scale, search & concurrency • Elasticsearch developer, doing trainings, support, blog posts, conferences, presentations Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Introduction Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
How do you decide? • What is the core asset of your company? Ideas, patents, employees, customers, warehouse, software, ... • Where to invest/develop next? • Data driven decisions Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
How do you decide? • What is the core asset of your company? Ideas, patents, employees, customers, warehouse, software, ... • Where to invest/develop next? • Data driven decisions logfiles for scaling up/down warehouse withdrawal triggers orders history for fraud detection assembly line, throughput improvement ... data explosion Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
More data is Big Data • More and more data Recommendations, page views, IoT, social media • Better decisions == more data? but ... Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The Big Data promise Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The Big Data promise problem Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The Big Data promise problem reaction time Time between storing and analysing an event Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The Big Data promise problem reaction time enrichment Increase event value by enriching Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The Big Data promise problem reaction time enrichment insights optimize for query, not for storage Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Requirements • Clean data to work on • Fast analysis chain near real-time • Easy to use user interface Everyone is able to create own reports Meet the ELK stack Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The ELK stack Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
The ELK stack Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash • Managing events and logs • Collect data • Parse data • Enrich data • Store data • Open Source: Apache License 2.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash architecture Input datastore stream log files files monitoring queues network Filter Output Logstash parse, enrich, tag, drop Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited datastore files email pager monitoring chat API queues
Logstash architecture Input datastore stream log files files monitoring queues network Filter Output Logstash parse, enrich, tag, drop Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited datastore files email pager monitoring chat API queues ip: 141.1.1.1 ip: 141.1.1.1 city: Zurich country: CH
Elasticsearch Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch • Schema-free, REST & JSON based distributed search engine • Open Source: Apache License 2.0 • Easy to understand, yet very powerful query language Full text search (phrase, fuzzy) Numeric search (support ranges, dates, ipv4 addresses) Highlighting Aggregations Suggestions Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana • Execute queries on your data & visualize results • Add/remove widgets • Share/Save/Load dashboards • Open Source: Apache License 2.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Samples, samples, samples Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Samples • Guardian case study • Web server logs • meetup.com RSVP stream • Wikipedia update stream • sysdig output Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Case Study: The Guardian • Ophan: In-house analytics software • Empower the organization Give the entire organization real-time insight into audience engagement Democratize analytics access for more than 500 users Encourage a culture of exploration and innovation for all employees • Leverage real-time analytics Easily query 360 million documents See traffic for all content as it happens Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Example: Web server log files Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Example: Web server log files input { stdin {} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => “clientip" } useragent { source => "agent" target => "useragent" } } output { elasticsearch { protocol => "http" host => "localhost" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Example: Web server log files input { stdin {} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => “clientip" } useragent { cat access.log | logstash agent -f logstash-logs.conf source => "agent" target => "useragent" } } output { elasticsearch { protocol => "http" host => "localhost" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Example: Web server log files { "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama-2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"", "geoip" => { "ip" => "83.149.9.216", "country_code2" => "RU", "country_code3" => "RUS", "country_name" => "Russian Federation", "continent_code" => "EU", "region_name" => "48", "city_name" => "Moscow", "latitude" => 55.75219999999999, "longitude" => 37.6156, "timezone" => "Europe/Moscow", "real_region_name" => "Moscow City", "location" => [ [0] 37.6156, [1] 55.75219999999999 ] }, "useragent" => { "name" => "Chrome", "os" => "Mac OS X 10.9.1", "os_name" => "Mac OS X", "os_major" => "10", "os_minor" => "9", "device" => "Other", "major" => "32", "minor" => "0", "patch" => "1700" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Example: Web server log files "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET / presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama- 2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36”", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/ kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama- 2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36"" Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited grok
Example: Web server log files "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET / presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama- 2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36”", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/ kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama- 2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36"" Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited grok date
Example: Web server log files "geoip" => { "ip" => "83.149.9.216", "country_code2" => "RU", "country_code3" => "RUS", "country_name" => "Russian Federation", "continent_code" => "EU", "region_name" => "48", "city_name" => "Moscow", "latitude" => 55.75219999999999, "longitude" => 37.6156, "timezone" => "Europe/Moscow", "real_region_name" => "Moscow City", "location" => [ [0] 37.6156, [1] 55.75219999999999 ] }, "useragent" => { "name" => "Chrome", "os" => "Mac OS X 10.9.1", "os_name" => "Mac OS X", "os_major" => "10", "os_minor" => "9", "device" => "Other", "major" => "32", "minor" => "0", "patch" => "1700" } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited geoip useragent
meetup.com RSVP stream • All RSVPs are written out to a HTTP stream • Each line is a JSON document • Available at http://stream.meetup.com/2/rsvps Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
meetup.com RSVP stream Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
meetup.com RSVP stream { response: "yes", member: { member_name: "Charlie “, member_id: 176530582 }, visibility: "public", event: { time: 1413270000000, event_url: "http://www.meetup.com/2EuroBootCamp/events/212054422/", event_id: “qsvrtkysnbsb", event_name: "Tuesday Morning Boot Camp" }, guests: 0, mtime: 1412774717000, rsvp_id: 1477279032, group: { group_name: "2 Euro Boot Camp!!", group_city: "Barcelona", group_lat: 41.4, group_lon: 2.17, group_urlname: "2EuroBootCamp", group_id: 17456462, group_country: "es", group_topics: [ { urlkey: "fitness", topic_name: "Fitness" } ] }, venue: { lon: 1.58728, venue_name: "Paque de la Espana Industrial", venue_id: 22845382, lat: 41.462646 } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
meetup.com RSVP stream # curl -s http://stream.meetup.com/2/rsvps | logstash agent -f logstash-meetup.conf input { stdin { codec => json_lines type => 'meetup' } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
meetup.com RSVP stream filter { if [venue][lat] and [venue][lon] { mutate { add_field => [ "[venue][lonlat]", "%{[venue][lon]}", "tmplat", "%{[venue][lat]}" ] } mutate { merge => [ "[venue][lonlat]", "tmplat" ] } mutate { convert => [ "[venue][lonlat]", "float" ] remove => [ "tmplat" ] } } metrics { meter => "meetup.country.%{[group][group_country]}" meter => "meetup.country.total" add_tag => "metric" flush_interval => 60 } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
meetup.com RSVP stream output { if "metric" in [tags] { stdout { codec => rubydebug } elasticsearch { host => 'localhost' index => 'metrics' protocol => 'http' } } if [type] == "meetup" { elasticsearch { host => 'localhost' index => 'meetups' protocol => 'http' } } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
wikipedia edits • wikipedia has a changes stream • constantly posted in an IRC channel Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
wikipedia edits input { irc { type => 'wikipedia' host => 'irc.wikimedia.org' nick => 'logstash-wikipedia' channels => ['#de.wikipedia'] } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
wikipedia edits filter { # remove some weird encoding stuff from IRC mutate { gsub => [ "message", "u000302", "", "message", "u000303", "", "message", "u000307", "", "message", "u000310", "", "message", "u000314", "", "message", "u00034", "", "message", "u00035", "", "message", "u0003", "" ] } # extract page and user grok { match => [ "message", "[[%{GREEDYDATA:page}]]%{GREEDYDATA} * %{GREEDYDATA:user} * %{GREEDYDATA}" ] } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
wikipedia edits output { stdout { codec => line { format => 'Page: %{page}' } } elasticsearch { host => 'localhost' index => 'wikipedia-edits' protocol => 'http' } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
wikipedia edits » logstash -f logstash-wikipedia.conf Page: Yamaha Aerox Page: Neues Beginnen - Blätter internationaler Sozialisten Page: Portal Diskussion:Fußball Page: Saputo Page: Portal:Phantastik/Mitarbeiten Page: Gesetz über den Einsatz der Informations- und Kommunikationstechnik in der öffentlichen Verwaltung Page: Spvg Plettenberg Page: Pflanzen gegen Zombies: Garden Warfare Page: Wasserstandsanzeiger Bremerhaven Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
kippo SSH honeypot Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig • sysdig is a system call tracer (tcpdump for syscalls) • powerful query language • very useful for system tracing (intrusions, performance tracing, weird behaviour) • See http://www.sysdig.org/ Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig • Easy to find things # sysdig -r dumpfile.scap "evt.type = open and evt.arg.name contains /usr/sbin" 2122 13:54:01.755117599 0 bash (1633) < open fd=3(<f>/usr/sbin/ hacked) name=/usr/sbin/hacked flags=262(O_TRUNC|O_CREAT|O_WRONLY) mode=0 • Now do this for all machines... Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig input { stdin { } } filter { grok { pattern => "^%{NUMBER:num:int} %{NUMBER:time:float} %{INT:cpu:int} % {NOTSPACE:procname} %{NOTSPACE:tid} (?<direction>[<>]) %{WORD:event} % {DATA:args}$" } date { match => [ "time", "UNIX" ] } if [args] { kv { source => "args" remove_field => "args" } } } output { elasticsearch { protocol => http index => "sysdig-%{+YYYY.MM.dd}" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Summary Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Summary • Do not create data silos. Free your data! • Make sure data is easy to query, not to store • Visualize • Find your use-case: Business, system administration, your app... it’s versatile! Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Soon... • Kibana 4... is going to be huge • Elasticsearch 1.4.0 has been released • Logstash going towards 1.5.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Demo! Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Getting up & running is easy • Download Elasticsearch, logstash & Kibana archives # elasticsearch-1.4.0/bin/elasticsearch # kibana-4.0.0-BETA2/bin/kibana # logstash-1.4.2/bin/logstash agent -f logstash.conf # open localhost:5601 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Thanks for listening! Q & A P.S. We’re hiring http://elasticsearch.com/about/jobs P.P.S. We’re helping http://elasticsearch.com/support http://elasticsearch.com/training Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Alexander Reelsen @spinscale alexander.reelsen@elasticsearch.com

More Related Content

Similar to OSMC 2014: Using elasticsearch, logstash & kibana in system administration | Alexander Reelsen (20)

PDF
DataHero / Eventbrite - API Best Practices
Jeff Zabel
 
PDF
Building the Eventbrite API Ecosystem
Mitch Colleran
 
KEY
The Open Web
Lachlan Hardy
 
PDF
James Higginbotham - API Design
John Zozzaro
 
PDF
Colin Carter - LSPs and APIs
sconul
 
PDF
Webcast: Pragmatic REST: The Next Generation
Apigee | Google Cloud
 
PDF
The Psychology of Security Automation
Jason Chan
 
PPTX
Building A Business-Facing Mobile Developer Community
ProgrammableWeb
 
PDF
USG Rock Eagle 2017 - PWP at 1000 Days
Eric Sembrat
 
PDF
How to use Salesforce Workbench like a Pro
Atul Gupta(8X)
 
PPTX
Building Social Tools
Anand Hemmige
 
PPTX
Why contribute to open source projects
Kranti Parisa
 
PDF
20190523 archiver fim
Archiver
 
PPTX
GALILEO virtual library and OpenAthens partnership
OpenAthens
 
PDF
Why we need oa infrastructure - STM Association Beyond Open Access Seminar
National Information Standards Organization (NISO)
 
PPTX
Biwa summit 2015 oaa oracle data miner hands on lab
Charlie Berger
 
PPTX
The Analytics Continuum
Rob Marano
 
KEY
Web API Basics
LearnNowOnline
 
PPTX
Global Azure 2022 - Architecting Modern Serverless APIs with Azure Functions ...
Callon Campbell
 
PPTX
Guide to open source
Javier Perez
 
DataHero / Eventbrite - API Best Practices
Jeff Zabel
 
Building the Eventbrite API Ecosystem
Mitch Colleran
 
The Open Web
Lachlan Hardy
 
James Higginbotham - API Design
John Zozzaro
 
Colin Carter - LSPs and APIs
sconul
 
Webcast: Pragmatic REST: The Next Generation
Apigee | Google Cloud
 
The Psychology of Security Automation
Jason Chan
 
Building A Business-Facing Mobile Developer Community
ProgrammableWeb
 
USG Rock Eagle 2017 - PWP at 1000 Days
Eric Sembrat
 
How to use Salesforce Workbench like a Pro
Atul Gupta(8X)
 
Building Social Tools
Anand Hemmige
 
Why contribute to open source projects
Kranti Parisa
 
20190523 archiver fim
Archiver
 
GALILEO virtual library and OpenAthens partnership
OpenAthens
 
Why we need oa infrastructure - STM Association Beyond Open Access Seminar
National Information Standards Organization (NISO)
 
Biwa summit 2015 oaa oracle data miner hands on lab
Charlie Berger
 
The Analytics Continuum
Rob Marano
 
Web API Basics
LearnNowOnline
 
Global Azure 2022 - Architecting Modern Serverless APIs with Azure Functions ...
Callon Campbell
 
Guide to open source
Javier Perez
 

Recently uploaded (20)

PPTX
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
PDF
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
PDF
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PDF
SAP Firmaya İade ABAB Kodları - ABAB ile yazılmıl hazır kod örneği
Salih Küçük
 
PDF
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 
PPTX
In From the Cold: Open Source as Part of Mainstream Software Asset Management
Shane Coughlan
 
PDF
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
PDF
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
PDF
Build It, Buy It, or Already Got It? Make Smarter Martech Decisions
bbedford2
 
PPTX
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
PPTX
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PDF
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PPTX
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
PPTX
Home Care Tools: Benefits, features and more
Third Rock Techkno
 
PDF
TheFutureIsDynamic-BoxLang witch Luis Majano.pdf
Ortus Solutions, Corp
 
PDF
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
SAP Firmaya İade ABAB Kodları - ABAB ile yazılmıl hazır kod örneği
Salih Küçük
 
Generic or Specific? Making sensible software design decisions
Bert Jan Schrijver
 
In From the Cold: Open Source as Part of Mainstream Software Asset Management
Shane Coughlan
 
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
Build It, Buy It, or Already Got It? Make Smarter Martech Decisions
bbedford2
 
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
Home Care Tools: Benefits, features and more
Third Rock Techkno
 
TheFutureIsDynamic-BoxLang witch Luis Majano.pdf
Ortus Solutions, Corp
 
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
Ad

OSMC 2014: Using elasticsearch, logstash & kibana in system administration | Alexander Reelsen

  • 1. Introduction into the ELK stack Alexander Reelsen [email protected] @spinscale Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 2. Agenda • Introduction • The ELK stack • Samples, samples, samples • Summary Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 3. About Elasticsearch • Founded 2012 in Amsterdam • Funded by Benchmark, Index Ventures and NEA Ventures • Distributed company Offices in Los Altos, Amsterdam, London, Berlin, Phoenix • Offering support subscriptions & trainings • We’re hiring Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 4. About me • Joined early 2013 • Interested in all things scale, search & concurrency • Elasticsearch developer, doing trainings, support, blog posts, conferences, presentations Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 5. Introduction Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 6. How do you decide? • What is the core asset of your company? Ideas, patents, employees, customers, warehouse, software, ... • Where to invest/develop next? • Data driven decisions Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 7. How do you decide? • What is the core asset of your company? Ideas, patents, employees, customers, warehouse, software, ... • Where to invest/develop next? • Data driven decisions logfiles for scaling up/down warehouse withdrawal triggers orders history for fraud detection assembly line, throughput improvement ... data explosion Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 8. More data is Big Data • More and more data Recommendations, page views, IoT, social media • Better decisions == more data? but ... Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 9. The Big Data promise Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 10. The Big Data promise problem Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 11. The Big Data promise problem reaction time Time between storing and analysing an event Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 12. The Big Data promise problem reaction time enrichment Increase event value by enriching Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 13. The Big Data promise problem reaction time enrichment insights optimize for query, not for storage Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 14. Requirements • Clean data to work on • Fast analysis chain near real-time • Easy to use user interface Everyone is able to create own reports Meet the ELK stack Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 15. The ELK stack Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 16. The ELK stack Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 17. Logstash Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 18. Logstash • Managing events and logs • Collect data • Parse data • Enrich data • Store data • Open Source: Apache License 2.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 19. Logstash architecture Input datastore stream log files files monitoring queues network Filter Output Logstash parse, enrich, tag, drop Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited datastore files email pager monitoring chat API queues
  • 20. Logstash architecture Input datastore stream log files files monitoring queues network Filter Output Logstash parse, enrich, tag, drop Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited datastore files email pager monitoring chat API queues ip: 141.1.1.1 ip: 141.1.1.1 city: Zurich country: CH
  • 21. Elasticsearch Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 22. Elasticsearch • Schema-free, REST & JSON based distributed search engine • Open Source: Apache License 2.0 • Easy to understand, yet very powerful query language Full text search (phrase, fuzzy) Numeric search (support ranges, dates, ipv4 addresses) Highlighting Aggregations Suggestions Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 23. Logstash Logstash Store/Search Data Visualize Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 24. Kibana • Execute queries on your data & visualize results • Add/remove widgets • Share/Save/Load dashboards • Open Source: Apache License 2.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 25. Kibana Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 26. Samples, samples, samples Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 27. Samples • Guardian case study • Web server logs • meetup.com RSVP stream • Wikipedia update stream • sysdig output Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 28. Case Study: The Guardian • Ophan: In-house analytics software • Empower the organization Give the entire organization real-time insight into audience engagement Democratize analytics access for more than 500 users Encourage a culture of exploration and innovation for all employees • Leverage real-time analytics Easily query 360 million documents See traffic for all content as it happens Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 29. Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 30. Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 31. Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 32. Case Study: The Guardian Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 33. Example: Web server log files Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 34. Example: Web server log files input { stdin {} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => “clientip" } useragent { source => "agent" target => "useragent" } } output { elasticsearch { protocol => "http" host => "localhost" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 35. Example: Web server log files input { stdin {} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => “clientip" } useragent { cat access.log | logstash agent -f logstash-logs.conf source => "agent" target => "useragent" } } output { elasticsearch { protocol => "http" host => "localhost" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 36. Example: Web server log files { "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama-2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"", "geoip" => { "ip" => "83.149.9.216", "country_code2" => "RU", "country_code3" => "RUS", "country_name" => "Russian Federation", "continent_code" => "EU", "region_name" => "48", "city_name" => "Moscow", "latitude" => 55.75219999999999, "longitude" => 37.6156, "timezone" => "Europe/Moscow", "real_region_name" => "Moscow City", "location" => [ [0] 37.6156, [1] 55.75219999999999 ] }, "useragent" => { "name" => "Chrome", "os" => "Mac OS X 10.9.1", "os_name" => "Mac OS X", "os_major" => "10", "os_minor" => "9", "device" => "Other", "major" => "32", "minor" => "0", "patch" => "1700" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 37. Example: Web server log files "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET / presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama- 2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36”", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/ kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama- 2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36"" Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited grok
  • 38. Example: Web server log files "message" => "83.149.9.216 - - [28/May/2014:16:13:42 -0500] "GET / presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama- 2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36”", "@version" => "1", "@timestamp" => "2014-05-28T21:13:42.000Z", "host" => "kryptic.local", "clientip" => "83.149.9.216", "ident" => "-", "auth" => "-", "timestamp" => "28/May/2014:16:13:42 -0500", "verb" => "GET", "request" => "/presentations/logstash-monitorama-2013/images/ kibana-search.png", "httpversion" => "1.1", "response" => "200", "bytes" => "203023", "referrer" => ""http://semicomplete.com/presentations/logstash-monitorama- 2013/"", "agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/ 537.36"" Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited grok date
  • 39. Example: Web server log files "geoip" => { "ip" => "83.149.9.216", "country_code2" => "RU", "country_code3" => "RUS", "country_name" => "Russian Federation", "continent_code" => "EU", "region_name" => "48", "city_name" => "Moscow", "latitude" => 55.75219999999999, "longitude" => 37.6156, "timezone" => "Europe/Moscow", "real_region_name" => "Moscow City", "location" => [ [0] 37.6156, [1] 55.75219999999999 ] }, "useragent" => { "name" => "Chrome", "os" => "Mac OS X 10.9.1", "os_name" => "Mac OS X", "os_major" => "10", "os_minor" => "9", "device" => "Other", "major" => "32", "minor" => "0", "patch" => "1700" } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited geoip useragent
  • 40. meetup.com RSVP stream • All RSVPs are written out to a HTTP stream • Each line is a JSON document • Available at http://stream.meetup.com/2/rsvps Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 41. meetup.com RSVP stream Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 42. meetup.com RSVP stream { response: "yes", member: { member_name: "Charlie “, member_id: 176530582 }, visibility: "public", event: { time: 1413270000000, event_url: "http://www.meetup.com/2EuroBootCamp/events/212054422/", event_id: “qsvrtkysnbsb", event_name: "Tuesday Morning Boot Camp" }, guests: 0, mtime: 1412774717000, rsvp_id: 1477279032, group: { group_name: "2 Euro Boot Camp!!", group_city: "Barcelona", group_lat: 41.4, group_lon: 2.17, group_urlname: "2EuroBootCamp", group_id: 17456462, group_country: "es", group_topics: [ { urlkey: "fitness", topic_name: "Fitness" } ] }, venue: { lon: 1.58728, venue_name: "Paque de la Espana Industrial", venue_id: 22845382, lat: 41.462646 } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 43. meetup.com RSVP stream # curl -s http://stream.meetup.com/2/rsvps | logstash agent -f logstash-meetup.conf input { stdin { codec => json_lines type => 'meetup' } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 44. meetup.com RSVP stream filter { if [venue][lat] and [venue][lon] { mutate { add_field => [ "[venue][lonlat]", "%{[venue][lon]}", "tmplat", "%{[venue][lat]}" ] } mutate { merge => [ "[venue][lonlat]", "tmplat" ] } mutate { convert => [ "[venue][lonlat]", "float" ] remove => [ "tmplat" ] } } metrics { meter => "meetup.country.%{[group][group_country]}" meter => "meetup.country.total" add_tag => "metric" flush_interval => 60 } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 45. meetup.com RSVP stream output { if "metric" in [tags] { stdout { codec => rubydebug } elasticsearch { host => 'localhost' index => 'metrics' protocol => 'http' } } if [type] == "meetup" { elasticsearch { host => 'localhost' index => 'meetups' protocol => 'http' } } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 46. wikipedia edits • wikipedia has a changes stream • constantly posted in an IRC channel Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 47. wikipedia edits input { irc { type => 'wikipedia' host => 'irc.wikimedia.org' nick => 'logstash-wikipedia' channels => ['#de.wikipedia'] } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 48. wikipedia edits filter { # remove some weird encoding stuff from IRC mutate { gsub => [ "message", "u000302", "", "message", "u000303", "", "message", "u000307", "", "message", "u000310", "", "message", "u000314", "", "message", "u00034", "", "message", "u00035", "", "message", "u0003", "" ] } # extract page and user grok { match => [ "message", "[[%{GREEDYDATA:page}]]%{GREEDYDATA} * %{GREEDYDATA:user} * %{GREEDYDATA}" ] } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 49. wikipedia edits output { stdout { codec => line { format => 'Page: %{page}' } } elasticsearch { host => 'localhost' index => 'wikipedia-edits' protocol => 'http' } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 50. wikipedia edits » logstash -f logstash-wikipedia.conf Page: Yamaha Aerox Page: Neues Beginnen - Blätter internationaler Sozialisten Page: Portal Diskussion:Fußball Page: Saputo Page: Portal:Phantastik/Mitarbeiten Page: Gesetz über den Einsatz der Informations- und Kommunikationstechnik in der öffentlichen Verwaltung Page: Spvg Plettenberg Page: Pflanzen gegen Zombies: Garden Warfare Page: Wasserstandsanzeiger Bremerhaven Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 51. kippo SSH honeypot Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 52. sysdig • sysdig is a system call tracer (tcpdump for syscalls) • powerful query language • very useful for system tracing (intrusions, performance tracing, weird behaviour) • See http://www.sysdig.org/ Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 53. sysdig • Easy to find things # sysdig -r dumpfile.scap "evt.type = open and evt.arg.name contains /usr/sbin" 2122 13:54:01.755117599 0 bash (1633) < open fd=3(<f>/usr/sbin/ hacked) name=/usr/sbin/hacked flags=262(O_TRUNC|O_CREAT|O_WRONLY) mode=0 • Now do this for all machines... Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 54. sysdig input { stdin { } } filter { grok { pattern => "^%{NUMBER:num:int} %{NUMBER:time:float} %{INT:cpu:int} % {NOTSPACE:procname} %{NOTSPACE:tid} (?<direction>[<>]) %{WORD:event} % {DATA:args}$" } date { match => [ "time", "UNIX" ] } if [args] { kv { source => "args" remove_field => "args" } } } output { elasticsearch { protocol => http index => "sysdig-%{+YYYY.MM.dd}" } } Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 55. sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 56. sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 57. sysdig Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 58. Summary Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 59. Summary • Do not create data silos. Free your data! • Make sure data is easy to query, not to store • Visualize • Find your use-case: Business, system administration, your app... it’s versatile! Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 60. Soon... • Kibana 4... is going to be huge • Elasticsearch 1.4.0 has been released • Logstash going towards 1.5.0 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 61. Demo! Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 62. Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 63. Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 64. Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 65. Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 66. Kibana 4 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 67. Getting up & running is easy • Download Elasticsearch, logstash & Kibana archives # elasticsearch-1.4.0/bin/elasticsearch # kibana-4.0.0-BETA2/bin/kibana # logstash-1.4.2/bin/logstash agent -f logstash.conf # open localhost:5601 Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
  • 68. Thanks for listening! Q & A P.S. We’re hiring http://elasticsearch.com/about/jobs P.P.S. We’re helping http://elasticsearch.com/support http://elasticsearch.com/training Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Alexander Reelsen @spinscale [email protected]