Exploring Splunk

cleverdata.ru | info@cleverdata.ru
EXPLORING SPLUNK>

Splunk to the Rescue in the Datacenter
It’s 2 AM, the web site is down.
Why did it fail?
Was it the web servers, the applications, the database servers, a full disk, or load balancers on the
fritz?
Relax. You deployed Splunk yesterday.
Search the log files from all your web servers, databases, firewalls, routers, and load balancers, as well
as search configuration files and data from all your other devices, operating systems, or applications of
interest.
A graph of web server show me when was a problem - at 5:03 PM, errors on the web servers spiked
dramatically -> check top 10 pages with errors:
The home page is okay.
The search page is okay.
The shopping cart is the problem. Starting at 5:03, every request to that page produced an error.
Shopping cart connected to a database -> logs shows the database is up -> ecommerce server logs.
At 5:03 PM, the ecommerce server cannot connect to the database server. -> changes to the
configuration files and see that someone changed a network setting.
It was done incorrectly. You contact the person who made the change, who rolls it back, and the system
starts working again.

Splunk to the Rescue in the Marketing Department
The promotions department of a large retailer.
Looking at the graph for the last few hours, you see a spike 20 minutes ago. Searches for your company
name and your latest product are way up.
You check a report on top referring URLs in the past hour and Splunk shows that a celebrity tweeted about
the product and linked to your home page.
You look at another graph that shows performance of the most frequently visited pages. The search page is
overloaded and slowing down. A huge crowd of people is coming to your site but can’t find the product they
are looking for, so they are all using search.
You log on to your site’s content management system and put a promotional ad for the new product at the
center of the home page.
Search traffic starts to drop, and traffic to the new product page starts to rise, and so does traffic to the
shopping cart page. You look at the top 10 products added to the cart and the top 10 products purchased;
the new product tops the list.
You send a note to the PR department to follow up. Incoming traffic is now converting to sales instead of
frustration, exactly what you want to happen. Your ability to make the most of an unforeseen opportunity
was made possible by Splunk.
Your next step is to make sure that you have enough of that product in stock, a great problem to have.

Approaching Splunk
As you use Splunk to answer questions, you’ll find that you can
break the task into three phases:
• First, identify the data that can answer your question.
• Second, transform the data into the results that can answer
your question.
• Third, display the answer in a report, interactive chart, or graph
to make it intelligible to a wide range of audiences.
Begin with the questions you want to answer:
• Why did that system fail?
• Why is it so slow lately?
• Where are people having trouble with our website?
• How many purchases?
• What is conversion?
• What are TOP/FLOP products?
• And many others questions

Splunk: The Company and the Concept
In 2002, Erik Swan and Rob Das started talking to companies about their
problems. They asked prospective customers, “How do you solve problems in
your infrastructure?”
These practitioners told Splunk’s founders that solving infrastructure problems
was like slowly crawling around in caves (their datacenters) with pickaxes, poor
lighting, and limited navigational power (old scripts and log management
technologies). In short, it was like spelunking—and so, the name “Splunk” was
born.
“Why couldn’t searching IT data be as easy and intuitive as a GoogleTM search?”

How Splunk Mastered Machine Data in the Datacenter
• Creating a central repository is vital: One of the major victories of Splunk is the way
that diverse types of data from many different sources are centralized for searching.
• Splunk converts data into answers: Splunk helps you find the insights that are buried
in the data.
• Splunk helps you understand the structure and meaning of data: The more you
understand your data, the more you’ll see in it. Splunk also helps you capture what you
learn to make future investigations easier and to share what you’ve learned with
others.
• Visualization closes the loop: All that indexing and searching pays off when you see a
chart or a report that makes an answer crystal clear. Being able to visualize data in
different ways accelerates understanding and helps you share that understanding with
others.

Operational Intelligence
Splunk has been at the forefront of raising awareness about operational intelligence, a new category of methods and
technology for using machine data to gain visibility into the business and discover insights for IT and the entire
enterprise.
Operational intelligence enables organizations to:
• Use machine data to gain a deeper understanding of their customers: For example, if you just track
transactions on a website, you see what people bought. But by looking closely at the web server logs you can
see all the pages they looked at before they purchased, and, perhaps even more important for the bottom
line, you can see the pages that the people who didn’t buy looked at. (Remember our new product search
example from the intro?)
• Reveal important patterns and analytics derived from correlating events from many sources: When you can
track indicators of consumer behavior from websites, call detail records, social media, and in-store retail
transactions, a far more complete picture of the customer emerges. As more and more customer interactions
show up in machine data, more can be learned.
• Reduce the time between an important event and its detection: Machine data can be monitored and
correlated in real time.
• Leverage live feeds and historical data to make sense of what is happening now, to find trends and
anomalies, and to make more informed decisions based on that information: For example, the traffic created
by a web promotion can be measured in real time and compared with previous promotions.
• Deploy a solution quickly and deliver the flexibility needed by organizations today and in the future—that
is, the ability to provide ad hoc reports, answer questions, and add new data sources: Splunk data can be
presented in traditional dashboards that allow users to explore the events and keep asking new questions.

Operational Intelligence at Work
Using machine data in Splunk helps solve vexing business problems. Here are a few examples:
• An operations team implemented a cloud-delivered customer-facing application and used Splunk for
diagnostics. They soon realized they could track user statistics and better plan capacity—a metric with
profound business implications.
• Web server traffic logs can be used to track shopping carts being filled and abandoned in real time. The
marketing department can use this information to determine where consumers are getting stuck and what
types of purchases are being abandoned so that any problems can be fixed right away and promotions can
target items that are abandoned.
• Organizations using Splunk to monitor applications for troubleshooting have realized that they can easily
provide views to their first-line support team to handle customer calls directly, versus escalating those calls to
expensive engineering resources.
• A major utility company was able to eliminate costly software maintenance fees by replacing six other
monitoring and diagnostic tools with Splunk, while enhancing their NERC and SOX compliance efforts.
• A major public media organization reduced the time it took to capture critical web analytics from months to
hours. They were also able to track their digital assets with a granularity and accuracy that they couldn’t have
otherwise, resulting in better royalty accounting and content marketing.
• A taco fast-food restaurant connected its points of sale (POS) to Splunk, and within an hour, business analysts
were able to begin answering questions like, “How many people are buying tacos in the midnight-to-2 AM
period, in this geography, during this time of the year?”
Operational intelligence enables organizations to ask the right questions, leading to answers that deliver
business insights, using combinations of real-time and historical data, displayed in easily digestible dashboards
and graphical tools.

Machine Data Basics
Splunk> mission is to make machine data useful for people.
Systems (such as web servers or load balancers or video games or social media platforms) write to log files when they
are running. This information (the machine data in the log files can use to understand what those systems are doing
as they run (or fail to run).
The universe covered by machine data is much more than log files—it includes data from configuration, clickstreams,
change events, diagnostics, APIs, message queues, and custom applications.
Some types of machine data:
• Applications Logs
• Web Access Logs
• Web Proxy Logs
• Call Detail Record (CDR)
• Clickstream data
• Message Queue
• Packet Data
• Configuration Files
• Firewall and IDS logs
• Database Audit Logs and Tables
• File System Audit Logs
• Management and Logging APIs
• OS Metrics, Status, and Diagnostic Commands
• Logs from DNS, DHCP, and other network services record
• Syslogs from your routers, switches, and network
• OS logs
• Tweets

Machine Data Contains Critical Insights

Splunk Data Sources
During indexing, Splunk can read machine data from any number of sources. The most
common input sources are:
• files: Splunk can monitor
specific files or directories. If
data is added to a file or a new
file is added to a monitored
directory, Splunk reads that
data.
• the network: Splunk can listen
on TCP or UDP ports, reading
any data sent.
• scripted inputs: Splunk can
read the machine data output
by pro- grams or scripts, such
as a Unix® command or a
custom script that monitors
sensors.
• apps

Platform for Machine Data

Splunk indexes raw data by creating a time-based map of the words in the data without
modifying the data itself by dividing a stream of machine data into individual events.
Each event has at leas 4 default fields:
Understanding How Splunk Indexes Data

Searching with Splunk

SPLTM: Search Processing Language
index="mygizmostoreindex" | chart count AS views, count(eval(method="purchase")) AS
purchases by categoryId | rename views as "Views", purchases AS "Purchases", categoryId
AS "Category”
index="csv_zakupki" | stats sum("Общая цена контракта тыс_ руб") as
Summa, sum("Количество допущеных заявок") as Qty by "Победитель"| eval
"Средняя цена контракта"=(Summa/Qty) |sort -"Средняя цена контракта" | head 10
| fields - Summa, Qty

Geospatial data based on IP

Web Store Dashboard

Web Site Monitoring Tool

Why Splunk fast?
The typical components of a Splunk deployment are made up of the following (all of
which can exist on a single machine):
• Search Head: The web service you login to through your browser and submit searches,
view dashboards, etc.
• Indexer: Does initial parsing of event data and stores it to disk
• Forwarder: Gathers the event data and delivers it to an Indexer
When a search is submitted by a user, the search head submits that search to each indexer
individually and they run in parallel (distributed).
A “map” function grabbing data in parallel execution fashion.
sourcetype=syslog src_ip=192.168.1.1 | chart count by host
The map function = “sourcetype=syslog src_ip=192.168.1.1”
The reduce function = “chart count by host”
Splunk is unique because it built a framework around MapReduce and setup a very handy
search language that easily and directly gets translated into a MapReduce job.

Splunk Architecture
• Scales to TBs/day and Thousands of Users
• Automatic load balancing linearly scales indexing
• Distributed search and MapReduce linearly scales search and reporting

Scaling and High Availability
The functionality of Splunk can be roughly broken
down into three basic areas:
• Search Head: The web service you login to
through your browser and submit searches,
view dashboards, etc.
• Indexer: Does initial parsing of event data and
stores it to disk
• Forwarder: Gathers the event data and delivers
it to an Indexer
Splunk can scale them individually.
Clustering Splunk indexers is the way to achieve
high availability. The coordination of the
replication, failure, and other clustering specific
items is done by a cluster master.

Exploring Splunk

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Exploring Splunk (20)

More from Dmitry Anoshin (20)

Recently uploaded (20)

Exploring Splunk

Editor's Notes