The Hitchhiker's Guide to Service Intelligence

Editor's Notes

• #3: Here is a 3.5-hr schedule: 00:00     (slides 1-6) Introductions and Setup 00:07     (slides 7-11) Splundamentals -- Core Splunk in IT Ops 00:15     IT Troubleshooting Demo 00:25 (slides 12-24) Splunk IT Service Intelligence (ITSI) 00:45     ITSI Tour 00:55     BREAK   01:00     (slides 25-39) Service Intelligence Design Practices 01:20     (slides 40-62 / demo) Let's Play – Setting up Service Intelligence 01:55     BREAK   02:00     (slides 62-67 / demo) Let’s Play -Troubleshooting exercise 02:15     (demo) Advanced Exercise #1 - Create a SmartPhone Service 02:15     (demo) explore the 'sourcetype=mint:network' data; what KPIs are possible? 02:20     (demo) Create new service ("SmartPhone"), create 3 KPIs 02:35     BREAK   02:40 Advanced exercises continued 02:40     (demo) Modify 'Online Store' GT to add the new service 02:45     (demo) Advanced Exercise #2 - Show Adaptive Thresholding 02:47     (demo) Advanced Exercise #3 - Show Anomaly Detection 02:50     (slide 70) Summary 02:55     BREAK   03:00 (slides 71-78) What our ITSI customers are doing 03:20     (slide 79-80 + sign up discussion) What is GTE?  Ideas for ITSI use cases? 03:25     (slides 81-82) Wrap up 03:30     DONE!
• #5: In order to solve these evolving needs, a certain class of customers began to leverage the platform and take advantage of the data they already had indexed within Splunk. They built some pretty sophisticated use-cases to improve operational efficiencies. And they way they are doing this is by adding a service perspective to the data they already have in Splunk. What became apparent as we spoke to those customers was that we have the ability to transform this age-old problem of troubleshooting and monitoring with a new approach driven by machine data. Given our customers had custom built service insights using the data they already had in the platform, it was a natural evolution for us to build an integrated solution based on our customers successes and make Splunk service-aware. This helps our customers to maximize the value they can get from Splunk with a machine data driven approach to monitoring and analytics.
• #6: This is the “IT Troubleshooting” demo, about 15-20 min Cover Splunk 101 basics
• #7: Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most machine data generated by all of the devices, control systems, sensors, SCADA, networks, applications and end users connected by industrial networks. There are many options: Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API and more. You can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis. The Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries   The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The data volumes supported by Splunk are ideal for IoT and industrial data. There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include: Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS.  Rest API Modular Input: Poll local and remote REST APIs and index the responses. Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data. Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper. DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection. MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics. AMQP Modular Input: Index data from message queues provided by AMQP brokers. JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ. COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server. SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution. Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events. Splunk isn’t the only technology that can benefit from collecting machine data, so let Splunk help send the data to those systems that need it. For those systems that want a direct tap into the raw data, Splunk can forward all or a subset of data in real time via TCP as raw text or RFC-compliant syslog. This can be done on the forwarder or centrally via the indexer without incrementing your daily indexing volume. Separately, Splunk can schedule sophisticated correlation searches and configure them to open tickets or insert events into SIEMs or operation event consoles. This allows you to summarize, mash-up and transform the data with the full power of the search language and import data into these other systems in a controlled fashion, even if they don’t natively support all the data types Splunk does.
• #8: Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most machine data generated by all of the devices, control systems, sensors, SCADA, networks, applications and end users connected by industrial networks. There are many options: Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API and more. You can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis. The Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries   The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The data volumes supported by Splunk are ideal for IoT and industrial data. There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include: Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS.  Rest API Modular Input: Poll local and remote REST APIs and index the responses. Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data. Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper. DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection. MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics. AMQP Modular Input: Index data from message queues provided by AMQP brokers. JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ. COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server. SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution. Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events. Splunk isn’t the only technology that can benefit from collecting machine data, so let Splunk help send the data to those systems that need it. For those systems that want a direct tap into the raw data, Splunk can forward all or a subset of data in real time via TCP as raw text or RFC-compliant syslog. This can be done on the forwarder or centrally via the indexer without incrementing your daily indexing volume. Separately, Splunk can schedule sophisticated correlation searches and configure them to open tickets or insert events into SIEMs or operation event consoles. This allows you to summarize, mash-up and transform the data with the full power of the search language and import data into these other systems in a controlled fashion, even if they don’t natively support all the data types Splunk does.
• #9: The rise of big data has forced IT organizations to transition from a focus on structured, relational data, to accommodate unstructured data, driven by the volume, velocity and variety of today’s applications and systems. As the data has changed from structured data to unstructured data, the technology approach needs to change as well. When you don’t know what data types you’ll need to analyze tomorrow or what questions you need to ask in a week, flexibility becomes a key component of your technology decisions. The ability to index any data type, search across silos and avoid being locked into a rigid schema opens a new world of analytics and business insights to your organization. Schema at Read – Enables you ask any question of the deal Search – Enables rapid, iterative exploration of the data along with advanced analytics Universal Indexing – Enables you to ingest any type of machine data Horizontal scaling over commodity hardware enables big data analytics
• #11: That brings us to Splunk IT Service Intelligence – a packaged solution that enables real-time visibility into services driven by machine data. Splunk ITSI speeds and simplifies service monitoring and analytics and enables IT to make better, smarter and informed business decisions. This solution allows you to gain a deep understanding of your services. With Splunk ITSI, you have real-time views into the health of your services, and can use advanced analytics to find patterns, detect anomalies and trends to proactively monitor and address issues. As a result you have improved service visibility, reduced resolution times, and a transformative approach to monitoring and analytics driven by machine-data.
• #26: FOR THE PRESENTER: This entire Tour section should last no more than 10 min. Describe how GTs can show KPIs & health scores to any audience/group/team: Show GTs: Buttercup Games Business Process (executives, business service owners) On Line Transaction Service (NOC, Tier2); “can use visio diagrams…” Buttercup Games Online Store (service flow, sub-services) Show saved Deep Dive “DB Deep Dive”; BRIEFLY describe DD functionality (you will be able to go into more depth later) Show Notable Event Review, BRIEFLY describe (you will be able to go into more depth later) Show Service Analyzer, briefly describe Ask if the students have questions.
• #28: So, let’s look at a simple visual to discuss how it works? In four simple steps, customers can achieve data driven service insights. They Get the data in. (all the data…) They quickly define services, entities, and KPIs They monitor and troubleshoot They analyze and detect Through these steps, the customers is able to realize the value of Data Defined, Data Driven Service Insights.
• #32: In the “real world”, it will probably be necessary to iterate up & down these steps a few times. For example, what if a KPI requires data which is not being collected by Splunk?
• #35: FOR THE PRESENTER: The next three slides set the students up for decomp discussion, later. GOALS: Get the students to open two specific GTs in separate browser tabs.
• #36: These actions set the student up for decomp discussion, later
• #37: These actions set the student up for decomp discussion, later
• #38: TO STUDENTS: You have this glass table on your own system. This Glass Table shows the high-level business process for Buttercup Games. Does anyone notice anything missing? (no info in Order Entry) We need better visibility into our Online Store, which is part of the Order Entry process.
• #39: TO STUDENTS: You have this glass table on your own system. This Glass Table shows a more detailed process flow for the Online Store service. Notice the sub-services which make up our Online Store service, and how the process flows.
• #40: Based on a recent DB outage which was caused by a saturated network interface, we’ve decided that network utilization would be a handy KPI for our Database Service. We’re also going to tweak the high-level Business Process Glass Table to provide more visibility into the Online Store service. And we’re going to do it in 15 minutes!
• #41: FOR THE PRESENTER: Remind the students that they can refer to their own locally-downloaded slides for “click-by-click” reference for the process of adding a KPI. Then switch to your own browser and demonstrate these steps “live”. Have fun with the concept that a roomful of people can build a new KPI in only a few minutes, and that “the clock is ticking”.
• #42: FOR THE PRESENTER: SHORT discussion of entities
• #43: FOR THE PRESENTER: Briefly cover “data model” vs “ad hoc search”. Don’t spend a lot of time here.
• #44: FOR THE PRESENTER: Briefly cover the concepts on this page, and point how the “Generated Search” window at the bottom, and how cool it is that Splunk builds the search for you; does anyone in the audience have users who could benefit from this? QUICK TANGENT: In the typical working environment, which often has a chasm between the “Business Types” and the “Tech Types”, how long would it take to map services to actual infrastructure?  "Many quarters, and possibly a year-- on the conservative side, right?" To quantify that, by show of hands, has anyone here been involved in an IT Service Management / Business Management team trying to map every server to a service or business function?  Did you sustain any long-term injuries?  And even IF you are successful in this effort, as soon as you finish you have to start over. ITSI is remarkable because it can allow the Business teams and Technical teams to map out the important services realistically and effectively– in DAYS and WEEKS. We offer a Glass Table Workshop to facilitate such an exercise on YOUR services and YOUR data– in a single day.
• #45: Keep moving…
• #46: FOR THE PRESENTER: This might take a while for “waiting for data” to produce an actual graph for the students (1-2 minutes, typically). Instruct the students that if will take a couple of minutes for the data to appear, and to not click on anything in the meantime. Then skip to the Adaptive Thresholds and Anomaly Detection slides and discussion, while the students wait. Afterwards, can be helpful to gauge progress by asking for a show of hands to see how many students are still waiting. If necessary, simply show the students how to set thresholds (on your own browser), then move forward.
• #47: FOR THE PRESENTER: Talk through-- NOT HANDS ON
• #48: FOR THE PRESENTER: Talk through-- NOT HANDS ON
• #49: FOR THE PRESENTER: Talk through-- NOT HANDS ON
• #50: Talk through NOT WORK
• #51: We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
• #52: FOR THE PRESENTER: As before, switch to your own browser and demonstrate these steps “live”. Have fun with the concept of saving a copy before editing– so that you don’t muck it up.
• #53: FOR THE PRESENTER: Have fun with this GT editor section. The GT editor is a bit twitchy, so exploit the humor and have fun with the students. GOALS (for the next 3 slides): Identify 2 “interesting/useful” KPIs from the Online Store service, to position in the gray “Order Entry” oval; let the students choose details and viz types Put a ServiceHealthScore widget (from Online Store) under the pony, to show overall health of the service. Modify “custom drilldown” to land on the “Buttercup Games Online Store” GT Encourage the students to use text boxes and other techniques to make the widget more readable, prettier to look at Remind the students that “the boss’ boss” will be looking at this GT, and we want to make sure that they’ve got good visibility into “our” service (Online Store).
• #55: FOR THE PRESENTER: When finished (after everyone have hit ‘Save’ and ‘View’, and are looking at their own beautiful GTs): How long did it take to create a new KPI and make major changes to a Glass Table? Pretty cool! Ask the students if this (ITSI) could be useful in their own environments If you have more than 15 min of remaining time, speak through some actual (referenceable) customer ITSI use cases.
• #57: FOR THE PRESENTER: This hands-on section can be very powerful for the students. This allows them to “put it all together”, driving ITSI with their own fingers. As before, switch to your own browser and demonstrate these troubleshooting steps “live”. The corresponding slides are intended as reference for the students. If pressed for time (i.e., less than about 10 min), talk through and show this process– but don’t have the students attempt to “click along” in real time.
• #58: If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
• #59: Note that this “drill down” has inherited the same time selection (i.e., an earlier outage)– pretty cool! FOR THE PRESENTER: The major points here: During the heat of battle, when troubleshooting an outage, being able to visualize the entire service flow is extremely valuable By being able to see health status of all the underlying services, we can quickly choose where and how best to proceed. Potentially huge time savings– customers report major reductions in MTTR
• #60: FOR THE PRESENTER: This is a good “variable time” section. You can spend as little or as much time as you choose, depending on how much time you have remaining in the session. Remind the students that they will have more time to play with DD later (yes, they might be confused by this, since only a few minutes remain in the session)
• #61: FOR THE PRESENTER: This is another good “variable time” section. You can spend as little or as much time as you choose, depending on how much time you have remaining in the session. Remind the students that they will have more time to play with this stuff later (yes, they might be confused by this, since only a few minutes remain in the session)
• #63: If time permits, try some or all of these: Create a new service based on app data coming from the customers’ smartphones (sourcetype="mint:network”) No identified entities, though we will show “pseudo entities” KPI 1: Total hits (sourcetype="mint:network" | stats count) KPI 2: Errors by device type (sourcetype="mint:network" statusCode>399 | stats count by device) KPI 3: Latency by carrier (sourcetype="mint:network" | eval latency=(latency/1000) | stats avg(latency) by device) (Show pseudo-entities in Service Details) Add the new service to the “Online Store” GT Show students how to access & play with Adaptive Thresholds in Configure Services -> Web Service -> Corporate Web Requests Show students how to access & play with Anomaly Detection in Configure Services -> Web Service -> CPU % Show students more details about modules, through Service Details Create a new GT, using a customer diagram as the background Use existing services & KPIs as mockups, renaming the widgets to fit the customer’s environment