Splunk live university of alberta 2015

Editor's Notes

• #2: Good morning everyone and welcome to SplunkLive Calgary Thanks so much for having me at your SplunkLive today
• #3: Graduated from CS at University of Alberta in 2002, worked for various research projects and did some contract development for a few years. Joined university again in 2007.
• #4: University of Alberta – friendly neighbor about 300 KM that away (for main campus) 18 faculties includes some big ones like Science, Medicine and Engineering The University ranking changes every year as well as different rankings get published. This seemed like a safe enough statement to make without going into half a page of small print.
• #5: Our starting point about 4 years ago. Over the last 4 years or so we’ve been consolidating about 300+ individual groups IT into central department. Originally this was envisioned as a 10 year plan, so we still have a ways to go. At this point we are supporting a lot of different institutional needs, lots of different technologies and life is very exciting.
• #6: Application Hosting. We manage the applications and databases for a lot of clients across campus. There are some big applications that are managed by others (LMS, Peoplesoft), but we make up for it with the number of different applications we support. I’ve stopped being amazed at the number of needs an institution of this size has. We have an application for printing out labels to put on file folders, tracking project time, billing and invoicing, databases supporting libraries, ticketing systems, wikis, departmental pages. Typically there is a piece of software behind a lot of business processes and all of that needs to be monitored, patched, upgraded. As our consolidation effort continues we will be using Splunk to look into how an application is used in order to determine how it could be consolidated with other applications of similar function. There is an amazing amount of information about usage patterns, what gets accessed and how often and who does the accessing.
• #7: We’ve re-organized ourselves along functional lines. OS Support, Networking, Application Hosting, etc. What that means is that some investigations spanning multiple teams become very time consuming and expensive (two people looking at logs). Some of that is unavoidable, and even desirable, but for a large number of errors we’re just missing that one piece of crucial information that “solves” the problem. That could be a log line from the VM host indicating physical hardware problem, log from authentication system detailing why connection was rejected, etc. etc. Splunk allows us access to that information. Here is where I need to bring up a big warning flag. Having access to the logs does not mean you can understand the logs. There are some errors where the team running a system is required to correctly interpret the logs, but in general having more eyes is a good thing. Some of the expertise can be developed over time, some more through developing dashboards and applications within Splunk.
• #8: I get a kick of these timelines, so I wanted to add our own. I’ve seen a few at a Splunk conference and they typically go something similar. An organization gets Splunk in limited capacity, something happens (systems get hacked, phishing attack, etc) and next year they are running 10x the license. We’ve had a bit of a different experience, where the Splunk importance snuck up on us. In September 2013 we’ve deployed Splunk as a pilot. Some of the conversations at that point were along the lines of “backups are not important as all the systems keep their own logs” Splunk is just a view into the logs, there is little to no new information contained. Let’s just send logs to see what we can make out of it. By March we were becoming concerned every time we needed to restart Splunk, since that meant data loss. Our installation was configured as a syslog target for networking devices and IMS logs so if Splunk was not there to receive the events, they went nowhere. By September (1 year later) we needed to notify management of Splunk outages because of the data loss. In April / May this year we are rebuilding the environment in clustered configuration with dedicated syslog servers. I don’t know if I can think of any one moment where Splunk suddenly became critical production environment, but it definitely is one now.
• #9: What follows is a gross generalization. I obviously understand the challenges within my team the best, so take this with a grain of salt. Although other groups do use and send logs to Splunk, the three main groups are Infrastructure Applications, Networking and Application Hosting. It’s interesting that we all have different needs and different use cases. Infrastructure has a small number of highly critical applications that they need to understand end to end (mail, central authentication, etc). Networking has a few data sources that are fairly similar (switches, firewalls, etc). Security looks at a few different data sources like VPN and IPS as well as authentication logs from a number of systems. Application Hosting currently has a relatively few number of sources, but a lot of them are different and unique. There is probably 5 different ways apache is configured to log access requests, we run every major database type and version. Postgresql, MySQL, MSSQL, Oracle. In addition we support applications themselves or interact with software vendors on behalf of clients. On my last monthly report there were 374 different relational databases in environments supported by my team.
• #10: An example from our Infrastructure Applications team. Our authentication system (approximately 180,000 accounts) generates around 12 GB of logs / day. Logs were stored on each individual node of a cluster in a text file. Trying to find logs related to a specific login id required signing on onto the 20 of so systems and using “grep” to identify individual log lines. That was not a quick task and required generating IO load on the servers. Doing anything more advanced than that was nearly impossible. After. We have summary indexes and reporting indexes on this data to quickly answer specific questions we know will be asked. We can correlate with data from other systems and alert in real-time for specific events. Users are no longer our main issue detection method.
• #11: This is something we’ve used with great effect in my team for a few performance investigations so far. We break down the web traffic by percentiles and return size. It allows us to pinpoint problems (in some cases) as well as provide an instant report to the client how their application is really performing. This query is both complex enough to be useful, yet simple enough that I can explain it to non technical clients.
• #12: This is a new initiative coming out of my team. It is called a First Responders App (even though it is currently a dashboard, I know it will end up an app eventually). First Responders is meant as the place to go at a start of an incident. It’s meant to put a lot of infrastructure information at analyst’s fingertips and it spans information from all of the operational teams. It allows an analyst to verify backup status, check logs, check tickets / change calendar, check monitoring system, who has logged in into the system last, etc. As we’re still rolling it out we do not have all the information we would perhaps wish, but the reception so far has been very positive.
• #13: Also part of First Responders we have a holistic look at the logs. Not only do we look at the logs from the system, we also look at logs about the system. If our IPS detects activity against the host, that will appear in the window at top left. Same if out authentication suddenly starts throwing a lot of errors or messages about the host. Sparklines are great for very quickly identifying patterns and seeing if something is unusual. Lastly we have a query I’ve been playing around for a some months. This query tries to generate a statistical baseline of events from a system, and then compare last full hour against that baseline to highlight issues. In this screenshot I relaxed my alerting thresholds, that’s why the z scores are so low, it does illustrate what the output looks like. I’m working a next generation of this type of query that will log all deviations from the entire environment every hour. Wish me luck.
• #14: This was probably our management’s first introduction to splunk, monthly reports for critical systems. We are still tweaking what goes on the reports, and will continue to do so. This particular dashboard shows our Adobe Connect, including a histogram of meeting sizes, which did include classes in the 240 – 245 participant range as well as an AppDex score. Eventually I’d like to standardize all application monthly reports and have them send automatically to each client / department.
• #15: Things that worked and things that did not work so well. The main things that were successes: You can really defuse a situation by being able to rapidly provide facts. If you’re able to provide a list of users who accessed a specific file in the first 15 minutes of a breach investigation, that really brings down the stress level of everyone involved and situation rapidly de-escalates. Similarly for performance investigations. Everyone in the system can see all logs, we try to have the system as open as possible to all IT. Challenges: There is so many different way of configuring logging. That makes getting consistent reports a challenge. Knowing what is “normal”. Having the ability to rapidly generate graphs of values and have data that goes back a few months (at least) is highly beneficial. Syslog. Where possible use universal forwarders, where not possible have a syslog collector.
• #16: Some of the “AHA moments” Using transactions to create message centric logs. It was nothing short of magical, especially when compared to the good old “grep” command across multiple systems. Generic alerts. Ability to create alerts that work for systems that are not in the system yet. The ability to look at the entire environment as a single event stream is incredibly powerful. An extension of the previous. While investigating a hiccup of some sort I performed a time constrained query across the entire environment. I wanted to see whether the error was limited to this system, or whether it appeared anywhere else. Using two windows I was able to run simple queries on specific messages and determine “normal event” or “not”. It was an amazing, and humbling, look at our environment. So many things happened within that 10 second window.