Splunk in Nordstrom: IT Operations
1 like•278 views
The document discusses Nordstrom's use of Splunk for log aggregation and analytics across their IT systems. Some key points: - Nordstrom uses Splunk to consolidate machine data from various systems like point-of-sale devices, web servers, and monitoring tools for unified visibility. - Splunk has been adopted organically by over 300 users across Nordstrom who use it for tasks like performance monitoring, troubleshooting, and building custom reports. - Nordstrom applies DevOps principles and tools to manage their large and distributed Splunk deployment, with components like search heads, indexers, and deployment servers. Configuration is managed through version control.
Splunk in Nordstrom: IT Operations
- 1. Copyright © 2014 Splunk Inc. Bernie Macias Applied Architect, Nordstrom Mobile POS, DevOps and the Role of Splunk
- 2. Disclaimer 2 During the course of this presentaIon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauIon you that such statements reflect our current expectaIons and esImates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presentaIon are being made as of the Ime and date of its live presentaIon. If reviewed aTer its live presentaIon, this presentaIon may not contain current or accurate informaIon. We do not assume any obligaIon to update any forward-‐looking statements we may make. In addiIon, any informaIon about our roadmap outlines our general product direcIon and is subject to change at any Ime without noIce. It is for informaIonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaIon either to develop the features or funcIonality described or to include any such feature or funcIonality in a future release.
- 3. Who Am I? ! Current PosiIon: 1+ years – Applied Architect ! Past Experience: Since 2005 – Tech support, Windows system admin, windows system engineer, system architect, Unix analyst ! Splunk Experience: 3+ years – Deployed approved Splunk architecture at Capital One – Started the Splunk> Sea^le user group ! Other Tech Interests: Ongoing – Python, Django, Data visualizaIon (d3.js), Open Stack • 3
- 4. Agenda ! About Nordstrom ! What s Nordstrom Doing with Splunk? ! Managing and Scaling Splunk ! DevOps on Splunk ! What About the Users? (Gecng the Users Involved) 4
- 5. About Nordstrom ! Founded by John W. Nordstrom in 1901 ! Over 260 stores in US and Canada ! eCommerce: Nordstrom.com, NordstromRack.com, HauteLook.com ! 65,000 employees; over 1,000 in IT ! MulIple data centers with an eye on the cloud or mulI-‐cloud ! DevOps mission: scale, manage, and quickly deliver on Nordstrom iniIaIves • 5
- 6. Omni-‐channel at Nordstrom Deliver a Seamless Customer Experience 6 Nordstrom Store Nordstrom Online Nordstrom Rack Nordstrom Rack Online / Haute Look Nordstrom Omni-‐Channel
- 7. What’s Nordstrom Doing with Splunk?
- 8. Web apps OperaIonal logs, system logs, Web logs, Crash logs Win/Unix metrics (2000+ servers) Chef data POS and Wi-‐Fi data NNMi, SCOM, Gomez Splunk @ Nordstrom ConsolidaIon of all our machine data for unified visibility 8
- 9. Example: Point of Sales (POS) 9 • New customer experience • Faster checkout anywhere in store • Small real estate footprint v. tradiIonal POS
- 10. Performance Monitoring of POS Devices 10
- 11. 11 POS Inventory Mapping 11
- 12. Splunk AdopIon Has Been Organic and Viral • AgnosIc to technology • Index any type of data from any data source • Scalable soluIon SINGLE SOURCE OF TRUTH • Accelerated adopIon with 300+ users across the organizaIon • Dev, App, NW, Ops and global offshore teams create their own data inputs, reports and dashboards EASE OF USE • Role-‐based access controls to provide dev teams access to producIon logs and metrics • Flexible reporIng across a variety of use-‐cases SECURE & FLEXBILE PLATFORM 12
- 13. AcceleraIng Value by Using Splunk Apps ! Technology Add-‐ons: – *nix – Windows – Cisco IOS – MicrosoT Exchange ! Homegrown App – Splunk for Gomez – Few Django Apps ! Splunk for simple XML 13
- 14. Secng the Stage for Managing and Scaling Splunk 14
- 15. How Many of You Would Consider Yourself…? 15 SYS ADMIN SPLUNK ADMIN SPLUNK USER
- 16. Distributed Splunk gets complex 16
- 17. 17 All-in-One Search Indexers FW HFWSYSLOG TCP HTTP FW HFW Pool Storage Search Pool #1 Search Pool #2 GA Indexers Pool Storage Secure Indexers Search Pool #3 Pool Storage Indexers Indexers FW HFW Managed Per Available Zone Forwarder/Agents only send data to Zone Specific indexers TOPOLOGIES OF SPLUNK
- 18. InstallaIon and ConfiguraIon 18
- 19. Using DevOps Principles to Manage Splunk 19
- 20. Why Not DevOps for Splunk? 20
- 21. DevOps Tools ! Config Management Plauorm ! Source Control ! Dev Environments ! ConInuous IntegraIon Tools for building tools spanning: 21
- 22. How Many Splunk Components? ! NFS Server – shared search head storage ! Search Head – searches indexed data ! Indexer – parsing and indexing data ! Deployment Server – App Deployment ! Intermediate forwarder – receiving or collect data where forwarder cannot be install directly ! Master – Cluster and ReplicaIon Master ! Universal Forwarder – local collecIon agent 22
- 23. nord_chef-‐splunk: a CHEF cookbook ! Ruby code that models distributed Splunk (search, index, etc) ! Reuse able code defined by a^ributes CONSISTENT, SCALABLE, REPEATABLE 23
- 24. What's Automagically Configured? • Splunk SSL • TCP and UDP listen Ports • Set system local configs • Distributed Search • Mounted Bundles • Indexers aware of all search pools • And More • Move default DB locaIons • Add user and change Admin • Splunk servers share Secret • Search pooling • Set Deployment Server • Search Heads aware of Indexers • Drives configured • Web server 24
- 25. What About the Users? 25
- 26. What Can Users Do In Splunk? • Create private objects • Cannot create/edit global objects directly in Splunk • Dashboards? Extracts? Saved searches? 26 USE GIT!
- 27. CreaIng a Custom GIT CLI • Downloaded Web Terminal for Splunk App • Installed on limited capability search head • Customized for GIT CLI 27
- 28. Sample Deployment Workflow 28 Insert Image(s)
- 29. 29 Splunk is a Journey What’s Next?
- 30. Special Offer: Try Splunk MINT Express for Free! Splunk MINT offers a fast path to mobile intelligence. How fast? Find out with a 6-‐month trial* • Register for your free trial: h^p://mint.splunk.com/conf2014offer • Download the Splunk MINT SDKs • Add the Splunk MINT line of SDK code and publish** • Start gecng digital intelligence at your fingerIps! *Offer valid for .conf2014 a5endees and coworkers of a5endees only. **Trial allows monitoring of up to 750,000 monthly acDve users (MAUs). 30
- 31. THANK YOU 31