![© 2018 SPLUNK INC.© 2018 SPLUNK INC.
Our Idea
“SplDevOps” Became the Solution
Version Control
Git[Lab] Utilized
Multiple Projects/Branches
Key Releases Tagged
Full Route-to-Live
Multi-Stage Environments
Dev > Pre-Prod > Prod
• Automated Testing
Agile Development
Short Sprints
Test Driven Development
Issue Management
& Feature Backlog
Configuration Management
Orchestrated Deployment
Centralized Config
Ansible used via SSH](https://image.slidesharecdn.com/fn1461foundationsmclarenspldevopsfinalv2-181010073526/85/SplDevOps-Making-Splunk-Development-a-Breeze-With-a-Deep-Dive-on-DevOps-Containerization-Version-Control-and-Automation-9-320.jpg)
SplDevOps: Making Splunk Development a Breeze With a Deep Dive on DevOps' Containerization, Version Control and Automation
- 1. © 2018 SPLUNK INC.© 2018 SPLUNK INC. SplDevOps Making Splunk Development a Breeze with a Deep Dive on DevOps, Containerization, Version Control & Automation Harry McLaren, Ilias Diamantakos, Tomasz Dziwok October 2018 | Version 1.3
- 2. © 2018 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2018 Splunk Inc. All rights reserved. Forward-Looking Statements
- 3. v © 2018 SPLUNK INC. HARRY MCLAREN Splunk Enablement Lead, Managing Consultant ILIAS DIAMANTAKOS Splunk Engineer, Associate Consultant Who Are We? cyberharibu ilias-diamantakos
- 4. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Best Security Company of the Year Employer: ECS (UK Splunk Elite Partner)
- 5. © 2018 SPLUNK INC.© 2018 SPLUNK INC. What’s It All About? ▶ Customer Challenges ▶ What Do We Want? ▶ Our Idea to Deploy Splunk ▶ Technical Deep Dive ▶ Project Roadmap ▶ Key Takeaways ~40mins
- 6. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Customer Challenges “The expansion of Splunk has increased operational complexity, as we manage it manually and can’t keep on top of project change requests.” – High-Street Retailer “We require a full route- to-live to maintain system integrity and can’t deploy changes fast enough in our current setup.” – National Bank “Multiple developers within the same DEV environment, causes repeated configuration conflicts and delays to planned changes.” – National Building Society
- 7. © 2018 SPLUNK INC.© 2018 SPLUNK INC. What Do We Want? Enterprises Want to Respond Quickly, Safely & With Less Risk Rapid Changes to Splunk Software 01 Orchestrated Deployment 02 Fragile Route-to-Live Fail Safe, Fast Backout 03 Development at Scale • Enterprise Scale Development • Synchronous Changes / Multiple Admins & Developers • Splunk Defined via Code • Familiar Approach (AKA: DevOps/Agile) Reduction in Custom Config • Every ’Custom’ Configuration Introduces Disparity • Inconsistent Dev, Test, Pre-Prod, Prod • Testing is “Best Endeavors” • Increased Risk, Changes Batched
- 8. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk for Agility Supporting Agile Methodology by Default Schema at Read, Supporting Multiple Use Cases Analytic Tools Exposed to UI, Empowering Users to Experiment Plain Text Configuration Files, Documented & Supported Splunk API is Enumerated, Dev Licenses, Labs Encouraged SPL Web UI Plain Text Config Open API Monitor InvestigateIntelligence
- 9. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Our Idea “SplDevOps” Became the Solution Version Control Git[Lab] Utilized Multiple Projects/Branches Key Releases Tagged Full Route-to-Live Multi-Stage Environments Dev > Pre-Prod > Prod • Automated Testing Agile Development Short Sprints Test Driven Development Issue Management & Feature Backlog Configuration Management Orchestrated Deployment Centralized Config Ansible used via SSH
- 10. © 2018 SPLUNK INC. Project: Internal Monitoring Ask: Deploy Splunk Internally for SecOps & ITOps Use Cases
- 11. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Automation Engine Containerization Version Control Brief Background Let’s talk tools!
- 12. © 2018 SPLUNK INC. What Tool Fits Where? a new Splunk infrastructure the DevOps way! ▶ Identical environments & route-to-live • Development, Pre-production, Production ▶ Eliminate fear driven development • It’s ok to make mistakes! ▶ Minimize direct production changes • Always go through route-to-live • Transparent change control ▶ Modern means of disaster recovery ▶ Security driven ← Ansible + Git + Docker + Python ← Docker + Git ← GitLab ← Ansible (IaC) ← Ansible Vault
- 13. © 2018 SPLUNK INC. How We Wanted It To Look Spoiler Alert: This is also the end result dev pre-prod prod IX: Splunk Indexer SH: Splunk Search Head DS: Splunk Deployment Server AS: Ansible Server
- 14. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Multiple Repositories Of /opt/splunk/etc for each instance Ansible & DS IX splunk_ix SH splunk_sh Syslog Collector syslog ansible splunk_ds
- 15. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Git Workflow aka “the change process”
- 16. © 2018 SPLUNK INC. Everything starts from our DevEnv So let’s spin one up
- 17. © 2018 SPLUNK INC. What’s going on in the background
- 18. © 2018 SPLUNK INC. What’s going on in the background
- 19. © 2018 SPLUNK INC. How It Looks
- 20. © 2018 SPLUNK INC. How It Looks
- 21. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Let’s Share Secrets No really, we are sharing! ▶ How to version sensitive information • Encryption ▶ How to decrypt automatically • Ansible Vault ▶ How to store Ansible Vault Password • More encryption
- 22. © 2018 SPLUNK INC. Let’s Decrypt One password to rule them all Ansible Server
- 23. © 2018 SPLUNK INC. Use Case Scenario Demo time
- 24. © 2018 SPLUNK INC.© 2018 SPLUNK INC. How it should have been done Integrating with our change process
- 25. © 2018 SPLUNK INC.© 2018 SPLUNK INC. How it should have been done Integrating with our change process
- 26. © 2018 SPLUNK INC. How it gets deployed
- 27. © 2018 SPLUNK INC. How it gets deployed
- 28. © 2018 SPLUNK INC. How it gets deployed
- 29. © 2018 SPLUNK INC. Lessons We Learned Not everything was easy… ▶ Multiple repositories • What goes where? • Many lines of history ▶ Identical code for different environments • There are always exceptions (Eventgen, production API calls) ▶ Data for different environments • Production data is sensitive ▶ Automated deployment of code • When do you restart?
- 30. © 2018 SPLUNK INC. Deployment Results Did it work!?
- 31. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Full Route-to-Live Implemented in Production Users & Admins Educated & Empowered Everything Under Version Control Promoting Changes in ~ 5mins (Dev>Prod) Foundations Built for Future Development End Result Prototype Success, Production Rollout
- 32. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Adaptable Framework Expressed in Software (Python + Git + Ansible) Environment Agnostic & Scales to Clustered Deployments& Hybrid Cloud Architecture User Friendly & End-to-End Integrated with Issue/Change Management Roadmap Introducing “Splunk Compiler” (v2.0+)
- 33. © 2018 SPLUNK INC.© 2018 SPLUNK INC. Splunk Supports Experimentation by Default Agile/DevOps Methodologies are Compatible Doesn’t Require Automation Expertise Version Control BEFORE Software Orchestration Key Takeaways Remember Four Things…
- 34. © 2018 SPLUNK INC. Don't forget to rate this session in the .conf18 mobile app Thank You