Devops (start walking in the same direction) by ops
- 1. DEVOPS START WALKING IN THE SAME DIRECTION
- 2. WHO I AM? Demis Rizzotto System Engineer,DevOps Linux Engineer @Exxoss Currently consultant @Lampiris
- 3. THE PROBLEM OF DOWNWARD SPIRAL
- 4. DIAMETRICALLY OPPOSED GOALS AND INCENTIVES DEV ▸ Respond to the rapidly changing competitive landscape OPS ▸ Provide stable, reliable, and secure service to the customer THE CORE, CHRONIC CONFLICT WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS
- 5. FRUSTRATION ▸Unnecessary process steps ▸Unnecessary rework ▸Unnecessary features ▸Waiting for someone else to act on an action or task ▸Waiting for an approval ▸Waiting for an environment ▸Creating a ticket for a manual task ▸Reporting to management by manually updating spreadsheets ▸Manual status reporting ▸…
- 6. RESULTS ▸ Fragile application are prone to failure ▸ Ever increasing backlog and technical debt ▸ Urgent security rework and remediation ▸ More urgent projects into the queue ▸ Big deployments very difficult to diagnose when fail
- 8. DEVOPS DEV OPS FACILITATE AND AUTOMATE DEPLOYMENT HELP DEV TEAM BECOME AUTONOMUS FACILITATE DIAGNOSTIC GET DEVS INTERESTED IN « PROD STUFF » GIVE ACCESS TO LOGS GIVE ACCESS TO MONITORING OPTIMIZE LOGS DEFINE METRICS QUALITY TESTS MONITOR YOUR APPS
- 10. DEV IN DEVOPS IMPROVEMENTS ▸ The ability to accurately track dependencies ▸ Services that are resilient and degrade gracefully ▸ The ability to archive data to manage the size of the production data set ▸ The ability to easily search and understand log messages across services ▸ The ability to trace requests from users through multiple services ▸ Simple, centralized runtime configuration using feature flags, …
- 11. DEV IN DEVOPS ▸ Make Dev responsible for their own services ▸ Integrate Dev into the incident management processes ▸ Have Dev cross-train Ops ▸ Embed Ops knowledge and capabilities into Dev ▸ Design for IT Operations EMBED IT OPERATIONS INTO DEVELOPMENT
- 12. OPS IN DEVOPS
- 13. OPS IN DEVOPS IMPROVEMENTS ▸ Self service ▸ Infrastructure as code ▸ Standardize infrastructure ▸ Make all infrastructure data visible ▸ Make application info data visible ▸ Modify the incident resolution process and blameless post-mortems ▸ Monitor the health of the deployment pipelines
- 14. OPS IN DEVOPS ▸ Single “repository of truth” containing both the code and environments ▸ Create the one-step Dev, Test and Production environment build process ▸ Extend the deployment pipeline processes into production EXTEND DEVELOPMENT INTO IT OPERATIONS
- 15. WE ARE ALL DEVOPS WE ARE ALL DEVOPS ▸ Stop think . Do ▸ Share responsability ▸ It’s Never Someone Else’s Problem ▸ Dedicated Designated Ops in every team
- 16. ▸ Different type of devops, but devops is possible for all type and size of companies ▸ Iterate! ▸ Identify bottleneck and remediate ▸ Define small target and share results DEVOPS IS FOR ALL DEVOPS IS FOR ALL
- 17. DEVOPS IS FOR ALL ▸ AUTONOMUS TEAMS ▸ Trust but verify ▸ Local autonomy, but global consistency ▸ SELF SERVICE ▸ How ? API, web portal, infra as code , AWS accounts, … ▸ AUTOMATE ▸ If you are tired to do it manually. Automate it. ▸ REDUCE NON-VALUE ADDED ▸ If isn't your business it's maybe better don't do it your self KEY OF SUCCESS : REDUCE FRUSTRATION
- 18. INFRASTRUCTURE
- 19. ON DEMAND ( IAAS ) EASIER TO REBUILD THAN TO REPAIR INFRASTUCTURE
- 20. INFRASTRUCTURE IMMUTABLE INFRASTRUCTURE ▸ Automate the setup and deployment for every part and every layer of your infrastructure. ▸ Never change any part of your system once it is deployed. If you need to change it, deploy a new system. AVOID HISTORY
- 21. INFRASTRUCTURE INFRASTRUCTURE AS CODE Versionning and code review Easier track changes Bootstrap your infrastructure from scratch Automate
- 22. CI/CD
- 23. TESTING
- 24. TESTING CONTINUOUS TESTING ▸ Automated tests ▸ code quality ▸ Unit tests ▸ Stress tests ▸ Don’t forget to test infrastructure (load banacer, recovery etc.) ▸ Extend you build pipeline
- 25. SECURITY
- 26. SECURITY INFORMATION SECURITY AS EVERYONE’S JOB, EVERY DAY ▸ Updates ▸ Use secrets management ▸ Integrate security into our CI/CD pipeline ▸ Check code libraries and their recommended configurations ▸ Dependency scanning ▸ Static analysis ▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)
- 27. SECURITY SECURITY METRICS MONITORING ▸ Successful and unsuccessful user logins ▸ User password resets ▸ User email address resets ▸ Database syntax error ▸ User credit card changes
- 28. DEPLOYMENTS
- 29. DEPLOYMENTS DEPLOYMENTS ARE ROUTINE AND PREDICTABLE No stress . It’s a routine. On business day when everyone is already in the office and without our customers even noticing
- 30. DEPLOYMENTS HOW ▸ Work in small batches and delivering quickly and frequently ▸ Blue green deployment ▸ Feature toggle ▸ Canary testing ▸ Dark launch ▸ Automated deployments
- 31. MONITORING
- 32. ALERTING ▸ Mesure service health not only server health (Health pattern) ▸ Reducing Alert Fatigue ▸ Cut alerts that aren’t actionable ▸ Adjust thresholds ▸ Consolidate related alerts ▸ Give alerts relevant names & descriptions ▸ Make sure the right people are getting alerts ▸ Multiple notification rules and choose right channel ▸ Use incident management tools(Pagerduty, Victoryops,..)
- 33. MONITORING IF IT MOVES, TRACK IT ANOMALY DETECTIONTRENDS
- 34. APM End user monitoring Application Performance Management
- 35. DELIVERY PIPELINE METRICS - TEAM LEVEL DASHBOARD
- 36. LOGS
- 37. LOGS ▸ Set your log severity right ▸ Remember you will not be the only one reading these logs ▸ Track your communication with other systems ▸ Log exceptions correctly ▸ Use an ID to track your events ▸ Normalize yours logs ▸ Separate and Centralize your Log Data
- 38. LOGS
- 40. DOCUMENT WRITING EFFECTIVE DOCUMENTATION ▸ Treat documentation like a requirement ▸ Put the Information in the Most Appropriate Place ▸ *Keep your documentation close to the code ▸ Keep documentation just simple enough, but not too simple ▸ Avoid creating large volumes of documentation of questionable detail which become obsolete shortly after they are written
- 41. MEASURE MEASURE PROGRESS ▸ KPIs ▸ Deployment frequency ▸ Deployment speed ▸ Failure rate ▸ Time from code to prod ▸ Share goals ▸ Increase the visibility of work
- 42. COMMUNICATE ▸ Find better ways to communicante ▸ Promote day to day direct communication ▸ Coffee machine meetings ▸ Chat (Slack, HipChat, Team,…) ▸ Use video (Slack, Skype, Hangout,..) ▸ Quality over quantity ▸ Post Mortem meetings ▸ Bootcamps,… IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY
- 43. COLLABORATE TIPS FOR BETTER COLLABORATION ▸ Change the seating and rotate people through development ▸ Improve the environment for collaboration (Breakout areas, whiteboards, nice meeting rooms, a decent coffee machine) ▸ Cross domain workshops for analyze your process ▸ Block time for process improvement work (lean) ▸ Run lunch and learn / brown bag sessions ▸ Put a persistent chat room in place
Editor's Notes
- #4: Spirale NEGATIVE
- #5: DEV: changes into production as quickly as possible OPS: stable, reliable, and secure, making it difficult or even impossible for anyone to introduce production changes that could jeopardize (Problème comptable et objectifs différents entre dev et ops )
- #6: même en tant que ops j’en peut plus d’attendre la création d’un environment
- #9: DevOps ne signifie pas q’un dev peut se passer des ops.Chauqun son metirer ; Collaborer Exemple de spécialisation .
- #11: Quick fix ok , mais il faut ensuite il faut améliorer
- #14: serveur = commodité (EX : prise de courant éléc) STOP Reboot
- #15: SINGLE REPOSITORY : application code , dependencies (e.g., libraries, static content, etc.), infrastructure, deploy, Any script used to create database schemas, application reference data, etc, All the environment creation tools and artifacts described in the previous step (e.g., VMware or AMI images, Puppet or Chef recipes,
- #17: What problems are we trying to solve?
- #18: REDUCE NON-VALUE ADDED : Refaire le Nieme service d’authentification, ou de gestion de fichiers, gestion des logs, IAAS , …
- #20: ON DEMAND EASIER TO REBUILD THAN TO REPAIR
- #23: Pipeline Important (aussi pour les Ops)
- #27: Mise à jour système ( Infra as code) , java, librairies ,… Open Web Application Security Project
- #28: Bcp d’infos dans les logs pour avoir une vue « sécurité »
- #30: Instead of starting deployments at midnight on Friday and spending all weekend working to complete them, deployments occur throughout the business day when everyone is already in the office and without our customers even noticing
- #31: Automatiser! Ansible, kubernetes, simple shell script, git,… Exemple Facebook (dark launch)
- #33: Notifier uniquement si nécessaire Moins de notifications plus d’actions
- #34: Analyser vos métriques
- #36: Si vous voulez aller plus loin et mieux comprendre les interactions
- #37: Vous devez aussi monitorer votre CI/CD
- #38: Ops: partages les logs!!! On perd juste du temps avec la boucle infernale des logs « protégées »
- #39: Utilises plus les logs! tenez compte que vous n’êtes pas les seuls à les lire! Mais c’est difficile d’examiner des fichiers de lois pour les humains donc…
- #40: From TEXT to visual data! Vous pouvez aller plus loin que des fichiers texte
- #42: La documentation c’est important SWAGGER API
- #43: Mesurez pour mieux comprendre ce qui change
- #44: Promote