Keeping Your DevOps Transformation From Crushing Your Ops Capacity
4 likes1,695 views
The document discusses the challenges faced by operations teams in a landscape driven by DevOps and digital transformation, resulting in what's termed 'the operations squeeze.' It emphasizes the need to replace silos with cross-functional teams and adopt an 'operations as a service' model to improve efficiency and responsiveness while managing security and stability. The document argues that failing to address these issues could lead to competitive disadvantages for businesses but also presents opportunities for operational reimagination.
Keeping Your DevOps Transformation From Crushing Your Ops Capacity
- 1. Keeping your DevOps Transformation from Crushing Your Ops Capacity Damon Edwards @damonedwards
- 2. Damon Edwards Ops Improvement DevOps Consulting Ops Tools Community
- 3. This talk is about what happens after deployment Deploy Everything Else What does Ops do with it’s time?
- 4. Let’s start with an (unfortunately) typical incident… Download the PDF here: https://rundeck.co/incident_does2017
- 5. And people still wonder where the time goes. DevOps, Containers, Continuous Deployment.. Why aren’t we getting more done? Executive Team
- 6. It’s only getting worse The Operations Squeeze Introduction “The Operations Squeeze” Go faster! Be flexible! Lock it down! Improved Quality Shorter Time-to-Market Fast Feedback From Users Availability Auditing Security Compliance Dev Ops Ops “The Ops Squeeze”
- 7. It’s only getting worse The Operations Squeeze Introduction “The Operations Squeeze” Go faster! Be flexible! Lock it down! Improved Quality Shorter Time-to-Market Fast Feedback From Users Availability Auditing Security Compliance Dev Ops Ops More errors More delays Less capacity Less flexibility “The Ops Squeeze”
- 8. Use a “Lean” lens to analyze operations activity
- 9. Lean: Empower those closest to the issue escalate escalate 1° 2° 3° escalate 4°
- 10. Lean: Empower those closest to the issue escalate escalate 1° 2° 3° escalate 4° Push the ability to take action this direction
- 11. Lean: Empower those closest to the issue escalate escalate 1° 2° 3° escalate 4° Push the ability to take action this direction But what gets in the way?
- 12. Lean: Empower those closest to the issue escalate escalate 1° 2° 3° escalate 4° Push the ability to take action this direction SilosBut what gets in the way?
- 13. Backlog Context I need X Backlog I do X Requests for X Silo A Priorities Context Priorities Silo B Tools Tools Silos ruin everything
- 14. Backlog Context I need X Backlog I do X Requests for X Silo A Priorities Context Priorities Silo B Tools Tools Silos ruin everything
- 15. How to spot silos? Look for Ticket-Driven Request Queues Team A (Dev) Team B (Ops) Ticket System ??
- 16. Team A (Dev) Team B (Ops) Ticket System ?? Silo Builder How to spot silos? Look for Ticket-Driven Request Queues
- 17. Team A (Dev) Team B (Ops) Ticket System ?? Silo Builder Snowflake Maker How to spot silos? Look for Ticket-Driven Request Queues
- 18. Request queues proven to have a huge economic impact
- 19. All of those little delays end up being really expensive RevenueperWeek Time COST OF DELAY Actual Revenue Market: Long Lifetime Value, Peak Unaffected by Delay Opportunity Ready
- 20. So lets get rid of silos & request queues!
- 21. Popular: Replace Silos with Cross Functional Team Dev/Test Release OperatePlanning
- 22. Popular: Replace Silos with Cross Functional Team Dev/Test Release OperatePlanning Cross-Functional Teams Cross-Functional Teams Cross-Functional Teams
- 23. Dev/Test Release OperatePlanning Popular: Replace Silos with Cross Functional Team Cross-Functional Teams Cross-Functional Teams Cross-Functional Teams
- 24. EnvironmentsDBAs Network Security NOC Dev/Test Release OperatePlanning Popular: Replace Silos with Cross Functional Team Cross-Functional Teams Cross-Functional Teams Cross-Functional Teams
- 25. EnvironmentsDBAs Network Security NOC Dev/Test Release OperatePlanning Team A (Dev) Team B (Ops) Ticket System ?? Popular: Replace Silos with Cross Functional Team Cross-Functional Teams Cross-Functional Teams Cross-Functional Teams
- 26. Team A (Dev) Team B (Ops) Ticket System ?? Silo Builder Snowflake Maker Get rid of the remaining ticket-driven request queues…
- 27. … by replacing with Operations as a Service design pattern Team A (Dev) Team B (Ops) Ticket System Operations as a Service Actual Exceptions On Demand On Demand
- 28. … by replacing with Operations as a Service design pattern Team A (Dev) Team B (Ops) Ticket System Operations as a Service Actual Exceptions On Demand On Demand
- 29. Change how your organization thinks about automated procedures…
- 30. Automated procedures are comprised of three parts Definition of the automated procedure Execution of the automated procedure Governance of the automated procedure Define Execute Govern
- 31. Automated procedures are comprised of three parts Definition of the automated procedure Execution of the automated procedure Governance of the automated procedure Define Execute Govern (security, oversight, compliance, etc.)
- 32. Traditional Ops Silo Define Execute Govern “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops
- 33. Rigid Self-Service Define Execute Govern “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops
- 34. Define Execute Govern Execute “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops Rigid Self-Service (limited)
- 35. High-Velocity Handoffs Define Govern Execute “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops
- 36. Self-Service Operations Define Govern Execute “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops
- 37. Self-Service Operations Define Govern Execute Govern “Consumers of Ops” (Dev, QA, Release, NOC, Security, etc.) Ops
- 38. fdfd Operations as a Service Operations as a Service ED G Team B (Ops) Vet Procedures Define Policies Execute On Demand Team A (Dev) Define Procedures Execute On Demand
- 39. fdfd Operations as a Service Move definition, execution, and governance to where you get the most effective use of labor and best flow of work Operations as a Service ED G Team B (Ops) Vet Procedures Define Policies Execute On Demand Team A (Dev) Define Procedures Execute On Demand
- 40. fdfd Lean: Standardize to eliminate waste and enable improvement Operations as a Service ED G Team B (Ops) Vet Procedures Define Policies Execute On Demand Team A (Dev) Define Procedures Execute On Demand Collaborate to create standard operating procedures
- 41. Example Operations as a Service Platform #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform
- 42. Example Operations as a Service Platform #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform
- 43. Example Operations as a Service Platform #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform
- 44. Example Operations as a Service Platform #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform
- 45. Example Operations as a Service Platform #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform
- 46. Let’s look at some companies who are leveraging this design pattern.
- 50. 90% Reduction in MTTR 50% Reduction in escalations 55% Reduction of overall support costs
- 52. 1. New org, support, and escalation model
- 53. escalate 1° 2° 3° 4° escalate escalate 1. New org, support, and escalation model
- 54. escalate 1° 2° 3° 4° escalate escalate 1. New org, support, and escalation model EMT ER Trauma Surgeon Specialist Surgeon TOC (NOC) SRE Production Eng. Scrum Teams Data Services Platform Eng. Global Network > 15 min > 30 min > 60 min
- 55. 2. Key: Push the ability to take action closest to the problem escalate 1° 2° 3° 4° escalate escalate 1. New org, support, and escalation model EMT ER Trauma Surgeon Specialist Surgeon TOC (NOC) SRE Production Eng. Scrum Teams Data Services Platform Eng. Global Network > 15 min > 30 min > 60 min
- 56. 2. Key: Push the ability to take action closest to the problem escalate 1° 2° 3° 4° escalate escalate 1. New org, support, and escalation model EMT ER Trauma Surgeon Specialist Surgeon TOC (NOC) SRE Production Eng. Scrum Teams Data Services Platform Eng. Global Network > 15 min > 30 min > 60 min 3. Longterm investment in operability (deployment, configuration, monitoring, automated runbooks)
- 57. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html
- 58. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html • Automated Ops procedures written/ vetted by the delivery teams
- 59. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy
- 60. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks
- 61. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations
- 62. “Support at the Edge” Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ http://rundeck.org/stories/mark_maun.html • Automated Ops procedures written/ vetted by the delivery teams • Ops remained in full control of what can run and security policy • Empowered NOC and other support teams with self-service ops tasks • Empowered developers with limited self-service operations • Combined with new incident response and escalation model
- 63. Fortune 100 Manufacturing & Services: “Force Multiplier” Team A Services Scripts/Tools ENV A Services Scripts/Tools ENV B Services Scripts/Tools ENV C Team B Team C Operations as a Service Platform Ops Team n… Services Scripts/Tools ENV n
- 64. Fortune 100 Manufacturing & Services: “Force Multiplier” Team A Services Scripts/Tools ENV A Services Scripts/Tools ENV B Services Scripts/Tools ENV C Team B Team C Operations as a Service Platform Ops Team n… Services Scripts/Tools ENV n • Manufacturing and service conglomerate with need to decouple IT
- 65. Fortune 100 Manufacturing & Services: “Force Multiplier” Team A Services Scripts/Tools ENV A Services Scripts/Tools ENV B Services Scripts/Tools ENV C Team B Team C Operations as a Service Platform Ops Team n… Services Scripts/Tools ENV n • Manufacturing and service conglomerate with need to decouple IT • Each line of business has different security and policy needs
- 66. Fortune 100 Manufacturing & Services: “Force Multiplier” Team A Services Scripts/Tools ENV A Services Scripts/Tools ENV B Services Scripts/Tools ENV C Team B Team C Operations as a Service Platform Ops Team n… Services Scripts/Tools ENV n • Manufacturing and service conglomerate with need to decouple IT • Each line of business has different security and policy needs • Handle 5-10x the scale without adding to central Ops organization
- 67. Recap Capacity crunch is only going to get worse Make explicit investment in process and tooling Operations as a Service: Reshaping IT Operations to Solve Today’s Challenges 4 D evOps and Digital Transformations are driving an unprecedented increase in the pace and volume of daily change. Who generally finds this to be welcome news? Development and Product teams. Who has reasons to be alarmed at the problems and challenges this might bring? Operations. Operations organizations in today’s enterprises are finding themselves squeezed between two unrelenting forces. On one side there are the business-driven demands of DevOps and Digital Transformation (“Go faster! Open things up!). On the other side there are the demands to maximize security and stability (“Don’t be the next hack! Don’t be the next outage! Lock things down!”). And there, in the middle, is an already over-burdened Operations organization doing their best to avoid being squeezed beyond the breaking point. Operations has reached an inflection point. To deliver what the business demands, Operations must find a way to provide increasing levels of organizational responsiveness and throughput — all while “locking things down” to sufficiently meet today’s risk profiles. A lot is riding on how Operations responds to this challenge. A failure here is not just a localized IT failure. A failure will undermine a business’s ability to operate. Failing to solve this will turn into a competitive disadvantage for the business. On the flip side, this challenge also presents a great opportunity. Operations can take this business mandate and use it to reimagine how both planned and unplanned work is handled. This is a chance to improve how Operations both serves the broader business and improves the day-to-day lives of Operations professionals. The Operations Squeeze Introduction “The Operations Squeeze” Go faster! Be flexible! Lock it down! Improved Quality Shorter Time-to-Market Fast Feedback From Users Availability Auditing Security Compliance Dev Ops Ops Use a “Lean lens” to analyze Ops activity Team A (Dev) Team B (Ops) Ticket System ?? Beware of ticket-driven request queues Leverage the Operations as a Service design pattern #! ! "# $ Scripts APIs Tools Cloud VMs Containers Orchestration Collect and Process Output Infrastructure details Config. Man. CMDB Monitor. Metrics Cloud Corp Directory Authentication ITSM Tickets >_ ● Collaborate on Standard Procedures ● Manage Access & Governance policies ● Execute Actions & Share Visibility Web GUI API CLI Operations as a Service Platform “Shift-Left” control and decision making.