GDG Cloud Southlake #2 Jez Humble DevOps Transformation:Building & Scaling High Performing Tech Orgs
- 1. Jez Humble Serverless SRE June 30, 2021 The DevOps Transformation: Building and Scaling High Performing Technology Organizations
- 2. ● We can measure software delivery & operational performance, and we know how to get better at it ● A combination of technical, process, management, and product development capabilities drive culture and performance ● Culture can be measured and changed ● Use community structures and transformational leadership to drive organizational change TLDR
- 3. DevOps An organizational and cultural movement that aims to increase software delivery velocity, improve service reliability, and build shared ownership among software stakeholders. cloud.google.com/devops
- 5. Software delivery as a competitive advantage Elite performers are twice as likely to meet or exceed their organizational performance goals: ● Profitability ● Productivity ● Market share ● Number of customers ● Quality of products or services ● Operating efficiency ● Customer satisfaction ● Quantity of products or services provided ● Achieving organizational and mission goals 2019 State of DevOps Report: cloud.google.com/devops
- 6. Deployment frequency Lead time for changes Software Delivery & Operations Performance Time to restore service Change fail rate Speed Stability
- 7. 2019 State of DevOps Report: cloud.google.com/devops
- 8. 2019 State of DevOps Report: cloud.google.com/devops Aspect of Software Delivery Performance Elite High Medium Low Deployment frequency For the primary application or service you work on, how often does your organization deploy code to production or release it to end users? On-demand (multiple deploys per day) Between once per day and once per week Between once per week and once per month Between once per month and once every six months Lead time for changes For the primary application or service you work on, what is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)? Less than one day Between one day and one week Between one week and one month Between one month and six months Time to restore service For the primary application or service you work on, how long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage or service impairment)? Less than one hour Less than one day Less than one day Between one week and one month Change failure rate For the primary application or service you work on, what percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch)? 0-15% 0-15% 0-15% 46-60%
- 9. Firmographics Industry Number of employees
- 11. bit.ly/dora-bfd
- 12. Org structure and culture Teams deliver results, not individuals. How do we build high performing teams and enable them to deliver with speed and stability? Continuous delivery Lean management & product development Mission-oriented culture, psychological safety 1 2 3 Autonomous teams 4
- 13. What is continuous delivery? “The ability to get changes—features, configuration changes, bug fixes, experiments—into production or into the hands of users safely and quickly in a sustainable way.” https://continuousdelivery.com/
- 14. State of DevOps Report: cloud.google.com/devops Continuous delivery Technical practices Trunk-based development Continuous integration Deployment automation Shift left on security Loosely coupled architecture Empowered teams Version control Continuous testing Test data management Monitoring and observability Proactive notifications Database change management Code maintainability Less burnout Continuous delivery Less deployment pain Less rework SDO Performance Software Delivery Performance Availability Organizational Performance Westrum organizational culture
- 15. Build quality in “Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place” — W. Edwards Deming
- 17. Lead time “How long would it take your organization to deploy a change that involves just one single line of code? Do you do this on a repeatable, reliable basis?” Mary and Tom Poppendieck, Implementing Lean Software Development: From Concept to Cash, p59.
- 18. Lead time and TTR in the enterprise When you discover a vulnerability in your stack, how long would it take you to find, patch and redeploy all impacted applications? https://arstechnica.com/information-technology/2017/09/massive- equifax-breach-caused-by-failure-to-patch-two-month-old-bug/
- 19. Low performers take weeks to conduct security reviews and complete the changes identified. Security as a technical practice Building security into software development improves performance and security quality. Elite performers build security in and conduct security reviews and complete changes in just days. Build security in by running security tests as part of the deployment pipeline. InfoSec can make it easy to consume pre-approved libraries, packages, toolchains, and processes.
- 20. Architectural outcomes: can my team... 01 ...make large-scale changes to the design of its system without the permission of somebody outside the team, or depending on other teams? 02 ...complete its work without needing fine-grained communication and coordination with people outside the team? 03 ...deploy and release its product or service on demand, independently of other services the product or service depends upon? 04 ...do most of its testing on demand, without requiring an integrated test environment? 05 ...perform deployments during normal business hours with negligible downtime?
- 21. Cloud is a differentiator Elite performers were 24 times more likely to have met all essential cloud characteristics than low performers*. But only 29% of respondents met all five! 2019 State of DevOps Report: cloud.google.com/devops *Five essential characteristics of cloud computing defined by NIST in Special Publication 800-145 On-demand self-service Broad network access Resource pooling 1 2 3 Rapid elasticity 4 5 Measured service
- 22. 2015, 2016 & 2017 State of DevOps Report: cloud.google.com/devops Lean management & product development Lean Product Development Work in small batches Make flow of work visible Gather & implement customer feedback Team experimentation Software delivery performance Organizational Performance Westrum organizational culture Lean Management Limit work in process (WIP) Visual management Feedback from production Lightweight change approvals Reduced burnout
- 23. Reed Hastings, CEO Netflix https://www.slideshare.net/reed2001/culture-1798664/94-Highly_Aligned_Loosely_Coupled_Highly Autonomous teams in practice
- 24. Culture How organizations process information Pathological (power oriented) Bureaucratic (rule oriented) Generative (performance oriented) Low cooperation Modest cooperation High cooperation Messengers shot Messengers neglected Messengers trained Responsibilities shirked Narrow responsibilities Risks are shared Bridging discouraged Bridging tolerated Bridging encouraged Failure leads to scapegoating Failure leads to justice Failure leads to enquiry Novelty crushed Novelty leads to problems Novelty implemented Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q How do we improve
- 25. Strongly Agree 1. On my team, information is actively sought. 2. Messengers are not punished when they deliver news of failures or other bad news. 3. On my team, responsibilities are shared. 4. On my team, cross-functional collaboration is encouraged and rewarded. 5. On my team, failure causes inquiry. 6. On my team, new ideas are welcomed. Strongly Disagree 7 6 5 4 3 2 1
- 26. https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/ Culture of psychological safety ● Predicts software delivery performance and organizational performance ● Implement by adopting continuous delivery and lean product management practices
- 27. Disaster recovery testing “For DiRT-style events to be successful, an organization first needs to accept system and process failures as a means of learning… We design tests that require engineers from several groups who might not normally work together to interact with each other. That way, should a real large-scale disaster ever strike, these people will already have strong working relationships” —Kripa Krishnan, Director, Cloud Operations, Google Only 40% of respondents perform disaster recovery testing at least annually on production infrastructure —State of DevOps Report 2019
- 28. 2018 & 2019 State of DevOps Report: cloud.google.com/devops SDO Performance Trust Software Delivery Performance Availability Organizational Performance Voice Westrum organizational culture Autonomy Functional outsourcing - Heavyweight change process Clear change process - Growing autonomous teams
- 30. It takes work.
- 31. It takes work. We don’t have to do it all at once.
- 32. It takes work. We don’t have to do it all at once. will probably fail if we try
- 33. Transformation strategies 2019 State of DevOps Report: cloud.google.com/devops
- 34. Elite teams favor strategies that create community structures Communities of practice Grassroots Proof of Concept (POC) as a template POC as seed 2019 State of DevOps Report cloud.google.com/devops
- 35. Communities of Practice at Google “The Testing Grouplet was a team of Google developers who worked together in their 20% time (time provided by Google to allow developers to work on Google-related projects of their choosing aside from their main projects) to address the challenges in promoting unit testing adoption throughout Google. An all-volunteer group with little funding and no direct authority, it relied on persuasion and innovation to convince Google developers of the value of unit testing, and provided them with the tools and knowledge needed to do it well.” — Mike Bland https://martinfowler.com/articles/testing-culture.html#google
- 36. Transformational leadership 2017 State of DevOps Report cloud.google.com/devops
- 37. “Double, half and quarter every year: double the frequency of releases, half the number of low impact incidents, and quarter the number of high impact incidents.” — David Knott, Chief Architect at HSBC https://bit.ly/hsbc-devops HSBC Technology goals