Better Governance Banking on Continuous Delivery
4 likesâ˘788 views
This document discusses how Capital One has transitioned from a traditional waterfall development model to a continuous delivery model over the past 5 years. It outlines Capital One's journey from outsourcing development to bringing it in-house, moving from vertical silos to product teams, and embracing practices like automated testing, deployment from public clouds, and open source. The document also discusses how Capital One ensures governance and compliance as it continuously delivers new features and changes through practices like a "software delivery clean room" where all changes are peer reviewed and tracked. It also discusses how the developer role has changed to include awareness of risks and following control best practices to ensure safety in continuous delivery.
Better Governance Banking on Continuous Delivery
- 1. Better Governance Banking on Continuous Delivery
- 2. Tapabrata âTopoâ Pal Sr. Director & Sr. Engineering Fellow [email protected] @TopoPal Jennifer Brady Director, Technology Governance [email protected] ! Former Audit Director ! Current IT Governance Director ! Responsible for both a control automation and data analytics team ! Work with Data Scientists, Data Engineers, and Developers ! Developer ! DevOps Evangelist ! Product Manager of Shared Continuous Delivery Tools Platform ! Creator and core contributor of Hygieia DevOps Dashboard
- 3. Capital One ! Millions of accounts ! One of the largest Digital Banks ! #1 Information Weekâs Elite 100 ! ~ 20 years old
- 4. Different DNA ! Build our own software ! Build on public cloud ! MicroServices ! Open Source ! Continuous Delivery
- 7. ⢠Waterfall ⢠Manual Build ⢠Manual Deployment ⢠Manual Test ⢠Data Center ⢠Closed Source First ⢠Agile ⢠Automated Build ⢠Automated Deployment ⢠Automated Test ⢠Public Cloud ⢠Open Source First 5 Year Journey
- 8. Mostly Out-Sourced Mostly In-Sourced Vertical Silos Product Team Dev, Ops, QA, RM Engineers 5 Year Journey
- 9. ! DOES 2014 Building out Automation steps ! DOES 2015 Scaling DevOps, Open Source, Cloud, Innovation ! DOES 2016 Measure, Improve, Mature
- 10. 2017 and beyond ! #SlayTheMonolith ! #NoFearRelease ! #YouBuildItYouOwnIt
- 11. #YouBuildItYouOwnIt ! YOU Coded It, YOU Build It ! YOU Built It, YOU Test It ! YOU Tested It, You Deploy It ! YOU Deployed It, YOU Own It
- 12. #NoFearRelease ! Fear of speed ! Fear of breakdown ! Fear of being out of control ! Fear of being non-compliant
- 13. We want thisâŚ
- 14. And not thisâŚ
- 15. Safety in Continuous Delivery
- 16. Former Auditorâs Perspective Welcome to the Wild West Image Credit http://www.freepik.com Designed by Freepik
- 17. And at Capital One⌠Image Credit: http://dkcoin8.com
- 18. Compliance
- 20. Compliance vs Governance Compliance = Checking the box Governance = Awareness of and active management of risk
- 21. Three Lines of Defense ! 1st Line : Who Owns the Risk ! 2nd Line: Sets Policy, Monitors the Risk ! 3rd Line: Independent Assurance
- 22. What is the Developerâs Role in Governance? ! Awareness ! Risk mitigation ! Follow control best practices
- 23. Why Controls? ! Controls are there to protect you and the company ! Provide assurance around ďŹnancial reporting ! Provide comfort to investors
- 24. "Uncontrolled variation is the enemy of qualityâ
- 25. Minimum Set of Controls ! Two Sets of Eyes ! Least Privilege ! Unauthorized Change Monitoring
- 26. Automation is easy, almost, such as⌠! Build on every commit ! Static code analysis on every build ! Scanning for open source vulnerability ! Static security scan ! Automated tests ! âŚ.
- 27. Biggest hurdle Ensure that a single developer can not make changes to production bypassing all controls
- 28. Options ! Separate team managing pipeline ! Separate team just to perform production deployment ! Hire professional âbutton pushersâ
- 29. Assumptions ! Enough âbutton pushersâ available ! They cannot code ! Cannot train them to do anything else ! But, they should know if it is okay to push the button
- 31. âthe secrets of change is to focus all your energy not on ďŹghting the old but on building the newâ
- 32. Clean Room A clean room or cleanroom is an environment, typically used in manufacturing, including of pharmaceutical products or scientiďŹc research, as well as semiconductor engineering applications with a lower level of environmental pollution such as dusts, airborne microbes, aerosol particles and chemical vapors. https://en.wikipedia.org/wiki/Cleanroom
- 33. Software Delivery Clean Room ! All product pipelines are identiďŹed and registered ! Everything is under source control ! Every change is peer-reviewed ! Production Changes occur only via code changes ! Nobody has access to production servers ! Every code change goes through various levels of testing and scanning ! Pipeline stops or alerts if things fail ! Evidences captured and evaluated at near real time ! Evidences are analyzed for discrepancies
- 34. DEVELOPMENT TEST MONITORINGIMPLEMENTATION App Code source controlled enforcing peer review and disallowing direct commits 1.1 1.2 2.1 No direct Access to the Binary Artifact 3.1 Static Code Analysis 4.1 Static Code Analysis Config reviewed 4.2 Functional Tests (Traceable to Story) 4.8 4.3 Test Automation jobs configuration peer reviewed and source controlled 4.4 Critical Business Transaction (Regression) Testing 4.5 Static Application Security Testing Infrastructure Code Security Scan Open Source Security Testing Secret Key Management Must not use production raw data for testing Prod/Non-Prod Deployments Scripts Tested 6.1 Prod Deployment and Test Results (UAT & Exploratory) approved by PO 6.3 6.2 Automated rollback process Tested in Prod/Non-prod No Connectivity b/w PROD and Non-PROD Environments 6.4 Separation between Dev/Test/Prod environments must be managed with access controls Developers will not have write access in Prod 7.1 RELEASE All Changes to pipeline code (build /deploy job scripts) must be recorded and peer reviewed Infrastructure Code source controlled enforcing peer review and disallowing direct commits Test Code source controlled enforcing peer review and disallowing direct commits Performance testing 5.1 5.2 5.45.3 4.7 4.1 Build will be successful only if it passes static code analysis 4.2 Static Code analysis configuration is reviewed by the team and approved by PO 3.1 No direct access to the binary artifact 4.3 Every testable story must have corresponding test case/scenario/ step 4.4 Test Code source controlled enforcing peer review and disallowing direct commits 4.5 Test Automation jobs configuration peer reviewed and source controlled 4.6 A core set of test cases that are considered critical (e.g regression testing) must meet the passing threshold defined by the team lead 4.8 Must not use production raw data for testing 4.7 Performance test results must meet the passing threshold defined by the team lead 5.4 Secret key Management 5.1 Static Security scan/testing results are reviewed and approved by the PO or Tech lead for App Code 6.1 Prod/Non-prod Deployment scripts/configurations tested by the team 7.1 Developers will not have write access in PROD 5.2 Security scan/testing results are reviewed and approved by the PO or Test lead for Infrastructure code 5.3 Open Source Security Testing (Build Artifact has ONLY approved libraries) 6.2 Prod Deployments and Test Results (UAT/Exploratory) must be approved by PO/Tech lead 6.3 Automated roll back process for production deployment must be tested in Prod and Non-prod 6.4 Must not have any connectivity or access between Prod and Non-Prod Environment 7.2 Separation between dev/test and prod environments enforced with access controls Source Control 1 Binary Repository and Application versioning 3 Security Checks 5 Quality Checks 4 Build 2 Deployment 6 Support 7 1.1 Application Code source controlled enforcing no direct commits to master/release without peer review 1.1 Infrastructure Code Source Controlled enforcing no direct commits to master/release without peer review 2.1 All Changes to pipeline code (build /deploy job scripts) must be recorded and peer reviewed 5.5 App Dynamic Security Scan/testing results are reviewed and approved by the PO/Tech lead 5.5 4.6 App Dynamic Security Testing 7.2 Software Delivery Clean Room
- 35. @TopoPal Result 2016 2017 # Products deploying multiple times a day ~20 ~300 Average #deployments per day ~1 ~4 Max #deployments for a product in a single day ~30 ~50
- 36. Automating Clean Room Monitoring Audit API https://github.com/capitalone/Hygieia/tree/master/api-audit
- 37. Are you well managed if you are doing Continuous Delivery?
- 38. Are you well managed if you are not doing Continuous Delivery?
- 39. Thank You!