ADDO 2021: Why and how to include database changes in the deployment pipeline
- 1. TRACK: CI/CD CONTINUOUS EVERYTHING NOVEMBER 12, 2020 Eduardo Piairo Why and how to include database changes in the deployment pipeline
- 2. TRACK: CI/CD CONTINUOUS EVERYTHING ● Engineering Operations Director @ Deeper Insights ● Co-Founder and Leader @ DevOps Porto About Me @EdPiairo https://pt.linkedin.com/in/eduardopiairo [email protected] https://www.eduardopiairo.com
- 3. TRACK: CI/CD CONTINUOUS EVERYTHING ● Deployment pipeline ● Database challenges ● Deployment pipeline stages ● Deployment pipeline scenarios ● Final thoughts Agenda
- 4. TRACK: CI/CD CONTINUOUS EVERYTHING A deployment pipeline should enable collaboration between the various groups involved in delivering software and provide everyone visibility about the flow of changes in the system, together with a thorough audit trail. by Martin Fowler @ https://martinfowler.com/bliki/DeploymentPipeline.html Definition
- 5. TRACK: CI/CD CONTINUOUS EVERYTHING ● Stage 1 – Change description (Source Control) ● Stage 2 – Change validation (Continuous Integration) ● Stage 3 – Change implementation (Continuous Delivery) Stages Source Control Continuous Integration Continuous Delivery
- 6. TRACK: CI/CD CONTINUOUS EVERYTHING Database & Deployment Pipeline Source Control Continuous Integration Continuous Delivery Database + Application
- 7. TRACK: CI/CD CONTINUOUS EVERYTHING What’s so special about databases?
- 8. TRACK: CI/CD CONTINUOUS EVERYTHING Database Challenges • Database and application changes (most of the time) are handled differently • Different deployment pipelines - synchronization needed • Database is not included (manual work)
- 9. TRACK: CI/CD CONTINUOUS EVERYTHING Database Challenges • Manual work … • Lack of traceability of database changes (changes history) • It’s the most expensive and risky work type • Prevent CI and CD utilization in their full extent • Promote the fear of changes
- 10. TRACK: CI/CD CONTINUOUS EVERYTHING Databases become a bottleneck in the delivery process
- 11. TRACK: CI/CD CONTINUOUS EVERYTHING
- 12. TRACK: CI/CD CONTINUOUS EVERYTHING The Value Of Automation • Enable control over database development • Increase speed of response to change • Greater reliability, repeatability and consistency • Enables failing in a controlled and known way • Remove/reduce human intervention in the release process
- 13. TRACK: CI/CD CONTINUOUS EVERYTHING Fearless database changes
- 14. TRACK: CI/CD CONTINUOUS EVERYTHING Source Control • Every database change can be described with an SQL script • Traceability through change history • SQL as documentation • Shared code-base and shared process • Enforce standards to reduce conflicts
- 15. TRACK: CI/CD CONTINUOUS EVERYTHING Migrations vs State • State based solutions • How the database should be • Migrations based solutions • How the database should change Migration Migration State Delta
- 16. TRACK: CI/CD CONTINUOUS EVERYTHING Scripting Guideline • One script, one operation type, one object (small batches) • Merge conflicts management • Patterns identification • File system scripts history search
- 17. TRACK: CI/CD CONTINUOUS EVERYTHING Continuous Integration • Integrate and validate changes • Unit and integration tests • Small batches, less risk • Considerations • What should be tested? And when? • How can the application contribute to this stage?
- 18. TRACK: CI/CD CONTINUOUS EVERYTHING Continuous Delivery • Delivering change(s) in the target environment • Considerations • Downtime • Time to recover • Affected applications
- 19. TRACK: CI/CD CONTINUOUS EVERYTHING Failed Deployment • Scripts (roll-back, roll-front) • Very fast • (Please) Do not make mistakes • Working backwards
- 20. TRACK: CI/CD CONTINUOUS EVERYTHING Failed Deployment • Backups (before deploy) • Safer but slower • Keep databases small • Avoid shared databases
- 21. TRACK: CI/CD CONTINUOUS EVERYTHING Scenario #1 Source Control Continuous Integration Continuous Delivery APP DB • Independent deployment pipelines • High need of synchronization between DBA and development team
- 22. TRACK: CI/CD CONTINUOUS EVERYTHING Scenario #2 Source Control Continuous Integration Continuous Delivery APP DB • Different code repositories • Independent CI process • The contact point will be the deployment moment
- 23. TRACK: CI/CD CONTINUOUS EVERYTHING Scenario #3 Source Control Continuous Integration Continuous Delivery APP DB • Different code repositories • Connected/dependent CI process • Short learning step towards scenario #4
- 24. TRACK: CI/CD CONTINUOUS EVERYTHING Scenario #4 Source Control Continuous Integration Continuous Delivery APP DB • Share the some code repository, CI process and CD process • No need of synchronization • Promotes learning within teams (database code reviews)
- 25. TRACK: CI/CD CONTINUOUS EVERYTHING My Definition Pipeline: Cultural and technical tool for managing changes in the software development process (should contemplate databases, applications and infrastructure).
- 26. TRACK: CI/CD CONTINUOUS EVERYTHING DevOps Way • Increase flow visibility • Increase feedback • Increase knowledge base and exploration
- 27. TRACK: CI/CD CONTINUOUS EVERYTHING Q&A @EdPiairo https://pt.linkedin.com/in/eduardopiairo [email protected] http://www.eduardopiairo.com/ #2021-ci-cd