The Future of Database Development
Download as PPTX, PDF0 likes41 views
The document discusses how containers can help with database development. Containers provide a consistent and isolated environment for applications and databases. They allow developers to easily branch and experiment without affecting other work. Containers also enable continuous integration by providing a known starting state for testing database applications. Developers can quickly reset containers to fix mistakes. Overall, containers streamline database development tasks and reduce infrastructure dependencies.
The Future of Database Development
- 1. The Future of Database Development (with containers) Steve Jones Editor, SQLServerCentral Redgate Software
- 2. Agenda Who am I? Getting Started on a Database Project Container Primer Branching for Experiments Fixing Mistakes Easy Continuous Integration
- 3. Steve Jones DevOps Advocate, Redgate Software Editor, SQLServerCentral 30 years SQL Server data experience DBA, developer, manager, writer, speaker in a variety of companies and industries Founder, SQLServerCentral Currently the editor in chief, with the goal of helping you learn to be a better data professional every day 13 year Microsoft Data Platform MVP I have been honored to be recognized by Microsoft for the as a Data Platform MVP working with SQL Server @way0utwest /in/way0utwest www.voiceofthedba.com [email protected]
- 4. Software Dev is Easy • Git clone • Start working
- 6. The Database Vision - Getting Started • I got a new gig… • They're letting me do web development!!!! • I don’t want to worry about databases • I don’t have SQL installed (No services running) • I have cloned the source code (git clone)
- 7. SQL Server Shared Account database PostgreSQL Todo database .NET Core API Frontend web
- 8. I get a login to Spawn (DBaaS)
- 10. The Database Vision – Minimal Work • Git clone xxxx (code) • Spawnctl auth (db service auth) • Yarn xxxx (start the app)
- 11. What are containers? Or how are these different from virtual machines?
- 12. Hardware Operating System Hypervisor VM VM Operating System Operating System Operating System VM App App App Hardware Operating System Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Container Image Docker runtime
- 13. What is a container? • Started in Linux as a lightweight way to provide a full fledged Linux environment without starting a full VM • LXC were early Linux containers • Docker built their own container engine • Uses the host kernel • Provides a separated environment for applications • Virtualizes OS resources • Namespaces • Control groups
- 14. What are Images? • In the container world, an image is a template of the filesystem • We build these with layers • A unified file system combines these layers to give a consistent view • The layers are read only • Images are downloaded and used to create an instance of a container
- 15. Images and Containers • The container is an instance of the Image • All containers share the image • Each container gets a separate writeable layer • Changes exist in this writeable layer • The unified file system keeps a consistent view • Changes made to the writeable layer persist through pause/stop • A volume may be mounted from the host inside the container
- 17. Containers Help with Getting Started • Containers are self-contained • Everything inside • Monolithic, or multiple containers linked • Remove the need for dependencies • Remove (some) of the need for infrastructure • Known versions ensure stability/repeatability • For databases • Smaller data sets work well inside containers • Larger data sets are a problem
- 18. Containers help with Branching • A typical concern is I want to try something • However, I don’t want to disturb other’s work • Branches allow me to experiment and commit separately • I can discard these, recreate these as needed, which means I want the database to do the same. • I can merge in changes that work
- 19. Containers and Branching • Containers are immutable • Restarting a container means that I return to a known state • I can branch, with a new container and make changes • Once work is done, I can commit the code to VCS
- 20. Demo Branching - Checkout new branch - Getting a new data container
- 21. Oops, I did it again • Often, I make some mistakes in development • Usually, I can’t remember or don’t know the code to reset the database • This is frustrating because app code just resets
- 22. Demo Reset Delete/change data Spawnctl reset data-container
- 23. Recap • DBaaS removes the need to manage the server component • Offloads resources elsewhere • Offers more features • Containers provide a known, consistent starting point • Integration with easy deployment as a part of app config • Reset can speed up testing of data issues
- 24. Continuous Integration Independent Verification and Validation
- 25. Independent Validation with CI • We want to check changes (committed) by developers • Avoid the "it works on my machine" • The CI process should be simple and seamless
- 26. CI with Containers • Known state for a database app • Schema • (test) data • Start quickly • Multiple containers for different data stores can be run in parallel • Lighter weight than VMs
- 28. Challenges • DBaaS • Subsetting • We need easy ways to get a partial set of data • Automation and adaptation to schema changes • Test Data • Size (subsetting) • Scale • Spread • Container Orchestration (AKS et al)
- 29. Summary • DBaaS is the future • Containers make development easy(-ier) • Handle branching easily for experimentation • Reset immediately • Parallelize testing with a database • Lightweight and reduce setup • DBAs still have control
Editor's Notes
- #2: DevOps is a hot topic in today’s software development world. However most of the knowledge and experience with DevOps is based around application software and ignores the database. We will examine how the concepts and principles of DevOps can be applied to database development by looking at both automated comparison analysis as well as migration script management. Automated building, testing, and deployment of database changes will be shown. https://unsplash.com/photos/IMUwe-p1yqs
- #4: steve
- #10: This is a recording of a demo system on my laptop. Notice at the beginning, I don't have any SQL Servers running. In fact, I have no data stores, including PostgreSQL, which this app uses. I have downloaded a repo prior to this, but I can use the yarn package manager to start my application. Once I do that, the yarn configuration starts to look for containers, doesn't find them and creates them. Notice there are MSSQL, SQL Server, images and PostgreSQL images, These are used to create new containers. This does take a little time, as this is a proof of concept, running in the UK, and not a lot of resources are allocated. Once up, notice that credentials and connection strings are returned. These are plain text, but they are also generated for this container, for this user. Once we have containers created, there is an upgrade, as migrations, or new SQL Scripts in the repository, are applied to the databases. Flyway is used to check the scripts and apply them to the schemas in the databases. Both SQL Server and PostgreSQL databases are migrated. As migrations are finished, the app then is started, and a URL returned. I can then switch to a browser and see the app running. I can connect, and begin working. Each time I start the app, the container status is checked, and if they are available, they are reused. If I need to reset my data store for some reason, say I removed or changed data, I can just remove a container with a command line call and a new one will be recreated the next time the app starts. If I add new SQL code for objects in a file, the new migration script is executed the next time the app starts up. If I branch, I get a new set of containers. If I switch back to my old branch, the old containers are there. Each one starting from the same, immutable image. At the end, we see this is from the Spawn project, from Redgate, which is a beta proof of concept for doing database development with the Database as a Service.
- #13: https://hub.docker.com/_/microsoft-mssql-server
- #18: https://hub.docker.com/_/microsoft-mssql-server
- #29: https://foundry-labs.visualstudio.com/spawn-demo-app/_build/results?buildId=17516&view=logs&s=6884a131-87da-5381-61f3-d7acc3b91d76