DevOps 101 for data professionals

Editor's Notes

• #2: Play this during intro (2 mins): https://www.youtube.com/watch?v=IW9K4PwrsBg
• #3: This is Alex, he writes software to help you achieve DevOps for databases. This is Alex,
• #5: Show of hands… Who is a developer? Project manager? Do we have any DBAs? DBAs care about… NEXT SLIDE
• #6: DBAs care about this stuff. Two buisness entities with different goals. It causes friction. NEXT SLIDE
• #7: Sometimes OPS fail
• #8: Sometimes OPS fail
• #9: Sometimes OPS fail
• #10: Sometimes OPS fail
• #11: Sometimes OPS fail
• #12: Sometimes OPS fail
• #13: Sometimes OPS fail
• #14: Velocity: John Allspaw Paul Hammond
• #15: Andrew Clay Shafer Patrick DeBois John Allspaw Paul Hammond Gene Kim
• #18: What about databases? The persistent data the our businesses our built on? Schema changes vs existing data There is an odd relationship here between how you want the code to look and how you update prod. You can’t simply drop and replace the DB because you’ll lose your data. You have to create some sort of upgrade script. Twice as much code normally means twice as many errors and if the state you want and your way to get there don’t match, which is right? What is your source of truth? Without a clear idea of your source of truth – how is DevOps even possible? Reference data vs production data But while the database has existing data, not all of it is ‘production’ data in that sense. What about country codes? Reference data, lookup data, static data, the data that makes your system work. This data needs to get deployed with your schema changes. And you also need to think about how to transfer data in the other direction for testing. How do you test the latest dev build with production (or-production-like) data? Teamwork and testing The word DevOps refers to the problems associated with siloed Dev and Ops teams. Nowhere is this more apparent than in the land of the database? Is there anyone here who has never heard of any problems between Dev and DBA teams? But it goes further than that. With application source code we use source control, we invented distributed source control systems and we debate about the optimal branching strategies and strategies for implementing continuous integration. We’ve barely begun to have these conversations about databases. We barely have strategies for how to provision individual developers with their own sandboxes. Sometimes different developers work in different ways, some working off scripts and others working directly on the database. We need a better way of working together. And finally testing. Our problems with database source control and acquiring suitable test data make it hard for us to provision test database environments manually, let alone automatically. Database drift And all these problems result in this final point. In DevOps and Continuous Delivery we often talk about cycle time. How long would it take you to make a one line change, run it through your normal testing process and get it to production? If the cycle time for your database is measured in days, weeks or months (or years?) then when you hit a production issue you don’t have time to go back to your source code. The business is haemorrhaging money and the DBA will make a decision: What is more expensive, the cost of delaying the fix, or the risk of making that low risk change now and fixing the problem right away. It’s all well and good wagging your finger at people who choose to make hot-fixes directly on production, but until we sort out the cycle time problem, it won’t go away. Production drift is a symptom of poor DevOps or DLM strategy. And drift causes more problems: Environment inconsistencies undermine your tests and can cause failed deployments either because code clashes or because important fixes are accidentally rolled back. Drift and poor DevOps processes are a vicious circle that needs to be broken.
• #19: Start with Lean Empower the team Focus on the customer, eliminate waste Continuously Improve Optimize the whole Plan for change Automate processes Build quality in
• #20: Schema changes vs existing data Reference data vs production data Teamwork and testing Database drift (change outside process, e.g. production hot-fixes)
• #21: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #22: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #23: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #24: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #25: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #26: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #27: Essentially, here is the challenge: get from one state to another, using some upgrade script. NEXT SLIDE…
• #28: Essentially, here is the challenge: get from one state to another, using some upgrade script. NEXT SLIDE…
• #29: Essentially, here is the challenge: get from one state to another, using some upgrade script. NEXT SLIDE…
• #30: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #31: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #32: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #33: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #34: For example, Paul Stovell (creator of Octopus Deploy) basically says the migrations approach is reliable and provides the functionality to handle the deployment whatever way works.
• #35: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #36: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #37: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #38: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #39: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #40: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #41: So what is better? Well, this is my attempt to summarise. The answer is neither is perfect. Generally, if the stuff on the left sounds more like your database, you’ll probably find a state based solution works better for you and visa verca. However, most people don’t live entirely on one side or the other. Let’s look at the tools available.
• #42: I’m here to say neither solution is perfect. I’m going to explain it with an example. Let’s say my team has two tickets/work items. Each is picked up by a different developer.
• #44: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #45: We’ve all done DevOps right. It looks like this. VCS, CI, RM. We are versioning, packaging and deploying our webapps. Maybe we’ve gone serverless? Everything is in source control and all our infrastructure is disposable. Right?
• #48: If no questions, Alex Y to kick of some Q+A by asking Alex T the following questions: - We talked about conflict resolution in programmable objects, but what about conflict resolution in migration scripts? I loved the demo, but demos are built on demo software and designed to make it all look so easy. What sort of problems am I going to run into in the real world? DevOps is about automation, and tools like ReadyRoll can help. But it’s also about Culture, Lean, Measurements and Sharing. Can you comment on how ReadyRoll supports DevOps in the wider sense?
• #49: This is Alex, he writes software to help you achieve DevOps for databases. This is Alex,
• #50: This is Alex, he writes software to help you achieve DevOps for databases. This is Alex,