Security Implications for a DevOps Transformation

Editor's Notes

• #2: Introduce yourself
• #3: When you’re learning something new, you don’t dive in at the end – you start at the beginning. So let’s take that approach to familiarizing ourselves with DevOps. We’ll take the case of a new startup, and build to a DevOps toolchain as we encounter – and solve – the common problems in a software development process.
• #6: This is by design
• #7: We need a more concrete frame of reference in order to assess operational impacts of a DevOps transformation.
• #8: Peopleoften talk about a DevOps Toolchain, and present it like this. There’s an awful lot going on here, and this can only overwhelm people who don’t have a clear conception of DevOps. Fundamentally, DevOps can be best understood as the latest step in a series of improvements to the way that IT organizations write software and deliver it to servers. By placing DevOps on a continuum in this way, we can give it context and define a company’s “DevOps transformation” in terms of the larger technological transformations that have led us to DevOps. Let’s strip away all this complexity now and get back to basics.
• #10: Okay, so how did we get here?
• #11: When you’re learning something new, you don’t dive in at the end – you start at the beginning. So let’s take that approach to familiarizing ourselves with DevOps. We’ll take the case of a new startup, and build to a DevOps toolchain as we encounter – and solve – the common problems in a software development process.
• #12: When we’re just getting started, building our application is pretty easy.
• #13: When we’re just getting started, building our application is pretty easy.
• #14: It’s almost not worth calling it a workflow at this point. I’m just logging on to my single server and writing the webpage live. Easy-peasy.
• #15: Inevitably, when we go live we discover complications. In this case, we can’t very well say “Hello, world!” to the world if we only say it in English. Looks like we need to do some more work. But I can’t keep up with this. I’m not a language expert and I don’t have the time to go around looking up translations while doing the business of running my incredibly popular website. Let’s hire some developers.
• #16: So we hire some developers to handle the increasing demand for features.
• #17: Now that we have two people, our initial approach is already starting to break down. We need a more effective way to collaborate so we aren’t endlessly stepping on each other’s toes or spending lots of time trying to figure out what’s changed and who did what. It’s a pain for people to keep track of all this
• #18: This is where a good version control system excels. It facilitates collaboration by handling all of the bookkeeping (change tracking, versioning, conflict resolution), and freeing the developers to focus more on writing code.
• #19: This is still mostly a manual process, and has some weaknesses. There’s no validation that I’m checking out the right thing, or putting it in the right place with the right permissions. I can’t manage dependencies. I need knowledge of the underlying VCS structure in order to manage versions. Some orgs have restrictions against having dev tools (such as VCS clients) on production systems. Ask: How can we solve this one?
• #20: With a version control server in place, we can make our first revision to the workflow. Now, instead of making changes directly on the production server, we commit our changes to the version control server and then check out the new files on the production server.
• #21: This is still mostly a manual process, and has some weaknesses. There’s no validation that I’m checking out the right thing, or putting it in the right place with the right permissions. I can’t manage dependencies. I need knowledge of the underlying VCS structure in order to manage versions. Some orgs have restrictions against having dev tools (such as VCS clients) on production systems. Ask: How can we solve this one?
• #22: We can overcome most of these limitations by introducing package management. Instead of copying files directly to our destination server, we can bundle them into a single, atomic package that contains metatdata (version info, installation location, dependencies) and provides facilities for automated installation and uninstallation activities.
• #23: This is still mostly a manual process, and has some weaknesses. There’s no validation that I’m checking out the right thing, or putting it in the right place with the right permissions. I can’t manage dependencies. I need knowledge of the underlying VCS structure in order to manage versions. Some orgs have restrictions against having dev tools (such as VCS clients) on production systems. Ask: How can we solve this one?
• #24: Now we revise our workflow further. After checking the latest version of code, we build a package that contains all of the files. We then deploy that single package to the server, rather than copying a lot of files. This results in deployments that are faster, more reproducible, and more reliable than was previously possible.
• #25: ThinWe now have effective collaboaration and easy deployments. We can get our multi-lingual greeting site out to the people. Let’s do it.
• #26: Whoops. Automation without validation just gets bugs to production faster.
• #27: The most straightforward way to validate our code’s quality is to set up a test server, and deploy to it first before we deploy to production. That allows us to validate the test application before it hits production.
• #28: This helps, but our test process has the same weaknesses that all manual processes share. They are slow and unreliable. People aren’t as good at repetitive, systematic tasks like validation as machines are. This testing also happens late. By the time we run the first test, we’ve already committed the code to version control, built a package, and deployed it to one environment. That’s a lot of work when we have no idea whether the code is any good. It would be better to catch the problems earlier.
• #29: If we can come up with a way to allow the computer to do our tests, then we gain all the benefits of automation. Because of this, people have written a variety of automated testing frameworks that do just that. Once we’ve automated our tests, it’s easier to apply them earlier (and more frequently) in the lifecycle.
• #30: If we can come up with a way to allow the computer to do our tests, then we gain all the benefits of automation. Because of this, people have written a variety of automated testing frameworks that do just that. Once we’ve automated our tests, it’s easier to apply them earlier (and more frequently) in the lifecycle.
• #31: So now we revise the workflow again. Rather that a single, monolithic, manual test late in the process, we inject a series of automated tests throughout. This gets the tests dome more quickly and helps to ensure the quality of the code at each step of the process. Remember, this is iterative.
• #33: Now, computers are doing the bulk of our work for us. They track the versions of the code, run tests on the code, bundle it into packages, deploy it to our servers. Outside of writing the code, just about the only thing that people are doing is telling the computers what to do and when. Turns out we can offload that to computers, too.
• #34: Continuous Integration servers, at their root, are job execution engines. They can monitor entities (like a VCS repo) and kick off jobs in response to events, or they can be made to kick jobs off by invoking an API. The jobs can be invoked on any servers to which the CI server has access (generally via SSH or an installed agent). This gives us a way to let the computers. Jobs can be chained to other jobs, creating what is commonly known as a “pipeline.” You can even define workflow and reporting logic, so a pipeline run can be stopped and the proper people notified automatically if a job fails. This allows us to take still more work out of the hands of people and transfer it to a machine.
• #35: If we can come up with a way to allow the computer to do our tests, then we gain all the benefits of automation. Because of this, people have written a variety of automated testing frameworks that do just that. Once we’ve automated our tests, it’s easier to apply them earlier (and more frequently) in the lifecycle.
• #36: Now we’ve automated all of the steps of the development process that lend themselves to automation. Our developers can focus solely on writing code, and as soon as they check something in, our automated workflow kicks in and does the rest. We’ve attained nirvana…
• #37: …almost. Things always fail, and usually at the worst time. We’ve got a great app development and delivery process, but what happens when a piece of the infrastructure breaks?
• #38: We call in the sysadmin, of course. And this is where agile dreams have been dashed for decades. For a surprisingly long time, there was no good way to write the instructions for (re)building a server in a way that a computer could interpret. Instead, sysadmins operated from memory or from giant runbooks (paper or electronic).
• #39: We call in the sysadmin, of course. And this is where agile dreams have been dashed for decades. For a surprisingly long time, there was no good way to write the instructions for (re)building a server in a way that a computer could interpret. Instead, sysadmins operated from memory or from giant runbooks (paper or electronic).
• #40: Over time, we created a series of progressively smaller anchors, but we almost always came back to the giant runbook. Sometimes a shell script would replace a paragraph or two, but we’d still have a line in our tome that said “and then run this script.” Golden images and VM snapshots removed some of the drift and repetition, but those starting points were still generally built by hand an then imaged. If anything in the underlying image needed to change, we made the change by had and rebuilt the image.
• #41: What we really needed was the same thing the developers had all along: a way to rewrite the instructions that we followed for building a machine – that is, our runbook – in a way that allows a computer to interpret and act upon them. This breakthrough was infrastructure as code.
• #42: Over time, we created a series of progressively smaller anchors, but we almost always came back to the giant runbook. Sometimes a shell script would replace a paragraph or two, but we’d still have a line in our tome that said “and then run this script.” Golden images and VM snapshots removed some of the drift and repetition, but those starting points were still generally built by hand an then imaged. If anything in the underlying image needed to change, we made the change by had and rebuilt the image.
• #43: Once we have an implementation of infrastructure as code, we can apply all of the same tools that our developers have been using with such success to system configuration. This was the shift from agile development with an anchor to DevOps. It’s important to recognize that there is almost nothing new here – most of the technology and practice has existed for years in software development circles. The one thing that changed was the introduction of a way to express system configurations in a manner that allowed us to bring those software development practices to bear on system configuration management.
• #48: Please don’t call it DevOpsSec But in a lot of ways, the DevOps community regards security orgs in a similar manner to ops in the past. We throw things over the wall for security to deal with later.
• #50: Even if a compromise doesn’t cause direct harm in the form of IP theft or financial loss, the negative publicity associated with disclosure can cause significant harm.
• #51: None of this is unreasonable. Though sometimes the processes to ensure them can be burdensome, the requirements themselves are sound.
• #52: The very things that make DevOps so successful as a development methodolgy undermine the confidence of security organizations.
• #56: Were you wondering when I’d get to the part where this is actually good for security?
• #60: Each of the testing and CI points in the pipeline represents an opportunity to incorporate security. We can also extend this pipeline in order to incorporate automated logging and monitoring systems.