The importance of a DevOps pipeline - Amazon Web Services (AWS) Tutorial

Before we start digging into AWS, getting technical and building out our pipeline, let's take a step back and talk about DevOps, what a pipeline is, and what the importance is of having a good DevOps pipeline. The software development process was completely different back when I started building software 20 years ago. Developers had to package all of their source code from the repository, package all of their dependencies. Then they had to write instructions on how to deploy it to a server, hand that over to a release manager, and then deploy it across the different environments. That took a long time, was very complicated and many times took several tries to get it right. Deployments were large scripted events. It could take a day or two to put all of this together on top of the development cycle. Meanwhile, developers are setting their idle, troubleshooting with the release manager hopeful that the release will work. The software release process itself could take weeks or months because every time you did a release, you had to put the same package together and make sure the environments were similar, so that as you release across these different environments, the application would work and would be consistent. It was not agile and it did not allow you to respond to the customer demands quickly. Today's world of modern software development works completely differently. By the nature of the cloud and the demands put on us by our customers, we perform frequent releases. This could mean anything from releases every two to three weeks to daily or hourly releases. Vulnerabilities happen all the time, so we have to be able to respond quickly. Customers want changes in the application. They want it to modernize and see that they're making an impact. So we have to make changes quickly to accommodate our customers. In order to do this, we have automation throughout the process from beginning to end. There has to be a way for changes to be efficient and quick. The easiest way to do this is by using automation: automation with the source code deployment, automation with the testing, automation with the release to production. And in order to do this, multiple teams have to collaborate together. This includes the development teams, the operation teams, otherwise known as the DevOps team. Multiple teams have to work together to make this happen. A good definition of a DevOps pipeline is that it's a set of automated processes and tools that allow developers and operations professionals to collaborate on building and deploying code. By the nature of its name, it's a combination of development and operations. So the tool set that you put together has to enable both teams to collaborate. The tooling that allows this to happen is called a pipeline. A pipeline, like we talked about before, is a set of tools that enables collaboration and frequent releases of the application across different environments. So what makes a good pipeline? Well, development needs to be fast. Development needs to be able to quickly check in source code, have the source code tested, release the code, go onto the next environment, and finally be released into production. All of that has to happen quickly and not in a matter of days. A good pipeline also needs to support automation. There should not be many manual steps in a pipeline. There should be pass/fail criteria built into the pipeline that allow you to accept or reject a build based on certain criteria. And finally, you have to have immediate feedback. Testing used to take a long time because we would do the testing manually or we have to review the results of the PDF manually. A good pipeline allows for immediate feedback. You can see the results from the pipeline, you can see the pass/fail, you can see why the build was rejected, and you can act on it quickly. So all of this is good for developers, but what about security? Security can't be the thing that holds up the delivery of the application. If it does, it will get left behind. That's where DevSecOps comes along. When you combine security with DevOps, the modern way of describing this is DevSecOps. It takes the burden off the security teams by giving more of that responsibility to the developer teams. It takes the test, the security team used to run, embeds them into the process or the pipeline, and allows the developers to take ownership of that. They're responsible for it, but they're not accountable for it. The securities team will still set the guidelines, but the DevOps teams will be the one to perform the testing inside of their environment and inside of their process. This makes security fast and not a burden. Like I said, the key to that is you have to take security, put it with the DevOps team, and enable the developers to run it alongside their other tests to make it easy. DevSecOps can work inside of the pipeline. You can use the pipeline to leverage automation for all of the security tools that you have to run. Whether they're static analysis or cloud security tools, leverage the pipeline to embed the tools in the process. Embed the pass/fail criteria inside the pipeline, and this will give you the ability to build security into the DevOps process. And also, balance speed versus accuracy. Sometimes security tools may take a long time to run. Decide if you need to break the build for that particular tool, or can you run the tool on a subset of the application, maybe only on code changes or a problematic part of the application. As you're going through and building DevSecOps into the pipeline, balance speed and accuracy versus the completeness of the security task. And then ultimately, this makes the developers responsible for security. Developers are the ones that are responsible for running the test. It's part of their process, and if they have metrics and guidelines that hold them to running the security tests, we just have to make it easy and embed into their process.