Dev Breakfast: Level up to DevSecOps

Editor's Notes

• #2: Good morning everyone, and welcome to the Readify Dev Breakfast. My name is Kieran Jacobsen, I am the Technical Lead for Infrastructure and Security within Readify’s Managed Services team. My responsibility is to manage and maintain Readify’s internal systems and infrastructure. I spent a lot of time working with developers, and customer operational and security teams to achieve positive outcomes. I have a fairly unique understanding of the pressures and unique challenges for each group. I have seen strategies that have worked, and quite a few that have failed.
• #3: By all accounts, 2016 was a massive year for information security. We saw a significant number of breach disclosures, breaking records in terms of the number and size of the breaches. We saw a number of older breaches appear for the first time, like those impacting LinkedIn, Myspace and Yahoo. We saw the mirai IOT bot net appear, hit Brian Krebs with a record breaking denial of service attack, then target DynDNS and knocking off Spotify, twitter, GitHub PayPal and more. We also saw a number of breaches where database backups made public, including one impacting the Australian Red Cross.
• #4: Yet 2017, politics aside, seems to be gearing up to be even worse. Attackers have laid waste to poorly secured Hadoop, MongoDB, ElasticSearch and CouchDB instances; deleting data and leaving ransom notes. We also saw a suspected DDOS take down Lloyds bank for 2 days. Recently, a 13 year old worm made a reappearance, That’s right, SQL Slammer is back!
• #5: Here we see a representation of development and operations before the introduction of DevOps. Development would catapult new builds at operations, and they would return with a volley of bugs and issues. Our applications were unstable, deployments were a complex mess and overall our organisations suffered.
• #6: Along came DevOps, with a promise that we would get two waring factions to act as one. DevOps has largely been a success, applications have become more stable, we now have a faster release cycle, with fixes deployed to production often at an hourly basis. Development and Operations is now moving at a speed to which the rest of the business was wanting. Gartner predicted that 2016 was the year DevOps goes mainstream, with 25% of the businesses using DevOps by 2017. HP Enterprise, my old employer went so firmly as to say that within 5 years DevOps will be the norm when it comes to software development.
• #7: With everything, we somehow left our security. Organisations rushed to embrace DevOps, often at times forgetting to bring their security teams along. This has lead to a variety of issues, including those we saw earlier. In someway, It feels like we are back in the late 90s or early 2000s. If 2016 is the year that DevOps goes mainstream, then 2017 will surely be the year that attacks against DevOps becomes the norm.
• #8: So how do we move to DevSecOps? Well, we start by creating clear communication pathways between the groups, and using streamlined communication techniques. We need to consider security as code, training and obviously, integrate security controls into our DevOps cycle.
• #9: This quote, from Queensland Microsoft MVP, Jess Dodson, succinctly sums up some of the issues in the relationship between the three groups. It is such a great quote. We are all in customer service, our users are our customers; I want to extend upon this and say that within the IT industry, we are each others customers. We need to understand them and their needs to do our job well; and this goes for the relationships between Dev, Sec and Ops as well. We both the customer and the service providers within our organisation.
• #10: So here are the three groups. <click> With DevOps we recognise that dev and ops are customers of each other. Dev cannot create without the assistance of Ops, and conversely ops needs stable high quality code from the dev teams to ensure that everything runs smoothly. What about Security? <click> Well obviously, Security is a client of dev and ops. Security needs both groups to fix identified security issues, if it security has any chance of ensuring that the organisation is safe. <click> The often forgotten relationship. Dev and Ops are customers of security. These two groups are not security experts, they need guidance, assistance and support from security specialist to ensure that they are developing secure applications and deploying secure infrastructure. For these relationships to succeed, we need to ensure that there is clear communication. How do we support these relationships?
• #11: Now I hope that by now, everyone is aware of the risks of using excel and word documents containing macros, and you should have them blocked within your organisation. Unfortunately many security activities still occur via excel checklists and word documents sent via email. Excel is clunky, it is slow and difficult to process, I recently completed a review process, all 12 thousand questions in macro enabled excel files, it was slow, and it wasn’t productive. Word documents have their challenges as well, it can be hard to difference them. If I knew I was compliant to version 1.15 of the security policy, and now there is 1.16, how do I know what changed in this word document? And don’t start me on change history entries at the start of documents, do they ever work? Email attachments are great for sharing, but bad for collaboration. You should also be aware of the risks of communicating sensitive security matters using email and email attachments. Now everyone’s security is up to your level. So what should we do?
• #12: Backlogs and boards have worked wonders for agile and devops. These tools promote a collaborative effort to complete tasks and fosters ownership. Teams work together to determine the priorities for tasks. Operation and security tasks can exist within backlogs just as development tasks can, I know, because this is how I work.
• #13: Don’t forget ticket, support or helpdesk tools, like ZenDesk here. These tools foster team collaboration and encourage communication with customers. These tools allow for incidents and tasks to be passed cleanly between different parts of your organisation, with history maintained as required. You can write automated responses, saving time and allowing issues to be closed faster and with less fuss. Here is a great example, a user calls your helpdesk, they have noticed an issue with an application, it doesn’t seem to be handling the correct input. Helpdesk takes a look, validates the issue and sends it across to the system admin team. The sysadmin team looks, does some more troubleshooting, it now looks like a security issue in the app, so they send the ticket across to security, who review it, confirm the issue and pass it, with additional information on what the cause of the issue is, across to the development team to fix the issue. Once a fix has been developed, it goes back to operations to ensure its deployed smoothly. This is a smooth progression between teams, its clear who has responsibility for the issue, and its clear to the user what is happening.
• #14: The last one, is something I myself has only just in the past 6 months come to realise. Documentation needs to be in a format where editing is easy, and for that reason we often fall back to word, excel and onenote, but its then hard to version and track changes. Writing documentation using markup, and using git workflows, helps this. The workflow can be simple, operations teams make changes to say, some security processes. Once they have finished making their changes, they commit them to a branch and create a pull request. Security reviews the changes, approves them and the changes are pushed to the rest of the organisation.
• #15: So the next big takeaway is security as code. Or as I like to put it, here is all of the code that could impact your organisations security. So obviously we have application source code, but what about the other parts of your environment? If your developers are using Azure or AWS, they are probably using templates to deploy infrastructure. Care needs to be taken that these templates are created in a way that the infrastructure they deploy is secure. Does operations review these? What about security? Server configuration tools like Chef, Puppet and PowerShell DSC are all the rage, but I often see two major issues. Firstly, have you secured these tools correctly? Secondly, is the configuration these tools are deploying secure?
• #16: What we see here, is an except from a template that deploys a virtual machine. This except covers the configuration of a Network Security Group, these are basically a ACL. I took this one from the Azure QuickStart Templates repository. In this example, we can see they locked down traffic to RDP only, but the allowed sources is the Internet. Guess this box will start seeing brute force traffic. Surprisingly, this isn’t the worst that I have seen, its actually pretty good. The majority of templates do not make use of NSGs, this results in an ANY:ANY rule being applied, resulting in all services, from RDP, to SSH,, to SMB to SQL left exposed to all in sundry. Why Microsoft when this way, I cannot explain. What I do know, is that your developers are probably using these templates or basing their own templates from these.
• #17: So let’s take a quick look at another example, this type its PowerShell DSC. This example is snipped of a much larger DSC configuration that all of our servers at Readify comply to. This will disable insecure ciphers being used for HTTPS connections. Imagine the effectiveness if you pushed this out to all of your web servers, or better yet, every server in your fleet? You would be close to having a A+ rating for all of your web properties.
• #18: Training is so important, and there is a massive return on investment to be gained from training your developers, operations teams and security teams. We cannot be experts at development and security and operations, but I can have an awareness and some working knowledge of the other areas. For instance, the developer who understands application vulnerability testing, the operations team member that knows some IOS development, or the security engineer that knows how to deploy cloud infrastructure. For developers, security awareness begins at day zero. When a new developer starts, they need more than the simple password and shoulder surfing but they need specialist training as well. You need to ensure that developers know what the quality expectations for code they write is, the basics of secure coding and the most common exploit vendors. Do not assume they come with this knowledge. Senior developers should go on training courses that cover secure code practices and common mistakes. This training needs to be hands on. Training like this encourages ownership of security by senior developers, and this is critically important. Senior developers are mentors for junior developers, they sent the tone of the team and the projects, and importantly they lead code review processes. Training isn’t a once off thing, it’s a multiple times per year thing. Training is not just a PowerPoint slide, nor is it the same video each year. Training needs to be reviewed regularly to ensure it covers the latest technology and the changes in the security landscape.
• #19: At Readify, we identified that phishing was a critical risk to an organisation. Technology can only solve so much, there isn’t a silver bullet. So we opted to organise some training. We selected an external vendor that had a three step process. The first is a baseline, every use gets an email, the same email, throughout a month long period. After baseline comes training, and then once training is complete, there is a period of continual training. During this final phase, staff get one phishing email each month to test them and to help hone their skills. Before we look at the results, lets look at the break down of staff at Readify. <click> As you can see, the majority, around 80%, of our employees are technical, be it developers, database people, business intelligence, SharePoint, or infrastructure people like me. My expectation was that the majority of those who fell for the baseline email, would be non-technical, I know, I am biased. <click> As you can see, the results are a tad more interesting. Overall, we had 26% of our staff click on the baseline email, surprisingly, only a small fraction of those, 2 people, were non-technical. Since the training, our click rate is down to about 2% each month. Moral of the story, don’t always expect your non-technical staff to be the source of your security issues.
• #20: So back to the infinity of DevOps. Where do we integrate security? The answer simply is, at every step. We need to consider security at every point, and security gates should exist between each step. They say security is as strong as the weakest link in the chain, well, this is your new chain, make it as you wish.
• #21: Security planning is critical, we need to involve security personnel and teams in sprint planning and reviews. Consider security stories early in projects, they can be complex and should not be rushed.
• #22: Training is what impacts our coding phase the post, but don’t forget to consider methodologies like test driven development. Creating unit tests that verify security aspects is highly affective at finding bugs early. Think about tooling, most new IDEs contain tools and plugins that can assist developers and alert them for particular code quality issues. Invest in the right tooling. Finally, establish code views via pull requests. People should never approve their own requests, no matter the urgency. Those performing reviews need the adequate training, they need to know and understand attack vectors in code and common security issues. Any piece of code, that contains issues, be they security, stability or quality issues, shouldn’t be approved.
• #23: During the build processes, static and dynamic code analysis tools can assist in finding security issues before the reach production. We can use static analysis before build, or dynamic after build. Now I am not going to enter the which one is better argument, its been going on for a while in the security industry. When it comes to these tools, you need to ensure that you have tuned them correctly. Issues raised need to be feed back into the development teams and prioritised based upon their risks and any mitigating factors. Just because a tool says an issue is of a high priority, doesn’t take into account any mitigation that may be in place within your application or infrastructure. Now I am going to point out, if these tools detect issues, or a specific number of issues, or issues of specific urgency or criteria, then the build fails, and the code doesn’t proceed any further.
• #24: We test to ensure that our code is of the right quality, and that it does what we need it do. Testing comes in all forms. A quick example might be automated interface testing that puts SQLi into user input on a web application and tests for an adequate response. Automated Fuzzing is become quite popular in large enterprises. A number of vendors, including Microsoft are offering cloud based fuzzing platforms. Now I can’t believe in 2017 I still need to talk about this, but people still don’t do load testing. Load testing comes in two forms, the first, validating that an application and its infrastructure can handle the expected load, and the other is to push an them to their breaking point. Both are crucial to security. Just look at the Australian Census, it couldn’t handle the legitimate traffic, let alone attack traffic. If an app fails here, it doesn’t proceed.
• #25: There are some unique opportunities in terms of finding potential security vulnerabilities. Typically after an app passes build, and testing, we install it to some sort of pre-production environments. It is here that we have the first clear vision of the application on the full stack, servers, databases, and load balancers. What I recommend, is that here, before proceeding to production, scan using integrated vulnerability tools. You will get a better picture of any infrastructure vulnerabilities here, so make use of the opportunity. You can automatically trigger off this scanning upon successful deployment. Depending upon what is found, you may proceed to production or block and wait for a fix.
• #26: Now the final stages operate and monitor are where some of our typical ops team activities come in. Obviously we need to monitor our applications to ensure they are available to users, we need to monitor logs for application exceptions as well. But we can also monitor the logs for security issues, look for malicious activities and suspicious patterns and trends. We should rescan using our vulnerability assessment tools on a regular basis, this is more critical where builds happen less frequently, as we need to ensure that no new vulnerabilities have since been discovered. Tracking dependencies in your infrastructure and your code is also critical, there are a number of platforms that can do this, and alert you to vulnerabilities. Do you know if all of your NPM packages are free of bugs? Track issues and remediate.
• #27: I want to thank you all for coming this morning, it has been a pleasure to speak to you. We have time for some questions before breaking up for networking afterwards. Does anyone have any questions?