The Human Side of DevSecOps

Editor's Notes

• #4: As noted up front, this is a talk about people and organizational factors. If you think that DevOps begins and ends at the pipeline and the technologies that plug into it, you may want to wait for the recap on Twitter. And this talk is definitely not for you if your job description for your AppSec team includes configuring, tuning, and running code and web app scanners by hand. For those of you who have had one or more transformation efforts fail due to change management failures, staff shortages, or other organizational issues—and for those who have one or more automated AppSec tools and are wondering how to make them successful—this talk is for you.
• #5: I think none of us would be at this webinar if we didn’t think there is transformational power in DevOps. But for a lot of teams facing the DevOps transformation, the potential comes with a lot of anxiety. After all, we’re talking about collapsing silos. In the real world, when that happens, people die. This goes double for integrating Security into DevOps. But given that DevOps has already absorbed development, operations, QA and release engineering, why should the security transition be harder?
• #6: The reason is culture clash. More than any other organization, Security has been publicly skeptical of the benefits of DevOps. They’ve seen a lot of development processes come and go, and insecure software keeps getting built. h/t @petecheslock, DevOpsDays Austin
• #7: Let’s put a finer point on the skepticism. Security has traditionally been a discipline of controls and gates. But there’s no room for this sort of manual, expert-driven check in DevOps, just as there’s no room for weeks of manual quality assurance testing if you want to ship code several times a day. That means developers have to get security conscious, and security folks have to stop looking down their noses at DevOps and figure out how to help it move faster, not stand in the way.
• #8: As with a lot of other things, Gene Kim’s Three Ways help us to see the problem: We need to see software delivery as an end to end system that includes security as part of the problem. Security needs to help create some of the feedback loops from production (and earlier stages of the process), and needs to help Dev interpret and act on them. And security needs to participate in the culture of continual experimentation and learning.
• #9: A big part of the challenge of meeting the need for security to participate in DevOps is math: specifically, for every four people employed in infosec, there are three additional job openings. You can’t hire an infosec team big enough to define security requirements, perform secure code reviews, interpret security testing results, and handle security alerts for all your applications in your entire portfolio if you’re operating at DevOps speed.
• #10: Theory of Constraints: in an end to end flow, identify the constraint, exploit, subordinate, elevate, then repeat What is the constraint in DevSecOps? Define requirements --> Develop code --> Review code --> Build code --> Test code --> Handle bugs (Almost) all have some touch between security and development First pass: Security is the constraint Exploit constraint – maximize throughput with processes like security reviews etc. Subordinate constraint – implicitly done either by minimizing the amount of security changes that happen or adding surge capacity Elevate constraint – Optimize handoffs, change the process
• #11: So to put it another way, once you’ve optimized your Dev and Ops processes, Security becomes the constraint.
• #12: Identify Exploit Subordinate (This theory, by the way, is why I say that this talk isn’t for you if you are still running your scanners by hand. If you’re doing that, you haven’t even begun to exploit the constraint of your security team yet.) Elevate? How do we do that? The definition is “In this step, more substantive changes are implemented to “break” the constraint. These changes may necessitate a significant investment of time and/or money” – and may involve targeted review of lost productive time, tactical actions, design and/or component upgrades, and supplementing the constraint with additional throughput. But how does that work if you don’t have enough people in the first place? You do it by making more people who can do the work that security is doing
• #13: Enter Security Champions – members of the DevOps team who can perform lower-skill, high frequency jobs to take load off the security team and help DevSecOps to scale. You can think of this role as essentially a new guild, like release engineers or folks who work on unit tests. Some examples of things security champions can take on (we’ll talk about a few of these): Peer reviews Security grooming Code reviews (very specific topics based on certain security controls) Data validation Encoding Parameterization Logging Error handling Product security incident response – new CVEs based on impact and severity Provide details of components, how to tell if you’re exposed, etc Known vulns e.g. Shellshock/Heartbleed/Struts-Shock – triage impact, provide remediation plan This sounds great, but how will we make this happen?
• #14: So how do we do this practically? There’s a lot to talk about here, but I want to focus on how you get started— -- how to pick the right people for the guild, how do you get the program started, and how do you set up the working relationship between the security champion and the central security team. We’ll be drawing examples from Veracode’s own experience standing up a security champion function.
• #15: Let’s assume you’ve made the case to management. The first step is drafting your team. Let’s talk about how not to pick the right people— -- You shouldn’t assume that only developers are going to be good security champions. A strong QA resource may be a great candidate. ---- They should just have clear expectations on the time commitment. ---- And make sure to loop the managers into the call for volunteers. ---- And don’t stop with just one per team – you want to make sure you’re not introducing a new single point of failure -- You should also have clear requirements. These will probably be specific to your culture and process, but some likely common requirements include: ---- Not new to company (ramping on day 2 day) ---- Not too junior (needed an influencer) ---- Not in an existing scrum role (PO, SM)
• #16: Now that you have the team in place, you need to give them a certain baseline knowledge. How you do this will differ according to what your security team looks like, but some of the basics are: 1. security fundamentals (2 day instructor led training) CIA, etc Trust no one Security controls must be server-side Deny by default Use white lists Defense in depth Threat modeling 2. Reinforce with eLearning to provide ongoing training and access 3. Ongoing education with CTF exercises Use stuff that’s readily available – https://overthewire.org
• #17: Set guidelines for common activities – Product specific grooming guidelines Update based on findings Self-service review guidelines (what to teach vs. what can be done by anyone) – automate if possible (pro tip) Does it need a review Does it have to be a security team person Crypto? Can it be security champion Dictates basic acceptance criteria
• #18: All of these things are great activities. To be really successful, you need to measure what you’re doing and how you’re making progress.
• #19: Product security maturity model Baseline and regular updates Open Samm and BSIMM useless – built our own Needed a vision of an ideal security program Code review certification eLearning, classroom training, code review ‘test’ First year – mentored transition – failure SC do first review Expert validates results Absence of findings was a problem Formal exercise with 1:1 meeting to discuss process and findings Goals for security champions Goals for teams (against maturity model) Positive and negative impacts Include eLearning Document inside of stories with security disposition
• #21: So this is the first step in easing security as a constraint in your process. You’ll probably find a lot of additional places to improve this process.
• #22: Last, a quick plug – if you would like to know more about how Veracode thinks about securing DevOps, you may want to look at our white paper, 5 Steps to Achieving Secure DevOps, which includes a few other topics around automation and Appsec in Production.