Lean Security - LASCON 2016

Editor's Notes

• #3: Hello LASCON! James Wickett and I created this presentation for RSA’s Rugged DevOps track this year. James and I are both actively involved with the DevOps community (we run DevOpsDays Austin, coming in May). We blog together at theagileadmin.com.
• #4: DevOps changed my life and I’m here to share how understanding the same Lean techniques that have revolutionized development and operations can improve the effectiveness of your security work.
• #5: In the recent book by Steven Bellovin, “Thinking Security,” he points out that we are failing in our attempts to turn security spend into real security because we’re spending limited resources on the wrong things.
• #6: Even when we have the bad guys outnumbered, they still seem to slip through our fingers, not necessarily through any skill of their own.
• #7: To talk about Lean, we also want to talk about Agile and DevOps because they’re closely interrelated.
• #8: Most of you probably have an understanding of Agile. You have seen it done right, seen it done wrong… There’s a lot more to it, but this is the core manifesto.
• #9: Agile has become widespread in the industry. Why? Because it’s provided clear benefits to software development in organizations.
• #10: Next, DevOps!
• #11: We had a problem with development and operations being siloed and chasing opposing goals.
• #12: DevOps was conceptualized as a way to correct that. Tom Limoncelli’s book The Practice of Cloud System Administration says that “DevOps is the application of Agile methodology to system administration.”
• #13: We are no longer having conversations about whether it is or isn't a thing. We are now talking about DevOps adoption and how it’s quickly growing, and is showing the same kind of benefits to adopting organizations.
• #14: Gene Kim’s IT Revolution Press sponsors a yearly study on DevOps, compiled by a data scientist, and it reveals that companies are getting real ORDERS OF MAGNITUDE value out of devops.
• #15: And that brings us to Lean. Who’s already familiar with Lean in some context (manufacturing, Six Sigma, software development, or product development)?
• #16: What is Lean? Lean started off by revolutionizing the world of manufacturing, as pioneered by W. Edwards Deming, and later popularized in the Toyota Production System, but since then it has been adapted to software development. Its practices include value stream mapping, waste, pull, queueing theory, human motivation, measurement and visualization of metrics, TDD… We’ll go over many of these in the context of improving security work later in the presentation.
• #17: Eric Ries also applied lean principles to product development in his book Lean Startup, which characterizes the core loop inside the product development cycle as “Build – Measure – Learn.” Lean is about bringing your effort to bear on the item with the highest leverage at any given time.
• #18: The Puppet State of DevOps Report says: “One can describe DevOps as the pattern that emerges when you apply these same lean principles to technology.” Lean product, lean software, agile, and devops all come together into a single mutually reinforcing picture for a technology organization.
• #19: So let’s get to the security part of all this.
• #20: If you look at every new innovation, whether it’s Lean, devops, cloud, social, mobile, etc. only through the filter of how “it is a threat” to security then you’ve adopted a losing mindset out of the gate. We’re not going to spend any time on this because it’s a red herring.
• #21: Every single field has to innovate to stay relevant, and InfoSec doesn’t get a pass on that. IT operations had stagnated in innovation for a decade, and part of the impetus behind the adoption of DevOps is that just plain wasn’t acceptable any more. Being the BOFH saying “no” is no longer a viable approach; businesses won’t stand for it any more.
• #23: We’ll now examine six common challenges faced by InfoSec organizations and explain how you might be able to bring Lean to bear on implementing security more effectively in your organization. Each of these is very similar to a problem that we had in Ops and that seemed intractable before we found DevOps.
• #24: Each one of these issues is a perception you have probably heard from someone at some point. While these are not all fair, they are also not completely unfounded. The first complaint is that you’re just there to check boxes and don’t do much to make the apps and systems really more secure.
• #25: In his book on browser security, Michael Zalewski has a great intro covering the history of information security and he poignantly notes that at some point we decided risk management was able to fill the development and operations gaps we experienced. We became experts of structured inadequacy and wrapping problems with policies and “Accept the risk” statements.
• #26: This doesn’t create value. Let’s talk about value.
• #27: The foundation of Lean is to understand what the real value add steps are in the creation of your product or service. This is an example value stream mapping representing the value creation steps between the initiation of a process and delivery to the customer. In Lean Software, this is called the “Concept to Cash” flow. So take a moment to think about this. Do you understand exactly what value you are creating and where? Is it just a compliance checkbox at the end of the process? Where could you be creating value?
• #28: Johan Bakker took a stab at what the true value of security management looks like to an organization. Note that it hinges on creating a solution that is custom to your organization. Whatever stock answers you got taught in Security School are not necessarily the value your customers need from you.
• #29: Where can you add real security value that improves the value of your organization’s core business? Years ago at National Instruments, I worked on the FPGA Compile Cloud product, a service where customers would upload their very sensitive IP to us to compile. By aligning our security work with end customer concerns, we had a security stance that was used as a positive marketing message and greatly facilitated adoption - which is another word for “sales”.
• #30: The second complaint is that security is just a bottleneck to getting “the real work done.”
• #31: Fortune Magazine reported just last month that the average time to deliver corporate IT Projects has increased from ~8.5 months to over 10 months in the last 5 years.
• #32: And the reason is… Security. Security has resulted in a proliferation of new work that can slow everything else down, if it’s “poorly coordinated.” What does that mean?
• #33: Waste. Lean focuses a lot on the identification and removal of waste; it’s the very first Lean principle. In today’s business environment time is a critical resource, and to be honest, Security is often guilty of squandering it.
• #34: Lean talks about three types of waste. Muda is the most discussed, though you can see how you might see muri (unreasonable demand, over-pressure) and mura (lack of flow) in security’s interaction with the rest of the organization.
• #35: The seven forms of muda can be seen in security operations. Here are a couple security-centric examples, but the takeaway is to analyze what you’re doing and identify the areas of waste in it.
• #36: Muda is separated into two types - “Type 1” is necessary - like compliance - but does not add value; “Type 2” is just plain unneeded. Even if you’re just focused on the “necessary” ones, do you really want to describe your career at family gatherings as “Type 1 waste?”
• #37: Your net value to the organization is the value you create minus the waste that you generate.
• #38: People are tempted to see security as a solution in search of a problem when they don’t see how it fits in to everything.
• #39: Security is everyone’s job, right?
• #40: In Operations, Performance used to be invisible and we would say performance was everyone’s job… And then no one did anything about it. Security has a lot of corollaries to application performance problems 5-10 years ago.
• #41: The Web and app performance field finally stopped passing the buck around and decided to help people actually see and address the problems, mainly by focusing on visualizing performance metrics directly in context to workers. Steve Souders, who served as head performance guru at Yahoo and Google, drove a movement to understand how to make high performance Web sites via tooling like YSlow, founding the Velocity conference, and urging companies like Microsoft and Google to share their research on performance impact to revenue.
• #42: We could do the same thing with security… Some of these things already exist, but are you using them?
• #43: This is an example of another Lean principle, “See the whole”; lean software development speaks extensively about metrics collection and visualization. You have to adopt a more open “broadcast” mentality to empower education and collaboration.
• #44: If you just add visibility to yourself and some executive manager - well, one of you better start writing the code, or no one else gives a rat’s ass.
• #45: It’s easy to get forced into reacting after there’s a breach, or even just after a vulnerability has gone live into production.
• #46: Deming addressed this issue very early in Lean. This gave rise to the “Build Integrity In” Lean software principle.
• #47: The more you can create fast feedback loops which detect and remediate security problems continually as part of your engineers’ normal work process, the less waste you generate.
• #48: Integrate into continuous integration and use test driven development (TDD) to rectify issues at the lowest waste point - upstream, or “shifted left” as the kids say nowadays. One tool that helps you do this is an open source project called gauntlt.
• #49: Many of you may have seen James present on gauntlt previously. It allows you to use the same gherkin syntax that several popular test driven development tools like cucumber and robot use to specify the security requirements of a piece of software and run it as a test along functional tests.
• #50: So as the code goes through the developers’ CI pipeline, it gets security tests applied to it prior to the build passing.
• #51: This provides continuous feedback at a very close point (not as close as the IDE, but close) - and therefore enables positive change with the least amount of waste.
• #52: There’s a whole workshop tutorial if you want to learn more about gauntlt.
• #53: Don’t wait till the end. Figure out what could be done in conjunction with each of those value creation steps.
• #54: It’s one thing to be a bottleneck due to constraints, but I think it’s fair to say many engineers consider security to be actively messing with them.
• #56: Adrian Cockroft from Netflix claimed they had “No process” in his talk at AppSec when it came here to Austin. What he really meant is that they have made doing the right thing a part of the systems everyone uses, so the perception is that there’s no process.
• #57: Use automation to build integrity in, don’t rely on mass inspection after things have already been done wrong.
• #59: Security is a product, like any other. And all products have to make tradeoffs about what they will do and what they won’t do.
• #60: If you listen to some folks you can’t “do security” without the direct backing of the CEO and a $1.2M budget to get the six or seven huge products you need, and that’s the way to achieve perfection. What if you can’t get that? Even if you can, many tools not only may be not a good fit for your needs, but also take a lot of time to implement. In a very real sense… Mo’ money, mo’ problems.
• #61: Lean Startup describes the solution to this efficiency problem. Rather than add additional waste via long analysis cycles, implement something small and fast, analyze the results, and iterate. Often layers of a couple imperfect items yields better security than one “perfect” item.
• #62: Security is a product. DevOps realized that performance, reliability, deployment, etc. was a product with defined customers and a need for active management and prioritization. Do the same thing with security. Manage it like a product manager manages all the other aspects of a product.
• #63: One of the reasons there’s been a DevOps track at this conference since 2012 is that not too many years ago we ops engineers were in the same situation and had all the exact same criticisms leveled at us. Operations was the at-best-invisible, beancounting bottleneck that was always a day late and a dollar short. But these time-tested principles has helped our entire industry begin to innovate its way out of that rut. Check them out and see if they can help you in the same way.
• #65: Thanks for your time! You can find more of our thoughts on DevOps, Lean, Security, and other topics at theagileadmin.com.