Using ThreadFix to Manage Application Vulnerabilities

Using ThreadFix to Manage Application Vulnerabilities!
!
Dan Cornell!
CTO, Denim Group!
@danielcornell

© Copyright 2013 Denim Group - All Rights Reserved

My Background
•  Dan Cornell, founder and CTO of
Denim Group

•  Software developer by background
(Java, .NET, etc)

•  OWASP San Antonio, Global
Membership Committee

© Copyright 2013 Denim Group - All Rights Reserved 2

Denim Group Background

•  Secure software services and products company
–  Builds secure software
–  Helps organizations assess and mitigate risk of in-house developed and third party
software
–  Provides classroom training and e-Learning so clients can build software securely
•  Software-centric view of application security
–  Application security experts are practicing developers
–  Development pedigree translates to rapport with development managers
–  Business impact: shorter time-to-fix application vulnerabilities
•  Culture of application security innovation and contribution
–  Develops open source tools to help clients mature their software security programs
•  Remediation Resource Center, ThreadFix
–  OWASP national leaders & regular speakers at RSA, SANS, OWASP, ISSA, CSI
–  World class alliance partners accelerate innovation to solve client problems


Agenda
•  Introductions
•  Application Vulnerability Management
•  ThreadFix Background
•  Use Cases / Demonstrations
–  Track Scan Results Over Time
–  De-Duplicate and Merge Multiple Scanners
–  Scanner Benchmarking
–  Virtual Patching
–  Turning Vulnerabilities into Software Defects
–  Program Benchmark Reporting
•  Future Directions
•  Questions


Application Vulnerability Management

•  Application security teams uses automated static and dynamic test results as
well as manual testing results to assess the security of an application
•  Each test delivers results in different formats
•  Different test platforms describe same flaws differently, creating duplicates
•  Security teams end up using spreadsheets to keep track manually
•  It is extremely difficult to prioritize the severity of flaws as a result
•  Software development teams receive unmanageable reports and only a small
portion of the flaws get fixed


The Result
•  Application vulnerabilities persist in applications:
**Average serious vulnerabilities found per website per year is 79
**Average days website exposed to one serious vulnerability is 231 days
**Overall percentage of serious vulnerabilities that are fixed annually is only 63%

•  Part of that problem is there is no easy way for the security team and
application development teams to work together on these issues

•  Remediation quickly becomes an overwhelming project

•  Trending reports that track the number of reduced vulnerabilities are
impossible to create

**WhiteHat Statistics Report (Summer 2012):
https://www.whitehatsec.com/assets/WPstats_summer12_12th.pdf


Vulnerability Fun Facts:
•  Average number of serious
vulnerabilities found per
website per year is 79 **
•  Serious Vulnerabilities
were fixed in ~38 days **
•  Percentage of serious
vulnerabilities fixed
annually is only 63% **
•  Average number of days a
website is exposed, at least
one serious vulnerability
~231 days

WhiteHat Statistics Report (Summer 2012):
https://www.whitehatsec.com/assets/WPstats_summer12_12th.pdf


Vulnerability Remediation Data
Vulnerability
Type
Sample
Count
Average
Fix
(minutes)

Dead
Code
(unused
methods)
465
2.6

Poor
logging:
system
output
stream
83
2.9

Poor
Error
Handling:
Empty
catch
block
180
6.8

Lack
of
AuthorizaKon
check
61
6.9

Unsafe
threading
301
8.5

ASP.NET
non-‐serializable
object
in
session
42
9.3

XSS
(stored)
1023
9.6

Null
Dereference
157
10.2

Missing
Null
Check
46
15.7

XSS
(reﬂected)
25
16.2

Redundant
null
check
21
17.1

SQL
injecKon
30
97.5


Where Is Time Being Spent?
70%
Indicates the weighted average
versus the average of
60% individual projects
59%

50%

44%
42%
40% 37%

30% 31% 28%
29% 24%
24%
20% 17%
20%
15% 15%
16%
10% 9%
3%
2%
0% 0% 0%
Setup Development Fix Vulnerabilities Confirm Fixes / QA 0%
Deploy Overhead
Environment


Enter ThreadFix
•  An open source software vulnerability aggregation and management system
•  Imports dynamic, static and manual testing results into a centralized platform
•  Removes duplicate findings across all testing platforms to provide a prioritized
list of security faults
•  Eases communication across development, security and QA teams
•  Exports the prioritized list into the company’s bug tracker of choice to
streamline software remediation efforts
•  Auto generates web application firewall rules to protect corporate data while
the software vulnerability is being fixed
•  Empowers managers with vulnerability trending reports that can pinpoint team
issues and illustrate application security progress


ThreadFix Background
•  An open source vulnerability management and aggregation platform that
allows software security teams to reduce the time it takes to fix software
vulnerabilities

•  Freely available under the Mozilla Public License (MPL)

•  Download available at: www.denimgroup.com/threadfix


ThreadFix Consolidates reports so managers can speak intelligently about
the status and trends of security within their organization

12

Vulnerability Import • Pulls in static and dynamic results
• Eliminates duplicate results
• Allows for results to be grouped

13

Real-Time Protection Virtual patching helps protect
organizations during remediation


Defect Tracking • ThreadFix can connect to common defect trackers
• Defects can be created for developers
Integration • Work can continue uninterrupted


Large Range of Tool Compatibility


Supported Tools:
Dynamic Scanners SaaS Testing Platforms
Acunetix WhiteHat
Arachni Veracode
Burp Suite QualysGuard WAS 2.0
HP WebInspect
IBM Security AppScan IDS/IPS and WAF
Mavituna Security Netsparker DenyAll
NTO Spider F5
OWASP Zed Attack Proxy Imperva
Tenable Nessus mod_security
Skipfish Snort
w3aF
Defect Trackers
Static Scanners Atlassian JIRA
FindBugs Microsoft Team Foundation Server
IBM Security AppScan Source Mozilla Bugzilla
HP Fortify SCA
Microsoft CAT.NET
Brakeman


Use Cases / Demonstrations
•  Track Scan Results Over Time
•  De-Duplicate and Merge Multiple Scanners
•  Scanner Benchmarking
•  Virtual Patching
•  Turning Vulnerabilities into Software Defects
•  Program Benchmark Reporting


Track Scan Results Over Time
•  Pretty basic, but many software security programs have problems
providing even basic metrics and trending graphs
•  Goal: Turn a “dude with a scanner” into a “dude with some data”

•  Notes:
–  Each new scan is diff-ed against the previous scan
–  Vulnerabilities are tracked as new, fixed, reopened
–  You can durably mark false positives


Track Scan Results Over Time
•  Demonstration


De-Duplicate and Merge Multiple Scanners
•  Q: What’s worse than handing a developer a 300 page PDF?
•  A: Handing a developer two 300 page PDFs!

•  Communicating vulnerabilities via PDF is a horrible interaction pattern
for security and development teams (more on this later)


What is a Unique Vulnerability?
•  (CWE, Relative URL)
–  Predictable resource location
–  Directory listing misconfiguration

•  (CWE, Relative URL, Injection Point)
–  SQL injection
–  Cross-site Scripting (XSS)

•  Injection points
–  Parameters – GET/POST
–  Cookies
–  Other headers

25

What Do The Scanner Results Look Like?

•  Usually XML
–  Skipfish uses JSON and gets packaged as a ZIP

•  Scanners have different concepts of what a “vulnerability” is
–  We normalize to the (CWE, location, [injection point]) noted before

•  Look at some example files

•  Several vendors have been really helpful adding additional
data to their APIs and file formats to accommodate requests
(thanks!)

26

Why Common Weakness Enumeration
(CWE)?
•  Every tool has their own “spin” on naming vulnerabilities
–  OWASP Top 10 / WASC XX are helpful but not comprehensive

•  We tried to create our own vulnerability classification scheme
–  Proprietary
–  Not sustainable
–  Stupid

•  CWE is pretty exhaustive
•  Reasonably well-adopted standard
•  Many tools have mappings to CWE for their results

•  Main site: http://cwe.mitre.org/

27

Challenges Using the CWE
•  It is pretty big (909 nodes, 693 actual weaknesses)
•  But it kind of has to be to be comprehensive…

•  Many tools provide mappings
•  And sometimes they’re even kind of accurate!

•  Some tools provide more than one CWE category for a vulnerability
•  So in ThreadFix we make a best guess

•  Some tools provide “junk” results
•  So in ThreadFix we collapse those into a single vulnerability

•  Some organizations have their own classification schemes


De-Duplicate and Merge Multiple Scanners


Scanner Benchmarking
•  Of the scanning technologies you are using, which is providing the
most value?


Scanner Coverage
•  You can’t test what you can’t see

•  How effective is the scanner’s crawler?

•  How are URLs mapped to functionality?
•  RESTful
•  Parameters

•  Possible issues:
•  Login routines
•  Multi-step processes
•  Anti-CSRF protection

31

Are You Getting a Good Scan?
•  Large financial firm: “Our 500 page website is secure because the
scanner did not find any vulnerabilities!”

•  Me: “Did you teach the scanner to log in so that it can see more
than just the homepage?”

•  Large financial firm: “…”

32

Did I Get a Good Scan?
•  Scanner training is really important
•  Read the Larry Suto reports…

•  Must sanity-check the results of your scans

•  What URLs were accessed?
•  If only two URLs were accessed on a 500 page site, you probably have a bad scan
•  If 5000 URLs were accessed on a five page site, you probably have a bad scan

•  What vulnerabilities were found and not found?
•  Scan with no vulnerabilities – probably not a good scan
•  Scan with excessive vulnerabilities – possibly a lot of false positives

33

Low False Positives

•  Reports of vulnerabilities that do not actually exist

•  How “touchy” is the scanner’s testing engine?

•  Why are they bad?
–  Take time to manually review and filter out
–  Can lead to wasted remediation time

34

Low False Negatives

•  Scanner failing to report vulnerabilities that do exist

•  How effective is the scanner’s testing engine?

•  Why are they bad?
–  You are exposed to risks you do not know about
–  You expect that the scanner would have found certain classes of vulnerabilities

•  What vulnerability classes do you think scanners will find?

35

Other Benchmarking Efforts

•  Larry Suto’s 2007 and 2010 reports
•  Analyzing the Accuracy and Time Costs of Web Application Security Standards
–  http://ha.ckers.org/files/Accuracy_and_Time_Costs_of_Web_App_Scanners.pdf
•  Vendor reactions were … varied
–  [Ofer Shezaf attended this talk at AppSecEU 2012 and had some great questions and comments. See his
reactions to the latest Larry Suto scanner report here :
http://www.xiom.com/2010/02/09/wafs-are-not-perfect-any-security-tool-perfect ]

•  Shay Chen’s Blog and Site
•  http://sectooladdict.blogspot.com/
•  http://www.sectoolmarket.com/
•  http://www.infosecisland.com/blogview/21926-A-Step-by-Step-Guide-for-Choosing-the-Best-
Scanner.html

•  Web Application Vulnerability Scanner Evaluation Project (wavsep)
•  http://code.google.com/p/wavsep/

36

Scanner Benchmarking


Virtual Patching
•  Connect vulnerability
scanners to IDS/IPS/
WAF systems

•  Map data from
sensors back to data
about vulnerabilities


Virtual Patches - Formats
•  Two approaches
1.  (vulnerability_type, vulnerability_location)
2.  (vulnerability_signature , vulnerability_location)

(1) “There is a reflected XSS vulnerability in login.php for the username parameter”
versus
(2) “Watch out for HTML-ish characters in login.php for the username parameter”

•  The snort and mod_security rules follow approach (2)
•  Integration with commercial solutions may use approach (1)


Trivia and Analysis
•  IDS/IPS/WAF has an impact on the scanning process
–  Snort breaks w3af scanning
–  mod_security CRS introduces some false positives into skipfish scanning

•  mod_security CRS is quite good
–  And getting better all the time: SQL Injection Challenge
–  http://blog.spiderlabs.com/2011/06/announcing-the-modsecurity-sql-injection-challenge.html

•  Virtual patching appears to win for injection flaws


Where Is This Useful?
•  Environments where you have little or no control over deployed code
–  XaaS – PaaS, IaaS
–  99% of all corporate data centers

•  Environments where you have a large “application security debt”
–  Actual code fixes: take time and can be hard to get on the schedule


What Are The Problems?
•  Current vulnerability data formats only allow for coarse-grained virtual
patches
–  Can lead to false blocks

•  Virtual patches likely will not stop well-informed, determined attackers
–  See the results of the mod_security SQL Injection Challenge


Virtual Patching


Turning Vulnerabilities Into Software Defects
•  Security teams talk about “vulnerabilities”
•  Software developers talk about “defects”

•  Developers Don’t Speak PDF
–  http://blog.denimgroup.com/denim_group/2012/11/hey-security-teams-developers-dont-speak-pdf.html

•  Why should developers manage 90% of their workload in defect
trackers
–  And the magic, special “security” part of their workload … some other way?

•  ThreadFix lets you slice, dice and bundle vulnerabilities into software
defects
–  And track their remediation status over time to schedule re-scans

But My Bug Tracker Isn’t Supported!

•  We are always working on supporting new technologies
–  Check out the current support list:
https://code.google.com/p/threadfix/wiki/DefectTrackers
–  Submit a bug to the TheadFix defect tracker
https://code.google.com/p/threadfix/issues/list

•  You can add new defect trackers as plugins
–  No changes to the core codebase required
–  For instructions and sample code check out the wiki article:
https://code.google.com/p/threadfix/wiki/CustomDefectTrackerGuide


Turning Vulnerabilities Into Software Defects


Program Benchmark Reporting
•  How does your software security organization stack up?
–  Look at publicly-shared data from WhiteHat and Veracode

•  Compare your progress
–  Percentage of vulnerabilities fixed
–  Time to fix different vulnerability types
–  Age of remaining vulnerabilities


Program Benchmark Reporting


Current Status
•  1.0 released September 17th, 2012
•  1.0.1 released October 19th, 2012
•  1.1 (release candidate) released January 28th, 2013
•  Final 1.1 coming in the next couple of weeks


Future Directions
•  Increase the audience that can find ThreadFix useful
–  Add native scanning capability
–  Add scan scheduling and coordination capability

•  Address “enterprise” concerns
–  Expanded security model available in version 1.1
–  Continue to grow this area

•  Improve the user experience

•  Dashboard and reporting


Common Usage Scenarios
•  Use ThreadFix to provide an “enterprise” console for a standalone
desktop scanning tool

•  Use ThreadFix to normalize and merge multiple sources of
vulnerability data
–  Including the results of manual code reviews, threat models, etc

•  Use ThreadFix as a base for a custom application vulnerability
management solution
–  We’ve already written a LOT of code and solved a LOT of problems


How Can You Help?
•  Use it and provide feedback
–  Bug reports
–  Usability recommendations
–  Feature requests

•  Scan file examples
–  Multiple tools, multiple versions, limited sample set
–  Help!

•  Contribute


How To Get ThreadFix
•  Denim Group ThreadFix homepage: www.denimgroup.com/threadfix

•  Google Code site: https://code.google.com/p/threadfix/

•  Google Group:
https://groups.google.com/forum/?fromgroups#!forum/ThreadFix


Conclusions / Questions

Dan Cornell
dan@denimgroup.com
Twitter: @danielcornell

www.denimgroup.com
www.denimgroup.com/threadfix
code.google.com/p/threadfix
(210) 572-4400


Using ThreadFix to Manage Application Vulnerabilities

More Related Content

What's hot

Viewers also liked

Similar to Using ThreadFix to Manage Application Vulnerabilities

More from Denim Group

Recently uploaded

Using ThreadFix to Manage Application Vulnerabilities