EXTENT-2017: Gap Testing: Combining Diverse Testing Strategies for Fun and Profit
Download as PPTX, PDF0 likes108 views
Dr. Ben Livshits discusses the integration of diverse testing strategies, such as fuzzing and crowd-sourced testing, to enhance software reliability and achieve higher statement-level coverage. The talk emphasizes the importance of combining automated and manual testing methods to address the gaps in developer-written test suites, which often miss significant behavioral aspects observed in the field. The ultimate aim is to create a coordinated testing approach that optimally utilizes resources and improves overall testing efficacy.
EXTENT-2017: Gap Testing: Combining Diverse Testing Strategies for Fun and Profit
- 1. DR. BEN LIVSHITS IMPERIAL COLLEGE LONDON GAP TESTING: COMBINING DIVERSE TESTING STRATEGIES FOR FUN AND PROFIT
- 2. MY BACKGROUND Professor at Imperial College London Industrial researcher Stanford Ph.D. Here to talk about some of the technologies underlying testing Learn about industrial practice Work on a range of topics including Software reliability Program analysis Security and privacy Crowd-sourcing etc.
- 3. FOR FUNCTIONAL TESTING: MANY STRATEGIES Human effort Test suites written by developers and/or testers Field testing Crowd-based testing Penetration testing Automation (Black box) Fuzzing White box fuzzing or symbolic execution We might even throw in other automated strategies into this category such as static analysis
- 4. MANUAL VS. AUTOMATED My focus is on automation, generally However, ultimately, these two approaches should be complimentary to each other Case in point: consider the numerous companies that do mobile app testing, i.e. Applause The general approach is to upload an app binary, have a crowd of people on call, they jump on the app, encounter bugs, report bugs, etc. Generally, not many guarantees from this kind of approach But it’s quite useful as the first level of testing https://www.slideshare.net/IosifItkin/extent2016-the-future-of-software-testing
- 5. MANUAL VS. AUTOMATED: HOW DO THEY COMPARE? Fundamentally, a difficult question to answer What is our goal Operational goals Make sure the application doesn’t crash at the start Make sure the application isn’t easy to hack into Development/design goals Make sure the coverage is high or 100%, for some definition of what coverage is Make sure the application doesn’t crash, ever, or violate assertions, ever? Do we have to choose?
- 6. MULTIPLE, COMPETING, UNCOORDINATED TECHNIQUES ARE NORMAL We would love to have a situation of when one solution delivers all the value Case in point: symbolic execution was advertised as a the best thing since sliced bread: Precision of runtime execution Coverage of static analysis How can this go wrong? The practice of symbolic execution is unfortunately different Coverage numbers from KLEE and SAGE
- 7. SO, MAYBE ONE TECHNIQUE ALONE IS NOT GOOD ENOUGH What can we do? Well, let’s assume we have the compute cycles (which we often do) and the money to hire testers (which we often don’t) How do combine these efforts? Fundamental challenges Overlap is significant, bling fuzzing is not so helpful Differences are hard to hit – for example, how do we hit a specific code execution path to get closer to 100% path coverage? Symbolic execution is a heavy-weight, less-than-scalable answer
- 8. DEVELOPER-WRITTEN TESTS VS. IN-THE-FIELD EXECUTION Study four large open-source Java projects We find that developer-written test suites fail to accurately represent field executions: the tests, on average, miss 6.2% of the statements and 7.7% of the methods exercised in the field; The behavior exercised only in the field kills an extra 8.6% of the mutants; finally, the tests miss 52.6% of the behavioral invariants that occur in the field.
- 9. LET’S FOCUS ON EXECUTION PATHS Need to coordinate our testing efforts Gap testing principles Avoid repeated, wasteful work Find ways to hit methods/statements/basic blocks/paths that are not covered by other methods Common paths: Covered multiple times Extra work is not warranted However, extra testers are likely to hit exactly this Occasionally encountered: How do we effectively cover this? Rarely seen: How do we hit this without wasting effort?
- 10. TWO EXAMPLES OF MORE TARGETED TESTING Crowd-based UI testing aiming for 100% coverage Targeted symbolic execution aiming to hit interesting parts of the code
- 11. GAP TESTING FOR UI Testing Android apps Goal: to have 100% UI coverage How to define that is sometimes a little murky But let’s assume we have a notion of screen coverage Move away from covered screens By shutting off parts of the app Aim is to to get as close as 100% coverage by guiding crowd-sourced testers
- 12. CROWD OF TESTERS WITH THE SYSTEM GUIDING THEM TOWARD UNEXPLORED PATHS
- 13. GUIDING SYMBOLIC EXECUTION Continue exploring the program until we find something “interesting’ That may be a crash or an alarm from a tool such as AddressSanitizer, ThreadSanitizer, Valgrind, etc. Suffers from exponential blow-up issues and solver overhead If we instead know what we are looking for, for example, a method in the code we want to see called, we can direct our analysis better Prioritize branch outcomes so as to him the target
- 14. ULTIMATE VISION A portfolio of testing strategies that can be invoked on demand Deployed together to improve the ultimate outcome Sometimes, manual testing in the right thing, sometimes it’s not We’ve seen some examples of complimentary testing strategies The list is nowhere close to exhaustive…
- 15. OPTIMIZING TESTING EFFORTS How to get the most out of your portfolio of testing approaches, minimizing the time and money spent It would be nice to be able to estimate the efficacy of a particular method and the cost in terms of time, human involvement, and machine cycles That’s actually possible with machine learning-based predictive models, i.e. mean time to the next bug found is something we can
- 16. THE END.
- 17. GAP TESTING: COMBINING DIVERSE TESTING STRATEGIES FOR FUN AND PROFIT We have seen a number of testing techniques such as fuzzing, symbolic execution, and crowd-sourced testing emerge as viable alternatives to the more traditional strategies of developer-driven testing in the last decade. While there is a lot of excitement around many of these ideas, how to property combine diverse testing techniques in order to achieve a specific goal, i.e. maximize statement-level coverage remains unclear. The goal of this talks is to illustrate how to combine different testing techniques by having them naturally complement each other, i.e. if there is a set of methods that are not covered via automated testing, how do we use a crowd of users and direct their efforts toward those methods, while minimizing effort duplication? Can multiple testing strategies peacefully co-exist? When combined, can they add up to a comprehensive strategy that gives us something that was impossible before, i.e. 100% test coverage?