Test Automation Research... Is That Really Needed in 2018?
1 like•595 views
The document discusses the Testomat project by Markus Borg, aimed at improving test automation in software development to balance quality and speed. It highlights methodologies such as visual analytics, mutation testing, and AI-assisted bug report assignment to enhance software testing processes. The initiative seeks to optimize performance and ease of use for modern software teams facing fast-paced release cycles.
Test Automation Research... Is That Really Needed in 2018?
- 1. Research Institutes of Sweden TEST AUTOMATION RESEARCH… IS THAT REALLY NEEDED IN 2018? @mrksbrg mrksbrg.com Markus Borg Swedish Institute of Computer Science
- 2. Intrinsic human fascination with automation Research helps to automate: 1) better 2) more 3) sustainably (CC Flickr: dalbera)
- 3. Development engineer, ABB, Malmö, Sweden 2007-2010 ▪ Editor and compiler development ▪ Safety-critical systems PhD student, Lund University, Sweden 2010-2015 ▪ Machine learning for software engineering ▪ Bug reports and traceability Senior researcher, RISE AB, Lund, Sweden 2015- ▪ Software engineering for machine learning ▪ Software testing and V&V 3 Who is Markus Borg?
- 4. 4 ▪ TESTOMAT project ▪ Test visualization ▪ Mutation testing ▪ AI-assisted bug report assignment Agenda
- 6. Problem Statement ▪ Contemporary dilemma. Modern software teams must optimize for both: ▪ bug free software ▪ ease of change ▪ Ever-faster release cycles => more automation 6 Project Goal ▪ Help software teams to increase the development speed without sacrificing quality ▪ Advance the state-of-the-art in test automation
- 8. TRL 0 TRL 1 TRL 2 TRL 3 TRL 7 TRL 8 TRL 9 WP4 Test Priorities WP6 Test Automation Maturity WP5 Automated Testing for Quality Standards WP3 Test Effectiveness From technology validated in research labs To technology demonstrated in relevant environment TRL 4 TRL 5 TRL 6 8
- 10. 10
- 11. Large amounts of test results Simple result matrices hide information 11 The backside… Wikimedia Commons, Brukar:Bep Creative Commons CC-BY 2.5
- 12. 12 Solution Approach Visual analytics (Illinois Applied Research Institute)
- 13. ▪ Why game engine? ▪ Interaction out-of-the-box ▪ Why Unity? ▪ Fairly simple ▪ Scales well ▪ Very popular ▪ Unity??? ▪ Cross-platform game engine and IDE ▪ Drag-and-drop 2D and 3D scenes ▪ Scripting in C# 13
- 14. 14
- 18. Do you trust your test cases? 18 Ali Parsai (Staff Sgt. Ryan Callaghan, US Air Force)
- 19. High Quality Test Suite Fewer Bugs How do you know that your test suite is good?
- 22. SurvivedKilled 2
- 24. a-ba+b a<<ba>>b a!=ba==b a<=ba<b !aa B::bA::b a(b,c)a(b) a|ba&b Very many test executions to kill mutants!
- 25. 25 Solution Approach Toward mutation testing in the cloud Sten Vercammen ▪ Goal ▪ Make mutation testing fast enough to fit during nightly build ▪ Approach ▪ Distribute the work ▪ Investigate bottlenecks and recommend optimizations
- 26. 26 Lessons learned from proof-of-concept tool ▪ Most steps are independent ▪ Speed-ups of 12x-13x with 16 workers ▪ Good chance to finish during nightly builds
- 30. 30
- 31. Bug tracker
- 32. 32 Model of bug report assignment Bug tossing!
- 33. ▪ Goal ▪ Useful tool deployable with minimum configuration effort ▪ Approach ▪ Bug reports = textual data + basic metadata ▪ Train machine learning classifiers on historical bug reports ▪ Combine them using state-of-the-art ensemble learning 33 Solution Approach AI-assisted bug report assignment
- 34. 34 Experimental setup Company A Company B Machine learning 4 x Pre-processing & feature selection 50,000+ bug reports
- 35. ▪ In line with human activity – But instantaneous! 35 Results
- 36. ▪ Productification of solution in internal tool ▪ Simplified solution without ensemble ▪ Deployed in bug tracker for large project ▪ Presents instantaneous recommendation of responsible team ▪ Accuracy 8% lower than manual work 36 Prototype deployed at Ericsson
- 40. Research on test automation = Good use of tax money 40
- 41. Intrinsic human fascination with automation Research helps to automate: 1) better 2) more 3) sustainably (CC Flickr: dalbera)
- 42. Research Institutes of Sweden Swedish Institute of Computer Science [email protected] mrksbrg.com @mrksbrg © Musée des arts et métiers-CNAM Research => 1) better 2) more 3) sustainable test automation