Mutation Testing: Start Hunting The Bugs
0 likes•159 views
The document discusses mutation testing, a structural testing method aimed at improving code quality by identifying effective test cases through small code alterations. It explains the basic ideas behind mutation testing, including how to generate and run tests, and demonstrates practical examples using Java. Additionally, it provides insights on frameworks for the JVM, lessons learned from using mutation testing, and guidance on how to implement it in projects.
![Lesson Learned
public static final String[] discardCommonPrefix(String a, String b) {
String[] ret = { a, b };
int l = a.length() < b.length() ? a.length() : b.length();
for (int i = 0; i < l; i++) {
if (a.charAt(i) == b.charAt(i)) {
if (i + 1 < l) { ret[0] = a.substring(i + 1); ret[1] = b.substring(i + 1); }
else {
if (a.length() < b.length()) { ret[0] = ""; ret[1] = b.substring(i + 1); }
if (a.length() == b.length()) { ret[0] = ""; ret[1] = „"; }
if (a.length() > b.length()) { ret[0] = a.substring(i + 1); ret[1] = „"; }
}
} else break;
}
return ret;
}](https://image.slidesharecdn.com/mutationtesting-starthuntingthebugssydneytestersgroup1-200618081547/85/Mutation-Testing-Start-Hunting-The-Bugs-42-320.jpg)
![Lesson Learned
public String[] discardCommonPrefix(String a, String b) {
final String[] ret = new String[2];
int l;
if (a.length() < b.length()) { l = a.length(); }
else { l = b.length(); }
int position = 0;
for (; position < l; position++) {
final char charA = a.charAt(position);
final char charB = b.charAt(position);
if (charA != charB) { break; }
}
if (position >= a.length()) { ret[0] = ""; }
else { ret[0] = a.substring(position); }
if (position >= b.length()) { ret[1] = ""; }
else { ret[1] = b.substring(position); }
return ret;
}](https://image.slidesharecdn.com/mutationtesting-starthuntingthebugssydneytestersgroup1-200618081547/85/Mutation-Testing-Start-Hunting-The-Bugs-43-320.jpg)
Mutation Testing: Start Hunting The Bugs
- 1. MUTATION TESTING @ SVENRUPPERT START HUNTI NG THE BUG S
- 2. Who is speaking? • Sven Ruppert • Developer Advocate – DevSecOps • JFrog Inc • Twitter: @SvenRuppert • eMail: [email protected]
- 3. THE AGENDA The story behind this talk The example – test yourself The basic idea behind Kinds of Mutations Frameworks for the JVM A small Demo Lesson Learned How to start in your project
- 4. Win a JFrog Shirt…
- 5. Win a JFrog Shirt… • JFrog.com/shownotes • Tonights Slides • JFrog T-Shirt Raffle – Win 1 of 3 JFrog T-Shirts https://bit.ly/SyndeyTestersTshirt
- 6. The story behind this talk
- 7. The story behind this talk no test coverage no dedicated refactoring budget decrease complexity was needed Project was >10 yrs old What is the right place to start? Where is the biggest risc? How to motivate the team?
- 8. The example – test yourself
- 9. The example – test yourself public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } How many tests are needed? What are the right input values?
- 10. The example – test yourself public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } How many tests are needed? What are the right input values? Write down - - the expected values - the used metric
- 11. The example – test yourself public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } how to find out, what will be enough? how to find the right tests? how to guarantee a constant quality?
- 12. The example – test yourself public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } @Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); } @Test public void testAdd02() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); }
- 13. The basic idea behind
- 14. The basic idea behind Mutation Testing is a structural testing method we want to find a way to write "good" tests how to find "good" tests? let the machine find the targets let´s mutate it... but how?
- 15. The basic idea behind a mutation is a small change in the code .. small enough to be a small defect P will be the program T will be the collection of all tests / Test Suite a sequence of mutations / P1,P2,P3... Px will have only one mutation compared to P
- 16. The basic idea behind running all tests from T against Px green: T will kill the mutation red: if all tests are green we kill k out of n mutants we are perfect if: k == n how to create all versions of Px ?
- 17. The basic idea behind generating the mutants and running all junit tests check the reports write more / better tests loop until quality target reached
- 18. The basic idea behind the amount of defects per 1k lines of P used to compare inside a project Module A Module B Module C 20 / 1k 10 / 1k 30 / 1k Module B Module A Module C Helps to decide how to spend the money
- 20. Kinds of Mutations Value Mutation changing constants, loop bounds (adding/subtracting values) Decision Mutation < will be changed to <= swapping/deleting/duplicating lines of code Statement Mutation
- 21. Kinds of Mutations for Java you could think about more language spec. mutations .. changing modifiers .. changing between static / non static .. delete member initialization .. delete this. .. argument order change
- 22. Kinds of Mutations mutation testing is an add-on to normal jUnit TDD tools are supporting it well generating and running all tests are time consuming but most important will effect your project structure
- 23. Frameworks for the JVM
- 24. Frameworks for the JVM muJava 2003. First released as JMutation (Java Mutation System). 2004. The name was changed to MuJava (Mutation System for Java). 2005. Software Copyright Registration, ALL RIGHTS RESERVED. 2005. Version 2 released with several fault fixes and modified mutation operators. 2008. Version 3 released with minimal support for Java 1.5 and 1.6. 2013. Version 4 released to support JUnit tests and Java 1.6 language features, including generics, annotations, enumerations, varargs, enhanced for-each loops, and static imports. 2015. Additional and improved error messages. Bug fixes for OpenJava. Licensing changed to the Apache license. https://cs.gmu.edu/~offutt/mujava/ https://github.com/jeffoffutt/muJava/graphs/contributors
- 25. Frameworks for the JVM 2012. started around 2012 with a small codebase. 2014. very active since 2014 pitest.org
- 26. A small Demo
- 27. A small Demo public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } How many tests are needed? What are the right input values? Write down - - the expected values - the used metric
- 28. A small Demo public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } How many tests are needed? For 100% LineCoverage Two tests One if – branch leads to two paths
- 29. A small Demo public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } @Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); } @Test public void testAdd02() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); }
- 30. A small Demo public class Service { public int add(int a, int b){ if( a<2 ){ return ( a+b ) * -1; } else { return a+b; } } } @Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); } @Test public void testAdd02() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); }
- 31. A small Demo @Test public void testAdd001() throws Exception { final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); }
- 32. A small Demo @Test public void testAdd02() throws Exception { final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); }
- 33. A small Demo final int add = new Service().add(0, 0); Assertions.assertThat(add).isEqualTo(0); final int add = new Service().add(3, 0); Assertions.assertThat(add).isEqualTo(3); we got 100% Line Coverage How good these tests are? How to measure if these test are good? How to find the good tests?
- 34. A small Demo final int add = new Service().add(0, 0); final int add = new Service().add(3, 0); let´s generate a the mutation report with maven : pitest: mutationCoverage >> Generated 54 mutations org.pitest……mutators.ConditionalsBoundaryMutator org.pitest……mutators.IncrementsMutator org.pitest……mutators.ReturnValsMutator org.pitest……mutators.MathMutator org.pitest……mutators.NegateConditionalsMutator Killed 3
- 35. A small Demo
- 36. A small Demo final int add = new Service().add(0, 0); final int add = new Service().add(3, 0); Killed 3
- 37. A small Demo final int add = new Service().add(0, 0); final int add = new Service().add(3, 0); final int add = new Service().add(2, 0); Killed 4
- 38. A small Demo final int add = new Service().add(0, 0); final int add = new Service().add(3, 0); final int add = new Service().add(2, 0); Killed 5 final int add = new Service().add(1, 1);
- 39. A small Demo final int add = new Service().add(0, 0); final int add = new Service().add(3, 0); final int add = new Service().add(2, 0); Killed 6 final int add = new Service().add(1, 1); final int add = new Service().add(2, 2);
- 40. A small Demo final int add = new Service().add(1, 1); final int add = new Service().add(2, 2);
- 41. Lesson Learned
- 42. Lesson Learned public static final String[] discardCommonPrefix(String a, String b) { String[] ret = { a, b }; int l = a.length() < b.length() ? a.length() : b.length(); for (int i = 0; i < l; i++) { if (a.charAt(i) == b.charAt(i)) { if (i + 1 < l) { ret[0] = a.substring(i + 1); ret[1] = b.substring(i + 1); } else { if (a.length() < b.length()) { ret[0] = ""; ret[1] = b.substring(i + 1); } if (a.length() == b.length()) { ret[0] = ""; ret[1] = „"; } if (a.length() > b.length()) { ret[0] = a.substring(i + 1); ret[1] = „"; } } } else break; } return ret; }
- 43. Lesson Learned public String[] discardCommonPrefix(String a, String b) { final String[] ret = new String[2]; int l; if (a.length() < b.length()) { l = a.length(); } else { l = b.length(); } int position = 0; for (; position < l; position++) { final char charA = a.charAt(position); final char charB = b.charAt(position); if (charA != charB) { break; } } if (position >= a.length()) { ret[0] = ""; } else { ret[0] = a.substring(position); } if (position >= b.length()) { ret[1] = ""; } else { ret[1] = b.substring(position); } return ret; }
- 44. Lesson Learned Version 1 Version 2 for { if { if else { if if if } } else } if else for { if } if else if else
- 45. Lesson Learned mutation tests are often leading to …cleaner code compared to jUnit only … smaller modules (shorter mutation runtime) and something nice… helps to find useless code
- 46. Lesson Learned
- 47. Lesson Learned Here the two lines near by In real life.. A few lines are in between
- 48. SecurityPayload Generator Injection Store a TestPlan with the BuildInfo Load binary data from Artifactory – Generic Repo SecurityPayload Generator Injection Store a TestResults with the BuildInfo Repo Testplan Repo SecurityPayload TestResultRepo
- 49. Artifactory / XRay and how to use it Internet REST API / WebUI RulePolicyWatch Repo
- 50. Try it : OnPrem and/or Cloud https://jfrog.com/platform/free-trial/ Just try it
- 51. How to start in your project
- 52. How to start in your project you need jUnit - to generate the reference add the pitest-plugin to the build section configure the plugin generate the reference -> clean , install run pitest: mutationCoverage report will be under target/pit-reports
- 53. How to start in your project <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <configuration> <outputFormats> <outputFormat>XML</outputFormat> <outputFormat>HTML</outputFormat> </outputFormats> <targetClasses> <param>org.rapidpm.*</param> </targetClasses> <targetTests> <param>org.rapidpm.*</param> <param>junit.org.rapidpm.*</param> </targetTests> </configuration> </plugin>
- 54. How to start in your project <reporting> <plugins> <plugin> <groupId>org.pitest</groupId> <artifactId>pitest-maven</artifactId> <reportSets> <reportSet> <reports> <report>report</report> </reports> </reportSet> </reportSets> </plugin> </plugins> </reporting>
- 55. How to start in your project Start with some tests generate the pitest report write more tests to kill mutations if you have time, eliminate useless tests do it one by one
- 56. How to start in your project
- 57. How to start in your project
- 58. How to start in your project
- 59. How to start in your project
- 60. How to start in your project
- 61. Win a JFrog Shirt… • JFrog.com/shownotes • Tonights Slides • JFrog T-Shirt Raffle – Win 1 of 3 JFrog T-Shirts https://bit.ly/SyndeyTestersTshirt
- 62. THANK YOU! • Sven Ruppert • Developer Advocate – DevSecOps • Jfrog Inc • Twitter: @SvenRuppert • eMail: [email protected]