![Can I see an example? 6
■ Here’s a simple method with a test public int Subtract(int a, int b)
{
return a – b;
}
[Test]
public void TestSubtract()
{
Assert.AreEqual(1, Subtract(3, 2));
}
Test passes](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-29-320.jpg)
![Can I see an example? 6
■ Here’s a simple method with a test public int Subtract(int a, int b)
{
return a – b;
■ We’re going to mutate the arithmetic }
operator, replacing it with addition and
[Test]
division public void TestSubtract()
{
Assert.AreEqual(1, Subtract(3, 2));
}
Test passes](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-30-320.jpg)
![Can I see an example? 6
■ Here’s a simple method with a test public int Subtract(int a, int b)
{
return a + b;
■ We’re going to mutate the arithmetic }
operator, replacing it with addition and
[Test]
division public void TestSubtract()
{
■ Let’s try addition: 3 + 2 = 5, so the }
Assert.AreEqual(1, Subtract(3, 2));
test fails—which is good
Test fails](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-31-320.jpg)
![Can I see an example? 6
■ Here’s a simple method with a test public int Subtract(int a, int b)
{
return a / b;
■ We’re going to mutate the arithmetic }
operator, replacing it with addition and
[Test]
division public void TestSubtract()
{
■ Let’s try addition: 3 + 2 = 5, so the }
Assert.AreEqual(1, Subtract(3, 2));
test fails—which is good
Test passes
■ Let’s try division: 3 / 2 = 1 (remember,
this is integer division), so the test
passes—which is bad](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-32-320.jpg)
![What does that show us? 7
■ The division mutant survived public int Subtract(int a, int b)
{
return a – b;
■ The existing test (suite) is inadequate }
[Test]
■ We need to improve the test(s)—the public void TestSubtract()
current test is far too simplistic with {
Assert.AreEqual(1, Subtract(3, 2));
just one simple assertion }
Unit test passes
Mutation testing fails](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-36-320.jpg)
![What does that show us? 7
■ The division mutant survived public int Subtract(int a, int b)
{
return a – b;
■ The existing test (suite) is inadequate }
[Test]
■ We need to improve the test(s)—the public void TestSubtract()
current test is far too simplistic with {
Assert.AreEqual(1, Subtract(3, 2));
just one simple assertion }
■ We add a second test assertion
Unit test passes
Mutation testing fails](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-37-320.jpg)
![What does that show us? 7
■ The division mutant survived public int Subtract(int a, int b)
{
return a – b;
■ The existing test (suite) is inadequate }
■ We need to improve the test(s)—the [Test]
public void TestSubtract()
current test is far too simplistic with {
Assert.AreEqual(1, Subtract(3, 2));
just one simple assertion Assert.AreEqual(5, Subtract(3, -2));
}
■ We add a second test assertion
Unit test passes
■ Here is the fixed test that now kills the Mutation testing passes
division mutant](https://image.slidesharecdn.com/anintroductiontomutationtesting-120822095633-phpapp02/85/An-introduction-to-mutation-testing-38-320.jpg)
An introduction to mutation testing
- 1. An introduction to mutation testing 1 David Musgrove Lead developer of NinjaTurtles Author of Tests and testability blog
- 2. Overview 2
- 3. Overview What is mutation testing? 2
- 4. Overview What is mutation testing? How do we do mutation testing? 2
- 5. Overview What is mutation testing? How do we do mutation testing? Why do we do mutation testing? 2
- 6. Overview What is mutation testing? How do we do mutation testing? Why do we do mutation testing? What are the disadvantages? 2
- 7. Overview What is mutation testing? How do we do mutation testing? Why do we do mutation testing? What are the disadvantages? Conclusions 2
- 8. What is mutation testing? 3
- 9. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail
- 10. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail ■ Failing is the new passing—the mutant should break the tests
- 11. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail ■ Failing is the new passing—the mutant should break the tests ■ The tests collectively should fail, not every individual test
- 12. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail ■ Failing is the new passing—the mutant should break the tests ■ The tests collectively should fail, not every individual test ■ It indicates how well your code is covered by your suite of tests
- 13. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail ■ Failing is the new passing—the mutant should break the tests ■ The tests collectively should fail, not every individual test ■ It indicates how well your code is covered by your suite of tests ■ Mutation testing doesn’t test your code, it tests your tests
- 14. What is mutation testing? 3 ■ It is the technique of running your tests over deliberately broken versions (mutants) of your code to make sure they fail ■ Failing is the new passing—the mutant should break the tests ■ The tests collectively should fail, not every individual test ■ It indicates how well your code is covered by your suite of tests ■ Mutation testing doesn’t test your code, it tests your tests ■ It is not new—the concept was introduced by Richard Lipton in 1971
- 15. How do we “break” the code? 4
- 16. How do we “break” the code? 4 ■ Example: swap arithmetic operators +, –, x, ÷, modulo
- 17. How do we “break” the code? 4 ■ Example: swap arithmetic operators +, –, x, ÷, modulo ■ Errors can be introduced, for example division by zero—but this should just produce a test failure, which is fine
- 18. How do we “break” the code? 4 ■ Example: swap arithmetic operators +, –, x, ÷, modulo ■ Errors can be introduced, for example division by zero—but this should just produce a test failure, which is fine ■ Example: swap comparison operators ==, !=, <=, <, >=, >
- 19. How do we “break” the code? 4 ■ Example: swap arithmetic operators +, –, x, ÷, modulo ■ Errors can be introduced, for example division by zero—but this should just produce a test failure, which is fine ■ Example: swap comparison operators ==, !=, <=, <, >=, > ■ Can introduce invariants, for example == and >= can be equivalent checking the upper limit of a loop counter, so handle with care
- 20. How do we “break” the code? 4 ■ Example: swap arithmetic operators +, –, x, ÷, modulo ■ Errors can be introduced, for example division by zero—but this should just produce a test failure, which is fine ■ Example: swap comparison operators ==, !=, <=, <, >=, > ■ Can introduce invariants, for example == and >= can be equivalent checking the upper limit of a loop counter, so handle with care ■ Example: delete “statements” (e.g. .NET IL sequence points)
- 21. Then what? 5
- 22. Then what? 5 ■ We save the mutated code and run the test suite against it
- 23. Then what? 5 ■ We save the mutated code and run the test suite against it ■ We hope the test suite fails and the mutant is “killed”
- 24. Then what? 5 ■ We save the mutated code and run the test suite against it ■ We hope the test suite fails and the mutant is “killed” ■ If the mutant survives, then further investigation is necessary
- 25. Then what? 5 ■ We save the mutated code and run the test suite against it ■ We hope the test suite fails and the mutant is “killed” ■ If the mutant survives, then further investigation is necessary ■ We may have to add tests to kill the mutant
- 26. Then what? 5 ■ We save the mutated code and run the test suite against it ■ We hope the test suite fails and the mutant is “killed” ■ If the mutant survives, then further investigation is necessary ■ We may have to add tests to kill the mutant ■ We may have to refactor the code to kill the mutant
- 27. Then what? 5 ■ We save the mutated code and run the test suite against it ■ We hope the test suite fails and the mutant is “killed” ■ If the mutant survives, then further investigation is necessary ■ We may have to add tests to kill the mutant ■ We may have to refactor the code to kill the mutant ■ Both are related to TDD—red, green refactor
- 28. Can I see an example? 6
- 29. Can I see an example? 6 ■ Here’s a simple method with a test public int Subtract(int a, int b) { return a – b; } [Test] public void TestSubtract() { Assert.AreEqual(1, Subtract(3, 2)); } Test passes
- 30. Can I see an example? 6 ■ Here’s a simple method with a test public int Subtract(int a, int b) { return a – b; ■ We’re going to mutate the arithmetic } operator, replacing it with addition and [Test] division public void TestSubtract() { Assert.AreEqual(1, Subtract(3, 2)); } Test passes
- 31. Can I see an example? 6 ■ Here’s a simple method with a test public int Subtract(int a, int b) { return a + b; ■ We’re going to mutate the arithmetic } operator, replacing it with addition and [Test] division public void TestSubtract() { ■ Let’s try addition: 3 + 2 = 5, so the } Assert.AreEqual(1, Subtract(3, 2)); test fails—which is good Test fails
- 32. Can I see an example? 6 ■ Here’s a simple method with a test public int Subtract(int a, int b) { return a / b; ■ We’re going to mutate the arithmetic } operator, replacing it with addition and [Test] division public void TestSubtract() { ■ Let’s try addition: 3 + 2 = 5, so the } Assert.AreEqual(1, Subtract(3, 2)); test fails—which is good Test passes ■ Let’s try division: 3 / 2 = 1 (remember, this is integer division), so the test passes—which is bad
- 33. What does that show us? 7
- 34. What does that show us? 7 ■ The division mutant survived
- 35. What does that show us? 7 ■ The division mutant survived ■ The existing test (suite) is inadequate
- 36. What does that show us? 7 ■ The division mutant survived public int Subtract(int a, int b) { return a – b; ■ The existing test (suite) is inadequate } [Test] ■ We need to improve the test(s)—the public void TestSubtract() current test is far too simplistic with { Assert.AreEqual(1, Subtract(3, 2)); just one simple assertion } Unit test passes Mutation testing fails
- 37. What does that show us? 7 ■ The division mutant survived public int Subtract(int a, int b) { return a – b; ■ The existing test (suite) is inadequate } [Test] ■ We need to improve the test(s)—the public void TestSubtract() current test is far too simplistic with { Assert.AreEqual(1, Subtract(3, 2)); just one simple assertion } ■ We add a second test assertion Unit test passes Mutation testing fails
- 38. What does that show us? 7 ■ The division mutant survived public int Subtract(int a, int b) { return a – b; ■ The existing test (suite) is inadequate } ■ We need to improve the test(s)—the [Test] public void TestSubtract() current test is far too simplistic with { Assert.AreEqual(1, Subtract(3, 2)); just one simple assertion Assert.AreEqual(5, Subtract(3, -2)); } ■ We add a second test assertion Unit test passes ■ Here is the fixed test that now kills the Mutation testing passes division mutant
- 39. How do we do mutation testing? 8
- 40. How do we do mutation testing? 8 ■ Can be applied at source code, intermediate code (Java ByteCode, Microsoft IL) or compiled code level
- 41. How do we do mutation testing? 8 ■ Can be applied at source code, intermediate code (Java ByteCode, Microsoft IL) or compiled code level ■ For Java and .NET, intermediate code is easiest to target in an automated solution as it has a relatively small set of well understood instructions and doesn’t require code parsing
- 42. How do we do mutation testing? 8 ■ Can be applied at source code, intermediate code (Java ByteCode, Microsoft IL) or compiled code level ■ For Java and .NET, intermediate code is easiest to target in an automated solution as it has a relatively small set of well understood instructions and doesn’t require code parsing ■ Can be automated using a mutation testing tool
- 43. What tools are available? 9
- 44. What tools are available? 9 ■ There are a number of abortive open source attempts, or skeleton projects; only a few projects are actively maintained and developed
- 45. What tools are available? 9 ■ There are a number of abortive open source attempts, or skeleton projects; only a few projects are actively maintained and developed ■ The following show signs of meaningful activity within the last six months at the time of writing (August 2012)
- 46. What tools are available? 9 ■ There are a number of abortive open source attempts, or skeleton projects; only a few projects are actively maintained and developed ■ The following show signs of meaningful activity within the last six months at the time of writing (August 2012) ■ Java ByteCode mutation: PIT, Javalanche
- 47. What tools are available? 9 ■ There are a number of abortive open source attempts, or skeleton projects; only a few projects are actively maintained and developed ■ The following show signs of meaningful activity within the last six months at the time of writing (August 2012) ■ Java ByteCode mutation: PIT, Javalanche ■ .NET IL: NinjaTurtles
- 48. Why do we do mutation testing? 10
- 49. Why do we do mutation testing? 10 ■ I like to think of it as a way of measuring a team’s test-drivenness
- 50. Why do we do mutation testing? 10 ■ I like to think of it as a way of measuring a team’s test-drivenness ■ Test-driven development (TDD): write test, write code to make it pass, refactor (red, green, refactor)
- 51. Why do we do mutation testing? 10 ■ I like to think of it as a way of measuring a team’s test-drivenness ■ Test-driven development (TDD): write test, write code to make it pass, refactor (red, green, refactor) ■ No code should be written that isn’t there to make a test pass
- 52. Why do we do mutation testing? 10 ■ I like to think of it as a way of measuring a team’s test-drivenness ■ Test-driven development (TDD): write test, write code to make it pass, refactor (red, green, refactor) ■ No code should be written that isn’t there to make a test pass ■ Therefore if you mutate any code you should make a test fail
- 53. Why do we do mutation testing? 10 ■ I like to think of it as a way of measuring a team’s test-drivenness ■ Test-driven development (TDD): write test, write code to make it pass, refactor (red, green, refactor) ■ No code should be written that isn’t there to make a test pass ■ Therefore if you mutate any code you should make a test fail ■ So perfect TDD implies 100% of (meaningful) mutants killed
- 54. YAMM—yet another maturity model 11
- 55. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users
- 56. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users ■ Level 1: code is tested by dedicated testers before release
- 57. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users ■ Level 1: code is tested by dedicated testers before release ■ Level 2: code is also tested by the developers; some automation
- 58. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users ■ Level 1: code is tested by dedicated testers before release ■ Level 2: code is also tested by the developers; some automation ■ Level 3: tests are written first and automated as far as is possible
- 59. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users ■ Level 1: code is tested by dedicated testers before release ■ Level 2: code is also tested by the developers; some automation ■ Level 3: tests are written first and automated as far as is possible ■ Level 4: mutation testing is applied to verify efficacy of test suites
- 60. YAMM—yet another maturity model 11 ■ Level 0: code is tested when it is released, by the users ■ Level 1: code is tested by dedicated testers before release ■ Level 2: code is also tested by the developers; some automation ■ Level 3: tests are written first and automated as far as is possible ■ Level 4: mutation testing is applied to verify efficacy of test suites ■ This could be a tree, with other ideas like BDD introduced, but it illustrates the position of mutation testing
- 61. What are the disadvantages? 12
- 62. What are the disadvantages? 12 ■ Good tooling is only just emerging
- 63. What are the disadvantages? 12 ■ Good tooling is only just emerging ■ It’s relatively slow and computationally expensive
- 64. What are the disadvantages? 12 ■ Good tooling is only just emerging ■ It’s relatively slow and computationally expensive ■ Each line of code can produce many mutants
- 65. What are the disadvantages? 12 ■ Good tooling is only just emerging ■ It’s relatively slow and computationally expensive ■ Each line of code can produce many mutants ■ Each mutant requires a suite of tests to be run
- 66. What are the disadvantages? 12 ■ Good tooling is only just emerging ■ It’s relatively slow and computationally expensive ■ Each line of code can produce many mutants ■ Each mutant requires a suite of tests to be run ■ But the suite of tests can be narrowed down, and it only needs to be run until the first failure is encountered
- 67. What are the disadvantages? 12 ■ Good tooling is only just emerging ■ It’s relatively slow and computationally expensive ■ Each line of code can produce many mutants ■ Each mutant requires a suite of tests to be run ■ But the suite of tests can be narrowed down, and it only needs to be run until the first failure is encountered ■ And it’s eminently parallelisable for multicore PCs
- 68. What are the disadvantages? 13
- 69. What are the disadvantages? 13 ■ A passing mutation test can fail fast, but a failing mutation test is likely to take significantly longer as it runs the entire set of relevant tests
- 70. What are the disadvantages? 13 ■ A passing mutation test can fail fast, but a failing mutation test is likely to take significantly longer as it runs the entire set of relevant tests ■ So should be applied continuously to keep quality high
- 71. What are the disadvantages? 13 ■ A passing mutation test can fail fast, but a failing mutation test is likely to take significantly longer as it runs the entire set of relevant tests ■ So should be applied continuously to keep quality high ■ Can only realistically be applied in this way to code bases and test suites that are already in fairly good shape
- 72. A shameless plug for NinjaTurtles 14
- 73. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased
- 74. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased ■ Open source tool working at a .NET IL level
- 75. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased ■ Open source tool working at a .NET IL level ■ Includes seven different types of mutation at time of writing
- 76. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased ■ Open source tool working at a .NET IL level ■ Includes seven different types of mutation at time of writing ■ Compatible with existing .NET languages and unit test frameworks
- 77. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased ■ Open source tool working at a .NET IL level ■ Includes seven different types of mutation at time of writing ■ Compatible with existing .NET languages and unit test frameworks ■ Optimisations include parallelised test runs, reduced test suites, fail fast (where the underlying test runner supports it)
- 78. A shameless plug for NinjaTurtles 14 ■ Disclaimer: I’m the lead developer on this so I’m biased ■ Open source tool working at a .NET IL level ■ Includes seven different types of mutation at time of writing ■ Compatible with existing .NET languages and unit test frameworks ■ Optimisations include parallelised test runs, reduced test suites, fail fast (where the underlying test runner supports it) ■ Console runner gives immediate results
- 79. Sample NinjaTurtles output 15
- 80. Sample NinjaTurtles output 15 ■ Simple console command applied to one class within the NinjaTurtles code base
- 81. Sample NinjaTurtles output 15 ■ Simple console command applied to one class within the NinjaTurtles code base ■ Over 1,000 runs of the test suite, each run as a separate process
- 82. Sample NinjaTurtles output 15 ■ Simple console command applied to one class within the NinjaTurtles code base ■ Over 1,000 runs of the test suite, each run as a separate process ■ Test results returned in under 90 seconds
- 83. Sample NinjaTurtles output 15 ■ Simple console command applied to one class within the NinjaTurtles code base ■ Over 1,000 runs of the test suite, each run as a separate process ■ Test results returned in under 90 seconds ■ Also XML or HTML output
- 84. Conclusions 16
- 85. Conclusions 16 ■ Mutation testing has a lot to offer teams looking to constantly improve the way they do things
- 86. Conclusions 16 ■ Mutation testing has a lot to offer teams looking to constantly improve the way they do things ■ Test code is a first class citizen—it deserves testing too
- 87. Conclusions 16 ■ Mutation testing has a lot to offer teams looking to constantly improve the way they do things ■ Test code is a first class citizen—it deserves testing too ■ Computing power and quality of tooling are improving so mutation testing is a realistic tool to use day to day
- 88. Conclusions 16 ■ Mutation testing has a lot to offer teams looking to constantly improve the way they do things ■ Test code is a first class citizen—it deserves testing too ■ Computing power and quality of tooling are improving so mutation testing is a realistic tool to use day to day ■ Passing mutation tests run a lot quicker than failing ones, so this should only be applied by relatively “mature” development teams
- 89. Conclusions 16 ■ Mutation testing has a lot to offer teams looking to constantly improve the way they do things ■ Test code is a first class citizen—it deserves testing too ■ Computing power and quality of tooling are improving so mutation testing is a realistic tool to use day to day ■ Passing mutation tests run a lot quicker than failing ones, so this should only be applied by relatively “mature” development teams ■ The barrier to entry is low, so you can start doing this today
- 90. Links and resources NinjaTurtles: http://www.mutation-testing.net Tests and testability: http://davidmusgrove.com/blog Email: david at davidmusgrove.com 17
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n
- #85: \n
- #86: \n
- #87: \n
- #88: \n
- #89: \n
- #90: \n
- #91: \n
- #92: \n
- #93: \n