Test-Driven Development.pptx
- 2. Outline ● What is TDD and why use it ● The TDD Process ● Effectiveness of TDD ● Designing for Testability
- 3. What is Test-Driven Development “Test-Driven Development (TDD) is a technique for building software that guides software development by writing tests.” - Martin Fowler Design Test Code Design Code Test Test-After Development Test-Driven Development
- 4. The Three Laws of TDD 1. You may not write production code unless you've written a failing test first 2. You may not write more of a unit test than is sufficient to fail 3. You may not write more production code than is sufficient to make the failing unit test pass
- 5. Why Test-Driven Development? ● Instant feedback ○ Faster debugging, more confidence ○ Squash larvae instead of hunting mature bugs ● Better development practices ○ Drive writing testable code ○ Decompose into manageable tasks ● Tests always up-to-date ○ Tests themselves are also tested! ● Increased value of tests as documentation “Test-first code tends to be more cohesive and less coupled than code in which testing isn’t a part of the intimate coding cycle” - Kent Beck
- 6. But! ● We don't have the time or money! ○ Short-term loss, long-term gain ○ Bugs in production are costly ● I am already writing unit tests… ○ Sometimes happy path only ○ In practice fewer total tests ● TDD feels too restricting and unnatural ○ Rules can be adjusted Initial Investment Waterfal l TDD Savings
- 8. The TDD process LED example ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 9. The TDD process James W. Grenning - Test Driven Development for Embedded C ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 10. The TDD process TEST(LedDriver, TurnOnLedOne) { LedDriver_TurnOn(1); TEST_ASSERT_EQUAL_HEX16(1,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { } 1 Tests 1 Failures 0 Ignored ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 11. The TDD process TEST(LedDriver, TurnOnLedOne) { LedDriver_TurnOn(1); TEST_ASSERT_EQUAL_HEX16(1,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress = 1; } ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 12. The TDD process TEST(LedDriver, TurnOnLedOne) { LedDriver_TurnOn(1); TEST_ASSERT_EQUAL_HEX16(1,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress = 1; } 1 Tests 0 Failures 0 Ignored ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 13. The TDD process TEST(LedDriver, TurnOnMultipleLeds) { LedDriver_TurnOn(8); LedDriver_TurnOn(9); TEST_ASSERT_EQUAL_HEX16(0x180,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress = 1; } 2 Tests 1 Failures 0 Ignored ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 14. The TDD process TEST(LedDriver, TurnOnMultipleLeds) { LedDriver_TurnOn(8); LedDriver_TurnOn(9); TEST_ASSERT_EQUAL_HEX16(0x180,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress |= 1 << (ledNumber - 1); } ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 15. The TDD process TEST(LedDriver, TurnOnMultipleLeds) { LedDriver_TurnOn(8); LedDriver_TurnOn(9); TEST_ASSERT_EQUAL_HEX16(0x180,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress |= 1 << (ledNumber - 1); } 2 Tests 0 Failures 0 Ignored ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 16. The TDD process TEST(LedDriver, TurnOnMultipleLeds) { LedDriver_TurnOn(8); LedDriver_TurnOn(9); TEST_ASSERT_EQUAL_HEX16(0x180,virtualLeds); } void LedDriver_TurnOn(int ledNumber) { *ledsAddress |= LedNumberToBit(ledNumber); } ● Define a test list ● Add a small failing test ● Implement minimal code ● Make the test pass ● Refactor
- 18. What do research studies say? ● Microsoft and IBM Case study ○ 15-35% initial time increase, 40-90% less defects ● George and Williams, Professional Pair programmers Java bowling game ○ 16% slower, 18% more test cases ● Choma, Study on developers perception with ~10 years of experience ○ 96% → reduces debugging effort ○ 92% → higher quality code ○ 71% → noticeably effective https://www.microsoft.com/en-us/research/wp-content/uploads/2009/10/Realizing-Quality-Improvement- Through-Test-Driven-Development-Results-and-Experiences-of-Four-Industrial-Teams-nagappan_tdd.pdf https://dl.acm.org/doi/10.1145/952532.952753 https://link.springer.com/chapter/10.1007/978-3-319-91602-6_5 Notable Studies
- 19. What do research studies say? ● Internal Quality: ● Weighted methods per class, Depth of inheritance tree, Number of children, Coupling between objects, Lack of cohesion in methods ● External Quality: Test cases passed, Defect density, Defects per test, Effort required to fix defects, Change density, Percentage of preventative changes ● Productivity: Amount of code/features produced per development effort ● Test Quality: Test density, Test coverage, Test productivity Metrics https://www.researchgate.net/publication/258126622_How_Effective_is_Test_Driven_Development https://arxiv.org/pdf/1711.05082.pdf
- 20. What do research studies say? ● Research results lack a definitive conclusion ● Increased test coverage, decreased defect density ● Productivity: inconclusive, short-term loss long-term gain? Industrial Semi-Industrial Academic Internal Quality o o o External Quality + + o Test Quality o + + Productivity — o + Results https://www.researchgate.net/publication/258126622_How_Effective_is_Test_Driven_Development https://arxiv.org/pdf/1711.05082.pdf Metric Field
- 21. ● Initial productivity loss → better external quality ● TDD developers get faster over time ● Confidence increases due to broad test coverage ● TDD requires monitoring adjusting dosage accordingly “somewhere around the 2-years in the mark, something magical started to happen: I started coding faster with unit tests than I ever did without them” - Eric Elliott Takeaways What do research studies say? ● TDD creates tests as documentation ○ Shorten onboarding and handoffs? ○ Increased resilience to losing knowledgeable people? ● Effect on job satisfaction and developer retention?
- 22. What do research studies say? ● Metric definitions lack detail or unrepresentative ● Limited scope and size of projects ● Different language, environment, and domain context ● Difficult to measure TDD adherence ● Differences in programmer skill levels Issues "TDD improves code quality" source: https://ieeexplore.ieee.org/document/5463691
- 23. ● Not a panacea nor failproof ● Steep learning curve, adapt mindest ● Can be difficult to predict course ● False sense of security ● Difficult to use in some situations Limitations “TDD helps with, but does not guarantee, good design and good code. Skill, talent, and expertise remain necessary” - Esko Luontola Possible variation: ● Code up fast prototypes for exploration to be thrown away (Spike) ● Write down important test cases and observations
- 25. The TDD process Real code has dependencies → break dependencies: ● Hardware independence ○ Stand-in for expensive hardware ● Inject difficult to produce inputs ○ Network failure ● Speed up a slow collaborator ○ Database ● Replace something under development ○ Software library Designing for Testability External Dependencies "Pull the plug now, Harry!"
- 26. The TDD process Test Doubles: ● Dummy, Stub, Spy, Mock, Fake ● Allow us to independently test application code Designing for Testability External Dependencies
- 27. The TDD process Test Doubles Designing for Testability External Dependencies Simple Complex Dummy Never called, allows program to compile Stub Returns a value as directed by test case Spy Returns a value and verifies parameters passed Mock Verifies function calls, call order, and parameters passed Fake Partial implementation of a real component
- 28. The TDD process ● Prescribe which calls to expect, then execute process Designing for Testability Mocking TEST(Flash, WriteSucceeds_ReadyImmediately) { MockIO_Expect_Write(CommandRegister, ProgramCommand); MockIO_Expect_Write(address, data); MockIO_Expect_ReadThenReturn(StatusRegister, ReadyBit); MockIO_Expect_ReadThenReturn(address, data); result = Flash_Write(address, data); LONGS_EQUAL(FLASH_SUCCESS, result); }
- 29. The TDD process ● Mocks should be simple ○ Complex mocks hard to read and maintain ● Heavy mocking can lead to brittle tests ○ When implementation changes tests need update ● Mocks can lead to overconfidence ○ Mocks may mask integration issues Designing for Testability Mocking Challenges
- 30. The TDD process ● SOLID principles keep modules flexible and testable ○ Dependency Inversion, Open-Closed, Liskov Substitution ● LightController should not know about concrete drivers Designing for Testability Interfaces
- 31. The TDD process ● Test-drive the interface before the internals ● Tests should test a single concept ● Focus on tests that increase confidence ● Legacy code: add tests before modification TDD Best Practices
- 32. The TDD process Acceptance Test-Driven Development ● Collaboratively define acceptance tests ● Focus on capturing the business requirements TDD Extensions Behaviour-Driven Development ● Define system behaviour from perspective of stakeholders ● Given-When-Then Extend TDD by involving different stakeholders TEST(LightScheduler, ScheduleOffWeekendAndItsSaturdayAndItsTime) checkLightState(lightNumber, LIGHT_OFF); then
- 33. The Future of TDD with AI? ● Use tests to communicate system requirements to AI ● Allow non-technical people to specify desired behavior ● The Two Disks Parable https://drpicox.medium.com/the-two-disks-parable-ac1a16803c58
- 34. In Summary ● TDD has potential but requires monitoring ● Reliant on developer experience and motivation ● No clear consensus regarding internal quality ● Better suited for some contexts than others
- 35. Questions! TDD?
Editor's Notes
- #2: Also known as TDD We're going to discuss what it is and what it is not, and also what it's good at and what it's not good at
- #3: roughly divided into 4 section We start with the general introduction what it is and why should care Quickly go through TDD process with an example as a refresher After that we'll look at some empirical studies on TDD and its effectiveness Lastly, I will go over how some more advanced topics regarding design and testability
- #4: So, What is test-driven development Martin Fowler gave following definition states that TDD is a method for developing software, NOT just testing software, which is guided by writing tests In the traditional test-after approach implementation first but with TDD we write our tests before the implementation so already test in place that tell you what the implementation should do
- #5: Proper TDD adheres to the following three laws. 1 2 this last point is where a lot of developers get disturbed sometimes this means writing an implementation that you know is wrong but it still allows you to pass the current test and your future tests will have to make sure you eventually come to the correct implementation
- #6: TODO: Image of crane building here TDD is really about steady, incremental progress. Whereas in test-after we might implement and test a big chunk of work at once, in TDD we do it in small but confident steps. TDD reduces the likelihood that we need to go back and fix things and also makes it easier to fix things, since we know the last change broke something.
- #7: Now that we have a definition for TDD, let's look at why you would use it in the first place Just to be clear, this presentation is not going to be a pure advertisement for TDD. Rather, I want to discuss the pros and cons One of the main benefits is Instant feedback: we want our feedback loops to be a short as possible Running test frequently, catch defects early, last change, more confidence as tests are passing If you write a big chunk of implementation first, it can be hard to properly test all of your code By writing the tests first you are driven to write testable code from the start, instead of trying to squeeze them in later It also encourages decomposing the problem into manageable tasks, one small test at a time Safer refactoring as you have already tests in place While non-TDD tests can also serve as documentation TDD tests may do a better job at capturing the original intentions When you go into new code you don't know much about implementation yet Non-TDD test may focus more on validating the code rather than specifying its requirements.
- #8: These are some common objections to TDD Most and foremost, we don't have the time or money for TDD With TDD you have an initial investment, but this should eventually pay out in the long run because of less debugging And bugs in production can be very costly Another argument is, I am already writing unit tests! Tests are usually different with test after, Often just the happy path is tested Also, in practice test-after usually ends up with less tests than TDD After the product has been implemented and shipped it's unlikely unit tests are going to be written For some people TDD feels to restricting, but like any other tool, there is some leeway and it's okay to adjust the granularity a bit to your liking Later on we're going to discuss the actual effects and costs
- #9: now, A quick refresher on the TDD process
- #10: For this example we have a an array of LEDs and we want to implement some driver functionality with TDD
- #11: The first step is to define a test list With all the relevant tests that you can think of currently These can also be deduced from a requirement specification May evolve as we implement features
- #12: By failing the test first we also confirm that the test doesn't give a false positive, thus we test the test These simple implementations test our tests. Watching the test case fail shows that the test can detect a wrong result. EXTRA: In essence, we’re closing a vice around the code under test, holding the behavior steady (see the sidebar on the following page). Don’t worry, the production code won’t be hard-coded and incomplete for long. As soon as you need to turn on a different LED, the hard-coded value will have to go. The real implementation is not much more diffi- cult, but I ask you to resist the temptation to put in more code than is needed by the current test. We’re evolving the design. The problem with adding more code than the current tests require is that you probably won’t write all the tests you need to keep future, and present, bugs out of the code. Adding code before it is needed by the tests adds complexity. Sometimes you will be wrong about the need, resulting in carrying the complexity unnecessarily. Also, there is no end to the thinking “I will need it.” Where should you stop? In practicing TDD, we stop when the code is not needed by the current tests. Loose ends are cataloged in the test list. TDD is structured procrastination. Put off writing the right production code until the tests force us to. Implementation completeness, the ulti- mate objective, is reached only after all the correct tests are in place.
- #13: Hard-coding the right answer shows that the test case can detect the right result. The test is right and valuable, even though the production code is incomplete. The problem with adding more code than the current tests require is that you probably won’t write all the tests you need to keep future, and present, bugs out of the code. In this simple example it may be obvious, but for more complex cases it may not be Adding code before it is needed by the tests adds complexity. Sometimes you will be wrong about the need, resulting in carrying the complexity unnecessarily. EXTRA: In essence, we’re closing a vice around the code under test, holding the behavior steady (see the sidebar on the following page). Don’t worry, the production code won’t be hard-coded and incomplete for long. As soon as you need to turn on a different LED, the hard-coded value will have to go. The real implementation is not much more diffi- cult, but I ask you to resist the temptation to put in more code than is needed by the current test. We’re evolving the design. The problem with adding more code than the current tests require is that you probably won’t write all the tests you need to keep future, and present, bugs out of the code. Adding code before it is needed by the tests adds complexity. Sometimes you will be wrong about the need, resulting in carrying the complexity unnecessarily. Also, there is no end to the thinking “I will need it.” Where should you stop? In practicing TDD, we stop when the code is not needed by the current tests. Loose ends are cataloged in the test list. TDD is structured procrastination. Put off writing the right production code until the tests force us to. Implementation completeness, the ulti- mate objective, is reached only after all the correct tests are in place.
- #14: Not much to refactor yet, so add next test
- #15: For the next tests we turn on multiple random LEDS And now we see that our previous implementation was wrong
- #16: At this point, the easiest way is simply to add the correct implementation
- #17: The Tests Are Right With the implementation being incomplete, you might think that noth- ing is being tested. Big deal! The test makes sure that a variable is set to one! Try to think about it a different way. The tests are right! They are a very valuable by-product of TDD. These simple implementations test our tests. Watching the test case fail shows that the test can detect a wrong result. Hard-coding the right answer shows that the test case can detect the right result. The test is right and valuable, even though the production code is incomplete. Later, as the implementation evolves, these seemingly trivial tests will test important behavior and boundary conditions. In essence, we’re closing a vice around the code under test, holding the behavior steady (see the sidebar on the following page). Don’t worry, the production code won’t be hard-coded and incomplete for long. As soon as you need to turn on a different LED, the hard-coded value will have to go. The real implementation is not much more diffi- cult, but I ask you to resist the temptation to put in more code than is needed by the current test. We’re evolving the design. The problem with adding more code than the current tests require is that you probably won’t write all the tests you need to keep future, and present, bugs out of the code. Adding code before it is needed by the tests adds complexity. Sometimes you will be wrong about the need, resulting in carrying the complexity unnecessarily. Also, there is no end to the thinking “I will need it.” Where should you stop? In practicing TDD, we stop when the code is not needed by the current tests. Loose ends are cataloged in the test list. TDD is structured procrastination. Put off writing the right production code until the tests force us to. Implementation completeness, the ulti- mate objective, is reached only after all the correct tests are in place.
- #18: And only when the tests pass do we refactor, so we add a helper function to clean up the code
- #19: Alright, now that we all know what TDD entails, I want to discuss what research has to say about TDD and its effectiveness
- #20: First just quickly a couple of notable studies Microsoft and IBM did a case study with multiple developments teams Development time increased somewhat (may be due to initial cost) but significantly less defects microsoft ibm teams agreed: more time, but offset by less bugs which means less time Another study by George and Williams had Professional pair programmers develop bowling game where the control group used waterfall Their results showed again lower development but more tests cases And Choma did research focusing on the developer's perception of TDD with on average 10 years experience And overall the developers were quite positive towards TDD
- #21: Maybe before looking at more results, maybe it is good to look at some metrics We have the internal quality which mostly relates to the intrinsic quality of the software itself as defined by some common measures such as Then we have external quality which says more about the performance and output of the process How much time and effort did it take And lastly test quality with metrics such as ..
- #22: Here I have aggregate the results of two comparative studies which each compared several different studies on TDD Where the four metrics are evaluated based on the environment We have industrial, we have semi-industrial, which involves either professional in a controlled setting or students in a industrial setting and then we have academic experiments There is no obvious consensus in the research However, test coverage and defect density were fairly consistent over all studies When it comes to productivity the results are quite inconclusive it's hard to say something over the very long-term
- #23: So mainly there seems to be an initial productivity loss in exchange for better external quality TDD is really something you have to learn: Our findings suggest, after overcoming initial difficulties: understand where to start, how to create a test for a feature that does not yet exist, participants gain greater confidence to implement new features and make changes due to broad test coverage. actionable advice, carefully monitoring, increasing or decreasing the dosage accordingly Erick Elliot, author of the book 'composing software', stated that after a magical 2-year mark he started writing faster with unit tests Then there are a few open questions which have not been studied yet as they are hard to measure but might still play a role For example, TDD tests may shorten developer onboarding or codebase handoffs? And lastly,
- #24: There are definitely some issues with the conducted research works on TDD metrics used for describing the findings have not been either defined in detail or lack the quality attribute they should be presenting. Another problem: small scope of many experiments consisted of small tasks Different studies used different languages.. The environment and domain context not always specified And it is also difficult to measure how well developers are following the TDD rules, which has a big impact on how effective TDD is Much of the inconsistency likely can be attributed to internal factors not fully described in the TDD trials. Thus, TDD is bound to remain a controversial topic of debate and research.
- #25: TDD definitely has a number of limiations First, TDD is not a panacea nor failproof, it still relies a lot on skill and experience of the developer TDD also has a steep learning curve and often requires developer to adapt a different mindset Many experienced developers have a mental model of the system they are building, TDD might interfere with this Can be difficult to predict course, more costly to throw away all tests when wrong, planning ahead too much False sense of security, if the tests are not sound then test coverage has little value Can be difficult to use in situations like GUIs, Relational Databases, Web Service
- #26: Alright, now let's look at how we can make TDD more effective and also deal with real-world dependencies
- #27: In the real world, our code often has external dependencies Hardware independence: especially useful for embedded system where your board may not have arrived yet or you have limited access to hardware It also allows us to inject inputs which are difficult to produce For example if we want to simulate a network failure, we could ask our colleague to pull the plug at a very specific moment However, this is hardly reproducible and can take a lot of time Also may want to break dependance on a slow collaborator as the speed of our tests is important we want our tests to run often and run fast Lastly, you may want to replace something
- #28: In order to break external dependencies we can use test doubles We can replace our dependent-on-components with test doubles This allows us to independently test the application code without relying on the any dependencies
- #29: There are some different definitions of test doubles going around but I am going to stick with these which are roughly ordered in terms of increasing complexity. A stub is a very simple entry point that returns a value Stub: e.g. last thrown exception Fake: e.g. in-memory database
- #31: Complex mocks hard to read and maintain Test setup becomes more complex If the tests are heavily coupled to the tests, then tests need to be updated often It's possible that mocks are hiding intergration issues integration tests should still be included
- #32: With the former implementation on the left, every time we want to add a new driver we need to modify the light controller But if we use an interface to decouple the lightcontroller from actual driver implementations, then it becomes easier to add new drivers in the future
- #33: Legacy code: • Test-drive new code. • Add tests to legacy code before modification. • Test-drive changes to legacy code. No ramble-on tests we can test all numbers from 1 to 100 but gives little value
- #34: There are two variants with both extend TDD by putting more emphasis on involving different stakeholder such as customers, business analysts, testers in ATDD developers and stakeholders define acceptance tests together in BDD, the focus is more on the system behaviour from the stakeholders' pespective Given that the schedule is turned off for the weekend And when it's saturday And when it's normally time to turn on the lights Then lights should remain, because it's weekend
- #36: Use tests to tweak behaviour TESTS: TDD. We wrote tests first, and tests created our code. Our code exists because of these tests, and we can repeat the process. CODE: But the code would be different. Uhm. Better? The second time that I write the code, I do it better. Let’s crash the CODE and let TESTS survive.
- #37: And while there is no clear consensus regarding internal quality, research does suggest several benefits of TDD including a decreased defect rate In the end, we should keep in mind that TDD is better suited for some contexts than others