Are Your Continuous Tests Too Fragile for Agile?
2 likes1,232 views
The document discusses the challenges and best practices of implementing continuous testing within agile development, emphasizing the importance of integrating quality assurance throughout the development process. It outlines various testing methodologies, including unit, functional, and regression testing, and highlights the need for disciplined maintenance of test suites. Additionally, it addresses common pitfalls in testing, such as resistance from developers and the neglect of testing practices, and proposes a 'spring cleaning' approach to improve testing conditions.
Are Your Continuous Tests Too Fragile for Agile?
- 1. Paraso& Proprietary and Confiden1al 1 11/13/14 Are Your Con*nuous Tests Too Fragile for Agile Arthur Hicken Paraso& Evangelist
- 2. Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2 Agenda § Challenges of Con1nuous Agile § Tes1ng defined § Tes1ng is hard (no one wants to do it) § Best prac1ces § A modest proposal: Spring cleaning
- 3. Paraso& Proprietary and Confiden1al 3 Paraso& Proprietary and Confiden1al 3 The role of QA in Agile § QA is part of development, not an a&erthought § Review developer tes1ng efforts § Addi1onal tes1ng of requirements § Itera1ve during development § Fast discovery and repor1ng of defects § Inves1gate root cause of defects § Influence development policy
- 4. Paraso& Proprietary and Confiden1al 4 Paraso& Proprietary and Confiden1al 4 Challenges of Agile § Need to be fast § Need to be flexible § Tes1ng can be overlooked if not baked in § Architectural compromises from schedule
- 5. Paraso& Proprietary and Confiden1al 5 Paraso& Proprietary and Confiden1al 5 Challenges of Con1nuous § Tests must produce binary decision go/no-‐go § Reuse unit test and func1onal tests from dev to QA § High level of automa1on § Requires disciplined mature process § Tes1ng must automa&cally answer § Is it stable § Will it do what it’s supposed to do
- 6. Paraso& Proprietary and Confiden1al 6 Paraso& Proprietary and Confiden1al 6 Importance of Tes1ng § How do you know when you're done? § How do you know if a fix for a minor bug broke a major func1on of the system? § How can the system evolve into something more than is currently envisioned? § Tes1ng, both unit and func1onal, needs to be an integrated part of the development process.
- 7. Paraso& Proprietary and Confiden1al 7 Paraso& Proprietary and Confiden1al 7 Tes1ng delivers A maintained suite of unit tests: § Represents the most prac1cal design possible § Provides the best form of documenta1on for classes § Determines when a class is "done" § Gives a developer confidence in the code § Is a basis for refactoring quickly
- 8. Paraso& Proprietary and Confiden1al 8 Paraso& Proprietary and Confiden1al 8 Why Tes1ng is O&en Ignored § Perceived as “burdensome” § Too much effort § Too much 1me § We’re on a schedule – no 1me to test. § Why is there always 1me to do it over, but never to do it right? § Developer Ego § “I don’t need to test my code – I don’t make mistakes.” § Tes1ng means I have to admit that I might have made a mistake. § I don’t like to test – it’s beneath me. § Tes1ng isn’t fun – I just want to code! § That’s for the QA dept.
- 9. Paraso& Proprietary and Confiden1al 9 Paraso& Proprietary and Confiden1al 9 Defini1ons § Unit § Func1onal § Regression
- 10. Paraso& Proprietary and Confiden1al 10 Paraso& Proprietary and Confiden1al 10 Unit Tes1ng Unit tests that validates your method behaves a certain way given a specific parameters § Regressions tes1ng § Validate behavior of exis1ng code hasn’t changed § Scheduled and run automa1cally § Test Driven Development § Validate behavior of new code to be wrijen § Created before developing, turned into regression tests a&er
- 11. Paraso& Proprietary and Confiden1al 11 Paraso& Proprietary and Confiden1al 11 Func1onal Tes1ng § Incarna1ons of func1onal tests § Unit tests (Junit, CppUnit, Nunit, etc) § Integra1on tests § Scripted tests § Manual tests § Ways to create func1onal tests § Manually wrijen § Machine generated from applica1on tracing § Machine generated from capturing traffic § Machine generated from recording clicks
- 12. Paraso& Proprietary and Confiden1al 12 Paraso& Proprietary and Confiden1al 12 Regression Tes1ng § Contributors to the regression test suite § Func1onal unit tests that pass § Machine generated unit tests for current behavior § Unit test wrijen by developers and testers § Tests wrijen to validate bug fixes § Regression test failures § Unwanted change in behavior – fix the code § Inten1onal change in specifica1on – update test controls
- 13. Paraso& Proprietary and Confiden1al 13 Paraso& Proprietary and Confiden1al 13 What’s the Difference? Unit Tests • The code is doing things right • Wrijen from programmers perspec1ve • Confirm code gives proper output based on specific input Func1onal Tests • The code is doing the right thing • Wrijen from users perspec1ve • Confirm the system does what the user expects it to do Regression Tests • It s1ll works
- 14. Paraso& Proprietary and Confiden1al 14 Paraso& Proprietary and Confiden1al 14 Why do Unit Tes1ng So&ware bugs are inserted during coding phase, but o&en aren’t found un1l func1onal tes1ng, or worse they are found by customers. The longer a bug stays in the system, the more expensive it is to fix it. Unit tes1ng is a strategy that allows you to find out if your system is behaving the way you expect early in the SDLC. Unit tes1ng allows for easy regression tes1ng, so you can find out quickly whether you have introduced some unexpected behavior into the system.
- 15. Paraso& Proprietary and Confiden1al 15 The Cost of Defects
- 16. Paraso& Proprietary and Confiden1al 16 Paraso& Proprietary and Confiden1al 16 Best Prac1ces
- 17. Paraso& Proprietary and Confiden1al 17 Paraso& Proprietary and Confiden1al 17 What’s Important § Unit tests require a LOT of care § Unit tes1ng is a con1nuous opera1on § Test suite maintenance is a commitment § Control data (asser1ons) in the code
- 18. Paraso& Proprietary and Confiden1al 18 Paraso& Proprietary and Confiden1al 18 Test incorpora1on § Take exis1ng unit tests and schedule them into build process § Run and fix tests as part of build process § Ideal goal: 0 failures and 100% code coverage § Realis1c goal: 10% test failures and 80% code coverage § If the unit test suite is noisy, don’t qualify the build based on asser1on failures
- 19. Paraso& Proprietary and Confiden1al 19 Paraso& Proprietary and Confiden1al 19 Unit Test Crea1on § Maintainability § Proper asser1ons § Nightly runs § Tests should be: § Automa1c (no human input or review) § Repeatable (no strange dependencies)
- 20. Paraso& Proprietary and Confiden1al 20 Paraso& Proprietary and Confiden1al 20 Legacy Test Suites § Leverage your exis1ng suite § Clean up incrementally § Set policy based on current results § Tighten policy when in compliance
- 21. Paraso& Proprietary and Confiden1al 21 Paraso& Proprietary and Confiden1al 21 Maintaining and Crea1ng UT § Maintaining is more of a problem than crea1ng. § Don’t create a test if you can’t or don’t plan to maintain it. § Crea1ng as you go creates a bejer test suite § Tests not run regularly become irrelevant
- 22. Paraso& Proprietary and Confiden1al 22 Paraso& Proprietary and Confiden1al 22 Write tests with the code § Create the test when you’re wri1ng the code § Have a test for every task you complete § Increases understanding § Improve asser1ons
- 23. Paraso& Proprietary and Confiden1al 23 Paraso& Proprietary and Confiden1al 23 Connect Unit Tests § Associate tests with Code and Requirements § Naming Conven1ons should have meaning § Use code comment tags @task @pr @fr § Use check-‐in comments @task @pr @fr § Improve change-‐based tes1ng
- 24. Paraso& Proprietary and Confiden1al 24 Paraso& Proprietary and Confiden1al 24 Specific Genera1on § Don’t be lured by 100% coverage § Auto asser1ons are not meaningful unless validated § Review the tests as you work on the code
- 25. Paraso& Proprietary and Confiden1al 25 Paraso& Proprietary and Confiden1al 25 Code Review Test § Code for unit tests should be reviewed § Consider doing test-‐design review § Ensures you’re tes1ng what you think you are § Helps reduce noise § Helps improve understanding § Review the code and the test code together
- 26. Paraso& Proprietary and Confiden1al 26 Paraso& Proprietary and Confiden1al 26 Tes1ng Excep1on Handling § Frequently Skipped § Prevents crashes § Prevents death spiral § Rule of thirds § 1/3 basic func1onality § 1/3 advanced features / infrequent serngs § 1/3 error handling
- 27. Paraso& Proprietary and Confiden1al 27 Paraso& Proprietary and Confiden1al 27 All About Asser1ons § Verifies that the code behaves as intended § True/false § Null or not null § Specific values § Complex condi1ons
- 28. Paraso& Proprietary and Confiden1al 28 Paraso& Proprietary and Confiden1al 28 Why Assert? § Excuses: § My code already has console output § I can check it visually/manually § Binary output (pass/fail) § Part of overall test suite § No human interven1on
- 29. Paraso& Proprietary and Confiden1al 29 Paraso& Proprietary and Confiden1al 29 Asser1on failures Introduced coding error • Fix the new code Func1onality changed • Update the asser1on Improper asser1on • Comment it out
- 30. Paraso& Proprietary and Confiden1al 30 Paraso& Proprietary and Confiden1al 30 Asser1on Policy § Assign failures as tasks § Author of the test or of the changed code § Try to reach developer that has the highest chance to understand the change § Developer assesses failure: § Bug (fix) § Changed func1onality (update asser1on) § Bad asser1on (improve or remove) -‐ don’t delete; instead, comment out
- 31. Paraso& Proprietary and Confiden1al 31 Paraso& Proprietary and Confiden1al 31 What to assert § Ease of crea1ng asser1ons leads to poor choices § Anything CAN be checked – but should it be? § Asser1ons should be connected to what you’re trying to test § Asser1ons should be logically invariant § Check values by range, not equivalence § “What do I expect here logically?”
- 32. Paraso& Proprietary and Confiden1al 32 Paraso& Proprietary and Confiden1al 32 Asser1on Example: Java public Integer getNumber() { Scanner input = new Scanner( System.in ); System.out.print( "Enter a number between 0 and 10: " ); int number = input.nextInt(); // assert that the absolute value is >= 0 assert ( number >= 0 && number <= 10 ) : "bad number: " + number; return number }
- 33. Paraso& Proprietary and Confiden1al 33 Paraso& Proprietary and Confiden1al 33 How to recognize bad asser1ons § How o&en do you WANT to manually verify an asser1on? § Daily or near-‐daily checking indicates poor asser1on. § Troublesome possibili1es: § Date § User or machine name § OS § Filesystem names and paths
- 34. Paraso& Proprietary and Confiden1al 34 Paraso& Proprietary and Confiden1al 34 Missing Asser1ons § Error was found a&er code changed § Test suite never caught the error § Update unit-‐test with new asser1on § Write new unit tests as necessary
- 35. Paraso& Proprietary and Confiden1al 35 Paraso& Proprietary and Confiden1al 35 Cleaning It Up
- 36. Paraso& Proprietary and Confiden1al 36 Paraso& Proprietary and Confiden1al 36 The Challenge: Management’s Perspec1ve § Need tests to reduce risks § Tes1ng takes 1me from features/fixes § Not enough 1me § Not enough people § Tests and reports lack context
- 37. Paraso& Proprietary and Confiden1al 37 Paraso& Proprietary and Confiden1al 37 The Challenge: Development’s Perspec1ve § Hundreds of failing or broken tests § Test requirements unclear § Exis1ng tests are dubious § Deadline pressure
- 38. Paraso& Proprietary and Confiden1al 38 Paraso& Proprietary and Confiden1al 38 Spring Cleaning Solu1on § Throw out the useless, old junk § Iden1fy and eliminate “noisy” tests § Organize the remaining chores § Priori1ze unit test tasks § Keep it clean § Establish and follow a rou1ne test maintenance plan
- 39. Paraso& Proprietary and Confiden1al 39 Paraso& Proprietary and Confiden1al 39 Throw out the Junk: Iden1fying Noisy Tests § Tests that are fragile § Pass/Fail/Incomplete results frequently change § Frequent changes are required to keep up-‐to-‐date § Tests that are outdated § More than “n” releases have occurred despite this test failure § The test applies to legacy code that is frozen § The test has not been executed for “x” period of 1me § Tests that are unclear § It takes more than 5 minutes to understand why the test exists § The failure message is incorrect/unhelpful/irrelevant § There is no associated requirement or defect § Be merciless!
- 40. Paraso& Proprietary and Confiden1al 40 Paraso& Proprietary and Confiden1al 40 Organize Chores: Priori1ze Tasks with Policy § All tasks are created equal § Some tasks are more equal than others § Policies establish priori1za1on and assignment
- 41. Paraso& Proprietary and Confiden1al 41 Paraso& Proprietary and Confiden1al 41 Keep it Clean: Establish a Rou1ne Process § Management and Development need to co-‐define a Policy § Clear tes1ng goals that 1e tests to business needs § Clear delinea1on of responsibility § Test crea1on and dele1on § Test failure resolu1on § Naming conven1ons § Peer review § Clear and frequent communica1on § Understand the technical and business impact § Enhance traceability to project requirements § Ask ques1ons § Focus on safety/quality, not coverage § Keep the number of tests minimal § Run all tests on a regularly scheduled, frequent basis § Clean as you go
- 42. Paraso& Proprietary and Confiden1al 42 Paraso& Proprietary and Confiden1al 42 Spring Cleaning Recap § Find and delete “noisy” tests § Use policies to priori1ze, assign, and plan § Co-‐design and follow a process to keep it clean
- 43. Paraso& Proprietary and Confiden1al 43 Paraso& Proprietary and Confiden1al 43 § Web § hjp://www.paraso&.com/jsp/resources § Blog § hjp://alm.paraso&.com § Social § Facebook: hjps://www.facebook.com/paraso&corpora1on § Twijer: @Paraso& @MustRead4Dev @CodeCurmudgeon § LinkedIn: hjp://www.linkedin.com/company/paraso& § Google+: +Paraso& +ArthurHickenCodeCurmudgeon § Google+ Community: Development Tes1ng Because its Bejer