Software Testing, Everyone's responsibility
- 1. SOFTWARE TESTING Everyone’s Responsibility Kurt Bliefernich
- 2. How can you know your code works? That’s easy. Test it. Test it again. Test it up. Test it down. Test it seven ways to Sunday. ~ Clean Coder Robert C. Martin (Uncle Bob)
- 3. Isn’t Software Testing Expensive? • Long Regression Cycles • Time to write tests • Lack of feature development
- 4. Cost Savings 1,000 hours of up-front implementation without TDD. TDD saved 623 man hours, or, for a team of 4, about a month of development. Here are the assumptions made: • Production bugs without TDD: 100 • Production bugs with TDD: 40 (60% fewer, middle ground between 40%— 80%) • Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost) • Average time to fix a production bug: ~15 hours
- 5. Testing Saves Money • Automate Regression • Write lots of smaller tests • Retire redundant tests • Less code rewrite because of bugs • Easier to refactor, grow code
- 6. The Testing Ice Cream Cone • This is an Anti-pattern • Sometimes called Testing Cupcake.
- 7. The Testing Pyramid • Write lots of fast tests, near 100% coverage • Push exception testing down the pyramid • Automate your happy path, full stack tests • Let QA specialists do what they do best (AdHoc, Exploratory, etc.)
- 8. The Testing Pyramid • Technology Facing Tests == Developer Responsibility • Business Facing Tests == QA && Product Responsibility • Best, QA and Developers cross pair and write tests
- 9. Unit tests are low-level, focusing on a small part of the software system. Unit tests are written by the programmers themselves using their regular tools - the only difference being the use of some sort of unit testing framework. Unit tests are expected to be significantly faster than other kinds of tests. https://martinfowler.com/bliki/UnitTest.html
- 10. A component test is a test that limits the scope of the exercised software to a portion of the system under test. https://martinfowler.com/bliki/ComponentTest.html
- 11. An integration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that they coherently work together. https://testing.googleblog.com/2015/ 04/just-say-no-to-more-end-to-end- tests.html
- 13. Contract/Consumer Tests Contracts should be about catching: • Bugs in the consumer • Misunderstanding from the consumer about end-points or payload • Breaking changes by the provider on end-points or payload https://docs.pact.io/best_practices/contract_tests_not_functional_tests.html
- 15. Automated UI Testing • Should cover sunny day scenarios • Testing through the UI is slow, increasing build times. • Most importantly such tests are very brittle.
- 16. Test DrivenDevelopmentDesign 1. You are not allowed to write any production code until you first written a failing unit test. 2. You are not allowed to write more of a unit test than is sufficient to fail – and not compiling is failing. 3. You are not allowed to write more production code that is sufficient to pass the currently failing unit test.
- 18. JUnit & AssertJ
- 19. Testing Terminology - Doubles Test Double is the generic term for any kind of pretend object used in place of a real object for testing purposes.
- 20. Testing Terminology - Doubles Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
- 21. Testing Terminology - Doubles Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production.
- 22. Testing Terminology - Doubles Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
- 23. Testing Terminology - Doubles Mocks are objects pre- programmed with expectations which form a specification of the calls they are expected to receive.
- 24. Testing Terminology - Doubles Spies are mock objects, as they record what is called. They allow for real methods to be called when they are not mocked.
- 26. References and Links • https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks • https://martinfowler.com/articles/mocksArentStubs.html • https://www.amazon.com/Clean-Coder-Conduct-Professional-Programmers/dp/0137081073 • https://docs.pact.io/best_practices/contract_tests_not_functional_tests.html • https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html • https://watirmelon.blog/ • https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software _Development_Life_Cycle_SDLC • https://medium.com/javascript-scene/the-outrageous-cost-of-skipping-tdd-code-reviews-57887064c412 • https://www.thoughtworks.com/insights/blog/introducing-software-testing-cupcake-anti-pattern
- 27. Questions ??
Editor's Notes
- #2: I'm going to talk about why developers should write tests. I want to cover some common myths about cost. I want to talk about the different kinds of developer written tests. I will cover some testing terminology AND. I want to talk about some of the mechanics of writing these tests.
- #3: Writing tests is part of being a professional developer. This may sound controversial. http://www.ambysoft.com/surveys/tdd2008.html 2008 – 71% of Agile developers claim to practice TDD 2014 – 76% https://www.infoq.com/news/2016/07/developers-tdd-bdd-atdd 2016 – A little over 50% developers have adopted TDD/BDD practices You are not agile if you are not doing TDD, don’t fool yourself. Test first / test last
- #4: https://medium.com/javascript-scene/the-outrageous-cost-of-skipping-tdd-code-reviews-57887064c412 initial project build-outs to take up to 30% longer with TDD
- #5: TDD reduces production bug density 40% — 80%, (computer.org 03/2007) According to the IBM System Sciences Institute, fixing a production bug costs 100x more than fixing a bug at design time, and over 15x more than fixing a bug at implementation time. (https://www.researchgate.net/publication/255965523_Integrating_Software_Assurance_into_the_Software_Development_Life_Cycle_SDLC)
- #6: Wouldn’t it be nice to have a near full regression on each code commit? Code as documentation. Developers document the code with tests. (Spock example) Writing tests while coding reduces bugs. This shifts re-writing to lack of or misunderstanding of requirements. Code refactoring is a huge cost. Having robust tests in place help ensure that refactors don’t break things. Helps define bounded context. Refactor to microservices.
- #7: Writing tests usually spans 2-3 teams: Developers write unit, integration and component tests. Another team writes black box tests through the GUI. Manual testers have a set of scenarios that they execute manually. Generally, these teams work in isolation and do not collaborate much. Work happens sequentially, as opposed to in-sync. First developers write their code and corresponding tests, then manual testers run their tests, then GUI testers will write their tests. What does this scenario look like? A mini-waterfall.
- #9: If QA knows and Trusts the tests that developers are writing, then why waste time re-testing This requires test reporting and metrics
- #10: Unit tests are small Unit tests are FAST There should be a unit test for each conditional logic point You should also look to remove redundant unit tests
- #11: Component tests exercise more than one thing. Can be used for testing how multiple methods interact Can be used to to test how different classes interact. Should NOT be used to test how separate systems interact
- #12: Sounds similar to component tests My definition: testing separate parts of the system in isolation. API to Service to Database, and back Should not use common things like a shared database. Mocking or Stubbing still exists for external systems.
- #13: Consumer Contact tests Becoming more important in Microservice environments
- #14: Contract tests should not contain exhaustive exception testing. Push those down to the lower level tests It is the CONSUMER of the API that is responsible for doing contract tests
- #15: Write tests to validate the UI Avoid record and run when ever possible
- #16: Test recording software can be dangerous. Little changes require entire new tests to be made. Tools like cucumber and gherkin help UI code should be tested by the UI unit/component/integration test suite. Best, Devs and QA work to write these tests together so we can trust they are right and we don’t need to re-test
- #17: This will drive you to write code that can easily be tested, hence the tests are driving your design.
- #18: Spock example. This is a form of documentation
- #19: Notice Assign, Act, Assert == Given, When, Then Junit & AssertJ
- #20: https://martinfowler.com/articles/mocksArentStubs.html
- #21: https://martinfowler.com/articles/mocksArentStubs.html A dummy may be a null, empty list, empty object.
- #22: A good example of a fake is an in-memory database. Not suitable for production, but good for verification
- #23: Stubs are more suitable when we care more about state verification than or outcome that code behavior, or whether something got called
- #24: Mocks are said to either be restrictive or permissive restrictive FAIL when a non-programmed interaction takes place permissive will return a sane default if a non-programmed interaction takes place, often null
- #25: Spies are useful when you want to stub or mock a method in the class that you are testing a different method in. Tax calculation, method to get tax rate by state
- #27: There is a testing library or framework for almost any language. There is no excuse to not write tests.