Functional testing patterns
2 likes4,927 views
The document provides an overview of functional testing, emphasizing its importance in verifying business requirements through automated tests, while also highlighting challenges like flakiness and maintenance issues. It suggests best practices for improvement, including better collaboration among technology, process, and people, and recommends specific strategies for developing effective test cases and managing team dynamics. Ultimately, it aims to refine the testing process to enhance quality and speed in software development.
![Workflow
• Sprint [N -1]
o Feature elaboration with entry level acceptance criteria
• Sprint [N]
o Feature kick-off meeting
o Developers implement feature
o Testers (optionally) add additional acceptance criteria
• Sprint[N, N + 1]
o Testers verify all acceptance criteria to be satisfied
o Testers refactor functional test suite
o Feature marked complete
© 2012 ThoughtWorks, Inc. 22](https://image.slidesharecdn.com/functionaltestingpatterns-120112062420-phpapp02/85/Functional-testing-patterns-22-320.jpg)
Functional testing patterns
- 1. Functional Testing Pa0erns Premanand Chandrasekaran ThoughtWorks, Inc. [email protected]
- 2. Types of Tests Business Facing Support Programming Functional Tests Showcases Acceptance Tests Exploratory Tests Critique Product Usability Tests Unit Tests Performance Tests Component Tests Stress Tests System Tests Security Tests Technology Facing h0p://www.exampler.com/old-‐‑blog/2003/08/21/#agile-‐‑testing-‐‑project-‐‑1 © 2012 ThoughtWorks, Inc. 2
- 3. Functional Tests -‐‑ The Pitch • What Are Functional Tests? o Verification that business requirements are met o Black Box o Automated • Why Functional Tests? o Maintain (high) external quality o Allow team to be bolder o Allow team to go faster © 2012 ThoughtWorks, Inc. 3
- 4. Functional Tests – The Reality • Flaky • Take long time to run • Cumbersome to maintain • Don’t catch many bugs • Some things are hard to automate • Don’t quite give you the feedback you were hoping for © 2012 ThoughtWorks, Inc. 4
- 5. Consequences • Low confidence in automation • Over-reliance on manual testing • Slipped Deadlines AND/OR • Lesser Testing AND/OR • Testing in Production :-) © 2012 ThoughtWorks, Inc. 5
- 6. How To Get Be0er? Technology Process People © 2012 ThoughtWorks, Inc. 6
- 7. Process © 2012 ThoughtWorks, Inc. 7
- 8. Functional Test – Example © 2012 ThoughtWorks, Inc. 8
- 9. Example -‐‑ Specification © 2012 ThoughtWorks, Inc. 9
- 10. Example – Implementation © 2012 ThoughtWorks, Inc. 10
- 11. Implementation Problems? • No correlation between specification and implementation • Intent lost in translation • Easy to write, hard to read, maintain • Too monolithic, lacks abstraction • Causes non-technical domain experts to tune out of the process © 2012 ThoughtWorks, Inc. 11
- 12. Executable Specifications Specification Step Tight Correlation Implementation Step © 2012 ThoughtWorks, Inc. 12
- 13. Functional Test -‐‑ Specification © 2012 ThoughtWorks, Inc. 13
- 14. More useful information in failure reports © 2012 ThoughtWorks, Inc. 14
- 15. Acceptance Criteria DSL • Given some initial context • When an event occurs • Then ensure outcome • Example – © 2012 ThoughtWorks, Inc. 15
- 16. Functional Test – New Implementation © 2012 ThoughtWorks, Inc. 16
- 17. Good Acceptance Criteria Scenario: Advanced Search for a hat! ! Given I am searching on "http://www.etsy.com"! When I click on the select box with css class "select.handmade"! And I select "Knitting"! And I click on the text box with id "#search_query"! And type in "hat"! Then there are search results! Focus on WHAT, not HOW Scenario: Advanced Search for a hat! ! Given I am searching on Etsy.com! And I am not logged in! And I specify the Knitting sub category! And I search for hat! Be@er Then there are search results! ! © 2012 ThoughtWorks, Inc. 17
- 18. How Many Tests? Manual Testing 10% Functional Tests Integration Tests 20% Unit Tests 70% h0p://blog.mountaingoatsoftware.com/tag/agile-‐‑testing © 2012 ThoughtWorks, Inc. 18
- 19. What About Legacy Systems? • Start with functional tests • Cover with unit tests for new functionality • Be on the lookout to replace functional tests more fine-grained tests at every given opportunity © 2012 ThoughtWorks, Inc. 19
- 20. What To Automate? • The MOST important scenarios o End-To-End system level scenarios o Happy paths • Avoid automating individual feature- level acceptance criteria • Make a distinction between “ephemeral” and “long-standing” tests © 2012 ThoughtWorks, Inc. 20
- 21. When do these tests run? • After every check-in • As part of CI build pipeline o Unit à Smoke à Regression à Deploy • Incubator Test Suites © 2012 ThoughtWorks, Inc. 21
- 22. Workflow • Sprint [N -1] o Feature elaboration with entry level acceptance criteria • Sprint [N] o Feature kick-off meeting o Developers implement feature o Testers (optionally) add additional acceptance criteria • Sprint[N, N + 1] o Testers verify all acceptance criteria to be satisfied o Testers refactor functional test suite o Feature marked complete © 2012 ThoughtWorks, Inc. 22
- 23. Technology © 2012 ThoughtWorks, Inc. 23
- 24. Test Characteristics • Be idempotent o At least re-runnable o Keep tests isolated from one another • Be runnable in parallel o No inter-dependencies between tests • Be deterministic o Quarantine/eradicate non-deterministic tests o Don’t “sleep” • Prefer callbacks or at least poll for asynchronous responses o Consider mocking remote third-party services • Have separate tests to verify interaction with remote services o Consider using relative dates/times or mock the system clock o http://martinfowler.com/articles/nonDeterminism.html © 2012 ThoughtWorks, Inc. 24
- 25. Test Sources • Don’t have any dependencies on production sources • Treat as first-class citizens o Keep DRY o Run static analysis metrics • Leverage design patterns like o Page Objects o Compound Steps • Consider the use of terser languages o Groovy, Ruby etc. © 2012 ThoughtWorks, Inc. 25
- 26. Test Data • Use application to create test data • Use test data creation steps o Frameworks like DBUnit • Use anonymized production data subsets o Requires lot of time and effort • Use anonymized production data copies o Huge data volumes may make it prohibitive • Avoid the shared database anti-pattern • Consider mocking remote third-party services © 2012 ThoughtWorks, Inc. 26
- 27. Test Infrastructure • Invest in fast build hardware o CI servers support multiple remote agents • Use same software configuration as production in all environments • Consider running tests on the cloud o http://www.saucelabs.com © 2012 ThoughtWorks, Inc. 27
- 28. People © 2012 ThoughtWorks, Inc. 28
- 29. Who Writes/Owns The Tests? • The specifications – acceptance criteria? o Domain Experts • Step implementations? o Developers & Testers • The functional suite? o The Team o Day-to-Day – Testers © 2012 ThoughtWorks, Inc. 29
- 30. Team Dynamics • Fix broken builds and tests immediately o OK to occasionally break the build o Not OK to leave the build broken for long • Do not allow check-ins over a broken build o Unless check-in is being made to fix the build • Co-locate teams as much as possible o At least create fully formed remote teams © 2012 ThoughtWorks, Inc. 30
- 31. Questions? [email protected] Thank You! © 2012 ThoughtWorks, Inc. 31