Use Automation to Assist—Not Replace—Manual Testing
- 1. T3 Test Automation 10/5/17 9:45 Use Automation to Assist "Not Replace"Manual Testing Presented by: Jeffrey Martin SmartBear Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
- 2. Jeffrey Martin SmartBear Working with companies and testing teams of all sizes and skills, Jeffrey Martin is product owner of SmartBear Software’s test management solution QAComplete. With more than fifteen years of experience in product management, quality, regulatory, sales, engineering, and marketing, Jeffrey brings his broad, cross-discipline skill set to uncover unique insights into obstacles facing his own teams and the QA teams of his customers. With a practical, results-first, focus on the problems facing the modern software development team, Jeffrey has helped organizations deliver better software more quickly, while making it easier to consume by the end-user.
- 3. 1 Use Automation to Assist —Not Replace— Manual Testing 22 Ideal Automation Return E f f o r t Return The Shark Fin Manual Automated Time
- 4. 2 33 Reality of Most Automation E f f o r t Return? Time Mountains of No Return Series1 Series2 44 Why the Gap? 1. Failure to align different goals across the team 2. Automated testing viewed as a replacement for existing processes 3. Automated testing approached like development 4. Not enough focus on high-return automation
- 5. 3 55 Goals - The Automation Dream • Less Time • No faulty tests / false defects • Less people • More like manufacturing The “Tester”The “Boss” • Less repetition • No faulty tests / false defects • More time to Test • More like craftsmanship 66 Goals - The Agreement 1. Less Time & Repetition 2. No faulty tests / false defects Focus on these two goals
- 6. 4 77 Replacing Existing Processes What Testers and the Business roles are familiar with: Workflows Business Processes Test Cases User Stories So this is what they automate 88 Misleading Early Results The obvious workflows get automated first Less Time & Repetition No (ok, less) faulty tests / false defects Success breeds repetition What gets overlooked is efficiency Break down workflows into component tasks
- 7. 5 99 Task Efficiency over 5 years Credit: XKCD hDps://xkcd.com/1205/ 1010 Automate like a Dev? Ok, so all Tasks in a process are not created equal One popular alternative: Think like a developer and break processes down 2 Developer approaches with the same problem when applied to testingObject Oriented Func.onal Stable operaLons Stable “things” New “things” New operaLons
- 8. 6 1111 Why Dev-focused approaches fail Developers are used to building “the whole thing” They can’t choose not to code something! Development and Tests are based on the same Requirements and therefore do not capture errors in design Additionally, Developers can only interact and check for expected results Interact Check Click Values Enter text Presence Drag Layouts / Coordinates 1212 An aside: Checking vs. Testing Automation when done right is more valuable than “Checking” Many non-checking functions are better done by computers Checks are based on design, Tests are based on the much messier Usage. Fully Automated Regression “Testing” is a screening tool that only Checks
- 9. 7 1313 Where are we with the Gap? 1. Failure to align different goals across the team Find the areas of agreement, focus on those 2. Automated testing viewed as a replacement for existing processes Break workflows into Tasks 3. Automated testing approached like development Recognize that development and testing, even automated testing, are different activities 4. Not enough focus on high-return automation 1414 Task Automation 1. Testing is hard work, automation should free up time for that hard work 2. Processes are made of tasks 3. Tasks do not always equal tests 4. Tasks have different returns on automation investment
- 10. 8 1515 Good Task Automation Candidates 1. Environment Setup / Teardown 2. Data Entry a) Form Filling b) Varying data inputs in a repetitive process 3. Exposing backend data (APIs, DB tables, etc….) 4. Repetitive / boring tasks that are prone to inattention errors 5. Tasks with high reuse value across many workflows 6. Tests with timing or screen responsiveness as a criteria for success 7. Many non-functional test types, such as Performance testing 8. Smoke Test Suites – key user workflows as a screening tool 9. Result Capture 1616 Where are we with the Gap? 1. Failure to align different goals across the team Find the areas of agreement, focus on those 2. Automated testing viewed as a replacement for existing processes Break workflows into Tasks 3. Automated testing approached like development Recognize that development and testing, even automated testing, are different activities 4. Not enough focus on high-return automation Deconstruct Tests in to Tasks Rate your tasks on Return
- 11. 9 1717 The Hard Part – Combining ExecutionCombining Automation and Manual execution of Tasks is difficult Many tools do exist to help combine execution and results: 1. Automation tools that pause and wait for manual testing either in a form or via a button 2. Test Management or Manual Testing tools that run scripts on demand Caution: Switching between execution methods has a cost that should be factored into Task selection. Questions?