Anti-Patterns for Automated Testing
- 1. 5/13/16 1 © 2016 LogiGear Corporation. All Rights Reserved STAREAST 2016, Track Session W3 Orlando, Florida Wednesday, May 4th, 2016 11.30 AM – 12.30 PM "Anti-Patterns" for Automated Testing Mr. Playback Hans Buwalda LogiGear [email protected] @hansbuwalda www.happytester.com © 2016 LogiGear Domain Language Approaches: Keywords 4 actions, each with an action keyword and arguments read from top to bottom fragment from a test with actions acc nr first last open account 123123 John Doe acc nr amount deposit 123123 10.11 deposit 123123 20.22 acc nr expected check balance 123123 30.33 • The test developer creates tests using "actions" with keywords and arguments • Checks are, as much as possible, explicit (specified expected values) • The automation task focuses on automating the keywords, each keyword is automated only once • This technique can be very scalable. A similar approach is behavior based testing, which also works with human readable tests, but is more verbose
- 2. 5/13/16 2 © 2016 LogiGear High Level Test Design - Test Development Plan Objectives Test Module 1 Test Cases Test Module 2 Test Module N Actions . . . AUTOMATION Objectives Objectives interaction test business test Overview Action Based Testing define the "chapters" create the "chapters" create the "words" make the words work Test Cases Test Cases window control value enter log in user name jdoe enter log in password car guy window control property expected check property log in ok button enabled true user password log in jdoe car guy first last brand model enter rental Mary Renter Ford Escape last total check bill Renter 140.42 © 2016 LogiGear § Business objects and business flows - objects are like cars, invoices, locations, etc - flows are like create, fulfill, pay and close an order § Other tests - functions and features, like premium calculation or PDF output - administration, users, security, authorizations - graphics - technologies, protocols, ... - customization, extensibility - interoperability - . . . § Business versus interaction - differentiate within business objects and all other categories - interaction can be further differentiated into: values, UI, keyboard, etc - also, for every category, look at negative tests, flows, aggressive tests, etc Examples of considerations
- 3. 5/13/16 3 © 2016 LogiGear Example Top Level Structure Project create, retrieve, update, delete / deactivate ("CRUD") copy, move categorize, enumerate, identify convert, serialize, export/import, ... UI, dialogs, forms, pages input (validation, defaulting, dependencies) flows (primary paths, alternate paths) keyboard shortcuts, keyboard controls, ... . . . <business object 1> Lifecycles, data operations Interaction Functions and Features Technologies, protocols, controls Data (handling, quality, ETL, ...) Security, authorization, admin Graphics, multi-media, charts, ... Load, performance Interoperability Customizing, extensibility Business Flows Concurrency, race conditions, ... Business Objects processes, transactions, end-to-end, day in the life, combinations of flows, ... calculations, analyses, PDF output, ... © 2016 LogiGear Eye on the ball, Scope § Always know the scope of a test module § The scope should be unambiguous § The scope determines many things: - what the test objectives are - which test cases to expect - what level of actions to use - what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)
- 4. 5/13/16 4 © 2016 LogiGear Example of a Test Step Description Expected step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner step 21 Enter "acme watch" in the search field The "Search" button is enabled step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2 step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2 step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping" step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches © 2016 LogiGear GWT scenario Scenario: Applicant is at Stop Test alert and chooses to Stop Test Given User turns off Password required option for Drive Test And User has logged in by Traffic Applicant account And User is at the Assessments Take a Test page And User clicks the Traffic Test link And User clicks the Next button And User clicks the Sheet radio button in Mode page if displayed And User clicks the Start button And User waits for test start And User clicks the Stop Test button When User clicks the Confirm Stop Test button And User enters the correct applicant password And User clicks the Confirm Stop Test button Then The Test is Over should be displayed in the Message label Then Value of the Message label should be The test is over. And The Welcome to Traffic Testing page should be displayed
- 5. 5/13/16 5 © 2016 LogiGear What is an "anti-pattern" § A practice considered harmful - opposite of a "pattern" § Has an easy to remember name § Can be helpful to spot or describe risks § In the case of automated tests, the risks you usually try to avoid are in areas like: - effectiveness - efficiency - readability - manageability - maintainability - etc © 2016 LogiGear Using the anti-patterns in this presentation § Experimental, input welcome § There are overlaps - in not-so-great test designs multiple patterns tend to apply simultaneously § Input for design decisions, not absolute truths - "decisions have consequences", but those can be acceptable in sometimes - not everything applies to all situations, not every anti-pattern occurrence is a bad thing - see them more as a starting point of a discussion § See anti-patterns as symptoms - lack of an overall high level test design - lack of experience of the test developer
- 6. 5/13/16 6 © 2016 LogiGear "Anti-patterns" (informal) § Interaction Heavy – Not having many business tests § Lame – No depth or variety, no testing techniques used § Enter Enter Click Click – Test steps are too detailed § No Life – Missing life cycle steps of business objects § Clueless – No clear scope for the tests § Cocktail – Interaction tests mixed with business tests § Over-Checking – Checks not relevant for the scope § Sneaky Checking – Checks hidden in actions § Action Explosion – Many actions, hardly different § Mystery Actions – Cryptic actions, unclear what they do § Techno – Actions and tests that look like code § Passive Timing – Hard-coded "sleeps" for timing issues © 2016 LogiGear Interaction Heavy § Not having many business tests § What to look for: - seeing only test cases like these: • can I successfully create a new customer • can I successfully update an order • ... - seeing the test tree follow the UI organization § The UI is the most visible part of an application. It is not uncommon to have a "what you see is what you test" tendency § What is missing is a clear understanding and verification if the application is fit for business
- 7. 5/13/16 7 © 2016 LogiGear Lame § No depth or variety, no testing techniques used § What to look for: - boredom - no suprises - no techniques used - not many, and trivial, bugs § Consider exploratory approaches, not just for manual testing via the UI, but also for test case design - in other words: make it more fun - involve domain experts © 2016 LogiGear Enter Enter Click Click § Test steps are too detailed § What to look for: - long sequences with only built-in low level actions like "enter", "click", "select menu", etc - high sensitivity to UI changes - not able to understand the goal of the tests easily § Example: - an application in the oil industry - over 6000 test cases with a length of 2000 test lines or more § Rather than trying to optimize, create a high-level test design first ("from scratch")
- 8. 5/13/16 8 © 2016 LogiGear No Life § Missing life cycle steps of business objects - like CRUD: Create-Read-Update-Delete § What to look for: - is it clear what the business objects are? - do they show in the hierarchy - in particular are there modifications ("update") and removal/closure ("delete") tests? - is there variation in such tests § Life-cycle tests are usually not difficult for define § I usually prefer to design life-cycle tests as business tests, not as interaction tests © 2016 LogiGear Clueless § No clear scope for the tests § What to look for: - look through the test lines. Assuming you know the domain, is it clear to you what the purpose of the test is? - is the scope mixed. For example a customer is created, and tested, and a order is placed by that customer in the same test § Having clear scopes for tests (in particular test modules) helps organize and maintain them § It also helps follow up - for example: "Credit Card Payments" has fails
- 9. 5/13/16 9 © 2016 LogiGear Cocktail § Interaction tests mixed with business tests § This is quite common § What to look for - a test is testing business rules and processes, like calculation of a home loan premiums in various situations - and at the same time is doing UI checks § This anti-pattern is usually a symptom of other anti-patterns (like "clueless") © 2016 LogiGear Over-Checking § Checks not relevant for the scope § Testers, and their management, often believe everything a test script goes through needs to be tested § Automation engineers want to make sure the navigation is still ok (consider using an "assert" action instead) § The result can be less clarity and harder maintenance § In a clear test design over-checking can quite easily be avoided
- 10. 5/13/16 10 © 2016 LogiGear Example of a Test Module TEST MODULE Order processing start system TEST CASE TC 01 Order for tablets user password login jdoe doedoe window check window exists welcome order id cust id article price quantity create order AB123 W3454X tablet 198.95 5 order id total check order total AB123 994.75 . . . © 2016 LogiGear Sneaky Checking § Checks hidden in actions § What to look for: - in action definitions or scripts there are checks that are not visible in the test module - it are often interaction checks, like "does this window exist", after a click § Such checks may influence your statistics, and increase maintenance sensitivity § If it is "just to make sure", use an "assert" action: - producing a warning or an error if the condition isn't met - in most cases this is not needed, the tool will complain
- 11. 5/13/16 11 © 2016 LogiGear Action Explosion § Many actions, hardly different § What to look for: - every step and variation of a step has its own action - many actions are used only once or twice - there are about as many actions as tests § It is the opposite of the "enter enter click click" - the challenge is to find a balance between not-enough- actions and too-many-actions § Example: open financials open financials - payments tab open financials - payments tab - approval sub tab open financials - payments tab - approval sub tab - approved changes panel © 2016 LogiGear Consider using "mid-level" actions § "Low level": detailed interaction with the UI (or API) - generic, do not show any functional or business logic - examples: "click", "expand tree node", "select menu" § "High level": a business domain operation or check on the application under test - hide the interaction - examples: "enter customer", "rent car", "check balance" § "Mid level": common sequences at a more detailed application level - usually to wrap a form or dialog - for use in high level actions - greatly enhance maintainability - example: "enter address fields" enter customer enter address fields enter select set . . .. . .
- 12. 5/13/16 12 © 2016 LogiGear Mystery Actions § Cryptic actions, unclear what they do § What to look for: - action or arguments that are not clear to those who know the domain - generic actions like "check invoice" - are not business related or UI related § Unclear actions: - might be used a wrongly - might not be found, delimiting their re-use - hinder impact analysis and follow up © 2016 LogiGear Techno § Actions and tests that look like code - "preferably" C++ code - work well to impress people, but not for much else § What to look for: - not English or another natural language - all kinds of special characters - lack of spacing - in general names that are _NOts0EasY_2REad § Often a result of developers at the helm of the test design
- 13. 5/13/16 13 © 2016 LogiGear § Hard-coded "sleeps" for timing issues § What to look for - "sleep" or "wait" actions - sometimes hidden in other actions - arguments are sometimes variables, this is not much of an improvement § Passive timing - big nono, the number one archenemy of automation J - at best slows down your tests - but if not long enough make the test unstable - in VM's the problem is even worse than in physical machines - if used, the team should clearly escalate it, define it as a risk § Better approach: Active timing - wait for a measurable event - usually the wait is up to a, generous, maximum time - for UI many such waits are built into the automation tool - consider involving developers to help you (see my slides of my "BIG Testing" tutorial for more suggestions) Passive Timing © 2016 LogiGear Be careful in communication! § Avoid even the hint of attack § Make sure all people involved understand, and appreciate, that anti-patterns are applied § In particular watch your writing, like emails and reports - good fuel for "flaming" - avoid writing something like: these tests are "lame" - talk about test design - teamwork rules suppreme
- 14. 5/13/16 14 © 2016 LogiGear Summary § Automation success depends on design decisions, most of which are test design decisions § Anti-patterns can help identify risks, and start discussions § Anti-patterns are not (meant to be) absolute rules § Test design and automation are team work, part of a larger route to success