Your Automated Execution Does Not Have to be Flaky
1 like1,705 views
Alan Richardson will present on how to remove intermittent failures from automated test execution. Common causes of intermittency include lack of synchronization, parallel test interference, long running tests, difficulty automating the system under test, outdated tools, uncontrolled test preconditions, incorrect assumptions in assertions, and uncontrolled data. Richardson will provide examples of each cause and recommend solutions such as adding synchronization, isolating test environments, controlling data, and investigating failures rather than assuming tests are "flaky". The goal is to convince attendees that intermittent failures are a result of unresolved issues, not flaws in tests, and can be addressed through proper testing practices.
Your Automated Execution Does Not Have to be Flaky
- 1. Your Automated Execution Does Not Have to be Flaky Eurostar Webinars Feb 2018 Alan Richardson www.eviltester.com/flaky www.compendiumdev.co.uk @eviltester @EvilTester 1
- 2. Have you experienced flaky test automation? In this webinar, Alan Richardson plans to convince you that you haven't. Instead you have experienced the result of not resolving the causes of Intermittent execution. Alan will explore some common causes and solutions of intermittent behaviour. Why? So you never say the phrase, "flaky tests", ever again. @EvilTester 2
- 3. Flaky Test Automation is Normal @EvilTester 3
- 4. Flaky Test Automation is Normal Because we have normalized Flaky Test Automation "Our Tests are Flaky" "Some tests fail Randomly" @EvilTester 4
- 5. "There is nothing so absurd that it has not been said by some philosopher." Cicero, On Divination, Book II chapter LVIII, section 119 ﴾44 BC﴿ "Truth happens to an idea. It becomes true, is made true by events." William James, Lecture VI, Pragmatism's Conception of Truth, Pragmatism: A New Name for Some Old Ways of Thinking ﴾1907﴿ @EvilTester 5
- 6. How to Normalize Flaky Test Automation We all know these Test Automation Truths "GUI Automation is Flaky" "We have to live with 'flakiness'" "We shouldn't automate at the GUI" "We can only remove flakiness under the GUI" "Flaky Tests" blames the tests. Not good enough. @EvilTester 6
- 7. It isn't even the "Tests" We don't automate Tests We automate the execution of steps in a workflow or process We automate the execution of a System We add condition assertions during that execution We don't have flaky Tests ‐ we have automated execution that fails. Sometimes on the steps, sometimes on the assertions. @EvilTester 7
- 8. 'Flakiness' does not reside at a 'level' I have seen 'flakiness' in Unit Tests in API Tests in Integration Tests in GUI Tests @EvilTester 8
- 9. It is too easy to say 'flaky' and then blame 'GUI execution' and then blame 'the tool' Living with flakiness is a choice. Choose a different approach. @EvilTester 9
- 10. I am not the only person saying this. see references at the end and name drops throughout try to cover something different in this talk @EvilTester 10
- 11. "We designed that flakiness. We are allowing that to happen. We engineered it to be that way. And its our fault that that exists." Richard Bradshaw, "Your Tests aren't Flaky, You Are!" Selenium Conference 2017 https://www.youtube.com/watch?v=XnkWkrbzMh0 @EvilTester 11
- 12. Take it more seriously. Describe it differently. Intermittent Occurring at irregular intervals; not continuous or steady. https://en.oxforddictionaries.com/definition/intermittent @EvilTester 12
- 13. Take it more seriously. Describe it differently. Nondeterministic Algorithm "a nondeterministic algorithm is an algorithm that, even for the same input, can exhibit different behaviors on different runs" https://en.wikipedia.org/wiki/Nondeterministic_algorithm @EvilTester 13
- 14. Flaky is not serious enough. We do not want to use nondeterministic algorithms for continuous assertions that we are relying on @EvilTester 14
- 15. Your Test Automation is not Flaky Your automated execution fails intermittently @EvilTester 15
- 16. Don't Blame Tests. Look For Root Causes. watch Alister Scott's GTAC 2015 talk @EvilTester 16
- 17. I have removed 'flakiness' from Unit Tests from API Tests from Integration Tests from GUI Tests Automated execution does not have to fail intermittently. @EvilTester 17
- 18. How to remove Intermittent Failure from your Automated Execution 1. Care 2. Investigate 3. Do something about it @EvilTester 18
- 19. How to remove Intermittent Failure from your Automated Execution 1. Decide Intermittent Failure is unacceptable 2. Investigate the cause of Intermittent Failure 3. Mitigate Remove the cause actually fix it Implement a retry strategy might obscure bugs Accept Intermittent Results might provide hints at solutions @EvilTester 19
- 20. Take it seriously We write Automate Assertion checking because we care that those assertions are true for each build of the system. Determinism is important. @EvilTester 20
- 21. High Level Grouping of Common Causes of Intermittency Synchronisation ‐ lack of or poor Parallel Execution ‐ interference Long Running Tests ‐ too long, too risky Automatability ‐ hard to automate system Tools ‐ inappropriate or out of date State Preconditions ‐ not controlled Assertions ‐ wrong or incorrect assumptions Data ‐ not controlled see also Richard Bradshaw and Mark Winteringham "SACRED" Mnemonic. @EvilTester 21
- 22. Will cover Top 3 for each Grouping @EvilTester 22
- 23. Top 3 Common Causes ‐ Synchronisation None Time Based Synchronisation Incorrect App State Synchronisation @EvilTester 23
- 24. Common Solutions ‐ Synchronisation Synchronise on States do not rely on framework Synchronisation Multiple Intermediate States Consider Latency Synchronise in Abstractions not the @Test methods unless @Test specific @EvilTester 24
- 25. Top 3 Common Causes ‐ Parallel Execution Framework not thread safe Tests Interfere Shared Test Environment @EvilTester 25
- 26. Common Solutions ‐ Parallel Execution Independent environments Independent Data Separate Suites rather than threaded execution Create Threadsafe, reusable code Create reusable library abstractions rather than Frameworks Avoid 'static' singleton objects @EvilTester 26
- 27. Top 3 Common Causes ‐ Long Running Tests Sequential rather than Model Based not delineating between: preconditions, process, assertion components tests in flow rather than isolation @EvilTester 27
- 28. Common Solutions ‐ Long Running Tests Understand that more actions == more risk Synchronise prior to each step Consider Model Based Testing Create component test and automated execution playgrounds Minimum assertions @EvilTester 28
- 29. Top 3 Common Causes ‐ Automatability, Automatizability Not Testability: Application has non‐deterministic behaviour Hard to Synchronise Application fails non‐deterministically in live @EvilTester 29
- 30. Common Solutions ‐ Automatability, Automatizability Build apps that can be automated Non‐Deterministic apps need step retry strategies rather than test retry strategies @EvilTester 30
- 31. Top 3 Common Causes ‐ Tools Out of Date Inappropriate Local Tool Infrastructure @EvilTester 31
- 32. Common Solutions ‐ Tools Use the right tool for the job Keep your tooling environment controlled and up to date Change your approach to take latency into account process on server return results return source, process on execution client @EvilTester 32
- 33. Top 3 Common Causes ‐ State Preconditions Not Checking State Preconditions at start of test Not controlling state preconditions prior to test Precondition setup using same tool @EvilTester 33
- 34. Common Solutions ‐ State Preconditions control data precondition state setup ‐ whatever works http, db, api ‐ 'hack it in' avoid dependencies between execution unless a long running test @EvilTester 34
- 35. Top 3 Common Causes ‐ Assumptions Encoded in Assertions Assert on an Ordered Set Assert on Uncontrolled Data Assertion Tolerences @EvilTester 35
- 36. Common Solutions ‐ Assumptions Encoded in Assertions Logging so you can interrogate failure afterwards Ability to re‐run tests with same data and setup @EvilTester 36
- 37. Top 3 Common Causes ‐ Data Missing Data Externally controlled data Uncontrolled Data @EvilTester 37
- 38. Common Solutions ‐ Data Create data for each test Avoid test dependencies Avoid re‐using data between tests Check data as a precondition Data synchronisation on all precondition data @EvilTester 38
- 39. Summary Your Test Execution is not 'flaky', it is failing intermittently It is possible to remove intermittent failures, even when automating through a GUI Commons solutions: synchronisation, data control, environmental isolation @EvilTester 39
- 40. Other talks to watch Alister Scott, GTAC 2015: Your Tests Aren't Flaky https://www.youtube.com/watch?v=hmk1h40shaE Richard Bradshaw, "Your Tests aren't Flaky, You Are!" Selenium Conference 2017 https://www.youtube.com/watch?v=XnkWkrbzMh0 @EvilTester 40
- 41. Other talks to watch Craig Schwarzwald, SAY GOODBYE TO THE “F” WORD … FLAKY NO MORE! https://www.youtube.com/watch?v=2K2M7s_Ups0 Mark Winteringham ‐ REST APIs and WebDriver: In Perfect Harmony https://www.youtube.com/watch?v=ugAlCZBMOvM Search also for: Flaky Selenium, Flaky Automation, Flaky Test Automation @EvilTester 41
- 42. End Alan Richardson www.compendiumdev.co.uk Linkedin ‐ @eviltester Twitter ‐ @eviltester Instagram ‐ @eviltester Facebook ‐ @eviltester Youtube ‐ EvilTesterVideos Pinterest ‐ @eviltester Github ‐ @eviltester Slideshare ‐ @eviltester @EvilTester 42
- 43. BIO Alan is a Software Development and Testing Coach/Consultant who enjoys testing at a technical level using techniques from psychotherapy and computer science. In his spare time Alan is currently programming a Twitter client called ChatterScan, and multi‐user text adventure game. Alan is the author of the books "Dear Evil Tester", "Java For Testers" and "Automating and Testing a REST API". Alan's main website is compendiumdev.co.uk and he blogs at blog.eviltester.com @EvilTester 43