Ten10 Seminar: Test Automation, Tooling and the Future (slides)
1 like•309 views
This document discusses test automation and the future of testing. It begins by asking questions about whether software engineers can replace test analysts, if automation can replace manual testing, and how AI tools will impact automation test engineers. It then discusses challenges with the "test" part of test automation. The next sections cover how test automation maturity has evolved from record and playback to more robust approaches. It also discusses when test automation may not be the answer, and how AI and machine learning can be used for visual validation, optimizing test execution, automating automation, and testing AI applications and systems. The document concludes by recapping the key questions.
Ten10 Seminar: Test Automation, Tooling and the Future (slides)
- 1. Copyright Ten10 2018 Test Automation, Tooling & The Future 21st November, 2018
- 2. Copyright Ten10 2018 Are Commercial Automation Tools Dead? Chris Thompson, Consultancy Services Director, Ten10
- 3. Copyright Ten10 20183 Does the relentless rise of open source functional automaton tools herald the end of the commercial tool market? How have commercial and open source functional automation tools evolved over the last ~20 years? How does the market look today, and what does the future hold? Introduction
- 4. Copyright Ten10 20184 Tools used for: • Functional test automation of application user interfaces Functional automation tools : definition • Functional test automation via APIs and service web interfaces
- 5. Copyright Ten10 20185 Commercial tools: “the early years” and beyond
- 6. Copyright Ten10 20186 2000 2005 2010 2015 2020 (2006) (2012) (2015) (2017) (2006) (2003) (2008)(2000) (2005) (2006) (2011)(2002) (2009) (2008)
- 7. Copyright Ten10 20187 2000 2005 2010 2015 2020 (2006) (2012) (2015) (2017) (2006) (2003) (2008)(2000) (2005) (2006) (2011)(2002) (2009) (2008) Typical characteristics: • Standalone, licenced, “one size fits all” • Multiple technology support • Framework and execution • Reporting and analysis • “Scriptless” automation • Data and keyword driven approaches • Code editors • Image based / OCR techniques • Aimed at “non technical” testers • Commercial support and training
- 8. Copyright Ten10 20188 The rise of open source
- 9. Copyright Ten10 20189 2000 2005 2010 2015 2020 (2004)
- 10. Copyright Ten10 201810 2000 2005 2010 2015 2020 (2004)
- 11. Copyright Ten10 201811 (2005) (2005) (2012) (2008) 2000 2005 2010 2015 2020 (2004) (2009) (2010) (2012) (2013) (2013) (2013) (2011) (2008) (2015) (2011)
- 12. Copyright Ten10 201812 (2005) (2005) (2012) (2008) 2000 2005 2010 2015 2020 (2004) (2009) (2010) (2012) (2013) (2013) (2013) (2011) (2008) (2015) (2011)
- 13. Copyright Ten10 201813 (2005) (2005) (2012) (2008) 2000 2005 2010 2015 2020 (2004) (2009) (2010) (2012) (2013) (2013) (2013) (2011) (2008) (2015) (2011) Typical characteristics: • Web and mobile technology focussed • Separate framework and execution • Limited analysis and reporting • Very limited Windows / legacy support • More developer / dev in test focussed, multiple language and IDE integration • Lightweight and agile • “Scripted” / coded automation • Community support
- 14. Copyright Ten10 201814 2000 2005 2010 2015 2020 (2008) (2011) (2010) (2010)
- 15. Copyright Ten10 201815 The “new breed” of tools
- 16. Copyright Ten10 201816 2000 2005 2010 2015 2020 (2016) (2014) (2015) (2018) (2017)
- 17. Copyright Ten10 201817 2000 2005 2010 2015 2020 (2015) (2015) (2016)
- 18. Copyright Ten10 201818 How have the “traditional” vendors responded?
- 19. Copyright Ten10 201819 2000 2005 2010 2015 2020 (2006) (2012) (2015) (2017) (2006) (2003) (2008)(2000) (2005) (2006) (2011)(2002) (2009) • Greater language support and IDE integration • Freemium models • Light(er)weight to support Agile/DevOps • Selenium and CI integration • Appealing to both Testers and Developers • Ground up re-write • Multiple language and IDE support • Lightweight to support agile / DevOps • Current and legacy technology support • Webtestit • IDE for building Selenium and Protractor tests
- 20. Copyright Ten10 201820 Is it working?
- 21. Copyright Ten10 201821 2018 Survey
- 22. Copyright Ten10 201822 • Traditional commercial tool vendors under increasing pressure and are not keeping pace with technology developments • Innovation is key to survival • Trends around agile, DevOps, continuous testing and continuous delivery have changed the landscape and user base significantly • Convergence of test and development driving an explosion in tools • New commercial vendors providing platforms for open source tools and freemium models, whilst supporting latest technology trends • Progress in the AI / machine learning automation space dominated by commercial players Are commercial automation tools dead?
- 24. Introducing Selenium to an Enterprise Presented by Adrian Brown 21/11/2018 Introducing Selenium to an Enterprise 24
- 25. An introduction to TP ICAP Introducing Selenium to an Enterprise 25
- 26. TP ICAP – what we do • We are the world’s largest interdealer broker • We have offices in 31 countries • Our businesses have over 73 offices globally and we operate in financial and energy and commodity markets across the world 26
- 27. Fusion - our journey to the cloud Strategic Solutions Private data centres Transformation; move to AWS cloud Manual everything, complex to change Automation and immutable infrastructure Legacy Challenges Fusion: “A unified trading application, covering multiple asset classes” 27 Unsupported Adobe Flex, monolithic stacks Rich HTML5 application, microservices
- 28. Summary Developing modern software requires adopting the right level of automation for testing. By creating simple tooling and removing complex tasks we make Selenium automation attractive and achievable for the team. Introducing Selenium to an Enterprise 28
- 29. Introducing Selenium to an Enterprise 29 What did we solve?
- 30. Before you even start • Enterprise challenges • IT security, compliance and policies • Proxies and firewalls • Zero admin rights, software install barriers • People that you will never meet… Introducing Selenium to an Enterprise 30
- 31. The actual problem • HTML automation testing (capture replay) • Application(s) and component(s) • Streaming data, an environment, cookies • People and process • Getting started… Introducing Selenium to an Enterprise 31
- 32. A solution • Simple tools straight to the desktop • Local development experience, CI with SauceLabs • Automating config, binaries and process management • Packaging it up, versioning and updating it • Running tests before we deploy Introducing Selenium to an Enterprise 32
- 33. Strategy • Set the vision, change it as we go, talk about it • Influence by demonstration • Invest in creating infrastructure, make it super easy • Satisfy IT security with automated controls • Make the team want to be part of creating tests Introducing Selenium to an Enterprise 33
- 34. Automation enablers • NodeJS tooling “@icap/automate” • JavaScript WebDriverIO API for Selenium • Simple BDD and assertion framework (Mocha, Chai) • DevOps tools; GitHub, Nexus, NPMO, Ansible, TeamCity • SauceLabs for our target user config Introducing Selenium to an Enterprise 34
- 35. Some principles • Write small discrete tests, fail fast • Tests live with application source code • Make it super easy for anyone to contribute • Target the browsers / OS that matter most • Remove redundant tests over time Introducing Selenium to an Enterprise 35
- 36. Test infrastructure Introducing Selenium to an Enterprise 36 CI GitHub SauceLabs Engineer Tooling
- 37. • Clone and install HTML repo (app) npm install • Describe and write test(s) under folder automation-test • Run test(s) grunt automate-local • Commit to GitHub • CI runs test(s) grunt automate-saucelabs The experience Introducing Selenium to an Enterprise 37
- 38. The result • 6 minutes to experience “hello world” • GitHub, Node, NPMO, then install and run • Upskilled the team • Greater involvement with UI developers • Valuable tests, more time exploratory testing Introducing Selenium to an Enterprise 38
- 39. Extending the tooling • Performance testing challenge • Distributed Selenium solution • A local experience for creating tests • Use Cloud SaaS (testable.io) • Reuse functional tests for OpenFin (desktop) • Generate config, manage binaries • Runnable from CI • Zero-effort for engineer Introducing Selenium to an Enterprise 39
- 40. Tackling Identity Today, Tomorrow, and its Vision for the Future Thank you 40
- 41. Copyright Ten10 2018 Is Automation Always the Answer? And What Does the Future of Test Automation Look Like? Ash Gawthorp, Director, Ten10
- 42. Copyright Ten10 201842 Ash Gawthorp Director, Ten10 Introductions 42
- 43. Copyright Ten10 201843 Key questions 43 Can a software engineer essentially replace the role of test analyst? Can automation replace the requirement for manual testing? How will the new generation of AI automation tools impact the role of the automation test engineer? 1 2 3
- 44. Copyright Ten10 201844 The “Test” in test automation 1. What should we be testing? 2. Are we achieving the right level of coverage? 3. Are we asserting on the correct elements? 4. Identifying, selecting, reading and interacting with the required elements 5. Maintaining scripts as build changes occur 6. Decommissioning the test script or refactoring/ combining into other tests ? 100% 75% 50% 25% ☑
- 45. Copyright Ten10 201845 ? The “Test” in test automation 100% 75% 50% 25% ☑☑☑ ☑ ☑1. What should we be testing? 2. Are we achieving the right level of coverage? 3. Are we asserting on the correct elements? 4. Identifying, selecting, reading and interacting with the required elements 5. Maintaining scripts as build changes occur 6. Decommissioning the test script or refactoring/ combining into other tests
- 46. Copyright Ten10 201846 2 31 Test Team Dev Team Test automation maturity Record and playback • Not Robust • Susceptible to change • Repeated code • Relies on waits • Unreliable and Slow Obstructive… • Unwilling to build in libraries • Unwilling to set automation IDs Good software development practices • Separating element code from test code • No repeated code • Waits for conditions • Faster – reduced maintenance Well meaning… • Willing to add automation IDs • Change IDs between builds Robust automation • Defensive code • Multiple selectors • Waits on next state and retries • Increased reliability and resiliency Useful… • Design for automatability • Sets consist IDs • Understands downstream impact • Makes recommendations
- 47. Copyright Ten10 201847 Is Test Automation Always the Answer?
- 48. Copyright Ten10 201848 How many scripts do you need to write? How many steps in each script? How long to write a step? How long does it take to execute each run? How many times do you run it? How much does the app change? How much maintenance is required? How long before the application is replaced/changed? Test automation maturity
- 49. Copyright Ten10 2018 AI and Test Automation
- 50. Copyright Ten10 201850 50 What do we mean when we talk about AI in testing?
- 51. Copyright Ten10 201851 51 What do we mean when we talk about AI in testing?
- 52. Copyright Ten10 201852 Machine learning is a form of applied artificial intelligence, based on the theory that systems that can change actions and responses as they are exposed to more data will be more efficient, scalable and adaptable for certain applications compared to those explicitly programmed (by humans). Machine learning
- 53. Copyright Ten10 201853 AI is already coming on fast, he notes, and although it’s fairly technical, AI is generic enough to have a wide range of applications. “And you can expect to see changes in the testing tools,” including predictive analytics in testing algorithms and in the applications being tested.” Michael Fritzius, president of Arch DevOps LLC https://techbeacon.com/state-test-automation-tools-top-trends-challenges-2018
- 54. Copyright Ten10 201854 AI and test automation How AI and machine learning can be used in testing How the testing can be automated for AI systems and apps
- 55. Copyright Ten10 201855 How AI and Machine Learning Can Be Used in Testing
- 56. Copyright Ten10 201856 • Checking things are the right colour, shape, size • Especially true a variety of devices, different screen resolutions pixel densities etc. • ML based visual validation tools > Applitools > mabl How AI and machine learning can be used in testing - Visual Validation
- 57. Copyright Ten10 201857 • Help to manage volume of reporting and log files • Identify existing, new and duplicate failures • Speed up process to prioritise failures How AI and machine learning can be used in testing - Optimising execution and effort to fix failures
- 58. Copyright Ten10 201858 • Application spidering, screenshots, html, load times, roundtrips • Compares app changes by comparing current scan to previous patterns • Useful for spotting non-visual changes • Safety-net rather than primary source of testing How AI and machine learning can be used in testing Automating the automation, part 1 – Application Spidering
- 59. Copyright Ten10 201859 • Elements automatically identified through a number of selectors / paths • Tool learns around changes and relationships in the DOM • Scripts adjust themselves automatically How AI and machine learning can be used in testing Automating the automation, part 2 – Reducing maintenance
- 60. Copyright Ten10 201860 Automating the testing of AI systems and applications
- 61. Copyright Ten10 201861 • Moving from deterministic to non-deterministic • Moving away from certainties into probabilities • Models will change in production • May require functional testing in live Using tools to test AI apps - What’s changed?
- 62. Copyright Ten10 201862 • Will require data knowledge of AI/ML to build models • Reliance on testing APIs with many datasets • Affinity with machine learning / data-science Using tools to test AI apps - Moving the skills goalposts
- 63. Copyright Ten10 201863 Recap and summary 63 Can a software engineer essentially replace the role of test analyst? Can automation replace the requirement for manual testing? How will the new generation of AI automation tools impact the role of the automation test engineer? 1 2 3
- 64. Copyright Ten10 201864 Recap and summary 64 Can a software engineer essentially replace the role of test analyst? Can automation replace the requirement for manual testing? How will the new generation of AI automation tools impact the role of the automation test engineer? 1 2 3
- 65. Copyright Ten10 201865 Recap and summary 65 Can a software engineer essentially replace the role of test analyst? Can automation replace the requirement for manual testing? How will the new generation of AI automation tools impact the role of the automation test engineer? 1 2 3
- 66. Copyright Ten10 201866 Recap and summary 66 Can a software engineer essentially replace the role of test analyst? Can automation replace the requirement for manual testing? How will the new generation of AI automation tools impact the role of the automation test engineer? 1 2 3
- 67. Copyright Ten10 2018 T 020 3697 1444 E [email protected] W www.ten10.com