Creating a successful continuous testing environment by Eran Kinsbruner
1 like201 views
This document discusses continuous testing and provides an overview of key concepts. It defines continuous testing as executing automated tests as part of the software delivery pipeline to rapidly obtain feedback on release candidates. The benefits of continuous testing discussed include fast feedback, continuous visibility, and testing that matches different skill levels. It also covers topics like test authoring best practices, key performance indicators for continuous testing, and the potential future role of machine learning. Continuous testing is presented as a way to mature an organization's path toward DevOps practices.
Creating a successful continuous testing environment by Eran Kinsbruner
- 1. Maturing Your Path Toward DevOps with Continuous Testing
- 2. AGENDA • State of Continuous Testing? What are the Benefits of CT? • Fast Feedback and Continuous Visibility • Testing that matches your skillset & What to Automate? • Test Authoring and Coding Best Practices • KPI for Continuous Testing Measurements • Bonus: ML and Future of CT Eran Kinsbruner Dir., Lead SW Evangelist andAuthor, Perfecto Twitter: @ek121268 (https://twitter.com/ek121268) Blog: http://continuoustesting.blog LinkedIn: https://www.linkedin.com/in/eran-kinsbruner-4b47a81/
- 3. Continuous Testing for Digital Has Never Been More Complex 3© 2018, Perfecto Mobile Ltd. All Rights Reserved. Augmented Reality Virtual Reality IOT Artificial Intelligence Mobile Progressive Web Continuous Testing for Digital
- 4. WHAT IS CONTINUOUS TESTING? 5/24/2018 4© 2018, Perfecto Mobile Ltd. All Rights Reserved. Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.
- 5. 5 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io What the DevOps Process Probably Looks Like Process Impact Organizational Impact 2-3 WEEKS 1-3 WEEKS ❌ Unstable ❌ Labor-intensive ❌ Cluttered ❌ Slow • Manual testing. • Unreliable and flaky executions. • Long time to analyze results. • Increases time to release. • Increases risk and reduces flexibility during the cycle. • Reduces innovation time versus bug fixes time. • QA isn’t part of the daily cycle since they have limited feedback to share. • Testing holds back innovation. ”End of Cycle” Testing
- 6. 6 | Automated Testing in DevOps perfecto.ioI. The State of Digital Transformation Mobile Testing Challenges Complex test creation leads to high percentage of manual testing Untrusted test automation leads to high percentage of manual testing Hard and costly to manage in- house lab Outdated mobile test Lab Reporting noise (too many false negatives) resulting in irrelevant test feedback Unable to execute parallel tests and scale with the right amount of coverage
- 7. Key Benefits Of a Mature DevOps State of DevOps Report, 2018 - DORA
- 8. Value of CT and DevOps State of DevOps Report, 2018 - DORA State of Software Testing - Panaya
- 9. Licenses In-use Disconnected Orchestration issues Network Availability Data Backend issues Objects Codding Time Other Scripts & FW issues • 10% of devices, causing 80% of lab issues Networking Stability Config Lock Lab issues Reasons for “Distrustful automation” (Functional UI Testing, mobile) 52% success rate Backend 30% Lab 15% Orchestration 15% Scripts & FW 40% 80% of issues have a pattern
- 10. Licenses In-use Disconnected Orchestration issues Network Availability Data Backend issues Objects Codding Time Other Scripts & FW issues • 10% of devices, causing 80% of lab issues Networking Stability Config Lock Lab issues Smart insights are key for Test Automation success Backen d 30% Lab 15% Orchestration 15% Scripts & FW 40% What’s wrong with my Scripts What’s wrong with my Executions What’s wrong with my Lab What’s wrong with my Backend
- 11. 11 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Let’s Focus on Testing Testing Is Led By 3 Main Personas B U S I N E S S T E S T E R S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G ) S O F T W A R E D E V E L O P E R Performs exploratory and structured manual testing Creates functional and nonfunctional code-based test automation scripts (via Selenium, Appium, etc.) Creates unit and build acceptance tests
- 12. How should testing types be dispersed among the three personas and throughout the DevOps pipeline? SDETDeveloper Business Tester: Manual? BDD? Codeless?
- 13. 13 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2/28/2019 13© 2015, Perfecto Mobile Ltd. All Rights Reserved. BDD/Codeless Scripting Flaky and common test failures and complex manual test cases Interactive Tests UI/UX manual tests High Skilled (Dev/SDET) code-based scripting Developers and SDETs Business Testers Ownership Business Testers Ownership Balancing Test Creation for the Three Different Personas with the Right Tools
- 14. 14 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io 2/28/2019 14© 2015, Perfecto Mobile Ltd. All Rights Reserved. Interactive Tests UI/UX manual tests Developers and SDETs Business Testers Ownership Business Testers Ownership Balancing Test Creation for the Three Different Personas with the Right Tools
- 15. Test AUTOMATION– What to Automate? 1. What’s the test engineer’s gut feeling 😊😊 2. Risk calculated as probability to occur and impact to customers 3. Value – does the test provide new information and, if failed, how much time to fix? 4. Cost efficiency to develop – how long does it take to develop and how easy is it to script? 5. History of test – volume of historical failures in related areas and frequency of breaks Source: Angie Jones
- 16. Some Recommended CT Measurements (KPIs) • How fast are testing activities moving, and what is slowing down these activities? • Test flakiness • Test duration • % of automated vs. manual tests • Application quality measurements • # of escaped defects and in which areas • MTTD – mean time to detection of defect • Build quality • Pipeline efficiency measurements • # of user stories implemented per iteration • Test automation as part of DoD across iterations • Broken builds with categories • CI length trending • Lab availability and utilization • Quality costs measurements • Operational costs, lab availability issues • Cost of hardware/software • Costs of defects by severity and stage
- 17. Key ML Use Cases In Test Automation • Recognize objects • Transcribe speech – Accessibility • Make quality related decisions based on data • Identify Trends and/or Patterns • Security use cases – Identify signatures e.g. 2/28/2019 17© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 18. 18 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Test Authoring Workflow and Skillset Required Define manual test scenarios, BDD & user stories style scenarios Create scripts in Java/JS etc. from within IDEs (IntelliJ, Eclipse) Define Objects using Object Spy tools, DOM viewers etc. Page based test steps creation Insert Visual validations and Assertions Typically takes longer to develop, more complex (~6 Hours per test) Medium-High code development skills required Test Maintenance Test changes are required proactively, tests are more error-prone to changes in production, objects Tests are managed and maintained in an SCM tool (GIT, Perforce) Tests are recorded with no coding in most cases Codeless tools UI used for creation Objects are “learned” and generated on-the-fly Test scenarios are often less structured and more exploratory based (flow based) Time to author is shorter, can be ~1 hour per test Test reusability is easy Business tester, lower to no-coding skills is sufficient Tools support self-healing with auto test correction Local or cloud test versioning is used, no GIT integration SDET/Developers Business Testers (&Developers?) Mindset and Workflow Changes in Test Automation
- 19. 19 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Tools Maturity Highly mature, includes samples, best practices, documentation Integrations exists to ALM tools, Defect management etc. Testing Types and App Types Supported Functional, API, Load, others Mobile Native (Appium e.g.) and Desktop Web (Selenium e.g.) Emerging technology, less mature, no well- defined guidelines and practices (e.g. guide to shift from standard to ML?) Web is more mature than mobile codeless, basic integrations only Mostly functional (E2E) and basic API Most support web, mobile is lagging behind SDET/Developers Business Testers (&Developers?) Test Execution Configured environment using tools like TestNG data provider Execution done locally, via CI, cloud-based Execution management is built-in the codeless tools Execution done locally, via CI, cloud-based Mindset and Workflow Changes in Test Automation
- 20. 20 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io Perfecto’s Smart Automation for Continuous Testing SMART Test Execution Fast and parallel test execution with multi-team orchestration abilities and management. SMART ML-Driven Test Reporting and Analysis "Single pane of glass" provides visibility and scales to support millions of test results. SMART Test Creation Automated creation that matches your team’s skillset (Appium, Espresso, XCUITest, Quantum BDD). SMART CLOUD Lab Always on and stable. Always up to date. Supports all mobile OS and platforms. The Perfecto human factor increases your chances to succeed. Black Belt Testing Experts Training Dedicated Success Manager 24/7 VIP Support
- 21. Q & A
- 22. Thank you