10 Emerging Test Frameworks for Cross Browser Testing
Download as PPTX, PDF6 likes1,567 views
The document outlines emerging test frameworks for cross-browser testing, highlighting criteria for selecting the right automation tools for various projects. It discusses the comparison between real and headless browsers, along with specific examples and methodologies used by USAA and Panera for web testing. Key considerations include the complexity of projects, available resources, and the importance of frameworks being user-friendly and well-documented.
10 Emerging Test Frameworks for Cross Browser Testing
- 1. 10 Emerging Test Frameworks for Cross-Browser Testing
- 2. Housekeeping • Please participate in our live poll and end-of-webinar survey • Ask us questions in the QA panel • We will send you the recording and slides 9/20/2017 2© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 3. Uzi Eilon CTO, Perfecto Eran Kinsbruner Dir., Product Evangelist Author: The Digital Quality Handbook Agenda How to select the right test automation framework? Cross browser testing landscape - overview USAA tool selection strategy Technical walkthrough demo – Panera RWD Angular site automation Protractor Integration with Perfecto Q & A Bryan Osterkamp Lead Technical Architect, USAA Chaley Stanfield Sr. Analyst, Test Engineer, Panera
- 4. How to select the right testing framework? 9/20/2017 4© 2015, Perfecto Mobile Ltd. All Rights Reserved. Organizational Fit Technical Fit
- 5. Selection Criteria: Organizational Fit Requirements 9/20/2017 5© 2016, Perfecto Mobile Ltd. All Rights Reserved. Project Complexity Quality analysis requirements Test Types & Practices Resources and Skills Availability Test Environment Existing Tech Stack
- 6. Selection Criteria: Technical Fit Requirements 9/20/2017 6© 2016, Perfecto Mobile Ltd. All Rights Reserved. SDLC Process Fit (Integrations, Plugins, etc.) Community size, support and Doc’s Feedback Loop and Reporting Automation Coverage Cloud and Automation at Scale Automation Robustness and Maintainability
- 7. Poll Question 9/20/2017 7© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 8. Web Testing Tools - Download Trends 9/20/2017 8© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 9. Web Market Testing Landscape 9/20/2017 9© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 10. The Biggest Diffs 9/20/2017 10© 2015, Perfecto Mobile Ltd. All Rights Reserved. Real Browser Testing Headless Browsers • Interact with a standard browser • Pros: • Match your users platforms • Easy to navigate and control objects • Cons: • Slow • Complicated setup • Uses Lightweight browser like SW • Pros: • Fast and easy to run • Easy CI setup • Cons • Doesn’t fit UI testing • Limited test coverage capabilities • Cannot cover diffs between browsers • Elements management too complex
- 11. Grouping The Tools 9/20/2017 11© 2015, Perfecto Mobile Ltd. All Rights Reserved. Real Browser (WebDriver) Headless Headless browser SW (can used as standalone) Used for: • End to end testing • Functional testing Used for: • Fast CI – Short cycle • Basic functional testing • Backend services testing • Performance / network traffic
- 12. Headless Example 9/20/2017 12© 2015, Perfecto Mobile Ltd. All Rights Reserved. The Script Connect to the Phantom Use Cases: • Har • Java/Selenium • Join script (headless validation)
- 13. WebDriver Example 9/20/2017 13© 2015, Perfecto Mobile Ltd. All Rights Reserved. The Script: Connect to the browser
- 14. WebDriver Grid 9/20/2017 14© 2015, Perfecto Mobile Ltd. All Rights Reserved. The Script: Connect to a grid
- 15. USAA- Bryan Osterkamp © 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 16. USAA Tool Selection • Define needed capabilities 9/20/2017 16© 2015, Perfecto Mobile Ltd. All Rights Reserved. Selection Criteria End to End Testing BDD/ATDD Friendly Tool Documentation Visual Navigation Testing
- 17. USAA Tool Selection • Define needed capabilities • Identify importance (weight capabilities) 9/20/2017 17© 2015, Perfecto Mobile Ltd. All Rights Reserved. Weight Key 1 – Low importance 3 – Medium importance 5 – High importance Weight Selection Criteria 5 (High importance) End to End Testing 3 (Medium importance) BDD/ATDD Friendly 5 (High importance) Tool Documentation 1 (Low importance) Visual Navigation Testing
- 18. USAA Tool Selection • Define needed capabilities • Identify importance (weight capabilities) • Define scoring key 9/20/2017 18© 2015, Perfecto Mobile Ltd. All Rights Reserved. Scoring Key 0 – Did not meet expectations 2 – Met expectations 3 – Exceeded expectations Weight Selection Criteria Tool X Tool Y Tool Z 5 (High importance) End to End Testing 3 3 3 3 (Medium importance) BDD/ATDD Friendly 3 2 3 5 (High importance) Tool Documentation 0 2 2 1 (Low importance) Visual Navigation Testing 3 3 2
- 19. USAA Tool Selection • Define needed capabilities • Identify importance (weight capabilities) • Define scoring key 9/20/2017 19© 2015, Perfecto Mobile Ltd. All Rights Reserved. Weight Selection Criteria Tool X (weighted) Tool Y (weighted) Tool Z (weighted) 5 (High importance) End to End Testing 3 5 x 3 = 15 3 5 x 3 = 15 3 5 x 3 = 15 3 (Medium importance) BDD/ATDD Friendly 3 3 x 3 = 9 2 3 x 2 = 6 3 3 x 3 = 9 5 (High importance) Tool Documentation 0 5 x 0 = 0 2 5 x 2 = 10 2 5 x 2 = 10 1 (Low importance) Visual Navigation Testing 3 1 x 3 = 3 3 1 x3 = 3 2 1 x 2 = 2 Total 27 34 36 Scoring Key 0 – Did not meet expectations 2 – Met expectations 3 – Exceeded expectations
- 20. Infrastructure as a Freeway 9/20/2017 20© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 21. Panera Bread - Chaley Stanfield © 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 22. Panera Web Testing Overview • What is our methodology/strategy? • TDD • Agile methodology • Short sprints • Planning that involves the full team • Early delivery • 2-3 days development, testing starts by 2nd or 3rd day • Testing for new functionality wrapped up 2- 3 days before end of sprint to allow time for bug fixes and regression testing • Allows for early bug finds and fast turnaround on fixes • Short sprints allow teams to assess processes and adapt quickly 9/20/2017 22© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 23. Panera Web Testing Overview • How do we decide what to automate? • Best candidates are: • Unchanging • Tested repeatedly • ‘Tried and true’ functionality • Our best candidates are regression testing and data testing • We normally do not script for new pieces of functionality until they have been live at least a couple sprints • We look to QA for input on what has the most ROI 9/20/2017 23© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 24. Panera Web Testing Overview • What were our requirements for an automation framework? • AngularJS-friendly • Used for integration and e2e testing • Support for desktop and MR apps • Ample, easy-to-understand documentation • QA-friendly reporting • Easy to configure • Multi-threaded testing • Supports POM • Similar to Selenium WebDriver (original automation framework) 9/20/2017 24© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 25. Panera Web Testing Overview • Test Structure (Jasmine) • conf.js file • Environments (baseUrl) • Selenium Address • Local • Perfecto • Suites • Reporters • Jasmine HTML Report • Perfecto Report • Multicapabilities • Browser • Platform (for Perfecto) • Multi-threaded testing 9/20/2017 25© 2015, Perfecto Mobile Ltd. All Rights Reserved. • appPage.page.obj.js files • Locates Web Elements • appPage.js files • Logic • Acts on Web Elements found in appPage.page.obj.js files • Assertions • testSuite.spec.js files • Tests • Pieces together methods in appPage.js files to form a coherent test flow • Broken down by functionality
- 26. Panera Web Testing Overview • Challenges • Maintenance • Can’t check third party services/our BOH • Browser updates • Difficult to change browser versions on the fly • Difficult to run on multiple platforms/devices • Making automation executable by anyone on the team 9/20/2017 26© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 27. Demo • Panera website testing using Protractor
- 28. Giveaway for Webinar Attendees/ Q&A
- 29. Thank You
Editor's Notes
- #4: Chaley Stanfield Sr. Analyst, QA Engineers Test Engineer Panera
- #17: Often, when evaluating a potential new tool, it can be overwhelming and difficult to quantify which tool to select. The approach our team takes is to outline success criteria very clearly, to ease the evaluation and decision process. We’ll use Eran’s selection criteria as an example. Walk through needed capabilities (selection criteria) definition
- #18: Once you identify the needed capabilities, then you’ll want to identify the importance of each capability. As in, how critical is the capability to your decision? This will help you “weight” each capability which you’ll see is very important when making a decision later on. Also useful when you have someone that wants to include criteria that you don’t think is as important (you just add it and give it a lower weight). Helps get away from “shiny penny” distractions.
- #19: Once you have identified the importance, then you’ll want to define your scoring key. This is how you will actually document whether or not the tool you are evaluating has met (or not met) your expectations for that particular capability. This can also be useful if you are asking multiple people to evaluate the tool and provide feedback. (I usually hide the weighting when asking others to evaluate the tool). The scores are pretty close…
- #20: The final step is to put it all together and score the applications together. You’ll see we just do basic math to determine the final decision. Walk through the math and show the total that takes into account all aspects. Note that it is always good to do a “reality check” to make sure the scores reflect overall needs (or even add additional capabilities discovered throughout the evaluation).
- #21: The Infrastructure as a freeway slide will go here… Essentially, I’ll talk about what happens when other teams want to utilize a different tool than the one you “selected” and use my “Infrastructure as a Freeway” concept to describe the concept and how I handle that. Here is the concept if you want more specifics: https://www.linkedin.com/pulse/software-infrastructure-freeway-bryan-osterkamp/