Automated Testing – Web, Mobile, Desktop - Challenges and Successes

Editor's Notes

• #2: It shares the progress Intuit has made to incorporate automated testing across all platforms, where there have been problems, and goals for the future. Also shares a journey of transition of manual testing to Automation testing for accessibility. This presentation is for other companies that are working towards automated testing. While this presentation provides information on 3rd party testing tools, it is not an endorsement or critique of any product. It reflects our experience as we’ve researched and implemented testing strategies. Photo: cup of robots by hobvias sudoneighm https://www.flickr.com/photos/striatic/1276095/
• #3: Coop Intern at Intuit with the TurboTax team and partly did a project with Ted Drake who introduced me to this amazing field. Later came back as an intern for the Live Community which is like help support Team and made their product accessible. Joined as a full time Employee in 2015 to drive accessibility automation initiative. Ted leads Intuit’s accessibility initiatives via education, training, and working with product managers. The hand represents Intuit’s Special Needs and Ability Network for employees and their family members
• #4: HTML components WorldSpace Accessibility library with Martini Tenon/PayPal packages Android components Lint Architecture Google accessibility checks IOS component Xcode plugin
• #5: Intuit is software company that develops financial and tax preparation software and related services for small business, Accountants and Individuals. One of the main product of Intuit is Turbotax. Since turbotax supports 1000+ plus forms, it is very difficult to evaluate accessibility for each form manually. Thus decided to automate compliance testing for accessibility.
• #6: Worldspace is external server which tests web based products for accessibility compliance. It crawls through webesites and test for all possible web pages. Intuit has used WorldSpace for several years to do automated testing of our production web sites. It could be installed on internal server, which provides access to non-production servers and JIRA integration WorldSpace tracks compliance. A single error makes a page non-compliant, so timelines are not useful for tracking progress. http://www.deque.com/products/worldspace/
• #7: If you want to test QA server or preprod server, fireeyes comes to rescue. FireEyes plugin allows manual scans and building automation scripts Takes input as a link to the product and QA server can access worldSpace with the help of this bridge. Basically fireyes acts like a bridge if you want to test QA or preprod server on Worldspace. Fireeyes upload all the results after the scan on Worldspace. So that developers can still go to Worldspace and see all the errors on QA or preprod server
• #8: QuickBase (http://quickbase.intuit.com/) Easy to use database for building custom projects. – within the organization keeps a tab on the status of all the products and their status on accessibility against latest rules Keeps a record of the stockholder’s name, all the scores from Worldspace, the last date of scoring.  Worldspace scores is no of critical errors/no of pages scaned. Also we use a system similar to a Credit score where depending on how well the scores are, they are indicated by different colored zones. The colors used are something like in a traffic signal - where green means all is good to go, Yellow indicates that we need to be cautious and watchful, things might become red soon which is a big STOP or danger that which means we need to stop and revisit the accessibility standards for that particular product.
• #9: Martini is an internal tool for UI Test Automation This includes a common set of tests that are shared across Intuit teams The Accessibility Library was built to focus on most common errors, while triggering minimal false errors The initial library was built by the Intuit.com team and has been improved with customer feedback Photo: Martini by Tory Barratt Crane https://www.flickr.com/photos/torimbc/4040067721/
• #10: Incorporating the accessibility library requires minimum effort. The tests will open a page, parse the dom, and begin searching for elements based on xpaths. The tests are run against these elements, such as checking all images for alt attribute. If there is a violation, errors are generated and reported to a text file for QE integration.
• #11: We needed a stable set of pages to run our tests against. Acme Widgets includes good and bad versions of basic HTML pages. If our tests were solid, the good pages would throw 0 errors or warnings. The bad pages should throw lots of different errors. Access the test site: http://fyvr.net/acme/ Download the code: https://github.com/7mary4/a11y-control-test Screenshot test results: tenon.
• #13: Important step was to be able to deliver to a customer and gain feedback. Fuego Web Player Team (CTG) – internal team has successfully incorporated this into their automation framework. FEEDBACK – Library helped identify issues like alt text missing on the player. Customer empathy sessions were conducted which helped gain deeper insight.
• #14: As customers started using this library, it helped us to get lot of feedback such as when the page structure is complex test takes more time to run. Thus it motivates us to improve performance. Since now there is no manual evaluation, product releases were faster and less pain to QE/Developer and catches any issues before we go live.
• #15: Since this library is using by all products we should go and see how all products doing in terms of accessibility.
• #16: Tenon.io is a new product that is API based. The team passes code to the Tenon server, which returns test results in JSON. These results can be easily manipulated to work with multiple bug tracking systems, automation tools, and CI/CD tools. http://tenon.io The tests are more sophisticated than our Martini library Tests provide guidance on error importance and solutions Intuit’s security team was worried about passing our non-production code to an external server. This can be done securely, but is still a concern. This could be solved by installing Tenon on an internal server, which is something we are still investigating. Pricing is reasonable. Support is great.
• #17: https://github.com/paypal/nemo-accessibility Nemo is a node.js based automation framework for browser automation. Nemo creates a Node-JS server to run your favorite accessibility automation library Easily integrates HTML Code Sniffer, Chrome Accessibility Library, or Deque’s aXe library This is something we are evaluating as an alternative to Tenon.
• #18: For mobile applications, the accessibility API is used to find elements for automated testing. The majority of documentation about mobile automation is about how to use these elements within the accessibility API to run automated tests. However, there is little information or libraries for checking if the app is actually accessible. Photo: iphone by Claudio Schwarz https://www.flickr.com/photos/purzlbaum/367889445/ https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UIAutomation.html http://developer.android.com/training/testing/ui-testing/index.html
• #19: http://appium.io/ Appium is an open source tool for automating native, mobile web and hybrid applications on IOS and Android platforms. Also Appium is cross platform. It allows you to write tests against multiple platforms such as IOS and Android using same API.
• #20: An equivalent to martini for the web is Appium for Android devices (Google platform). Appium is an open source automation framework which makes things more easily available for integration. Google suggests test frameworks like Espresso and Robolectric. Google Accessibility provides checkers like Espresso and Roboelectric in order to test the accessibility amongst Android devices. https://google.github.io/android-testing-support-library/ https://google.github.io/android-testing-support-library/docs/espresso/ http://robolectric.org/ AccessibilityChecks: automated accessibility tests: http://developer.android.com/reference/android/support/test/espresso/contrib/AccessibilityChecks.html
• #21: Intuit Mobile testing standard framework Work to integration google a11y checks with appium Decided to make it as CI/CD service Service will run behind the scenes!. So teams who will on board to CI/CD will load there project and we will run that service for all mobile apps. Thus we will let them know of possible accessibility errors in their project.
• #22: Android Lint is a static code analysis tool built into Eclipse and Android Studio. Each problem detected is reported with a description message and a severity level, thus easy to prioritize.
• #23: As you can see here source files for app and .xml needs to parse to the lint tool. And when we build and run the tool we get the output in the form of issue category such as correctness usability, Security, performance, internationalization and accessibility.
• #24: Video on YouTube: https://www.youtube.com/watch?v=jMnWwz6ZFwU
• #25: Interestingly, Appium is available on iOS too. Would be interesting to see how integration with that would serve both platforms (Android and iOS). https://developer.apple.com/library/tvos/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UIAutomation.html
• #26: https://www.youtube.com/watch?v=euEsfNR5Zw4 Xcode is the development environment for any iOS product. As a part of the same, they provide a Xcode plugin that helps identify a11y errors. Is integrated as a part of the build process of the application and like the other warnings/errors in Xcode, we get the one’s related to A11y too. Very less overhead and simple installation by just connecting to the worldspace server through the project. Deprecated due to fragility with new xcode builds
• #27: For the OSX platform, development for the Mac, we are still in the very preliminary stages of researching on how this integration could be done. There are no obvious potential choices for integration into libraries too for this platform. Current automation products PyAtom is first python based library which fully enables GUI testing for mac applications via the apple accessibility API. It has direct access to the API. Frankk allows you to write structured text tests or requirements using cucumber and have them execute against your Mac OS application https://github.com/pyatom/pyatom http://www.testingwithfrank.com/mac.html https://developer.apple.com/videos/play/wwdc2015/406/ https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/01-introduction.html
• #28: Windows is a challenge considering every product is massively different. Tests are created per product. Strategies around accessibility are simplifying our testing strategies. Windows Desktop product is built using C++.
• #29: Since quickbooks desktop is both for mac and Windows there is common layer we use known as QbForms. Now since QBForms was also built using C++ there is very little chance of making it accessible. Thus now effort is to refactor the codebase with C# and .net to make it accessible.
• #30: Long Tail Monitoring: You can’t give everything equal attention, especially within a large company. Automated testing provides awareness for the long tail, those products that often get ignored. For instance, we had a product update their wordpress theme and the error rate skyrocketed. We were able to quickly evaluate the product, provide guidance, and they fixed many of the issues the same day. This wouldn’t have been discovered as quickly without automated testing. https://en.wikipedia.org/wiki/Long_tail Implementing accessibility testing has become significantly easier after incorporating this from a centralized resource. Automated testing forces engineers, QE, and product teams to recognize accessibility errors. This is something they may not have realized prior to the testing. Good testing also provides context and solutions. Testing at the continuous integration stage greatly reduces the cost for fixing accessibility issues.