Persona Testing - Deep Dive

Editor's Notes

• #3: Focus on Banking/Financial Use Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Credentials in code: To run the sample remember to add your credentials and Lab URL String host = “myLab.perfectomobile.com"; capabilities.setCapability("user", “myUser"); capabilities.setCapability("password", “MyPassword"); Selenium and Perfecto Tree Uses BB&T U application – tried to login with username, password and validates error message for unsuccessful login Also opens device browser, goes to perfecto sample page, logs in and validates successful login
• #4: The challenge  Customers using a mobile application are themselves mobile. They enter an elevator and switch from WiFi to Cellular throughout the day. Calls and messages pop up, applications run in the background, location changes as users are on the go. A standard functional test is not enough, the testing needs to consider the environment that affects the application.
• #5: Different users at different points of the day go through different experiences (user flows of the app) in different environments. For example, a travelling business person who is on and off planes and is exposed to poor reception; and a gamer will have a lot of CPU consuming apps running in the backgrounds. Different devices are configured differently, in terms of permissions, access, the number of background apps. For example, there may be apps using location. There is a lot happening outside the control of my app.
• #6: In person interactions with this example Financial Institution ( B of A) has declined 23% from 2009 to present and online mobile transactions have increased 94%. Online/Mobile Banking is growing exponentially!
• #7: Let’s dig into Quality. Mobile quality differs dramatically from web. Specifically it is the user conditions. Therefore Quality teams must look beyond testing function toward testing functions across all user conditions. The objective is to set defining and controlling user conditions as a key required capability. This can be expressed as a day in the life of how you use your favorite app. Think about traveling from point a to b and various network, device, device resources consumed and potentially conflicting apps that are also running on your device. App performance is influenced by a variety of factors, such as degraded network conditions, frequent network transitions back and forth between cellular and WiFi, on board sensor or resource conflicts along with a non-stop stream interruptions – SMSs, notifications, system pop-ups and phone calls. These factors impact app performance. Mobile app quality programs must account for these factors during testing to consistently deliver high-quality end user experience.
• #8: Meet real user experiences and bring the real user into the lab. Test conditions, unplanned events, environment changes and expand the test coverage and predictability for production via the continuous quality lab. ** Adding Wind Tunnel to an existing script will effortlessly expand your test coverage to include real user experience. Testing what is going on outside the control of the application and not only what is inside the application. This includes the application server, the carrier, and the device OS. For example, testing what happens when an incoming call event occurs during a transaction in a bank application. The solution Wind Tunnel offers the following abilities to extend coverage to real user conditions. Pre-defined Personas that simulate network, location and other properties Points of interest that identify crucial application flows Timer reporting ability to track performance of the application All these abilities are easy to add to the test and visible in a new reporting interface designed to help understanding and drill down of issues.
• #9: Defining the real user environment is a challenge. Perfecto has created the concept of Persona in which a type of user is defined. Each persona contains a series of properties that together simulate a real end user. By setting a persona all its pre-defined properties are set. It is possible to use the Perfecto provided profiles and also to create custom profiles. There are five different available personas ready to use (with unique capabilities). For example, Georgia, a 47 year old high income, corporate road warrior living in Manhattan, traveling to San Fran & London. Her traits include: Network: Constantly connected, frequent network switches across various WiFi and cellular networks Brand of choice: Apple & Verizon App loyalty: Intense social user, WhatsApp, Mobile Banking and PO & Expense Report approval. Georgia’s list of background apps allows testing your app, considering the resources consumed by the other apps constantly running on Georgia’s device, such as high CPU, network consumption, camera, etc. These traits are translated into capabilities, creating Georgia's persona that can be easily incorporated into your test.
• #10: There are five different available personas ready to use. See above image and detailed table for their unique capabilities. Existing tests can be executed across the set of personas. The personas can be parameterized and run utilizing common continuous integration servers, such as Jenkins or test execution managers such as TestNG.
• #11: Perfecto agnostically supports a wide variety of languages and test frameworks
• #12: To set a persona use the windTunnelPersona capability. Example - java To tailor an existing persona with specific values, define new capability values to override those that have already been defined in the persona.
• #13: What is a persona? Continued… A persona is a set of capabilities stored in the format of a JSON file located in the Perfecto lab repository. The pre-defined persona JSON files are located in PUBLIC:Personas folder (for example: PUBLIC:Personas/Georgia.json). When you set values of elements common to a persona definition, you override the persona’s original corresponding value (e.g. setLocation or mobile:location:set override the persona location). Notice Georgia has more than one device, therefore, unless you have overridden that value, one of her devices is randomly selected for the test execution. The network profile can be more easily defined using the constants listed in the Wind Tunnel post listing the Network Conditions for Wind Tunnel. https://community.perfectomobile.com/posts/1110522 Don’t forget to override this value by null, if you’re using a cloud with no network, or your test will fail. If you do not need any specific persona attributes, yet you do wish to add the Wind Tunnel mechanism to your test, you can avoid adding a persona altogether, use other Wind Tunnel commands, and still generate the report. A reference persona with mostly empty fields is automatically added to your capabilities, with some outline image appearing in the Wind Tunnel report. When using a reference persona remember to: Set the device, or your test will fail. Use the vitals start/stop command to ensure the generated Wind Tunnel report includes the device vitals.
• #14: Note that while the view of the device is still in portrait, the display of the application is in landscape. The report will show the device in the correct orientation.
• #15: The Wind Tunnel Point of interest (POI) allows mobile teams to pin point real user specific issues around performance, functionality and more. By using POI users can get a detailed dashboard with view into the actual error for a specific transaction. For example, a purchase transaction that takes too long (see the Wind Tunnel’s Timer functionality), or the CPU consumption of the running Facebook app (in the vitals’ CPU graph). This allows identifying app quality issues in specific environment under specific conditions before app delivery to production, assuring end-user satisfaction and eliminating redundant long debugging cycles.
• #16: Points of interest are the points in the script that the user may want to log, in order to highlight in the Wind Tunnel report. For example, loading a new page. The user can also use the point of interest to indicate a logical failure such as incorrect data showing in the app. ** For details and code snippets see the Adding Wind Tunnel to your test post in the Wind Tunnel guide. https://community.perfectomobile.com/series/24692/posts/1048049
• #17: Wind Tunnel introduces a new interactive test report enabling real time analysis. Using a persona in your test will create a Single Test Report. Download the Perfecto report from your test, using the downloadReport method, and click on the link to open the Single Test report. This report is opened in a browser. Alternatively, you can open the Report from the link that appears on the report page in the Reporting tab of the CQ Lab interface. You can also retrieve the report URL via: String reportURL = (String) (driver.getCapabilities().getCapability(WindTunnelUtils.WIND_TUNNEL_REPORT_URL_CAPABILITY)); Understanding the results: When reading the Single Test Report, first take a look at the points of interest reported from the test and identify any failures (commands and timers). Take a look at the previous command, the app & OS vitals (graph), the device screenshots and video. (When selecting a specific point in the Single Test Report timeline, the device video is synced to that time.) ** For details, the Wind Tunnel story, and code snippets, see the Viewing results post in the Wind Tunnel guide. https://community.perfectomobile.com/series/24692/posts/1048048
• #18: Notes: next slide we’ll run the sample test. Focus on Banking/Financial Use Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Credentials in code: To run the sample remember to add your credentials and Lab URL String host = “myLab.perfectomobile.com"; capabilities.setCapability("user", “myUser"); capabilities.setCapability("password", “MyPassword"); Selenium and Perfecto Tree Uses BB&T U application – tried to login with username, password and validates error message for unsuccessful login Also opens device browser, goes to perfecto sample page, logs in and validates successful login
• #19: First – show basic test in Eclipse Project Named TrainingProjectExample RemoteWebDriverTest.java – run as java test Then show Lufthansa video with callouts
• #20: Additional references: Measure the time it takes for an application to open, using the new Timeout parameter in the Application start command. https://community.perfectomobile.com/posts/1008846 Measure application responsiveness with user experience timing. https://community.perfectomobile.com/series/23468