How to Create a Risk Based Testing Strategy With Simulators, Emulators, and Real Devices

Perfecto by Perforce © 2020 Perforce Software, Inc.
How to Create a Risk-Based Testing Strategy
With Simulators, Emulators, & Real Devices

2 | Advanced Codeless Testing for Web Apps
Today’s Speakers:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268
• https://www.linkedin.com/in/eran-kinsbruner-4b47a81/
Tzvika Shahaf • VP Product Management at Perfecto
• Software professional with over 12 years in product
management, development, and testing
• Expert in implementing and advising on test strategies for
Fortune 500 enterprises and leading tech companies
• http://linkedin.com/in/tzvikashahaf/
• @Tzvika_Shahaf

perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
Test automation coverage considerations
Definitions of emulators and simulators
3 Differences between iOS/Android virtual platforms
6 Q&A
5 Live demo: testing in parallel and at scale on virtual platforms
https://ter.li/y7jfmv
4 Strategy for testing on both real and virtual platforms

Balancing Test Automation Coverage
TEST SCENARIO ONE
Test Case – 1 Test Case – 2 Test Case – 3 Test Case – N
2 sides of the same coin

Digital Platform Coverage Strategy
Android OS Market Share
Android 6.x (Marshmallow) Android 7.x (Nougat) Android 8.x (Oreo)
Android 9 (Pie) Android 10 (Q)

Real Devices Run Different Custom OS Versions
Device Model Custom OS Implementation Versions Base Stock OS
Samsung Galaxy S20 Ultra One UI 2 Android 10
Samsung Galaxy S7 TouchWiz UI Android 8.0
Samsung S8 One UI Android 9.0
Samsung Note 9 One UI 2 Android 10.0
Google Pixel 4 XL STOCK Android 10
Huawei Mate 20 Pro EMUI 9.1 Android 9.0
LG G8 ThinQ LG UX 9.0 Android 10.0
Oppo Reno 5G Color OS 6 Android 10
Huawei Y9 Prime Magic UI 2.1 Android 10
Huawei Enjoy 20 Pro EMUI 10.1 Android 10

The Agile Testing Manifesto
T H E D E V O P S M A N I F E S T O
How to determine test value?
 Continuous testing over testing at the end.
 Embracing all testing activities over only automated functional testing.
 Testing what gives value over testing everything.
 Testing across the team over testing in siloed testing departments.
 Product coverage over code coverage.

Virtual Platform Definitions
• Simulators
• iOS (Apple)
• A simulator is software that imitates the workings of
software on your system that was made for mobile.
• Mobile simulators can be used in app testing and cross-
browser or responsive testing to simulate a mobile device
and running your app or website in it.
• Emulators
• Android (Google)
• An emulator is software that imitates the working of
software on your system that was made for mobile.
• Emulators also take into consideration the hardware part
of the mobile device (memory, processor, battery
performance, etc.).
Real Device Behavior

Differences Between Android & iOS Virtual Platforms

Real Device Issues — Logs, Analysis, and Reporting Can Help
It is not working on my Samsung s10
BAD UPDATE !!! Now the APP does not open on my Xiaomi Mi A1
Source: App Annie

Real Device Issues: COVID-19 PathCheck App

Working Together: Test Data is Generated by Multiple Personas
D E V Q A / S D E Ts N F T/ O t h e r

US Mobile Carrier Performance, Q1 2020 Carriers Vary
— Test Against Edge Performances
Source: OpenSignal

How to Keep Your Test Coverage In Sync With Your End Users

Perfecto’s Mobile Testing Pyramid Combo
Testing Pyramid Within Your DevOps Pipeline Requirements
U N I T T E S T S
M A N UA L
T E S T S
Testing isolated units of code.
I N T EG R AT I O N T E S T S
Testing across integrations of more than one system.
U I T E S T S
Testing through a user interface.
Per code commit/ Post commit
Build Acceptance/Regression Testing
E2E Testing
Exploratory/Manual Testing
Dev
Triggered
Mix Real Devices & Emu./Simu.
CI
Triggered+
Real Devices Only
CI
Triggered
Real Devices OnlyProd. Monitoring
Non Functional Testing:
Performance (UX)
Testing/Security
Real Devices Only

Leverage Tagging for Test Impact Analysis (TIA) and Fast Debugging
Tags are key enablers for slicing
and dicing test data, reducing
MTTR (Mean Time To Resolution),
and making data-driven decisions.

Considerations
for Using iOS and
Android Virtual
Platforms vs. Real
Devices
Stage and activity in
the software
development cycle
Dev
Test
Debug
Design
Cost avoidance
Coverage
Platforms
Real user
conditions
Test types
Functional/Regression
/Integration Testing
Unit
Non Functional
Production Monitoring

Requirements for Perfecto’s Mobile App Testing
Pyramid Combo Within Your DevOps Pipeline
Early SDLC stages fit the use of
virtual platforms
Easy setup
Fast feedback
Cost effective
Unit testing types
Build acceptance and
integration testing should
involve a mix
Advanced scenarios across iOS/Android
are limited on virtual platforms (no app
store, diff HW)
Real user experience required to guide
next software development phase
Such phase requires a combination of
environments (real & virtual)
UI and UAT phases fit the use
of only real devices
Real UX feedback
Greater mix of platform coverage against
official OS versions (Index)
Risk-free release gateway decision
Please Refer to Perfecto’s Test Coverage Index

Coverage Index Across Geographies

Pipeline Testing Coverage Example
Virtual Platforms
Mix of
Virtual Platforms & Real Devices
Only Real Devices

LIVE DEMO – Virtual Platform Testing With Perfecto

Thank You!
Visit perfecto.io or follow us @perfectomobile

How to Create a Risk Based Testing Strategy With Simulators, Emulators, and Real Devices

More Related Content

What's hot (20)

Similar to How to Create a Risk Based Testing Strategy With Simulators, Emulators, and Real Devices (20)

More from Perfecto by Perforce (17)

Recently uploaded (20)

How to Create a Risk Based Testing Strategy With Simulators, Emulators, and Real Devices