Mobile Applications Testing: From Concepts to Practice

rial

Presented by:
Max Saperstone

rought to you by:

340 Corporate Way, Suite Orange Park, FL 32073
888‐2
MB
AM Tuto
4/7/2014
8:30 AM

“Mobile Applications Testing:
From Concepts to Practice”

Coveros, Inc.

B

300,
68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

or almost a decade, Max Saperstone has been a test engineer focusing on test automation
ng

Max Saperstone
Coveros, Inc.

F
and the continuous integration/continuous delivery process. Max specializes in open source
tools—Selenium, JMeter, AutoIT, Cucumber, and Chef. He has led several testing automation
efforts, including developing an automated suite focused on web-based software to operate
over several applications. Max also headed a major project developing an automated testing
structure to run Cucumber tests over multiple test interfaces and environments, while
developing a system to keep test data “ageless.” He is currently developing a new testi
architecture for SecureCI to allow testing of multiple interfaces, custom reporting, and
minimal test upkeep.

Mobile Application Testing Tutorial
Student Handbook
February 2014

Copyright 2014 Coveros, Inc. All rights reserved.
Copyright © 2014 by Coveros, Inc. All rights reserved. No part or parts of this Coveros, Inc.
documentation may be reproduced, translated, stored in any electronic retrieval system,
transmitted in any form or by any means, electronic, mechanical, photocopying, recording or
otherwise, without prior written permission of the copyright owner.
Coveros, Inc.
4000 Legato Rd
Suite 1100
Fairfax, VA 22033
Phone: 703-349-6109
www.coveros.com

Mobile Application Testing Tutorial Welcome
Copyright 2014 Coveros, Inc. All rights reserved. Page 1
Welcome
Slide 1
Welcome to the Mobile Testing for Test Professionals training course. In this course, you will learn all
about how to apply Mobile Testing to your Software Testing efforts.
1© Copyright 2013 Coveros, Inc.. All rights reserved.
Mobile Application Testing
Tutorial

Mobile Application Testing Tutorial Course Trainers
Course Trainers
Slide 2
Slide 3
Trainer
Jeffery Payne
Jeffery Payne is CEO and founder of Coveros, Inc., a software company that
helps organizations accelerate the delivery of secure, reliable software. Coveros
uses agile development methods and a proven software assurance framework to
build security and quality into software from the ground up. Prior to founding
Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc.
Under his direction, Cigital became a leader in software security and software
quality solutions, helping clients mitigate the risk of software failure. Jeffery is a
recognized software expert and popular speaker at both business and technology
conferences on a variety of software quality, security, and agile development
topics. He has also testified before Congress on issues of national importance,
including intellectual property rights, cyber-terrorism, Software research funding,
and software quality.
Trainer
Alan Crouch
Alan Crouch is a Senior Software Security Specialist with Coveros, Inc. a Virginia-
based firm focusing on agile, software quality, and application security. Alan has
worked closely with federal agencies and private companies to advise, audit, and
support IT security and governance teams. In addition to his cybersecurity
experience, he has a strong background in software engineering, test analysis,
test automation, and security testing. With a passion for software and security,
Alan has focused his career on building secure software and developing better
software security practices.

Course Trainers
Slide 4
Slide 5
Trainer
Max Saperstone
Max Saperstone has been working as a Software and Test Engineer for almost a
decade, with a focus on Test Automation and the CI/CD process. He specializes
in open source tools, including the Selenium Tool Suite, JMeter, AutoIT,
Cucumber, and Chef. Max has lead several testing automation efforts, including
developing an automated suite focusing on web-based software to operate over
several applications for Kronos Federal. He also headed a project with Delta
Dental, developing an automated testing structure to run Cucumber tests over
multiple test interfaces and environments, while also developing a system to keep
test data 'ageless.' He is currently developing a new testing architecture for
SecureCI™ to allow testing of multiple interfaces, custom reporting, and minimal
test upkeep.
Trainer
Richard Mills
Richard Mills has more than 20 years of experience in software engineering
specializing in software development with a concentration on pragmatic software
process and tools. Rich has a specific focus in Agile development methods, static
and dynamic software analysis tools, configuration management, engineering,
automated software build management, and continuous integration. He currently
works as a Senior consultant with Coveros, Inc. Helping his customers build
software better, faster and more securely. As a consultant, Rich helps customers
improve software delivery by coaching and mentoring in the Agile development
methodologies, automating software build, test, deployment and employing
secure development techniques. Prior to joining Coveros, Rich spent 16 years
with Cigital, Inc. fulfilling a wide range of roles including consulting, training, and
software engineering.

Course Trainers
Slide 6
Trainer
Jeffrey Pierce
Jeff Pierce is a Managing Consultant with Coveros, Inc., a software company that
helps organizations accelerate the delivery of secure, reliable software. Coveros
uses agile development methods and a proven software assurance framework to
build security and quality into software from the ground up. Jeff is a leader with
over twenty years experience in information technology (IT) program/project
management and client management in all phases of business investment and
system development lifecycle. He has proven technical excellence in
development and deployment of strategic web-based enterprise information
systems utilizing cloud computing and continuous integration technologies. Jeff
also has experience with strategy and roadmap development, policy and
procedure development, program management office implementation, IT and
business management, AGILE SCRUM and XP management, systems
engineering, technical team building and leading large and small on-shore and off-
shore full-time and contract staff with an outstanding record of accomplishment.

Mobile Application Testing Tutorial About Coveros
About Coveros
Slide 7
Coveros is a consulting company that helps organizations build better software. We provide software
development, application security, QA/testing, and software process improvement services. Coveros
focuses on organizations that must build and deploy software within the constraints of significant
regulatory or compliance requirements. The primary markets we serve include: DoD, Homeland Security
& associated critical infrastructure companies, Healthcare providers, and Financial services institutions
 Coveros helps organizations accelerate the delivery of secure, reliable
software
 Our consulting services:
– Agile software development
– Mobile application development
– Application security
– Software quality assurance
– Software process improvement
 Our key markets:
– Financial services
– Healthcare
– Defense
– Critical Infrastructure
About Coveros
Corporate Partners

Mobile Application Testing Tutorial Agenda
Agenda
Slide 8
Agenda
 Introduction to Mobile Testing
 Mobile Web Applications
 Native Mobile Applications
 Unique Mobile Testing Considerations
 Mobile Testing Platforms
 Mobile Testing Techniques
 Tools for Mobile Testing
 Wrap-Up

Mobile Application Testing Tutorial Introduction to Mobile Testing
Introduction to Mobile Testing
Slide 9
Slide 10
Introduction to Mobile Testing
Agenda
 What is Mobile Application Testing?
 Why is Mobile Application Testing Important?
 What Do You Need to Know to Be a Good Mobile Tester?
 Types of Mobile Applications
 Mobile Application SDLC

Mobile Application Testing Tutorial Definition of Mobile Application Testing
Definition of Mobile Application Testing
Slide 11
Companies everywhere are rapidly building and deploying mobile applications. Testing these applications
must be accurate, reliable, usable and rapid – keeping up with the pace of development and changing
mobile platforms. Additionally MAT will look at testing areas that have typically only been tested by
hardware engineers and will need to testing focus in order for mobile applications to be built, delivered,
and executed.
Title
Month Year
Definition of Mobile Application Testing
 Mobile Application Testing is a process by which
application software developed for mobile devices is tested
for functionality, security, usability and consistency.
 Although Mobile Application Testing is a relatively new
aspect of software testing, all existing standard testing
practices and techniques can be applied.
What is Mobile Application Testing?

Mobile Application Testing Tutorial Mobile Application Testing Importance
Mobile Application Testing Importance
Slide 12
App stores can reduce the time in the feedback loop, and customers can be utilized for early and
frequently feedback. Without it, you may be developing an application your customers don’t want
As with all software, code quality verification is important. Users are quicker than ever to abandon an
application that doesn’t work perfectly the first and every time.
Users have high expectations and are App stores also make it extremely easy to get software out to the
public at a rapid pace. You want to ensure your company is putting their best foot forward to maintain a
good reputation
Having a consistent, predictable release process will ensure good software is released every time
The extensive choice of browsers, operating systems, devices and applications make it possible for
enterprises to develop new, interactive and highly personal ways for customers and employees to access
products, services and applications. Mobile devices are at the same time sources of rich information,
powerful entertainment and purchasing platforms, and important work tools.
Companies that successfully implement mobile strategies engage both customers and employees in new
ways, and create an environment of personal interaction that enhances sales and productivity. What
makes mobile so attractive to consumers—instant access to a multitude of apps; a wide range of rapidly
evolving devices and functions; a choice of operating systems; new, cloud-based services that require no
storage on the device—also makes it tricky for companies to deploy effectively.
The fourth annual World Quality Report—from Cap Gemini, Sogeti and HP—reveals that only 31% of
organizations surveyed currently have formal processes in place to test their mobile applications, while
just 18% of those have a focus on security
 App stores reduce the user feedback loop.
 User feedback is both early and frequent.
 Usability and quality is king, when users are quicker than
ever to abandon an application that doesn’t work the first
time.
 Users can get updates rapidly and expect seamless
updates and integration.
 Mobile devices are changing quickly and constantly getting
new capabilities and features.
Why Is Mobile Application Testing Important?

Mobile Application Testing Tutorial Mobile App Testing Difference
Mobile App Testing Difference
What Do You Need To Be a Good Mobile Tester?
 Mobile Application Testing Tools aren’t Mature or Available
– Many tools are created for specific device/OS combinations
– Cross-platform tool support is limited
– No one tool will do
 Mobile Application Testing Requires Specialized Skills
– UI and usability focused
– Strong testing background
– Understand networking
– Engineering/hardware focus
– Constantly learning
 Cloud Components
 Cost of Entry
 Automation Immaturity
– Automation image based and time consuming
– Automation difficult to leverage outside the framework
What Makes Mobile App Testing Different?
What Do You Need To Be a Good Mobile Tester?
 History of Mobile Devices and Emerging Trends
 Basic Software Testing Skills
 Basic Understanding of the Mobile Software Lifecycle
 Basic Understanding of Telecommunications
 Awareness of Mobile Operating Systems/Platforms
– Experience in virtualization for Linux, or VMWare, and data center
operations
 Experience with cloud infrastructure and SaaS
 Knowledge of Mobile testing tools and frameworks
 Knowledge and experience with test automation tools

Mobile Application Testing Tutorial Understand the Basics of Software Testing
Understand the Basics of Software Testing
Slide 15
This course is not meant to teach you the basics of software testing, it assumes you know them and can
apply them.
Understand the Basics of Software Testing
 Knowledge of Software Testing basics is essential to testing
any platform. Even if you just starting your career, you need
to know:
– Software Testing Goals and Objectives
– Software Testing Principles
– Software Testing processes, methodologies, and techniques
 If you don’t have a strong foundation in software testing, we
suggest you start simple and enroll in some basic training
programs.
What You Need to Know

Mobile Application Testing Tutorial Understand the Basics of Telecommunications
Understand the Basics of Telecommunications
Slide 16
Understanding the basics of Telecommunications will always give you advantages when doing testing
mobile applications.
Title
Month Year
Understand the Basics of Telecommunications
 Mobile App Testing involves networks, not just the app or
device itself.
 Can you identify and understand the differences in these
technologies?
– 2G, 3G, 4G, Edge, LTE, CDMA, GPRS, GSM, HSCSD, SIM, SMS,
WAP, TCP, UDP, Wi-Fi
 Telecom is global
– …therefore, so is your app.

Mobile Application Testing Tutorial Mobile Operating Systems/Platforms
Mobile Operating Systems/Platforms
Slide 17
Slide 18
Awareness of Mobile Operating Systems/Platforms
 OS Design
– Integration Points
– Device Capability Usage
– Security
Features/Vulnerabilities
 Device Operation
– Standalone
– Offline
– Wi-Fi
– Interruptions
Awareness of Mobile Operating Systems/Platforms
 Device Design
– Variable Screen Size/Resolution
– Device specific Browser Limitations/Behaviors
– Variable input/navigation methodologies
 Touch Screen
 Keyboard/virtual keyboard
 Ball/Pointer
 Voice
 Menu Structures
– CPU Constraints/Variability
– Memory Constraints/Variability
– Power utilization constraints
– Keypad lock

Mobile Application Testing Tutorial Emerging Trends
Emerging Trends
Slide 19
It’s important to stay on top of emerging trends!
Staying on top of emerging trends will better help you
understand what you may need to test for when your
business wants to add or integrate a new piece of
technology into your mobile application. In addition, it
will make your products more competitive in the
marketplace.
Emerging Trends

Mobile Application Testing Tutorial Types of Mobile Applications
Types of Mobile Applications
Slide 20
Slide 21
 Mobile Web Apps
– Non-Responsive
– Responsive
 Native Apps
– Device-Specific
– Responsive
– Hybrid Apps
Mobile Web Applications
 Web apps are really websites
that, in many ways, look and feel
like native applications. They are
run by a browser and typically
written in HTML5.
 Users access them as they
would access any web page:
they navigate to a special URL.

Mobile Application Testing Tutorial Native Applications
Native Applications
Hybrid Applications
Native Applications
 Native apps live on the device and are accessed through
icons on the device home screen.
 Native apps are installed through an application store and
are developed specifically for one platform.
 They can take full advantage of all the device capabilities
and can work offline.
Hybrid Applications
 Hybrid apps are application that
combines elements of both native
and web applications.
 Hybrid apps are installed through
an application store but integrate
with a large amount of content
from Web-based services.
 They can often function without an
Internet connection, but may have
limited capabilities when
disconnected
 The majority of apps fall into this
category.

Mobile Application Testing Tutorial Mobile Application SDLC
Mobile Application SDLC
Slide 24 - [Source: Harty, Julian. “Mobile Apps”]
Title
Slide 25
The Waterfall Mobile Application SDLC
Requirements
Implementation
•Design
•Code
Verification
•Unit Tests
•Internal Installation
•System Tests
Launch/Release
•Pre-Publication
•Publication
Engagement
•Search
•Trust
•Download
•Installation
•Upgrade
•Downgrade
Validation
•Payment
•Use
•Feedback
The Agile Mobile Application SDLC
Launch
Releases1-n
Sprint 1Sprint 0 Sprint 2
End Game Engagement
Requirements
Design
Implementation
Testing
Launch/Release
Engagement/Validation

Mobile Application Testing Tutorial Mobile Web Applications
Agenda
Agenda
– Overview
– Types of Mobile Web Apps
 Non-Responsive Applications
 Responsive Applications
– Testing
– Technologies

Mobile Application Testing Tutorial What is a Mobile Web Application?
What is a Mobile Web Application?
Exercise
Overview
What is a Mobile Web Application?
 A mobile web application is an application that is accessed
by users over a network such as the Internet or an intranet
on a mobile device.
 Mobile Web applications are generalized for multiple
platforms and not installed locally but made available over
the Internet through a browser.
 e.g. Online Banking Services
 Mobile Web Apps can be written with HTML, CSS and
JavaScript
 Check your HTML 5 capabilities
 With your Mobile Device, go to:
– http://webdirections.org/deviceTest/
Exercise

Mobile Application Testing Tutorial Advantages
Advantages
Non-Responsive Apps
Overview
Advantages
 You can use simple HTML, CSS, JS for most mobile
applications
 You can maintain and fix bugs in real time.
 Quick upgrade and migration
Disadvantages
 You can’t sell your application
 Must be responsive to multiple resolutions, operating
systems and browsers
 Struggle with performance and testability
Types of Mobile Web Apps
Non-Responsive Apps
 A non-responsive web application
that responds the same way on a
laptop’s web browser as it would on
a mobile device.
 More and more website’s are
moving away from non-responsive
apps, but there are many reasons
why someone would choose to
have a non-responsive app.

Mobile Application Testing Tutorial Responsive Apps
Responsive Apps
Responsive Web Apps - Example
Responsive Apps
 Responsive Web design is an approach that suggests that
an app should respond to the user’s behavior and
environment based on screen size, platform and
orientation.
 The practice often consists of flexible screens, grids, and
layouts, images and an intelligent use of CSS media
queries, automatically switching to accommodate for
resolution, image size and scripting abilities.
1920x1080 Screen Resolution – Windows 7
www.coveros.com

Mobile Application Testing Tutorial Responsive Web Apps - Example
1920x1080 Screen Resolution – Windows 7
www.coveros.com
1136 x 640 Screen Resolution – iPhone5
www.coveros.com

Mobile Application Testing Tutorial Testing a Mobile Web App
Testing a Mobile Web App
Slide 36
Testing
Testing a Mobile Web App
 Testing a mobile web app is fairly similar to how you would
test any other web application. In fact, you can often use
the same tools.
 Using emulators and user-agents you can often replicate
resolutions and device types on your laptop.
 However, there are some things you won’t be able to
replicate (gestures, shakes, etc.), so don’t think that you
can avoid testing on an actual mobile device.
 For responsive web apps, you can often reuse tests suites
for various resolutions, but no good test plan for a
responsive app will have one set of identical tests for each
resolution.

Mobile Application Testing Tutorial
This Page Intentionally left blank

Mobile Application Testing Tutorial Native Mobile Applications
Native Mobile Applications
Agenda
Native Mobile Applications
Agenda
 Overview
 Types of Native Apps
– Device Specific Applications
– Responsive Applications
– Hybrid Applications
 Testing
 Technologies
– OS API
– APP Store
– Device Capabilities

Mobile Application Testing Tutorial What is a Native Application?
What is a Native Application?
Device-Specific Applications
Overview
What is a Native Application?
 Native applications are developed for a specific platform
and installed on a mobile device.
 Native applications can be pre-installed or installed through
an app store on demand.
 e.g. Phone app vs. Instagram.
 Native applications are built in a language for a specific
mobile device
Types of Native Apps
Device-Specific Applications
 Device-Specific Applications are native apps built for not
only one platform (e.g. iOS) but also a device (e.g. iPhone).
Responsive Applications
 Responsive Applications are platform-specific native apps
that are built to respond to various screen size, resolutions
and other built-in device capabilities based on the mobile
device the application is installed upon.

Mobile Application Testing Tutorial Hybrid Applications
Hybrid Applications
Testing a Native App
Types of Native Apps
Hybrid Applications
 By a strict definition, Hybrid apps are technically native
apps. However, they are part native apps, part web apps;
they live in an app store and can take advantage of the
many device features available but rely on HTML being
rendered in a browser embedded within the app itself.
 Hybrid apps allow an organization to have consistency
between different device platforms.
 They can also provide presence in the app store without
having to build much of a native app.
 They can provide some cross-platform development,
because HTML components can be reused to reduce costs.
Testing
Testing a Native App
 Testing a native app is a mix of software and hardware
testing. A tester has to test the app in all the ways a user
may use the app (not just standard user behavior).
 Using emulators you can often replicate resolutions,
gestures and device platforms on your laptop.
 However, there are some things you won’t be able to
replicate, so don’t think that you can avoid testing on an
actual mobile device.
 Since each of the selection points are determined by the
developer, test that there is enough space to perform
actions and that you are properly applying OS Gestures.

Mobile Application Testing Tutorial Testing a Hybrid App
Testing a Hybrid App
Slide 43
Testing
Testing a Hybrid App
 Arguably, testing a hybrid app is the most complicated to
plan and strategize. A good tester must know how to test
both native app and mobile web app components to test a
hybrid app successfully.
 You can test many components like you would a mobile
web app (even the same tools).
 Testing the components built with the native code, should
be performed just like a typical native app.
 However, don’t forget about your integration points!
 There are native mobile browser functionality that can have
features that can cause unintended behavior in the
embedded browser.

Mobile Application Testing Tutorial Unique Mobile Testing Considerations
Unique Mobile Testing Considerations
Agenda
Agenda
 User Expectations
 Network
 Data Storage
 Operating System
 Jail Breaking
 Hardware/Device Capabilities
 Security

Mobile Application Testing Tutorial User Expectations Challenges
User Expectations Challenges
Network Challenges
User Expectations Challenges
 Global Users
– Global User Community
 Internationalization/Localization requirements
 Instant Feedback (online ratings)
 Expectations
– Intuitive UI
– Performance – 60% of mobile app users will abandon app or site if
load time > 3 seconds
– Seamless across multiple devices
– Seamless across various connectivity options (3g, 4g, Wi-Fi)
– Security
– Data utilization
– Power utilization
– Easy Installation and Launch
Testing Mobile Applications
Network Challenges
 Networks
– More than 400 mobile networks globally
 Numerous Global Standards
– NMT (Nordic Mobile Telephony)
 FDMA (Frequency Division Multiple Access)
– GSM (Global System for Mobile Communications)
 TDMA (Time Division Multiple Access)
 FDMA (Frequency Division Multiple Access)
– UMTS (3GSM) (Universal Mobile Telecommunications System)
 W-CDMA (Wideband Code-Division Multiple Access)
– IS-95 (CDMA one) / IS-2000 (CDMA 2000)
 CDMA (Code-Division Multiple Access)
– WIMAX / LTE (Long Term Evolution)
 OFDMA (Orthogonal Frequency Division Multiple Access)

Mobile Application Testing Tutorial Network Challenges
Network Challenges
Network Challenges
 Multiple Carriers
– Verizon, AT&T, Sprint, T-Mobile, Clearwire, US Cellular, Leap
Wireless, nTelos, etc.
– Not all 4G Networks are created equal. 4G for Verizon is not the
same as 4G for AT&T
 Network Hopping
– Cell to Cell Hopping
– Roaming
– Cell to Wi-F
– 3G to 4G, Edge to 3G, etc.
– International Networks
 Near Field Communication
 Bluetooth
Network Challenges
 Implementation variability and issues
– Tunneling
– Web Proxies
 limiting site connectivity
 Transcoding
 http header information filtering
 Several Bandwidth/Connection Modes
– 1G, 2G, 3G, 4G
– Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)

Mobile Application Testing Tutorial Data Storage Challenges
Data Storage Challenges
Operating System Challenges
 Small Capacity
 Data Privacy / Shared Storage
 Data Encryption (or lack there of)
 Cloud Data Storage
 Shared Data Resources
– Photos
– Calendar
– Video
– Mail
– Messages
– Notifications
Data Storage Challenges
 App Store Integration
 OS API
 Fragmentation
 Backwards Compatibility
 No Industry-wide standards
 Capability, Capacity and
Performance

Mobile Application Testing Tutorial Operating System Challenges
 Interruptions are events that would interrupt normal
operations
 There are two types of Interruptions
– Application Interruptions:
 Installation, Launch, Run, Exit, Upgrade, Uninstall/Delete
– Device/OS Interruptions
 Incoming Call
 Receiving Incoming Call
 Receiving Message
 Device Shutdown
 Device Restore
 Remove Battery
 Camera Activated
 Loss of Connectivity
 Regain Connectivity

Mobile Application Testing Tutorial About Jailbreaking
About Jailbreaking
Jail Breaking Challenges
About Jailbreaking
 Jailbreaking (iOS) or Rooting (Android) is the process of
removing the limitations on mobile devices through the user
of software or hardware exploits.
 Jailbreaking permits root access to the operating system,
allowing download of non-approved apps, extensions and
themes not available otherwise.
 Large populations of users utilize Jailbroken/rooted mobile
devices.
 Corporate policy may permit devices from being Jailbroken
Jail Breaking Challenges
 Providing escalated privileges by overcoming limitations in
a computer system or device by deliberating overriding
security, administrative and marketing restrictions by the
creator
 Non-Approved Applications
 Malware, Worms
– First iOS Worm, named iKree, was released to affect Jailbroken
iPhones in 2009

Mobile Application Testing Tutorial Hardware Challenges
Hardware Challenges
Slide 55
Slide 56
Hardware Challenges
 Screen Resolution
– Size
– Clarity
– Portrait vs. Landscape
 Bluetooth / NFC
 Camera
 Processing Power
 Speaker/Microphone
 Flashlight
 Accelerometer
 Magnetometer
Hardware Challenges
 Touch Screen
– Single Touch Inputs
– Multi-Touch Inputs
– Virtual Keyboard
– Multiple Inputs Simultaneously
 Physical Inputs
– Keypads
– Track Ball
 GPS / Location Services
 Power Consumption
 Memory Usage
– Launch, Background, Foreground, Exit
– Simultaneous Applications
– Memory Sweeping

Mobile Application Testing Tutorial Security Challenges
Security Challenges
Slide 57
Slide 58
Security Challenges
 Encryption
– Data at Rest
 On Device
 On Middle-Tier Servers
 On Databases
 Cached Content
– Data in Transit
 Privacy
– Application Privileges
 Features
 Data
 Capabilities
– 3rd Party Application Integration
 User Configurable Trust Settings
Security Challenges
 Operating System
– Malware
– Critical Bugs
– Out of Date (Upgrading)
– Support ends after two years typically
 Network
– No Firewalls
– Few devices limit Internet connections
– Unsecure Communication Channels
 Bluetooth Discovery Mode
 Unsecure Networks
 Unauthorized Modification, Jail Breaking, Rooting

Mobile Application Testing Tutorial Mobile Testing Platforms
Mobile Testing Platforms
Agenda
Mobile Testing Platforms
Agenda
 Introduction
 Desktop Browsers
 Mobile Emulation
 Cloud Testing
 Physical Device Testing
 Remote-Device Testing
 Recommendations

Mobile Application Testing Tutorial Choosing the Right Platform
Choosing the Right Platform
What is possible with Desktop Browsers?
Introduction
Choosing the Right Platform
 There are many different device platforms, that are often
fragmented and grow at different paces
 iOS, Android, Blackberry, Windows phone, Symbian,
etc.
 Multiple form factors and screen resolutions
 Users get new devices every 24 months
 There are often carrier restrictions on devices (beyond
network considerations)
Desktop Browsers
What is possible with Desktop Browsers?
 Desktop browsers can be extremely useful when
developing and testing mobile web apps.
 Desktop browsers like Chrome and Firefox have many tools
to effectively use user-agents to simulate multiple mobile
browser types with little or no cost.
 Leveraging desktop browsers does not replace device
testing but they can provide robust debugging options and
can be used to simulate multiple screen sizes.

Mobile Application Testing Tutorial User-Agent Switcher for Chrome
User-Agent Switcher for Chrome
Slide 63
https://chrome.google.com/webstore/detail/user-agent-switcher-for-
c/djflhoibgkdhkhhcedjiklpkjnoahfmg/reviews?hl=en-US
You can also add user-agent strings as necessary (highly recommended)
Title
Month Year
Desktop Browsers
User-Agent Switcher for Chrome
Example: iPhone Agent on Google
Chrome in Incognito Mode

Mobile Application Testing Tutorial What is Mobile Emulation?
What is Mobile Emulation?
iPhone Emulator
Mobile Emulation
What is Mobile Emulation?
 Emulators offer the less expensive testing option, but they
have many limitations for mobile testing and may not give
an accurate depiction of the mobile user experience.
 However, as part of the mobile testing process the use of
emulation has an important role. There is certain testing that
you can conduct with emulators that provides value. For
example, testing for screen size or visual form factor.
 However, emulation is often an approximation of the
rendering and can only get you so far.
 Leveraging emulators is a positive step towards real device
testing, however, not sufficient for comprehensive testing
needs.
Emulator
iPhone Emulator

Mobile Application Testing Tutorial iPad Emulator
iPad Emulator
Android Emulator
Emulator
iPad Emulator
Emulator
Android Emulator

Mobile Application Testing Tutorial What is Cloud Testing?
What is Cloud Testing?
Sauce Labs
Cloud Testing
What is Cloud Testing?
 Cloud Testing platforms allow testers and developers to run
their applications on cloud-based emulators and simulators
in virtual environments.
 This eliminates the need to maintain emulator, simulators
for all your testing/development teams.
 You can easily test on all types of OS versions and device
models.
 Some cloud-based tools allow mobile teams to receive real-
time alerts against any measurement criteria, allowing them
to address issues before they impact end-users
 There are often APIs that allow automated testing through
these platforms.
Cloud Testing
Sauce Labs

Mobile Application Testing Tutorial What is Device Testing?
What is Device Testing?
Crowdsourcing
Physical Device Testing
What is Device Testing?
 Testing on a real, physical device with full functionality to the
tester.
 Physical device testing will always provide the most
accurate testing results, but is also the most costly.
 Without access to the real processor and hardware to
understand the quirks that some real device have, testers
may not discover some real life issues that do not present
themselves in emulators
 Crowdsourcing is the practice of obtaining testing services
by soliciting contributions from a large group of people, and
especially from an online community, rather than from
traditional employees, outsourcing or suppliers.
 It combines the efforts of numerous self-identified
volunteers or part-time workers, where each tester of their
own initiative adds a small portion to the greater testing
efforts.
 Crowdsourcing is different from outsourcing in that the work
comes from an undefined public rather than being
commissioned from a specific, named group.
Crowdsourcing

Mobile Application Testing Tutorial What are Remote Device Access (RDA) Service?
What are Remote Device Access (RDA) Service?
Which Platform is Best for My Mobile Testing?
Remote Device Access
What are Remote Device Access (RDA) Service?
 RDA services enable access to a live device over the
Internet. As a Mobile Apps Tester, you should be aware of
such services and should suggest your managers about the
capabilities of such services.
 Due to the large number of devices available in the market,
it is not feasible to buy a new device every time. That
makes RDA a much more cost effective solution.
Recommendations
Which Platform is Best for My Mobile Testing?
Type of Testing Browser Emulator Device RDA Cloud Emulator
Debugging x x x
Feature Testing x x x
Usability Testing x x x
Exploratory
Testing
x x x x x
Performance
Testing
x x x x x
Automated Testing x x x
UAT x x
Security Testing x x x x x
Pre-Prod Testing x x

Mobile Application Testing Tutorial Mobile Testing Techniques
Mobile Testing Techniques
Agenda
Agenda
– Exploratory Testing
– Device Access
– Security Testing
– Performance Testing
– Compatibility Testing
– Submission Guideline Testing

Mobile Application Testing Tutorial Mobile Testing Techniques
Exploratory Testing
Test
Management
Exploratory
Testing
Service/API
Testing
Device Access
Security Testing
Back-End
Testing
Mobile UI
Automation
Performance
Testing
Submission
Guidelines
Testing
Compatibility
Testing
Exploratory Testing
 Exploratory Testing is still the most widely used mobile
testing technique
 Necessary for some validation
 Frequent key in outsourcing and crowdsourcing scenarios
 Advantages
– Tester can assess performance of application under actual
operating conditions
 Determination/Evaluation of Performance
 Determination/Evaluation of Ease of Use
 Identification of unexpected behaviors and errors
 Does it feel right?
– Test-As-You-Fly
 A Goddard Space Flight Center Technical Standard which essentially means
that you test the system under test as it is intended to be used.

Mobile Application Testing Tutorial Standard Mobile App States
Standard Mobile App States
Interruption Testing
 App Store
– Upload
– Download
– Update Broadcasting
– Monetization scenarios
 Installation and Removal
– Multiple Modes
– Verification of App Logo and Splash
 Operation
– Start/Stop
– Pause/resume
– Credentials
– Auto-Start
 Operational Modes
– Standalone
– Offline
– Wi-Fi
– Cabled
– Airplane Mode
Standard Mobile App States
Exploratory Testing
 Incoming Call
 SMS Received
 Voicemail
 Email
 Social media notifications
 Alarms
 Calendar reminders
 Battery low
 Battery Critical
 Power Sleep Cycle
 Battery Removed
 Battery Charging
 Low Memory
 Device Switched Off
 Device Reboot
 Network outage and Recovery
 Bluetooth Connection request
 Phone/System Updates
 Data Transfer Cable Insertion
 Camera Activation
 Home Button
 Search Button
 Volume Control Changes
 Screen Lock
Interruption Testing
Exploratory Testing

Mobile Application Testing Tutorial User Interface Testing
User Interface Testing
User Inputs
 Layouts, alignment, colors,
fonts
– Views/Tables
– buttons, Menus, Links,
Pickers, Toolbars
– Alerts/Messages
– Themes, Color Schemes,
Styles
– Brightness Settings
 UI Elements
– Position
– Size
– Inputs
– Actions
 Images and Symbols
– Clarity
– Alignment
– Density (load performance)
 Error Message Displays
 Consistency
– Across the application
– Across various devices
User Interface Testing
Exploratory Testing
 Gestures
– Tap
– Press/Hold
– Double Tap
– Drag
– Pinch
– Spread
– Swipe/Flick
– Rotation
– Shake
– Tilt
User Inputs
Exploratory Testing
http://uxdesign.smashingmagazine.com/2011/10/06/not-your-parents-mobile-phone-ux-design-guidelines-smartphones/

Mobile Application Testing Tutorial Accessibility and Usability Testing
Accessibility and Usability Testing
Slide 82
Slide 83
 Usability
– Efficiency
– Accuracy
– Recall
– Emotional Response
 Accessibility
– Screen Size
– Lighting Conditions and brightness settings
– Multi-screen capability and compatibility
– Touch Interface size
Exploratory Testing
 Workflows
– Information Presentation
– Ease of Task Completion
– Collaborative Task Completion (Sequential and Parallel)
 Application Integration and Interactions
– Native Application/Device
– Web Application/Device
– Hybrid Application/Device/Web App
Exploratory Testing

Mobile Application Testing Tutorial Mobility
Mobility
Connectivity
 Geo aware (GPS, Tower Triangulation)
 Accelerometer (Motion, Gestures)
 Magnetometer (Compass)
 Traversing zones/cells/networks
– GPS Signal fade/interference
– WiFi Signal fade/interference
– Cellular Signal fade/interference
Mobility
Exploratory Testing
 Near Field Communication
– Android Beam
– Bluetooth
– Wi-Fi
 Networks
– Variable Networks Data throughput support
– Variable network connectivity protocol types
 4g/3g/analogue
 1G, 2G, 3G, 4G
 Wi-Fi, Wi-Fi/3G, Wi-Fi/4G)
– Signal Strength
 No Network
 Intermittent connectivity
 Strong Signal
 Varying signal strength
– Trust level settings (Security/Data Exposure)
Connectivity
Exploratory Testing

Mobile Application Testing Tutorial Usability: Deadly Application Failures
Usability: Deadly Application Failures
Testing Device Sensors
Exploratory Testing
Usability: Deadly Application Failures
• Insufficient features and functionality
• Consumption of resources (Battery, CPU, Storage)
• Failure to operate well under sub-optimal connectivity
situations
• Slow Performance
• Interoperability
• doesn’t integrate with native device features
• causes errors in other existing applications
• Difficult to use
 Rotation/Tilt
– Not supported (user expectations problem)
– Could cause screen res/layout issues (switching from Portrait to Landscape mode)
– Other unexpected results
 Movement/Shake Inputs
– Not supported (user expectations problem)
– Unexpected results
 Does jogging on a treadmill cause song over-shuffling on iPod?
 Geo-Location Capabilities
– Verify location mapping correct
– Verify location capabilities while in motion
– Verify lack of freeze-ups while in motion
– Verify re-location upon loss/re-gain of signal
 Magnetometer/Rotation
– Verify Compass direction correct
 Light
– Automatic brightness adjustments
 Heat
– Device function shutdown, reduced performance
Testing Device Sensors
Device Access Testing

Mobile Application Testing Tutorial Security Testing
Security Testing
Performance Testing
Security Testing
 Network Attacks
– Traffic Analyzers
 Server Side Attacks
– API/Service Attacks
– Back-end Server
– Database Attacks
 Client-Side Attacks
– Cross Site Scripting
– Data Privacy Exploitation
– Buffer Overflows
 Injection and Data Flow Analysis
 Jail-breaking/Rooting
 Penetration Testing
Performance Testing
 Mobile apps are often supposed to scale to massive
volumes. We have to delve into lots of different realistic
scenarios for performance over more than just WI-FI.
 Performance testing should cover real world mobile
networks (Not all networks are created equal).
– 4G LTE, 3G, 2G, EDGE, GPRS
 Don’t forget to test middle-tier and back-end for
performance
 If you’re not testing your mobile app out in the real world like
a client would experience, your performance testing is not
complete

Mobile Application Testing Tutorial Performance and Stress
Performance and Stress
Compatibility Testing
 Low resources
– Optimization Modes
– Power saving Modes
– Variable Battery Levels (Full,
Low, Critical, Charging)
– Memory/CPU utilization
 Launch, Background,
Foreground, Exit
 Simultaneous Applications
utilization
 Continuous Keypad entry (across
various keypad controls/events)
 Memory Sweeping
– Storage Space
 Installation
 Application Utilization
– Battery Consumption
 Server Load
– Stress testing
 Multi-user
 Multi-functional
– Backups
 Crowd-Source Testing
 Network Testing
– Internetwork
– Network  Roaming
– Wife  Cell
Performance and Stress
 Testing for Compatibility:-Testing the compatibility of your
application with native device features (i.e. To make sure
your application is not hampering native device
functionality)
 Platform
 Device
 Other Applications
 Application backwards compatibility
 Browsers
 Networks
Compatibility Testing

Mobile Application Testing Tutorial Submission Guideline Testing
Submission Guideline Testing
Slide 92
Just an excerpt from the Apple Guidelines:
2. Functionality
1. Apps that crash will be rejected
2. Apps that exhibit bugs will be rejected
3. Apps that do not perform as advertised by the developer will be rejected
4. Apps that include undocumented or hidden features inconsistent with the description of
the app
5. will be rejected
6. Apps that use non-public APIs will be rejected
7. Apps that read or write data outside its designated container area will be rejected
8. Apps that download code in any way or form will be rejected
9. Apps that install or launch other executable code will be rejected
10. Apps that are "beta", "demo", "trial", or "test" versions will be rejected
11. iPhone apps must also run on iPad without modiﬁcation, at iPhone resolution, and at
2X iPhone
12. 3GS resolution
13. Apps that duplicate apps already in the App Store may be rejected, particularly if there
are many
14. of them
15. Apps that are not very useful or do not provide any lasting entertainment value may be
rejected
16. Apps that are primarily marketing materials or advertisements will be rejected
 Google Play Store
– Quick review
– Simple standards to meet
– http://play.google.com/about/developer-content-policy.html
 Apple App Store
– Notoriously difficult standards
– Long review time
– Stringent Requirements
– https://developer.apple.com/appstore/guidelines.html

Mobile Application Testing Tutorial Submission Guideline Testing
17. Apps that are intended to provide trick or fake functionality that are not clearly marked
as such
18. will be rejected
19. Apps larger than 20MB in size will not download over cellular networks (this is
automatically
20. prohibited by the App Store)
21. Multitasking apps may only use background services for their intended purposes: VoIP,
audio
22. playback, location, task completion, local notiﬁcations, etc.
23. Apps that browse the web must use the iOS WebKit framework and WebKit JavaScript
24. Apps that encourage excessive consumption of alcohol or illegal substances, or
encourage
25. minors to consume alcohol or smoke cigarettes, will be rejected
26. Apps that provide incorrect diagnostic or other inaccurate device data will be rejected
27. Developers "spamming" the App Store with many versions of similar apps will be
removed from
28. the iOS Developer Program
Title
Month Year

Mobile Application Testing Tutorial Android Core Quality Requirements Categories
Android Core Quality Requirements Categories
Apple App Store Requirements Categories
 Functionality
– Permissions
– Installation
– Audio
– UI and Graphics
– User / App State
 Performance and Stability
– Stability
– Performance
– Media
– Visual Quality
 Google Play
– Policies
– App details page
– User Support
Android Core Quality Requirements Categories
 Functionality
 Metadata
 Locations
 Push Notifications
 Game Center
 iAds
 Trademarks and trade dress
 Media Content
 User Interface
 Purchasing and Currencies
 Scraping and Aggregation
 Device Damage
 Personal Attacks
 Violence
 Objectionable Content
 Privacy
 Pornography
 Religion, Culture, and ethnicity
 Contests, sweepstakes,
lotteries, and raffles
 Charities and Contributions
 Legal Requirements
Apple App Store Requirements Categories

Mobile Application Testing Tutorial Tools
Tools
Agenda
Tools
Agenda
 Introduction
 Multi-platform Testing Tools
 Android Testing Tools
 IOS Testing Tools
 Blackberry testing tools
 Windows Mobile Testing Tools
 Remote Device Access Services
 Outsource/Crowd Source Testing Services

Mobile Application Testing Tutorial Multi-platform Test Tools
Multi-platform Test Tools
Multi-Platform Test Tools
Selenium 2 (WebDriver)
 Can run on emulators and devices
 Native support exists for Android, iPhone and iPad but is
deprecated
 Third party drivers have been developed to fill in these gaps
– iOS-Driver
– Selendroid
– Appium
 Support for all major browsers
– Browsers downloaded and installed on devices to supplement
native browsers can be fully tested with WebDriver.
Multi-platform Test Tools
Testing Tools
Gomez, mobiReady, and
W3C mobileOK Checker
 Tests web based apps
and evaluates their
mobile performance.
 Examines mobile best-
practices and usability,
and provides steps to
correct
 Web-based testing, no
download or install
needed
Multi-Platform Test Tools
Testing Tools

Mobile Application Testing Tutorial Android Tools
Android Tools
Android Debugging Tools
 Monkey Runner
– Runs on simulator or device
– Generates pseudo-random streams of user events such as clicks,
touches, or gestures, as well as a number of system-level events.
– Can run an automated start-to-finish test of an Android application.
You provide input values with keystrokes or touch events, and view
the results as screenshots.
 Robotium
– Support for native and hybrid applications
– Can write function, system and acceptance test scenarios, spanning
multiple Android activities
– Integrates smoothly with Maven or Ant to run tests as part of
continuous integration.
Android Tools
Testing Tools
 Dalvik Debug Monitor Server (DDMS)
– Port-forwarding services
– Screen capture on the device
– Thread and heap information on the device
 Logcat
– Process, and radio state information
– Incoming call and SMS spoofing
– Location data spoofing and more
 TaskKiller
– A utility application used to kill process
– Useful to skip device reboot.
Android Debugging Tools
Testing Tools

Mobile Application Testing Tutorial iOS Tools
iOS Tools
Blackberry Tools
 UIAutomation
– Lets you run an automated set of tests against an application
– Can be used to perform automated tests on real devices and on the
iPhone Simulator
– It enables you to quickly track regressions and performance issues
 NUnit
– Unit level testing for iOS apps
 Test Studio for iOS
– Lets developers test native iOS apps as well as hybrid and Web apps
– The tool can test common iOS controls and record test scripts for
gestures such as taps, multiple finger gestures such as zooms, swipes
and taps.
 iPhone test automation
– Fonemonkey
– UISpec
– Bromine (not recommended)
iOS Tools
Testing Tools
Testing Tools
 BlackBerry device simulator – “Fledge”
 Jamo Solutions – “M-eux Test”
 Borland Silk Mobile
Blackberry Tools
Max1

Mobile Application Testing Tutorial Windows Tools
Windows Tools
Remote Device Access Services
 MSDN Hopper Test Tool
– Simulates input stress on Windows Mobile devices
– Hopper will stress all applications that are available through the
menu system by rapidly sending keystrokes and screen taps in a
random fashion
 MSDN Device Emulator
– Mimics the behavior of Windows Mobile
– Provides a virtual hardware platform that you can use to test
applications on multiple virtual devices
– Runs code compiled for ARM microprocessors
 MSDN Cellular Emulator
Windows Tools
Testing Tools
 DeviceAnywhere
 Perfecto Mobile
 Keynote
 Paca Mobile Center
 pCloudy
Remote Device Access Services
Testing Tools

Mobile Application Testing Tutorial Security Testing Tools
Security Testing Tools
Performance Testing Tools
 Kryptowire (Native App Testing Tool)
 OWASP ZAP (Mobile App Testing Tool)
 For more information and training on Security
Testing Tools, check out the Security Testing
Course from SQE.
 Additional resources at owasp.org
Security Testing Tools
Testing Tools
 NeoLoad
 BlazeMeter
 SOASTA
 Mobitest
Performance Testing Tools
Testing Tools

Mobile Application Testing Tutorial Crowd Source Testing Services
Crowd Source Testing Services
Outsourced Mobile App Testing Services
 GetJar
 Mob4Hire
 uTest
 Ubertesters
 App Lover
 Apphance
 Beta Bait
 Bugcrowd
Crowd Source Testing Services
Testing Tools
 Coveros
 Mobica
 Intertek
 RapidSoft
 Belatrix
 Ranorex
 HP
 TestQuest
 Utopia Solutions
 Wipro
Outsourced Mobile App Testing Services
Testing Tools

Mobile Application Testing Tutorial Wrap-Up
Wrap-Up
Agenda
Wrap-Up
Agenda
 Wrap-Up
 Recommendations
 Questions

Mobile Application Testing Tutorial Mobile Testing Books we recommend
Mobile Testing Books we recommend
Questions?
Mobile Testing Books we recommend
 “Tap Into Mobile Application Testing”
Kohl, Jonathan
 “A Practical Guide to Testing Wireless Smartphone Applications”
Harty, Julian
 “Mobile Application Security”
Dwivedi, Himanshu
 “Mobile Developer’s Guide To The Galaxy”
http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Mobile_De
vGuide_13.pdf
Questions?
Contact Information:
training@coveros.com

Mobile Applications Testing: From Concepts to Practice

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Mobile Applications Testing: From Concepts to Practice (20)

More from TechWell (20)

Recently uploaded (20)

Mobile Applications Testing: From Concepts to Practice