Appurify Performance Automation Whitepaper FINAL

The State of
Mobile Development
& Performance
in 2014
An Appurify Study for Mobile App & Website Developers
© 2014 appurify

Mobile developers understand the importance of performance and quality.
In our fast-paced mobile marketplace, consumers have no patience for apps
and sites that can’t meet their expectations. Performance testing is essential
to deliver the best possible mobile experience—but what kind of testing tools
and processes are most effective? Technologies and best practices for app
performance are developing rapidly, but to date there has been little data
available on the actual state of the industry.
To capture data for our study, Appurify worked with
Market Cube, a panel research company, to recruit
a total of 400 respondents between Dec. 10 – Dec.
20, 2013. All participants were based in the U.S. and
held responsibility for mobile development strategy
or mobile testing at organizations with 50 or more
employees. Each respondent also had purchasing
decision-making power for performance testing
and automation tools—so their views are especially
important for understanding the needs, preferences,
and plans of companies in this regard. The margin of
error for survey results is +/- 4.9%.
The results of the study reveal key concerns in mobile
app and site performance today—as well as a clear
picture of developers’ plans to create better mobile
experiences for users moving forward.
Appurify initiated a study to shed light on several key issues:
2© 2014 appurify
What are mobile
developers doing today
to ensure quality and
performance?
What challenges do
we face in meeting the
expectations of consumers
for their favorite mobile
apps and sites?
What kind of testing and
automation tools are
proving most useful and
popular—and what would
the ideal tool look like?

3© 2014 appurify
Performance counts.
An overwhelming majority of developers—
nearly 90%—have seen a direct connection
between the performance of their apps and the
reviews and ratings they receive in app stores.
Keeping abreast of user reviews.
Developers have become too reliant on user
reviews to spot quality issues, with 60% of
developers scanning user reviews at least daily.
Testing and optimization is getting harder.
Top challenges cited include making sure apps
work across proliferating devices and platforms,
keeping pace with rapid development cycles,
and the sheer amount of time testing takes.
Updates are too frequent for manual testing.
App updates have increased 62% over the past
year, and 42% of apps and mobile websites
update weekly, which is too rapid for effective
manual testing.
Data is power.
Historical and trending data is recognized as an
important indicator in a company’s optimization
strategy. 98% of companies see testing over time
as important.
Automation is the answer.
Looking for a smarter way to ensure app
performance, 76% of respondents plan to invest
in test automation.
Stability — can the app or website function as
intended without crashes?
UI responsiveness — does the interface render
accurately and respond quickly to user inputs
such as panning, scrolling, and taps?
Network performance — Can the app work well
with available network connectivity, even in
less than ideal signal conditions?
Load time — does the app or website load
quickly or does the user have to wait?
CPU usage — can the app make efficient use
of available processing power, or does it lag on
anything but high-end devices?
Memory usage — can the app make efficient
use of available memory, or does is have issues
on anything but high-end devices?
Users perceive the performance of an app
or website in terms of a variety of factors.
These include:
Developers should pay close attention to these
factors when testing and optimizing the mobile
experiences they create across device types,
platforms, and usage conditions.
Key Findings Definition

Part 1:
The
Mobile Development
Landscape
4© 2014 appurify

Total
50-499
500-999
1000-4999
5000+
# of
Employees Percent of Companies with Mobile Apps / Websites by Company Size
MOBILE APPS MOBILE WEBSITE
5© 2014 appurify
48%
28%
24%
28%
Mobile today encompasses a wide range of experiences for business and
consumers, a variety of development platforms, and a continued commitment
to both apps and websites. Across this diverse space, though, two consistent
themes emerge: companies are releasing updates faster than ever to keep
users satisfied— and they’re paying close attention to the strong tie between
performance and ratings.
Apps vs. Websites—or both
The mobile landscape is shared nearly equally
by mobile apps (72%) and mobile websites (76%),
with 48% of organizations using both. While
mobile sites have a sight edge overall, among
larger companies the advantage goes to apps—
used by 88% of companies with more than 5,000
employees, compared with 78% for sites.
MOBILE
APPS
MOBILE
WEBSITES
BOTH
72% 76%68% 73%69% 77%69% 79%88% 78%
Distribution of Mobile Apps / Websites by Company

Type of Applications & Websites Developed
Total
Consumer
Apps
90%
PRODUCTIVITY
RETAIL
MEDIA
NEWS
LIFESTYLE
GAMES
BUSINESSAPPS
TRAVEL
EMPLOYEE-FACING
APPS
BUSINESSSITES
OTHER
OTHER
CONSUMERSITES
SOCIAL
NETWORKING
Total
Business
Apps
56%
Total
Mobile
Websites
72%
Website for
Business;
Apps for Life
6© 2014 appurify
When it comes to mobile apps, consumers are the name of
the game. When asked to select all types of apps and mobile
websites developed, 90% of companies that were surveyed focus
on consumer apps, 56% focus on business users and 72% focus on
mobile websites. The most popular categories of consumer apps
include productivity, social networking, and retail.
Mobile web development, on the other hand, tends to be more
business-oriented, with 50% of sites targeting business users and
only 42% serving consumers.
49% 33%46% 30%42% 27%38% 50%40%34% 42%40%
1%1%

If you could create the ultimate
mobile testing solution, what
would it include?
I would like a fully automated, fully
integrated system that can replicate testing
on all major mobile devices and OSs so that
we wouldn’t have to spend so much time
running redundant tests.
Release Frequency Change vs. 12 Months Ago
INCREASED STAYED THE SAME DECREASED
7© 2014 appurify
To satisfy demanding users, companies are releasing updates faster than
ever—at least weekly at 42% of organizations—putting more and more
pressure on developers and QA to get things right. Nearly two-thirds of
organizations have accelerated their release cycle in the past year, while
only 1% has slowed the pace.
Faster and faster—and better?
“
”
62%
37%
1%
Frequency of Releases
35%
28%
34%
21%
WEEKLY
MONTHLY
QUARTERLY
LESS THAN
WEEKLY
YEARLY
3%7%

Frequency of Scanning
User Reviews
8© 2014 appurify
As organizations scramble to refresh their apps,
they’re also keeping a close eye on the results of their
efforts. Of the 73% of developers with apps available
in the Apple App Store or on Google Play, 60% scan
their user reviews at least daily—and 19% do so every
hour. Only 10% check their reviews monthly or less.
When they do, they see a direct correlation between
the performance of their app and the ratings given
by users. An overwhelming 87% of organizations
with apps available through the Apple App Store
or Google Play report an impact of performance on
ratings—and 47% of these described the effect as a
large one.
Extent App Performance
Affecting Ratings
LARGE EFFECT
HOURLY DAILY WEEKLY MONTHLY LESS
SOME EFFECT NO EFFECT
47%
41%
29%
7%
52%
2%
3%
Seen App Performance
Affecting Ratings
87%
13%
NO
YES
19%

9© 2014 appurify
If you could create the
ultimate mobile testing
solution, what would it
include?
My ultimate testing solution would involve the
ability to test different types of phones with
different CPUs, different amounts of memory,
operating systems, and platforms. It would also
show which ones have more bugs and which
have better results.
“
”

Part 2:
Performance & Testing
Approaches and
Challenges
10© 2014 appurify

Different companies approach testing differently. This is true in terms of the headcount
assigned to this task, the testing methods used, the elements tested, and even, to an extent,
the importance of tracking testing over time. But however they test their apps, they often
wish they had a better way of doing it. Current tools and process leave something to be
desired—and take valuable time that could better go to creation and innovation.
The largest companies tend to have the most people involved with testing mobile apps
and sites, as one would expect. Of companies with more than 5,000 employees, 31% have
more than 101 people in this role. At smaller businesses of 50 – 499 employees, the norm
is to have 10 or fewer. But the correlation isn’t completely consistent—15% of the largest
companies surveyed also have 10 or fewer people regularly involved in testing, while
18% of companies with fewer than 500 employees involve more than 101 people. Clearly,
different companies can assign varying levels of importance to performance testing.
Companies also use a wide variety of methods for testing. First, we’ll look at functional
testing, which verifies whether given input produces the expected output, such as
testing the login process for an app. By far, the most popular approach involves manual
testing on real devices (78%), followed by the ad hoc discovery of issues by employees
or customers—a far from ideal approach (41%). Relatively few outsource (31%) or
crowdsource (27%) testing.
11© 2014 appurify
The Who & How of Testing
Number of People Involved with Mobile Testing
at Your Organization on a Regular Basis
By Company Size (# of Employees)
1-10
11-50
51-100
101+
Total 50-499 500-999 1000-4999 5000+
39% 54% 36% 30% 15%
24% 27% 19% 27% 22%
24% 14% 34% 27% 32%
12% 4% 4% 16% 31%

A similar range can be seen in approaches to performance testing, which examines
metrics such as load time, battery consumption, and network usage. Again, the
majority use manual testing on real devices (75%) followed by ad hoc testing by
employees or customers (40%).
Significantly, the top two methods for both functional and performance testing involve
real devices, showing a clear preference for this approach over simulators and/or
emulators. At the same time, two of the top three methods in each area rely on automated
testing. From these results, we can infer a strong preference for two factors in testing: real
devices and automation.
12© 2014 appurify
Functional Testing of Apps / Mobile Sites
Testing the Performance of Apps / Mobile Sites
Manual Testing on Real Devices
Ad Hoc as Employees or Customers Find Issues
Outsourced Service
Crowdsourced Solution (e.g. uTest)
Automated Testing
Manual Testing on Real Devices
Ad Hoc as Employees or Customers Find Issues
Outsourced Service
Peformance Testing Software
(e.g. HP, SOASTA, Appurify)
Crowdsourced Solution (e.g. uTest)
Open Source or Home Grown Tools
78%
75%
41%
40%
31%
29%
26%
27%
29%
10%
28%

Taking a closer look at performance testing, we see the issues of
most concern for developers—and, presumably, for users as well.
The performance elements actively tested at most organizations
include memory usage (71%), load time (71%), CPU usage (64%) and
network usage (64%).
Historical and trending data plays an
important role in almost every company’s
testing strategy. A full 98% of organizations
see testing over time as at least somewhat
important, and 56% see it as very important.
Functional test results are tracked over time
at 91% of companies, and performance test
results are tracked at 86% of companies.
13© 2014 appurify
Performance Elements Actively Tested
MEMORY
USAGE
VERY IMPORTANT SOME-
WHAT
NOT
VERY
NOT
AT ALL
NETWORK
USAGE
LOAD
TIME
BATTERY
USAGE
CPU
USAGE
FRAMES
PER SEC.
71%
64%
71%
59%
64%
48%
Importance of Testing
Over Time
35%
7%
1% 1%
56%

Mobile performance is a high-level concern at a majority of businesses. Management
at 62% of organizations reviews performance reports—more than developers (59%),
QA (57%), or operations (46%).
14© 2014 appurify
Tracking Test Results Over Time
When It Comes To Mobile Performance, Who In Your
Organization Is Involved In Reviewing The Results
/ Any Performance Reports?
Functional Testing Performance Testing
91% 86%
9% 14%
NO
YES
MANAGEMENT OPERATIONS
(OPS)
DEVELOPERS QUALITY
ASSURANCE
(QA)
62%
46%
59%
57%

15© 2014 appurify
would it include?
Accelerate mobile application testing with
automated functional testing features. Manage
complexity and change while testing on multiple
devices and platforms. Prevent service outages
and disruptions through load and performance
testing. Incorporate location and network
speed into functional and performance testing.
Reuse scripts to reduce rework and to monitor
performance in production.
While businesses clearly understand the importance of performance and functional testing,
they’re far from fully satisfied with their current testing processes. 61% rate their testing
processes for device coverage as less than ideal, and 58% for operating system coverage.
Nearly two-thirds see room for improvement in their ability to pinpoint and address crashes
(64%) and in their ability to know how an app will actually perform in the wild in terms of
load time, memory usage, and other key factors (65%).
When asked about their top three toughest
challenges for mobile testing, companies
produced a list that will sound familiar to
every mobile developer:
Respondents also provided a rogue’s gallery
of the most difficult elements to test using
their existing tools, citing the worst as:
“
”
Making sure apps work across all device/
operating system possibilities
It’s very time-consuming
Keeping pace with quick development cycles
Hard to pinpoint and/or recreate the exact
location of bugs and other issues
Accounting for different user and device
conditions, such as geographies or low
memory on devices
Accounting for different network conditions
(networks, signal strengths, etc.)
Crashes
Network usage
CPU usage
Load time
Basic pass/fail functionality
Memory usage
Room For Improvement

16© 2014 appurify
It seems obvious that the most strategic aspect of an app developer’s work is the creation of
compelling and differentiated mobile experiences—not routine performance and functional
testing. Strikingly, though, a full 78% of companies spend at least as much time testing their
mobile apps and sites as they do developing them—and 15% actually spend more time on
testing. This underscores the importance of testing automation tools to allow companies to
focus their attention and resources on creation rather than troubleshooting.
As discussed above, companies commonly see
testing on real devices as the most ideal approach.
Nearly half of respondents report doing so very
often (47%), while nearly as many say they do so
often (42%). But that doesn’t mean it’s easy. The
biggest challenges involved in testing on real
devices include:
Automating the process
Connecting into continuous integration
practices
Management of devices, including charging,
storage, and keeping them up to date
Getting access to phones running on older
operating systems
Buying new devices
Making Time For Testing
would it include?
Have all the devices need for testing in one
place for automatic testing so there would be
no need for manual testing.
“
”

Part 3:
The Tools
of the Trade
17© 2014 appurify

0
1
2
3
4
5
18© 2014 appurify
22%
15%
24%
3%
9%
27%
Where testing tools are concerned, companies prefer not to limit their options.
Most use a variety of free and paid tools, and the more people they devote to
testing, the larger their toolbox grows.
Free and paid—there’s room for both
Number of Free and Paid Tools Used
Number of Free Tools Number of Paid Tools
Companies generally use multiple tools for testing, showing no
strong preference for either free or paid options. Almost every
company uses more than one free tool (88%)—and almost every
company uses more than one paid tool as well (90%). Both kinds
have a role to play in the process.
As one would expect, companies with a larger
number of people involved in the testing process
tend to use a larger number of free and paid tools.
20%
19%
26%
3%
7%
26%

19© 2014 appurify
would it include?
Consistent repeatable task or scripts that
measure criteria exactly the same way each
time, and eliminate the subjectivity of the test.
“
”
Number of Free and Paid Tools Used
% Using 4 or More
% with 4+ Free Tools % with 4+ Paid Tools
TOTAL TOTAL
1-10
TESTERS
1-10
TESTERS
11-50
TESTERS
11-50
TESTERS
51+
TESTERS
51+
TESTERS
28%
32%
38%
43%
50%
62%
42% 45%

21© 2014 appurify
When Do You Plan To Invest In Test Automation?
IN THE
NEXT 3
MONTHS
IN 3-6
MONTHS
IN 6
MONTHS
TO 1 YEAR
IN 1-2
YEARS
IN MORE
THAN 3
YEARS
29%
34%
15%
1%
21%
As testing gets more demanding, automation becomes more critical. Most companies
have already adopted test automation, and they’re continuing to expand their
capabilities in this area. Those that haven’t yet are only looking for the right way to
move forward.
The Time Is Now
Faced with a dynamic marketplace, demanding consumers, and
accelerated release cycles, it’s no surprise that a strong majority of
companies—74%—have adopted test automation. A similar number
report plans to make further investments in test automation. Time is
of the essence; 21% will make these investments within the next three
months, and another 29% within the next six months. Only 16% plan
to wait longer than a year.
Irrespective of the popularity of free testing tools, most companies
are prepared to devote significant budget resources to these tools.
Of those planning to invest in test automation, 70% plan to spend at
least $50,000, and 12% have earmarked at least $500,000.

22© 2014 appurify
How Much Do You Plan To Spend On Implementation?
LESS
THAN
$10k
$10k
to
$50k
$50k
to
$100k
$100k
to
$500k
$500k
-
$1mm
MORE
THAN
$1mm
19%
35%
23%
8% 4%
11%
Irrespective of the popularity of free testing tools, most companies
are prepared to devote significant budget resources to these tools.
Of those planning to invest in test automation, 70% plan to spend
at least $50,000, and 12% have earmarked at least $500,000.
would it include?
Automating your mobile application testing is the
best way to achieve the quick, precise results you
need to accommodate fast development cycles.
For the most realistic results, you must also test
on actual devices—without optical recognition,
emulation, jailbreaking or tethering.
“
”

23© 2014 appurify
If The Process Of Writing Tests Was Significantly
Easier, How Likely Would You Be To Adopt Automation
Practices Into Your Organization?
VERY
LIKELY
IN 3-6
MONTHS
IN 6
MONTHS
TO 1 YEAR
IN 1-2
YEARS
IN MORE
THAN 3
YEARS
29%
34%
15%
1%
21%
If Only We Could...
Of the 26% of organizations not using test automation, reasons cited include uncertainty about how
to get started or which framework to use, lack of budget, and lack of resources. Significantly, only 1%
has no interest in it—which means that for the vast majority of companies lacking test automation,
this is a matter of circumstances, not of choice.
The question then becomes: how could the implementation of test automation be made more
feasible for the companies who have yet to take this step? Test scripts emerge as a key factor. Asked
how an easier process for writing tests would affect their decision to adopt automation practices,
35% said that they would be very likely to do so, and another 41% said that they would be more
likely. Only 1% said that they would remain unlikely to adopt automation even with a significantly
easier test writing process.

24© 2014 appurify
For organizations that have adopted
test automation, the benefits have
been diverse and substantial. Asked
about the top benefits they’ve seen,
respondents reported:
Saving time
Increasing quality
Reducing cost
Increasing speed of development
Minimizing hours spent on other
methods of testing
Greater test coverage
Repeatability of tests
Testing Pays Off

Conclusion
26© 2014 appurify
The results of the Appurify study show that, while companies
take a variety of approaches to the mobile experiences they create
and the resources they use to do so, they share the need for active
testing to meet user requirements in terms of both functionality
and performance.
As release cycles accelerate, organizations can end up spending
an inordinate amount of time on testing, increasing the need for
automation. At the same time, they are often less than satisfied with
the tools they’re using, even when they’re already using several free
and paid tools. What they most want is a way to test on real devices
while simulating the full range of devices, platforms, and usage
conditions—simply and efficiently.
A significant number of companies
plan to invest in test automation. Time
is of the essence; 21% will make these
investments within the next three
months, and another 29% within the next
six months. Only 16% plan to wait longer
than a year. Organizations are finding
that automation is critical to improve
the optimization and performance of
their apps and have received significant
benefits including the following:
1 Saving time
2 Increasing quality
3 Reducing cost
4 Increasing speed of development
5 Minimizing hours spent on
other methods of testing
6 Greater test coverage
7 Repeatability of tests

About Appurify and AMP
27© 2014 appurify
Designed to meet the needs of mobile app and website developers, AMP (Appurify
Mobile Platform) is the mobile app optimization solution that boosts the quality
and performance of mobile apps and sites pre-release in a controlled, measurable
environment. With automated analysis across a comprehensive set of real devices
and user conditions, dev, QA, ops and management teams can now get the
visibility and actionable intelligence they need to shorten dev cycles, maintain
brand credibility and build better mobile experiences.
To learn more about AMP, please contact Appurify at:
sales@appurify.com.

Appurify Performance Automation Whitepaper FINAL

More Related Content

What's hot (20)

Similar to Appurify Performance Automation Whitepaper FINAL (20)

Appurify Performance Automation Whitepaper FINAL