Using JMeter and Google Analytics for Software Performance Testing

Using JMeter and Google
Analytics
How to avoid
performance
testing
yourself into a
corner...
Ed Curran
Performance Testing Considerations

XBOSoft, Inc. All Rights Reserved.
XBOSoft
Dedicated to software quality improvement
Founded in 2006
We speed products to market with our expertise:
• Software QA consulting
• Software testing
Global team with offices in San Francisco, Beijing
and Amsterdam
“Thorough, accurate and fast”
2

House Rules
▪ Participants other than the speakers are muted
▪ Questions via the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
▪ Questions may be asked throughout the webinar -
we’ll try to answer them at the end
▪ You will receive info on recording after the
webinar
3
Webinar Hashtag: #XboPerformance

Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
Relevant specialties and
passions
• Software quality process, evaluation,
measurement and improvement
• Software quality in use / UX design
• Mobile User Experience and usability
• Cycling and travel
4
XBOSoft Team

Ed Curran
VP of Engineering
5
• Over 25 years of experience: technology;
international; BSEE/MBA
• Much of career dedicated wireless industry,
as well as software development
• Developer of performance management
monitoring systems for wireless networks for
use in capacity planning
• VzW Representative for Verizon Quality Council
• QuEST Forum / TL9000 Speaker
Meet Our Speakers
XBOSoft Team

But Today is All About You
… Why are you here?
6

Introduction
Software Testing Challenges
• Lack of Standard Processes
• 30% Overruns in Development Time
• Time to Market Pressures
Shanthakumaran, Prasanth. "Trends in Software Testing." KPMG - Business Excellence
(2016): 5-6.
>> A Real Need to Automate
7

Automated Testing Trends
Rehn, Cecilia, ed. "European Software Testing Benchmark Report 2016 - Automation." Editorial.
Test Magazine Aug. 2016: 1-2. Web.
Types of Automation Utilized
Automation Survey Respondents
8

Automated Testing Trends
Automated Testing Benefits
• 90% of companies have seen ROI in year 1
• Large range of automated processes
employed
• Focus on UI Testing
Rehn, Cecilia, ed. "European Software Testing Benchmark Report 2016 - Automation." Editorial.
Test Magazine Aug. 2016: 1-2. Web.
9

Automation Testing Trends
• Test Environment (configuration,
maintenance, stability)
• Many types of automated processes available
– Difficult to determine tests to perform
– Scripts need to be created and maintained
Practical Challenges
10

What is Wrong with My
Performance Test?
Automation Shortcomings
• 82% Automation on UI
– Customer-facing UI
– Potential to overemphasize
• 53% Performance
– Critical Components Left Out (e.g.
Ecosystems)
11

• Load Tests
– Identify bottlenecks, check applications ability to perform under anticipated
loads
• Stress Testing
– Find breaking point
• Endurance Testing
– Support expected load over time
• Spike Testing
– Test software reaction to sudden load spikes
• Volume Testing
– Test software reaction to sudden load spikes
• Scalability Testing
– Determine software effectiveness when scaling up to support user load
Performance Test Types
"Performance testing : Process & Important Metrics - Guru99." 3 Nov. 2016,
http://www.guru99.com/performance-testing.html.
12

Additional Considerations
• Different Types of Tests
– Tests do not stand in a vacuum
– Can be related (e.g., Performance Testing
within the Ecosystem)
• Performance Testing is Unusual
– Time-component dimension
– Exploited through automated testing
13

Software
• Part of a system
• Needs to be designed to specific
requirements
… Critical to consider software operation within
the larger system.
System Complexity
14

What is it you Really Want to Test?
“... the totality of features and characteristics of a
product or service that bear on its ability to satisfy
stated or implied needs.”
"What is Quality | IGI Global." http://www.igi-global.com/dictionary/quality/24254. Accessed 16 Dec. 2016.
Quality in Technology is:
15

What Is It You Really Want to Test
(cont.)
High Level Software Test Concepts
• Smoke Test / Happy Path
• Negative / Boundary Tests
• Timing/Performance/Load dimension
16

What Is A Test?
● h1 (Blue) - A happy path, or
smoke test, set to test a
given function that exists at
t=0.
● b1a, b1b (Red) - This
outward pressure adds to the
smoke test by expanding the
test to include additional
negative tests using
boundary/equivalence value
analysis.
● t=0, t=n - used to denote a
linear time where the
Blue/Red tests can
repeatedly execute over the
course of the time.
● th1 - shows that, during
performance testing, the
blue/red tests can repeatedly
be performed, but not
necessarily at periodic
intervals.
17

Load Testing Example
Stepping Thread Group - Design

Load Testing Example
Stepping Thread Group - Response Results - Part 1

What Is a Test? (cont.)
Real-World Software System
• Simple Example
– Box metaphor
– Simple
– Well organized
• Real World
– Much more complex
20

What is a Test (cont.)
Multiple Performance Tests - Metaphor
21

What is a Test?
… Where Do You Stand?
22

Knowing Where You Stand
Why Drive Load?
… Determine capacity bottlenecks for System under test
• Key Parameters/Factors
– Simultaneous varying inputs
– Simultaneous varying conditions
• Simulate Production Environment
• Run Load Tests to Exceed Requirements
– Ensures known level of buffer
23

Key Prioritization Parameters/Factors
XBOSoft Performance Prioritization Matrix
24

• User
– View of Person using application
• Platform Section
– Function within application under test
• Covered in .jmx
– Currently a script covering
Key Prioritization Parameters/Factors (cont.)
25

• Usage (XBO)
– XBO review of function usage levels
• Tier 2 Customer Input (a.k.a. “Customer
Service Complaints”)
– Generated by end-users
– Holds weight as “real customers”
– Determine relevance
• related to software under evaluation?
• related to a load/performance related issue?
Key Prioritization Parameters/Factors (cont.)
26

• Google Analytics Usage %
– Most Widely Used Analytics Service
– Key insights to activity taking place on site
… GA Pageviews Report
Key Parameters/Factors (continued)
… Behavior Flow Report
27

GA Pageviews Report
"How to Use Google Analytics Behavior Reports to Optimize Your ...." 27 Oct. 2014,
http://www.socialmediaexaminer.com/google-analytics-behavior-reports/. Accessed 12 Dec. 2016.
28

Key Parameters/Factors
GA Pageviews Report
• Simple representation of an application
performance
– Page Views
– Average Time on Page
• How Functions/pages are being accessed
29

GA Behavior Flow Report
30

GA Behavior Flow
• Can assist in optimizing the application
• Can be directly employed into Performance/
Load Script
– Volume
– Path
• Predictor of future traffic load
31

• Functional Isolation
– Knowledge of Software Architecture
• Negative Test Required
– Required for a given function?
– Level and type
• General Importance Factor
– Stakeholder view of a given function (e.g.,
monetary transaction, school grades)
32

... What Do You Need?
33

Knowing What You Need
• Type of Business
• Customer Expectations
– 20-minute outage from Amazon costs $3.75 Million
– 28% of users will not return to a slow site
– 9% of users will not return after an outage.
– Bounce rate for pages loading in 2 seconds = 26%
– Bounce rate for pages taking more than 10 seconds
resulted in 41%
Business Requirements
"2016 Holiday Retail Insights Report." SOASTA, Performance Is Everything. SOASTA, 2016. Web.
… Consumer expectations are increasing.
34

Freedom from Production Servers
• Benefits
– Limits risk to real customers
– Freedom to schedule tests any time
– Supports time-dependent tests that can go on for hour
• Challenges
– Must be properly sized
– Must be maintained (e.g., backups, database
reconfigurations)
Test Application Server
35

JMeter Master/Slave Configurations
36

"How to use distributed JMeter to test netty server performance | Code ...." 29 Mar. 2016,
http://shengwangi.blogspot.com/2016/03/how-to-use-distributed-jmeter-to-test-netty.html.
JMeter Master/Slave Configuration
37

Benefits
• Slaves Offload Master as primary load driver
• Emulate users from different areas and
locations
• Allows graphical real-time viewing on Master
of tests in progress
JMeter Master/Slave Configuration
38

Application
Server 1
JMeter M/S Configuration (Regional)
By Thesevenseas (talk) - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=7376761
Application
Server 2
JMeter Slaves
39

Benefits
• Simple method for driving load to web
application servers
• Allows test support from centralized location
JMeter M/S Configuration (Regional)
40

JMeter M/S Configuration (Multi-Regional)
Map By Thesevenseas (talk) - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=7376761
Application
Server
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
JMeter Slave
41

Benefits
• Emulate users from different areas and
locations in “real world” scenarios
• Allows graphical real-time viewing on Master
of tests in progress
JMeter M/S Configuration (Multi-Regional)
42

API vs. HTML
43

• Required at the Start of Performance Analysis
– Current and anticipated utilization of CPU
– memory requirements
– disk drive capacity needs
• System Profile = System View
– >>> should include
• Application Server
• Performance/load computers
• All associated networks
System Profile for JMeter
44

Getting Real
Create Realistic Tests
• Mirror Production Environment
– To the extent practicable
• Focus on Critical Areas
– Utilize Performance Prioritization Matrix
• Beware of Too Test Scripts
– Avoid Non-Critical Areas
– IDEs create scripts fast (too many?)
– Can create test management challenge
45

Getting Real
Standalone Performance Tests Have Merit ...
A Game of Statistics
…. You can’t test everything
46

For more, please download:
Performance / Load Testing
Using JMeter and Google Analytics
http://xbosoft.com/knowledge-center/software-testing
-white-paper/
Edwin H. Curran
@ecurran
ed.curran@xbosoft.com
Thank You!
47

Using JMeter and Google Analytics for Software Performance Testing

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to Using JMeter and Google Analytics for Software Performance Testing (20)

More from XBOSoft (20)

Recently uploaded (20)

Using JMeter and Google Analytics for Software Performance Testing