Load and Performance Testing for J2EE - Testing, monitoring and reporting using Open Source Tools

LOAD & PERFORMANCE TESTING FOR J2EE
Alexandru Ersenie
Senior Load & Performance Test Engineer
Edict eGaming GmbH, Hamburg
alexandru.ersenie@googlemail.com
http://www.alexandru-ersenie.com
TESTING, MONITORING, ANALYSING AND REPORTING USING OPEN SOURCE TOOLS

07/25/12 Alexandru Ersenie - Load & Performance Testing for J2EE 2
AGENDA
SAMPLING/PROFILING
3 TIER TUNING
VIRTUALIZATION
TESTING IN THE CLOUD
SECURITY
LOGGING
BEST PRACTICES
NETWORKING
........
PERFORMANCE BASICS
Scope
Metrics
Factors on Performance
Performance Reports
Generating Load
I
MONITORING
Monitoring types
CPU Monitoring
Other Performance Monitors
Reactive Monitoring/Reporting
II
JVM GC and Heap Monitoring
TOOLS
Monitoring
Reporting
Analysis
III

PERFORMANCE BASICS
1.1

1.1 DEFINITION AND SCOPE
PERFORMANCE
Short response time for a given piece of
work
High throughput (rate of processing work)
Low utilization of computing resources
High availability of the computing system
or application
Efficient usage of
hardware resources;
Storage
Continue to operate
despite errors
SCALABILITY
Handle growing amount
of work;
Ability to enlarge
to accommodate more
work
EFFICIENCY RELIABILITY
Recover after
failure;
Mean time
between failures
ROBUSTNESS
Non functional
requirements

1.1 REAL WORLD NUMBERS
1 second of slower performance on pages could cost Amazon $1.6 billion in sales each year
25% of users will leave a site if a page takes more than 4 seconds to load.
http://performance-testing.org/performance-testing-statistics
http://en.wikipedia.org/wiki/BigTable
RESPONSE TIMES
Facebook serves more than 2 million ‘Like’ buttons per second (June 2010).
Facebook held a clear lead in total page views during March 2011, recording about 85
billion. This was more than three times as many as number two Google, which had about
25.6 billion
THROUGHPUT
Google uses a compressed, high performance, proprietary data storage file system, (...)
designed to scale into the petabyte range (1000 terabytes)
EFFICIENCY

METRICS
1.2

1.2 METRICS
OPTIMISTIC
SLA are already defined
Concurrent users
Transactions / Second
Response Times
Storage
Concurrent users
10 000 logged in users
50 000 visitors
1 000 business transactions / second
4 000 web requests / second
Response Times
Landing page total load time less than
8 seconds for 70 % of the users
Transaction Response time less than
2 seconds for 90 % of all transactions
Storage
Maximum 100 bytes
per transaction
TEST TO ACCHIEVE DEFINED SOFTWARE LEVEL AGREEMENTS
How many transactions can
the system handle
How many sessions can
the system handle
What is the average
response size
What is the 90 % value /
business case
How much space will a
transaction use in the database
REAL WORLD PROJECTS
SLA are not defined
Concurrent users
Response Times
Storage
?
TEST TO DEFINE
SOFTWARE LEVEL AGREEMENTS
?

RESPONSE TIMES
What are the response times that users
are receiving when performing specific
transactions in the system:
Minimum / Maximum / Average response
time
50 to 90 percent line
1.2 METRICS
THROUGHPUT
How many users can the system handle,
and how many transactions can the
system handle in a unit of time
Transactions per second
As the number of users using the system increases,
the throughput increases as well. The system is busy
satisfying user requests
When the systems limit is reached, the throughput
decreases, and waiting occurs, since the system is
busy managing itself in order to satisfy all user requests

FACTORS ON
PERFORMANCE
1.3

SOFTWARE
Software design, architecture and configuration have great impact on performance:
PROGRAMMING CONFIGURATION
JVM
Config
Thread
Pool
EJB
container
JDBC
Max-connections=”5”;
jdbc-connection-timeout=”5”
Sync.
errors
Deadlocks
Race conditions
Arithmetic
errors
Buffer overflow
Arithm. exceptions
Redundant
operations
Dead code
Redundant assign
Other
-Xmx:2G
-XX:NewSize:1.8G
-XX:NewSize:1.8G
max-threadpoolsize=”5”
Max-cachesize=”512”;
cache-timeout=3600
Memory
leaks
80 % of all performance
problems in JAVA
1.3 FACTORS ON PERFORMANCE

HARDWARE
WINDOWS5000
cons
LINUX
1024
Open
files
Database Server and Hardware
Hardware /
Configuration
OS /
File System
1 Server
APP
Server
Web
Server
DB
Server
Single 3 Tier Server
DB Server /
Hardware
Oracle
Standard
CPU
CPU
Oracle
Enterprise
CPU
CPU
CPU
CPU
CPU
CPU
●No partitioning
●Limited online
operations
●Limited indexing
MSSql MySQL
●Different locking
implementation
●Different
Index performance
●default range for dynamic
ports in Windows is 1024
to 5000
●Unix systems have a
default maximum open
files limit of 1024
1.3 FACTORS ON PERFORMANCE
Shared resources:
●CPU
●Memory
●Network
●Disk
Virtualization

GENERATING LOAD
1.4

Web
&
Application
Server
LOAD
AGENT
LOAD
AGENT
LOAD
AGENT
LOAD
AGENT
LOAD
DISPATCHER
1.4 GENERATING LOAD
WHAT DO WE NEED?
Test Results
DB
Reporting
Server
Reports
MONITORS

1.4 GENERATING LOAD
IMPLEMENTATION
MODEL
Web
&
Application
Server
JMETER
JMETER
JMETER
JMETER
JMETER
CONTROLLER
MY SQL
DB
JASPER
SERVER
JASPER
REPORTS
JMX;
REST
MONITORS

LOAD AGENTS
Virtual Users
Ramp up time
Pause times
Number of transactions
Number of repetitions
Increasing rate/repetition
Server and Port
MONITORS
Object usage
EJB Resources
JDBC Resources
CPU Time
HEAP Monitoring
REPORTING
Process for DB Import
Process for Maven
Process for other formats
Generate Load
Collect Response Times
Monitor Hardware Resources
Monitor System Resources
Process Results
Import results into database
Generate Reports
DISPATCHER
1.4 GENERATING LOAD
'users=200 ramptime=200 pausetime=2000
pausetimedev=500 transactions=100 repeats=1
loopsinrepeat=1 warmup=no'
'hostname=myserver port=8080 protocol=https'
'trace_objects=yes
monitor_server=yes'
'generatereport=yes '
/start_test.sh

FUNCTIONS
configure_server
empty_logs
monitor_resources
generate_report
start_test
server_warmupJMETER DISPATCHER
FOLDER
/includes
/scripts
SCRIPT REPO
PLACE ORDER
EXPORT
LOGIN
HOMEPAGE
REGISTRATION
Scripts call
functions to
control the test
workflow
/testplans
TESTPLAN REPO
T_Place_Order
T_Export
T_Login
T_Homepage
T_Registration
Scripts start test
plans by using
functions
/results
RESULTS
Results are
stored here
MY SQL
DB
Processed
results are
exported to
a database
JASPER
REPORTS
Reports are
generated
from the DB
1.4 GENERATING LOAD
IMPLEMENTATION
MODEL

PERFORMANCE
REPORT
1.5

1.5 Performance Report
SUMMARY METRICS / TRANSACTION JVM CONFIGURATION
CPU USAGE HEAP USAGETHROUGHPUT & RESPONSE TIMES

MONITORING
2

Allows real time monitoring of critical resources
Active Monitoring
CPU UsageCPU Usage Heap UsageHeap Usage
Thread UsageThread UsageAllows real time interfering with the system:
Execute Garbage Collection
Generate Thread Dump
Analyze thread activity
Generate Heap Dump
Enables monitoring the system's limits
Allows a better understanding of how the system behaves depending on the load scenario, and the
user behavior, thus enabling to determine what the system can perform
Allows quickly modifying either the test configuration,or the software/hardware configuration, repeat the
test and compare the results
Allows parallel analysis of the system and the load scenario, thus enabling to identify root causes and
behaviors quickly
JDBC Usage EJB Cache Usage
2.1 Monitoring Types
AJP Thread UsageAJP Thread Usage

Object Usage
Reactive Monitoring
2.1 Monitoring Types
Throughput Response Times
Resource StatisticsGarbage Collection
Allows reactive analyze and detailed interpretation of
collected data
Data collectors can easily be extended using a modular approach (function based):
●Monitor JDBC; Monitor EJB; Monitor Thread Usage; Monitor Heap; etc
1. Enables monitoring the system's
behavior and resources over a
prolonged period of time
2. Easily extendable by adding collectors
3. Storable information for future comparing
4. Tool independent
5. High definable granularity
6. Automated
ADVANTAGES
1.High amount of data and complexity in
analyzing the results
2.Requires high proficiency with tools,
monitors, collectors for the purpose
of extending the monitoring system
DISADVANTAGES

REACTIVE MONITORING
2.2

2.2 Reactive Monitoring
RESPONSE TIMESWhat are the for specific transactions
THROUGHPUTWhat is the maximum for specific transactions
RESOURCE USAGEWhat is the for a load scenario
GARBAGE COLLECTIONHow efficient is the for a load scenario
50% line60% line70% line80% line90% line
Min
Resp
Time
Avg
Resp
Time
Max
Resp
Time
No. of
transactions
Average
Trans /
Second
Max
Trans /
Second
CPU HEAP JDBC EJB WEB OTHER.................
.............
.............
NO OF
YOUNG
GC
NO OF
OLD
GC
DURATION
OF
GC
%
IN
GC
OTHER.................
OTHER
OTHER

RESPONSE TIMES
THROUGHPUT

RESPONSE TIMES DRILL
THROUGHPUT DRILL

RESOURCES
CPU HEAP
JDBC CONNECTIONS

GARBAGE COLLECTION

GARBAGE COLLECTION STATISTICS

OBJECT USAGE
See the behavior of objects in time
Drill down on the object to see detailed information like Instances and Bytes occupied

OBJECT USAGE - PRIMITIVES
See the behavior of objects in time
Drill down on the object to see detailed information like Instances and Bytes occupied

JMX
Java Management Extensions
Supplies tools for managing/monitoring:
● Applications
●,System objects
● Devices
● Service oriented networks.
Resources are represented by
objects called MBeans
REST MONITORING
Monitoring and management data
Exposed by the application server
(example: Glassfish)
SYSTEM
●Top
●Vmstat
●Iostat
●jmap
SERVER LOGS
Garbage Collection Logs
COLLECTORS
LOAD GENERATOR
●Transactions
●Response Times
●Response Codes
●Response Sizes
MY SQL
DB

ACTIVE MONITORING
CPU
2.3

INSERT MOVIE WITH RUNNING TEST AND
MONITORING

●Represents the first input in identifying throughput decrease
Allows real time monitoring of CPU usage
Allows identifying hotspots(spikes) and relate them in time using graphical timestamp representation
Identify Garbage Collections and their effects on CPU Usage
Allows creating “Snapshots” and future reactive analysis based on CPU usage patterns
Allows identifying outside contributors (Database, JMS, I/O)
2.3 Monitoring Types – CPU Monitoring

Server resources properly configured – EJB Cache, EJB Pool, Thread Pool, JDBC
Connection Pool
OPTIMAL BEHAVIOR
Load configuration adapted for optimum usage of resources: Requests are being
Processed, JDBC connections are used optimally

Depending on the configured Garbage Collection Strategy, on the Heap usage of your application, and the
properly configuration of the Heap according to business scenarios and memory footprints, Garbage Collection
can have a huge impact on throughput
GARBAGE COLLECTION
Parallel Collections with only two CPU
Stop the world strategy instead of CMS
Poor Heap Configuration (too small, too big, wrong rations of young to old
Manual Garbage Collections

Long running queries – Occupying the JDBC resource for too long. Requests are being queued, translating
in decreased throughput and increased response times
Oracle Redo Log failed checkpoints – Database stops servicing for satisfying internal needs
Pause
Redo Log Statistics
Average of 1 Redo Log File / Minute : Redo Configuration needs update
DATABASE WAITS

Number one cause for thread contention: wait times induced by synchronized methods (threads have to wait in
a queue for acquiring the lock)
Contention
THREAD CONTENTION

ACTIVE MONITORING
HEAP
2.4

Allows real time monitoring of HEAP usage
Allows identifying memory leak trends
Correlate Garbage Collection with CPU Usage:
CPU usage
Time in GC
Overall system behavior on GC
Determine Garbage Collection rates and durations
2.4 Monitoring Types – HEAP Monitoring
HEAP MONITORING

PERMANENT
GENERATION
YOUNG
GENERATION
OLD GENERATION
●Internal representations of the
JAVA classes
●Objects describing classes and
methods
●Information used for
optimization by the JIT
compilers
●Initially objects are allocated in
the Young Generation
●When collected, objects are
moved between survivors
(default 32 collections)
●Objects that have survived the
maximum allowed number of
collections
●Dead objects waiting to be
collected
S1 S2
1 16
17 32
JVM HEAP STRUCTURE

JVM HEAP STRUCTURE

HEAP MONITORING
EXAMPLES
2.4

MEMORY LEAK ?
Used Memory increases as the number of
live objects surviving collections increases
(particularly under load)
An increasing trend is not necessarily a
memory leak. Objects can survive several
collections (max threshold), and can be
already dead, waiting for the old collector
In test systems wait for several full
garbage collections and build a trend line
in order to see if a memory leak is showing
Wait for a full garbage collection to see
if the memory decreases
Test systems can be provided with the
option of manually triggering a garbage
collection to see if objects are being
released after the test is over

A full Garbage Collection goes over all
generations, parsing the entire object
structure to identify removable objects
Usually the old generation is at least
twice the size of the young generation
Collection time for the old generation
significantly increased (object tree parsed)
Dead or marked for GC objects are
removed
Space is reclaimed
Garbage Collection strategy of critical
importance: Parallel or Concurrent GC
NO LEAK

Objects that cannot be collected are moved
to the old generation
Old Garbage Collections cannot remove
the objects; space cannot be reclaimed
Total Heap used increases constantly
Eventually the system performs only
Garbage Collection
Used Memory increases as the number of
live objects surviving collections increases
(particularly under load)
The system is busy managing itself,
instead of running your application
Both throughput and response times
affected
Server restart is required
MEMORY LEAK
The right time to trigger a HEAP DUMP
and see what objects are leaking
ACTION

ACTIVE MONITORING
OTHER MONITORS
2.5

User Profile
Session Information
File contents
Caching shared data in a Hash Map
Every request will want to acquire and
hold the lock on the hash map,
becoming a bottleneck
Brian Goetz: Threading lightly : Reducing Contention
Synchronized Methods
One thread is executing a synchronized
method for an object, all other threads that
invoke synchronized methods for the same
object block (suspend execution) until the first
thread is done with the object.
Java Tutorials - Concurrency
Contention
Threads can be monitored, and blocking
or long running threads can be identified
THREAD CONTENTION
2.5 Monitoring Types – Other Monitors

^
Most often contention problems appear
only under load, due to concurrency effects
(more requests on the same
object/method)
The right time to trigger a thread dump
and see the root cause of contention
ACTION
BEHAVIOR
CPU usage drops dramatically, for it is
busy with solving the locks instead of
satisfying business requests
Several threads in “Monitor” state, at
the same time, indicate thread
contention
THREAD CONTENTION

Allows quickly identifying
blocked/blocking threads
Lists all existing threads and their
state when the thread dump was
triggered
Threads are listed together with
their stack traces, allowing
identifying of:
What resource is being locked
Who locks the resource?
What is the locking thread
doing?
THREAD DUMPS

THREAD DUMPS
Kill -3 PID : writes
stack trace to
jvm.log
jvm.log can be
imported into IBM
Thread Analyzer

Actively monitor the number of
JDBC resources in use, and other
statistics like:
● Connections created
● Connections removed
● Connections in pool
JDBC MONITORING
Actively monitor the number of EJB
Beans in cache and other statistics
like:
● Cache hits
● Passivation statistics
● Number of beans created
● Number of beans removed
EJB MONITORING
Actively monitor the number of
requests, and other statistics like:
● Error count
● Max request time
WEB MONITORING

Database Call on each Transaction – In this case each transaction checks if the user is valid, by creating
and using a new JDBC Connection / check;
2.5 Monitoring Types – Case Study

TOOLS
3

3. Tools
MONITORING
VISUAL VM
CheckJMX
REST Interface
System Interfaces
PROCESSING
PENTAHO
Shell Scripts
LOAD
JMeter
Shell Scripts
REPORT DESIGN
IReport
REPORTS
Jasper Reports
HEAP ANALYSER
MAT
YourKit
THREAD ANALYSER
VisualVM
IBM Thread Analyzer
GARBAGE COLLECTION
HP Jmeter
PROFILING
YourKit

THANK YOU FOR LISTENING
?

Load and Performance Testing for J2EE - Testing, monitoring and reporting using Open Source Tools

More Related Content

What's hot (20)

Similar to Load and Performance Testing for J2EE - Testing, monitoring and reporting using Open Source Tools (20)

Recently uploaded (20)

Load and Performance Testing for J2EE - Testing, monitoring and reporting using Open Source Tools