Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005)

Jonathan Misurda (jmisurda@cs.pitt.edu)
James A. Clause, Juliya L. Reed, Bruce R. Childers
Department of Computer Science
University of Pittsburgh
Pittsburgh, Pennsylvania 15260 USA
Mary Lou Soffa
Department of Computer Science
University of Virginia
Charlottesville, Virginia 22904 USA
Demand-Driven Structural Testing
with Dynamic Instrumentation

Software Testing
n  What is Software Testing?
q  Gather information about the behavior of the software being
developed or modified.
n  Why test software?
q  Gain confidence and insights into software correctness
q  Create quality, robust programs
n  Why is testing hard?
q  Testing is expensive
n  50-60% of the total cost of software development
q  Adds complexity to development process
n  One testing technique is not enough

Structural Software Testing
n  Structural Testing
q  The process of discovering run-time properties of a
program specifically pertaining to control-flow
q  Demonstrate adequacy of the test cases
q  Different granularities of structures:
n  Node coverage records statements (basic blocks)
n  Branch coverage records control-flow edges
n  Def-Use coverage records pairs of variable definitions
and uses
q  Over multiple test cases until coverage criteria

Current Testing Tools
n  E.g.: JCover, Clover, IBM Rational PurifyPlus
n  Static instrumentation with test probes
q  Probes are injected instrumentation to gather coverage
information
n  Limitations:
q  Expensive in both time and space
q  Only one type of test
q  Limited code regions and scopes
q  Cannot define their own ways to test

Our Approach
n  Demand-driven structural testing
q  Specification driven: User written tests
q  Test plans: Recipe of how & where to test
q  Path specific: Instrument only what is needed
q  Dynamic: Insert & remove instrumentation
n  Jazz: A scalable & flexible framework for testing
q  Automatically apply multiple test strategies
q  Handle large programs
q  Allows user customization of tests

Jazz Overview
Application
IDE & Test GUI
Test Specification
Test Planner
Test Plan
Test Analyzer and Result Reporter
Test Dynamic Instrumenter
Test Results
Select the regions and test types
Allows user customization
Automatically generate where and
how to instrument
Run the program with demand-driven
instrumentation
Collect the results

Test Specifier
1a
1b
3
2
Specify the regions to test
1a. Highlight line regions
1b. Select classes and methods
List of desired tests
Click to create a test

Test Planner
n  Test plan: Where and how to test program
n  Test storage, probe location table,
instrumentation payload
n  Can combine payloads to create custom strategies
Application
Test
Planner
Test
Specification
Test Instrumentation
Payload
Test Plan
Global Storage & Probe Location Table

Test Planner
n  Challenges:
q  When to insert test probes
q  Where to test/instrument
q  What to do at each probe
n  A planner determines three things:
q  Seed Probe Set – probes which are initially inserted in a
test region.
q  Coverage Probe Set – probes that can be inserted and
removed on demand along a path.
q  Probe Lifetime – whether a probe must be re-inserted
after being hit by its “control flow successor” basic blocks.

Branch Coverage Example Plan
public class simpleDemo {
public static void main(String[] args) {
int evenSum = 0;
int oddSum = 0;
for(int i = 0; i < 100; i++) {
if(i%2==0) {
evenSum += i;
}
else {
oddSum += i;
}
}
System.out.println("The sum of the even numbers from 0 to 100 is: " + evenSum);
System.out.println("The sum of the odd numbers from 0 to 100 is: " + oddSum);
}
}
DEFINITIONS {
NAME: d_method, REGION_D,
LOCATION: FILE simpleDemo.java {
CLASS simpleDemo, METHOD main
}
}
BODY {
DO BRANCH_TEST ON REGION d_method UNTIL: 85%
}
simpleDemo.java
simpleDemo.testspec

Branch Coverage Planner
n  Record which edges are executed
q  Determine (source, sink) edge pairs hit at run-time
q  Source is a branch
q  Sink can be taken & not-taken target of branch
Probe Set Members
Seed Probe First block in testing region
Coverage Probe Sinks of currently instrumented block
Probe Lifetime Until all incoming edges have been
covered

Branch Coverage Example
Probe Loc Tab Storage
Block Next Hit
1 2,3
2 4
3 4
4 1
Test Payload
Mark edge hit
Insert at next point
Remove instrumentation
1
2 3
4
Coverage probe
Hit
N, N
N
N
N
Y, Y
Y
Y
Y
Seed probe

Demo
n  Branch coverage of music player
q  Test region is whole program (JOrbis)
q  Test input is a song
n  Compared
q  Traditional approach with static instrumentation
q  Demand-driven approach with dynamic instrumentation
With Dynamic
Instrumentation
With Static
Instrumentation

Other Planners
n  Node Coverage Planner
q  Record which statements are executed
Probe Set Members
Seed Probe First statement in the testing region
Coverage Probe Next reachable statement
Probe Lifetime Removed as soon as statement is executed
n  Def-Use Coverage Planner
q  Record which variable definition reaches an executed use
Probe Set Members
Seed Probe All variable definitions
Coverage Probe Uses associated with all definitions
Probe Lifetime Defs removed after all reachable uses are
covered. Uses are removed as executed

Dynamic Instrumentation
n  Test plan targets an instrumentation API
n  FIST instrumentation engine [WOSS’04]
q  Retargetable & reconfigurable
q  Dynamic insertion & removal of instrumentation
q  Binary level instrumentation (post JIT)
n  Uses fast breakpoints [Kessler]: Replace existing
instruction with a jump to instrumentation

Experimental Methodology
n  Test Specification
q  Implemented as an Eclipse 3.0 plugin
n  Test Planner & Dynamic Instrumenter
q  Implemented using the Jikes RVM version 2.1.1
q  On the x86 platform
n  SPECjvm98 benchmarks test input
q  unloaded 2.4 Ghz Pentium IV, 1GB of memory
q  RedHat Linux 7.3
n  Traditional vs. Jazz (in same implementation)
q  Compared coverage, run-time, memory

Coverage Results
n  Coverage – same reported by both tools
Branch Node Def-Use
compress

58% 90.6% 89.8%
jess

46.8% 80.3% 71.8%
db

44.4% 76.9% 75.0%
javac

38.9% 75.0% 66.9%
mpegaudio

60.9% 88.7% 90.5%
mtrt

50.6% 90.3% 87.3%
jack

55.6% 82.2% 73.4%

Run-time Performance
Static Branch
Dynamic Branch
Static Node Static Def-Use
Dynamic Node Dynamic Def-Use

Memory Requirements
Branch Node Def-Use
Demand Static Demand Static Demand Static
compress

7.9

7.5

4.4 3.7 8.8 5.6
jess

50.2

60.3

34.1 29.7 70.3 44.8
db

9.7

8.9

5.0 4.1 9.8 5.1
javac

178.9

186

107.3 85.3 202.0 125.3
mpegaudio

24.7

29.5

15.0 14.7 34.8 22.3
mtrt

22.4

23

12.5 11.0 26.1 16.1
jack

73.4

78

46.5 37.6 89.0 46.8
• In Kilobytes

Summary
n  New demand-driven approach
q  A tool (Jazz) for structural testing
q  Dynamic instrumentation guided by a program’s
execution
q  Minimally intrusive
q  User configurable and flexible testing
n  Very low overhead
q  E.g., branch coverage tool is 3-4x faster than
traditional approaches

For Further Information…
jmisurda@cs.pitt.edu
http://www.cs.pitt.edu/coco

Test Results
1 Click to run test suite
2 Results

Static Instrumentation
n  Shortcomings of static instrumentation:
q  Not scalable: Instrumentation left in place causes
unnecessary increase in overhead
q  Inflexible: Only certain tests, languages, and
platforms
q  Cumbersome: Requires recompilations for
dedicated testing
q  Intrusive: Addition of testing code may alter or
mask defects

Run-time Performance
-10%
30%
70%
110%
150%
190%
230%
270%
compress jess db javac mpeg mtrt jack avg
%slowdownoverbaseline
Static Branch Dynamic Branch Static Node
Dynamic Node Static Def-Use Dynamic Def-Use

Branch Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
Dynamic Static None

Node Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
Dynamic Static None

Def-Use Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
Dynamic Static None

Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005)

More Related Content

Similar to Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005) (20)

More from James Clause (14)

Recently uploaded (20)

Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005)