20051019 automating regression testing for evolving gui software

Automating Regression Testing
For Evolving GUI Software

Zheng-Wen Shen
2005/10/19

1

References
Automating regression testing for evolving
GUI software
– Atif Memon, Adithya Nagarajan and Qing Xie
– JOURNAL OF SOFTWARE MAINTENANCE
AND EVOLUTION: RESEARCH AND
PRACTICE 2005; 17:27–64.

2

Outline
1. Introduction
2. The DART process
3. Design of DART
– 3.1. GUI Representation
– 3.2 Modules of DART
4. Experiments
5. Conclusions

3

1. Introduction
Today’s competitive software development
market demands that several developers,
who are perhaps geographically
distributed, work simultaneously on large
parts of the code during maintenance.
Rapid quality assurance feedback
– nightly/daily building and smoke testing.
– To build the product and test it every day

4

DART
(Daily Automated Regression Tester)
A new framework that address the needs of re-
testing frequent builds of GUI software.
Automation!!
– structural GUI analysis
– test-case generation
– test-oracle creation
– code instrumentation to
test execution
coverage evaluation,
regeneration of test cases
Test cases re-execution.

5

2. The DART Process

6

Roles of the developer/tester and DART

Maintenance
cycle

8

3. Design of DART
 Driving philosophy behind the design
1. Automated
2. Efficient
3. Robust
4. Portable
5. General

10

Runs all
test cases
Expected
results
How much
testing

“glue”

Coverage
information Reverse
Engineering

Test Cases

11

3.1 GUI Representation
A formal model of the AUT’s GUI

GUI Representation
1 Objects & Properties

2 Events 3 Components

4 Event-flow Graphs 5 Integration Tree

12

3.1.1 Objects and properties
objects O = {o1, o2, . . . ,om}
properties P = {p1, p2, . . . ,pl }
A GUI = O + P
The state of a GUI = a set P at time t.
valid initial state set = iff the GUI may be in
any state Si ∈ SI when it is first invoked.

13

3.1.2 Events
The events E = {e1, e2, . . . , en} associated
with a GUI are functions from one state to
another state of the GUI.
legal event sequence = e1;e2;e3; . . . ; en

14

3.1.3 Component
A GUI component C is an ordered pair (RF, UF)
– A modal window + a set of modeless windows
RF represents a modal window in terms of its
events.
UF is a set whose elements represent modeless
windows also in terms of their events.
Each element of UF is invoked either by an
event in UF or RF.

15

3.1.4 Event-flow graphs
All possible interactions among the
events in a component.
An event-flow graph for a component C is
a 4-tuple <V, E, B, I> where:
– V : all the events in the component
– E ⊆ V × V : directed edges
– B ⊆ V : available to the user when the
component is first invoked
– I ⊆ V : invoke other components.

16

EFG for part of MS WordPad.
“main” component

17

3.1.5 Integration Tree (IT)
To identify interactions among
components.
Invocations
– Component Cx invokes component Cy iff
Cx contains an event ex that invokes Cy .

18

Definition of IT
An IT is a triple <N, R, B>
– N : components in the GUI.
– R ∈ N: a designated component called the
Main component.
– B : directed edges , the invokes relation
between components

19

IT for part of MS WordPad.

20

3.1.6 Event Classification
1. Restricted-focus events
2. Unrestricted-focus events
3. Termination events
4. Menu-open events
5. System-interaction events

21

1 Restricted-focus event
open modal windows

22

2 Unrestricted-focus events
open modeless windows

23

3 Termination events
close modal windows

24

4 Menu-open events
Open menus

25

5 System-interaction events
Interact with the underlying software
to perform some action

26

3.2 Modules of DART
2
1 Test Case Generator
3
GUI Ripper Test Oracle Generator

GUI Representation

4 6
5 Test Executor
Coverage Evaluator
Code/Event Instrumenter

27

3.2.1 GUI Ripper
Step1: GUI traversal and extraction (DFS)
Step2: Manual inspection by Tester
Step3: Generating the event-flow graph and
Integration Tree

28

3.2.2 Test-case generator
A GUI test case T is a pair <S0, e1; e2; . . . ;
e n>
Generate test case by traversing the
event-flow graphs

29

3.2.3 Test-oracle generator
Test oracles are used to determine
whether or not the software executed
correctly during testing.
<S0, e1; e2; . . . ; en>  S1;S2;…;Sn

expected state or not

30

Approach: execution extraction
A test case is executed on an existing,
presumably correct version of the software
and its state is extracted and stored as
oracle information.

31

Level of Oracle Information
(LOI)
Complete (LO1)
– {(w, p, o), ∀w ∈ Windows, ∀o = objects ∈ w, ∀p =
properties ∈ o}
Complete visible (LO2)
– {(w, p, o), ∀(w ∈ Windows)&(w is visible), ∀o = objects
∈ w, ∀p = properties ∈ o}
Active window (LO3)
– {(w, p, o), (w = active Window), ∀o = objects ∈ w, ∀p =
properties ∈ o},
Widget (LO4)
– {(w, p, o), (w = active Window), o = current object, ∀p
= properties ∈ o}

32

3.2.4 Coverage evaluator
Code coverage and event coverage
Event-based coverage criteria
– Event coverage (length-1)
– Event-interaction coverage (length-2)
– Length-n event-sequence coverage

33

3.2.5 Event instrumenter
All event sequences that are executed on
the GUI be collected.
The key Idea: To detect the existing
listeners and attach our own listeners.

34

3.2.6 Test executor
Executing an entire test suite
automatically on the AUT.
– perform all the events in each test case
What properties should be compared?
– level of testing (LOT1 - LOT4)
– The test designer may choose to employ
partial oracle information.

36

Possibilities available to the test designer for level
of detail of oracle information.

10 different combinations

37

4. Experiments
The subject applications: TerpOffice

http://www.cs.umd.edu/~atif/newsite/terpoffice.htm
38

4.1 Results
Space Requirements
Time requirements
Code coverage

45

Space requirements

500mb

46

Time requirements
All times are reported for a 2.2 GHz
Pentium 4 machine with 256 MB of RAM.
The results show that the smoke testing
process is practical, in that it can be
performed in one night.

47

TerpPaint
51.9 Hours

48

TerpPad

4.7 Hours

49

TerpCalc

11 Hours

50

TerpSpreadSheet

18.6 Hours

51

TerpDraw
8 Hours

52

TerpManager

2.7 Hours

53

Code coverage
We instrumented the application before
running all of the smoke test cases.
We recorded the statements that were
executed for each user-implemented class
during test-case execution.

54

5. Conclusions
We define a formal model of a GUI
derived from specifications that is
useful for smoke testing.
We develop a new process for re-
testing nightly builds of GUI software.
Our regression testing process
cannot only be used for nightly builds,
but also for general GUI re-testing.
61

In the future
We will study the effectiveness of the DART
process by analyzing the number of faults
detected.
We will also integrate DART in a higher-level
process that involves executing other types non-
GUI) of smoke tests on the software.
We will also investigate the application of DART
to other software systems that take events as
input. (e.g. Web applications)

62

20051019 automating regression testing for evolving gui software

More Related Content

What's hot (20)

Similar to 20051019 automating regression testing for evolving gui software (20)

More from Will Shen (15)

Recently uploaded (20)

20051019 automating regression testing for evolving gui software

Editor's Notes