mtATL

Towards effective mutation testing for ATL
Esther Guerra, Juan de Lara
Universidad Autónoma de Madrid (Spain)
Jesús Sánchez Cuadrado
Universidad de Murcia (Spain)
Esther Guerra Towards effective mutation testing for ATL MoDELS 2019 1 / 30
miso.es

Introduction
Model transformations
Importance of correctness of model transformations
Testing of transformations helps in detecting errors
How do you know if you did enough tests?
Mutation testing to the rescue!
It permits quantifying the quality of a test suite
Based on injecting artiﬁcial faults in the program under test
If test suite detects the artiﬁcial faults, it will likely detect real errors
Our focus is on mutation testing for ATL

Introduction
Mutation testing
Input
ATL transformation to test
Test suite made of:
input models (manual vs automatic)
oracle function (partial vs total)
Challenge: Identify the best test input
model generation technique
good at ﬁnding errors
few models if possible
test cases
program
under test
INPUT
1

Introduction
Mutation testing
Mutation operators
Mimic errors of competent developers
Used to create mutants of the program
Operators for ATL exist, but do not
mimic real errors
Challenge: Design operators based on
errors made by ATL developers
mutant
programs
mutation
operators
test cases
2
program
under test
INPUT
2

Introduction
Mutation testing
Mutation score
For each mutant, execute test suite
against mutant and original program
Compare results
if diﬀerent, the mutant is killed
otherwise, it is alive (undetected error)
Mutation score =
killed mutants / total mutants
The higher the score, the better the
quality of the test suite
mutant
programs
mutation
operators
live mutants killed mutants
mutation score
test cases
yes no
3
program
under test
INPUT
mutant test
==
original test

Introduction
Mutation testing
On eﬀectiveness
Mutation testing is very expensive
(many potential mutants)
Careful selection of mutation operators
do not produce trivial mutants
produce hard-to-kill mutants
Challenge: Analyse the eﬀectiveness of
operators for ATL, which is unknown
mutant
programs
mutation
operators
mutation score
test cases
yes no
3
program
under test
INPUT
mutant test
==
original test

Introduction
Mutation testing
Improving the test suite
Add input models that kill live mutants
Challenge: Devise a technique to
synthesize mutant-killer models for ATL
mutant
programs
mutation
operators
mutation score
additional
test cases
test cases
yes no
program
under test
INPUT
mutant test
==
original test
4

Contributions
Mutation operators for ATL
new operators that mimic frequent ATL developer errors
evaluation of eﬃcacy of these and other operators
Test suite
evaluation of eﬃcacy of three test model generation techniques:
random, meta-model coverage, transformation coverage
novel technique to generate input models that kill live mutants
Open-source tool for mutation testing of ATL

A Brief Tour
of
ATL

ATL
Atlas Transformation Language
Widely used model transformation language
Dynamic, testing is needed to avoid runtime errors

ATL by example
create OUT : Relational from IN : Class
helper context Class!Attribute def: multiValued : Boolean =
if self.upperBound = −1 then true
else self.upperBound > 1 endif;
rule Class2Table {
from c : Class!Class ( not c.isAbstract )
to out : Relational!Table (
name <− c.name,
col <− Sequence{key}→union(c.att→select(e | not e.multiValued)),
key <− Set{key} ),
key : Relational!Column ( name <− ’objectId’ )
}
rule MultiValuedDataTypeAttribute2Column {
from a : Class!Attribute ( a.type.oclIsKindOf(Class!DataType) and a.multiValued )
name <− a.owner.nameOrEmpty + ’ ’ + a.name,
col <− Sequence {thisModule.createIdColumn(a.owner), value} ),
value : Relational!Column ( name <− a.name )
}
lazy rule createIdColumn {
from ne : Class!NamedElt
to key : Relational!Column ( name <− ne.name )
}

ATL by example
create OUT : Relational from IN : Class // input and output meta−models
rule Class2Table {
name <− c.name,
}
}
}

ATL by example
rule Class2Table { // matched rule
from c : Class!Class ( not c.isAbstract ) // input pattern with ﬁlter
to out : Relational!Table ( // output pattern
name <− c.name,
}
}
}

ATL by example
rule Class2Table {
name <− c.name, // binding
col <− Sequence{key}→union(c.att→select(e | not e.multiValued)), // binding
}
}
}

ATL by example
helper context Class!Attribute def: multiValued : Boolean = // helper, similar to a function
rule Class2Table {
name <− c.name,
col <− Sequence{key}→union(c.att→select(e | not e.multiValued)), // helper invocation
}
}
}

Mutation testing for ATL
rule Class2Table {
from c : Class!Class (not c.isAbstract)
to out : Relational!Table … }
CFCP
:Class
isAbstract=true
name=‘Person’
original transformation mutant transformation
Min
Mout M’out

rule Class2Table {
from c : Class!Class (not c.isAbstract
and c.name=‘’)
to out : Relational!Table … }
:Class
isAbstract=false
name=‘Female’
:supers
:Table
name=‘Female’

Mutation Operators
for
ATL

Syntactic operators
Create-update-delete operations on language elements
Troya et al. [1]: 18 operators for main elements of ATL meta-model
Concept Operator
Matched rule addition
deletion
name change
In and out pattern element addition
deletion
class change
name change
Filter addition
deletion
condition change
Binding addition
deletion
value change
feature change
1
Troya et al., Towards systematic mutations for and with ATL model transformations, ICST Workshops, 2015, pp. 1-10

Semantic operators
Operations mimic faults a developer may incur, based on authors’
experience not on empirical evidence
Mottu et al. [1,2]: 10 operators which are language-independent
Navigation operators
Relation to the same class change (RSCC) Replaces navigation by another to the same class
Relation to another class change (ROCC) Replaces navigation by another to a different class
Relation sequence modification with deletion (RSMD) Removes last step of a navigation sequence
Relation sequence modification with addition (RSMA) Adds navigation step in a navigation sequence
Filter operators
Collection filtering change with perturbation (CFCP) Modifies filter, e.g., acting on property or type of class
Collection filtering change with deletion (CFCD) Deletes filter
Collection filtering change with addition (CFCA) Adds filter, e.g., returning an arbitrary element
Creation operators
Class compatible creation replacement (CCCR) Replaces type of created object by a compatible one
Classes association creation deletion (CACD) Deletes creation of association
Classes association creation addition (CACA) Adds relation between two target objects
1
Mottu et al., Mutation analysis testing for model transformations, ECMDA-FA, LNCS 4066, Springer, 2006, pp. 376-390
2
Aranega et al., Towards an automation of the mutation analysis dedicated to model transformations, STVR 25(5-7), 2014

Typing operators
Operators introduce typing errors or force runtime errors
Sánchez et al. [1]: 27 operators
used to test a static analyzer for ATL
coverage of ATL mm + operators for programming languages
Type Targets
Creation binding
source/target pattern element
rule inheritance relation
Deletion rule, helper, binding
source/target pattern element
rule filter, rule inheritance relation ..............
Type modification type of source/target pattern element
helper context type, helper return type
type of variable, collection or parameter ..............
Feature name modification navigation expression, target of binding
Operation name modification predefined operator (e.g., and) or operation (e.g., size)
collection operation (e.g., includes), iterator (e.g., exists, collect)
operation/helper invocation
1
Sánchez et al., Static analysis of model transformations, IEEE TSE 43(9), 2017, pp. 868-897

Operators based on errors made in practice (zoo)
Operators emulate errors made by competent developers
Zoo set (new!!!): 7 operators emulating the 5 most frequent typing
errors in the ATL zoo [1]
Error Frequency Operator
No binding for compulsory target feature 48.8% Remove binding of compulsory feature (RBCF)
Invalid actual parameter type 11.9% Replace helper call parameter (RHCP)
Feature access over possibly undefined receptor 11.22% Remove enclosing conditional (REC)
Add navigation after optional feature (ANAOF)
Feature found in subtype 3.75% Replace feature access by subtype feature (RSF)
Binding possibly unresolved 3.7% Restrict rule filter (RRF)
Delete rule (DR)
1
Sánchez et al., Static analysis of model transformations, IEEE TSE 43(9), 2017, pp. 868-897

Tool Support

Tool support
Java framework for mutation testing of ATL:
https://github.com/jdelara/MDETesting
Implementation of all presented mutation operators (extensible)
Operators can use transformation typing info (anATLyzer)
Generator of test models (random, mm coverage, path coverage)
Generator of mutant-killer models (explained later)
DifferentialTester
MutationRegistry Mutation
ModelGenerator
Path
Coverage
MM
Coverage Random
RSCC RSF
> 50 operators
…
mutations
*
SemanticTyping Syntactic Zoo
MutantGenerator
1
1
ATL Transformation
Mutant
KillerModel
Generator
1 mutant provider
original
1
TestDriver
0..1
1 executes
uses
Type
Information
has
EMF random
instantiator
USE Validator
model finder
AnATLyzer
static analyzer
1

Eﬃcacy of
Operators and
Test Model Generation Techniques

Evaluation of mutation operators
Experiment design
6 syntactically correct transformations from the ATL zoo
For each transformation:
create test suite with models generated by our three techniques
(random, meta-model coverage, path coverage)
create transformation mutants
compute mutation score
Overall, >32 000 mutants, >1 million executions

Applicability of operators
61% operators were applicable to all transformations
4 operators with poor applicability (i.e., frequently useless):
[typ] deletion of parent rule (1 application)
[typ] modiﬁcation of type of variable (1 application)
[zoo] deletion of enclosing conditional (1 application)
[syn] deletion of input pattern element (0 applications)

Resilience and stubbornness of mutants
overall, 88% mutants were killed
most operators produced stubborn mutants (killed by few models)
Id Type Operator Mutants %Killed mutants %Killer models
0 syn In pattern element deletion 0 - -
1 sem Classes association creation addition (CACA) 14 100.00 17.87
2 zoo Replace feature access by subtype feature 48 100.00 3.97
3 typ Parent rule deletion 21 100.00 3.72
4 typ Variable modification 48 100.00 2.08
5 sem Relation sequence modification with deletion (RSMD) 72 100.00 1.94
........
18 typ,syn In pattern element creation 3818 100.00 0.03
........
21 typ,syn Remove binding 724 99.45 0.13
........
24 zoo Remove binding of compulsory feature 260 99.23 0.36
........
52 typ Helper deletion 780 89.87 0.12
53 typ Parameter deletion 513 89.47 0.15
54 typ Parameter type modification 570 89.47 0.13
55 zoo Add navigation after optional feature 44 83.33 1.04

operators 1 to 18 only produced trivial mutants (always killed)
........
........
........
........

operators 52 to 55 produced the hardest-to-kill mutants
mutants of 52–54 were among the stubbornnest
mutants of 55 were crash-prone and not so sttuborn
........
........
........
........

operators 21 & 24 had similar resilience, but 24 created fewer mutants
similarly, matched rule deletion preferrable to rule deletion
........
........
........
........

no operator set was more eﬃcient than the others
zoo operators: 1 hard-to-kill, 3 trivial, 3 intermediate
........
........
........
........

Evaluation of test model generation techniques
Mutation score of test suites generated by:
random instantiation (50 models with 100 objects)
coverage of input meta-model
coverage of transformation execution path
Meta-model Transformation path Random
#M. typ sem syn zoo #M. typ sem syn zoo #M. typ sem syn zoo
t1 62 69.98 80.73 100.00 97.14 27 77.53 75.23 100.00 97.14 50 67.15 58.33 59.86 97.14
t2 200 87.44 31.68 45.54 30.33 18 82.47 71.66 89.31 95.08 50 65.64 56.71 48.36 23.58
t3 50 81.21 81.44 84.50 97.04 1 84.85 87.37 90.56 99.26 50 64.18 81.05 69.01 95.56
t4 50 98.71 68.52 86.17 100.00 92 98.14 98.60 96.33 100.00 50 99.22 98.33 95.74 100.00
t5 710 73.48 82.05 73.37 92.16 24 76.83 84.62 81.41 92.16 50 15.67 65.81 22.22 78.00
t6 26 75.91 72.84 60.96 87.88 6 82.70 70.37 60.56 87.88 50 21.60 19.28 23.51 45.45
Transformation path is the best option
produced the highest-quality test suite more often
produced the smallest test suites (less testing time)

Synthesis
of
Mutant-Killer Models

Synthesis of mutant-killer models
Method (intuition)
if self.upperBound = -1 then true
false Mutation
live mutant
1) AST node of mutated code

Method (intuition)
false Mutation
live mutant
MVDataType
Attribute2Column
Class2Table
multiValued
control flow graph
upperBound = -1 / then
true false
Helper call
IF
Matched rule
Mutation
Matched rule
2) execution paths from matched rules to mutation

Method (intuition)
false Mutation
live mutant
Class.allInstances()-> -- rule input
select(c | not c.isAbstract)-> -- rule filter
exists(c | c.att->exists(e |
not e.upperBound = -1)) -- inlining of helper
or
Attribute.allInstances()->
exists(a | a.upperBound = -1)
OCL path condition
MVDataType
Attribute2Column
Class2Table
multiValued
control flow graph
true false
Helper call
IF
Matched rule
Mutation
Matched rule
3) requirements for a test model to reach the mutated code

Method (intuition)
model finding
false Mutation
live mutant
or
OCL path condition
: Class
isAbstract = false
: Attribute
upperBound = -1
att
input test model
MVDataType
Attribute2Column
Class2Table
multiValued
control flow graph
true false
Helper call
IF
Matched rule
Mutation
Matched rule
4) synthesis of input test model by model ﬁnding

Method (intuition)
model finding
false Mutation
live mutant
or
OCL path condition
: Class
isAbstract = false
: Attribute
upperBound = -1
att
input test model
MVDataType
Attribute2Column
Class2Table
multiValued
control flow graph
true false
Helper call
IF
Matched rule
Mutation
Matched rule
However, reachability is necessary but not suﬃcient...

Evaluation of eﬃcacy
For each mutant:
generate a killer test model
execute mutant and original transformation with the test model
compare the results
typ sem syn zoo
Mutants %Killed Mutants %Killed Mutants %Killed Mutants %Killed
t1 266 93.94 58 96.55 187 87.57 15 100.00
t2 2200 97.14 550 83.82 4045 97.11 60 81.67
t3 151 93.10 32 100.00 175 94.08 21 95.24
t4 3161 82.76 649 71.19 5993 78.66 65 73.85
t5 382 92.25 75 94.67 363 86.76 41 92.68
t6 153 59.48 42 54.76 143 58.04 16 43.75
Overall, 85% mutants killed
By cases, killed mutants >90% in 12/24 (in green)
Altogether, reasonable good results

Wrap-up

Wrap-up
Today’s presentation
Analyse some steps in mutation testing for ATL
mutation operators
test model generation techniques
synthesis of mutant-killer models
to make it more eﬀective
(and ATL transformations less buggy)

Wrap-up
Today’s presentation
Analyse some steps in mutation testing for ATL
mutation operators
test model generation techniques
synthesis of mutant-killer models
to make it more eﬀective
(and ATL transformations less buggy)
Future plans
Replica of evaluation with partial oracles
Method to detect equivalent mutants for ATL
Improve synthesis of mutant-killer models

Towards effective mutation testing for ATL
Esther Guerra, Juan de Lara
Universidad Autónoma de Madrid (Spain)
Jesús Sánchez Cuadrado
Universidad de Murcia (Spain)
esther.guerra@uam.es
Questions?

mtATL

More Related Content

More from miso_uam (20)

Recently uploaded (20)

mtATL