Model-based mutation testing-Approach and case studies

Contents lists available atScienceDirect

Science

of

Computer

Programming

www.elsevier.com/locate/scico

Model-based

mutation

testing—Approach

and

case

studies

✩

Fevzi Belli

a

,

Christof

J. Budnik

b

,

Axel Hollmann

c

,

Tugkan Tuglular

d

,

∗

,

W. Eric Wong

e

a_{DepartmentofComputerScience,ElectricalEngineeringandMathematics,UniversityofPaderborn,Germany} b_{SiemensCorporation,CorporateTechnology,USA}

c_{andagonGmbH,Cologne,Germany}

d_{DepartmentofComputerEngineering,IzmirInstituteofTechnology,Turkey} e_{DepartmentofComputerScience,UniversityofTexasatDallas,USA}

a

r

t

i

c

l

e

i

n

f

o

a

b

s

t

r

a

c

t

Articlehistory: Received 19 July 2014

Received in revised form 7 December 2015 Accepted 14 January 2016

Available online 21 January 2016 Keywords:

Mutation testing Model-based testing Model-based mutation testing Mutation operator

Fault detection capability

This paper rigorously introduces the concept of model-basedmutation testing (MBMT) andpositionsitinthelandscapeofmutationtesting.Twoelementarymutationoperators,

insertion and omission,are exemplarilyapplied toahierarchyof graph-basedmodels of increasingexpressivepowerincludingdirectedgraphs,eventsequencegraphs,finite-state machinesandstatecharts.Testcasesgeneratedbasedonthemutatedmodels(mutants)are used todetermine not onlywhethereach mutantcan be killedbutalsowhetherthere areanyfaultsinthecorrespondingsystemunderconsideration(SUC)developedbasedon the original model.Novelties of ourapproach are: (1) evaluationof thefault detection capability(in termsofrevealingfaults in theSUC) oftest sets generated basedonthe mutatedmodels,and(2) supersedingofthegreatvariety ofexistingmutationoperators by iterations and combinations of the two proposed elementary operators. Three case studies wereconducted on industrial and commercial real-life systems to demonstrate the feasibilityofusingthe proposedMBMTapproachin detectingfaults inSUC, and to analyzeitscharacteristic features.Ourexperimentaldatasuggest thattestsetsgenerated basedonthemutatedmodelscreatedbyinsertion operatorsaremoreeffectiveinrevealing faults inSUCthanthose generatedbyomission operators.Worthnotingis thattestsets followingtheMBMTapproachwereabletodetectfaultsinthesystemsthatweretested bymanufacturersandindependenttestingorganizationsbeforetheywerereleased.

©2016ElsevierB.V.All rights reserved.

1. Introduction

Testingisoneofthemostcommonlyusedtechniquesforassuringqualityinthesoftwareindustry.Itentailsthe execu-tionofthesoftware1initsrealenvironment,underreal-lifeconditions.Theinitialstepofsoftwaredevelopmentisusually therequirementselicitation, andits outcomeisthespecificationofthesystem’sbehavior.It isimportanttogeneratetest cases anddefine appropriate testing processesat thisearly stage, long before theimplementation begins,in compliance

✩ _{Authors are listed in the alphabetical order of their last names.}

*

Corresponding author.

E-mailaddresses:[email protected](F. Belli), [email protected](C.J. Budnik), [email protected](A. Hollmann), [email protected](T. Tuglular), [email protected](W.E. Wong).

1 _{The terms “program,” “software,” “application,” and “implementation” are used interchangeably, whenever appropriate.} http://dx.doi.org/10.1016/j.scico.2016.01.003

withtheuser’sexpectancyofhowthesystemshouldbehave.Anadvantageofthisistoavoidfaultsbeingdetectedatlate stages,whentheycouldhavebeendetected muchearlierandfixedlessexpensively.There aretechniquestovisualizeand representtherelevantfeaturesofthesystemunderconsideration(SUC)initsenvironmentalcontext,leadingtoamodel of

theSUC.Thesemodelsareincreasinglybeingusedasabasis(inthecontextofmodel-basedtesting(MBT))togeneratetest cases,whichsatisfycertaincriteria,forexample,coverageofeverynodeinthemodel.Thesetestsarethenusedtovalidate theSUC.Theassumptionisthatthemodeliscorrect,andifthebehaviorofSUCisconsistentwiththatofthemodelduring conformancetestingagainsttestcasesgeneratedabove,thentheSUCmustalsobecorrect.

A broadvarietyof formalor informalmodels existformodeling softwareasrecommendedinstandardssuch asUML

[1] orTTCN-3 [2]. Thesemodels describe theSUC atdifferentlevels of granularityand preciseness.Graph-basedmodels consist of nodesandarcs.The semanticmeaning associatedwiththesenodes andarcsdetermines thelevel ofprecision oftheSUCdescription.Theissueofwhichmodeltouseisnotdeterminedsolelybyidentifyingwhichismostsuitablefor modelingtheSUC,butalsobytakingintoaccountotherfactors(especiallywhenconsideringtheproposedMBMTapproach inSection3)suchashoweasilythemodelcanbemutatedwithrespecttocertain mutationoperatorsandhowtestcases can begeneratedforthemodel.Inthispaper,wechoosethe followingformalgraph-basedmodelsinorderofincreasing expressive power:(i)Directedgraphs(DG)includenosemantics;thatis,nodesandarcshavenointerpretation.Thepaper usesthemtointroducenotions,especiallymutationoperators,syntactically.(ii)Eventsequencegraphs(ESG)interpretnodes aseventsandarcsassequencesofevents[3].(iii)Finite-statemachines(FSM)interpretnodesasstates,whereasarcslabeled byeventssymbolizingstatetransitions.(iv)Statecharts(SC)considerconcurrencyandhierarchyaspects[4,1].

A fundamental problemof testingstems from the huge variety ofpossible software faults to be considered. For this purpose,mutationtesting techniquesgeneratefaultyversionsofasoftwaresystem(namely,mutants)byintroducingsimple butrepresentativefaults usingmutationoperators(which arealsoknownasmutantoperators intheliterature).Different strategies havebeenproposed formutationtesting. Someare appliedtosourcecodeofthe SUCwhileothersareapplied tomodelsderivedfromspecifications.Mutantsgeneratedcanbeusedattheunittesting[5,6],aswellasattheintegration testinglevel[7](furtherdetailsonrelatedworkaregiveninSection6).Acommonproblemofthesestudiesisthattheyuse alargenumberofmutationoperatorstogeneratemutantsbecausetheyempirically determinethemutationoperatorsbased onselectedfaultmodels.Thismakesitarguableastotheexactnumberofoperators thatshouldbedevelopedforagiven fault model. A differentapproach is used inthis paperby introducing two elementary mutation operators: insertion and

omission. StartingwithaDGastheunderlyingmodel,theelementaryoperators areappliedtoother models,suchasESG,

FSM,andSCinthesamemanner.Thisisalsoamajordifferencebetweenthispaperandotherstudies,suchas[7–16].Many mutation operatorsknownfromliterature canbe reducedto thesetwoelementaryoperators andtheircombinationsand iterations.Forexample,the“sdl”operatorintheMothratoolset[17]deletesastatementthatisequivalenttoan“omission” operator,andthe“svr”operatordoesascalarvariablereplacementthatisan“omission”followedbyan“insertion.”Infact, mutantsgeneratedbythemutationoperatorsintheaforementionedliteraturecouldbeviewedasspecialcasesofthefault modeldevelopedinthispaper.However,mutantsgeneratedbythetwoelementaryoperators(withmultipleiterations)and their combinationspresentedinthispapermightnot begeneratedbytheoperatorsintroduced byotherauthors.Referto Section4.4formoredetails.

Theprimaryobjectiveofthispaperistoproposeaprecisemodel-basedmutationtesting (MBMT)approach.Weviewthe SUCasablackboxandassumethatthesourcecodeisnotavailable.Hence,itcannotbemutated.Instead,amodel(say

M

) isavailableandwillbemutated.Wefurtherassumethattheconformancebetween

_M

andthecorrespondingSUChasalso been validatedbya testset

T

createdbyapplyingan appropriate testgeneration algorithm(say

Φ

) to

M

.Theproposed elementarymutationoperators,insertion andomission,generateasetofmutatedmodels(mutants)whenappliedto

M

.The same test generationalgorithm

Φ

which hasbeenapplied to

M

will beapplied to thesemutantsto generatea test set foreachmutant.Whileusingtestcasesfromthesesetstodeterminewhetheramutantcanbe killed,thesametestcases canhelpusdetectfaultsinSUCwhichhavenotbeenrevealedduringtheconformancetestingdescribedaboveusing

T

.In otherwords,weattempttoassessthefaultdetectioncapabilityof

T

.Thisisdefinitelyadifferentviewthantheonesfound in theliterature,e.g., [18].Note alsoincontrasttothecode-basedmutation testing,mutantsin MBMTare classifiedinto more categoriesasdescribed inother approachestoMBMT. Thelater sections,especiallySection 3,willpreciselyexplain thisnovelinterpretationofMBMT.

Three casestudieson industrialandcommercialapplicationscoveringinteractive,reactive, andproactivesystemswere conductedto demonstratethefeasibility ofusingtheproposed MBMT approachandto discussitscharacteristic features. Theresultsofourexperimentssuggestthattestsgeneratedbytheproposedapproachcanbeeffectiveindetectingfaultsin thecorrespondingSUC.Forexample,inthethirdcasestudy(thecontrolterminalofamarginalstripmower),thetestcases generatedby theapproachdescribedherecoulddetectfaultsinthereleasedversion thatwerenot foundbytheprevious (andindependent)testers,suchasatechnicalcontrolboardinGermany.

Theremainderofthepaperisorganizedasfollows.Section2summarizesMBTandreviewsthemodelsandtest gener-ation processesselectedforourstudy.Section 3discussesMBMT andexplainsthedifferencebetweenthisnewapproach andthecode-basedmutationtesting.Section 4definestheelementarymutationoperators andcomparesthemwithother mutation operators reported in the literature. Section 5 presents case studies including results, analysis and discussion. Section6givesrelatedwork.OurconclusionandfutureworkappearinSection7.

Fig. 1. A sample ESG.

2. Modelingandmodel-basedtesting

In modeling and model-basedtesting (MBT), features of an SUC are described by a model

M

that is constructed in accordancewiththespecification(notethatspecification-basedtestingdoesnotnecessarilyrequireamodel).Asmentioned in theintroduction, it is assumedthat

M

iscorrect, andits correctness hasbeen assured, forexample, by usingmodel checking techniques [19] with respect to some properties derived fromthe requirements. A testgenerationalgorithm

Φ

usingatestselectioncriterion[20],forexample,coveringeverynodeif

M

isgraph-based,isappliedto

M

suchthat

Φ(

M )

generates a testset

T

= {

t1, t2,

. . . ,

tn

}

with each ti (i

=

1

,

. . . ,

n) being a testcase as an ordered pair ti

=

(inputtothe

SUC,expectedoutputfromtheSUC)

∈ T

.IfnofailuresareobservedwhenSUCisexecuted against

T

,weconcludethatSUC

is consistent with

M

(with respect to

T

). Since we assume

M

iscorrect, we may consequently assume SUC is correct. However,thisassumptionmaynotextendtotestcasesthatarenotin

T

.Hence,eveniftheexecutionof

T

doesnotreveal anyfaultsinSUC,itdoesnotimplySUCiscorrect.Thisleadstoanimportantquestion:howgoodis

T

intermsofdetecting faults? To answer thisquestion, we need to have a way to assessthe fault detectioncapability of

T

. To exemplify our approach,asummaryofourmodelsispresentedbelow.

Definition1.AdirectedgraphDiGraph

=

(N, A)isanorderedpairwhereN isafinitesetofnodesandA

⊆

N

×

N isasetof arcs.

InDefinition 1,thenodesandarcsdonothaveanysemantics.

2.1. Eventsequencegraphs

Forexemplifyingmodeling,model-basedtesting,andmodelmutation,weinterpretthenodesofaDGaseventsandarcs assequences ofevents leading toeventsequencegraphs (ESG) [3].Generally,an event represents an externally observable phenomenon,suchasanenvironmentalorauserstimulus,orasystemresponsepunctuatingdifferentstagesofthesystem activityofanSUC,whichisassumedtobedeterministic.ESGarecommonlyusedforanalysisandvalidationofuserinterface requirementsandaresimilar totheconcept ofeventflow graphs[21].WechooseESG becauseitintensivelyuses formal graph-theoreticalnotionsandalgorithms[22].

Definition2.Aneventsequencegraph ESG

= (

DG

,

Ξ,

Γ )

isadirectedgraph,where

•

DG

= (

E

,

A

)

isadirectedgraphasdefinedinDefinition 1withE beingafinitesetofevents and A

⊆

E

×

E beingaset ofarcs

• Ξ

(entry events)

⊆

E and

Γ

(exit events)

⊆

E aresetsofdistinguished eventswith

|Ξ|

≥

1 and

|Γ |

≥

1

The syntaxof avalid ESG requires thatevery eventhasto be reachedbyan entryandthat fromevery eventan exit hastobereached;otherwisetheESGisinvalid.ToidentifytheentryandexiteventsofanESGgraphically,every

ξ

∈ Ξ

is precededbyapseudo-event “[”

∈

/

E andevery

γ

∈ Γ

isfollowedby apseudo-event“]”

∈

/

E.Pseudo-eventrepresentedby

node“[”

∈

/

E andpseudo-event representedby node“]”

∈

/

E.Pseudo-eventsare annotationsanddonotbelong toDG.The

semanticsofthearcsofanESGareasfollows.Fortwoevents,

ψ

and

ψ



∈

E,

ψ

mustbeenabledaftertheexecutionof

ψ

ifandonlyif

(ψ,

ψ



)

∈

A.ItisassumedthataDGhastobestronglyconnected.Evenifatmostonearcin A ismissingto havethegraphstronglyconnected,thegraphisstillvalid.AnexampleofanESGisgiveninFig. 1thatmodelsasimpleCD player.

InFig. 1,the DG

= (

E

,

A

)

,whereE

= {

Play, Pause, Stop

}

and A

= {

Play, Pause

,



Play, Stop

,



Pause, Play

,



Pause, Stop

,



Stop, Play

}

.DG ofthesampleESGgiveninFig. 1isstronglyconnected.

Definition3.LetE and A bethefinitesetofeventsandarcsofanESG.Anysequenceofevents(e1,

. . . ,

ek)withei

∈

E for

i

=

1

,

. . . ,

k isaneventsequence (ES)oflengthk,if

(

ei

,

ei+1

)

∈

A fori

=

1

,

. . . ,

k

−

1.

Definition4.AnES

= (

e1,

. . . ,

ek

)

isacompleteeventsequence (CES)ife1

∈ Ξ

andek

∈ Γ

.

EachCESrepresentsawalk fromtheentryofanESGtoitsexit,realizedbyachainofuserinputsandsystemresponses. AtestcaseisanorderedpairofaninputandanexpectedoutputoftheSUC.Atestsetcancontainanynumberoftestcases.

Input: An ESG and a number n∈ N

Output: A test setT= {t1,t2, . . . ,tm}to cover all event sequences of length n. Each test case inT is a CES.

Generate a set of CES to cover all event sequences of length n in the ESG such that

m  i=1

length(ti)is minimal

Fig. 2. An ESG-based test generation process (details of the algorithms can be found in our previous work[22]).

A CESofan ESG can beused asatest caseforthecorresponding SUC.The testcasesare thereby formedby theevents, particularlybytheuseractivities(inputs)andtheexpectedsystemresponses(outputs).Atestexecutionwithrespecttoa

CES fails ifandonlyiftheCEScannot reachthecorresponding exiteventdueto afailure,such asasystemcrashorifit

reachesanexiteventbutdoesnotproducetheexpectedoutputs.

Fig. 2presentsacoverage-basedtestgenerationprocesstoproduceasetofCEScoveringalleventsequencesofarequired lengthinagivenESG.Letthelength ofeachtestcase(eachCES)bethenumberofeventsithas.Thetotallengthofatest set is the sumofthe lengthsof all its test cases. Tominimize thislength, solutions of the ChinesePostmanProblem can

be used;thatis,findingtheshortestpathorcircuitinagraphbyvisitingeacharc. Polynomialalgorithms supportingthis testgenerationprocesshavebeenpublished before,e.g.,inourpreviouswork[22].Thereasonforsuchminimizationisto reducethecostoftestexecution.

ReferringtotheESGinFig. 1,wehaveeventsequencesetsofdifferentlengths.Forexample: ES2

= {

event sequences of length 2

}

= {

(Play, Pause), (Pause, Play), (Play, Stop), (Stop, Play), (Pause, Stop)

}.

TocoverallthesequencesinES2,atestset

T

= {

(Play, Pause, Play, Pause, Stop, Play, Stop)

}

withoneCESisgeneratedusingtheprocessinFig. 2.

_T

hasatotallengthof7,whichistheminimumamongallthetest setsthatcancoverthefiveeventsequencesinES2.Similarly,wehave

ES3

= {

event sequences of length 3

}

= {

(Play, Stop, Play), (Play, Pause, Play), (Play, Pause, Stop), (Pause, Play, Stop), (Pause, Play, Pause)

,

(Pause, Stop, Play), (Stop, Play, Stop), (Stop, Play, Pause)

}.

TocoverallthesequencesinES3,anothertestset

T

isgeneratedwithtwoCESasfollows

T



= {

(Play, Pause, Play, Pause, Stop, Play, Pause, Stop), (Play, Stop, Play, Stop, Play, Pause, Play, Stop)

}

T

hasatotallengthof16

(

=

8

+

8

)

,whichistheminimumamongallthetestsetsthatcancoveralltheeventsequences inES3.

2.2. Finite-statemachinesandstatecharts

Tomodelmorecomplexsystems usingstates,hierarchy,concurrency, etc.,we exemplary2 _{chooseFSMandstatecharts.}

Based on the notions of DG,statecharts [4] are definedasfollows. Notethat the following definitionof statecharts also includesthedefinitionofFSM.

Definition5.Abasicstatechart isgivenbySC

= (

FSM

,

H

,

g

)

,where

•

FSM

= (

DG

,

E

,

f

,

Ξ,

Γ )

isafinite-statemachine

– DG

= (

S

,

TR

)

asdefinedinDefinition 1withS beingasetofstatesandTR

⊆

S

×

S beingasetoftransitions – E isafinitesetofevents

– f

:

TR

→

E isafunction,mappinganeventtoeachtransition

–

Ξ

⊆

S and

Γ

⊆

S arefinitesetsofinitialandfinalstates

•

H

⊆

S

×

S isahierarchyrelation

•

g

:

S

→ {

Ssimple

,

Sand

,

Sxor

}

isatotallabelingfunction

AnFSMorSCcanbeviewedasa DGwithsemanticallydistinguishedstates(nodes)andstatetransitions(arcs).Nodes represent states.Arcs, labeled byevents, aretransitions betweenstates.In ourstudy, onlydeterministic FSMandSC are considered. Wealso requirethat each statecan be reachedfromthe initialstate andafinal state isreachablefromeach state.Theset H isusedtorepresentthehierarchyrelationbetweenstatesinstatecharts.Eachstate s

∈

S isidentifiedasa

Input: An FSM or SC and a number n∈ N

Output: A test setT= {t1, . . . ,tm}to cover all transition sequences of length n. Each test case inT is a CTS.

Generate a set of CTS to cover all transition sequences of length n in the FSM or SC such that

m  i=1

length(ti)is minimal.

Map each CTS to the corresponding CES

Fig. 3. An FSM and SC-based test generation process (for more details see[4]).

Fig. 4. Code-based mutation testing.

simplestate(Ssimple)oracompositestate,andthelatterisfurtherclassifiedasanAND-state(Sand)oranXOR-state(Sxor). The sub-statesofan AND-stateare calledregions.It meansthat thestatechart residesconcurrentlyin eachregion ofthe AND-state.

Definition6.Asequenceoftransitions(tr1,

. . . ,

trk) withtri

∈

TR fori

=

1

,

. . . ,

k (whereTR isdefinedinDefinition 5)isa

transitionsequence (TS)oflengthk,if(tri

= (

sα

,

sβ

)

,tri+1

= (

sγ

,

sδ

)

) denotesa pairoftransitionsfori

=

1

,

. . . ,

k

−

1 with

sβ

=

sγ .

Definition7.ATS

= (

tr1,

. . . ,

trk

)

isacompletetransitionsequence (CTS)ifitstartsattheinitialandendsatafinalstate.

Fig. 3gives a coverage-based test generation process for FSMandSC to generatea set of CTS coveringall transition sequencesofarequiredlength.SimilartotheprocessforESG(seeFig. 2),eachCTScanbeusedasatestcaseandthetotal lengthofalltheseCTSisminimizedtoreducethecostoftestexecution.Detailedalgorithmscanbefoundin[4].Notethat eachCTS

= (

tr1,

. . . ,

trk

)

ismappedtoacorrespondingCES

= (

e1,

. . . ,

ek

)

whereei

=

f

(

tri

)

fori

=

1

,

. . . ,

k.

3. Model-basedmutationtesting(MBMT)

This section discusses the approach for conducting MBMT, followed by a discussion ofhow to classify (Section 3.1), generate,andexecute(Section3.2)mutants(namely,mutatedmodels).

Mutation testing, asintroduced in the late seventies of the last century,is a fault injection-based white-boxtesting technique [23–25] (seeFig. 4).It relieson twohypotheses: the competentprogrammerhypothesis[26,27] andthecoupling

effect[28].Thecompetentprogrammerhypothesissuggeststhatexperiencedprogrammerstendtowriteprogramsthatare

“close”to beingcorrect.Thatis,althougha programwritten bya competentprogrammer maybe incorrect, itwilldiffer from the correct version by only relatively simple faults in terms of their syntax and semantics. The taskof testing is thereforereducedtovalidatingaprogramthat isprobablynotcorrectbutisclosetothecorrectone.Thecouplingeffect assumesthatatestsetthatdetectsallsimplefaultsinaprogramisalsolikelytodetectmorecomplexfaults[28].Mutants generatedby introducingexactly onechange intoa programare knownasfirst-order mutants[28].Second-order mutants

aregeneratedbymakingtwosimplechanges.Mutantswithmorethanonechangearealsoknownashigher-order mutants.

Accordingtothecouplingeffect,higher-ordermutantsarelikelytobedetectedbytestcasesthatdetectfirst-ordermutants. Ingeneral,therefore,onlyfirst-ordermutantsaregeneratedandusedinmutationtesting.

Forexplanatorypurposes,assumethat aprogram

_P

anda testset

_T

aregiven. Asetofmutationoperators X isthen appliedto

P

togeneratemutantsbyintroducingoneormoresyntacticalchangesinto

P

.Basedonthecouplingeffect,the focusisonthefirst-ordermutants.Amutantiskilled (“distinguished”)bya testcaset in

T

iftheobservedbehaviorof

P

differsfromthatofthemutantwhenexecutedagainstt.Amutantisequivalent to

P

ifitalwaysbehavesthesameas

P

for everycase(i.e.,every possibleinput).Thetestset

T

issaidtobemutationadequate[20]withrespectto

P

and X ifevery non-equivalentmutantof

P

generatedbyapplyingthemutationoperatorsin X canbekilledbyatleastonetestcasein

T

.

Fig. 5. Model-Based Mutation Testing.

Mutationtestinghasalsobeenextendedtovalidateaspecification S.Todothis,asetofmutationoperatorsisappliedto S to generatemutatedspecifications(S∗).The adequacyof

T

ismeasured byhowmanynon-equivalentS∗ canbe killedby test casesin

T

.Fig. 4depictstheabove discussion.Such mutationtesting(hereafterreferredto as“code-basedmutation testing”)isnotalwaysfeasiblebecause,forexample,theprogramsourcemaynotbeavailable.

TheMBMTproposedinthispaperassumesthat(i)theconformanceof

M

andthecorresponding SUChasalreadybeen validated a priori by model-basedtesting (MBT) usinga test set

T

,and (ii) there maybe still latent faults inSUC that were notrevealedby

T

.ReferringtoFig. 5,thesolidlinesarefortheMBTandthedashed linesforthefollow-on MBMT.

Let the setofmutationoperators be definedas X

= {

x1,

. . . ,

xi

,

. . . ,

xk

}

. Applying an operator xi to

M

(denoted by xi

(M )

)

generates asetofmutatedmodels

M

i∗

= {M

i1∗

,

M

i2∗

,

. . . ,

M

i j∗

,

. . . ,

M

ip∗

}

,where p

≥

1.These mutantsarecalledmodelmutants of

M

.Forthe restofthepaper, they aresimply referred to asmutants whenthere isno ambiguity.From each

M

i j∗ (the

jth mutantgenerated by the ith operator), a test set

T

i j∗ is generated based on a test generation algorithm

Φ

, that is,

Φ(

M

i j∗

)

= T

i j∗

= {

t∗i j1

,

ti j2∗

,

. . . ,

ti jn∗

}

.Forexplanatorypurposes, weuse

M

∗ and

T

∗ asa generic representationforamutant generated from

M

anda testset withrespect to

T

_{i j}∗,respectively. TheobjectiveofMBMT istouseall the

T

∗ tohelpus detect possible additional bugs inSUC which cannot be revealedpreviously by

T

(generated by applying the same test

generation algorithm

Φ

to themodel

M

). Notethat weassume the competentprogrammer hypothesis andthecoupling

effectalsoapplytoMBMT.Thisisrecognizedasapotentialthreattothevalidityofourapproach(seeSection5.4).

3.1. MutantclassificationforMBMT

Thefollowingexplainshowtodeterminewhether

M

∗ isavalidoraninvalidmutantof

M

,andwhether

M

∗hasbeen

killed orisstilllive withrespecttotestcasesin

T

∗,orwhetherornot

M

∗isequivalentto

M

.

Definition8.valid

(

M

∗

)

:= {M

∗

_|M

∗_{satisfies the syntactic and semantic requirements imposed by the model type of}

_M

_}

_.

Forexample,if

M

isan FSM, then

M

∗ (amutatedmodel)must alsobea validFSM.Asyntactical requirementforan FSMisthat all transitionsmustbeassociated withstates.Semanticsmight requirethat an FSMmust bedeterministic or thatallstatesarereachablefromaninitialstate.

Definition9.killed

(M

∗

,

SUC

,

Φ)

:=

valid

(M

∗

)

∧ ∃

t∗

∈ Φ(M

∗

),

Output

(

SUC

,

t∗

)

=

ExpectedOutput

(

t∗

)

.

Ifthereisatleastonetestcaset∗ in

T

∗ such thattheoutputs of

M

∗ andSUCaredifferent(which isalsotheoutput of

M

astheconformancebetweenSUC and

M

hasbeenvalidatedalreadyby

T

= Φ(M )

asshowninFig. 5), then

M

∗ is killedbyt∗.NotethatExpectedOutput (t∗)istheoutputof

M

∗ ont∗.

Definition10.equivalent

(

M ,

M

∗

)

:= ∀

t intheinputdomainof

M

,Output

(

M ,

t

)

=

Output

(

M

∗

,

t

)

.

A mutant

_M

∗ is equivalent to

_M

ifandonlyifonevery elementoftheir inputdomain

_M

∗ and

_M

producethesame outputs.

Fig. 6. Mutant classification (arcs represent “or”).

Amutant

M

∗ islive withrespectto

T

∗

= Φ(M

∗

₎

_{ifandonlyifnotestcaset}∗_in

_T

∗_cankill

_M

∗_.

With respect to the code-based mutationtesting in Fig. 4, a non-equivalentmutant is either killed (i.e., the injected fault in

P

∗ isrevealed) orlive (i.e.,the faultin

P

∗ cannot be detected)with respectto agiven testset

T

. However,the classification ofmutantsforMBMTis muchmorecomplicated. Forexample,atest caset∗ killingamutant

M

∗ doesnot necessarilyimply thatt∗ detects thefault injected into

M

∗. Basedon Definition 9,

M

∗ is killedbecause Output(SUC,t∗) differsfromExpectedOutput (t∗).However,suchdifferencemaybeduetoafaultinSUCwhichwasnotdetectedpreviously whenwe validatedtheconformancebetween

M

andSUCusingthetestset

T

= Φ(M )

.Underthiscondition,t∗ helpsus findafaultinSUCwhichhasnotbeendetectedby

T

insteadofrevealingtheinjectedfaultin

M

∗.Fig. 6givesthecomplete classification.

Amutant

M

∗canbelive foroneofthefollowingreasons.First,itispossiblethat

M

∗ isequivalentto

M

(seeBox(a)in

Fig. 6).Second(seeBox(b)),

T

∗isunabletodetectthepreviouslyinjectedfaultin

M

∗.Therearetwopossiblereasons.Test casesin

T

∗donotexecutethemutatedpartof

M

∗ortheexpectedoutputof

T

∗ isunabletorevealthedifferencebetween

M

and

M

∗.Third(seeBox(c)),itispossiblethatSUCbehavesthesameas

M

∗ (whichisnotequivalentto

M

)withrespect totest casesin

_T

∗.Thisindicates thattheimplementationofSUCdeviatesfrommodel

_M

eventhoughaconformanceis establishedbetweenSUCand

_M

withrespectto

_T

= Φ(M )

.Thisalsoimpliesthat SUCcontains“code” whichintroduces some “unexpected” behavior –differentfrom thatspecified by

_M

.This helpstestersdetect additionalfaults inSUC that have not beenrevealed by

T

= Φ(M )

. Note that forthe code-based mutation testing, we only havethe first two cases (namely,Boxes(a)and(b)),butnotthethirdcase(Box(c)).

When a mutant

M

∗ is killed by

T

∗, then SUC behaves differently from

M

∗ on at least one test case t∗

∈ T

∗ (see

Definition 9).Therearethreepossibilities.First(seeBox(d)),theinjectedfaultin

M

∗isnotinSUC.WhenSUCisexecuted againstt∗,itdoesnot show certainfaultybehaviorof

M

∗.Second (see Box(e)), SUChassomefaults that arenot in

M

∗

norin

M

.ThisispossiblebecausetheconformancebetweenSUCand

M

isonlyvalidatedby

T

= Φ(M )

ratherthanevery possibleinput.Asaresult,faultsinSUCmaycauseitsexecutionagainstatestcaset∗toproduceanoutputdifferingfrom theexpectedoutputof

M

∗,whichdoesnotcontainthesamefault.Third (seeBox(f)),theinjected faultin

M

∗ isalsoin SUC,butnotin

M

.Thisispossibleforthesamereasondescribedabove. However,althoughthe faultisthesameinSUC and

M

∗,itwillmanifestindifferentways, whichmakes theoutputofSUCdifferfromtheexpectedoutput of

M

∗ forat leastone testcaset∗

∈ T

∗

= Φ(M

∗

)

.Notethat forthe code-basedmutationtesting, weonlyhavethe firstcase(namely, Box(d)),butnotthelasttwocase(Boxes(e)and(f)).

Interpretationofwhetheralivemutantfoundabugornotisperformedatline9ofAlgorithm 1,whereOutput(SUC,t∗) ischeckedforbeingequaltoExpectedOutput(t∗).IfthatcheckisTRUEforallt∗

∈ T

_{i j}∗,then

M

_{i j}∗ isalivemutant.Thenthe classificationtreeinFig. 6isused.

_M

_{i j}∗ ischeckedforequivalenceto

_M

.

1. If theyare equivalent,then (sinceweassumedthat SUCbehaves thesameas

M

)we concludethatSUC behavesthe sameas

M

i j∗.Thatmeanstestset

T

i j∗ didnotfindanewfaultinSUC.

Algorithm1Mutantgeneration,executionandclassification.

Input: ModelMthat describes SUC Set of mutation operators X= {x1, . . . ,xk}

Test generation algorithmΦ

Output: A list of killed and live mutants

1 foreach xi∈X

2 { M∗

i =xi(M ) // apply the ith mutation operator xitoM

3 foreachMi j∗∈ Mi∗

4 { if valid(Mi j∗) then // check the validity of each mutant

5 { Ti j∗:= Φ(Mi j∗) // test case generation

6 killed:=false

7 foreach t∗∈ T∗

i j

8 { Execute SUC against t∗

9 if Output(SUC,t∗)=ExpectedOutput(t∗)then

10 { AddMi j∗to the list of killed mutants

11 killed:=true

12 continue // as soon as a test case t∗kills the mutantMi j∗,

13 // the remaining test cases inTi j∗are executed.

14 }

15 }

16 if killed=false then

17 AddMi j∗to the list of live mutants

18 }

19 }

20 }

i. SUCbehavesthesameas

_M

_{i j}∗,whichis(c)leafofclassificationtreeinFig. 6.

ii.

T

∗ isunable todetectthepreviouslyinjectedfaultin

M

∗.Therearetwo possiblereasons.Testcasesin

T

∗ donot executethemutatedpartof

M

∗ortheexpectedoutputof

T

∗isunabletorevealthedifferencebetween

M

and

M

∗, whichis(b)leafofclassificationtreeinFig. 6.

To determine which of the above two possibilities, i.e. (2.i) and (2.ii), occurred,

M

is tested with test set

T

i j∗. If Output(

M ,

t∗) beingequal to ExpectedOutput(t∗) is TRUE forall t∗

∈ T

_{i j}∗, then (sincewe assumed that SUC behaves the sameas

M

) SUCbehavesthesameas

M

i j∗.IfOutput(

M ,

t∗)beingequaltoExpectedOutput(t∗)isnotTRUEforallt∗

∈ T

i j∗, thenSUCbehavesdifferentthan

M

_{i j}∗.

In summary, Boxes (a), (b),and (d) are the code-basedinterpretation ofkilled and live mutants with respect to

M

∗

and

T

∗.Ontheother hand,Boxes(c),(e),and(f)inFig. 6representcaseswherefaultsinSUCcanbedetectedbyMBMT. Stated differently,ifthereareanymutantsinthesecategories,then SUChassomefaults thatarenotin

_M

andcannotbe detected by thetest set

T

generatedby applying atest generation algorithm

Φ

to themodel

M

.This alsoimpliesthat if

|

LMne1

|

+ |

KMna1

|

+ |

KMna2

|

=

0, then

T

should be improved asitfailsto detect some faults inSUC. Also, ifthereare any mutants in Box (b) (i.e., LMne2 is not empty), then some non-equivalent mutants are still live, butmay be killed if

T

∗

= Φ(M

∗

)

isimprovedwithadditionaltestcases.Eachofthesecases(either

T

or

T

∗shouldbeimproved)suggeststhat thecorrespondingtestgenerationalgorithm

Φ

shouldbeimproved.Basedontheabovediscussion,wedefinethefollowing scores.

Definition12.Givenamodel

M

,thecorrespondingSUC,atestgenerationalgorithm

Φ

,andasetofmutationoperators X ,

the mutationfaultdetectionscore MFDS

(

M ,

SUC

,

Φ,

X

)

:=

|LMne1|+|KMna1|+|KMna2|

|M∗_|−|LMe|

=

|LMne1|M∗_|−||+|LMeKMna| |, where

|M

∗

|

is thenumber

ofallthemutantsgenerated.

MFDSgivestheratioofthenumberofmutantsidentifiedaskilled orlive thathelpdetectfaultsinSUC(thenumerator) tothetotalnumberofnon-equivalentmutants(thedenominator).RefertoSection 5.2formoreexplanation.

Definition13.Givenamodel

M

,thecorrespondingSUC,atestgenerationalgorithm

Φ

,andasetofmutationoperators X ,

themodified mutationscore MMS

(

M ,

SUC

,

Φ,

X

)

:=

|KMa|

|M∗_|−|LMe|,where

|M

∗

|

isthenumberofallthemutantsgenerated. MMS,similartothecode-basedmutationscore,givestheratioofthemutantsthatarekilled(butdonothelprevealany faults inSUC) tothetotalnumberofnon-equivalentmutants.Assumingthereisatleastonenon-equivalentmutant, MMS

hasavalue of1if

|

LMne1

|

+ |

KMna1

|

+ |

KMna2

|

=

0 (i.e.,MBMTdoesnot helpthedetectionofanyadditionalfaults inSUC which havenotbeenrevealedby testcasesin

T

= Φ(M )

) and

|

LMne2

|

=

0 (i.e.,no mutantsarelivewherethereasonof livenessisbecause

T

∗cannotdetecttheinjectedfaults).

3.2. MutantgenerationandexecutionforMBMT

Inthefollowing,amodel-independentalgorithm(Algorithm 1)formutantgeneration,executionofSUC,andclassification of mutants into killed or live is presented. The input consists of a model

M

that describes the underlying SUC, a set of mutationoperators X , anda test generation algorithm

Φ

. The output isa list ofmutants generatedby the mutation operators in X that are killed by test casesgenerated by

Φ(

M

∗

)

anda list of mutantsthat are still live. Eachmutation operator xi

∈

X is applied to

M

to generatea setof mutants denoted as

M

i∗.For each valid mutant

M

i j∗

∈ M

i∗,SUC is executed against test cases (one by one) in the corresponding

T

i j∗ generated by applying the algorithm

Φ

to

M

i j∗. The mutant

M

_{i j}∗ismarkedaskilled iftheexpectedoutputofatestcaset∗ differsfromthatofSUC.Notethatinfutureworkwe willexaminetheconditionsunderwhichtheexecutionofremainingt∗

∈ T

_{i j}∗ isnecessary.Ifnotestcasein

T

_{i j}∗cankill

M

_{i j}∗, themutantismarkedaslive.Detailsof

Φ(

M )

foreventandstate-basedmodelssuchasESG(eventsequencegraph),FSM (finite-statemachine)andSC(statecharts)havebeenpresentedinSection2.

4. Elementarymutationoperators

TheproposedMBMTapproachisexemplifiedusingthegraph-basedmodelsintroducedinSection3.Weexplainindetail thetwoproposedelementarymutationoperators(insertion andomission)andtheirapplications.

4.1. Syntacticdefinitionbasedondirectedgraphs

Inthefollowing,thetwoproposedelementarymutationoperatorsaresyntacticallyintroducedbasedondirectedgraphs; theirbasicfeaturesarealsodiscussed.Twotypesofmutants(node andarc mutants)canbegeneratedforeachDG.Foreach type, two elementary mutation operators are defined, namely insertion andomission. Togenerate a first-ordermutant, a mutation operator isapplied exactly once to a DG.Note that even in thisinitial step, themutated DG∗

= (

N∗

,

A∗

)

may becomeinvalid (e.g.,unconnected). Onlyinthecasethatan operatorwouldalwaysgenerateinvalidmutants,aminimum setofadditionaloperationsarealsoappliedtomakethemutantsvalid.Higher-ordermutantscanbegeneratedbyapplying theelementaryoperatorsortheircombinations,multipletimes.

Definition14.Thenodeinsertion operatornI

(

DG

, α)

→

DG∗

(

N

∪{

α

},

A

∪{(β,

α)

∪(

α

, γ

)

})

,where

α

,

β

and

γ

arethreenodes suchthat

α

∈

/

N,

β

and

γ

∈

N.

The nodeinsertion operator (nI-operator)inserts a node

α

∈

/

N into a given DG.To keep the resulting graphstrongly

connected,it alsoinserts anincoming arcto

α

,say(

β

,

α

), andanoutgoing arcfrom

α

,say (α,

γ

),where

β

and

γ

are arbitrarilychosenfromN suchthat

β

=

α

and

α

=

γ

.

Definition15.The nodeomission operator nO

(

DG

, α)

→

DG∗ (N

\{

α

}

, A

\{(β,

α)

|(β,

α)

∈

A

, α

and

β

∈

N

}\{(

α

, γ

)

|(

α

, γ

)

∈

A

, α

and

γ

∈

N

}

).3

Thenodeomission operator(nO-operator) removesanode

α

∈

N fromagivenDG.Italsoremovesthearcsthat have

α

aseitherthestarting orending node.IftheresultinggraphviolatesthevalidityofDG(e.g.,becoming unconnected),itis

discarded.

Definition16.Thearcinsertion operatoraI

(

DG

,

(

α

,

β))

→

DG∗ (N

,

A

∪ {(

α

,

β)

}

),where

α

and

β

∈

N but

(

α

,

β) /

∈

A.

Thearcinsertion operator(aI-operator)insertsanarc

(

α

,

β) /

∈

A betweennodes

α

and

β

inthegivenDG.

Definition17.Thearcomission operatoraO

(

DG

,

(

α

,

β))

→

DG∗ (N

,

A

\{(

α

,

β)

}

),where

(

α

,

β)

∈

A.

Thearcomission operator (aO-operator)removesan existingarc(α,

β

)fromagivenDG.Iftheresultinggraphbecomes

invalid(unconnected),itisdiscarded.

Definition18.Asubgraph DGofadirectedgraphDG isdefinedasDG

⊂

DG

:= ((

N

⊂

N

∧

A

⊆

A

)

∨ (

N

⊆

N

∧

A

⊂

A

))

.If twographsDG1andDG2arenotsubgraphsofeachother,itiswrittenasDG1



DG2.

UsingDefinition 18wedefineincrescent,decrescentandcrossmutantsasfollows.

Fig. 7. An I -mutant of the ESG given inFig. 1.

Fig. 8. A D-mutant of the ESG given inFig. 1.

Fig. 9. A C -mutant of the ESG given inFig. 1.

Definition19.

•

DG∗isanincrescent mutant (I-mutant)ofDG ifDG

⊂

DG∗

•

DG∗isadecrescentmutant (D-mutant)ofDG ifDG∗

⊂

DG

•

DG∗isacrossmutant (C -mutant)ifDG



DG∗

AgivenDGisasubgraphofitsincrescentmutants,andanyofitsdecrescentmutantsaresubgraphsoftheoriginalDG. There isnosubgraphrelationbetweenaDG andits crossmutants.UsingthenotionsofDefinition 19,wehavefollowing lemma.

Lemma1.

•

MutantsDG∗,generatedbythenI andaI-operators,areI-mutants.

•

MutantsDG∗,generatedbythenO andaO-operators,areD-mutants.

•

C -mutantscanonlybegeneratedbyapplyinganinsertionoperator(nI oraI)followedbyanomissionoperator(nO oraO),orvice

versa.Eachoperatorcanbeappliedmultipletimes,ifnecessary.

The first two parts of the Lemma are straightforward. The third part says to generate a C -mutant of a DG, we not only insert “a node or an arc” into the DG butalso delete “a node or an arc different fromthat being inserted” from the DG.Whethertheinsertion is beforeorafterthe deletionisirrelevant. Forexample,a C -mutantcan begenerated by first applyingthe nI-operator to insert a node (say

α

) and the associated arcs, followed by the nO-operator to delete a node (differentfrom

α

) andtheassociatedarcsfromtheDG.Similarly,aC -mutantcanalsobegeneratedbyapplyingthe

aO-operatortodeleteanarc(say(α,

β

))followedbytheaI-operatortoinsertanarc(differentfrom(α,

β

)).

Example1. MutantsinFig. 7,Fig. 8 andFig. 9give examplesofan I-mutant,a D-mutantanda C -mutantoftheESG in

Fig. 1,respectively.

Aproblemwithmutantgenerationisthattheapplicationofdifferentmutationoperatorscanleadtothesamemutants. WhenapplyingtheproposedMBMT,testcasesaregeneratedwithrespecttoeachmutant(seeAlgorithm 1inSection3.2). Hence, multipleoccurrences ofany duplicate mutantsintroduce inefficiency and unnecessary waste oftestingresources. We nowdiscusswhetherthereare anyduplicates amongthefirst-ordermutantsgeneratedby applyingthenI,aI, nO,or

aO-operatorexactlyonceto agivenDG.LetDG∗nI,DG∗aI, DG∗nO,andDG∗aO bethemutantsgeneratedbythe nI,aI, nO,

andaO-operators, respectively.Also, let A∗nI, A∗aI, A∗nO and A∗aO be theresultingsetofarcs dueto theapplicationof

the nI,aI,nO,andaO-operators,respectively,and N∗nO and N∗aO betheresultingsetofnodesduetotheapplicationof

Theorem1.First-ordermutantsofaDGgeneratedbythenI,aI,nO,andaO-operatorsaredifferentfromeachother.

Proof.

•

nI versusnO,nI versusaO, aI versusnO,andaI versusaO:fromLemma 1itfollowsthatmutantsaredifferent.

•

nI versusaI

:

DG∗nI



DG∗aI as

(

A∗nI

⊂

A∗aI

)

∧ (

A∗aI

⊂

A∗nI

)

Hence,first-ordermutantsgeneratedbythenI-operatoraredifferentfromthosegeneratedbytheaI-operator.

•

nO

(

DG

, α)

versus aO

(

DG

,

(β, γ

))

: if(α

= β ∨

α

=

γ

) then DG∗nO

⊂

DG∗aO as

(

N∗nO

⊂

N∗aO

)

∧ (

A∗nO

⊂

A∗aO

)

else

DG∗nO



DG∗aO as

(

A∗nO

⊂

A∗aO

)

∧ (

A∗aO

⊂

A∗nO

)

Hence,first-ordermutantsgeneratedbythenO-operatoraredifferentfromthosegeneratedbytheaO-operator.

2

Thenodecorruption operator(nC-operator)firstdeletesanodefromagivenDG,theninsertsanothernodewhichisnot

currentlyintheDG. Itcan beviewedastheapplicationof annO-operator followedby an nI-operator.The arccorruption

operator (aC-operator)changes the directionof an existing arcof agiven DG.It can be viewedas theapplication ofan

aO-operatorfollowedbyan aI-operator.ThenI,nO,aI,aO, nC andaC-operatorscanbeappliedrepeatedlytoaDGmultiple

times.Theycanalsobecombinedwitheachother.

4.2. ESG-basedmutation

Iftheunderlyingmodelinthe proposedMBMTapproachisan ESG,itiscalledan ESG-basedmutation.Letusassume anESGcorrectlyspecifiestheexpectedbehaviorofthecorrespondingSUC.ThisESG canthenbeusedtogeneratemutants

ESG∗

= (

E∗

,

A∗

,

Ξ

∗

,

Γ

∗

)

fortheproposedMBMT.

Eventmutants of an ESG canbe generated by using theeventinsertion operator(eI-operator) to insertan event e

/

∈

E

intotheESG resultingin E∗

=

E

∪ {

e

}

orthe eventomission operator(eO-operator)todeletean evente

∈

E from theESG resultingin E∗

=

E

\{

e

}

.TheeI and eO-operatorsareapplied similarlytoESGasthecorrespondingnI and nO-operatorsto

DG.FortheeI-operator,similartothenI-operator,thereisaneedtoinsertanadditionalincoming/outgoingarctoandfrom

theinsertedevent.Accordingly,fortheeO-operatorthereisaneedtoremovethearcsconnectedwiththedeletedevent.

Sequencemutants of an ESG can be generated by using the sequenceinsertion operator (sI-operator) to insert an arc

(whichisnotin A) intotheESGorthesequenceomission operator (sO-operator)todeletean arc(which isin A)fromthe

ESG.The sI and sO-operatorsare appliedsimilarlytoESG asthecorrespondingaI and aO-operatorstoDG.Afterapplying

the sO orsI-operators to an ESG, the validity ofthe resulting ESG hasto be checked.We also havethe eventcorruption

operator(eC-operator)andthesequencecorruption operator(sC-operator) appliedsimilarlytoESG asthecorrespondingnC

andaC-operatorstoDG(seeSection4.1).

TheeI,eO,sI,sO,eC andsC-operators,andtheircombinations,canbeappliedrepeatedlytoanESGmultipletimes.

4.3. SC-basedmutation

IftheunderlyingmodelintheproposedMBMTapproachisanSC,itiscalledanSC-basedmutation.Statemutants ofan FSMorSCcanbegeneratedbyusingthestateinsertion operator(stI-operator)orthestateomission operator(stO-operator).

ThestI-operatorinsertsastate(whichisnotcurrentlyinthestatechart)intoS.Italsorequirestheinsertionofanadditional

incoming/outgoingtransition-event pair to/fromthe inserted state. The stO-operator deletesan existing state from S.All transition-eventpairsconnectedwiththedeletedstateshouldalsoberemoved fromtheFSM.Similarly, transitionmutants

ofanFSMorSCcanbegeneratedbyusingthetransitioninsertion operator(tI-operator)orthetransitionomission operator

(tO-operator). The tI-operator inserts atransition tr

= (

s

,

s

)

into TR,where s and s

∈

S, but

(

s

,

s

) /

∈

TR. The tO-operator

deletesan existing transitiontr from TR.The stI, stO,tI and tO-operatorsaredone ina similarwayasthenI, nO,aI and

aO-operators for DG, respectively. After the application of the stI, stO, tI and tO-operators, the function f needs to be

updatedaccordinglytoaddordeleteappropriatemappingsbetweentransitionsandevents.

Wealsohavethestatecorruption operator(stC-operator)andthetransitioncorruption operator(tC-operator)forFSMand

SC,whichworkthesamewayasthenC andaC -operatorsforDG.Finally,thestI,stO,tI,tO,stC andtC-operators,andtheir combinations,canbeappliedrepeatedlytoanFSMorSCmultipletimes.

4.4. Elementarymutationoperatorscomparedwiththosefromliterature

Table 1summarizesthedifferencesbetweenthecode-basedmutationoperatorsreportedintheliteratureandthe ele-mentarymutationoperatorsproposedinthispaper.

Fordiscussionpurposes,weusethemutationoperatorsreportedbyFabbrietal.[10,12]astheexample.Theseoperators generatemutantsforFSMandSC.ThefirsttwocolumnsofTable 2giveoperatorsdefinedin[10]and[12],whereasthelast threecolumnsshowhowthey canbereplacedbyusingan insertionoromissionoperatororacombinationofthesetwo.

Table 1

Comparison between code-based mutation operators and the proposed elementary operators.

Code-based mutation operators Elementary mutation operators Number of operators typically a small to medium-sized set of operators (e.g.,

[29]); some studies use a large set of operators (e.g., [5]) two (insertion and omission) Fault model operators are chosen to simulate faults introduced by

common programming mistakes no explicit fault models Construction by experience and empirical results systematically

Order of mutants typically used first-order mutants first-order mutants; second-order mutants (corruption operators)

Reducibility code-based operators can be derived from elementary

operators and their combinations atomic; cannot be further reduced to other operators

Table 2

Mutation operators in [10]and [12]and their equivalence using the elementary operators.

Mutation operators in[12](first column) and[10](second column) Insertion Omission Artifact wrong-start-state wrong-starting-state (default state) × × start-state

arc-missing arc-missing × arc

event-missing event-missing × event

event-extra event-extra × event

event-exchanged event-exchanged × × event

destination-exchanged – × × destination

output-missing output-missing × output

output-exchanged output-exchanged × × output

– output-extra × output

state-missing – × state

– state-extra × state

Forexample,referringtothefirstrow,thereisan operatorin[10]and[12]togeneratemutantswith“wrong-start-state”. Thesemutantscanalsobegeneratedby applyingeitheran insertionoranomissionoperatortothe“start-state”.Another exampleistheoperatortogeneratethe“event-exchanged”mutantsin[10]and[12]canbereplacedbyanomissionoperator followedbyaninsertionoperatorontherelatedevents.

There are two advantages ofusing theproposed elementary mutationoperators over thosesuggestedby Fabbri etal. First, thenumberofoperators canbereducedtomake mutationtestingeasiertounderstand.Second,a systematic appli-cationoftheelementaryoperatorstoFSMandSCcanhelptestersavoidpossibleinconsistenciesofcertainoperatorsbeing appliedtoFSMbutnottoSC,orviceversa.Ithasbeennotedthattheoperatortogeneratemutantsbydeletingoneofthe existing statesis includedin[12],butnot in[10] (see thesecond rowfromthe bottomin Table 1), andtheoperator to generatemutantsby insertingan additionalstateisincludedin[10],butnotin[12](seethelastrow).Suchinconsistency is counter-intuitiveasonewouldassume ifmutantswitha missingstateare generated,thenmutantswithan additional state wouldalso be generated.There is no explanation in [10] and[12] asto why such inconsistency occurs. However, a systematicapplicationoftheelementaryoperators canhelp testerseasilydiscoverthiskindofinconsistency.The same informationcanalsobeusedtoidentifywhatkindofadditionalmutantsmaybegenerated.

5. Casestudies

Three casestudieswere conductedusingindustrial andcommercialreal-life systems:an interactive commercialmusic managementsystem,areactive/adaptive,real-timecruisecontrolsystem,andaproactivecontroldisplayunitofamarginal strip mowermountedonatruck.The firsttwostudiesuseESG (seeDefinition 2)tomodelthesystembehavior,whereas thethirdstudyusesESGandSC(seeDefinition 5).Objectivesofthesestudiesare:

•

EvaluatethefaultdetectioncapabilityoftestsetsgeneratedusingtheproposedMBMTapproachasdescribed inFig. 5

withESGandSCasthemodelswhichisdoneby

– determiningwhethertherearestillremainingfaultsinSUCthatcanbedetectedbysuchtestsets – measuringthescoresMFDS (seeDefinition 12)andMMS (seeDefinition 13)

•

ComparethefaultdetectioncapabilityoftestsetsgeneratedbytheESG-basedmutationversustheSC-basedmutation – testsetsgeneratedbasedontheeventandsequencemutantsofanESG

– testsetsgeneratedbasedonthestateandtransitionmutantsofanSC

For ESG-basedmutation,first-order mutantsweregenerated byapplying thesI,sO, andeO-operators exactlyonceto the ESG.TheprocessinFig. 2wasusedtogenerateatestsetforthesemutantscoveringalltheeventsequencesoflength2(i.e., coveringalltheeventsandarcsinanESG).Similarly,forSC-basedmutation,first-ordermutantsweregeneratedbyapplying

thetI, tO,andstO-operatorsexactlyoncetotheSC.The processinFig. 3wasusedtogeneratetestsetsforthesemutants

Fig. 10. RJB.

Fig. 11. ACC.

stI-operatorswouldgeneratealargenumberofmutants.Moreover,thesemutantsrepresentasimilarfaultmodelassI and

tI-mutantsinthecontextofgraphicaluserinterfaces.Hence,forreasonsofcostandpracticability,eI andstI-mutantswere

excluded inthecasestudies.As a result,the numberoftestsets fortheESG-mutation isthenumberof validfirst-order mutantsgeneratedbythe eO,sI and sO-operators,whereas thenumberoftest setsfortheSC-mutationisthe numberof validfirst-ordermutantsgeneratedbythestO,tI andtO-operators.However,we emphasizethatthecostoftest execution primarilydependsonthesize(intermsofthenumberofevents)ofallthetestcasesineachsetratherthanthenumberof testsetsitself.ThissizevariesforeachESGorSCandisdeterminedbythealgorithmsusedtosolvetheChinesePostman Problemwithrespecttotheunderlyingmodel.AsexplainedinFig. 2andFig. 3,thetotalsizeofeachsetisminimizedin ordertoreduce thecostoftestexecution. Alsonote thatMBTwas conductedbeforehandbasedonESG andSCusingthe testgenerationprocessesdescribed inFig. 2(withn

=

2)andFig. 3(withn

=

1).Allthefaultsdetected inSUCbyMBMT werenotfoundbyMBT.

5.1. DescriptionoftheSUC

Real Jukebox (RJB) is an interactive personal music management program developed by Real Networks [30] for PCs running the Microsoft Windows operating system. The basic English version ofReal Jukebox 2 (build 1.0.2.340) of Real Networks was used asthe SUCfor thefirst casestudy. The main GUIofRJB (see Fig. 10) has severalpull-downmenus thatinvokeothercomponents.Inthisstudy,RJBwasinstalledonaPCrunningWindowsXPwithServicePack2satisfying therecommendedhardwarerequirements(300MHzCPU,64MBRAM, fullduplexsoundcard,16bitcolorvideo card)as describedintheproduct’smanual.Duetothelackofamanufacturer’ssystemspecification,thatis,afunctionaldescription

ofRJB,theHelp contentsandhandbookofRJBwereusedtoproduceESGforgeneratingtestcases(eachofwhichisaCES).

The second case study comes from the automotive industry. It is a driver assistance system manufactured by Hella Corp.[31],one ofthemajor German suppliersforelectronic devices.Thefocus isonthe adaptivecruise control(ACC) – anelectroniccontrolunit (ECU),whichisanembeddedreactivesystem(seeFig. 11).Toautomaticallycontrolthedistance betweenvehicles,ACCissupplementedwitha24GHzradarandconnectedoveraCANbus(controllerareanetwork)tofive otherECUandtherebyindirectlycoupledtomorethan7sensors.AppropriateeventswereselectedforconstructingESGto modeltheSUCthatarespecific tothesystemandtestenvironment,forexample,shortcircuits.Arcsconnecttheseevents accordingtothesystemspecificationandlogicalrelations.Forexecutionofthetests,acompany-developedtestenvironment wasusedthatincludedaPC,SUC,aPLC(programmablelogiccontroller),aprogrammablepowersupply,andthestandard softwareCANoe[32]whichprovidessimulationandmonitoringofspecifictrafficscenariosaswellassimulationofallother

Fig. 12. RSM13.

Fig. 13. An ESG to model the switch of RSM13 between its transport and working positions.

involvedECUandcreation oftest reportsbasedonXMLandHTML.While generatingthetest sequencesforthesoftware, CANoestate-basedinformationwasaugmentedbyconditionsandvariables.

The third case studyis a proactive systemto control a marginal strip mower (RSM13, see Fig. 12) mounted on the Unimog, a truckmanufactured by Mercedes-Benz[33].Considering safetyaspects,the mostsignificantcomponent ofthe strip mowerare revolving knives.Operation is affected eitherby the powerhydraulic ofthe light truck orby the front powertake-off.Thepositionofthemowheadchangescontinuously.Analterationfromworkingontheleftsidetoworking ontherightsideisalsopossible.Besidesthepositioning,theinclineandsupportpressureofthemowingunitcanalsobe controlled.Thecontrolterminalhasfourdifferentmodes.Thefirstmodeisusedtocontroltheactuators,thesecondmode is used to switch betweentransport andworkingposition, the third and the fourthare two differentoperation modes. These fouroperationmodeshavebeenmodeledby ESG andSC, respectively.An ESGandan SCforthe secondmode are shown inFig. 13 andFig. 14,respectively. Referto http :/ /quebec .upb .de /MBMT.pdfformore information.Notethat these twomodelswereconstructedindependentlyforcomparingtestsetsgeneratedusingtheESG-basedmutationagainstthose usingtheSC-basedmutation,withrespecttotheirfaultdetectioncapabilities.

Allthree SUTsarerealworldproducts wherethesourcecodewasunavailableduetobeinga commercial-off-the-shelf product. As MBMT isa black-boxtesting technique,this workdid not needto havethe sourcecode available.Since test case generation isbased on models, test datahas been includedin the models themselves,wherever appropriate. More precisely,thesealgorithmsarebasedontransitionsornodesrandomlyselectedfromthemodels,theamountofinputdata depending on thenumberoftestcasesgenerated.ForRJB, manualtest executionisperformedonawindows client.Test casesaremanuallyencodedforACCandsubsequentlyexecutedautomatically.ForRSM13,testcasesaremanuallyexecuted viacontrolterminal.

5.2. Results,analysisanddiscussion

Havingestablished asetofmutationoperators, onequestion ofinterestiswhethermutantsgeneratedwithrespect to a particularoperatorleadtoamoreeffectivedetectionoffaultsintheSUC.Furthermore,aresometypesofthegenerated mutantseasiertokillthanothers?Thedatathatfollowsaimstoaddressthesequestions.

Mutants generated for RJB, ACC andRSM13 are classifiedbased on the scheme in Fig. 6.Tables 3, 4 and 5 give the numberoflive (LMne1,LMne2),killed (KMa,KMna)andequivalent (LMe)mutantswithrespecttoeachmutationoperator.Also includedare themutationfaultdetectionscore(MFDS)andthemodified mutationscore(MMS).Forexample,referring to

Fig. 14. A statechart to model the switch of RSM13 between its transport and working positions.

Table 3

Classification of ESG mutants generated for RJB, and the corresponding MFDS and MMS.a

LMne1 LMne2 LMe KMa KMna Sum1 MFDS MMS

sI 0 0 0 182 18 200 0.09 0.91

sO 0 142 0 55 3 200 0.02 0.28

eO 0 59 0 118 1 178 0.01 0.66

Sum2 0 201 0 355 22 578 0.04 0.61

a _{The notation Sum}

1gives the total number of mutants with respect to each mutation operator, whereas Sum2gives the total number of mutants with respect to each classification as presented in Fig. 6. The same applies to Tables 4 and 5.

Table 4

Classification of ESG mutants generated for ACC, and the corresponding MFDS and MMS.

LMne1 LMne2 LMe KMa KMna Sum1 MFDS MMS

sI 0 21 0 57 4 82 0.05 0.70

sO 0 46 0 15 1 62 0.02 0.24

eO 0 11 0 25 0 36 0.0 0.69

Sum2 0 78 0 97 5 180 0.03 0.54

Table 5

Classification of ESG and SC mutants generated for RSM13, and the corresponding MFDS and MMS.

LMne1 LMne2 LMe KMa KMna Sum1 MFDS MMS

sI/tI 0/0 0/0 0/0 179/36 5/5 184/41 0.03/0.12 0.97/0.88

sO/tO 0/0 5/8 0/0 16/17 1/1 22/26 0.05/0.04 0.73/0.65

eO/stO 0/0 3/3 0/0 12/12 0/0 15/15 0.0/0.0 0.80/0.80

Sum2 0/0 8/11 0/0 207/65 6/6 221/82 0.03/0.07 0.94/0.79

thesecondrowofTable 5,thereare22sO-mutants(ofwhich5areclassifiedasLMne2,16asKMaand1asKMna)and26

tO-mutants(ofwhich8areclassifiedasLMne2,17asKMaand1asKMna).TheMFDS andMMS forthesO-mutantsare0.05

and0.73,respectively,and0.04and0.65forthetO-mutants.

Tables 6, 7 and 8 give a briefdescription ofeach fault in SUC detected by theproposed MBMTapproach, butwhich werenotdiscoveredbeforepublicrelease.Tosavespace,onlyanexcerptofthecorrespondingfaultsispresentedhere;fora completelistrefertohttp :/ /quebec .upb .de /MBMT.pdf.Thefactthatourapproachcoulddetect6faultsinareleasedversion ofRSM13(theSUCusedinthethirdcasestudy)thatwerenotpreviouslyfoundduringthetestingconductedbyatechnical controlboardinGermanyisespeciallyimportant.Twoofthesefaultsareextremelysafety-critical.Fault1indicatesthatthe cuttingunitcanbeactivatedwithouthavinganypressureonthebottom,whichisverydangerousifpedestriansapproach

Table 6

Description of 5 (of the 22) faults detected in RJB by the proposed MBMT approach.

Fault ID Description of the fault Operator

1 Track Position cannot be set anymore and the application freezes when the end of track is reached. sI

2 GUI shows state Paused but the music is still playing. sI

3 PushingPlay button plays a wrong track. sI

4 AutoPlay does not start playing when a CD is inserted. sI

5 Pause button has no impact despite the fact that the system is playing. sI

Table 7

Description of the faults detected in ACC by the proposed MBMT approach.

Fault ID Description of the fault Operator

1 The system resides in the error state and the speed of the vehicle drops to 5 km/h. If a simulation is now

requested, the system remains in the error state (instead of switching to the simulation state as required). sO 2 If an error occurs, the system does not switch in time to the error state. sI 3 If the system resides in the error state and the error vanishes, the system does not switch to the requested

active state in time. sI

4 If the system resides in the error state and the error vanishes, the system does not switch to the requested

inactive state in time. sI

5 The system does not switch from initial state to active/inactive state in time. sI

Table 8

Description of the faults detected in RSM13 by the proposed MBMT approach.

Fault ID Description of the fault Operator

1 When the function bearingPressure is

deactivated, the function

cuttingUnit can

be activated.

sI for ESG or tI for SC 2 When the function cuttingUnit is

activated, the function

bearingPressure can

be deactivated.

sI for ESG ortI for SC 3 When Engine I is activated, a change from the view RSM_Actuator to

the view

Start is

possible.

sI for ESG or tI for SC 4 When Engine II is activated, a change from the view RSM_Actuator to

the view

Start is

possible.

sI for ESG or tI for SC 5 When Axis Lock is activated, a change from the view RSM_Actuator to

the view

Start is

possible.

sI for ESG or tI for SC 6 A change from the view RSM_Operation_II to

the view

RSM_Operation_I while

function

ArmShiftLeft is

active is

possible if the function ArmShiftLeft is

activated.

sO for ESG or tO for SC

the workingarea.Fault 6isobserved bykeepingthe buttonforshiftingthe mowheadpressed andchangingtoanother screen. Themowerheadwithitscuttingunitcannotimmediatelybestoppedinanemergency.Furthermore,restartingthe hydraulicgearwhileitisalreadyrunningcancauseseriousdamage.

The last columnof Tables 6, 7 and 8 indicate which type of mutants the fault revealingtest sets (the test sets that detected theremainingfaultsintheSUC)weregenerated.Forexample,ofthe22faultsdetectedinRJB,18faultrevealing test sets were generated withrespect to thesI-mutants, 3 test sets withrespect tothe sO-mutants, and1 test setwith respect totheeO-mutants.Ofthe5faults detected inACC,4fault revealingtest setswere generatedwithrespectto the

sI-mutants,1testsetwithrespecttothesO-mutants,andnonewithrespecttotheeO-mutants.Ofthe6faultsdetectedin

RSM13, 5faultrevealingtestsetswere generatedwithrespecttothesI or tI-mutants,1testsetwithrespecttothesO or

tO-mutants, andnonewithrespecttotheeO orstO-mutants.Withrespecttoourdata,itisclearthattestsets generated

with respect to the sI-mutants are more effective in detecting faults in SUC than those generated with respect to the

sO-mutants,whichinturnaremoreeffectivethanthosegeneratedwithrespecttotheeO-mutants.Itisalsoobservedthat

testsetsgeneratedusingtheSC-basedmutation(withrespecttothetI,tO andstO-mutants)havethesamefaultdetection capabilityastestsetsgeneratedusingtheESG-basedmutation(withrespecttothesI,sO andeO-mutants).

Comparing the data in the last columnof Tables 6, 7 and 8 withthe data in the 5thcolumn (marked asKMna) of

Tables 3, 4 and 5,averyimportantpointworth notingisthatthenumberoffaultrevealingtestsetswithrespecttoeach type ofmutantisthesameasthenumberofmutantsclassifiedasKMna foreachtype.Also,thetotalnumberoffaultsin SUCdetected byusingMBMTisthesumofallthemutantsclassifiedasKMna andLMne1.Forexample,thereare22faults detected forRJBwhichisthe sumof18(number ofsI-mutantsclassifiedasKMna)

+

3(numberofsO-mutantsclassified as KMna)

+

1 (number ofeO-mutants classifiedasKMna)

+

0 (numberof anymutants classifiedasLMne1). Thisis also the reasonwhythemutantfaultdetectionscoreMFDS isdefinedastheratioof(

|

LMne1

|

+ |

KMna

|

)to (

|M

∗

|

− |

LMe

|

) (see

Definition 12).

Inallcases,thesetsLMne1 andLMe (equivalentmutants)areempty.WealsoobservethatthemajorityofthesI-mutants (182 of 200forRJB, 57of82 forACC,and179 of184forRSM13) are classifiedasKMa.The reasonis that insertingan additional sequence(especially thoserelatedto the GUIsofRJB andRSM13) will mostlikelyintroduce a non-executable sequenceofevents,whichcanbeeasilydetected.Oneexampleisabuttonthatshouldbedisabledinacertaincontextbut becomes enabledduetothisadditionalsequence(arc).Consequently,themodified mutationscores (MMS)forsI-mutants