Test case prioritization approaches in regression testing: A systematic literature review

Contents lists available at ScienceDirect

Information

and

Software

Technology

journal homepage: www.elsevier.com/locate/infsof

Test

case

prioritization

approaches

in

regression

testing:

A

systematic

literature

review

Muhammad Khatibsyarbini

, Mohd Adham Isa, Dayang N.A. Jawawi, Rooster Tumeng

DepartmentofSoftwareEngineering,FacultyofComputing,UniversitiTeknologiMalaysia,81310JohorBahru,Johor,Malaysia

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received29December2016 Revised2August2017 Accepted25August2017 Availableonline1September2017 Keyword:

Testcaseprioritization Regressiontesting Softwaretesting

Systematicliteraturereview

a

b

s

t

r

a

c

t

Context: Softwarequalitycanbeassuredbygoingthroughsoftwaretestingprocess.However,software testingphaseisanexpensiveprocessasitconsumes alongertime. Byschedulingtestcasesexecution orderthrough aprioritization approach, softwaretestingefficiency can beimprovedespeciallyduring regressiontesting.

Objective: Itisanotablesteptobetakeninconstructingimportantsoftwaretestingenvironmentsothat asystem’scommercialvaluecanincrease. Themainideaofthisreviewistoexamine andclassifythe currenttestcaseprioritizationapproachesbasedonthearticulatedresearchquestions.

Method: Setofsearchkeywordswithappropriaterepositorieswereutilizedtoextractmostimportant studiesthatfulfillallthecriteriadefinedandclassifiedunderjournal,conferencepaper,symposiumsand workshopscategories.69primarystudieswerenominatedfromthereviewstrategy.

Results: Therewere40journalarticles,21conferencepapers,threeworkshoparticles,andfive sympo-sium articlescollectedfromtheprimarystudies.Asfortheresult,it canbesaidthatTCP approaches arestillbroadlyopenforimprovements.EachapproachinTCPhasspecifiedpotentialvalues,advantages, andlimitation.Additionally,wefoundthatvariationsinthestartingpointofTCPprocessamongthe ap-proachesprovideadifferenttimeline andbenefittoprojectmanagertochoosewhichapproachessuite withtheprojectscheduleandavailableresources.

Conclusion: Testcaseprioritizationhasalreadybeenconsiderablydiscussedinthesoftwaretesting do-main.However,itiscommonlylearnedthattherearequiteanumberofexistingprioritizationtechniques thatcanstillbeimprovedespeciallyindatausedandexecutionprocessforeachapproach.

© 2017ElsevierB.V.Allrightsreserved.

1. Introduction

Softwareengineeringisnotjustprogrammingandsoftware de-velopment.Softwareengineeringitselfisanimplementationof en-gineeringproceduresinthedevelopmentofanysoftwareina sys-tematicway[1].Withinasoftwaredevelopmentprocess,software testingconsumesalongertime inexecutionandcanbethemost expensivephase[2].Softwaretestingitselfisnormally,repetitively, carried out even when there are time constraint and fixed re-sources. Software engineering groups are regularly compelled to endtheir testingactivities becauseof financial andtime necessi-ties,whichwilltriggersomedifficultiessuchasproblemswiththe softwarequalityandclientagreement.However,theapplicationof test caseprioritization (TCP) appears to enhance test viability in softwaretestingactivity[3].

∗ _{Correspondingauthor.}

E-mail addresses: [email protected], [email protected] (M. Khatibsyarbini).

Regressiontestingisanactivitytoconfirmthatprogressionsdo notharmthepreviouslyfunctioningsoftware[4,5].Asthesoftware evolves, asoftwaretestsuitehasthetendencytoincrease insize which frequently makes it expensiveto execute. Research shows regressiontestingisanexpensiveprocesswhichmayrequiremore than 33% of the cumulative expenses of the software [6]. In the work of Yoo and Harman[7], various regression test approaches wereexaminedtosupplementtheimportanceoftheaccumulated test suitein regressiontesting.Those studieswere thenclassified intothreedomains;minimization,selection,andprioritization.

Testsuiteminimization(TSM)approachesintendtodistinguish repetitiveexperimentsandtoeliminatetestcasesfromatestsuite executionwithaspecific goalsuch astodecrease thenumberof teststorun[8].Minimizationissometimescalled‘testsuite reduc-tion’,meaningtheeliminationoftestcasesarepermanent.

Test case selection (TCS) approach also aims to decrease the number of test cases to be executed, however, the main idea of selectionapproach isthatit isintendedtobe modification-aware [9].TCStriestorecognizethetestcaseswhichwouldbeimportant tothelatestchangesonasoftware.

http://dx.doi.org/10.1016/j.infsof.2017.08.014 0950-5849/© 2017 Elsevier B.V. All rights reserved.

Table1

Regressiontestapproaches.

Component Regressiontestapproach

Minimization Selection Prioritization

Strategy Eliminatetestcase. Modification-awaretestcase. Testcasepermutationbyorderingandprioritizing. Strength Effectiveinreducingtestcases. Effectiveinselectingmodification-awaretestcases. Usefulwhennewtestcaseswillalwaysbe

consideredinthetestcasepermutation. Limitation Testcasesarenotmodification-aware. Newtestcasesmightbemissedoutinthetemporary

selectionthatismodification-aware.

Time-consuming,largertestsuite.

Table2

Summaryofrelatedstudiesinregressiontesting.

Studytype Studyreferences Studyfocus Yearofpublication Totalstudiesreviewed Yearscovered

SLR Singhetal.[13] TestCasePrioritization 2012 65 1997–2011

Mapping CatalandMishra[14] TestCasePrioritization 2013 120 2001–2011

Surveys YooandHarman[7] RegressionTesting 2012 159 1977–2009

KumarandSingh[15] LiteratureSurveyonTCP 2014 19 NA KiranandChandraprakash[16] LiteratureSurveyonTCP 2015 90 NA

Lastly, testcaseprioritization(TCP) aims toordera setoftest casestoachieve anearlyoptimizationbasedonpreferred proper-ties [3,10]. Itgives an approach the abilityto execute highly sig-nificant test cases first according to some measure, andproduce thedesiredoutcome,suchasrevealingfaultsearlierandproviding feedbacktothetesters.Italsohelpstofindtheidealpermutation ofaseriesoftestcasesandcouldbeexecutedaccordingly[7].

Table 1 shows a general comparison of three approaches in regression testing. Test case minimization reduces the test case amount in a set of test suite continuously while selection tech-nique performs atemporary selection ofseveraltest caseswhich relatedaretomodificationawareness.Fromselectiveselection, im-portant testcasesmightbe missedoutfromthetest suite.These test casescould possiblycontainan importantprioritythat needs to be executed to reveal certain faults. Intest caseprioritization, everysingletestcaseincludingnewtestcasesthatareaddedinto present test suite execution will be considered in prioritization. This iscrucial asnewtest caseswill be executed to test a mod-ified part of the software,hence, any abnormalities in the func-tionaloutputcouldeasilybeobserved.

Despite the fact that there are numerous TCP approaches in the literature, there are no latest progressive literature reviews which illustrate recent TCP importance in software testing re-search.Therefore,thisreviewattemptstoperformasystematic lit-eraturereview(SLR) onthelatestTCPapproachesasproposed by Kitchenham [11]. SLR is a specialized, uncompromising, studyof research evidence[12].Thepoint ofan SLRisnot tosimply sum-marize all current proofs based on research questions, it is also expected to bolster the improvement of evidence-based research recommendationsforresearchers.

This systematic literature review is structured as follows. Section2considersthepreviousstudiesrelatedtoTCPapproaches. Section3describesthestrategyembracedtodirectthisSLR. Next, result and discussion based on the research questions were dis-cussed in Section 4. Research findings were then elaborated in Section 5.InSection 6,thethreat ofvalidityto thisSLRwas dis-cussed. Finally, Section 7presents conclusion withregard to this systematicliteraturereview.

2. Backgroundstudiesoftestcaseprioritization

This section will discuss the previous studies that are related toTCPinregressiontesting.There werea fewsystematicreviews originatedundertheregressiontestcaseprioritizationtechniques domain.Fromtheliteraturegathered,theauthorscollatedoneSLR,

one mapping study, andthree survey studies that are related to regressiontestingandTCP,astabulatedinTable2.

Theonly SLR,work by Singh[13],offered asystematicreview inregression test case prioritization studycovering the time pe-riod from1997 to 2011.In their work, from 65 studies,49 were identifiedtoinitiateadifferentapproach,twoonaugmentationof prior studies,and14 onanalyzingback earliertestified study re-sults.TheSLRalsoanalyzedandidentifiedabouteightbroad prior-itizationapproaches.Theseapproachesinclude;genetic-based, cov-erage,requirement,modification,history,fault,composite,and oth-erswhichincludeseveralapproaches.TheSLRconcludesthateven asthereweredifferentkindsofapproaches,themainobjectiveof TCPin regressiontesting remains the same,which isto increase faultdetection.

Ontheother hand,inthework of[14],the authorspresented asystematicmappingintestcaseprioritizationwithaspecific fo-cusonTCPstudies.Itcoveredthetime periodbetween2001and 2011.Amajorityofthereviewsrecordedintheir workwereabout theapproval ofdifferent looks intoTCP andsolution recommen-dations.The resultsarethesame asreportedintheprevious SLR withasimilarinadequacy.Theauthorsmanagetoidentify16 stud-ies out of120,which correlated withthestrength andweakness ofsomeprioritizationtechniques.Tolocatethefinestprioritization method, additional reviews on the analysis of prioritization sys-temsareneededtobedoneasdifferentapproachesareconstantly beingproduced.

InadditiontothisSLRandonemappingstudy,thereisone sig-nificant survey in regression testingdomain and two minor sur-veys. The first significant survey is by Yoo and Harman [7], fo-cusingon regression testingarea which includes test suite min-imization (TSM), test case selection (TCS), and test case prioriti-zation (TCP). For TCP, they classified the current state-of-the-art approaches into several categories. Those approaches were clas-sified based on requirement, model, coverage covered, historical data,probabilisticcalculation,costawareness,andotherswhich in-clude several minorapproaches. The authors analyzed 159 stud-iescoveringthetimeperiodbetween1977 and2007.Theauthors alsomentionedthatexperimentaldesignsandevaluationsprocess were still not harmonizedbutitis gettingthe attentionofsome researchers lately.In the first minor survey [15], whichonly an-alyzed19 studies andonly focused on coverage-based technique andcost-effectivetechnique. Theauthors ofthesurvey suggested that having a new technique applied during the early stages of softwaredevelopmentlifecyclemaysignificantlyreduce develop-mentcost.The next minorsurvey[16],identified 90papersfrom IEEEpublication whichsummarized that there isa need to have

Table3

Summaryoffindingsbyrelatedstudies.

Studyreferences Coveredfindings SimilarfindingscomparedtothisSLR UncoveredfindingsaddedintothisSLR Singhetal.[13] -EmpiricalevidenceforseveralTCP

approaches(8mainapproaches) -Gapsregardingusageoftools,metrics, andartifactused

-EmpiricalevidenceforTCP approaches

-Gapsregardingusageevaluation metrics

-EmpiricalevidenceforotherrecentTCP approaches

-GapsregardingusageofartifactsonTCP approach

CatalandMishra[14] -TrendsinTCPapproaches -TrendsofTCPpublication

-TrendsofmetricanddatasetusedinTCP

-TrendsinTCPapproaches -Trendsofmetricused

-ReasonsbehindthetrendsofeachTCP approaches

-Reasonsbehindtheusesofevaluation metric

YooandHarman[7] -Overalloverviewforregressiontesting (TCP,TSM,TCS)

-Detailedoverviewofseveralpopular approachesinTCP(4approaches) -TypeofevaluationmetricusedinTCP

-DetailedoverviewofTCPapproaches -TypeofevaluationmetricusedinTCP

-Detailedoverviewofotherrecentpopular TCPapproaches

-Uncoveredevaluationmetricavailable withitsreasonsofcreations

KumarandSingh[15] -OverviewontwoTCPapproaches (coverageandcostoriented)

-DetailedoverviewofTCPapproaches -Detailedoverviewofotherrecentpopular TCPapproaches

KiranandChandraprakash[16] -Overalloverviewforregressiontesting (TCP)

-ListofseveralTCPtechniques,strategies, metrics,andalgorithm

-OverviewofTCPapproaches (includedtechniques,strategiesand algorithmused)

-Detailedoverviewofotherrecentpopular TCPapproaches

-Reasonsbehindtheusesofevaluation metric

anunderstanding ofcostcomponentsandtheadvantagesof hav-ingdifferentparametersthatcouldbetakenintoaccount.To sum-marizethebackgroundstudies inprevious relatedworks,Table 3 showsthesummaryoffindingsinrelatedstudiesincomparisonto thisSLRpaper.

From Table3, itcan be seen that only one studyprovides an empiricalevidence forsome TCPapproaches, while other studies onlyprovidean overviewofsomeapproaches.Therefore,there is an incomplete detailedoverview such asthe reasons behindthe trendsregardingsomeTCPapproaches,whichneedtobecovered. Italsocanbenoticedthatmostoftheworksonlysummarizethe numberofusageforeachevaluationmetric,butdidnotinclude in-depthdiscussions.Work bySingh[13]coveredevaluationmetrics butthegapsregardingusage ofartifactsonspecificTCPapproach werenotwelldiscussed.Inshort,thereareseveraluncovered find-ingsthatcanbeaddedtothecurrentSLRwork.

3. Researchmethod

With aspecific endgoal,a structured methodtoperform this SLR, as shown in Fig. 1, was implemented in order to examine thestudiesthat arerelatedtoTCP.The systematicandstructured methodwasinspiredbyKitchenham[11,12]andAchimugu[17].

Referring to Fig. 1, there are fivemain phaseswithin the re-view protocol, itemized as follows. Research Question, Selection ofRepositories,Search Strategy,StudySelection,andData Synthe-sis and Extraction. In the first phase, four main research ques-tionsweregeneratedtoanswerthemainaimofthispaperreview. Next,selectionofrelevantrepositorieswasperformed. Thisis fol-lowedbyemployingasearchstrategycomprisingspecifyingsearch stringsandsearchprocess,whichwereplannedbasedonthe artic-ulatedresearchquestions.Theoutputofthesearchstagewasthen movedintothestudyselection phase.Inthisphase,theoutcome ofthe search process underwent inclusion and exclusion criteria scrutinytoextractrelevantstudies.Qualityassessmentswerethen appliedtoevaluatethescrutinizedstudiesfurther.Finally,thelast phasedealtwithdatasynthesisandextractionofprimarystudies utilizedforthisSLR.

3.1.Researchquestionsandtheirmotivations

This SLRaimsto comprehendandreview recentexperimental evidencewithrespecttothemostrecentprioritizationapproaches inTCPareaforfurtherinvestigation,keepinginmindtheendgoal istoimprovisetheabilityofpresentapproaches.Atthesametime,

theauthorswishtoreviewtheempiricalevaluationsusedineach reviewed approach. To accomplish this goal, four research ques-tionswithrespectivemotivationswere articulatedaspresentedin Table4.

All theseresearch questions,that frame the reasonfor under-takingthisresearchare relativelyconnectedandconcurrently ex-plored.Theseresearchquestionsareusedtoanswertheextra find-ingsthatwillbecoveredinthisSLR,astabulatedinTable3.Tobe clear,Table 5mapseach research questionto its respectiveextra findingandthefinding’ssignificance.

From Table5,eachRQ answerssome uncoveredfindingsfrom previousworks,exceptforRQ3.ForeachRQ,thequestionsarenot onlydesignedtoanswertheuncoveredfindings,buttheyareused tocoversome extrafindingsthat canbe addedtothisSLRstudy. ThesignificanceofthefindingsforeachRQhasalsobeendetailed outasaguidancetoachievethegoalofthisSLRstudy.

3.2. Studystrategy

AstudystrategyiscrucialineverySLRtoguaranteethe broad-nessof theselectedstudies.The value oftheSLRisgenerally re-alizedaccordingto theselected primarystudies. Strategyforthis reviewdependedonthesethreestages:

a) Literaturerepositoryselection b) Searchstringidentification

c) Studyselectionprocess 3.3. Literaturerepositoryselection

The authors initiated this selection process by entering ‘Test Case Prioritization’ as search strings with the exact phrase on Google Scholar database. This database returns 2760 of studies available. From the search result, the authors identified several popularrepositoriesinTCPresearchareaanddecidedtogatherthe primary studiesoriginatedfromrecognized repositories.The cho-senrepositoriesare:

a) IEEEXplore b) ACMDigitalLibrary

c) ScienceDirect(Elsevieronly) d) WileyOnlineLibrary e) Springer

Thejustificationbehindtheselectionoftheseonlinedatabases isthat IEEEXploreoffers anumberofimportantconference arti-cles and symposium articles, while ACM Digital Library provides

Fig.1. Phasesofreviewprotocol. Table4

Researchquestionsandtheirmotivations.

Researchquestions RQstatement Motivation RQ1 Whatarethetaxonomiesoftestcase

prioritizationinregressiontesting? RQ1.1 WhatistheresearchtrendofTCP

techniquesinregressiontesting?

Theseresearchquestionsfocuson characterizingthecurrentdomainoftest caseprioritization.Thereasonistoknow thedevelopmentofTCPinregression testingthroughoutthepastyears. RQ1.2 Whatarethedistributionsofapproachesin

TCPtechniques?

RQ2 Whatarethedifferencesintermsof approachesforeachTCPtechnique? RQ2.1 Whatarethedescriptions,strength,and

limitationsofexistingprioritization approaches?

Theknowledgeofdifferencesinapproaches isnecessarytogiveaglimpseonhow eachprioritizationapproachfunctions, whilethestrengthandweaknessserve asthebasisforimprovement. RQ2.2 Howweretheseapproachesappliedand

howdidtheyaffectTCPresults? RQ3 WhataretheprocessesinvolvedinTCPin

regressiontesting?

Thisresearchquestioncanalsohelpto illustratethebasicprocessofTCP executioninalldifferentapproaches. RQ4 Whataretheevaluationmetricsand

suitabletypesofartifactsinvolvedinTCP withthereasonsfortheircreation?

Thisresearchquestionhelpsother researcherstochoosewhichevaluation metricissuitablefortheircontrolled experimentorcasestudy.

more articles from workshops used forthe primary studies. The remaining repositories areequally importantastheyhost journal articles that are related totest case prioritization.There are also important journals extracted from IEEE Xplore and ACM Digital Library.

3.4. Searchstringidentification

SLRisawell-knownreviewtechniqueforreviewingthe litera-turewithanextensivesearch aspectofthesubjectinthe

discus-sionfromallrelevantsources.Therefore,asystemicmethodto for-mulatesearchkeywordsinthisSLRconsistsofthefollowingsteps:

a)DeterminationofsignificanttermsbasedonRQs. b) Determinationofequivalentwordsforsignificantterms. c) Determinationofkeywordsinapplicablestudies.

d)UsageoftheBoolean‘OR’and‘AND’operatorsasanalternative linkbetweenterms.

Asthefocusistoexaminerelatedstudiesinregressiontesting, TCParea tobe precise, theresults fromprevious studies are uti-lizedinordertodeterminesignificantstudies. Theauthors

inten-Table5

Mappingofresearchquestionstouncoveredfindingwithitssignificance.

Researchquestions Uncoveredfindingsanswered Extrafindings Significanceofthefindings RQ1 ReasonsbehindthetrendsofeachTCP

approaches

DistributionsofapproachesinTCP techniquesanditslogic

Toprovideinsightsinthedevelopment ofTCPinregressiontestingwithin therecenttrends.

RQ2 Adetailedoverviewofotherrecent popularTCPapproaches

EmpiricalevidenceforotherrecentTCP approaches

Thestrengthandlimitationsofexisting prioritizationapproaches

Toprovideaglimpseofhoweach prioritizationapproachfunctions,and serveinformationforimprovement RQ3 – ProcessesinvolvedinTCP ToillustratethebasicprocessofTCP

executionforimprovement RQ4 Anuncoveredevaluationmetricavailable

withitsreasonsofcreations

GapsregardingtheusageofartifactsonTCP approach

MetricmostlikelyusedforspecificTCP approach

Informationofavailableevaluation metricsandartifactswiththe reasonsofcreationandrelationtoa specificTCPapproach

Fig.2. Searchstringsforselectingstudiesfromallrepositories.

tionallyusedtestcaseprioritizationastheexactphraseinmostof thesearch queries since thereare numerous research worksthat arerelatedtotestingtestcasesinregressiontesting.

As illustrated in Fig. 2, different search strings were used in eachrepository.Withaspecificgoaltoretrieveconceivable signifi-cantreviews,theauthorsutilizedtheseterms:“testprioritization”, “regressiontesting”,“test caseprioritization”,and“regressiontest prioritization”.It ought to be noticed that ifthe authors utilized anexactphrasesuchas“testprioritization” alone,itwillreturnan excessivenumberofunimportantreviews.Duetothis,‘AND’ oper-atorwasusedtolink“testprioritization” phraseand“testcase pri-oritization” phrasetoincorporate themintoan alternative search term.Asaresult,quiteanumberofrelevantstudieswereextracted

fromthe combination ofthe phrases. Torefine search outcomes, theauthorsused‘OR’operatorforthephrasesthatappearin doc-ument titles and authorkeywords. The range ofyears published wassetstartingfrom1999until2016.

3.5. Studyselectionprocess

Asmentioned inthe previoussection, SLRrequiresto be con-ductedinanappropriatemannerinordertoproduceahighimpact review tothe researchdomain itself.The SLRsearch processwas initiatedduringtheselection oftherepositories.Thefirst process wasto identifyseveral popularrepositories inTCP research area. The next stage was a search stage, where, an exhaustive search wasperformedonallsixselectedrepositoriesandallthe

prospec-Fig.3. Searchandselectionprocess. Table6

Inclusionandexclusioncriteria.

Inclusioncriteria Exclusioncriteria

Englishaspublicationlanguage Non-Englishaspublicationlanguage

Paperfocusingontestcaseprioritizationapproaches PaperdoesnothaveanyrelationwithTCPapproaches Paperwithcompletebibliographicinformationfrom1999to2016 Paperwithoutbibliographicinformation

Paperisabletocorrespondtoatleastoneresearchquestion Identicalstudies(latestpaperisincluded)

tivepaperswereassembledtogethertoeasetheselectionprocess. Then,alltheprospectivepapersunderwentaselectionstagein or-dertochooseforrelevantpapersthatweregoingtobeusedinthe primary studies.Fig.3illustratesthesearch andselection process conductedforthisSLR.

From Fig. 3, 707 potential studies were identified from the search stage. To narrow down on the number of the papers to bereviewed,alltheprospectivestudieswererequiredtobe justi-fiedtogetthemostsignificantstudies.Firstofall,theprospective studieswererequiredtogothroughinclusionandexclusion crite-ria.Thisprocesswasessentialtoremove duplicatesandunrelated studies. Adetailed overviewofthe inclusion andexclusion crite-ria utilized forscrutiny is shown in Table 6.The first criteriato besatisfiedwerepapersprintedandissuedinEnglishonlywillbe chosen. Studies thatwere not available inEnglish were removed. Then, their abstract wasbriefly studied.The paper that doesnot haveanyassociation withresearch questionswere excluded from themajor studieslist.Forduplicated papersthatappeared in dif-ferentcopies,themostrecentoneswouldbe themostcompleted and improved copies. They were selected while the others were removed.

Aftertheinclusionandexclusionstage,qualityassessmentwas applied. Thequality evaluationofthechosen studies was accom-plished by utilizing a weighting approach to examine significant studies that are adequate enough toanswer all the RQs.The au-thors articulated five assessment questions shown in Table 7 to

Table7

Qualityassessmentquestions. No Question

1 Weretheresearchobjectivesstatedprecisely? 2 Weretheplannedapproachesstatedclearly? 3 Wastheexperimentalstrategyappropriatelydesigned?

4 Didtheexperimentapplyonacasestudyoracontrolledexperiment? 5 Doestheexplorationenhancethescholarlyworld?

evaluate the comprehensiveness, reliability, and applicability of thenominated papers.Three optionalanswers withtheir respec-tive score were given for each question: “Yes”=1, “Partly”=0.5, and“No”=0.Subsequently,variouspaperswererejectedfromthis qualityassessment.Consequently,only80paperswere chosenfor theprimarystudies.Thetotalscoresofthesechosenstudiesis por-trayedinTableA1inAppendixarea.

Upon the completion of this selection stage, 80 studies were identifiedto manifestthecapability toanswerall oftheresearch questionsderivedearlier.

3.6.Datasynthesisandextractionmethod

Theprinciple ofdatasynthesis istosimplifyevidence presen-tationfromthenominatedpaperstoeasedataextractionprocess. Thisisinordertoansweralloftheresearchquestions.80selected primary studies underwentan additionalinspection with respect

Table8

Contentsassessmentmeasures. Nominatedpaper Description

Paperreferences Papertitle,publicationyear,andsources

Typeofpaper Journalarticle,conferenceproceeding,symposiums,andworkshop Paperfocus Mainideas,complications,inspirationandpurposes

Researchmethodology Casestudy,experimentation,reviews,andliteraturesurveys Applicationdomain Depictionofthespecificsituation

Constraints Study’slimitationsforfutureimprovement

Table9

Datacollectionforresearchquestionsframed. Researchquestions(RQs) Typeofdataextracted

RQ1 RQ1.1 Typesofregressiontestingtechniques,studies, andbibliographicreferences.

RQ1.2 TypesofTCPapproachessolutiontypesusedto conductthetechniques.

RQ2 RQ2.1 Briefdescriptionandtestpurposeofeach approach,formalhypothesis,teststrategy, threatstovalidity.

RQ2.2 Formalhypothesis,advantages,andathreatto validity.

RQ3 Tool/environmentforexperimentalsetup,data collectionmethod,process,andevidence typeormeasurefortheeffectivenessofTCP approaches.

RQ4 Evaluationmetricsandsoftwareartifacttype.

Fig.4. Percentageofcollatedstudies.

tothecontentassessmentmeasuresasshowninTable8to deter-minethespecifiedmattersforeachpaper.

Thepointofthismeasureistosynchronizetheprimarystudies andenhancetheassessedpapersforclarity.Thiswillhelpforthe purposeofdataextractionfrompaperswhichresponseexactlyto theresearchquestions.Briefimportanttypesofdatashowingthe mappingof synthesizeddata to research questions are shown in Table9below.

4. Resultanddiscussion

This section outlines the results with respect to the research questions.Thesummaryoftheprimarystudieswaspresentedfirst, followed by each research question, answered in different sub-section.

4.1.Overviewofprimarystudies

80 primary studies in total were nominated for this review. Fromtheprimary studies,there were44 journalarticles,26 con-ferencepapers,fiveworkshoparticles,andfivesymposiumarticles. ThepercentagesofthecollatedstudiesarepresentedinFig.4,and thetotalpapersissuedperyeararepresentedinFig.5.Asforthe overviewsofthe primary studies,the informationis tabulated in TableA2inAppendixsection.

4.2. WhatistheresearchtrendofTCPtechniquesinregression testing?(RQ1.1)

Asprioritizationontest caseshadonlygonethroughtestcase selectioninearlystudies[3],TCPwasthensuggestedandassessed inabroadercontext.ThefirstaspectofRQ1wastodeterminethe current trend of TCP techniques’studies. Referring to Fig. 5, the numberofpaperspublishedthroughtheyearshowsapositive in-crementbeginningfrom1999until2016.Itcanbeconcludedthat TCP has been recognized as an important element in regression testingamongresearchers.

In realworld scenarios, it is quitehard torealize whichtests will actually detect faults. Hence, that is why test case prioriti-zationapproachneeds tohaveotherapproachbackups,expecting that acertain numberofbackupapproacheswillendinboosting faultdiscovery indifferentways.There aremanytestcase priori-tizationapproacheswhich havebeen proposedby researchers.As manyaseightbroadapproachesweredescribedbySingh[15].All these TCP techniques approaches were originally grouped based on some commonalities such as phases selected, available re-sources(requirement,testcases),anddesiredoutput(time execu-tion,APFD).Forexample,Requirement-basedTCPcanbeinitiated duringorattheend ofrequirementphase,by usingthe require-mentresourceitself.Fig.6,showsotherdiscoveredapproaches.

As shown in Fig. 6, the authors grouped the TCPapproaches intosevenmaindimensions,whichseemtobepopularamong re-searchers. Thesecategories are reportedin recentmapping stud-ies and they have various commoncharacteristics [14]. However, the authors combined some approaches under ‘others-based’ di-mension,whichwerenotreallypopularamongresearchers includ-ingtopicmodelbased[18],multi-criteriabased[19,20],workflow based [21] and others [22–24]. Each approach presents different processanddatasetinperformingregressiontesting.

4.3. WhatarethedistributionsofapproachesinTCPtechniques?(RQ 1.2)

The second aspect of RQ1 is related to TCP techniques cate-goriesthathavebeenproposedbypreviousresearchers.The distri-butionsforeachapproacharedepictedinFig.7,whileFig.8shows themostutilizedapproaches.Thelistofcitationsforeach discov-eredTCPapproachistabulatedinTableA3inAppendixsection.

Theresults showedthatsearch-based TCPisthemostutilized methodamongthecollatedstudies.Itwasusedin25%ofthe stud-ies.Withinsearch-based TCPitself,there wereseveralalgorithms used including Genetic Algorithm (GA) [25–33], Greedy [34,35], Ant-Colony [36–38], StringDistance [39] and others[40,41]. Sev-eral observations are noted for artificial intelligence (AI) utiliza-tion. First, there are many publications on AI application and it is used in solving different problems contexts. Second, empirical dataiseasilyavailable forAIexperimental setup.Thisencourages researcherin executing andcompiling results using search-based approach.

The second largest portion of the reported approach in the primary studies is coverage-based TCP with an 18% distribution

Fig.5. Numberofpaperspublishedperyear.

Fig.6. ThetaxonomyofTCP.

[3,29,42–49,49–52].Coverage-basedTCPwasthefirstproposed ap-proach that resulted in high utilization until this day. Coverage-based TCP’smainidea isto covera 100%coverage ofthe system during regression testing, which logically will detect all possible faults withinthesystem. However,togain a100%coverage,there is a time cost that needs to be considered, as testing execution time willbedraggedandmoreresourceswillbe required.Thisis the motivationthat hasled morepublications oncoverage based toimproveitfromtimeandresource-relatedcostsaspects.

Fault-basedTCPcomes asthethird mostutilized approach re-portedintheprimarystudies[53–60].Theauthorsbelievethis ap-proachhitquitea numberofpublicationsince fault-based priori-tizationapproach wasmentioned byRothermel[3],intheirwork inTCP.Thefault-basedapproachisabletoconstructaspecifictest suitethat aims todetect theexactly estimatedfaults [58], which mayeaseresearcherstotargetonfaultsthatshouldbedetected.

Requirement-based and history-based TCP share fourth most utilized approach in TCP studies with a 9% distribution respec-tively. From the primary studies, requirement-based TCP as

ap-Fig.7. Percentagesofcollatedstudiesapproaches.

Fig.8. Mostutilizedapproaches.

pearedin[52,61–67]utilizerequirementsinformationtoprioritize testcases.Asystemisbuiltbasedonits requirements.Therefore, byusingtherequirementsinformation,itmaypossiblyaidto clas-sifyseveralcrucialtestcasesmorethanjustbyusingcode-related data.Thisstatementmayhavebeenthefactorfora requirement-basedapproachtobeconsidered.Ontheotherhand,history-based TCPappearedin[38,68–73],whichutilizepreviousdataor histori-caldatainordertoprioritizetestcases.Basedontheobservation intheprimary studies,history-based utilizationfactors hasto do withitsdatathatareeasilyavailableforanysearch-based experi-mentalsetup.

Risk-basedTCPwerereportedwitha7%distributioninthe pri-marystudies,whichappearedin[65,66,74–77].Theycanbeplaced undertwo categories,either requirement risk ormodel risk. For requirementrelatedrisks,theyusedrequirementtocalculatetheir riskindexforeachrequirement,whileformodelrelatedrisks,they

calculatedriskforeachnode inamodeldiagram(i.e.activity dia-gram,statetransitiondiagram,etc.)andprioritizetestcasesbased onthesecalculatedrisks.Ontop ofthat, BayesianNetwork-based TCPappearedin[78–81],whichisalmostsimilar tomodel-based TCP,witha5%distributionintheprimarystudies.Insteadofusing themodelinformationonly, BayesianNetwork-basedTCPforesees thepossibilityofeachtestcasetodiscoveranyerrorbymeansof differentaccessibledata[81].

Forothers-based TCPwhichdid not seemto be popular, they were available in the literature, in the quantity offour or fewer. As mentioned before,the authorscombined all theseapproaches under others-basedTCP whichare not reallypopular among the researchers such as cost-effective based [60,73,82], topic model based[18],multi-criteria based[19,20],workflow based[21],and others[22–24].The authorsalso tabulated all thenumber ofthe papersforeachapproachdepictedinFig.8.Itshowsthehierarchy

Table10

OverviewsofTCPapproaches.

Approaches Description

Search-based -Searchbasedapproachcanvaryintermsofimplementation

-Forexample,Greedyalgorithmwasfoundhighlyrelatedtocoverage-basedobjective[35,45] -TheapplicationofAIstrategymaydiffer,basedontheselectedtestsuite,inputcriteria,fitness function,andothers

Coverage-based -Thisapproachisafundamentaltestingapproachandinspectsthecodedirectly[43]

-Code,statement,branch,orfunctioncoveragelieswithintwomaincategories:totalandadditional coverage[7,45]

Fault-based -Fault-basedprioritizationapproachwasinitiallymentionedbyRothermel[3]

-Fault-basedstrategiesproduceasequenceoftestcasestodetecttargetedfaults[58],whichmayease researcherstohaveatargetonwhichfaultsthatshouldbedetected

Requirement-based -Requirement-basedutilizesrequirementsinformationduringrequirementelicitationsuchas customerpriority,errorpronevalue,volatilityvalue,andexecutiondifficultiesforprioritizationcriteria [61,67]

History-based -History-baseduseshistorydataasaninputtoexecutetestcaseprioritization

-Basedontheobservationintheprimarystudies,history-basedutilizationfactorshastodowithits datathatiseasilyavailableforanysearch-basedexperimentalsetup

Risk-based -Thisapproachismainlyusedinaprojectthattypicallyconcernsonriskvaluesrelatedtothe softwaretobedeveloped[61]

-Basedontheprimarystudies,riskscanbecalculatedfromrequirementitselfandsystemmodel Bayesian-Network-based -UtilizationofBayesianNetworksforeseesthepossibilityoferrortobediscoveredineverysingletest

casebythemeansofdifferentaccessibledata[81]

Cost-Effective-based -Cost-awaretestcaseprioritizationtechniquewasreportedbyHuang[73],whichintegrateshistorical datatoreducetestingcost

Topic-Model-based -Techniquethatuseslinguisticdatasuchasidentifiermarkers,remarks,andstringliteralstohelp differentiatetheirfunctionality[18]

ofthemostutilizedapproachesinTCPandwithinit,search-based andothersbasedapproacheswere separatedinto theirrespective categoryforeasieranalysis.

4.4. Whatarethedescriptions,strength,andweaknessofpresent prioritizationapproaches?(RQ2.1)

The overview of description for each TCP approach as illus-trated in Fig. 7 is listed in Table 10. The overview of these ap-proaches are important, as it allows researchers to gain insights onhoweach prioritizationapproachworks.However,forstrength andlimitations ofeach approachin TCP, they aredetailedout in TableA4inAppendixsection.Theoverviewsofthesestrengthsand weaknessesserveasamotivationforresearcherstolookfora po-tentialresearchareathatcanbefocusedon,inthefuture. 4.5. HowweretheseapproachesappliedandaffectTCPresults?(RQ 2.2)

Inordertoanswerthisquestion, theprimarystudieswere ex-amineddeeperintotheirexperimentalsetupandresults.Foreach approach, the authorselaborated certain work to give a view on howtheapproacheswereappliedandtheireffectontheTCP pro-cess.Thenextsub-sectionswilldiscusseachapproach.

4.5.1. Coverage-basedTCP

Thisapproachisafundamentaltestingapproachwhichinspects the code directly [43]. The fundamental measures such as func-tion, branch, and statement were the most widely used criteria in coverage approach [45]. Allof thecriteria were evaluated and canbecategorizedintotwomaincoverage-basedprioritization ap-proaches[7].Thosecoveragebasedaretotalandadditional cover-agebased.

The total coverage approach organizes the test cases which manage to coverthe mostportions of thesystem accordinglyby the mean of some preferred criterion. In the event that the test cases bear a similar coverage area, the conflict will be resolved arbitrarily.While foradditionalcoverage approach,themain idea of this approach is to attain a complete coverage earlier. There-fore,thisapproachdiffersfromthefirsthighestcoveragetestcase.

Thenexttestcaseswillconsiderthehighestcoverageofuncovered codeorstatement.

4.5.2. Requirement-basedTCP

Asystemisbuiltbasedonitsrequirements,therefore,byusing therequirements information itmay possibly aidto classify sev-eralcrucial test casesmore thanjust byusing code-related data. Thework in [67],assumes that prioritizationof requirementsfor test(PORT)isaprocessofestablishingthevalueofimportanceof asoftwarerequirement. EvaluationofPORT isbasedonfour crit-icalfactors: clientpriority,errorprone value,volatility value,and executiondifficulties.RecentstudybySrikanth[61],reportedthat acombinationoftwoormorefactorsmayproduceabetterresult thanasinglefactorprioritizationintermsoftestingeffectiveness.

Theworkin[62],proposedaprioritizationalgorithmbasedon traceability, completeness, the impact ofa fault in requirements, changesinrequirements,customer priority,anddevelopersviews. Theapproachwasclaimedtoyieldabetterresultintermoffault detectionrateincorrelationwithrandomlyorderedapproaches.A prioritizationapproachthatusesstaticdataandrequiresaprecise mappingfromdiscretetestcasemaynotbereliable.

4.5.3. Search-basedTCP

Search-basedprioritizationapproachhasquiteanumberof im-plementationalgorithms such asGeneticAlgorithm(GA) [25–32], Greedy[34,35],Ant-Colony [36–38],StringDistance[39] and oth-ers[40,41]. Experiment by Li [35] statedthat GA application ap-proach works poorer when compared to a greedy algorithm on computer-generated data. However, the application of a search-based algorithm may differ, based on the selected test suite, in-put criteria, fitness function, and others.While the collected re-sultshowedthemajorbenefitofGAapplicationinTCPapproaches, therearecertaindisadvantagesthatexist,suchas, executiontime isavastanxietyforGAapplicationsandtheyaretypicallyslowin theprocessofcompletion[28].

4.5.4. Risk-basedTCP

Risk-basedprioritizationapproachwasmainlyusedinaproject thattypicallyconcernsonrisk-relatedvalueswiththesoftwareto be developed [61]. Due to its prominence and practicality, some

researchers used requirements risk information in TCP process tocategorize crucial test cases that were expected todistinguish thefaults related tothe system’srisks [66,75]. The work of Het-tiarachchi[65]proposedfivemainstepstoprioritizetest casesby using requirement risks values. Requirements priorities and risk valueswere calculated within the first foursteps, while the test caseprioritization process only startedupon obtaining the result from the first four steps. It claims to have the ability to detect faults earlier since it is capable of being executed prior to code availability.

4.5.5. Fault-basedTCP

Fault-basedstrategiestrytoproduceasequenceoftestcasesto detectcertainlytargetedfaults AccordingtoRothermel[3],a spe-cificfault canbe revealedwhenexecuting aparticularstatement. Therearealsopossibilitiesthattheparticularstatementcanreveal othererrorinothertestcases.Infault-basedprioritization demon-stratedbyYuandLau[59],theyproposedFaultAdequateTestsize (FATE)focusing inthespecification. FATE isan effectiveness met-ricusedtodeterminethesizeoftheminimal faultadequate sub-set.FATEvalues rangebetween0andn.The lower theFATE, the higheristhechanceofdetectionofallaimedfaults.Theirtestcase prioritizationcan be implementedbeforethe codeinformationis available.

4.5.6. History-basedTCP

History-basedprioritizationuseshistorydataasaninputto ex-ecute test case prioritization. Work presented by Khalilian [71], claimedthat their improved methodin prioritizing test cases by includinghistoricaltest datamanage toaccelerate fault detection rate.Theimprovedmethodisanextensionoftheexisting history-based TCP approach put forward by Kim and Porter [72]. Their suggestedwork analyzes the weight for each test case by utiliz-inghistorydataincludingthecount ofexecutionswhichrevealed afault.ThisproposedtechniquewasmeasuredusingAFPDc utiliz-ingSiemenssuiteandSpaceprograms.Theirtechniqueshowedan improvementovertheir previous method ofperforming selection probability calculation.It is agreedthat historical based isan ef-fectiveTCPapproach, however,thereare some limitationsto this approach,includinglimitedhistoricinformation availabilityanda smallnumberofdefectinformation.

4.5.7. Bayesian-Network-basedTCP

For Bayesian-Network-based (BN) approach, a Bayesian Net-work’smodelforeseesthepossibilityofeachtestcasetodiscover any error by means of different accessible data [80]. Work pre-sented by Mirarab and Tahvildari [80], proposed a BN approach byaddressing threeTCPdifficulties: (1)determiningthe different collectionofproof fromthecodingpart,(2)incorporating allthe dataintoasoleBNmodel,and(3)utilizingprobabilistic interpre-tationtoassessthe possibilityofaccomplishment. Asthe results, this case study claimed to perform better than a random tech-nique.However, standard executionof BN is not atthe finest as error-pronedataarenotincludedanddoesnottakeadvantage of anyfeedback.

4.5.8. Cost-aware-basedTCP

A cost-aware-based test case prioritization techniquewas re-ported by Huang [73]. In their proposed work, Modified Cost-Cognizant Test Case Prioritization(MCCTCP) was utilized to esti-matetheunitsoffaults discoveredper unittestingrateby utiliz-ingGenetic Algorithm(GA).Theirexperimental setup was imple-mentedonflexandsedUNIXutilityprograms.Theycomparedtheir techniques against random, optimal, historical fault, time-aware,

Fig.9. StandardflowofTCPtechnique.

totalfunctional coverage,andcost cognizanttotalfunction cover-age.Theirprioritizationwasclaimedtohavea bettervaluein av-eragefaultdetectionrateifthenumberofgenerationsofsoftware codeishigher.

4.6. WhataretheprocessesinvolvedinTCP?(RQ3)

Softwareengineering highlyconcernson how theengineering processes are applied to software development in a systematic way.Therefore,itis necessarytohavethisRQto beinvestigated. Inordertoanswerthisquestion,theprimary studieswere exam-ined furtherintotheir experimentflow. Every experimentshould have their own process, however, inthis SLR, processes involved in TCPare only highlighted from severalstudies. The reason be-hindthisisbecause,numerousworksexhibitsimilarprocessflow, withtheonlynotabledifferenceintermsoftheadditionofone ex-trasteptoanexistingprocessflow.Fromtheprimarystudies,the authors determine 19 processes proposed by several researchers. All these19 works were selected astheir processes were clearly presented.Therewere alsootherclearlystatedprocesses inother studies, butthey were not coveredby the RQ’s.Each ofthe pro-cessisdetailedoutinTableA5inAppendixsection.Basedonthe 19processes realized,theauthors illustratethe basicflowofTCP processasshowninFig9.

As showninFig 9,TCPprocess startswith thepreparationof targeteddata.Eventhoughthereisnosinglepaperstatingclearly thattheTCPprocessstartswiththepreparationofdata,itis com-pulsory for any experiment or research to identify which infor-mation ordatathatwill beused. Thedataorinformationin TCP canbeintheformofrequirementstatement,systemmodels,and sourcecode.Theprocessisfollowedby,determiningand calculat-ing prioritization criteria ordependency based on the data cho-sen.From theprimary studies,Bryce [49]andEghbali[41] deter-mine,first,the testcasesthat havemostcoverage andwhichare uniquein coverage-basedapproach.While inrisk-basedapproach in[65,66,74],the riskvalue relatedto eachtest casewasdefined andcalculatedbeforeanyprioritizationwasperformed.After prior-itizationcriteriadeterminationstage,nextisprioritizationprocess. Testcaseprioritizationactivitycanonlybestartedafterthe crite-riahave been determined ormeasured. As in requirementbased approach in[61,67],prioritization oftest cases start afterthe re-quirementcriteriasuchascustomerpriorityisdeterminedand cal-culated. Inthe work ofMa [63] and Wang[56], TCPprocess can berepetitiveuntilallrequirements,testcases,orprescribedfaults are fullycovered orsatisfied.Finally, theresultis monitoredand theperformanceofacompleteTCPprocessismeasured.However, eachapproachmaystartthisprocessinadifferenttimelinewithin

Fig.10. TCPevaluationmetricstype.

aprojectschedulebasedonitschosendata.Forrequirement state-mentdata,TCPcanstart duringorrightafterrequirementphase. For systemmodels’ data, it startsright afterthe design phase is completedandsourcecodeneedstowaituntilitisavailable. 4.7. WhatistheevaluationmetricusedinTCPalongwiththe softwareartifact?(RQ4)

To answer thisresearch question, this section is divided into two sub-sections,comprisingevaluationmetricandsoftware arti-factsub-sections,withtherelationshipamongthem.

4.7.1. Evaluationmetrics

It isessential foranyapproachesproposed intestcase priori-tization toperform metricmeasurement toassess their effective-ness.Evaluationmetricisimportanttomeasuretheefficacyofany proposed TCPapproaches inprioritizing test casesandto bench-markitseffectivenessagainstother existingapproaches.Fig.10 il-lustratescurrentwidelyutilizedevaluationmetricsinTCParea.

FromFig.10,itshowsthatthemostwidelyusedmetricisAPFD with a 51% distribution, followed by Coverage Effectiveness (CE) 10%,APFDc(APFDwithcost consideration)9%,timeexecution7%, and others 23%. Average Percentage of Faults Detected (APFD) is evidently the most utilized metric favored by researchers in the primary studies. This metric was originally introduced by [3] in early TCPresearch,andlater usedmassively byother researchers [18,19,21,23–32,36,37,39,42–52,62,63,65–67,74,75,82–88].APFD isa metricusedto quantifyonhowquickanarranged andoptimized test suite can discover defects [3,89]. The result of APFD ranges from zero to 100, where a greater value indicates a better fault revealing rate. From this metric, researchers have come up with othermetricthatevolvesAPFDmetricbyconsideringotherfactors suchascostortimeconstraintstofulfilldifferentobjectivesof pri-oritization.

Coverageeffectiveness (CE)comesasthe secondmostutilized metric [18,21,23,24,28,39,48,49,55,56,58]. CEisa metricthat inte-grates the size oftest suites andthe coverage ofeach test case. It is the ratio between the size of the whole test suite andthe coverage of reordered test suite that reveals all faults or meets all requirements[90].CEvalues rangefromzerotoone, wherea higher value indicates a better effectiveness in coverage. Next is APFDcwitha9% distribution,whichisanAPFD comparable met-ric withan inclusionofa cost factor.APFDchasbeen utilizedby researchers [15,23,28,49,56,69,70,73,82,83], to measure the effec-tivenessof faultdetection rateovercost. Thismetric isthe most widely used metric in history and fault-based approaches, since history-based concerns on improvising previous testing that

in-Fig.11. TCPrelatedartifactassessed.

cludes cost factor, while fault-based focuses on prescribing fault thataremostlikelytobethecostlyone.

Timeexecutionmetricisusedby 7% ofthedistribution of re-searchers[18,25,28,36,37,39,58,88]inthisSLR.Timeexecution met-ricisprimarilyusedinsearch-basedTCPapproachtoverifythe ef-fectivenessofa proposed algorithminreducing TCPtime. Finally, othermetriccomprises a23% distribution,whichincludesseveral types of metric available. Most of the metrics are adapted from APFD,whichhasbeenturnedintoASFD,WPFD,TSFD,APBC,APDC, APSC,andNAPFDtoanswerdifferentTCPobjectives.

To further support the significance of all evaluation met-rics, ANOVA test has been utilized by several researchers [18,22,32,39,42,44,46,50,51,54,61,79,81,82]. Analysis of variance (ANOVA), is a statistical method that was introduced by Fisher [92].ANOVAisastatisticaltestthataimstoevaluatethedifference ofthemeansofthreeormoregroups.Thegroups’datamustbein numericaltype. ANOVA hypothesizesthe findings withboth null hypothesisandalternative hypothesis.A nullhypothesisis where alltreatmentsarehavingequalvalueswhilealternativerejectsthe nullhypothesis.ANOVAtestinTCPhsbeenusedtoverifythe im-pactofthe proposed workby measuring a statisticalsignificance of the investigated approach,by comparing any metrics, such as APFDorexecutiontime.

4.7.2. Artifact

An artifact is required in completing a controlled research in testingpractices. Artifacts can be inthe form oftest cases, soft-ware, coverage information, software requirements, fault records, historyevidence,andothers.Theutilizationofartifactdependson thetypeofresearchanditssuitabilitytotheexperiment.Acareful examinationoftheartifactsutilizedbydifferentTCPapproachesis presentedby Singh[13].The studyhighlighted anumberof arti-factsassessedby researchers.InthisSLR, thedistribution of arti-factsutilizedispresentedalong withtheirusage indifferentTCP approaches.Fig.11showstheTCP-relatedartifactsassessedby re-searchers.

On the subject of the artifacts assessed, our results conform thefindings reportedin [13], wherea 50%distribution ofthe ar-tifactsare accessiblewithno restrictionfromSoftware Infrastruc-ture Repository(SIR) [91]. In this SLR, a 36% distribution of the software artifacts has been extracted from SIR, as illustrated in Fig.11.Meanwhile,realcasestudyhasbeenusedinseveral stud-ies [42,43,47,49,52–55,61,63,65,67], which is considered as an ar-tifact in this SLR. The artifacts fromSIR are mainly divided into three typescomprising, siemen,space, andTLS. Siemen set con-sistsofsevenbenchmarkprogramwhichiswrittenCprogramming language.

Table 11 shows the list of programs developed by Siemens Corporate Research, which constitutes a 17% distribution of

ar-Table11

Siemenbenchmarkprogram. Program LOC Description

tcas 173 Aircraftcollisionavoidancesystem schedule2 374 PriorityScheduler

schedule 412 PriorityScheduler

tot_info 565 InformationStatisticsMeasure printokens 726 LexicalAnalyzer

printokens2 570 LexicalAnalyzer replace 564 PatternReplace

Fig.12. UtilizationofSIRprogramsinTCPapproaches.

tifacts assessed by researchers [3,13,28,34,39,44,45,66,88]. The next popular SIR program among researchers is Test Specifi-cation Language (TSL) with an 11% distribution, as cited in [13,23,24,27,28,44,58,73,82,88].Thiscategory isnamedTSLaftera TSLtoolthat isusedinexecutingtheprogram. TSLprogram con-sistsoffourprogramswhichareflex,grep,gzip,andmake,where all of them are UNIX utilities. The last SIR is a ‘Space’ program with9564linesofcode(6218executables),whichmakesup8%of thetotalartifactsassessedandappliedin[13,27,28,33,34,44,45,71]. Thespaceprogramhasbeenusedtovalidatethecontentwithina fileandfunctionsasaninterpreterforanarraydefinitionlanguage (ADL) grammar. In short, SIR provides information for each of C object,excluding requirement specification documentation. Over-all,SIR programs have beenwidely utilized in TCPapproach, for studiesthatarenotrelatedtorequirements.Fig.12showsthe uti-lizationpercentageofSIRprogramacrossdifferentTCPapproaches. FromFig.12,wecanseethat68%ofutilizationofSIRprograms havebeen utilized under search-based TCPapproach [25–40,93], while16%ofthestudieshaveutilizedSIRprogramsinfault-based TCP[56–60].Other approaches that utilizeSIR include, coverage-based4%,history2%,andothers10%.Thereasonbehindthis dis-tributionofstudies concerningutilizationSIR isbecause SIRonly providesartifactsintheformofsource code,testsuites,fault in-formation, coverage information, and some history evidence. An artifactsuch asrequirementis not provided,which explainswhy requirement-based TCP does not utilize SIR programs. Since SIR doesnotprovidesuchinformation,arealcasestudyorother con-trolledexperimenthasbeenusedasshowninFig.11.

5. Researchfinding

In software development, prioritizing test case is an essential activityintestingphase[3,94].Atthepointwhenclienthopesare high, promisedtime isshort, assetsare constrained,andthe de-velopedsoftware must havethe capability to fulfill the software requirementswithfewerfaults, TCPtechniqueisbeneficial to re-duce the cost, time, and software fault since TCP itself concerns toordertestcasesforearlyoptimizationinfault detection.A

de-tailed approach ofTCPis very important,not only tobe used to optimizethe testcaseexecution, butalso toaid projectmanager toorganizeprojectdeliveries,andmaybetomakesomenecessary adjustment.Therefore, the TCPimpact inregression testingmust beemphasizedmore.

Inanswering RQ1,alloftheprimary studieswere usedto an-swerthisresearch question. As forthe result, itcan be said that TCP approaches are still broadly open for improvements. There were positive increments of TCP publication starting from 1999 until 2016.New approachesin TCPare introduced constantly ev-eryyear,andimplementationofartificialintelligenceelementhas beenatrendamongresearchersinthelateststudy.However,other approaches have their own supporters, with their own advan-tages. Forexample, inother approaches, a multi-objectiveor cri-teria techniquehasshownquite a numberofsupporters ina re-cent publication [95,96] asit has the capability to tackletwo or more different kinds of objectives in one prioritization. This ap-proachcanalsobe easily combinedwithother techniques,which makesita moreinteresting andpromisingapproach.Therefore,it canbeconcludedthat TCPisrecognizedasanimportantelement inregressiontestingamongresearcherscurrently,asithasthe ca-pability to increase the effectiveness of testing in terms of fault detectionrate,cost,andtime.

ForRQ2,42%oftheprimarystudieswereexaminedthoroughly. Asa conclusion,eachapproachhasspecifiedpotential values, ad-vantages,andlimitation.The inputsanddataset typeplay an im-portant role inthe determinationoftheir advantagesand limita-tion.Forexample,therequirement-basedapproachusescustomer priority duringrequirementelicitation asinputs toprioritize and generateatest case.Risk-based mayalsouserequirementriskas one ofthe inputsto execute prioritizationprocess. This indicates thatbothmayhavetheirownadvantagesagainstotherapproaches intermsofTCPexecutionstartingpointsincebothmaystartprior tocodeavailability.The differencesintermofstrengths and limi-tationsoftheseapproachesarerequiredtogiveahintonhowTCP approachesfunctionandserveasa motivationforanychangesin thefuture.

ForRQ3,19 out of80primary studies were evaluated regard-ing their experimental setup. The implementationofTCP process shows a significant role in certain project (RQ3). TCPtechniques benefit project managers in adjusting their project schedules in orderto counter theconstraintthat exists within theproject de-velopmentprocess.VariationsinthestartingpointofTCPprocess amongtheapproachesprovideadifferenttimelineandbenefitto projectmanagertochoosewhichapproachessuitewiththeproject scheduleandavailableresources.

For RQ4,all of 80 primary studies havebeen examined thor-oughly. The evaluation metrics utilized in these primary studies withthereasons behindtheir creations havebeencovered. APFD metric remains as the main metric used in all TCP approaches. However, thereisquite anumber ofnewmetricsthat havebeen introduced, whichis adapted fromAPFD tosupport different ob-jectivesindifferentstudies.Asfortheartifacts,theevidence sug-geststhatprograms extractedfromSIR remainsthemostpopular choiceamongresearchers.However,SIRdoesnotsupport require-mentandmodelbasedTCPapproaches.

6. Threatstovalidity

ThisSLRhassomelimitationsrecognizedthatmaythreatenits validity.Similar to previous reviews, the potential threats ofthis systematic review are associated with an imperfect collection of primarystudiesandimprecisedatasynthesisandderivation.

6.1. Selectionofprimarystudies

InSection3,theauthorspresentedtheresearchmethodin de-tailinthe orderusedtoselecttherelevantstudies. However,the authors cannot completely ensure that the authors have gained all theaccessiblereviewsrelatedto TCPto answerall RQs.There are two possible issues. Firstly, a possible issue is the difficulty in finding appropriate search strings for differentrepositories. In Section3.4,detailedexplanationsweregivenfortheprocessof se-lection ofrepositories and search strings were used to find rele-vantstudiestobeusedinthisSLR.However,theremightbeother significantstudieswhichmayuseother keywords.Secondly,some significant or related studies might have existed in non-English publication.Itmissesouttherelevantnon-Englishstudiesalthough itcannotbeavoided.

6.2. Imprecisedatasynthesisandextraction

Imprecise data synthesis and fragmentary extraction process fromtheprimarystudiescouldbethenextpotentialthreattothe validityofthisSLR.Thismaybeduetounsystematicdata extrac-tionorinvalidclassificationofdata.Toreducethisproblem, man-ual scrutiny was applied to reduce the possibility of inaccurate data extraction by focusing on the data elements collected from theselectedstudiesrepresentedinSection 3.6.Furthermore,aset ofspecificqualityassessmentswereappliedtoavoidinaccurate in-clusionofdesiredstudies.

6.3. TCPrelatedfield

Withinregression testing,there are threetechniques compris-ing,TCP, TCS,andTSM.Testcaseselection (TCS)isalmostsimilar toTCPastheideabehindTCSistolocalizefault,selectrelatedtest cases,andexecuteitasasuite.However, inthisSLR,theauthors didnotdiscussthecurrenttrendofTCPinsupportingfault local-ization issues which maybe a potential threat to the validity of this SLR. To avoid this, the authors suggest that thisissue could bediscussedinfutureinSection7.Asanoverview,TCPdoes sup-port infaultlocalizationasdemonstratedin severalrecentworks [97–100].ATCP approachmayalso be turnedinto TCSafter pri-oritizationiscompleted andselectioncan bemadebasedon pri-oritization result. Work by [97] proposed a technique known as diversity maximization speedup(DMS), which ranks up program unit while performinga fault localizationforTCPandtheresults demonstrated that the technique was able to realize the aimed cost.

7. Conclusion

As the purpose of this studywas to classify andcriticize the currentstate andtrendoftest caseprioritizationapproaches, SLR

scheme was used to conduct this study. With this SLR scheme, someapplicableRQswereformulatedaccordingtotheaimofthis study. This research was conducted through finding, classifying, evaluating,andunderstandingall oftheprimary studies.The mo-tivationofthisreviewwastodiscoveranyareasorfieldsthat are likelyto undergoanysortof improvementvia systematic assess-mentofapplicableandimportantstudiesinTCPapproaches.

During the review process, the appropriate primary studies wererecognizedandevaluated.Thedatapulledoutfromthe stud-ies weresynchronized. Theauthors structured the outcomesinto tablesandfigures, whichare intended toease theunderstanding amongthe differentresearch groupsthat work on the sameTCP area.Fromthestudy,itwasdiscoveredthat:

a)Therewerequiteanumberofprioritizationtechniquesthat ex-istandimprovementsarestillrequired.

b) AIapplicationisquitepopularsinceitcanbeusedfordifferent problemsandtheempiricaldataavailabilityforAIexperimental setupishigh.

c) The data or informationclassification in TCP such as require-ment statement, systemmodels, andsourcecode needs tobe investigatedfurtherinfuture.

d)The specific time frame for the execution process of TCP for eachapproachneedstobedetailedout.

e)Earlyprioritizationintheearlystage ofa systemdevelopment life cycle is worth to be investigated further to ease project manager and developmentteam in making the necessary ad-justment.

f) Human involvementin decision-makingand estimationneeds tobechangedintoacomputerizedandautomatedreasoningto reducehumanerror.

g)HowdoesTCPsupportfaultlocalizationcomparedtoTCS?This issuecouldbe aninterestingpointtobe discussedinthenext work.

Acknowledgements

The work is fully funded by Fundamental Research Grant Scheme, vote number 4F836 and Research University Grant Scheme, vote number 11H86 under the Ministry of Education Malaysia-KPM.Wewouldalsoliketothankthemembersof Em-bedded & Real- Time Software Engineering Laboratory (EReTSEL) andSoftwareEngineeringResearchGroup(SERG),Facultyof Com-puting,UTMfortheirfeedbackandcontinuoussupport.

Appendix

TableA1

Resultqualityscoresofselectedstudies.

PaperRefs. Q1 Q2 Q3 Q4 Q5 Score

Rothermeletal.[3] 1 1 0.5 0.5 1 4

YooandHarman[7] 1 0 1 0 1 3

Singhetal.[13] 1 0 1 0 1 3 Thomasetal.[18] 1 1 1 0.5 1 4.5 Sampathetal.[19] 1 1 1 0.5 1 4.5 Sanchezetal.[20] 1 1 0.5 0.5 1 4 Meietal.[21] 1 1 1 0.5 1 4.5 Fangetal.[22] 1 1 1 0.5 1 4.5

MirandaandBertolino[23] 1 1 1 0.5 1 4.5

Koreletal.[24] 1 1 0.5 0.5 1 4

Maheswarietal.[25] 1 1 1 0.5 1 4.5

Louetal.[26] 1 1 0.5 0.5 1 4

Yuanetal.[27] 1 1 0.5 0.5 1 4

Catal[28] 1 1 0.5 0.5 1 4

KaurandGoyal[29] 1 1 1 0.5 1 4.5

Junetal.[30] 1 1 0.5 0.5 1 4 Sabharwaletal.[31] 1 1 0.5 0.5 1 4 Doetal.[33] 1 1 1 0.5 1 4.5 Debetal.[32] 1 1 1 0.5 1 4.5 Lietal.[34] 1 1 1 0.5 1 4.5 Lietal.[35] 1 1 0.5 0.5 1 4 Solankietal.[36] 1 1 0.5 0.5 1 4 Gaoetal.[37] 1 1 0.5 0.5 1 4 Noguchietal.[38] 1 1 0.5 0.5 1 4 Ledruetal.[39] 1 1 1 0.5 1 4.5 Jiangetal.[40] 1 1 1 0.5 1 4.5 Eghbalietal.[41] 1 1 1 0.5 1 4.5 DiNardoetal.[42] 1 1 0.5 1 1 4.5 Nardoetal.[43] 1 1 1 1 1 5 Haoetal.[44] 1 1 1 0.5 1 4.5 Haoetal.[45] 1 1 1 0.5 1 4.5 Zhangetal.[46] 1 1 0.5 0.5 1 4 Miller[47] 1 0 1 0 1 3 Fangetal.[48] 1 1 1 0.5 1 4.5 Bryceetal.[49] 1 1 0.5 0.5 1 4 Bryceetal.[49] 1 1 1 0.5 1 4.5 Jonesetal.[50] 1 1 1 0.5 1 4.5 Leonetal.[51] 1 1 1 0.5 1 4.5 Krishnamoorthietal.[52] 1 1 1 1 1 5 Tahvilietal.[53] 1 1 1 0.5 1 4.5 Alvesetal.[54] 1 1 1 0.5 1 4.5 Meietal.[55] 1 1 1 0.5 1 4.5 Wangetal.[56] 1 1 0.5 0.5 1 4 Qietal.[57] 1 1 0.5 0.5 1 4 Jiangetal.[58] 1 1 1 0.5 1 4.5 YuandLau[59] 1 1 1 0.5 1 4.5 Doetal.[60] 1 1 1 1 1 5 Srikanthetal.[61] 1 1 1 1 1 5

MuthusamyandSeetharaman[62] 1 1 1 0.5 1 4.5

Maetal.[63] 1 1 0.5 0.5 1 4 Arafeenetal.[64] 1 1 0.5 0.5 1 4 Hettiarachchietal.[65] 1 1 0.5 0.5 1 4 Yoonetal.[66] 1 1 1 1 1 5 Srikanthetal.[67] 1 1 0.5 1 1 4.5 Srikanthetal.[68] 1 1 1 1 1 5 Linetal.[69] 1 1 0.5 0.5 1 4 Marijanetal.[70] 1 1 0.5 1 1 4.5 Khalilianetal.[71] 1 1 0.5 0.5 1 4

KimandPorter[72] 1 1 0.5 0.5 1 4

Huangetal.[73] 1 1 0.5 0.5 1 4

Hettiarachchietal.[74] 1 1 1 0.5 1 4.5

YoonandChoi[75] 1 1 1 1 1 5

Stallbaumetal.[76] 1 1 0.5 0.5 1 4

Feldereretal.[77] 1 0 1 0 1 3

Ufuktepeetal.[78] 1 1 0.5 0.5 1 4

Zhaoetal.[79] 1 1 0.5 0.5 1 4

MirarabandTahvildari[80] 1 1 0.5 0.5 1 4

MirarabandTahvildari[81] 1 1 0.5 0.5 1 4

TableA2

Overviewofprimarystudies.

PublicationType PublicationYear PublicationName Refs.

Journal 2012 SoftwareTesting,Verification,Reliability [7]

Journal 2012 Informatica [13]

Journal 2014 SoftwareEngineering [18]

Journal 2013 EEETransactionsonSoftwareEngineering [19]

Journal 2015 IEEETransactionsonServicesComputing [21]

Journal 2014 SoftwareQualityJournal [22]

Journal 2016 JournalofSystemsandSoftware [23]

Journal 2015 IndianJournalofScienceandTechnology [25]

Journal 2011 Internationaljournaloncomputerscienceandengineering [29]

Journal 2002 IEEEtransactionsonevolutionarycomputation [32]

Journal 2006 IEEETransactionsonSoftwareEngineering [33]

Journal 2007 IEEETransactionsonsoftwareengineering [34]

Journal 2016 EmergingResearchinComputing,Information,CommunicationandApplications [36]

Journal 2012 AutomatedSoftwareEngineering [39]

Journal 2015 JournalofSystemsandSoftware [40]

Journal 2016 IEEETransactionsonSoftwareEngineering [41]

Journal 2015 SoftwareTesting,VerificationandReliability [43]

Journal 2014 TransactionsonSoftwareEngineeringandMethodology(TOSEM) [44]

Journal 2016 IEEETransactionsonSoftwareEngineering [45]

Journal 2013 IEEEtransactionsonsoftwareengineering [47]

Journal 2012 ScienceChinaInformationSciences [48]

Journal 2011 InternationalJournalofSystemAssuranceEngineeringandManagement [50]

Journal 2003 IEEETransactionsonSoftwareEngineering [51]

Journal 2009 InformationandSoftwareTechnology [53]

Journal 2016 InformationTechnology:NewGenerations [54]

Journal 2016 SoftwareTesting,VerificationandReliability [55]

journal 2015 IEEETransactionsonServicesComputing [56]

Journal 2012 InformationandSoftwareTechnology [59]

Journal 2012 InformationandSoftwareTechnology [60]

Journal 2010 IEEETransactionsonSoftwareEngineering [61]

Journal 2016 InformationandSoftwareTechnology [62]

Journal 2014 InternationalJournalofApplied [63]

Journal 2012 JournalofSoftwareEngineeringandApplications [67]

Journal 2016 JournalofSystemsandSoftware [69]

Journal 2012 ScienceofComputerProgramming [72]

Journal 2012 JournalofSystemsandSoftware [74]

Journal 2016 InformationandSoftwareTechnology [75]

Journal 2011 InternationalJournalofSoftwareEngineeringandKnowledgeEngineering [76] Journal 2014 InternationalJournalonSoftwareToolsforTechnologyTransfer [78]

Journal 2004 SoftwareQualityJournal [83]

Conference 1999 ProceedingsIEEEInternationalConferenceonSoftwareMaintenance [3] Conference 2014 IEEESeventhInternationalConferenceonSoftwareTesting,VerificationandValidation [20] Conference 2007 Proceedingsofthe3rdinternationalworkshoponAdvancesinmodel-basedtesting [24] Conference 2011 InternetComputing&InformationServices(ICICIS),2011InternationalConference [30]

Conference 2010 ComputerandCommunicationTechnology(ICCCT) [31]

Conference 2010 InternationalConferenceonQualitySoftware [35]

Conference 2015 SoftwareEngineeringandServiceScience(ICSESS) [37]

Conference 2015 nternationalConferenceonSoftwareTesting,VerificationandValidation(ICST) [38] Conference 2013 InternationalConferenceonSoftwareTesting,VerificationandValidation [42]

Conference 2013 InternationalConferenceonSoftwareEngineering(ICSE) [46]

Conference 2015 SoftwareEngineeringandServiceScience(ICSESS) [57]

Conference 2013 SoftwareMaintenance(ICSM) [58]

Conference 2013 InternationalConferenceonSoftwareTesting,VerificationandValidation [65]

Conference 2014 SoftwareSecurityandReliability(SERE) [66]

Conference 2013 EngineeringofComplexComputerSystems(ICECCS) [70]

Conference 2013 SoftwareMaintenance(ICSM) [71]

Conference 2002 SoftwareEngineering,2002.ICSE [73]

Conference 2016 ComputerSoftwareandApplicationsConference(COMPSAC) [79]

Conference 2015 ComputerSoftwareandApplicationsConference(COMPSAC) [80]

Conference 2007 InternationalConferenceonFundamentalApproachestoSoftwareEngineering [81] Conference 2008 InternationalConferenceonSoftwareTesting,Verification,andValidation [82] Workshop 2012 Proceedingsofthe2ndinternationalworkshoponEvidentialassessmentofsoftwaretechnologies [28] Workshop 2007 WorkshoponDomainspecificapproachestosoftwaretestautomation:inconjunctionwiththe6thESEC/FSEjointmeeting [49]

Workshop 2008 WorkshoponAutomationofsoftwaretest [77]

Symposium 2015 SoftwareReliabilityEngineering(ISSRE) [26]

Symposium 2015 InternationalSymposiumonSearchBasedSoftwareEngineering [27]

Symposium 2003 SoftwareReliabilityEngineering,2003.ISSRE [52]

Symposium 2016 ArtificialIntelligence,NetworkingandParallel/DistributedComputing(SNPD) [64]

TableA3

Existingapproachesandtheircitationindexes.

No. Approaches Source

1 Searchbased [25–40] 2 Coveragebased [3,29,42–52] 3 Faultbased [53–60] 4 Requirementbased [52,61–67] 5 Historybased [38,68–73] 6 Riskbased [65,66,74–77]

7 BayesianNetworkbased [78–81]

8 CostEffectivebased [60,73,82]

9 Multi-Criteriabased [19,20], 10 Topic-Modelbased [18] 11 Workflow [21] 12 LexicographicalOrdering [41] 13 Similaritybased [22] 14 Scope-Aidedbased [23] 15 Modelbased [24] TableA4

TheadvantagesandlimitationofTCPapproaches.

Approaches Advantages Limitation

Search-based-GA Acquiredresultsuchascoverageandfault detectionisthemajorbenefit[28]

SpeedexecutionofGAislikelytobeslow [28,35]duetomutationprocess Searchbased-Greedy Resultsinhighcoverageandperformanceagainst

othersearchalgorithms[34,35]

Toomuchgreedytypewithdifferentpurposes[34] Search-based

Antcolonyoptimization(ACO)

ACOmethodhasbeenconfidentlyusedinvarious optimizationproblemsthatbenefitsTCPasithas variouskindoffactorsthatmaybecombined [37]

Asantstendtofollowtheshorterpath,total coveragewouldbetheissuesforthisapproach [37]

Search-basedStringDistance Managetodetectandkillthestrongestmutantsin regressiontesting[39]

Notabletoproduceanoptimumresultwhenthere aretwooptimalsequencesproduced[39] Coverage-based Performfullcoveragewhichintheoryisableto

detectallpossiblefault[3,7,43]–[45]Coverage approachesaremuchpredictableintermsof APFD[44]

OverallAPFDscoreisstillnotoptimal[43,44] Requiresextrainformationsuchastestcasefault

historydataormodificationinformationto enhancefaultdetectionrate[42,45]Inother words,itisbettertocombinewithother approaches

Fault-based Highpossibilitytodetectthefaultsinasystem earlier(includinghiddenfaults)[53,59] Guaranteedtobefreefromprescribedfaults[56,58]

Accuracyoffaultdetectionissubjectivethatis pronetohumanerror[53]

Prescribedfaultsmightnotwelldefined[59] Requirement-based Usesofrequirementsinformationduring

requirementelicitationsuchascustomerpriority, errorpronevalue,volatilityvalue,andexecution difficultiesforprioritizationcriteria[61,67] Claimtoachievehighefficiency(earlierexecution

startingtime,nocodedependency)[61,63,67]

Suchinformationrequireshumaninvolvement resultinginapossibilityofhumanerror[61,67] Usesstaticdatathatneedstohaveanperfect

mappingmaynotreliable[62]

History-based Utilizinghistoricaldataincludingthecountof executions,thecountwhichrevealsafault[71] Acceleraterateoffault-detection[71,72]

History-basedhasseveralbottleneckssuchas historicinformationavailabilityandsmall numberofdefectinformation

Risk-based Highpossibilitytodetectspecificerrorrelatedto system’srisks[66,75]

Effectiveinearlydetectionofseriousfault[66,74]

Humanexpertdependency[75,76]

Bayesian-Network-based Canusemultiplekindsofinformationsuchas coverageinfo,faultpronenessandetc.topredict theprobabilityoffaultdetectionusingBayesian networkmodel[78,81]

Timeconsuming,requiresalgorithmimprovement [81]

Cost-Effective-based Promotecostawarenessthatleadstocost reduction[82]

Hasseveralbottleneckssuchashistoric informationavailabilityandsmallnumberof defectinformation

Multi-Criteriabased Outperformedsinglecriteriaprioritization[19] Betterinindustrialcasestudy[7]

Topic-Modelbased Canindicatetestcases’functionality,andableto identifymorerelatedevidence[18]

Fullnaturallanguageprocessingandcompletewith grammars,needtohaveatraineddatawhichis ineffective,andiserrorprone[18]

TableA5

Majorprocessproposedbyresearcher.

No Source Process

1 Miller[47] 1)Divideintotwocategoriesdependency 2)Testcasedependentswerecalculated

3)Longestpathoftestcasedependentwerecalculated 4)CalculatingTestCasePriorities

2 Bryceetal[49] 1)Selecttestthatcoversthemostunique 2)Iteratebyrandomselecttestfortiebreaking

3)Iteratetestcoverthemostnewpairsuntillastremaining 3 LeonandPodgurski[51] 1)Executethetestsbasedmaximumcoverage

2)Executethetestsbyclustersampling 3)Selecttestcasebyfailurepursuit 4)Testcaseleftorderedinrandom 4 EghbaliandTahvildari[41] 1)Choosehighestcoveragetestcase

2)Choosethehighestcoverageandnotcoveredyet 3)Repeatstep2

4)Repeatuntiltheorderingiscomplete,randomselectionforties 5 Tahvilietal.[53] 1)Specificcriteriaisdetermined

2)Calculatethecriteriavalues

3)Prioritizethetestcaseusingthecriteriavalues 6 Wangetal.[56] 1)Calculatefaultseverity

2)Selectingahighestfaultseverityoftestcaseandrecordthecoveredtestcaseandfaults 3)Updatingfaultseverityoftheremainingtestcases

4)Repeatingstep2to4untilallthetestcasesaresortedtoend 7 Jiangetal.[58] 1)Calculatecosttodiscoverfirstfaultlocalization

2)Calculatecosttodiscoversecondfaultlocalization 3)Repeatforthirdandsoon

4)Reportthemeanandvarianceofthecostforlocalizingfaults 8 YuandLau[59] 1)Determinefaultfromprescribemodel

2)GeneratetestcasesusingMUMCUT 3)Theeffectivenessisevaluated

9 Doetal.[60] 1)Obtaincoverageinformationforeachobjectprogram 2)Reordertestsuites

3)Computerateoffaultdetectionandmissedfault

4)Prioritizeagainwithfaultdetectioninformationandothercost 10 Srikanthetal.[68] 1)Analyzehistoricalfieldfailures

2)Identifyrarityofusecases 3)Tagusecases

4)Prioritizeusecases

11 Khalilianetal.[71] 1)Studyhistoricaldata(testpriority)

2)Testcasecoveragedataisgatheredwithrespecttocoveragecriteria 3)Prioritizebasedonpercentagecodecoveragewithitspriority 4)Recordthefinaldata

12 Maetal.[63] 1)Calculatedprioritiesoftestcases 2)Selectionoftestcasewithhighestvalue 3)Prioritizeandrecalculationpriorities 4)Repeatuntilallrequirementsarecovered

13 Srikanthetal.[61] 1)Getcustomerpriorityscoresandfaultpronenessscores 2)Findaverageofallscores

3)Dividedtheaverageofscoreswithsumofbothtoobtainweight 4)Computecombinedscoresbymultiplyingeachfactorscore 14 Arafeenetal.[64] 1)Clusterrequirement

2)Mappingtherequirement 3)Prioritizeincluster 4)Fullprioritization 15 Krishnamoorthietal.[52] 1)Factoridentification

2)Calculatetotalrequirementandtestcases 3)Calculatefactorvalues

4)Calculaterequirementweight

5)Analyzeandmapthetestcasetorequirement 6)Recalculaterequirementweight

7)Sortingthetestcases

16 Srikanthetal.[67] 1)Determinetherequirementcriteria 2)Determinetherequirementpriorities 3)Defectandfailuremappedtorequirement 4)Determinetherequirementcomplexity 5)Testcaseisrun

17 Hettiarachchietal.[74] 1)Riskisestimated 2)Riskrequirementcalculated 3)Riskitemcalculated 4)TCPprocessstarts 18 Hettiarachchietal.[65] 1)Riskisestimated

2)Riskweightcalculated 3)Riskvaluecalculated 4)Evaluateextrafactors 5)TCPprocessstart 19 Yoonetal.[66] 1)Identifyriskitem

2)Estimatestheriskvalues 3)Calculatetestcasepriority 4)Prioritizebasedontestcasepriority