Motivation

Sincemalwareispresentlyoneofthemostseriousthreatsto computersecuritywiththenumberofnewsamplesreaching140 millionin2015(AV-Test,2016a),battlesagainstit arefoughton manyfronts.Signaturematchingremainsthe coredefensetech- nology,butduetoevasiontechniquessuchaspolymorphism,ob- fuscation,andencryption,keepinggoodrecallisdifficultforstatic analysisandmethodsbasedpurelyonstringmatching.Apopular approachtotackletheseproblemsistoexecuteabinaryinacon- trolledenvironment(sandbox)(Oktavianto&Muhardianto,2013), monitoritsbehavior,andbasedonthisbehaviorclassifythesam- pleintobenignormalwareclass(orasaparticularmalwarefam- ily).The assumption ofthesedynamicanalysis methodsisthat behaviorshouldbemoredifficulttorandomizeand thereforeit shouldconstituteamorerobustsignal.

Most approaches to dynamic analysis rely on system calls (Ahmadi, Ulyanov, Semenov, Trofimov, & Giacinto, 2016;Naval, Laxmi, Rajarajan, Gaur, & Conti, 2015; Wüchner, Ochoa, & Pretschner,2014),astheyaretheonlymeanshowthebinarycan interactwiththeoperatingsystemandotherresources.Thispop- ularity has howeveralreadytriggered manyevasion techniques, suchasshadowattacks(Ma,Duan,Liu,Gu,&Liu,2012),system-

∗_{Correspondingauthor.}

E-mail addresses: [email protected](J.Stiborek),[email protected](T.Pevný),

[email protected](M.Rehák).

callinjectionattacks(Kc,Keromytis,&Prevelakis,2003),orsand- boxdetection(Garcia,2016).

Aperpendicularapproachtomodelingsystemcallsistomodel resourcesthebinaryhasinteractedwithtogetherwiththetypeof the resources.Therationaleisthatifmalwarewantstoprovide revenuetoitsowner,ithastoperformactions,suchasdownload- ingadvertisementsinthecaseofadware,encryptingharddrivein thecaseofransomware,exfiltratingsensitivedatainthecaseof credentialstealersorRemoteAdministrationTrojans(Chen,Wei,& Delis,2008),etc.Thisworkassumesthatexecutionoftheseactions involvesinteractionswithresourcesvisibleattheoperatingsystem level,andthisinteractioncanbeviewedasasignalwhichishard tohideandwhichcanbeindicativeofmalwarefamilies.

Modeling interactions with system resources has been al- ready exploited by the prior art. Mohaisen, Alrawi, and Mo- haisen(2015)extractsamanuallypredefinedsetoffeaturessuch asnumberoffilescreatedinspecificfolders,numberofHTTPre- quests, etc.,and useit insupervisedclassification. However,we believethattherapidlychangingthreatlandscapemakesitdiffi- culttomanuallydesignfeaturesthatareindicativewhilealsobe- ingstableovertime.Analternativeparadigmistoavoidmanual designandtouseabag-of-wordsmodel(BoWmodel),whereev- eryinteractionwith aparticularresourceidentifiedbyitsname isconsideredasauniquefeature(Rieck,Holz,Willems,Dussel,& Laskov,2008).Thepricepaidforcircumventingmanualfeaturede- signusingBoWisanexplosionoftheproblemdimension,which caneasilyreachmillions.

https://doi.org/10.1016/j.eswa.2017.10.036

J. Stiborek et al. / Expert Systems With Applications 93 (2018) 346–357 347

Thisworkcircumventstheproblemofmanuallydesigningfea- tureswhileatthesametimeavoidingtheproblemdimensionex- plosion.Theapproachistofirstclusterresourcenameswithsim- ilarityfunctionstailoredforeachresourcetype(filenames,mu- texes,registrynames,anddomainnames),andthenusethisclus- teringtorepresentasample(abinaryexecutedinthesandbox)in alower-dimensionalspace.Thisenables ustousearandomfor- estclassifier(oranyotherclassifierofchoice)toseparatemalware from legitimatesamples.The clusteringalsoeffectivelyremoves randomizationusedtoevadedetection.

The proposed approach is extensively evaluated on a large numberofsamples(morethan230,000) andcomparedtorele- vant priorart.Experimentalresults showthatthe proposedap- proachindeedimprovestheaccuracyofdetectingmalwarebina- rieswith thelow complexityrequiredfor deploymentonlarge- scaledatasets.

The contributions of this paper are manifold. This paper presents anovelapproachtorepresentingmalware samplesus- ingtheirinteractionswithsystemresources.Itemploysvocabulary- based method from the field ofmultiple instance learning (MIL) combinedwithadefinitionofnovelsimilaritymeasuresfordif- ferent typesofresourcetypesreflectingtheiruniqueproperties (structure of file paths, network communication, etc.). Next, it presentsafastapproximationofLouvainclusteringmethodused toautomaticallydefinethevocabularythatallowstoscaletheMIL methodtolargescaledatasets.Tothebestofourknowledge,this workis the firstthat employstechniques for multipleinstance learninginthefieldofmalwareanalysisandpresentsaviableal- ternativetotraditionalapproachesformalwareclassification.

2.Relatedwork

Since the analysis of malicious binaries and recommending them for further analysis has important practical applications, thereexistsrichpriorart.Althoughit isfrequentlydividedinto twocategories,staticanddynamic,theboundariesbetweenthem are blurred sincetechniques such as analysis ofthe execution graphisusedinbothcategories.

2.1.Staticmalwareanalysis

Staticmalwareanalysistreatsamalwarebinaryfileasadata filefromwhichitextractsfeatureswithoutexecutingit.Theear- liestapproaches(Lo, Levitt,&Olsson,1995)lookedfor amanu- allyspecified setofspecificinstructions(tell-tale)usedbymal- waretoperformmaliciousactionsbutnotusedbylegitimatebi- naries.Latterworks,inspiredbytextanalysis,usedn-grammod- elsofbinariesandinstructionswithin(Li,Wang,Stolfo,&Herzog, 2005).Malwareauthorsreactedquicklyand begantoobfuscate, encrypt,and randomizetheir binaries,which renderedsuchba- sicmodels(Sharif,Lanzi,Giffin,&Lee,2008a)useless.Sincere- versing obfuscationand polymorphictechniquesis intheoryan NP-hard problem(Moser,Kruegel, &Kirda,2007),moststateof theart (Ahmadi etal.,2016;Christodorescu&Jha,2006;Sharif, Yegneswaran,Saidi,Porras,&Lee,2008b)movedtoahigher-level modelingofsequencesofinstructions/systemcallsandestimating theiractionoreffectontheoperatingsystem.Therationalebehind isthathigher-levelactionsaremoredifficulttohide.Anexample ofhigher-levelmodelingisacallgraph(Hu,Chiueh,&Shin,2009; Kinable&Kostakis,2011;Kong&Yan,2013)whereverticesrepre- sentindividualfunctionsandedgescapturedependenciesbetween them.Theanalysisisthenformulatedasaproblemoffindingap- propriatesimilaritymeasure(fixedgrapheditdistance(Kinable& Kostakis,2011),normalizedmaximalcommonsubgraph(Huetal., 2009)ortrainedsimilarity(Kong&Yan,2013).Eventhoughsuch

analysisprovidesdeepunderstandingofmalwarebinaries,itsscal- abilityislimitedduetocomplexityofthesimilaritymeasures(typ- icallyNP-completeproblem).

2.2.Dynamicmalwareanalysis

Analternativesolutiontoanalyzingobfuscationandencryption istheexecutionofabinaryinacontrolledenvironmentandan- alyzingitsinteractionswiththeoperatingsystemandsystemre- sources.

Alargeportionoftheworkrelatedtodynamicmalwareanal- ysisutilizesystemcalls,sinceinmodernoperatingsystemssys- temcallsaretheonlyway forapplicationstointeractwiththe hardwareandassuchtheycanrevealmalwareactions.Thesim- plestmethodsviewasequenceofsystemcallsasasequenceof stringsandusehistogramsofoccurrencestocreatefeaturevectors fortheclassifierofchoice(Hansen,Larsen,Stevanovic,&Pedersen, 2016).Thebiggestdrawbackofthesenaivetechniquesislowro- bustnesstosystemcallrandomization.Similarlytostaticanalysis, thisproblemcanbetackledbyassigningactionstogroups(clus- ters)ofsystemcalls(syscalls)andusingthemtocharacterizethe binary(Bayer,Comparetti,Hlauschek,Kruegel,&Kirda,2009;Naval etal.,2015;Wüchneretal.,2014).Another

A wideclass of methodsidentifying malware binaries from sequences of syscalls rely on n-grams (Canzanese, Mancoridis, & Kam, 2015; Lanzi, Balzarotti, Kruegel, Christodorescu, & Kirda, 2010; O’Kane, Sezer, McLaughlin, & Im, 2013). Malheur (Rieck,Trinius,Willems,&Holz,2011)usesnormalizedhistograms ofn-gramsasfeaturevectors,whicheffectivelyembedssyscallse- quencesintoEuclideanspaceendowedwithL2norm.Inthisspace thealgorithmextractsprototypesZ={ z1, . . . , zn} usinghierarchi- calclustering.Eachprototypecapturesthebehaviorofthecluster, whichshouldmatchcorrespondingmalwarefamily.Aninteresting featureofMalheuristhatifaclusterhaslessthenacertainnum- berofsamples,theprototypeisnotcreated.Theclassificationofan unknownbinaryisdeterminedbysearchingforthenearestproto- typewithincertainrange.Ifthenearestprototypeisoutsideofthis range,thesampleisnotclassified.

To counter dynamic analysis advanced malware detects the presence of a sandbox and does not execute within it. Since most sandboxes rely on a detectable system call interposition,

Das,Liu,Zhang,andChandramohan(2016)proposetoextendhard- warewithFPGAthatwouldextractsystemcallsfromtheirexe- cutiononprocessor.Syscallsarethengroupedbycomprehensive yethanddesignedrules,andthesegroupsarethenfedintomulti- layerneuralnetworkclassifier.Theclassifieritselfisalsopartof theFPGA,suchthatthesystemcansimultaneouslyextracttraining samplesandclassifythem.

AMAL uses its custom sandbox to extract features de- scribing files, network communication and registry features (Mohaisenetal.,2015)andtunesvariousclassificationalgorithms. The main difference between AMAL and thiswork is the con- structionoffeatures.WhereasAMALusesnumericfeaturessuch ascountsorsizesofcreated,modifiedordeletedfiles,countsof created,modifiedordeletedregistrykeys,countsofuniqueIPad- dresses,etc.,we assumethatindividualresources (files,registry keys,mutexesandnetworkcommunication)havespecificrolein theoperationsystem,whichcanbedifferenteventhoughthechar- acteristicsexhibitedbythefilearethesame.

TheapproachproposedbyRiecketal.(2008)createsarepre- sentationoftheanalyzedbinariesdirectlyfromthedatawhichis atthefirstsightsimilartotheproposedapproach,howeverthere aretwokeydifferences.Thefirstoneisthesourceofdata,because Riecketal.modelactionstriggeredbythemalware(writingintoa file,communicationwithremoteserver,readingdatafromregistry keys,startingnewthread,etc.), whereastheproposed approach

348 J. Stiborek et al. / Expert Systems With Applications 93 (2018) 346–357

models onlyaffected resources.This enablestodeploythe pro- posedapproachinenvironmentswithoutaccesstolow-levelac- tions(VMswithoutsuchaccess,usermachineswithoutAPIhook- ing).Anotherdifference isinhandlingtherandomizationofre- sourcenames.Insteadofclusteringresourcenamesusedinthis work,Riecketal.removeparametersofactions,whichincreases thedimensionalityofthemodelsinceforeveryactionwithnpa- rameters itcreatesn+1featuresrepresentingthe actionatdif- ferentlevelsofgranularitybyremovingparametersfromtheend: fromfulldescriptionwithallparameterstothemostcoarsede- scriptionwhereonlynameoftheactionisused.Thisleadstoa massive increaseinthealreadylargenumberoffeatures.1 _Even

thoughtheresultingfeaturespace,issparsethescalabilityofsuch anapproachislimited.

Anderson,Storlie, and Lane(2012)propose tocombinetech- niquesfromstaticanalysiswiththedataobtainedusingdynamic analysisinordertocountertechniquesfrequentlyusedbymal- wareauthorstoavoiddetection,e.g.packingorexecutionstalling. Authorsproposesixdifferenttypesofinputdata,threebasedon techniquesfromstaticanalysis:(1)featuresextractedfromrawbi- narymodeledasn-grams,(2)opcodesextracted fromdisassem- bledbinaryand(3)controlflowgraph—agraphofallpossibleexe- cutionpaths;twobasedondynamicanalysis:(4)instructiontraces (Anderson,Quist, Neil,Storlie, &Lane,2011)and (5)systemcall traces;and(6)variousinformationextractedfromthebinary it- self suchaspackeridentification,entropyofthebinary,number ofinstructionsindisassembledfile,etc. Foreverytypeofinput authorsdefineakernelwhicharethencombinedusingmultiple kernellearning(Gonen&Alpaydin,2011)toobtainoptimalcombi- nation.TheoptimizedkernelcombinationisthenusedwithSVM classifier.

TheapproachproposedbyChenetal.(2012) utilizesknowl- edgefromthemalwareencyclopediasformalwareclassificationas itextractsmodelofthemalwaredescriptionandappliesittothe outputproducedbythesandbox.Usingsuchapproachauthorsare abletoannotatemalwarewithvariouslabels(exploit,dropper,etc.) whichisthenusedforthreatestimationandclassification.

2.3.Algorithmsformultipleinstancelearning

Classificationbasedonmultipleinstancelearningparadigmis apopulartopicwithmanypossibleapplicationssuchasclassifi- cationofimages,classificationofcandidatesdrugs,onlineobject tracking,etc.

Recently,Amores(2013)proposedtaxonomyofapproachesfor multipleinstancelearningthatdivides proposedalgorithmsinto threeclasses:(1)algorithmsbasedoninstance-spaceparadigm(IS), (2) algorithmsbasedon bag-space paradigm (BS)and (3) algo- rithmsbasedonembedded-spaceparadigm(ES).

TheISparadigmstatesthatthemultipleinstanceclassification isbasedsolelyoninformationextractedfromindividualinstances. The IS-basedmethodsfirstclassify theindividual instancesand thenpropagatetheestimatedlabelstothewholebags.Algorithms incorporatingIS paradigmincludeDiverse density(DD) proposed byMaronand Lozano-Pérez(1998),Expectation-Maximization Di- verse density(EM-DD) proposed byZhangand Goldman (2002),

Multiple Instance SupportVector Machines (MI-SVM) proposedby

Andrews,Tsochantaridis,andHofmann(2003),MIForestsproposed by Leistner, Saffari, and Bischof (2010) or G3P-MI proposed by

ZafraandVentura(2010).SincetheIS-basedmethodsoperateon the levelofindividualinstances,theyrequireanumericalrepre- sentationoftheinstancesoranappropriatekernelfunction.How- ever,thekernelizedmethodsapplicableinourscenario(MI-SVM)

1_{Accordingtotheexperiments,thenumberoffeaturesgeneratedforabout6000}

samplesreachesover20million.

donotscalewelltolargedatasets(O(M3_{)whereMistotalnumber} ofinstances)whichmaketheirusecomputationallyinfeasible.An interestingideatoimprovethescalabilityoftheIS-basedmethods isproposedbyCano,Zafra,andVentura(2015).Authorspropose toleveragetheperformanceoftheGPUsfortrainingofMIclassi- fiers.However,sinceGPUsaredesignedforfastmatrixoperations, theperformanceboostofthesimilaritymetricsproposedinthis paperwouldbelimited.Nevertheless,suchapproachprovidesan interestingoptionandwillbeconsideredinfurtherresearch.

InalgorithmsbasedontheBSparadigmistheinformationex- tractedfromthewholebags.Typically,itinvolvesdefinitionofa distancefunctionD(X,Y)thatcomparestwobagsXandY.Such functionsincludeminimalHausdorff distanceusedinWangand Zucker(2000),MI-GraphproposedbyZhou,Sun,andLi(2009)or kernel function proposed by Gärtner, Flach, Kowalczyk, and Smola(2002).However,asdiscussedinAmores(2013),thecom- plexityofapproachesbasedonBSparadigmisO(N2_·m2_)≈O(M2₎ whereNisnumberofbags,misaveragenumberofinstancesin abagandMistotalnumberofinstances.Eventhoughthecom- putationalcomplexityislowerthanthecomplexityofMI-SVM,it isstilltoomuchforlargedataset(thetestingdatasetinthispaper hasmore7millioninstances).

ThelastclassofapproachesarebasedonESparadigm. They transformthebagsintonumericalvectorsusingamappingfunc- tion M:X→Rd _{and train a supervised machine learning al-} gorithm on the transformedvector representation.Thesimplest mapping functions proposed by Dong (2006)and Bunescuand Mooney (2007) are defined as M(X) = ( f1, . . . , fd ) , where fi =

1

|X |x ∈ X xi, or fi=maxx∈ Xxi, or fi=minx∈ Xxi. More complex mappingfunctions definethe embeddingusingavocabularyV= { ( θ1, c1) , . . . , ( θd, cd) } , asetofconcepts (set ofinstances)where eachconceptisidentifiedbyidentifiercj andparametersθj .The mappingfunctionisthendefinedasM(X, V) =( f1, . . . , fd) withfi definedasfi (X, V) =1

Z

x∈ Xp(x| ci ) , ∀ i∈1, . . . , d, whereZisanor- malizationconstantensuringfi=1, and p(x|ci)representsthe likelihoodthataninstancexbelongstoaconceptci orhardassign- mentofinstancexintoconceptc_i .Regardingthedefinitionofthe vocabularyV, atraditionalapproach,adoptedinthisworkaswell, istoclusterallinstancesextractedfromallbagsusingasuitable clusteringalgorithm(typicallyK-Means)andtousetheresulting clustersastheconcepts.However,suchapproachdoesnotreflect relationsbetweenbagsandindividualinstances.Toaddressthisis- sue,Weidmann,Frank,andPfahringer(2003)proposepartitioning ofinstancesbasedondecisiontreesthatconsiderstheknowledge ofthelabelsofcorrespondingbags.However,asthedecisiontrees assumenumericalrepresentationoftheinstances,thisapproachis notapplicableinourscenario.

Cano(2017)proposesan interestingapproachthatcombines multiple-instancewith multi-viewlearningapproach. Themulti- viewapproachallowstofusevarioussourceofinformationabout thedatawhichinourcasecorrespondstovarioustypesofsystem resourcesandthusboosttheclassificationperformance.Further- more,authorsproposetoreplaceasinglemultipleinstanceclassi- fierwithanensembleapproachtofurtherimprovetheclassifica- tionperformance.However,sincethebaseclassifiersintheensem- blearestandardmultipleinstanceclassifiersdescribedabove,they sharethesamelimitations,whichlimitstheuseofthisapproach inourscenario.