Distributed Load Testing by Modeling and Simulating User Behavior

(1)

Louisiana State University Louisiana State University

LSU Digital Commons

LSU Doctoral Dissertations Graduate School

12-30-2020

Distributed Load Testing by Modeling and Simulating User

Behavior

Chester Ira Parrott

Louisiana State University and Agricultural and Mechanical College

Follow this and additional works at: https://digitalcommons.lsu.edu/gradschool_dissertations

Part of the Artificial Intelligence and Robotics Commons, Data Science Commons, and the Software Engineering Commons

Recommended Citation Recommended Citation

Parrott, Chester Ira, "Distributed Load Testing by Modeling and Simulating User Behavior" (2020). LSU Doctoral Dissertations. 5436.

https://digitalcommons.lsu.edu/gradschool_dissertations/5436

This Dissertation is brought to you for free and open access by the Graduate School at LSU Digital Commons. It has been accepted for inclusion in LSU Doctoral Dissertations by an authorized graduate school editor of LSU Digital Commons. For more information, please [email protected].

(2)

DISTRIBUTED

LOAD

TESTING

BY

MODELING

AND

SIMULATING

USER

BEHAVIOR

ADissertation

SubmittedtotheGraduateFacultyofthe LouisianaStateUniversityand AgriculturalandMechanicalCollege

inpartialfulfillmentofthe requirementsforthedegreeof

DoctorofPhilosophy in

TheDivisionofComputerScienceandEngineering

by

ChesterIraParrott

B.S.,ComputerScience,SoutheasternLouisianaUniversity,2013 May2021

(3)

Dedicatedtothestrength,spirit,andsacrificesofthefiercewomeninmylife.

(4)

Menhavebecomethetoolsoftheirtools. --HenryDavidThoreau

Walden

(5)

Acknowledgements

This workwaspartiallyfundedby theDonald W. ClaytonEndowmentintandemwiththe Di-visionofComputerScienceandEngineering. Anyjourneyworthtakingcanonlybecompleted throughthegenerosity,help,andeffortofmanyotherpeople;assuch,Ihavebeengivenandowe muchtomanypeople. ThankyoutoDr. JianhuaChen,Dr. Nash Mahmoud,andDr. TaoJinfor takingthetimetoserveonmycommitteeandprovidefeedbackonmyresearch.Tomyadvisor andcommittee chair,Dr. Doris Carver–thankyoufor yourconstancyandpatience. Youhave beenaninspirationandaninvaluablefontofwisdom.

ThankstothefacultyandstaffoftheDivisionofCSEfortimelysparksofencouragementand help. Thanks also tomy undergraduate professors: Dr. TheresaBeaubouef altered the course of my education with one conversation on the power of mathematics; Dr. Aron Culotta, Dr. PatrickMcDowell,Dr.GhassanAlkadi,andDr.JohnBurrisrecommendedmetoLSUwhenIwas searchingfor a graduate program. Thank you toRaghu Dodda, who exposed me tolog-based metricsandwasasoundboardformyideas.

Specialthanks toallmy friendsandfamilywhohavebeenthereforme,even whenIcould notreciprocate. Mygrandparentsgaveupmuchformy education,althoughIwastooyoungto appreciateitatthetime. Myparentsimpartedtomeastubborntenacityinthefaceofadversity andaninsatiablethirstforknowledgeandunderstanding.MyolderbrotherHarlandaredmeto re-imagine myselfand todream ofwhat might be accomplishedwithdiscipline appliedto my wildcreativestreak;myoldersisterJuneperiodicallypulledmyheadoutofthecloudstoremind meoftheimportanceoffamilyandtonevergiveup.

Thankyoutomywife,Robin,forpushingmetofinishmy bachelor’sdegree,forsupporting mydecision topursuegraduatestudies,forhonoringmewithyearsofloveandunderstanding, andforsharingthisjourneywithme.Yourresilientcapacityforgoodnessamazesmeeveryday. Thank you toour wonderful kids Georgia, Ian, Collin, and Molly-Lou –you did not start this journey with us, but you have given up so much time with me so that we could complete it, together.Thiswouldnothavehappenedwithouteachofyou!

(6)

Table

of

List

of

Tables

1.1. QualitativeComparisonofExistingLoadTestingProcesses . . . 7

3.1. ConditionalProbabilityMatrixforExampleUserBehaviorModel . . . 31

3.2. WeekIntheLifeofaBrowser-UsersTableSample . . . 40

3.3. WeekIntheLifeofaBrowser-EventsTableSample . . . 41

3.4. WeekIntheLifeofaBrowser-SurveyTableSample . . . 42

3.5. MSNBC.comAnonymousWebData . . . 44

3.6. AnonymousMicrosoftWebData . . . 45

4.1. ExampleWikipediaClickstreamData . . . 61

5.1. ConditionalProbabilityFeatureVectorsforExampleUsers . . . 80

7.1. QualitativeComparisonoflodestonewithExistingLoadTestingProcesses . . . 115

(9)

List

of

Figures

1.1. SequenceDiagramoflodestoneProcess . . . 8

1.2. ResearchRoadmap . . . 11

3.1. SampleSystemDefinition . . . 25

3.2. LegalApplicationStateSpaceforUserProfile: ‘SuperAdministratorUser’ . . . 28

3.3. LegalApplicationStateSpaceforUserProfile: ‘User’ . . . 28

3.4. FrameworkforSimultaneousModelingandSimulationofUserBehavior . . . 29

3.5. MarkovChainDirectedGraphforExampleUserBehaviorModel. . . 31

3.7. HeatmapofTransitionCountMatrixforExampleUsers . . . 34

3.8. MarkovChainDirectedGraphforUserBehavior . . . 34

3.9. HeatmapofConditionalProbabilityMatrixforExampleUsers . . . 35

3.10. ConditionalProbabilityforClusteredUBBstoTransitionfrom𝜎_𝑖to𝜎_𝑗 . . . 36

3.11. MarkovChainDirectedGraphforExampleUserBehaviorModel. . . 37

3.12. MSNBC.comonNovember9,1999 . . . 43

3.13. MSNBC.comNumberofUsersvs.NumberofClicks . . . 44

3.14. Microsoft.cominNovember,1998 . . . 46

3.15. ClassDiagramforTinyERPSpringBootServices . . . 48

3.16. TinyERPandLokiRuntimeConfiguration. . . 49

3.17. SequenceDiagramofSimulatedDataGenerationProcess . . . 51

3.18. TinyERPUserSimulationScriptTemplate . . . 52

3.19. SampleUserDataStream . . . 53

3.20. SystemMetricscapturedforTinyERP . . . 54

4.1. CompressedTextSizeofAllWikipediaArticlesOverTime[130] . . . 58

4.2. ExampleWikipediaBehavioralClickstreamGraph . . . 60

4.3. ExampleWikipediaLinkGraph . . . 60

4.4. ProcesstoExtractKnowledgeCommunities . . . 63

(10)

4.5. Tarjan’sStrongConnectAlgorithmandConnectComponentSubRoutine[116] . . . . 64

4.6. Full-DepthConnectedComponent . . . 66

4.7. ConstrainedStrongConnectAlgorithmandConnectComponentSubRoutine[116] . . 67

4.8. StronglyConnectedComponentsWithDepthConstraint: 10000 . . . 69

4.9. WeaklyConnectedComponentsWithDepthConstraint: 10000 . . . 70

4.10. StronglyConnectedComponentsWithDepthConstraint: 10 . . . 70

A.1. RoleTableDDLforTinyERPApplication . . . 118

A.2. UserTableDDLforTinyERPApplication . . . 118

A.3. AccountTableDDLforTinyERPApplication . . . 119

A.4. InvoiceTableDDLforTinyERPApplication . . . 119

A.5. OrdersTableDDLforTinyERPApplication . . . 120

A.6. User_RoleTableDDLforTinyERPApplication . . . 120

A.7. User_RoleTableDDLforTinyERPApplication . . . 121

A.8. Widget_OrderTableDDLforTinyERPApplication . . . 121

(11)

Acronyms,

Terms,

and

Definitions

Acronym Term Definition

S _System _A_{goal-oriented}_collection_of_entities_and_procedures N/A Agent/Entity An executor of some series of tasks within a given Systeminordertoaccomplishgiven(ordiscoverable) goals

N/A Machine Amechanicalorelectronic entitycapableof execut-ingroutinetasks

N/A Human/User A biologicalentity capableofexecuting routine and higher-ordertasks

HMS Human-Machine System(s)

Systems which are comprised of humans and ma-chines

MAS Multi-AgentSystem(s) Systemswhichare comprisedofhuman Agentsand machineAgentsworkingtowardacommongoal N/A Model Amathematicalrepresentationofanentityora

Sys-tem SUO,S𝑜 System(s)under

observation

AproductionHMSwhererealisticdatafortypical be-haviors from Agents can be captured for modeling andmining

SUT,S𝑡 System(s)undertest An HMS strictly reserved for pre-release testing of

code and infrastructural changes to detect hidden flaws

LT LoadTesting A series of procedures for identifying System-wide Load-relatedflawswithinanSUT

(12)

Abstract

Modern human-machine systems such as microservices rely upon agile engineering practices whichrequirechangestobetestedandreleasedmorefrequentlythanclassicallyengineered sys-tems.Acriticalstepinthetestingofsuchsystemsisthegenerationofrealisticworkloadsorload testing. Generated workload emulates the expectedbehaviors ofusers and machines within a systemundertest inordertofindpotentiallyunknown failurestates. Typicaltestingtoolsrely onstatictestingartifactstogeneraterealisticworkloadconditions.Suchartifactscanbe cumber-someandcostlytomaintain;however,evenmodel-based alternativescanpreventadaptationto changesinasystemoritsusage. Lackofadaptationcanpreventtheintegration ofloadtesting intosystemqualityassurance,leadingtoanincompleteevaluationofsystemquality.

Thegoalofthisresearchistoimprovethestateofsoftwareengineeringbyaddressingopen challengesin loadtesting ofhuman-machinesystems witha novel processthat a)models and classifiesuserbehaviorfromstreamingandaggregatedlogdata,b)adaptstochangesinsystem anduserbehavior,andc)generatesdistributedworkloadbyrealisticallysimulatinguser behav-ior. ThisresearchcontributesaLearning,Online,DistributedEngineforSimulationandTesting basedontheOperationalNormsofEntitieswithinasystem(lodestone):anovelprocessto dis-tributed loadtestingby modelingandsimulatinguser behavior. Wespecifylodestone within thecontextofahuman-machinesystemtoillustratedistributedadaptationandexecutioninload testingprocesses.lodestoneuseslogdatatogenerateandupdateuserbehaviormodels,cluster them into similar behavior profiles, and instantiatedistributed workload on software systems. Weanalyzeuserbehavioraldatahavingdifferingcharacteristicstoreplicatehuman-machine in-teractionsinamodernmicroserviceenvironment.Wediscusstools,algorithms,softwaredesign, andimplementationintwodifferentcomputationalenvironments:client-serverandcloud-based microservices. Weillustrate the advantagesoflodestone througha qualitativecomparisonof keyfeatureparametersandexperimentationbasedonshareddataandmodels.lodestone con-tinuouslyadapts tochangesinthe systemtobetestedwhichallows forthe integrationofload testingintothequalityassuranceprocessforcloud-basedmicroservices.

(13)

Chapter

1. Introduction

Mylifeseemedtobeaseriesofevents andaccidents.YetwhenIlookback,I seeapattern. --BenoîtMandelbrot AFractalLife Withtheproliferationofthehumanpopulation,ourinterconnecteddevices,andthesystems which nowcompose our infrastructure and everyfacet ofmoderncivilization, our societyhas evolved intoanecosystemofhumansand machinesemittinga profoundamountofdataevery dayabouthowitsdenizensliveandinteract. Onewayinwhichourhuman-machineecosystem recordsandechoeshumanbehavioristhroughdigitaltracesproducedasasideeffectofhumans changing or interacting with other people, processes, and technology. From a human-centric perspective,suchtracesofbehaviorcanbeusedtodescribeapopulationofusersaswellasthe deviceswithwhichtheyregularlyinteract.However,thescaleofdatarecordinghuman-machine eventscanbeoverwhelmingineventhesimplestofscenarios;moreover,attemptstoexpandand usetheknowledge insidestreamsoftrace datacanoverwhelmthe resourcesorganizationsare willing to dedicate toward analysis. As the amount of data produced by our human-machine ecosystemoutpaces the abilityto consumeandunderstandit, a pressing needexists tochange thecomputationalapproaches,tool-sets,andmind-setsweusetocreateandmaintainthesystems uponwhichoursocietyrelies.

Alandmarkstridetowardimprovingtheapproachesassociatedwithcreationandanalysisof systemsisthe adventofthe cloud. The rapidpropagationofcloudtechnologyhassparkedthe widespreadadoptionofarchitecturalpatternssuchasstatelessmicroservicesbasedon represen-tationalstatetransfer(REST)andsimilarprotocols[56]. Microservicesrelylessfrequentlyupon heavyweight, synchronous, stateful operationspreferring lightweight, asynchronous, stateless operationsbuiltonserverlesscomputingfunctionalitysuchasLambdaFunctionsfromAmazon WebServices(AWS)[3]orsimilarofferingsfromothercommercialcloud-serviceproviders[2,8,

(14)

51,61,91].Thedecreaseincostforcomputationalcyclesjuxtaposestheincreaseinavailabilityof technologicalsolutionsandknowledge.Dynamicallyevolvingsoftwaredesignchoicesreplacing classical architecturesmotivatea rationale tochange the manner and extenttowhich modern systemsshouldbeevaluatedforquality.

1.1.ResearchQuestions

The goal of this research is to improve the state of software engineering by addressing open challengesinloadtestingofhuman-machinesystemswithanovelapproachthat: a)modelsand classifiesuserbehaviorfromstreamingandaggregatedlogdata,b)adaptstochangesinsystem anduserbehavior,andc)generatesdistributedworkloadbyrealisticallysimulatinguserbehavior. Thisresearchismotivatedbythefollowingquestions:

𝑅𝑄₁: Canadata-drivenprocessforMarkovChain(MC)model-basedloadtestingbedeveloped thatoffersadvantagesoverexistingdata-drivenprocesses?

𝑅𝑄₂: Canrawstreamingbehavioraldatainahuman-machinesystembecontinuouslyand effi-cientlymeasured,modeled,andstoredasMCmodels?

𝑅𝑄₃: Can aggregated batched behavioral data in a human-machine system be systematically groupedintosemanticallyrelatedMCmodels?

𝑅𝑄₄: CanMCmodelsofahuman-machinesystembeextendedtoadapttochangesinasystem underobservation?

1.2.ChallengesofMarkovChainModel-basedLoadTesting

Currentloadtesting(LT)processesarenotcapableofswiftlyadaptingtochangeswithinasystem underobservation, requiringmanualworkby qualityassuranceengineer(s)(QA)for staticand modeled processes [13]. Indeed, an automated load testing system capable of mimicking the capabilitiesoflivehumantesterstoadapttothechangesinasystemundertestdoesnotexist[68]. Mostclassical automatedtesting systemsare eitherstrictly“recordand play” based onsystem traces,randomizedwithabackingMCmodel,oruseprobabilistictimedautomatonswhichextend thecapabilities ofaMCmodelbyaddingthe dimensionofthinktimetoshowhowusers might pausewhilemovingfromstatetostatetoincreaseworkloadrealism[68,126].

(15)

However, evenrecentprocessestoLTarenotadaptiveorreactivetochangesinthesystem to be tested [13, 77]; moreover, LT of modern microservice architectures provides additional challengesoverclassicalarchitectures[56]. Thecurrentcapabilitiesofcloud-nativetestsystems canpartiallyaddresstheperformanceconcernslistedin[56,98]throughreadyavailability,swift scale-out/scale-in,distributedcomputationspanninggeographicandinfrastructureboundariesto mimicactual usersofcloud-based systems, andfine-grained component-levelmonitoring [88]. Thematicchallengesfromtheworksaboveare:

𝐶₁: As users become more familiar with a system, usage patterns are likely to change which meritsanupdatetoarealistic-basedtestingload.[68]

𝐶₂: As a system evolves over time, model building offault-inducing loads might be improved throughincrementalanalysisofthesysteminternals.[68]

𝐶₃: LTprocessesshouldalignwithagiledevelopmentpracticessuchascontinuousintegration, dynamicdeployment,andtestautomation.[56]

𝐶₄: LTprocessesshouldcontinuouslyupdatetestingartifactssuchasmodelsandscenariosbased onchangesintheunderlyinginfrastructure,system,anduserbehavior.[56]

The researchsuggests thatviableprocessesaddressing thesechallengesshouldberobustin differingproperties;towit,suchprocessesshouldusemultipleagentsandbeadaptive,realistic, efficient,andoperationallybased.

𝑃₁(Adaptive): Aprocessforloadtestingwhichisadaptiveshouldbeabletoautomaticallyevolve, basedonchangesintheunderlyingsystemunderobservation(SUO).

𝑃₂(Realistic): Aprocessforloadtestingwhichisrealisticshouldproduceorrunloadtestswhich arerepresentativeofhowtheSUOisused,orwillbeused.

𝑃₃(Efficient): Aprocessforloadtestingwhichisefficientshouldnotrelyuponalgorithmsordata structureswhichprecludetestingofarbitrarilylargesystems.

𝑃₄(MultipleAgents): AprocessforloadtestingwhichusesMultipleAgentsshouldbeagent-based andshouldrelyonmorethanoneoperatingagent.

(16)

𝑃₅(OperationallyBased): Aprocessforloadtestingwhichisoperationallybasedshouldusedata representativeofthepeople,process,andtechnologyoftheSUO.

Insummary,data-drivenprocessesforrealisticmodel-basedloadtestingshouldcontinuously andautomaticallyadapt tobehavioral andinfrastructuralchanges ina systemtobetested. We examinehowexistingprocessesaddressthesechallenges.

1.3.ExistingProcesses

Previousattemptsatautomatedloadtestingprovidevaluableinsightanddirection;however,no processexistswhichaddresseschallenges𝐶₁−𝐶₄orexhibitsproperties𝑃₁−𝑃₅listedinSection1.2. Aloadtestingprocesswhichexhibitsthesepropertieswouldexhibitadvantagesoverexistingload testingmethodologies. Wedescribepreviousattemptstobetterframethescopeofthisresearch. (2002) Kant et al. [70] show the design of Geist as a means ofshowing how a MC model candriveasystemforstress testingweb servers. WhileGeistis‘dialable’,itdoesnotprovidea mechanismforautomaticallyadaptingtochangesintheSUO.Geistisbasedonoperationaltraffic and provides a requisite level ofrealism, but does not provide a suitable means for efficiently scaling to‘large’-scalesystems. Moreover, Geistdoes notallow formultiple runningagentsor modelingofuser behavior; rather, itreplaystraffic which haspreviously beenobserved inthe systemsimilartoSelenium[19].

(2002)Menasce[84] showsmethodswhicharenotadaptiveinnature,butrelyuponmanual changestotheload testingprocess. Menasce’sprocessisbasedonvirtualuserswhichrunina loadgeneratingprogram. However, thisprocessisnotefficient,astheloadgeneratingprogram isnotdesignedfor distributedcomputingorforusingmultipleagentstogeneratetherequisite workload. Italsois notspecificallydesigned forinducing faults;rather, itfocusesonmeansto replay traffic behavior based on manually created customer/user models. As the load testing processisbased onreplayingofbehavior whichhasbeen previouslyobserved,itis realisticin nature.

(17)

(2005) Canfora et al. [23] use genetic algorithmsin order to provide adaptive responsesto changeinthesystemundertest (SUT).Theirmethodsarenotbased inoperationalbehavioror behavioralmodels, preferringa prescriptiverule-basedprocessinstead. Geneticalgorithmsare unconstrained[23], thus efficiencyisnot baked intothis process,although the authorsseekto constraintheboundariesofthegeneticalgorithmsused.Thegeneticalgorithmsuseddonotuse multipleagentsduetoperformancelimitations.However,themethodisdesignedtoelicitfaults throughthedynamicgeneticalgorithmprocessandwilloptimizetowardfindingsuchfaults.

(2007)Barrosetal.[9]showaprocessforloadtestingwhichisbasedonoperationaldataand MCmodelstoperformcachedreplayingofthebehavioroftheSUO.Theprocessisnotefficient inthatmodelshavethepotentialforbecominglargerthaniscomputationallyfeasibleformany organizations,andtheauthorsdonotprovideameansforcompensating;moreovertheprocess doesnotusemultipleagentsoradistributed frameworkforexecutingworkload. Theprocessis designedforbenchmarkingsystems. However,itisoperationallybasedas ituseslogsfrom the SUOasinputfortheunderlyingMCmodels.

(2007)Pentaetal.[37]usegeneticalgorithmstooptimizeagainsttheQualityofService(QoS) constraints given. Their algorithm is capable of changing and optimizing toward the goal of violating those constraints, thus the load testing process is adaptive. However, the process is not realistic in that it is focused on violating QoS and is not based on any realistic behavior patterns. Genetic algorithmsarenotinherentlyefficientduetothelargestatespacetheycover. The processby Penta etal. is not designedto usemultiple agentsordistributed computation. Finally, the processseeks toinduce faultsrelated toQoS and isnot basedon operationaldata, ratheronstatesintheSUO.

(2009)Gu &Ge[53] usegeneticalgorithmsas ameansfor loadtestingasthey seek tohave theprocessadapttochangesintheSUO.Thisprocessisnotrealisticasitsearchesovertestsfor suitablecandidatesratherthanworkingwithrealisticbehaviorpatterns.Geneticalgorithmsare notinherentlyefficientduetothelargestatespace. Theprocessisnotdesignedfordistribution

(18)

as it runs offline. Genetic algorithms are not by default agent-based; also, the process is not distributed.AswithPentaetal.[37],theloadtestingprocessusedinthisprocessseekstoviolate QoSrestrictions.TheprocessfocusesonsystemmodelingbasedexplicitlyonQoS,notonactual programexecutionoroperationaldata.

(2012) Zhangetal. [143]havea compositional processforload testingwhich isadaptivein naturebasedontheanalysisofsystemcomponents. Theirfocusonthesystemcomponentsdoes not providea realisticuser-behavioralpointofreferenceforloadtesting; however,itdoeslend itself gracefully to distributed workflows. However, the statespace of the symbolic execution mechanismproposedcangrowbeyondtheboundariesofwhatwewouldconsideran‘efficient’ load testingprocess. The compositional processis not inherently multi-agentbased; however, eachsubcomponentcouldbeanalyzedbyanagentwithdifferentgoalsandrewards.Theprocess isnotbasedonlogdata;rather,theirmethodisbasedonsystemcomponentinformation.

(2015)Cotreneoetal. [35]describe RELAIwhichdoesnot adapt tochanges inthe SUOasit does notingest additionaldata forbehavior modelingafter thefirst ingestion. RELAIsupports usageofoperational/usageprofiles,butisstructuredforrunningonasingletestmachine,rather thanbeingdistributed. RELAIisnotmulti-agentbased,neitherisitdesignedtospecificallyfind faultsintheSUT;however,RELAIisbasedonoperationaldatasuchasuserprofilesormachine profiles.

(2015) Schuretal. [105] do notprovidea mechanismforautomaticallyadapting tochanges inthe SUO.WhendefiningProCrawl,theyspecified challengesrelated to‘adapting’the under-lyingmodels toa new SUO-based operationalprofile. The process canbe considered realistic as is based on user/system behavioral data to automate testing. ProCrawl does not work in a distributedfashion,thusscalingtheprocesstoalarge-scalesystemisinfeasible. ProCrawluses operational data; however, it is not designedto work with multiple agents or ina distributed manner. ProCrawl focuses on extracting operational models from software in which multiple usersareworkingthroughconcurrentworkflows.

(19)

Table1.1.QualitativeComparisonofExistingLoadTestingProcesses

𝑃₁ 𝑃₂ 𝑃₃ 𝑃₄1 𝑃₅

AuthorAdaptiveRealisticEfficientMultipleAgentsOperationallyBased

Barrosetal.[9] ✓ ✓ Canforaetal.[23] ✓ Cotreneoetal.[35] ✓ ✓ Gu&Ge[53] ✓ Kantetal.[70] ✓ ✓ Menasce[84] ✓ ✓ Pentaetal.[37] ✓ Schuretal.[105] ✓ ✓ vanHoorn[58] ✓ ✓ Vogeleetal.[126] ✓ ✓ Zhangetal.[143] ✓

(2008, 2016) van Hoorn,Vogeleetal. [58, 126]proposedMarkov4JMeter asa MCmethodfor realisticloadtesting;however,Markov4JMeterdoesnotprovideamechanismforautomatically adapting tochanges inthe SUO.Manual pre-processing isrequired tocreate theoperationally basedmodelsusedforexecuting LTinMarkov4JMeter;however, theseMCmodelsarerealistic when executing against the SUT. Markov4JMeter is based on JMeter [117] - a tool known in industry for having efficiency limitations [15] and operating on a single machine rather than multiple agents. This process was extended in2016 withWESSBAS [126] which centers on a domain specific language for mining MC models from operational data. However, WESSBAS doesnothavethecapabilityforcontinuousadaptationtotheSUO,andstillrequiresoffline pre-processingandmodelingofdata.

Asummaryoftheseprocessesandhowtheyrelatetoproperties𝑃₁−𝑃₅definedinSection1.2 islistedinTable1.1.Uponreviewoftheexistingprocessesforautomatedloadtesting,aprocess todistributeloadtestingbymodelingandsimulatinguserbehaviorwhichsuccessfullyaddresses allofthechallenges𝐶₁₋₄ listedinSection1.2doesnotexist.

(20)

Parse and Clean Data

Define & Update User Bases of Behavior Preliminary Analysis of Behavioral Data II Generate Behavioral Data Behavioral Data Stream Behavioral Data Batch Perform Configuration and Customization Schedule or Initialize Load Generation Agents Lodestone Knowledge Store

Define & Update Profile Bases of

Behavior Define & Update Semantic Bases of

Behavior Define & Update

Global Bases of Behavior Define Constant and Random Bases of Behavior Behavior Modeling SUO Lodestone Knowledge Store

Define & Update Agent Configurations Emit Operational Metrics Execute Behavioral Responses Execute Behavioral Load III VII IV I V VI VIII X XI XII XIII Pruning of Behavioral Patterns Q-Learning of Behavioral Patterns XV XIV IX.b

QA Load Generation SUT

I.a

I.b IX.a

Figure1.1.SequenceDiagramoflodestoneProcess

(21)

1.4.LODESTONEProcess

WepresentaLearning,Online,DistributedEngineforSimulationandTestingbasedonthe Op-erational Norms ofEntitieswithin a system (lodestone). The processcanbe implementedin anydistributedcomputingenvironmentwherethereexistsaSUTseparatedfromaSUOand op-erationallogscanbeminedfromtheSUO.Amoredetailedoverviewofourprocessisillustrated bythediagraminFigure1.1.lodestonerequiresthefollowingend-to-endsequentialsteps con-tinuouslyoperatingtogenerateworkloadagainsttheSUT:

I.GenerateBehavioralData: Event data areproduced by the interactions betweenpeople, pro-cess, and technology within the SUO. These data can be aggregated into a preliminary formoratomicallyencapsulatedinastreamofevents.

A.BehavioralDataBatch: Apre-aggregatedsetofeventsperiodicallyemittedbytheSUO. B.BehavioralDataStream: ArawsetofeventscontinuouslyemittedbytheSUO.

II.PreliminaryAnalysisofBehavioralData: Performedbysubjectmatterexperttoinformthe parsing,cleaning,configuration,andinitializationofloadgenerationagents.

III.ParseandCleanData: Afterpreliminaryanalysisofinput dataiscompleted, anappropriate parsingmodelformatsthedata,andextraneousdatapointsarecleanedfromtheinputdata. IV.DefineConstantandRandomBasesofBehavior: BasedonthepossiblesetofstatesintheSUO, defineMCmodelswhichmayrandomlytransitionbetweenstatesorexecuteobservedstate transitionswithequallikelihood.

V.Define&UpdateGlobalBasesofBehavior: BasedonthepossiblesetofstatesintheSUO,define MCmodels which mayexecute observed statetransitions based onpreviously observed likelihoodacrossallusers.

VI.Define&UpdateSemanticBasesofBehavior: Basedon the possibleset ofstates inthe SUO, define MC models which subdivide the global basis of behavior based on semantically linkedportionsoftheSUO.

(22)

VII.Define&UpdateUserBasesofBehavior: BasedonthepossiblesetofstatesintheSUO,define MCmodels which mayexecute observed statetransitions based onpreviously observed likelihoodforindividualusers.

VIII.Define&UpdateProfileBasesofBehavior: BasedonthepossiblesetofstatesintheSUO, de-fineMCmodelswhichmayexecuteobservedstatetransitionsbasedonpreviouslyobserved likelihoodforuserswithsimilarbehaviorpatterns.

IX.ManageLoadGeneration: TheQAmayperformadditionalstepstomanageloadgeneration. A.PerformConfigurationandCustomization: The QA may perform configuration to

im-provetheperformanceoroperationofloadgeneration.

B.ScheduleorInitializeLoadGenerationAgents: TheQAmayscheduleorchangeload gen-eration.

X.Define&UpdateAgentConfigurations: GiventhesetofobservablestatesintheSUO,andthe MCmodelsgeneratedinIV-VIII,defineanappropriatelysizednumberofagentsbasedon expectedbehaviorfortheSUO.

XI.ExecuteBehavioralLoad: Multipledistributedagentsinteract withtheSUTbyexecuting be-haviorandvolumetricstatetransitionspreviouslyrecordedorcustomizedbyQA.

XII.ExecuteBehavioralResponses: The SUT provides responses for eachstate transition to the loadgenerationagents,toincludeiftherequestedstatetransitionresultedinfailure. XIII.EmitOperationalMetrics: Theobservednumberofresponses,errors,andresponsetimeare

recordedbytheloadgenerationagentsforadditionalanalysisandimprovement.

XIV.Q-LearningofBehavioralPatterns: ThemetricsgeneratedbytheSUTaremodeledaccording totheQ-Learningreinforcementlearningalgorithmforloadgenerationagentusage. XV.PruningofBehavioralPatterns: As additional data comefrom the SUO, behavioral patterns

andstatetransitionswhicharenolongerinoperationshouldberemovedtoconserve op-erationalresources.

(23)

Extract User Models from Raw Log Data

Infer User Models from Aggregated Log Data Learn Adaptive User Models from Streaming Log Data Distribute User Modeling and Load Testing Investigate Anonymous User Log Data Define Required Formalisms Generate Behavioral Profiles with Clustering Investigate Wikipedia Clickstream Data Simulate Behavioral Data Define Methodology to Extract Semantic Models Improve Process with Reinforcement Learning Improve Process with Laplace Smoothing Improve Process with LRU Cache

Evaluate Methodology and Semantic Models Define LODESTONE Logical Architecture Define LODESTONE Physical Architecture Distribute Modeling and Load Testing in Amazon Cloud Evaluate LODESTONE with Markov4JMeter A B C D

Figure1.2.ResearchRoadmap

lodestonereliesupontheusageofmultipledistributedagents,operationaldata,andefficient algorithms to create a realistic and adaptive workload against the SUT. It combines methods for modeling user behavior from raw and aggregated data with algorithmic optimizations for adaptingloadgenerationastheSUOchanges.

1.5.Contributions

Theexpenseanddifficultyassociatedwithclassicalloadtestingmethodshavebecomeoutmoded bytheubiquityandspeedwithwhichsoftwaresystemsaredevelopedanddeployed. Atitscore, thisresearchhasfollowedthestepsshowninFigure1.2tocontributelodestone:anovelprocess for testingsystems based onthe data generated through interactions betweenusers and those systemsthatexhibitstheproperties𝑃₁₋₅fromSection1.2.

(24)

A.ExtractUserModelsfromRawLogData: The inceptionof this researchwas an investigation ofanonymoususerdataandadefinitionofformalismssupportingtheoverallprocess. Fol-lowingthisinvestigationwasthecreationofmodelsfromtheanonymousdataandshowing thereductionofthequantityofmodelsthroughclustering.Inordertoexaminethequality oftheclusteredmodels,itwasnecessarytosimulatebehavioraldatawithanindependently developedmechanism.

B.InferUserModelsfromAggregatedLogData: Theuserandprofilemodelscreatedwiththe pre-vious step of this research cannot be directly inferred from aggregated data due to the lackofindividualidentifiersfordisambiguatingobservedbehaviors. Aninvestigationinto Wikipedia’spublicly availableclickstreamdatawasnecessarytoaddressthosesituations whereonlyaggregateddataareavailable.Amethodologyandalgorithmwereconstructed to extractsemantic models from such aggregated data and wasevaluated as part of the research.

C.LearnAdaptiveUserModelsfromStreamingLogData: In ordertoallow themodels produced intheprevious twostepsofthisresearchtoadapt tochangesinthe SUO,improveddata structuresformedthecore ofthecomputationalrequirements. These datastructuresmet therequirementsoftheformalisms previouslydescribed,andwereaugmentedwith rein-forcementlearning,Laplacesmoothing,andaleast-recently-usedcachetosolveforvarious scenarioswhereadaptationwerenecessary.

D.DistributeUserModelingandLoadTesting: Thefinalstepofthisresearchcombinedtheresults oftheformalisms,analyses,algorithms,datastructures,andtoolingdevelopedinthe pre-vioussteps.Alogicalandphysicalarchitectureforhowlodestoneshouldoperateandbe structuredprecededanimplementationonAmazonWebServicesandanevaluationagainst acontemporarytoolwidelyusedforloadtesting.[3]

Increatingandevaluatinglodestone,severaldirectedaspectsoftheresearchwererequired fortheprocesstobeinstantiated,completed,andevaluated;theseaspectswereasfollows:

(25)

• Asetofterminologyandformalismsdefiningtheparameters,data,andmodelstobe amal-gamatedbylodestone andinformedbyanad-hocanalysisofseveralanonymouspublic userbehavioraldatasets.

• AdataingestionprocessusingtheDBSCANclusteringalgorithmtodetectandmodel sim-ilarusersbytheirbehaviorpatterns.

• A functionalmicroservice(TinyERP) andtestoracle (Loki)writteninaproduction-grade softwareframework(SpringBoot)forrealisticevaluationofloadtestingprocesses, contin-uousgenerationofperformance andbehavioraldata,generationofrule-based behavioral traffic,andcontrolledsimulationoferrorsinthesystem.

• AcollectionofstreamingstatisticsalgorithmswhichprovidedatasetstatisticsinO(1)space andO(1)time.

• ADepthConstraintextensionofTarjan’sSTRONGCONNECTalgorithmforefficient seg-mentationofaggregatedlogdataintosmallersubgraphsofrelateduserbehaviorand exe-cutedonaggregatedWikipediadata.

• AnextensionoftheformalismsofChapters2and3throughtheQ-Learningreinforcement machinelearningalgorithm,supportedbyaproofofoptimality.

• An extension of SparseVector and SparseMatrix data structures to address the “sunrise problem” of events which have never been observedbut should havea non-zero proba-bilityofbeingsimulatedthroughLaplacesmoothing.

• Additionof a least-recently-usedevent cache to allowfor efficientreduction ofinactive, invalid,orunlikelyhistoricaleventtransitions.

• ProvisionfortheKullback-LeiblerDivergencefordeterminingifclusteredmodelsprovide thesamelevelofinformationalrepresentationastherawdatausedbyclassicalloadtesting processes.

(26)

• Requirementofasystemtomonitorandlearnfrominadditiontothesystemtobetested; sucharequirement,notappearingelsewhereintheliterature,enablesanimplementationof lodestonetoacceptstreameddatafromaproductionenvironmentforcontinuousadaptive testingofasystemundertest.

• Thelodestone processasanimplementationinAmazonWebServiceswithalogical ar-chitectureandaphysicalarchitecture.

• QualitativeandquantitativecomparisonagainstaubiquitousloadtestingtoolJMeterusing identicalmodelslearnedfromrule-basedbehavioraldatageneratedbyTinyERP.

• Adistributedloadtestingprocesswhichisadaptive,realistic,efficient,operationallybased, andsupportsmultiple-agentexecution.

These aspects of the lodestone process delineate how raw system data may be collected in their various forms to efficiently learn and simulate interactions between the humans and machines ina real human-machine-system. lodestone maybe usedtoward solvingproblems infieldssuchascybersecurity,emergencyresponse,productmanagement,anduserexperience. However,withthisresearchwetendertheoreticalconceptsofandconcreteevidenceconcerning theunderstandingofuserbehaviordatatosolveloadtestingchallengesinsoftwareengineering.

(27)

Chapter

2. Background

Thelongeryoucanlookback,the fartheryoucanlookforward. --WinstonChurchill RoadtoVictorySpeech,March1944 Webrieflyoverviewthemodelingofhuman-machinesystemsandthesimulationofAgency withinhuman-machinesystems; additionally,we describeapproachestorealisticgenerationof workloadonsoftwaresystemswhichusesimilartheories,mathematicalstructures,and compu-tationalstructuresasourframework.

2.1.TheModelingofHuman-MachineSystems

Wedefineahuman-machinesystem(HMS)tobeacollectionofpeople,process,andtechnology workingtogetherwithinalargerorganizationofentitiestoaccomplishaprescribedoremergent goal. Work hasbeen done to model different entitieswithin HMS in order tounderstand and predict how the HMSwill operate under changing conditions. Weposit that the growing co-dependencyofhumansandmachinesinourtechnologicalsocietyallowsforbothsetsofentities tobemodeledandsimulatedasAgentswithinanHMS.Multi-AgentSystems(MAS)aredetailed in texts such as the work on by Shoham and Leyton-Brown [109] as models with distributed Agency whichare intended for the simulationand solving ofproblems. For more detailed ex-plorationofmining forbusinessprocessesandbusinessroles, wereferthe readertothe works byvan derAalst[120, 121] for processmining, andColantonioetal. [30] for rolemining. For additionalbackgroundinloadtestingofsoftwaresystems,wedirectthereadertowardJiangand Hassan’ssystematicliteraturereviewonthesubject[68]. Weoverviewselectworksinmodeling ofpeople,process,andtechnology.

(28)

2.1.1.People

Wediscusstraditionalmethodsofmodelinguser-specificAgentsresponsibleforcreatingchange withintheSUO.OneoftheprimaryworksonuserprofilingwasdonebyElaineRich,whodetailed adata-drivenapproachtomodelingmultipleusersforclustering[102]resultinginastaticmodel akintoadecision treeforuser representation. Suchuser-profiling hasbeenappliedtosecurity bytheNationalInstituteofStandardsandTechnologystandarddefinitionofRole-BasedAccess Control[44]whichextendedFerraiolo’soriginalworkonthesubjectin[43].Franket.al.define two creatingrole-basedprofilesofusers: the top-down prescriptiveapproach andthe bottom-up descriptive approach [47]. The prescriptive approachis usually an effort-intensivemanual process[54];thebottom-upapproachisusuallydata-driven,butsuffersfromover-generalityor over-specificresults.Middletonet.al.showaknowledge-basedontologicalmethodforprofiling userswithinrecommendersystemssuchasusedbyNetflix[86].

2.1.2.Process

Thecollectionofworkflowsteps,conditionalrules,andreactionsisanotherkeyelementto move-mentofdataandprogresswithinanHMSwhichisdonethroughprocessmanagementand min-ing[36]. AccordingtovanderAalst,suchworkflowsaretypicallyusedforsimulationpurposes orgeneral intelligence onthe behavioral characteristics ofan HMSfor improvement purposes [121]. PecarinadescribedAPSATasaframeworkforminingandmanagingworkflowprocesses fromnetworktraffic[95,96].Li,Kang,andLvshowamethod[78]fordynamicbusinessprocess miningfrom eventdata. Fei,Liqun,GuangYun, andXiaoleidescribeanalgorithmfordetecting conceptdriftinprocessmining[42]. Suchdetectionisusefulfordeterminingchangesinexisting businessprocesses.Boseet.al.discussamethodfordealingwithconceptdriftsinprocessmining meanttodealwithchangesovertime[16].Assuch,data-miningisafeasiblemeansofmodeling processesinanHMS;however, caremust betakentoaddresschanges intheHMSinorder for simulationandstudiestoremainaccurateaftertheminingprocesshascompleted.

(29)

2.1.3.Technology

Modelingtechnical components in anHMS istypically usedfor anomaly detection of system-level behaviors and performance characteristics[26, 57, 60, 69, 107, 111, 141]. Static means of usingoperationalmodelshavebeenusedinsoftwareengineeringpracticessuchasWhittaker’s approachtoCleanroomSoftwareEngineering[128].ToolssuchasProCrawlcanextractbehavior modelsfrom webapplications[105],and RELAIframeworkisabletouseaprobabilistic model based onoperational profilesinorder toimprovesoftware reliability [35]. Genetic algorithms areusedtotestServiceLevelAgreementsandQoS throughLT [23,37,53,143];however,these approachesarenotdirectlyusinguser-basedbehaviormodelstogenerateapplicationworkload. Anderson’s novel approach using a MC representation of sequential instructions is described in [4] where instruction combinations are usedas clustering features for categorizing various familiesofmalware.

2.2.MarkovModelingforLoadTestingofSystems

MarkovChainmodelsareasubsetofstochasticprocessesbasedondescriptivestatistics;however, they canbeadditionally usedinvarious real-lifecontexts suchasanomaly detectionmodeling andAgentbehavior.KarlinandTaylordefineastochasticMarkovProcessas:

AstochasticprocessisaMarkovprocesswhen 𝑃 𝑟{𝑎 <𝑋_𝑡 ≤𝑏|𝑋_𝑡

1 =𝑥1, 𝑋𝑡₂ =𝑥2, . . . , 𝑋𝑡𝑛 =𝑥𝑛}

=𝑃 𝑟{𝑎<𝑋𝑡 ≤𝑏|𝑋𝑡𝑛 =𝑥𝑛}

where𝑡_𝑖 ∈ 𝑇 , and𝑡₁ < 𝑡₂ < . . . < 𝑡_𝑛 < 𝑡. A discrete time Markov Chain (MC) {𝑋_𝑛}isaMarkovprocesswhosestatespaceisacountableorfiniteset,andforwhich 𝑇 =(0,1,2, ...).Itiscustomarytospeakof𝑋_𝑛beinginstate𝑖if𝑋_𝑛 =𝑖.Theprobability

of𝑋_𝑛₊₁ beinginstate𝑗,giventhat𝑋_𝑛 isinstate𝑖,isnotedby𝑃𝑛,𝑛+1

𝑖 𝑗 ,i.e.

𝑃𝑛,𝑛+1

𝑖 𝑗 =𝑃 𝑟{𝑋𝑛+1= 𝑗|𝑋𝑛 =𝑖}.

(30)

As such, 𝑃𝑛,𝑛+1

𝑖 𝑗 = 𝑃𝑖 𝑗 is independent of𝑛, and 𝑃𝑖 𝑗 is the probability that the state

valueundergoesatransitionfrom𝑖 to𝑗 inonetrial. Itiscustomarytoarrangethese probabilitiesasamatrix,thatis,asquarearray. WefurtherrefertoP= ||𝑃_{𝑖 𝑗}||asthe Markov matrix, ortransition probabilitymatrix ofthe process. As the measurable quantityofstatesisfinite,thequantityofrowsisequivalenttothequantityofstates. [72]

ProbabilisticmethodssuchasshownbyMenascein[84], Barrosetal. in[9], andKantetal. in [70], use a MC model for generating background web traffic. However, in [70], we observe a loomingfaultintypicalMCapproaches: thatindividualusersaremodeledindividually. Individ-ualmodelingofuserscannoteasilyscaleinmodernweb-basedsoftwareLT.Theinherentlackof scalabilitythatsuchover-abundanceofindividualizedmodelsimpliesleadsustosuggestmeans ofclustering userbehavior profiles, suchas those shown byBarth todiscover groups inwikis [10], Xiongetal. forclusteringsequences ofcategoricaldata [137],Melnykovforclusteringof clickstream data [83], and Benson for clustering ofhigher order mathematical models such as tensors[12]. Suchclusteringisperformedtoreducethenumberofindividualmodelsbeingused andthus unnecessaryperformance overhead. The directiononMCclusteringfor model-based LTbyMenasce[84]wasfollowedbyvanHoornwithMarkov4JMeter[58]andWESSBAS[59].

The workbyBarros etal. [9]showsthatstatic LTmethodsmust beabletoreactto chang-ing user behaviors, and the underlyingprobabilistic models should be updated periodicallyto accuratelyrepresenttheSUT.Suchstaticmethodsprovidevaluableinsightintohowthesystem performsundercertaintypesofload; however,dynamicinteractionwiththe SUOprovides ad-ditionalinsightintohowtheSUTreactstodifferenttypesofloadwhichmightnotbeaccounted for by QA. While static methods might provide information onone particular part of the sys-tem,theydonot alwaysprovideabroadviewofhowthesystemmight holisticallyreactunder arealisticproduction-levelloadorreacttochange withinanobservedHMS.Changewithinan observedHMSisalsostudied innewerworkbyBezemer etal. andSchulzetal. [13,104] who statethattheproblemofstaledataexistseveninmodernDevOpsenvironments,preventing au-tomatedmodel-basedLTfrombeingadoptedintoDevOpsandAgiledevelopmentenvironments. TheworkbySchulzetal.representsaprescriptiveeffortsuchasthetop-downapproachinrole

(31)

mining tomappingthe changedelta C𝛿 withinacontinuousintegration environment [104]. In

order toprovideadynamicapproach toaddressingstale, missing,and incompletedata, we ex-tenddescriptiveMCmodelswithformalisms,efficientalgorithms,andinferentialmethodssuch asthosegivenbymachinelearning.

2.3.ConceptModeling

Weshowrelevantmethodsandideasfromprocessmining,userbehavioranalysisandmodeling, web application modeling, and conceptual models of knowledge (ontology). Additionally, we showpreviousworkusingWikipediadatawhichisrelevanttoourmethodsandresults.

The term‘ontology’founditsmathematical underpinningswiththe workofWille,who es-tablishedfoundationsforwhathetermed‘ConceptLattices’[131,132,133]. Thisseriesofworks focused on providing an applications-centric bridge to the field of lattice theory withthe key contributionbeinga formaldefinitionofhowconcepts canbestructured. Thecommonlyused descriptionofinformation-specific ontologycamewhenGruberlisted[52] aformalprocessfor the design and subsequent care ofknowledge representation inthe form of ontology. Gruber describedontologyas“aspecificationofaconceptualization: inotherwords,aformal meansof representinghowknowledgeandconceptsareinterrelated”[52].WiththecreationofWikipedia as an online knowledge store (ca. 2001), the global user community of the site has generated large amounts of domain-specific content. While Voss [127] statesthat Wikipedia articles do not havea strictsemantic structure as docomputer scientificontologies (as describedby Gru-berin[52]),suchstructurecanbeinferredfromthesimilaritiesbetweenarticlesandcategories. SinceWikipedia’sinception,effortshavefocusedonextractingsuchformalconceptspecifications fromtheWikipediacontent. PunuruandChen[97]showmethodsforextractingconceptsfrom domain-specificdocumentation,andSchönhofen[103]detailshowcross-referencingWikipedia titles and categories with document words can be used to classify the domain of documents. Similareffortsusingnaturallanguageprocessing[63],[135],[134]havefocusedonextracting on-tologicalstructurefromthesitebythetextinsidethe articles. However,otherapproachessuch asNakayama,Hara,andNishioin[89]andSyed,Finin,andJoshiin[113]focusonthecategories

(32)

associatedwitheacharticle,the resultanthierarchyofconceptsonthesite, andgraphsofboth thelinksandconceptsextractedfromthearticles.OneuserstudyrelatedtoWikipedia consump-tionby Yu,Thom,andTamin[142]describeshowusersofthe sitetendedtohaveexploratory patterns withconsistent ‘backtracking’ (navigationfrom the currentpage to theprevious one) whenthenavigationalinformationofthesitewasnotclearenoughtocompletevarious research-relatedtasks(assignedaspartofthestudy).Theresultsindicatethatbacktrackingrepresentsthe singlelargestpercentageofuserbehaviorontheWikipediasite:userslocatea‘gateway’page(a topicalanchorpointforbasingfurthersearches)anditeratethroughthepagelinksuntilthenext ‘gateway’canbelocated[142].

Wikipediahasreleasedaggregatedclickstreamdatatoaidindeeperunderstandingof Wikipedia users’ behavior; this data may be used to improve upon previous Wikipedia-based ontologicalefforts. From[102],Richproposesoneoftheoriginalworksonusinga data-driven approachtomodelingmultipleusersforclustering. Thisknowledgebasedapproachresultsina staticmodelakintoadecision treefor userrepresentation. Middleton,Shadbolt, andDeRoure in[86]showaknowledge-basedontologicalmethodforprofilingusersinrecommendersystems. Anderson’s novel approach [4] using the MC representation of sequential actions is used for classificationofmalware. Instructioncombinationsareusedas featuresfor clusteringmalware families.

Staticmeansofusingsuchmodelsofprocessesandusagehavebeenusedinsoftware engi-neeringpracticessuchasCleanroomSoftwareEngineering[128]. Schur,Roth,andZellershow howtheirtool‘ProCrawl’canextractbehaviormodelsfromwebapplications[105],andCotreneo etal.showthattheir‘RELAI’frameworkisabletouseaprobabilisticmodelbasedonoperational profilestoimprovesoftwarereliability[35]. ProbabilisticmethodsproposedbyMenascein[84], Barros et al. in[9], Kant, Tewari, and Iyer in [70] usea MCmodel for generating behavioral traffic.However,intheseapproaches,eachuser’sbehaviorisrepresentedbyauniquemodel:an approachwhichdoesnotscalewhentakingintoaccountthevolumeoftrafficforWikipedia.

(33)

In order toreducethe number ofindividualmodels beingused(and thusunnecessary per-formance overhead),many methodshavebeen showntocluster user behavior. Notable works includeBarthin[10]whousedsocialnetworkanalysistoidentifygroupsofusersinwikis(such asWikipedia); Xiong,Wang,Jiang, andHuangin[137]showedhowtouseMCmodelsto clus-tersequencesofcategoricaldata(suchasuserbehavioraltracedata);Melnykovin[83]showeda methodforclusteringclickstreamdata;andBenson,Gleich,andLeskovecin[12]describeameans ofsegmentingnetworkstructures(suchasourclickstreambehavioralgraph)usingatensor-based representationofthedataandspectralclustering.Lee,Ellis,andLouiin[75])showaMCmodel for clusteringwhich iscapable ofdetectingsemantic concepts inaudiodata. Ahlers, Dirk and Mehrpoorshowmethodsofsemanticallysearchingsharedinformationin[1].

Communitydetectionalgorithmsareanothermeansofdetectingsubgroupsorclustersinside ofgraphs. Yangetal. discuss differentcommunitydetectionalgorithms andtheirperformance in[140]. These algorithms havebeenimplemented andtestedin theR programmingplatform withinthe‘igraph’package.

As nomenclatureformeasuringalgorithmic complexity,let𝐸 asthenumberofedgesinthe subjectgraph,and𝑁 tobethenumberofnodes,orvertices.The‘Edgebetweenness’algorithm focusesonusingFreeman’s betweennesscentralityas ameansoffilteringoutedges whichare highlysharedbetweendifferentcommunitiesinordertoextractthosehighlysharededgesfrom the larger graph. The algorithm runs in𝑂(𝐸2𝑁)[140]. ‘Fastgreedy’ runs in𝑂(𝑁 𝑙 𝑜𝑔2(𝑁)) and startswitheverynodeinthesubjectgraphrepresentingasingle-memberclusterorcommunity then proceeds to pairwise match communities until a ‘modularity’ metric is no longer maxi-mal[140]. ‘Infomap’ runsin𝑂(𝐸) and usesdecoders toreconstruct a viewof the graph based on information collected from randomwalks on the network graph[140]. ‘Label propagation’ alsoruns in𝑂(𝐸) and reliesupona majorityheuristicby assigningtokenstoeachnode inthe graph and iteratinguntil each node hasthe same token as the majority of itsneighbors[140]. ‘Leadingeigenvector’runsin𝑂(𝑁2)onsparsegraphs(wherethenumberofedgesismuchlower than the number ofnodes), and𝑂(𝑁(𝑁 +𝐸)) otherwise; itrelies uponusing eigenvalues and

(34)

eigenvectorsofthemodularitymatrixtomaximizemodularityiterativelyuntilthereislessthan an𝜖 level ofincrease inmodularity[140]. The ‘Multilevel’algorithm issimilar to ‘Fastgreedy’, butrunsin𝑂(𝑁 𝑙 𝑜𝑔𝑁)[140].‘Spinglass’runsin𝑂(𝑁3.2)andcanusesimulatedannealingtofind communities[140]. ‘Walktrap’runsin𝑂(𝑁2𝑙𝑜𝑔(𝑁)) forsparsegraphsand𝑂(𝐸 𝑁2)otherwise;it alsoreliesuponrandomwalksofthenetworkandworksonedgedegreedistancebetweennodes tohierarchicallymergecommunitiespairwise[140].

Another effective algorithm to segment a graph is the STRONGCONNECT algorithm de-scribedin[116];thisalgorithmworkstoextractstronglyconnectedcomponentsfromadirected graph, and weakly connected components from an undirected graph(all of which are further discussedin[38]). STRONGCONNECTrunsin𝑂(𝑁 +𝐸) andisdeterministic innature. Build-ingonthe approaches usedinthese algorithms, we showhow theWikipedia clickstreamdata arestructuredandhowMCuserbehaviormodelingandgraphclusteringcanbeusedtoextract semanticclustersfromtheWikipedia.comwebsite.

Therearemany approachestoLTofscale-conscioussoftware systems; forrecentliterature reviewsofacademicwork,we recommendthereaderexaminetheworkofJiang[67]aswellas thecollaborationbetweenJiangandHassan[68].Inaddition,wesuggestareviewofLeitnerand Bezemer’swork[77]forarecentoverviewofindustrialtooling.ChenandShang[27]performa longitudinalstudyofseveralreleasesofthesame opensourceproductstoanalyzeregressionin performancequality, assertingthat mostperformanceregressionscomefrombug fixesandare notnoticeduntilaftertheyaredeployed toproduction-akey reasonforQAeffortssuchasLT beforesoftwareisdeployedtoproduction. Chenetal.[28]discussmanyopenproblemsrelatedto performanceandregressiontestingpartiallyinformingthedirectionofourwork.Performanceis alsoasecurityconcernintheformofDistributedDenialofServiceattacks,asaddressedbyJiang etal.in[66]. Itisimportanttobeabletoevaluatetheperformanceandqualitycharacteristicsof theloadandperformancetestingframeworksthemselves.

(35)

2.4.MetricsandValidation

WhenperformingautomatedperformanceorLT,thereareanumberofapproachesforvalidating andcomparingthespecificsortoftestingmethodsbeingexecutedandoptimizingthevarietyand volumeofthedata beinganalyzed. Maliket. al[82]describe meansofusingPrincipal Compo-nentAnalysistoreducethetotalnumberofmetricsrequiredforanalysisbyperformancequality analysts.Nikraveshetal.[90]showhowvariousworkloadpatternscanbeusedtodeterminethe comparativeaccuracyofneuralnetworkandsupportvectormachineapproachestooptimizing scaleinandscaleoutofcomputingresources. OnemeansofcomparingvariousLTmechanisms is proposedbyGao etal. [49] inorder tomeet large-scale performance characteristic goalsin cloudbasedsystems.Ferreiraetal. [45]proposeatooltoperformsimilarlevelsofoptimization inServiceOrientedArchitectural(SOA)environments.Twoadditionalapproachestovalidating and assessing LT frameworks areproposed by Avritzer etal. in [7] and Laaber etal. in [73]. WhereAvritzeretal. focusontheapproach,Laaberetal. showhowvarioussuitesperformfor continuous assessment. For additional context related toour work, we describe model-driven andadaptivemethods.

2.5.AdaptiveLoadTesting

Realisticscenariosandrapidlyclosingthefeedbackresponselooparetwoofthemostimportant qualitiesfor legitimate systemtesting; inaddition, naturalfeedbackis whyhuman testingwill continue to be critical for performance testing ofproduction-critical systems. The ability of a testingsystemtoadapttoincomingdataisparamounttowardbridgingthegapbetweenhuman effortsandrotemachinescriptexecution.Lenzetal.[99]describeamachine-learningapproach to clusteringof various load and performance testingmetrics toward improving results of the quality assurance lifecycle; however, their approach is offline and does not mention dynamic generationoftestsortestdatafromthelearnedinformation. Incloud-basedsystems,Shariffdeen etal.[108]describeadaptiveauto-scalingstrategiestomeetperformancegoals;similarly,Iqbalet

(36)

al.[62]describesuchadaptivescalingapproachesinwebapplications.Thetendencyofmodeling typicallyreliesuponamanualprocessconsistingofaperiodofobservation,followedbybuilding orrebuildingmodels,andthenexecutingtheperformancetestingitself.TheprocessofLTmust remainadaptiveandreactivetocurrentdata(insteadofrequiringmanualinterventionfromQA). 2.6.Model-BasedLoadTesting

Modelingsystem or user behavior is a powerful means ofusing data from anSUO which can impact approaches toward testing the SUT. Gao and Jiang describe an ensemble-model based approachtoperformancetestingandshowhowitcanoutperformbaselinemodelsunder envi-ronmentalchangesintheSUT[50]. Apteetal.describeAutoPerf [5]formodelingperformance metrics of a system under test whilesimultaneously driving the performance testing process. While AutoPerf requiresprofiling and testingtobe two separateprocesses, lodestone allows foronlinestatisticalperformanceprofiling.Kapooretal.describeastatisticalanalysisandmodel ofuserbehavior withinawebsite[71] whichisbased onthehazard functionandhasbeen ob-served within various production systems (the authors have maintained) as a valid means of approximating the amountoftimeusers spend interactingwitha system. Ramakrishnanetal. [100]discuss metricsfor trackinguserinteraction times. Vögele etal. aswell asvanHoornet al.describeseveralmeansofmodelingofusersandworkloadinsession-basedsystems[59,122, 124,125,126]. Trubianietal. in[119]discussusingoperationalprofilesinLTinordertodetect andcorrectperformance-basedanti-patterns.Wienkeetal.describeadomainspecificlanguage approachin[129]formodelingperformancetestinginrobotics.

Insummary,the modelingofuser,machine,operational,andperformancecharacteristicsof asystem isapowerfulmeansofimproving thatsystem’smeasurablequality and maintainabil-ity.Forbetterresultsinaprospectiveautomatedperformancetestingprocess,adaptivemethods shouldbeincludedforimprovedspeedintestingfeedback.

Thischapter hasdescribedvarioustheoreticalunderpinnings,processes,andimprovements tothecoreofsystemmodelingforLT whichinformandcatalyzethe techniquesinlodestone formodelinguserbehaviorfromsystemlogs.

(37)

Chapter

3. Extracting

User

Models

from

Log

Data

Inminorwayswediffer,inmajor we’rethesame. --MayaAngelou

HumanFamily WedefineterminologyforMCbehaviormodelsandsupportingstructures. Withtheexception ofnecessaryclarificationbetweenthe SUOandthe SUT,theunderlyingsystematicframework we describe below is similar to that usedby Menasce [84] and van Hoorn[58]. However, the separationofsystemsisinthe samespirit asthe continuousintegrationpipelinesdescribedby Schulz etal. [104]. We recallthe underlyingmathematical structures for extensionand show how they fit into a logical framework for learning and extending MCbehavior models for LT fromeventdata.

3.1.SystemsUnderObservationandTest

We employ the following formalisms to define how behaviors within a system S _can _be rep-resented inthe context ofthis work. Simplyput, S should consist atthe minimum ofa setof actions, a set of states, and a transition function in order to match a minimalistic finite state machine definition. We expectthat operationofS produces outputlogsrepresenting the cur-rentoperationalstatusandAgentbehaviors. WeextendSwithadditionalformalismstosupport streaminglog-basedbehaviorprofiling.

Actions(𝐴)={𝐵𝑟 𝑜𝑤 𝑠𝑒(𝐵), 𝑅𝑒𝑎𝑑(𝑅), 𝐴𝑑𝑑(𝐴), 𝐸𝑑𝑖𝑡(𝐸), 𝐷𝑒𝑠𝑡 𝑟 𝑜𝑦(𝐷)} States(𝑆)={𝐸𝑥 𝑡 𝑒𝑟 𝑛𝑎𝑙 , 𝐿𝑜𝑔𝑖𝑛, 𝐿𝑜𝑔𝑜𝑢𝑡 , 𝐻 𝑜𝑚𝑒, 𝐴𝑐𝑐𝑜𝑢𝑛𝑡 , 𝐶𝑜𝑛 𝑓 𝑖𝑔𝑢𝑟 𝑎𝑡 𝑖𝑜𝑛} Roles(𝑅)={𝐴𝑢𝑑𝑖𝑡 𝑜𝑟 , 𝑈 𝑠𝑒𝑟 , 𝐵𝑢𝑠𝑖𝑛𝑒𝑠𝑠 𝐴𝑑𝑚𝑖𝑛𝑖𝑠𝑡 𝑟 𝑎𝑡 𝑜𝑟 ,

𝑇 𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝐴𝑑𝑚𝑖𝑛𝑖𝑠𝑡 𝑟 𝑎𝑡 𝑜𝑟 , 𝑆𝑢𝑝𝑒𝑟 𝐴𝑑𝑚𝑖𝑛𝑖𝑠𝑡 𝑟 𝑎𝑡 𝑜𝑟}

Proficiency(𝑃)={𝑈 𝑛𝑡 𝑟 𝑎𝑖𝑛𝑒𝑑 , 𝑁 𝑜 𝑣𝑖𝑐𝑒, 𝐶𝑜𝑚𝑝𝑒𝑡 𝑒𝑛𝑡 , 𝐸𝑥 𝑝𝑒𝑟 𝑡 , 𝐺𝑢𝑟 𝑢}

Figure3.1.SampleSystemDefinition

(38)

For an example ofhowa sample system canbestructured, refertoFigure 3.1. The sample system represents an enterprise resource planning website and contains six stateswhich may be accessed by users interacting withthe system. The users are subdivided into several roles which represent how they might need to accessdifferent states within the system. Each user hasalevelofproficiencywhich representshowfamiliartheyarewiththesystembased onthe amountoftimetheyneedtoperformthedifferentactionsinthesystem.Theactionsallowedtoa userenablesthemtointeractwiththesitebychangingtheunderlyingdatastoragemechanisms whilemaneuveringbetweenthestatesofthesystem.

WiththeexamplefromFigure3.1inmind,wedefinethesystemS suchthat:

S ={𝐴, 𝑆 , 𝑅, 𝑈 , 𝑃 ,E_𝑘,E𝑢, 𝜙𝑝, 𝜙𝑙, 𝜙𝜇, 𝜙𝑘, 𝜙𝑢}.

Let𝐴bethesetofallactionswhich canbetakenwithinS,𝑆 bethesetofallobservablestates insideS,𝑅bethesetofallbehavioralrolesinS,𝑈 bethesetofuserswhichinteractwithS,and 𝑃 betheproficiencythateachuser𝑢 ∈𝑈 canassumewhileinteractingwithS. LetE betheset ofallpossibleerrorsinS. WeclassifysucherrorsintotwodisjointsubsetsofE: knownerrors E𝑘,andunknownerrorsE𝑢.

Thefollowingfunctionsrepresenthoweachoftheabovesetsareusedtomodelthebehaviors ofentitiesinS_.

TransitionFunction

Let𝜙_𝑝 :𝑆×𝑆 ×𝐴×𝑅 →_R_[₀_,₁_] beaprobabilityfunctionsuchthat,givenasourcestate,atarget state,anaction,andarole,returnsaprobabilitybetween0and1thattheactionwilltakeplace.

(39)

ProficiencyFunction

Let𝜙_𝜇 :𝑆×𝑆×𝐴×𝑅×𝑃 → _N_[₀_,_∞)beaprobabilityfunctionsuchthat,givenasourcestate,atarget state,anaction,arole,andaproficiency,returnsthediscretemeantransitiontimebetweenthe statesforthegiveninputtupleinsecondsbetween0and∞.

RoleFunction

Let𝜙_𝑙 :𝑆×𝑆×𝐴×𝑅 → {0,1}beabinaryfunctionsuchthat,givenasourcestate,atargetstate, anaction,andarole,returnsa1iftheinputtuplerepresentsalegaltransitionanda0iftheinput tuplerepresentsanillegaltransition. SuchrolescanbevisualizedasinFigure3.2andFigure3.3 usingthesample systemdefinedinFigure3.1. InFigure3.2,the ‘SuperAdministrationUser’is capable ofperformingevery actionwhile instates𝜙₂, 𝜙₄, and𝜙₅; incontrast, Figure3.3shows howasample‘User’iscapableofreadingandbrowsingstate𝜙₅,butcannotevenaccess𝜙₄.

KnownErrorFunction

Let𝜙_𝑘 :𝑆×𝑆×𝐴×𝑅 → {0,1}beabinaryfunctionsuchthat,givenasourcestate,atargetstate, anaction, anda role,returns a1 if theinput tuplerepresentsa knownerrortransition anda 0 otherwise. Aknownerrortransitionrepresentsastaticsetofdiscoverableorpre-definedlogical flawsinS.

UnknownErrorFunction

Let𝜙_𝑢 :𝑆×𝑆×𝐴×𝑅 → {0,1}beabinaryfunctionsuchthat,givenasourcestate,atargetstate, anaction,andarole,returnsa1iftheinputtuplerepresentsanunknownerrortransitionanda 0otherwise. AnunknownerrortransitionrepresentsaflawwhichisnotstaticallydefinedinS throughtheuseof𝜙_𝑘.

ThechangedeltaC𝛿betweenS𝑡,andS𝑜mayaffectanyofthesub-componentsof𝑆,including

the functions. It isofnote that𝜙_𝑢, 𝜙_𝑘 are not explicitlydefined inS_𝑜,but they implicitly exist if the restof theconditions abovearefollowed; however, inS𝑡, the errorfunctions𝜙𝑢, 𝜙𝑘 may

specifyfailure logicfor testingAgentstodiscoveras partofevaluatingan LTapproach. There mayalso exist atesting-error oracle separatefrom S which iscapable ofalerting observersof

(40)

𝜎₀ 𝜎₁ 𝜎₂ 𝜎₃ 𝜎₄ 𝜎₅ 𝐵 𝐵 𝐵 𝑅, 𝐸, 𝐴, 𝐷 , 𝐵 𝐵 𝐵 𝐵 𝑅, 𝐸, 𝐴, 𝐷 , 𝐵 𝐵 𝐵 𝑅, 𝐸, 𝐴, 𝐷 , 𝐵 𝐵 𝐵 𝐵

Figure3.2.LegalApplicationStateSpaceforUserProfile: ‘SuperAdministratorUser’

𝜎₀ 𝜎₁ 𝜎₂ 𝜎₃ 𝜎₄ 𝜎₅ 𝐵 𝐵 𝐵 𝐵 𝐵 𝐵 𝐵 𝑅, 𝐵 𝐵 𝐵

Figure3.3.LegalApplicationStateSpaceforUserProfile:‘User’

(41)

Knowledge Store Data Processing Pipeline

Event API System Under Observation Process Technology People Event Data Behavior Model Generator Behavior Models System Under Test Event Data Behavioral Agents

Figure3.4.FrameworkforSimultaneousModelingandSimulationofUserBehavior errors through additional event data. For systems such as S𝑜, oracles are useful for showing

operational metrics and failures in production. However, for systems such as S𝑡, oracles are

useful fortrainingand detectingissuesinthe SUT.The datasimulationtestbedwe describein Section3.7providesmoredetailabouttestingoraclesandtheirplaceintheloadtestingprocess. 3.2.EventDataCollection

Theinteractionsbetweenpeople,process,andtechnologywithinanHMSresultinthegeneration ofsystem logs orstreams ofevent data. The difficulty ofextracting events from the SUO and decipheringinformationfromsucheventscanvary;however,wecanassumesomebasicstructure toeventdatastreams. Aneventisafive-tuple (𝜎_𝑠, 𝜎_𝑡, 𝜏 , 𝑢, 𝑒, 𝑠) suchthat𝜎_𝑠 ∈𝑆 isasourcestate, 𝜎_𝑡 ∈𝑆 isatargetstate,𝜏 ∈𝑇 isatime-stamp,𝑢 ∈𝑈 isauniqueidentifierforeachuserorsome distinctentityinS𝑜,𝑒 ∈ ErepresentsadetectederrorinS𝑜,and𝑠isadditionaleventinformation.

ForS𝑜 inFigure3.4,theEventAPI representsa mechanismforcollectingeventsfrom rawweb

responses,filesystems,ordatabases.TheEventAPIisusedtocollectdatafortheDataProcessing Pipeline to parse and pass event tuples into the Knowledge Store. The Knowledge Store is a centrallocation usedfor storing transition probabilitieswithin the 𝜙𝑝 function,mean timesin

the𝜙_𝜇 function,and anyadditional behavioral data for generatingbehavior profiles. For S_𝑡 in

(42)

Figure3.4,the same EventAPI used forthe SUO representsa mechanismforcollecting events from theSUT.Where the datafrom S𝑜 areusedtomodel behaviorpatterns, the datacollected

fromS𝑡 areusedtocalculatetheeffectivenessofanLTapproachandrewardtestingAgentsfor

successfullydetectingflawsintheSUT.

3.3.DescriptiveModelsforAgentBehavior

We conceptualize users and machines within S,S_𝑜, and S_𝑡 as Agents, a simulated collection of which can operate within a MAS. Individual agents and groups of agents can follow vari-ous behavior models or bases. It is important for application stakeholders, technologists, and researchers to understand how behaviors ofsystems and users of systems [11] interrelate; to that end, operationaland simulationmodels should be understandable byspecialists and non-specialistsalike.WedefinetheDescriptiveModelforAgentBehavior(DMAB)asanystatistically descriptivemodelofbehaviorusedtogoverntheactionsofanAgent.

Forexample,lettwoAgents𝐴,and𝐵,beresponsibleforgeneratingrandomnumbersfroma distribution𝐷 .LetussaythatAgent𝐴operatesusingthefivenumbersummarystatisticsof𝐷 : {𝐷_min, 𝐷_𝑄₁, 𝐷_𝜇, 𝐷_𝑄₃, 𝐷_max}.Assuch,Agent𝐴operatesusingaDMAB.Incontrast,letussayAgent 𝐵 operatesusing anopaquedeep-learning modelwhich cannotbeeasily described. Assuch,𝐵 does not operate usinga DMAB.We assertthat aDMAB ismore likelytobe acceptable inan enterpriseMASduetoitsdescriptivenature. Althoughlessdescriptivemodelshavethepotential ofrepresentingcomplexbehaviorpatterns,theoverheadofmaintaininguserconfidenceinsuch models can outweighany perceived value of increased accuracy. The cost analysis between a modelandhowunderstandableitisshouldbeperformedinanysituationwherethereisa ques-tion ofjustifying errant MASbehaviors. Such anexercise isa crucial stepin the development life-cycleofanyautomatedMAS.

(43)

𝜎₁ 𝜎₂ 𝜎₃ 0.4 0.3 0.3 0.4 0.3 0.3 0.4 0.3 0.3

Figure3.5.MarkovChainDirectedGraphforExampleUserBehaviorModel Table3.1.ConditionalProbabilityMatrixforExampleUserBehaviorModel

𝜎₁ 𝜎₂ 𝜎₃ 𝜎₁ 0.40.30.3 𝜎₂ 0.30.40.3 𝜎₃ 0.30.30.4

TheMCisaDMAB,duetothecause-effectnatureofitsmathematicalstructure;assuchithas beenusedinmanyMAS, includingtheLT toolMarkov4JMeter[58]. Markov4JMeter, although notbuiltonfullydistributedandindependentoperatingmodels,doesmodelusersorAgentsas ThreadswithinJMeter[58]. ADMABbasedonMClogiccanbecategorizedintodifferentbases ofbehavior,dependingonavailabledatafromanSUO.

3.4.BehaviorBases

WithinanautomatedMAS,MCbasedAgentscanoperateaccordingtoaprescriptiveor descrip-tivebasisofbehavior.ItisimportanttorememberthatalthoughweuseanunderlyingMC,there are many different approaches to storing behavioral profiles in memory based on event data. Additional methods involve constructing hidden Markov models, Bayesian networks, decision trees/forests, neuralnetworks, Petrinets, as well asmany other abstractions. It isconceivable thenthatourmethodscanbeportedtothosemodelsaswell;however,themainbehaviorprofile structure we explore here is the MC. The MC canbe equivalently represented as a connected

(44)

graphasinFigure3.5oranadjacencymatrixasinTable3.1.Wedefineandfocusonsixofmany possible distinctcategories ofthe MC,which we term: constantbasis, randombasis, user ba-sis,profilebasis,globalbasis,andsemantic basis. Wewillusemodelandbasisinterchangeably throughouttherestofthisdocument.

3.4.1.ConstantBasis

TheConstantBasisofBehavior(CBB)isaprescriptiveDMABrepresentingafixedsetof sequen-tial behaviorsina HMSwithstaticprobabilityoftransition betweentheindividual states. The CBBisrepresentativeofrule-based approachestoLTsuchas inJMeter [117],orincode-based LTmechanisms.

3.4.2.RandomBasis

The RandomBasis ofBehavior(RBB) is a prescriptiveDMABrepresenting random transitions betweenstatesinaHMS.TheRBBisrepresentativeofrandomizedapproachestoLTsuchasin JMeter[117].

3.4.3.UserBasis

TheUserBasisofBehavior(UBB)isaDMABrepresentinghowanindividualusermaybehavein aHMS.ThesebasesmaybelearnedbyobservingindividualusersandmaybeusedtobuildaMC modelwhichrepresentshoweachindividualusertransitionsbetweenstatesinagivensystem.

3.4.4.ProfileBasis

TheProfile BasisofBehavior(PBB)isa DMABwhich comprisesasetofsimilar UBB.ThePBB representsthebehaviorpatternspresentwithinaUBBwithoutrequiringtheadditionalstorage spaceorcomputationalcostsnecessaryforanagentusingthePBBtooperate.

3.4.5.GlobalBasis

TheGlobalBasisofBehavior(GBB)isaDMABwhichcomprisesallUBB.TheGBBiscalculated similarly totheUBBwiththe exceptionofuseridentitydistinction beingremoved. Inessence, theGBB isthatPBBwhichcomprisesthe aggregatedbehaviorpatternsofallusers observedin anSUO.

(45)

logout inventory view role view login start 4 1 2 1 4

3.4.6.SemanticBasis

TheSemanticBasisofBehavior(SBB)isaDMABwhichcomprisesasetoflogicallydivided sub-graphswithinaGBB.The SBBiscalculatedthroughtheuseofgraphpartitioningorclustering methodssuchasTarjan’salgorithm[116]andrepresentsbehaviorpatternswhichareconnected through semantic means. Semantic connections can be procedure calls within an application library, orAPI calls withinthe same module. When onlyaggregateddata suchas theGBB are availablefromtheSUO,theSBBcandividetheGBBsuchthatAgentbehaviorinanautomated MASmaybecontrolledandobservedwithfinergraineddetail.

3.5.BehaviorModelGenerationandClustering

InFigure3.4,weshowthatunderlyingeventdataiscollectedbythe EventAPI;eachMCbasis is constructed from these events. We constructeach MC DMAB by tracking and aggregating the temporaldeltasandfrequenciesofinput-statetooutput-statetransitions for eachuniquely identifiabledatastreamandcreatingasetofUBB.Forexample,inFigure3.6,therunningcounts oftransitionsbetweeneachstate𝜎_𝑖arerepresentedasweightededgesintheKnowledgeStoreas theadjacencymatrixillustratedwiththeheatmapinFigure3.7.Bydividingeachoftheweightsof alloutgoingedgesforeachsourcestatebythetotalnumberofeventsoriginatingfromthesame state, we compute the resultant MCshown in Figure 3.8– the UBB for the observed behavior stream.WecanvisualizesuchaUBBastheheatmapinFigure3.9withcolumnsrepresentingthe

Distributed Load Testing by Modeling and Simulating User Behavior

LSU Digital Commons

LSU Digital Commons

Distributed Load Testing by Modeling and Simulating User

Distributed Load Testing by Modeling and Simulating User

Behavior

Behavior

DISTRIBUTED

LOAD

TESTING

BY

MODELING

AND

SIMULATING

USER

BEHAVIOR

Acknowledgements

Table

of

Contents

List

of

Tables

List

of

Figures

Acronyms,

Terms,

and

Definitions

Abstract

Chapter

1.

Introduction

Chapter

2.

Background

Chapter

3.

Extracting

User

Models

from

Log

Data