Validation of a microsimulation of the port of Dover

(1)

Validation

of

a

microsimulation

of

the

Port

of

Dover

Chris

Roadknight

∗

, Uwe

Aickelin, Galina

Sherman

IntelligentModelling&AnalysisResearchGroup(IMA),SchoolofComputerScience,TheUniversityofNottingham,JubileeCampus,

WollatonRoad,NottinghamNG81BB,UnitedKingdom

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received30March2011

Receivedinrevisedform19July2011 Accepted21July2011

Available online 28 July 2011

Keywords:

Simulation Transport Operationsresearch Agents

a

b

s

t

r

a

c

t

Modellingandsimulatingthetrafﬁcofheavilyusedbutsecureenvironmentssuchasseaportsandairports areofincreasingimportance.Errorsmadewhensimulatingtheseenvironmentscanhavelongstanding economic,socialandenvironmentalimplications.Thisarticlediscussesissuesandproblemsthatmay arisewhendesigningasimulationstrategy.DataforthePortispresented,methodsforlightweight vehi-cleassessmentthatcanbeusedtocalibrateandvalidatesimulationsarealsodiscussedalongwitha diagnosisofovercalibrationissues.Weshowthatdecisionsaboutwheretheintelligenceliesinasystem hasimportantrepercussionsforthereliabilityofsystemstatistics.Finally,conclusionsaredrawnabout howmicrosimulationscanbemovedforwardasarobustplanningtoolforthe21stcentury.

1. Introduction

Simulationsoftware is beingappliedtodiverseareas and is becomingincreasinglycomplex,parameterizedandconﬁgurable

[18,19].Regardlessofhowgraphicallyrealistictheendproductmay appear,thecorestatisticsgeneratedbyanysimulationstillneeds tobevalidatedandveriﬁed.Simulationtoolkitsforseaportsexist

[20]butthesefocusmoreontheshipsandcontainertransfer.In thisstudywearemoreinterestedintheroadleadingtoandfrom theboarding.Eventsandstatisticsthatshowupwhensimulations aretestedmustalsoappearinreallifeandviceversa.Realworld validationofsimulationresultscanbeanexpensive,time consum-ing,subjectiveand erroneousprocessanddecidingexactlyhow muchvalidationtocommissionisusuallyanimpreciseart. Exist-ingmethodsformicrovalidationsuchasnumberplaterecognition andmanualsamplingareexpensiveanderrorprone.Weighingup thecost/rewardratioofvalidationisanimportantbutnon-trivial process.

Trafﬁcmicrosimulationsuseadiscreteevent[17]approachto themovementofvehiclesovertimewherethebehaviourofa sys-temisrepresented asa chronologicalsequence ofevents. Each eventoccursatauniqueinstantintime,witheachnewinstance ofthesystemviewedasnewstate.Theycombinethiswithsome degreeofagentbasedbehaviourwhereelementswithinthe sim-ulationhave a setof parameters and policiesthat they useto

∗_{Corresponding}_author._Tel.:₊₄₄_01158466591.

E-mailaddresses:[email protected],[email protected]

(C.Roadknight),[email protected](U.Aickelin),[email protected] (G.Sherman).

cometodecisions.Agentbasedapproachesaresuccessfullyusedin trafﬁcandtransportmanagement[7].However,despitetheir suit-abilityresearchinthisareaisnotmatureenoughandmoreover “someproblemareasseemunder-studied,e.g.,theapplicabilityof agenttechnologytostrategicdecision-makingwithin transporta-tionlogistics”[7].

Microscopictrafficsimulationmodelshaveunique characteris-ticsbecauseoftheirrepresentationofinteractionbetweendrivers, vehicles,androads.Theincreasingavailabilityofpowerfuldesktop computershasallowedsophisticatedcomputersoftwaretobeused tomodelthebehaviourofindividualvehiclesandtheirdriversin realtime.Microsimulationcanbeappliedtoanyscenario involv-ingcomplexvehicleinteractionsandhasbeenusedtomodelroads, rail,airandseaports[1,2].Ifvalidationisnotproperlyperformed, atrafficsimulationmodelmaynotprovideaccurateresultsand shouldnotbeusedtomakeimportantdecisionswithfinancial, environmentalandsocialimpacts.

Microsimulationbreaksa simulationdowninto thesmallest sensibleconnectedcomponents.Inthecaseofthesimulationof atrafficscenariothatwouldbevehiclesandthesmallestsensible stations(i.e.,tollbooths,roundabouts,junctions,stopsigns).Each micro-componentneedstobeaccuratelymodelledbutitisalso importanttocorrectlydefinedependenciesandflows.Ithasbeen shownthatquestionablesimulationpredictionscanresultfroma lackofdependenciesthatresultfromindependently microsimulat-ingelementsofalargersimulation[14],thisbringsintoquestion howbesttovalidateasimulationmadeupoflargenumbersof sub-componentsandhowtoensurethesimulationdoesnotcontain smallbutsignificanterrors.

Many scientiﬁc search and optimization approaches have analysed the subject of overtraining and the resulting lack of

Open access under CC BY license.

(2)

Fig.1.PlanofthePortofDover.

generalization.Forinstance,neuralnetworksusuallyhavea stop-pingconditionwhichwhen reachedsignalstheend oftraining, beyondthispointtherepresentationmodelcontinuestoimprove onthetrainingsubsetofinstancesbutdecayswhentestedonan unseendataset[15].Asimilarsituationoccursinstatisticswhena statisticalmodelstartstodescriberandomerrorornoiseinsteadof theunderlyingrelationship,hereitiscalledoverfitting[8]. Over-fittingor overtraining ismore likely tooccurwhen a model is unnecessarilycomplex,suchashavingtoomanyparameters rel-ativetothenumberofobservations.Whilelesswellresearched, similarsituationsmayariseinsimulationswhereasimulationis constructedtosuchanaccuracyastocompletelymimicthe sit-uationusedasanexample.Itiseasiertocreateovercalibration errorsusingmodern,componentisedsimulationsoftwarewhere eachindividualelementcanbehighlyconfiguredtobe representa-tiveoftheisolatedsub-systemwithoutrequiringanysystemwide validity.

Theresearchinthispaperinvolvessimulationsandrealworld datafromthePortofDover.Itwaschosenforthisresearchasit isthemostimportanttradingroutebetweentheUKandmainland Europe,hasanintricateandmultilevellayout(Fig.1)andhasa sub-stantialamountofexistingdataandsimulations.Overthepast20 years,thenumberofroadhaulagevehicles(RHVs)usingthePort ofDoverhasmorethandoubledtoover2.3million[4].Looking aheadoverthenext30years,boththePortandUKGovernment haveforecastsubstantialgrowthinRHVfreighttrafﬁc. Approxi-mately3milliontouristvehiclesalsopassthroughtheferryport annuallymakingitakeyEuropeanandglobaltouristgateway.

Thispapersets out toidentifytheperformance and charac-teristicsofamicrosimulationapproachtoclosedsystemvehicle simulation withparticular referencetothe stability and repro-ducibilityofthesimulations.Thenextsectionofthisarticleoutlines existingdata,statisticsandgraphsforthePortofDover,Section

3 discusses the simulation software package VISSIM,Section 4

summarisesthecharacteristics,beneﬁtsanddrawbacksofusing probabilisticroutingand/oragentbasedrouting,Section5 intro-ducesanovelvalidationprocedurewhichistestedatthePortof DoverandSection6offerssomeconclusions.

2. Doverexistingdata

LookingattheRHVtrafficforthe24hcycleoverafullyearit isapparentthatsystematicflowvariationsoccur,Fig.2highlights somekeyfactsaboutthis flowfor 2009.Forinstance the max-imumRHVflowisapproximately4timestheminimumflowat between1and4vehicles/min.Thelowestflowwasbetween2:00 amand2:15amandthehighestflowbetween3:15pmand3:30 pm.Thesemeasurementsweretakenattheweighbridge,hereall RHVsareweighed,timestampedandthedriversideofthe vehi-clenoted.Thisshowedthat1,194,973RHVsexitedtheUKviathe portin 2009,ofthese960,878werelefthanddrive.Thearrival statisticsattheweighstationsisrelatedtothearrivalattheport in generalbut modified atpeak periodsduetothefirst bottle-neckattheport,theCustomscheck,wherepassportsareinspected andalsobythequeuingattheweighbridgesthemselves.This ini-tialcheckhastheeffectofsmoothingtheflowtotheweighbridge andticketingkiosksbecauseatpeaktimesqueuesbuildup,this effectivelyreducesthebursty-nessofthetrafficflow.Thisdoesnot changetheoverallnumbersgoingthroughtheport,justthearrival dynamics.

(3)

Fig.2.AggregatedRHVﬂowsfor15mintimewindowsforthewholeof2009.

Fig.3.CCTVofvehiclesarrivingattheport.

Fig.4.SampleofinterarrivaltimesofvehiclesatthePortofDover.

(4)

Fig.6. VISSIMsimulationofthePortofDover.

3. UsingVISSIMasasimulationtoolkit

Trafficsimulationsoftransportnetworkstraditionallyusea dis-creteevent, cellularautomatastyle approach. Examples of this include TRANSIMS [9], PARAMICS[10], CORSIM [11] and more recentlyVISSIM[12].Fig.6showsadetailedexampleofaVISSIM simulationtheportandtheapproachingroadsandroundabouts. VISSIM[3]isaleadingmicroscopicsimulationprogramfor multi-modaltrafficflowmodelling.Ithasahighlevelofvehiclebehaviour detail that can beused to simulateurban and highway traffic, includingpedestrians,cyclistsandmotorisedvehicles.Itisahighly parameteriseddesignsystemthatallowsalotofflexibility.

VISSIMmodelsprovidedetailedestimatesofevolvingnetwork conditionsbymodellingtime-varyingdemandpatternsand indi-vidualdrivers’detailed behaviouraldecisions[3].Severalmodel inputs(suchasoriginflows)andparameters(car-followingand lane-changingcoefficients)mustbespecifiedbeforethese simula-tiontoolscanbeapplied,andtheirvaluesmustbedeterminedso thatthesimulationoutputaccuratelyreplicatestherealityreflected intrafficmeasurements.Havingaccesstoanaccuratesimulation ofthePortofDoverhaslargebenefits

There are several significant choke points around the port, places where queues appear and significant delays can arise, namelythepassportcheckingarea,theRHVweighbridgeandthe ticketing booths.Delayscan alsobeintroducedwithadditional security checks to a randomlyselected percentage of vehicles. TherearefiveweighbridgesthatallRHVsmuststopat,RHVsare guidedintothelefttwolanescomingofftheEasternDocks round-aboutfeedingintothethreeleftmostcustomschannelsastonot impedetheflowofothervehiclesintotheport.Thewaittimeat theweighbridgeismodelledasanormaldistributionwithamean of20sandstandarddeviationof2s.

VISSIMallowsthespeciﬁcationof aninitialrandomnumber seed,thisallowsforthesamesimulationtoberepeatedlystressed withadifferentsequenceofrandomnumbersbutalsoallowsdirect comparisonsofdifferentscenariosusingthesamerandom num-bers.Variabilitybetweenrunswithdifferentseedsisagoodmetric forhow robustthesystemis. Largedifferencesin runstatistics whenusingdifferentrandomnumberssuggestseithersomekind ofchaoticdata/environmentinteractionoranillogicaland patho-logicalfaultinthesimulationdesign.

Eachsection ofroad, link,junctionetc. hastobeaccurately modelled.Thesimulationmightdevelopaninbuiltfaultwhereby asmalldesignaspectthatappears(onsomelevels)tobesensible produces considerable variation in validation statistics just by modifying the random number seed. For instance, an integral

componentoftrafficsimulationsisthedecisionsmadebydrivers astheynavigatethedesiredroadsections.Laneselection, over-taking, acceleration, deceleration, follow gaps are all examples of driver behaviour parameters. The Port of Dover has many laneselectionpoints,andthenumberpossibleoflaneschanges repeatedly.Bymonitoringthelaneusagewecanseethedesired occupancyratesoflanesbutconfiguringthesystemtocorrectly reflectthisisnon-trivial.Forinstance,weknowtheoccupancyof thefiveweighbridgesoverthewholeof2009wasforbays1–5 were22.3%,25.4%,22.8%,15.6%and13.9%respectively.

Onewaytoenforcethisratioistouseaprobabilistic,‘roulette wheel’stylelaneselectionpolicy.VISSIM,along withmost sim-ulation toolkits, offersmethods tospecify probabilistic routing wherebyadefinedpercentageofvehiclesaresentdownunique routes. This is a piecewise technique that can be reapplied at variouslocationsaroundasimulation.While thesemethodsare attractivefromacalibrationperspectiveasexactrepresentations ofexisting statisticscan beensured, theprocess isan unrealis-tic one as it assumesthat drivers make probabilistic decisions atprecise locations.Soin this case whena vehiclearrives ata point prior totheweighbridges it is allocatedone ofthe lanes basedontherespectiveprobabilities.Itturnsoutthatthismethod leadstosignificantvariationsintriptimesdependingonthe ini-tialrandomnumberseed,thiscanbeseeninagraphicofthekey areas ofthesimulation for the2 differentruns(Fig.7).Oneof thebenefitsofgraphicalmicrosimulationisthatthe2Dand3D simulationshelp theresearcher tovisualise a newschemeand itspotentialbenefitsbutalsotohighlightunrealistic behaviour.

Fig. 7 shows the congestion at the decision point for 2 differ-ent runs.Using probabilistic routing toenforce correct routing percentagesisaclearcaseofovercalibrationaffectingsimulation brittleness.

Theserunshaveidenticalandrealisticinboundtrafficflowrates thathavebeenconstructedbasedonobservationsofflowsatpeak rates,yetconsiderabledifferenceisbehaviour.Theflowswere gen-eratedbyrecordingarrivalattheportusingCCTVcamerafootage andconstructinganarrivalprocessbasedon2minsegments.Each 2minsegmentproducedexactlythenumberofvehiclesrequired (Fig.8),vehiclesenterthelinkduringthat2minperiodaccordingto aPoissondistribution,ifthedefinedtrafficvolumeexceedsthelink capacitythevehiclesarestackedoutsidethenetworkuntilspaceis availableagainbutthiswasnotrequiredintheDoversimulation.

Fig.8showshowonebusy90minperiodwasrepresented.Fig.9

(5)

Fig.7.VISSIMsimulationsatidenticaltimes,withidenticaltrafﬁcﬂowsbutdifferentrandomnumberseedsshowinghowthiscaneffectcongestionifprobabilisticrouting isused.

Fig.8. Typical90minarrivalprocessaggregatedinto2minintervalbins.

numbersareusedwithinthesimulationtodecideexact arrival timeswithin2minwindows,decidelaneselectionwhenthatis anoptionandtodecideinitialdriverparameterisationwith cer-tainbounds.Fig.10showsthetriptimesforthesametworunsas

Fig.9withaveragetriptimesforallvehiclesvaryingbysigniﬁcantly betweenthetwosimulationsduringsomepointsofthesimulation. Theprobabilisticapproachhasalackofﬂexibilitywheredriverswill sticktotheirrandomnumberallocatedlaneevenifitiscongested, thissuggeststhelargedifferencesinrunstatisticsarearesultofan inappropriatelaneselectionheuristic,soofthe2optionsproposed earlierthemostlikelyanswerisan“illogicalandpathologicalfault

Fig.9.Queuelengthsforidenticalﬂowsbutdifferentrandomnumber(RN)seeds.

inthesimulationdesign”.Themostimportantissuehereisnot thatpoorsimulationdesigndecisionscanbemadeusingin rela-tiontooneaspectofasimulationwhileusingseeminglycommon senseassumptions.Wefoundmuchmorerepeatableresultscould begainedbyallowingVISSIM’sinbuiltdriverbehaviourfeaturesto selectthebestlane.Eventhoughthedesiredpercentage occupan-cieswerenotenforced,similarweighbridgeratiosweregenerated assimulateddriversavoidedthecongestedcentrelanesatpeak timesand effect of randomnumber seedswas much less.This clearlydemonstratesthatimplementingaheuristicthatappears logical when seekinga desired ratio(lane selection) cancause

(6)

ware(3.16GHzIntelDualCore,2GBRAM),whenrunatmaximum speeda2hsimulationtakesabout7mintocomplete.Theremay besomescopeforuseofamoreparallelenvironmentandthismay beespeciallyusefulifthedriverbehaviourwastobemademore complexandrealistic.

4. Algorithmevaluation

Allsimulationsrequiredecisionstobemade,frequentlyandat manylocations.Forvehiclesimulationsthesedecisionsrelatetothe behaviourofdriversandtheirvehiclesinresponseto environmen-talstimulusandtheirowngoals.Anagentbasedsolutionwould migratethesedecisionstoeachindividualdriverandvehicle.This approachissensibleandrealisticbutextremelydifficulttomanage andconfigure,therearealsocomputationissueswhendecisions arenotaggregated.Decisionscanbemadeonahigherleveland mostmodernsimulationsoftwaresystemsallowformethodssuch asconditional,probabilisticanddeterministicrouting.Thebenefit ofaggregatedhighleveldecisionmakingisthatcalibrationand con-figurationismucheasiertomanageandenforce,forinstanceifwe knowthat30%ofvehiclestakeonerouteand70%takeanotherroute thenasimplerandomorroundrobinselectionprocedurewould ensureanearexactreflectionofthis inthesimulation. Achiev-ingprecisebehaviourstatisticsusingalowlevelagentapproach ismuchmoredifficult.

ThePortofDoverisanidealexampleofthedifﬁculttrade-offs regardingthelocationofdecisionmakingprocesses.Firstlythereis thedriverspeciﬁcdecisionsthataffecthowtheirvehiclebehaves, howmuchspacedotheyleavetothevehicleinfront,whendothey overtake,howharddo theybrake/accelerate etc.Therearealso lanechoicedecisionsthataremade,howquicklythesedecisions aremade,wheretheyaremade,howaggressiveetc.Inthissection

the Portof Dover. Here everyRHV must come to a halt to be weighed,andeverydriverchoosesfrom5lanes.Thelaneselection dataforthewholeof2009wasmadeavailablesowehadexcellent statisticsforhowwhichdriverschosewhichlanesatwhichtimes. Sowhenmakingthedecisiononwhichlanetochooseweassessed twooptions:

1. Probabilisticrouting.Atasetpointpriortotheweighbridgea randomnumberisgeneratedandabiasedroulettewheel selec-tionapproach[16]isusedtotellthedriverwhichlanetohead for.Soifweknew10%ofdriversusedlanefourwecould gen-eratearandomnumberbetween0and1andifthenumberwas greaterthanzeroandlessthanorequalto0.1thenthedriver wouldselectlane4.Theratiosofvehiclestolanesisdifferentat differentflowrates,soacoarseapproachwouldbetousethe averagelanesectionratiosforthewholeoneyearperiodforall flowrates(non-flowspecificprobabilisticrouting)andamore precisemethodwoulduseflowspecificlaneoccupancyratios (flowspecificprobabilisticrouting)

2.Agentbasedrouting[7].Eachdriverhasasetofconfiguration parameters that decides when they shouldovertake, change lanes,slowdownetc.Theseconfigurations,alongwiththeroad layoutandvolumeoftrafficwouldbeallowedtodictatetheflow ofvehiclesthroughtheweighbridge

Itisimportanttorememberthatevenwhenvehiclesarebeing probabilisticallyroutedtheremaystillbesomeﬂexibilityinhow theyarriveattheirdestination.Inthisscenariovehiclesarerouted tooneof5possiblelanes,butthedecisiononwhichlanetheymust occupyistakenseveralhundredmetersfromthestartofthatlane givingthemtimetochangelanessafely,usingtheirsuiteofdriver behaviourparameters.

(7)

Fig.12.Laneoccupancyratiosatvariousarrivalratesforrealworlddata.

Weuse5differentﬂowratesintheseexperiments,theseare basedonrealworlddataextractedfromvideoimagesofvehicles enteringthePortandthetimestampedweighbridgedata.Therates rangefrom30vehicles/hto800vehicles/h.Allgraphsshowaverage laneselectionortriptimestatisticscapturedover21replications and errorbars based on standard deviationor quartiles where appropriate.Fig.11showsanoverheadplanoftheweighbridges withlanenumbering,withlaneﬁvebeingfarthestfromthesea andlane4thenearest.

Fig.12showswhatwasactuallyobservedattheweighbridge, againuseoflanes1and2predominatebuttoalessmarkeddegree withtheratioofvehiclesusinglane3almoststaticatallrates.

Fig.13ashowsthelaneoccupancyobservedfortheagentbased simulationapproach.Atlowflowlevelslanesoneandtwomake upnearly100%oflaneusage,thisisaresultofthelowoccupancy oflanesmeaningthevirtualdriversseldomneedtochangelanes. Astheflowincreasesall5lanesareusedmore,asqueuesforlanes 1and2develop.Fig.13bshowsthesamestatisticsforthe prob-abilisticroutingapproach.Fig.14showstheaverageerrorinlane occupancypredictionforthetwomethodsatdifferentflows,itis apparentthatasflowincreasestheaccuracyoftheagentbased approachimproves.

Whilethecorrectreflectionofrealworldoccupancyisan impor-tantgoalforthissimulation,it isalsoimportanttocheckother metricsofevaluation,onesuchmetricisthetriptimeorhowlong ittakesforRHVstotravelbetween2pointsofthePort.Whilethe lastsectionshowshowcloselytheweighbridgeoccupancy statis-ticscanbeengineered,thisexperiment willshowhowwellthe differentsimulation methodsreflect therealpoint topointtrip times,regardlessofhow welltheweighbridgeselectionis per-formed.Forthisexperimentweuseamuchshortertripdistance thatstartsafterthecustomscheckandcontinuesuntilshortlyafter theweighbridge,thiscapturesthedelaycausedbycongestionat andapproachingtheweighbridgeanddiscountstheeffectsofthe ticketingandcustomsstoppoints.Thenon-flowspecific proba-bilisticroutingroutesvehiclestodifferentlanesbasedonrealworld

dataoflaneoccupancy,ensuringnearperfectlaneoccupancy statis-ticswhereastheagentbasedapproachreliesontheinbuiltdriver behaviourprogrammedintoeachvehicleagenttodecidewhen tochangelanesandhenceoccupylanes.Fig.15showshowboth theagentbasedandprobabilisticroutingapproachesareaccurate reflectorsoftriptimeatlowandmediumflowratesbutalsoshows how theprobabilisticrouting approach gives muchhigher pre-dictedtriptimesthanthoseactuallyobservedathighflowrates. Thisisduetoaninflexibilityoftheprobabilisticroutingapproach wherebyoncethedesiredlanehasbeenselectedthedriverhas noabilitytoover-rulethisdictateandmaycauseconsiderableand unnecessarycongestionbypursuingtherequirelane.Table1gives anoverviewofhowmucherrorisassociatedwhicheach simula-tionmethodwhenmeasuredagaintriptimeandlaneoccupancy. Thetwolargesterrors(X)aretheagentbasedapproachperforming poorlyatlowflowratelaneselectionandtheprobabilisticrouting approachperformingpoorlywhentriptimesaremeasuredatvery busytimes.Theperformanceofbothofthesescenarioscouldbe improvedbyintroducingmoredriverintelligencebutnotwithout asignificantincreaseincomplexityandsimulationruntimes.How tobestconfigurethesimulationgiventheavailableroutingoptions isverymuchdowntomodellersrequirementsbutknowledgeabout theprosandconsofdifferentroutingmethodsisstillessentialeven ifa“perfect”solutionacrossallflowratesisnotavailable.

5. ValidationusingBluetooth

Validatingmicroscopictrafﬁcsimulationmodelsincorporates severalchallenges becauseof the incompleteness and rareness of validation data. Validation data is also usually measured in aggregate forms and not at the level of the individual vehicle. Thecost-performancerelationshipofvalidation isanimportant functionthatshouldbewellunderstoodandusedwhendeciding whatextentanyvalidationshouldbetakentoo[13].Mostofthe modelvalidationresearchusesaveragelinkmeasurementsof traf-ﬁccharacteristics[6].However,theseapproacheshavelimitations

Table1

Performanceoftwosimulationmethods,using2metricsand4arrivalrates(X=poor,√=ok,√√=good,√√√=verygood,√√√√=excellent).

Arrivalrates

Low(30–90vehicles/h) Medium(300vehicles/h) High(600vehicles/h) V.high(800vehicles/h)

Agent(laneselection) X √ √√ √√

Probabilisticrouting(laneselection) √√√ √√√√ √√√√ √√√

Agent(triptime) √√√√ √√√√ √√√ √√√

(8)

Fig.13.(a)Laneoccupancyratiosatvariousarrivalratesfortheagentbasedsimulationapproach.(b)Laneoccupancyratiosatvariousarrivalratesfortheprobabilistic routingapproach.

including possible non-obvious inconsistency between the observedandsimulationestimatedvariables.Herewedecidedto usepassiveBluetooth[5]monitoringtosampleavehicleslocation byuniquelyidentifyingitusingtheirBluetoothsignalid.Notall vehiclesemitaBluetoothsignal,sowewantedtotesthowuseful itwasasasamplingmetric.In termsofcost,passiveBluetooth monitoringcouldbedeployedatmuchlowercoststhancamera

Fig.14.Errorsinlaneoccupancypredictionforvarioussimulationmethods.

basedsolutionssuchasautomatednumberplatereadingorhuman videosampling.

Twosamplinglocationswerechosennearthebeginningand endofthedrivers’tripthroughthePortofDover,fromentrytothe Porttowaitingtoembarkontheferry.Thesesamplinglocations arelabelled1and2onthediagramofthePort(Fig.1).Thefirst samplinglocationusedwasthesecuritycheckareaclosetothe beginningoftheroutethroughthePortofDover.Here someof thedrivershavetheirpassportscheckedandalldriversslowdown tofindoutiftheyaretobechecked.Bluetoothcanbedetected atrangesupto100m[3]withoutsophisticatedequipmentbutit isheavilydependentonconditions.Thisareaisagoodlocation forBluetoothmonitoringbecauseitisundercover(allowingsome degreeofsignalreflection),driversaredrivingslowly,thecapture pointisclosetothevehiclesanddriversusuallyhavetheirwindows openforpassportchecks.Theonlydownsidetocapturinghereis thatthereare6possiblelanesfordriverstotake,butmosttake the3centralones.Duringthecaptureperiodof4happroximately 1200vehiclespassedthroughthisareaand796Bluetoothdevices wereregistered.SomevehiclesmayhavemorethanoneBluetooth devicesotheexactpercentageofvehiclessampledisnottrivialto ascertainbutdiscardingallbutoneofmultiple,timesynchronised Bluetoothsignalslargelyremovesreplication.

(9)

Fig.15. Triptimesthroughtheweighbridgecomplexatdifferentarrivalratesforrealdataandsimulationmethods.

Bluetoothsignalsthanlocation1(125vs.796)forasimilarperiod, eventhoughtherearefewerlanes.Thereasonsforthisincluded muchfeweropenwindows,fasterspeeds,andapropensitytouse themiddlelaneof3.WhentheuniqueBluetoothaddresseswere comparedwefoundatotalof104examplesofaBluetoothdevice beingfoundatbothlocations.Fig.16showsthetriptimesregistered forthese104Bluetoothemittersthatwerecapturedatthestartand ﬁnishofthetripthroughthePortofDover.Visuallyitisapparent thattherearegeneraltrendsofincreasinganddecreasingtriptime associatedwithqueuesandcongestioninterspersedwithlonger orshorterindividualtripsthatcouldbeexplainedbyeventssuch assecuritychecks,vehicles parkingtoretrieveitemsfromtheir bootormotor bikes/fastmoving vehiclesnegotiatingthecircuit quicker.

Arrivalofanonymousvehiclesatvariouslocationswasalso cap-turedusingloggingattheentrycamera,theweighbridgeandthe ticketingkiosks.Thesecaptureswereallpartial,thevideocapture missedsomevehiclesbecauseofobscuring,theweighbridgeonly capturedRHVsandtheticketingwasonlyavailableforasubset oftheferryoperators.UsingtheserealﬂowsthecurrentVISSIM

simulationwasstressedwithatrafficflowthatwasanaccurate representation of the real flow. Video cameras werealso used tocapturevehicledetailsatthesamelocationsattheBluetooth capture, enablingthederivation oftriptimes thatcanbe com-paredwiththeBluetoothderivedtriptimes.Fig.17showshowthe SimulationcomparedwiththeBluetoothcaptureandthevisual inspectioncapture.Thesametimeframeandarrivalratewasused foreachtrialandover100vehiclesweresimultaneouslysampled foreachexample.TheBluetoothandVisualCaptureareveryclosely correlated.Thecorrelationbetweenthesimulationresultsandthe captureresultsislessstrongwithalargerpeakatthe5minbin. Thiswouldsuggestthesimulationallowingvehiclestoprogressin aslightlytooregularway.Onecouldspeculatethatthisisdueto thesimplifieddriverbehaviour.Lookingatthestatisticsforthe3 datasets(Table2)itappearsthatthesimulationproducesslightly lowertriptimesbutthelargestandarddeviationsmeanthisisnot toastatisticallysignificantdegree.Thereisavisibledifferencein thesizeofthepeakat5minforthesimulationvs.therealworld capturesbutagain,giventhestandarddeviations,thisisinteresting butnotsignificant.

(10)

Fig.17.ProbabilitydensityfunctionoftriptimesasmeasuredbyBluetoothandvisualsampling.

Table2

Meansandstandarddeviationsfortriptimemeasurements.

Triptimedata(s)

Simulation Bluetooth Camera

Mean 319 358 343

Median 291 301 306

Standarddeviation 110 181 137

Max 729 1250 860

Themore complexa microsimulationisthemore validation optionsthereare.FortheDoversimulationwecouldvalidatethe triptimesatthestartandﬁnishoftheDoverroute.Itisimportant thatthesimulationshowssimilartriptimesandtripstatisticsto realobservationsasshowninFig.17andTable2.Withcomplex microsimulationsitisalsoimportanttovalidateindividual com-ponentsofthesimulation.Itisimportanttodothistoensurethat correctgrossstatisticsarenotbeingachievedbyaggregationof incorrectmodellingofsubcomponents.Forinstance,insome cir-cumstancesthesummationof2incorrectlymodelledsectionsmay giveaccurateoveralltriptimesbutwhenthesystemsisstressed indifferentwaystherearenoassurancesthattheoverallstatistics willremainaccurate.

6. Conclusions

A procedure for validation of microscopic traffic simulation modelsistested,anditsapplicationtothesimulationtoolbox VIS-SIMisdemonstrated.Thevalidationeffortsareperformedatthe microscopiclevelusingBluetoothmonitoringandvisualcounting ofvehicles.Analysisofvarianceofthesimulationresultsversus thefielddata showsBluetoothcapturetobea usefulapproach for micro-monitoringof vehicles but caremust betaken when choosingacollectionsite.WhileBluetoothmaynotremain pop-ularforeverit isentirelyplausiblethatsomekindofdetectible wirelessprotocolwillcontinuetobeavailablefortheforeseeable future.Theprocessdetailedheremaybeconsideredasteptowards thedevelopmentofasystematicapproachforvalidationoftraffic simulationmodels.Wehavetakenthisworkforwardbyusingthe refinedmodeltoassesstheperformanceofmodificationstothe Port.Wealsoplantoinvestigatehowaccuratelya Microsimula-tionmodelcancapturerarebutimportantevents,thekeytothis willbeassuringthatanomalousbehaviourisnotduetosimulation construction.

Whenbuilding a microsimulation great care must be taken to ensure each component is as accurate as possible as small errorsindesigncanleadtodisproportionatelylargeerrors.Thisis

especiallythecaseifactualbehaviourisreplacedwith probabilis-tic approaches, while thesecan ensurerepresentative statistics they can also introduce gross errors when coupled with strict lane discipline and can also be an example of overcalibration. UnlikeareassuchasNeuralNetworksandStatistics,overﬁttingor overcalibrationofasimulationismuchlesswellunderstoodand thereforemethodstoavoiditarelesswellknown.

Thereis arequirementinanagentbasedsimulation tohave appropriatelyintelligentagentsthatbestreflectactualbehaviour withoutintroducingsignificantoverheadsinterms of complex-ityandhardwarerequirements.Havingagentswithrepresentative behaviourreducestheneedtoovercalibratethesystembyusing popularmethodssuchasprobabilisticrouting.Thisisevidenced whenprobabilisticroutingisreplacedbyallowingdriverstomake naturaldecisionsonlaneselectionwhich resultsin muchmore consistentsimulationperformanceathighflowlevels.

Alternative methods for validation of simulation resultsare shown,withBluetoothcaptureprovingtobeaviable,low main-tenancemethodoftriptimesampling.WhetherornotBluetooth emissionisacompletelyindependentmethodofsamplingneeds furtherresearch,this willshow whethercertain subgroupsuse BluetoothmorethanothersorifcertainvehiclesallowBluetooth totransmitmorefreely.AmorecomprehensiveBluetoothcapture maygenerateenoughtriptimestotesttheexistingsimulationat amoreconclusivestatisticallevel.Bluetoothisonlyoneofmany wirelessprotocolsandmaynotbealongtermstandardforvehicle wirelesscomesbuttheapproachshouldbeequallyeffectivewith futurewirelesscommunicationtechniques.Thereisscopeforsome futureworkintheareaofcost-beneﬁtanalysisofvisualrecognition methodssuchasautomatednumber-platerecognitionvs.wireless protocolsamplingasitappearsfromthisworkthatbothmethods offerviablesolutionswithdifferentstrengthandweakness.

Acknowledgements

WewouldliketothankthePortofDover,particularlyDaniel Gillett and Paul Simmons, for their valuable contributions, for supplyingexistingVISSIMmodelsandforallowingaccessto loca-tions within the Port. This project is supported by the EPSRC (EP/G004234/1)andPTVAGtrafﬁcmobilitylogistics.

References

[1] D.Charypar,K.W.Axhausen,K.Nagel,Anevent-drivenparallelqueue-based microsimulationforlargescaletrafﬁcscenarios,in:Paperpresentedatthe 11thWorldConferenceonTransportationResearch,Berkeley,June,2007. [2] L.Ting,E.Heck,P.Vervest,J.Voskuilen,F.Hofker,F.Jansma,Passengertravel

(11)

(b)D.M.Hawkins,Theproblemofoverﬁtting,J.Chem.Inf.Comput.Sci.44 (2004)1–12.

[8]G.C.Cawley,N.L.C. Talbot,On over-ﬁtting inmodelselection and subse-quentselectionbiasinperformanceevaluation,J.Mach.Learn.Res.11(2010) 2079–2107.

[9] L.Smith,R.Beckman,D.Anson,K.Nagel,M.E.Williams,Transims: transporta-tionanalysisandsimulationsystem,in:ProceedingsoftheFifthNational ConferenceonTransportationPlanningMethods,Seattle,Washington,1995. [10]G.Cameron,Paramics:parallelmicroscopicsimulationofroadtrafﬁc,J.

Super-Comput.(1996).

[11]P.D.Prevedouros,Y.Wang,Simulationofalargefreeway/arterialnetworkwith integration,tsis/corsimandwatsim,Transport.Res.Rec.1678(1999)197–207. [12]I.T. Concepts, Vissim Simulation Tool (2001),

http://www.itc-world.com/VISSIMinfo.htm.

[13]R.G.Sargent,Verifyingandvalidatingsimulationmodels,in:1996Winter Sim-ulationConference,Coronado,CA,1996,pp.55–64.

[14]O.Rose,Whydosimplewaferfabmodelsfailincertainscenarios?,Proc.2000 WinterSimulationConf.,2000,pp.1481–1490.

[15]I.V.Tetko,D.J.Livingstone,A.I.Luik,Neuralnetworkstudies.1.Comparisonof overﬁttingandovertraining,J.Chem.Inf.Comput.Sci.35(1995)826–833. [16]D.E.Goldberg,K.Deb,Acomparativeanalysisofselectionschemesusedin

geneticalgorithms,in:G.J.E.Rawlins(Ed.),FoundationsofGeneticAlgorithms, MorganKaufmann,SanMateo,CA,1991,pp.69–93.

[17]S.Robinson,Simulation–ThePracticeofModelDevelopmentandUse,Wiley, 2004.

[18] J.Southern,G.J.Gorman,M.D.Piggott,P.E.Farrell,M.O.Bernabeu,J.Pitt-Francis, Simulatingcardiacelectrophysiologyusinganisotropic meshadaptivity,J. Comput.Sci.1(June(2))(2010)82–88.

[19]D.Balcan,B.Goncalves,H.Hu,J.Ramasco,V.Colizza,A.Vespignani, Model-ingthespatialspreadofinfectiousdiseases:theglobalepidemicandmobility computationalmodel,J.Comput.Sci.1(August(3))(2010)132–145. [20]M.R.Nevins,C.M.Macal,R.J.Love,R.J.Bragen,Simulation,animationand

visu-alizationofseaportoperations,Simulation71(1998)96–106.

conferences.Articlesonmyworkhavealsoappearedinmoremainstream publi-cationssuchasNewScientistandBBConline.AfterleavingBTIhavepursueda researchcareerinitiallyatLancasterUniversityandnowatNottinghamUniversity, workingasaResearchFellowintheareasofsimulationandAI.

UweAickelinUweAickelinisEPSRCAdvancedResearch FellowandProfessorofComputerScienceatThe Univer-sityofNottingham,whereheleadsoneofitsfourresearch groups: Intelligent Modelling and Analysis(IMA).His long-termresearchvisionistocreateanintegrated frame-workofproblemunderstanding,modellingandanalysis techniques,basedonaninter-disciplinaryperspectiveof theircloselycouplednature.Asummaryofhiscurrent researchinterestsismodelling,artiﬁcialintelligenceand complexitysciencefordataanalysis.