CiteSeerX — Distributed Algorithms for Multi-Robot Observation of Multiple Moving Targets

Distributed Algorithms for Multi-Robot Observation of

Multiple Moving Targets

LYNNE E. PARKER

[email protected]

CenterforEngineeringScienceAdvancedResearch(CESAR)

ComputerScienceandMathematicsDivision

OakRidgeNationalLaboratory

P.O.Box2008,Mailstop6355,OakRidge,TN37831-6355USA

Abstract. An importantissue that arises in theautomation of many security, surveillance, and recon-

naissancetasks is that ofobserving themovementsof targetsnavigatingin abounded areaof interest.

A keyresearchissuein theseproblems isthat ofsensorplacement|determiningwhere sensorsshould

belocatedtomaintainthetargetsin view. Incomplexapplications involvinglimited-rangesensors, the

useof multiple sensors dynamicallymovingovertimeis required. Inthis paper, weinvestigatethe use

ofacooperativeteamofautonomoussensor-basedrobotsfortheobservationofmultiplemovingtargets.

Inotherresearch,analyticaltechniqueshavebeendevelopedforsolvingthisproblemincomplexgeomet-

rical environments. However, these previous approaches are verycomputationally expensive {at least

exponential in the number of robots | and cannot be implemented on robots operating in real-time.

Thus,thispaperreportsonourstudiesofasimplerprobleminvolvingunclutteredenvironments|those

with either no obstaclesor with randomly distributed simpleconvex obstacles. Wefocusprimarily on

developingtheon-linedistributedcontrolstrategiesthatallowtherobotteamtoattempttominimizethe

totaltimeinwhichtargetsescapeobservationbysomerobotteam memberintheareaofinterest. This

paperrst formalizestheproblem (which we termCMOMMT forCooperative Multi-Robot Observation

ofMultipleMovingTargets)anddiscussesrelatedwork. Wethenpresentadistributedheuristicapproach

(which wecall A-CMOMMT) for solvingtheCMOMMT problem that uses weighted local force vector

control. Weanalyzetheeectivenessoftheresultingweightedforce vectorapproachbycomparingitto

three other approaches. Wepresentthe resultsof our experimentsin both simulationand on physical

robots that demonstratethesuperiorityoftheA-CMOMMT approachfor situationsin whichthe ratio

oftargetstorobotsisgreaterthan1=2. Finally,weconcludebyproposingthattheCMOMMTproblem

makesan excellent domain for studying multi-robot learning in inherentlycooperativetasks. This ap-

proachis therstof itskindforsolvingtheon-line cooperativeobservation problem andimplementing

itonaphysicalrobotteam.

1. Introduction

Manysecurity,surveillance,andreconnaissancetasksrequireautonomousobservationofthemovements

of targets navigating in abounded areaof interest. A key research issue in these problems is that of

sensorplacement|determiningwheresensorsshouldbelocatedtomaintainthetargetsinview. Inthe

simplest versionof this problem, the numberof sensors and sensor placement can be xed in advance

to ensure adequatesensory coverage of the areaof interest. However,in morecomplex applications, a

numberoffactorsmaypreventxedsensoryplacementinadvance. Forexample,theremaybelittleprior

informationonthelocationoftheareatobeobserved,theareamaybesuÆcientlylargethateconomics

prohibittheplacementofalargenumberofsensors,theavailablesensorrangemaybelimited,orthearea

maynotbephysically accessiblein advanceof themission. Inthegeneralcase, thecombinedcoverage

capabilitiesoftheavailablerobotsensorswillbeinsuÆcientto covertheentireterrainofinterest. Thus,

Thereare manyapplicationmotivations forstudyingthis problem. Targetsto be trackedin amulti-

robot or multi-agent context include other mobile robots, items in a warehouse or factory, people in

a search and rescue eort, and adversarialtargets in surveillance and reconnaissance situations. Even

medical applications, such as moving cameras to keep designated areas (such as particular tissue) in

continuousviewduring anoperation,requirethistypeof technology[18].

In this article, we investigate the use of a cooperative team of autonomous sensor-based robots for

applications in this domain. Ourprimary focus ison developingthe distributed control strategiesthat

allow the team to attempt to minimize the total time in which targets escape observation by some

robot team memberin thearea of interest, given the locations of nearby robots and targets. Because

we are interested in real-time solutions to applications in unknown and dynamic environments, we do

notviewtheproblemfrom theperspectiveofcomputinggeometric visibility,which islargelyaproblem

of planning from a known world model. Instead, we investigate the power of a weighted force vector

approach distributed across robot team members in simple, uncluttered environments that are either

obstacle-free or havearandomdistributionof simpleconvexobstacles. Inother work [32],wedescribe

theimplementation ofthis weightedforcevectorapproach in ourALLIANCEformalism([26], [30])for

fault tolerant multi-robot cooperation. In this article, we focus on the analysis of this approach asit

compares to three other approaches: (a) xed robot positions (Fixed), (b) random robot movements

(Random),and (c)non-weightedlocalforce vectors(Local).

Of course, many variations of this dynamic, distributed sensory coverage problem are possible. For

example, the relative numbers and speeds of the robots and the targets to be tracked can vary, the

availabilityofinter-robotcommunicationcanvary, therobotscandierin their sensingandmovement

capabilities,theterrainmaybeeither enclosedorhaveentrancesthatallowtargetstoenterandexit the

areaof interest, theterrain may beeither indoor (and thus largely planar)or outdoor(and thus 3D),

andsoforth. Manyothersubproblemsmustalsobeaddressed,includingthephysicaltrackingoftargets

(e.g., using vision, sonar, infrared, orlaserrange), predictionof targetmovements, multi-sensorfusion,

andsoforth. Thus,whileourultimategoalistodevelopdistributedalgorithmsthataddressallofthese

problem variations, werst focus onthe aspects of distributed control in robot teams with equivalent

sensingandmovementcapabilitiesworkinginanuncluttered,bounded area.

Thefollowingsectiondenesthemulti-targetobservationproblemofinterestinthispaper,whichwe

termCMOMMT forCooperative Multi-RobotObservationof Multiple Moving Targets. Wethendiscuss

relatedworkinthisarea,followedbyadescriptionofthedistributed,weightedlocalforcevectorapproach,

whichwecallA-CMOMMT. Next,wedescribeandanalyzetheresultsoftheA-CMOMMT approachin

bothsimulationandonphysicalrobots,comparedtothreeotherpoliciesforcooperativetargetobservation

{Local, Fixed,and Random. Wepresentaproof-of-concept implementation ofourapproachonateam

offourNomadrobots. Finally,wediscussthemeritsoftheCMOMMTproblemasageneraldomainfor

multi-robotlearninginvolvinginherentlycooperativetasks.

2. CMOMMTproblem description

Theproblemofinterestinthispaper|CooperativeMulti-RobotObservationofMultipleMovingTargets

(CMOMMT)|isdened asfollows. Given:

S : atwo-dimensional,bounded,enclosedspatialregion

V : ateam ofmrobotvehicles,v

i

;i=1;2;:::m,with360 0

eld ofviewobservationsensors

thatarenoisyandoflimitedrange

O(t): aset ofntargets,o

j

(t), j=1;2;:::;n,such thattargeto

j

(t)islocatedwithin

regionS attimet

We say that a robot, v

i

, is observing atarget when thetarget is within v

i

's sensing range (dened

DeneanmnmatrixB(t),asfollows:

B(t)=[b

ij (t)]

mxn

suchthat b

ij (t)=



1 ifrobotv

i

isobservingtargeto

j

(t)inS attimet

0 otherwise

Then,thegoalisto developanalgorithm,whichwecall A-CMOMMT, that maximizesthefollowing

metricA:

A= T

X

t=1 n

X

j=1

g(B(t);j)

T

where:

g(B(t);j)=



1 ifthereexistsanisuch thatb

ij (t)=1

0 otherwise

under the assumptionslisted below. Inother words, thegoal ofthe robots is to maximizethe average

numberof targetsin S that arebeingobservedbyat leastone robotthroughout themission that isof

lengthT timeunits. NotethatwedonotassumethatthemembershipofO(t)isknowninadvance. Also

notethatifwelaterwantedtouseanadaptivealgorithmsthat improveditsperformanceovertime,this

metriccouldbechangedtosumoverarecenttime window,rather thantheentiremission. This would

eliminatethepenaltyfortheearlyperformanceofprocessesthatimproveovertime.

Inaddressingthisproblem,wedenesensor coverage(v

i

)astheregionvisibletorobotv

i

'sobservation

sensors, for v

i

2V. Thenweassumethat, in general,the maximumregion coveredbythe observation

sensorsoftherobotteamismuchlessthanthetotalregiontobeobserved. Thatis,

[

vi2V

sensor coverage(v

i )S:

Thisimpliesthatxedrobotsensinglocationsorsensingpathswillnotbeadequateingeneral,andthat

instead,therobotsmustmovedynamicallyastargetsappearinordertomaintaintheirtargetobservations

andtomaximizethecoverage.

Wefurtherassumethefollowing:

 Therobotshaveabroadcastcommunicationmechanismthat allowsthemto send(receive)messages

to(from)eachotherwithinalimitedrange. Therangeofcommunicationisassumedtobelargerthan

thesensingrangeoftherobots,but(potentially)smallerthanthediameterofS. Thiscommunication

mechanismwillbeusedonlyforone-waycommunication. Further,thiscommunicationmechanismis

assumedtohaveabandwidthoforderO(mn)formrobotsandntargets 1

.



For all v

i

2 V and for all o

j

(t) 2 O(t), max vel(v

i

) > max vel (o

j

(t)), where max vel(a) gives the

maximumpossiblevelocityofentitya,fora2V[O(t). Thisassumptionallowsrobotsanopportunity

to collaborateto solvethe problem. Ifthetargets couldalwaysmovefaster,then theycould always

evade the robots and the problem becomes trivially impossible for the robot team (i.e., assuming

"intelligent"targets).



Therobotteammembersshareaknownglobalcoordinatesystem 2

.

In somesituations, theobservation sensor on each robot is of limitedrange and is directional (e.g.,

acamera),and canonlybeused to observetargetswithin that sensor'seld of view. However,in this

article,wereporttheresultsthatassumeanomni-directional2Dsensorysystem(suchasaringofcameras

orsonars,theuseofaglobalpositioningsystem,orasingleomnidirectionalcamerasuchasthoserecently

prototypedbySRI,CMU/Columbia,etc.), inwhichtherobot sensorysystemisoflimitedrange,butis

o

3. Relatedwork

Researchrelatedtothemultipletargetobservationproblemcanbefoundinanumberofdomains,includ-

ingartgalleryandrelatedproblems,multi-targettracking,andmulti-robotsurveillancetasks. Nearlyall

ofthepreviousworkinthisareainvolvesthedevelopmentofcentralizedalgorithmsforcomplexgeometric

environmentsusingidealsensors. Whilethesepreviousapproachesprovidetheoreticallyprecisesolutions

tocomplexgeometricalproblems, theyareextremelycomputationallyexpensive(atleastexponentialin

thenumberofrobots)and donotscalewellto numbersofsensorsandtargets greaterthanoneortwo.

Theprevioussolutions alsodonot providea on-line, real-timesolutionsto the cooperativeobservation

problem. Our work diersin that, while we do not solve for complexgeometric environments, we do

provide a heuristic on-line solution that works in simple environments, and that scales well to larger

numbersoftargetsandrobotsinthemidstofsensorandeectorerrors.

Whileacompletereviewofthepreviousresearchisnotwithin thescopeofthispaper,wewillbrie y

outlinethepreviouswork that ismostcloselyrelatedto thetopicof thispaper. Thework mostclosely

related to the CMOMMT problem falls into the categoryof the art gallery and related problems [25],

which dealwith issues related to polygon visibility. The basic art galleryproblem is to determine the

minimum number of guards required to ensure the visibility of an interior polygonal area. Variations

on the problem include xed pointguards ormobile guards that can patrol aline segmentwithin the

polygon. Several authors have looked at the static placement of sensors for target tracking in known

polygonalenvironments(e.g., [8]). These works dierfrom theCMOMMTproblem, in that ourrobots

mustdynamicallyshifttheirpositionsovertimetoensurethatasmanytargetsaspossibleremainunder

surveillance,andtheirsensorsarenoisyandoflimitedrange.

Sugiharaet al.[36] addressthe searchlight scheduling problem, which involvessearching foramobile

\robber"(which wecall target) inasimplepolygonbyanumberof xedsearchlights,regardlessof the

movementofthetarget. Theyshowthattheproblemofobtainingasearchscheduleforaninstancehaving

atleastonesearchlightonthepolygonboundarycanbereducedtothatforinstaceshavingnosearchlight

onthe polygonboundary. They alsopresentanecessaryand suÆcient conditionforthe existenceof a

searchscheduleforexactlytwosearchlightsintheinterior. Theydonotaddressissuesofcomputational

complexity.

SuzukiandYamashita[37]addressthepolygonsearchproblem,whichdealswithsearchingforamobile

target in asimplepolygon by amobile searcher. Theyexamine two cases: one in which thesearcher's

visibility is restricted to k rays emanating from its position, and one in which the searchercan see in

all directions simultaneously. Theirwork assumesasingle searcher. Thepaperpresentsnecessaryand

suÆcientconditionsforapolygontobesearchablebyvarioussearchers.Theyintroduceaclassofpolygons

for which thesearcherwith two ashlightsis ascapable asthe searcher with apoint lightsource, and

givenecessaryandsuÆcientconditionsforpolygonsto besearchablebyasearcherwithtwo ashlights.

LaValleetal. [19]introducethevisibility-basedmotionplanningproblemoflocatinganunpredictable

targetinaworkspacewithoneormorerobots,regardlessofthemovementsofthetarget. Theydenea

visibilityregionforeachrobot,withthegoalofguaranteeingthatthetargetwilleventuallylieinatleast

onevisibilityregion. Aformalcharacterizationoftheproblemandsomeprobleminstancesarepresented.

For a simply-connected free space, a logarithmic bound on the minimum number of robots needed is

established. A completealgorithm for computingthemotionstrategy ofthe robotsis presented,based

on searching anite cellcomplex constructedbased oncritical information changes. In this approach,

thesearchspaceisatleastexponentialin thenumberofrobots,andappearstobeNP-hard.

LaValleet al. [18] addresses the relatedquestion of maintainingthe visibility of amovingtarget in

a cluttered workspaceby asingle robot. Theyare able to optimize thepath along additional criteria,

such as the total distance traveled. They divide the problem into two cases: predictable targets and

partiallypredictable targets. Forthepredictable case,analgorithm thatcomputestheoptimalsolution

is presented; this algorithm is exponential in the dimension of the robot conguration space. For the

remains in viewin a subsequenttime step, and thesecond maximizes theminimum time in which the

targetcouldescapethevisibilityregion. Forthese approaches,thesearchspaceisalsoexponential.

Anotherlargeareaof relatedresearchhas addressedthe problemof multi-targettracking (e.g., Bar-

Shalom[2],[3],Blackman[7],Foxetal.[13]). Thisproblemisconcernedwithcomputingthetrajectories

ofmultipletargetsbyassociatingobservationsofcurrenttargetlocationswithpreviouslydetectedtarget

locations. Inthegeneralcase,thesensoryinputcancomefrom multiplesensoryplatforms. Other work

related to predictingtarget movementsincludes stochastic game theory, such asthehunter and rabbit

game([5],[6]),whichistheproblemofdeterminingwheretoshoottominimizethesurvivalprobabilityof

therabbit. Ourtaskinthispaperdiersfromthisworkinthatourgoalisnottocalculatethetrajectories

ofthetargets, but ratherto nddynamicsensor placementsthat minimizethecollectivetime thatany

targetisnotbeingobservedbyatleastoneofthemobilesensors.

In the area of multi-robot surveillance, Everett et al. [10] have developed a coordinated multiple

securityrobot controlsystemforwarehousesurveillanceand inventoryassessment. Thesystemissemi-

autonomous, and utilizes autonomous navigation with human supervisory control when needed. They

proposeahybridnavigationalschemewhichencouragestheuseofknown\virtualpaths"whenpossible.

Wesson et al. [39] describe a distributed articial intelligence approach to situation assessment in an

automated distributed sensor network, focusing on the issues of knowledge fusion. Durfee et al. [9]

describe adistributed sensor approach to target trackingusing xed sensory locations. As before, this

relatedresearchin multi-robotsurveillancedoesnotdealwiththeissueofinterestin this article| the

dynamicplacementofmobilesensors.

4. Approach

4.1. Overview

Our proposed approach to the CMOMMT problem is based upon the same philosophy of control as

our previously-developed ALLIANCE architecture for fault tolerant multi-robot control ([30]). In this

approach, weincorporate adistributed, real-time reasoningsystemutilizing motivations ofbehaviorto

control the activation of task achieving control mechanisms. For the purposes of fault tolerance, we

utilize nocentralized control,but rather enableeach individual robot to select itsown current actions.

Wedonotmakeuseofnegotiationamongrobots,butratherrelyuponbroadcastmessagesfromrobotsto

announcetheircurrentactivities. This approachtocommunicationandactionselection,asembodiedin

theALLIANCEarchitecture,resultsin multi-robotcooperationthat gracefullydegradesand/oradapts

to real-worldproblems, suchasrobot failures, changesin theteam mission,changesin therobot team,

or failures or noisein the communicationsystem. This approach has been successfully applied in the

ALLIANCE architecture to a variety of cooperative robot problems, including mock hazardous waste

cleanup([27],[30]), boundingoverwatch([29],[28]), janitorialservice[29], boxpushing[26],and simple

cooperativebaton-passing[31],implementedonbothphysicalandsimulatedrobotteams.

In[32],weprovidethedetailsofthebehaviororganizationthatimplementsourA-CMOMMTapproach

in the ALLIANCE architecture. In the current article, we focus on an analysis of the approach and

compare it to three other control mechanisms to determine its usefulness in solving the CMOMMT

problem. Inthe A-CMOMMT approach, robots use weightedlocal force vectorsthat attract them to

nearbytargetsandrepelthemfromnearbyrobots. Theweightsarecomputedinreal-timeandarebased

on the relative locations of the nearby robots and targets. The weights are aimed at generating an

improvedcollectivebehavioracrossrobots whenutilized byallrobotteam members.

Inthefollowingsubsections,weprovidemoredetailsof thisapproach,describingtheissuesoftarget

androbotdetection,computationofthelocalforcevectors,forcevectorweights,andthecombinationof

communication range

predictive tracking range sensing range

vi

Fig.1. Denitionofthe sensingrange,predictivetracking range,andcommunication rangeofarobotv

i

. Althoughthe

exact rangevaluesmaychange fromapplication toapplication, weassumethatthe relativeorderingofrange distances

remainsthesame.

4.2. Target androbotdetection

Ideally,robotteam memberswould beableto passivelyobserve(e.g., throughvisualimage processing)

nearby robots and targets to ascertain their current positions and velocities. Research elds such as

machine vision have dealt extensively with this topic, and have developed algorithms for this type of

passive position calculation. However, since thephysical tracking and 2D positioning of visualtargets

is not the focus of this research, we insteadassume that robots use a global positioning system (such

asGPSforoutdoors,orthelaser-basedMTI Conacindoorpositioning system[15]that isin useat our

laboratory) todetermine theirown positionand theposition of targetswithin theirsensing range,and

tocommunicatethisinformationto therobot teammemberswithin theircommunicationrange.

Inourapproach,eachrobotcommunicatestoitsteammatesthepositionofalltargetswithinitseld

ofview. Toclarifythisidea,Figure1depictsthreerangesthataredenedwithrespecttoeachrobotv

i .

Theinnermostrange isthesensing rangeofv

i

,within whichthe robotcanuseasensor-basedtracking

algorithmtomaintainobservationoftargetsinitseldofview. Themiddlerangeisthepredictivetracking

rangeoftherobotv

i

,whichdenestherangeinwhichtargetslocalizedbyotherrobotsv

k 6=v

i

canaect

v

i

'smovements. Thus,arobotcanknowabouttwotypesoftargets: thosethataredirectlysensedorthose

thatare\virtually"sensedthroughpredictivetracking. Theoutermostrangeisthecommunicationrange

oftherobot, which denestheextentoftherobot'scommunicatedmessages. Forsimplicity, weassume

that the sensing, predictivetracking, and communications rangesare identical (respectively) across all

robotteam members.

Whenarobotreceivesacommunicatedmessageregardingthelocationandvelocityofasightedtarget

that is within its predictive tracking range, it begins a predictive tracking of that target's location,

assumingthat thetargetwillcontinuelinearlyfrom itscurrentstate. Thispredictivetrackingwillthen

give the robot information on the likely location of targets that are not directly sensed by the robot,

so that the robot can be in uenced not only by targets that are directly sensed, but also by targets

thatmaysoonenter therobot'ssensingrange. Weassumethatifnotenoughinformationisavailableto

disambiguatedistincttargets(e.g.,duetoahighdensityoftargets,alowfrequencyofsensorobservations,

alow communicationbandwidth, etc.) then existing trackingapproaches(e.g., Bar-Shalom [3]) should

Magnitude of force vector ^-1 0 1

do1 do2 predictive

tracking range Distance between robot and target

Robot-Target

do3

Fig.2. Functiondeningthemagnitudeoftheforcevectorsactingonarobotduetoanearbytarget.

-1 0 1

dr1 dr2

Distance between robots

Magnitude of force vector

Robot-Robot

Fig.3. Functiondeningthemagnitudeoftheforcevectoractingonarobotduetoanothernearbyrobot.

Notethat thiscommunicatedinformationis notequivalentto globalinformation. Wedonotassuem

that all robots haveinformation bout all targets, since this assumption does not scale as the number

oftargets androbotsincreases. Inthefollowing, robotsonly useinformationaboutnearbytargetsand

nearbyrobotsin theircontrol approach. Thus,ourapproachisnotequivalentto aglobalcontroller.

4.3. Local forcevectorcalculation

Inperformingtheirmission,therobotsshouldbecloseenoughtothetargetstobeabletotakeadvantage

of their(i.e., robots') moresophisticated tracking devices(e.g., cameras),while remainingdispersed to

covermoreterrain. Thus,thelocalcontrolof arobotteam memberisbaseduponasummationofforce

vectorswhichareattractivefornearbytargetsandrepulsivefornearbyrobots. ThefunctioninFigure2

denestherelativemagnitudeoftheattractiveforcesofatargetwithin thepredictivetrackingrangeof

agivenrobot. Notethattominimizethelikelihoodofcollisions,therobotisrepelledfromatargetifitis

toocloseto thattarget (distance<do

1

). Therangebetweendo

2 anddo

3

denes thepreferredtracking

rangeofarobotfromatarget. Inpractice,thisrangewillbesetaccordingtothetypeoftrackingsensor

usedanditsrangeforoptimaltracking. Intheworkreportedhere,wehavenotstudiedhowtooptimize

thesettingsofthesethresholds,leavingthistofuture work. Theattractiontothetargetfallsolinearly

asthe distance to the target varies from do

3

. The attraction goesto 0beyond thepredictivetracking

range,indicatingthatthistargetistoofartohaveaneectontherobot'smovements. Therobotsensing

rangewill liesomewherebetweendo

3

andthepredictivetrackingrange.

Figure 3 denes the magnitude of the repulsive forces between robots. If the robots are too close

together(distance <dr

1

),theyrepelstrongly. Iftherobots arefarenoughapart(distance>dr

2 ),they

haveno eectupon eachother in termsof theforce vectorcalculations. Themagnitude scaleslinearly

4.4. Weightingthe force vectors

Using only local force vectors for this problem neglects higher-level information that may be used to

improvethe team performance. Thus, weenhance thecontrol approachbyweightingthecontributions

of each target'sforce eld onthe totalcomputed eld. Theweightw

lk

reduces robot r

l

's attractionto

anearbytarget if that target is within the eld of viewof another nearby robot. Using these weights

helps reducetheoverlapofrobot sensoryareastowardthegoalofminimizing thelikelihoodof atarget

escapingdetection. Here,eachrobotassumesthatitsteammateshavethesamesensingrangeasitsown.

Ifrobotv

l

detectsanotherrobotv

j

nearbyandwithin sensingrangeoftargeto

k

,then w

lk

shouldbe

setto alowvalue. Inthesimplestcase,sincewedene(insection2)arobotv

l

tobeobserving atarget

o

k

whenitiswithin v

l

'ssensingrange,wecouldassignw

lk

tobezerowheneveranotherrobot iswithin

sensing rangeof o

k

. However, this will increase thelikelihood that atarget will escapedetection, and

thusitmaygivebetterresultstosetw

l

ktosomenon-zerovalue.

Thepropersetting ofw

lk

is alsodependentupon theestimated densityof targets in thevicinity. If

targetsaresparselylocatedinthearea,thentherobotteam riskslosingtrackofahigherpercentageof

targetsifanytargetsareignored. Ontheotherhand,iftargetsaredenselydistributed,thentherisksare

lower. Inrelated work,weare currentlyexploring thepropercomputation ofthese weightsbasedupon

theseissues. Theresultsofthisrelatedworkwillbereported inthefuture.

These weights have the eect of causing a robot to prefer the observation of certain targets over

others. InmorecomplexversionsoftheCMOMMT problem,robotscouldalsolearnabouttheviewing

capabilities oftheir teammates,and discounttheir teammates'observationsifthat teammatehasbeen

unreliableinthepast.

4.5. Combination ofweightedlocal forcevectors

Theweightedlocalforcevectorsarecombinedtogeneratethecommandeddirectionofrobotmovement.

Thisdirectionofmovementforrobotv

l

isgivenby:

n

X

k =1 w

lk f

lk +

m

X

i=1;i6=l g

li

where f

lk

is the force vector attributed to target o

k

by robot v

l and g

li

is the force vector attributed

to robot v

i

by robot v

l

. To generate an(x;y) coordinate indicating the desired location of the robot

correspondingtotheresultantforcevector,wescaletheresultantforcevectorbaseduponthesizeofthe

robot. Therobot'sspeedandsteeringcommandsarethencomputedtomovetherobotinthedirectionof

thatdesiredlocation. Bothofthese computedcommandsarefunctions oftheanglebetweentherobot's

current orientation and the direction of the desired (x;y) position. The larger the angle, the higher

thecommanded rate of steeringand the lowerthe commanded speed. Forsmall angles, thespeedis a

function ofthedistancetothedesired(x;y)location,withlongerdistances translatingto fasterspeeds,

upto amaximum robot speed. Anew commandis generated each time theforce vectorsummation is

recomputed. Whilethis approach does not guaranteesmoothrobot paths, in practice, we have found

thattheforcevectorsummationsyieldadesired(x;y)locationthatmovesrelativelysmoothlyovertime,

thusresultinginasmoothrobotmotioninpractice.

We note here that the velocity and steering command can be overwritten by an Avoid Obstacles

behavior, which will move the robot away from any obstacle that is too close. This is achieved by

treatinganysuchobstacleasanabsoluteforceeld thatmovestherobotawayfromtheobstacle.

It isalso importantto note that theissue ofnoisysensors playsarole in helping to ensurethat the

robotbehaviorcorrespondingtothevectorsummationsisappropriate. Thestochasticnatureofarobot's

vi

v2

v1 o1

o4

o2 o3

communication range of v

i

predictive tracking range of v i

sensing range of vi

(a)

(b)

o5 v2

o1 o4

o2 o3

resultant direction of movement

vi

Fig.4. Exampleofforcesactingonrobotvi. Inthisgure,robotsarerepresentedbysmallblackdotsandtargets

are represented by smallgrey squares. (a) Robot v

i

hasfour targets within its predictive tracking range and

twootherrobotswithin itscommunicationrange. Oneothertarget,o5,isoutsidevi'spredictivetrackingrange.

(b)Forcesacting onrobot vi are attractive forcesto targets o1, o2,o3. and o4,and arepulsive forcefrom v2.

Notethattheattractiontoo

1

islessthanthattoo

2 oro

3

,sinceo

1

isoutsidethesensingrange,butwithinthe

predictive trackingrange. The attractionto o4 is less thanthat to o2 and o3 because o4 is within thesensing

rangeofanotherrobot,v

1

. There isnoattractiontotargeto

5

,sinceo

5

isoutsidev

i

's predictivetrackingrange.

Thereisalsonorepulsionactingonvi duetov1 becausev1 isalreadysuÆcientlyfarfromvi.

locatedequidistantbetweentwotargets thatmoveawayfromthe robotat thesamerate,sensorynoise

canpreventtheforcevectorsfromcancellingtheattractionto zero. Instead,slightdeviationsin sensing

cancausetherobottobegintobeattractedmoretowardsone ofthetargets,leadingtherobottofollow

thatsingletarget,ratherthanlosingbothtargets.

Figure4givesanexampleofthegenerationoftheweightedlocalforce vectorsforrobot v

i

atagiven

point in time. In part(a) of this example, there are ve targets, o

1 , o

2 , o

3 , o

4 , and o

5

, and twoother

robots,v

1 andv

2

,within v

i

'scommunicationrange. Themagnitudeoftheforcevectorsattractingrobot

v

i

to targets o

2 and o

3

isequivalentto the maximumvalue, since those targets arewithin v

i

's sensing

rangebutnotwithinanyotherrobot'ssensingrange. Theattractionofv

i

totargeto

1

islessthanthatfor

o

2 ando

3

,becauseo

1

is outsidethesensingrangeofv

i

(althoughstillwithin predictivetrackingrange).

Theforcevectortotargeto

4

isweightedlessduetoo

4

'spresencewithinv

1

'ssensingrange. Finally,there

isnoattractiontotargeto

5

,becauseitisoutsidev

i

'spredictivetrackingrange. Robotv

i

alsoexperiences

arepulsiveforceduetorobotv

2

,becauseofthecloseproximityoftherobots. Thereisnorepulsiondue

to v

1

,sincev

1

issuÆciently distant. Part(b) ofFigure 4showstheresultantdirection of movementof

v dueto theweightedforcevectors.

4.6. Seeking outtargets

Of course, for any cooperativeobservation technique to be of use, therobots must rst ndtargets to

observe. Alltechniquesaretriviallyequivalentifnotargetsareeverinview. Thus,therobotsmusthave

some means of searching for targets if none are currently detected. We have not yet implemented an

approachforseekingouttargets,butherewediscussafewideasastohowthis mightbedone.

In the A-CMOMMT and Local approaches, when a robot does not detect any target nearby, the

weightedsumoftheforcevectorswill causeeachrobotto moveawayfromitsrobotneighborsandthen

idleinonelocation. Whilethismaybeacceptableinsomeapplications,ingeneral,wewouldliketohave

the robots actively and intelligently seek outpotential targets in the area. Suzukiand Yamashita [37]

addressthisproblem throughthedevelopmentofsearch schedulesfor \1-searchers". An \1-searcher"

isamobilesearcherthathasa360 o

inniteeldofview. Asearch scheduleforan1-searcherisapath

through asimple polygonalarea that allowsthe searcher(or robot) to detect a mobile \intruder" (or

target),regardlessofthemovementsofthetarget. WhileclearlyrelatedtotheCMOMMTproblem,this

earlierworkmakesanumberof assumptionsthat donothold intheCMOMMTproblem: inniterange

of searchervisibility, only asinglesearcher, onlyasingle target, and anenclosed polygonal areawhich

does not allow any targets to enter or exit the area. Nevertheless, this approach could potentially be

extendedandadaptedforuseintheautomatedobservationproblem.

Anotherapproach would haverobots in uence their search to concentratetheir movementsin areas

previouslyknownto haveahigher likelihood oftarget appearance. Forinstance, sincenew targets are

morelikelytoappearnearentrances,arobotcouldspendahigherpercentageofitstimenearentrances.

A challenge would bedevelopingan eÆcientmechanismfordealing withthemoreglobal nature of the

problemwhendealingwithknowledgeofprevioustargetmovementsandtargetconcentrations.

5. Experimental resultsand discussion

Toevaluatethe eectiveness of the A-CMOMMT algorithm in addressing theCMOMMT problem, we

conducted experiments both in simulation and on a team of 4Nomad mobile robots. The simulation

studies allowed us to test larger numbersof robots and targets, while the physical robot experiments

allowedusto validatetheresultsdiscoveredinsimulationin therealworld. Theresultsthatwepresent

below summarize the outcomes of nearly 1,000,000 simulation test runs, and over 750 physical robot

experimentalruns.

Inbothsetsofstudies,wecompared fourpossiblecooperativeobservationpolicies: (1)A-CMOMMT

(weighted forcevectorcontrol), (2) Local (nonweightedforce vectorcontrol), (3) Random (robots move

random/linearly), and (4) Fixed (robots remainin xed positions). TheLocal algorithm computes the

motionoftherobots bycalculatingthesamelocal forcevectorsof A-CMOMMT, butwithouttheforce

vectorweights. This approach was studied to determine the eectiveness of the weights on the force

vectorsin theA-CMOMMT algorithm. The last twoapproachesare control casesfor the purposes of

comparison: theRandomapproachcausesrobotstomoveaccordingtoa\random/linear"motion(dened

below),while theFixedapproachdistributestherobotsuniformly overtheareaS, wheretheymaintain

xedpositions. Intheselast twoapproaches,robotmovementsarenotdependentupontargetlocations

ormovements. Whilewerecognizethatthesecontrolapproachesare"strawmen"approachesinthesense

thatanynumberofothercontrolapproachescouldbeimagined,theyseemtobeobviouschoicesthatare

importanttoevaluate. Theobjectiveofourongoinglearningresearch(brie yintroducedin Section6)is

todevelopaprincipledapproachtosearchthedomainofpotentialsolutionstothischallengingproblem.

The set of experiments conducted in simulationvaried somewhat from the experimentations on the

physical robots. In simulation, westudied twotypesof target movements{(1) random/linear and(2)

evasive{whereasin thephysical robot experiments,weonlystudied random/lineartarget movements.

Inaddition,thesimulationexperimentswereconductedusingacircularregionSwithnoobstacles(other

in onecollection of experimentson the physical robots, wealso populated S with varying densities of

rectangularobstaclesofsizeapproximatelyequaltotherobotdiameter. Ofcourse,thenumbersofrobots

andtargetswecouldexperimentwithwaslimitedonthephysicalrobotteam;insimulation,wetestedup

to10robotsand 20targets,whereasonthephysicalrobot team,wetestedupto3robotsand3targets

(butwithalimitationofonlyatotalof4robotsandtargetsat atime).

Thenextsubsectionsdescribetheexperimental setupandresultsforthesimulationstudies, followed

bythethephysical robotstudies.

5.1. Simulationexperimentalsetup

Weperformedtwosetsof experimentsin simulation. Intherstset, targetsmoveaccordingto a\ran-

dom/linear" movement, which causeseach targetto movein astraightline, with a5%chance at each

step(i.e., everyÆt =1second)to havethe directionof movementaltered randomlybetween 90 Æ

and

+90 Æ

. When theboundaryof S is reached, the targetsre ect o theboundary. Targets are randomly

assignedaxedspeedatthebeginningofeachexperimentbetweenthevaluesof0and150units/second,

whilerobotsareassignedaxedspeedof200units/second. Atthebeginningofeachexperiment,robots

andtargetsarerandomlypositionedandorientedin thecenterregionofS (exceptfortherobotsin the

Fixedapproach).

Inthesecondsetofsimulationexperiments,targetsmoveevasively,inwhichtheytrytoavoiddetection

bynearbyrobots. The calculationsforthe evasivemovementswere thesameasthe local force vectors

betweenneighboringrobots, exceptthat targets were given theadded benetof alargerrangeof view

than that ofthe robots, in order to magnify the behavioral dierences in evasiveversus random/linear

targetmotions. Thus,intheseexperiments,thetargetshadasensingrangeof1.5timesthatoftherobots.

If a target did notsee arobot within its eld of view, it moved linearly along its current direction of

movement,withboundaryre ection. Allotherexperimentalsetupswerethesameasfortheexperiments

withtherandom/lineartargetmovements.

We compared these four approaches in both sets of experiments by measuring the A metric (see

section2forthedenitionofA)duringtheexecutionofthealgorithm. Sincethealgorithmperformance

isexpectedtobeafunctionofthenumberofrobotsm,thenumberoftargetsn,therobotsensingrange,

and the relative size of the areaS; we collecteddata for awide rangeof values of these variables. To

simplifytheanalysisofoursimulationresults,wedenedtheareaS astheareawithinacircleofradius

R ,xed therangeofrobot sensingat2,600 units ofdistance, treatedrobotsand targetsaspoints,and

includednoobstacleswithin S (otherthantherobotsand targetsthemselves, andtheboundaryofS).

Fortheforcevectorfunctionsdenedin Figures2and3,weusedthefollowingparametersettings:

do

1

= 400

do

2

= 800

do

3

= 2600

predictivetracking range = 3000

dr

1

= 1250

dr

2

= 2000

Wevariedmfrom 1to 10,n from 1to 20, andR from 1,000to 50,000units. For each instantiation

ofvariablesn,m,andR ,weevaluatedtheAmetricforruns oflength2minutes. Wethenrepeatedthis

processfor250runsforeachinstantiationtoderiveanaverageAvalueforthegivenvaluesofn,m,and

R .

Sincethe optimumvalueof theA metricforagivenexperimentequalsthenumberoftargets, n,we

normalizedtheexperimentsbycomputingA=n,whichistheaveragepercentageoftargetsthatarewithin

somerobot'sviewatagiveninstantoftime. Thisallowsus tocomparetheresultsofexperimentsthat

5.2. Statistical signicance

Toverify the statistical signicanceof ourresults, we used theStudent's t distribution, comparing the

policiestwoat atime forall sixpossiblepairings. Inthese computations, weused the null hypothesis:

H

0 :

1

=

2

,andthereisessentially nodierence betweenthe two policies. UnderhypothesisH

0 :

T = X

1 X

2

 p

1

n

1 +

1

n

2

where = q

n

1 S

2

1 +n

2 S

2

2

n

1 +n

2 2

Then,onthebasisofatwo-tailedtestata0.01levelofsignicance,wewouldrejectH

0

ifT wereoutside

therange t

:995 to t

:995

, which for n

1 +n

2

2=250+250 2=498degreesoffreedom, is therange

-2.58to2.58. Forthedatagivenintheguresinthissection,wefoundthatwecouldrejectH

0

ata0.01

levelofsignicanceforallpairingofpoliciesthatshowavisibledierenceinperformanceinthesegures.

Thus,wecanconcludethatthevariationinperformanceofthepoliciesillustratedbythefollowingresults

issignicant.

5.3. Qualitativecomparison

Our results can be discussed both from a qualitative perspective of variation in behavior, and from

a quantitative perspective, by evaluating and comparing metrics of performance of each approach. In

thissubsection,wepresentanddiscussthebehavioraldierencesamong thefourapproaches. Thenext

subsectiondiscussesthequantitativecomparisonsof theapproaches.

Figure5showssnapshotsofthebeginningoftheexperimentsforeachofthefourapproachesforthree

robots and six targets for random/linear target movements; Figure 6 shows similar snapshots for ve

robotsandtwentytargets. Allofthesnapshotguresshowresultsforaworkspaceradius,R ,of10,000.

Inthesegures(andsimilaronestofollow),thesmallblackcirclesgivethepositionsoftherobots,while

thesmallgraysquaresgivethepositionsofthetargets. Thegraycirclesaroundeachrobotrepresentthe

sensingrange of that robot. Theboundaryof the areaS is thelarge black circle. Traces of therobot

and targetpathsare shown; thespacingof thedotsalongeachtraceindicates therelativespeedof the

correspondingrobotor target.

BothoftheseFigures5and6illustratethe\harder"observationproblem(buttypicaloftheproblems

we are interested in), which occurs when there are moretargets than robots. Thus, an assignmentof

onerobotpertargetwouldnotbeastraightforwardsolutiontotheproblem. IntheLocalapproach,the

robots tendto clusternear thecenter ofmassof thetargets, withsomeseparationdueto therepulsive

forcesbetweentherobots. Thisleadstoseveraltargetsbeingunderobservationbymultiplerobots,while

other targets escape observation. In the A-CMOMMT approach, the robots exhibit more distribution

due totheweightsontheforce vectorsthat causethem to belessattracted totargetsthat are already

underobservationbyanearbyrobot. Thus,moretargetsremainunder observation.

Figures7and8showsimilarresultsforevasivetargetmovements. TheFixedandRandomapproaches

domuchworsehere,becausetheydonotgetobservation\forfree"fromtargetswanderingthroughtheir

sensing ranges. Instead, the targets avoid them, and verylittle time elapses during which targets are

underobservation. TheLocalandA-CMOMMTapproachesperformsimilarlytobefore,exceptthatnow

they suer astronger penaltyfor losing atarget because, once lost, targets tend to remain out of the

sensingrangeoftherobots.

5.4. Quantitativecomparison

Wecanmoreeectivelyevaluatetheapproachesbyquantitativelycomparingtheapproachesbasedupon

(a) (b)

(c) (d)

Fig.5. Simulationresultsof3robotsand6targets,withtargetsmovingrandom/linearly,forfouralgorithms: (a)Fixed,

(b)Random,(c)Local, and(d)A-CMOMMT.Inthisandsimilarguresthatfollow,therobotsarerepresented byblack

circles,andthetargetsarerepresentedbygraysquares. Thegraycirclessurroundingeachrobotindicatetherobotsensing

range. Tracesofrobotandtargetmovementsarealsoshown.

Fig.6. Simulationresultsof5robotsand20targets,withtargetsmovingrandom/linearly.

Fig.7. Simulationresultsof3robotsand6targets,withtargetsmovingevasively.

Fig.8. Simulationresultsof4robotsand20targets,withtargetsmovingevasively.

Percentage improvement in simulation of A-CMOMMT and Local over baseline, for R > 10,000, r = 2600, and random target movements

A-CMOMMT 654% 550% 600% 570% 511%

Local n

m 1 1 4 10

5 1

2

708% 550% 525% 450% 389%

Table1. PercentageimprovementinsimulationofA-CMOMMTandLocalalgorithmsoverthebaseline(whichwedeneto

betheRandomapproach). Theseresultsareforradiusofworkarea,R,notlessthan10,000,radiusofrobotsensingrange,

r,of2600,andtargetsmovingrandom/linearly.Thevaluesaregivenintermsoftheratioofthenumberoftargets(n)to

thenumberofrobots(m).

Percentage improvement in simulation of A-CMOMMT over Local, for R > 10,000, r = 2600, and random target movements

A-CMOMMT -8% 0% 14% 27% 31%

m n 1 1 4 10

5 1 2

Table2. PercentageimprovementinsimulationofA-CMOMMToverLocal,forradiusofworkarea,R,notlessthan10,000,

radiusofrobotsensingrange,r,of2600,andtargetsmovingrandom/linearly.Thevaluesaregivenintermsoftheratioof

thenumberoftargets(n)tothenumberofrobots(m).

random/linearly. Table1givesthepercentageimprovementofA-CMOMMTandLocalovertheRandom

approach for theresultsshownin Figure 9. Table2showsthis improvementin termsof A-CMOMMT

overLocal.

The results are given as a function of the radius of the work area, R , for ve dierent ratios of

the number of targets to the number of robots (n=m). As expected, all of the approaches degrade in

performance asthe size of the areaunder observation increases. The morenaiveapproaches| Fixed

and Random | degrade veryquickly asR increases, which is expected since these approaches use no

informationabouttargetpositions. Inthelimit,theRandom approach performsbetterthanFixed,due

totheproximityoftheinitialstartinglocationsoftherobotsandtargetsintheRandomapproach. Thus,

forreal-worldapplicationsinwhichtargetsmovealongmorepredictablepathsratherthanrandomly,this

suggeststhatasignicantbenetcanbegainedbyhavingrobotslearnareasoftheenvironmentSwhere

targetsaremorelikelytobefound,and forrobotstoconcentratetheirobservationsin thoselocations.

More interesting is the comparative performance of A-CMOMMT versus Local. As Table 2shows,

A-CMOMMTperformsfrom8%worsethanLocalforn=m=1=5,to31%betterforn=m=10. Thus,for

n=m;<1=2(i.e.,lessthanhalfasmanytargetsasrobots),LocalperformsbetterthanA-CMOMMT,while

for n=m>1=2, A-CMOMMT is thesuperior performer. Sincethe \harder"versionof theCMOMMT

problem is when there are many more targets than robots, these results show that the higher-level

weightingisbenecialforsolvingtheharderproblems|upto a31%improvement.

However,whenthere areonlyafewtargets,theweightfactorsworktothedisadvantageoftheteam,

resultinginuptoan8%degradation. Thisisbecausetheweightscausetherobotteamtoruntheriskof

losingtargetsiftheirattractiontothosetargetsisin uencedbyotherrobotsbeingnearby. Thisisanot

aproblemifthereareasuÆcientnumberoftargetssuchthat mostrobotshaveothertargetobservation

choices;butitisaproblemforcasesofrelativelyfewtargets. Figure10givesanexampleofthissituation,

inwhichthereareverobotsandonetarget. Averagedover250runsofthissituation,theA-CMOMMT

approach yields anormalizedvalue ofA of 0.82, whilethe Local approach yields anormalizedvalueof

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(a) n/m = 1/5, Targets move randomly A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(b) n/m = 1/2, Targets move randomly A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(c) n/m = 1, Targets move randomly

A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(d) n/m = 4, Targets move randomly A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(e) n/m = 10, Targets move randomly A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

Fig.9. Simulationresultsof4distributedapproachestocooperativeobservation,forrandom/lineartargetmovements,for

Percentage improvement in simulation of A-CMOMMT and Local over baseline, for R > 10,000, r = 2600, and evasive target movements

A-CMOMMT 770% 618% 700% 525% 429%

Local n

m 1 1 4 10

5 1

2

870% 655% 656% 438% 343%

Table 3. Percentage improvement of A-CMOMMTand Local algorithmsoverthe baseline(which wedene to be the

Randomapproach). Theseresultsareforradiusofworkarea,R,notlessthan10,000,radiusofrobotsensingrange,r,of

2600,andtargetsmovingevasively. Thevaluesaregivenintermsoftheratioofthenumberoftargets(n)tothenumber

ofrobots(m).

Percentage improvement in simulation of A-CMOMMT over Local, for R > 10,000, r = 2600, and evasive target movements

A-CMOMMT -11% 0% 7% 20% 25%

m n 1 1 4 10

5 1 2

Table4. PercentageimprovementofA-CMOMMTovertheLocalapproach,forradiusofworkarea,R,notlessthan10,000,

radiusofrobotsensingrange,r,of2600,andtargetsmovingevasively. Thevaluesaregivenintermsofthe ratioofthe

numberoftargets(n)tothenumberofrobots(m).

(a). However,in several occurrences, theteam loses the target, asshown in Figure 10 (b). TheLocal

approach, ontheother hand,nearlyalwaysallowstheteam tomaintainthetargetinview,asshownin

Figure10(c). Thus,aeldedversionoftheA-CMOMMTapproachshouldtakeintoaccounttherelative

numbersof targets to revise theweighting in uence dynamically to betterdeal with small numbersof

targets.

Figure11showstheresultsofoursecondsetofexperimentswherethetargetsmoveevasively. Relative

to the random/lineartarget movementresults from Figure 9, theperformance ofall of theapproaches

degrades,asexpected,withtheFixedandRandomapproachesdegradingquicker. Otherwise,therelative

performances remain thesame. Table3 gives the percentage improvementof A-CMOMMT and Local

overtheRandomapproachfortheresultsshowninFigure11. Table4givesthepercentageimprovement

ofA-CMOMMT overLocalforthese results,showingthat A-CMOMMT performsfrom 11%worsethan

Localforn=m=1=5,to25%betterforn=m=10.

5.5. Robotimplementation

WeimplementedtheA-CMOMMTalgorithmonateamoffourNomad200robotsandperformedexten-

siveexperimentationon the physical robots. The purposesof these experimentswere: (1) to illustrate

thefeasibilityofourapproachforphysicalrobotteams, (2)tocomparetheresultswiththeresultsfrom

simulation, and (3) to determine the impact of random scattered obstacleson the eectiveness of the

proposed approach. Weconducted severalhundredexperimentalruns onthephysicalrobotsto compile

statistically signicant data on the performance with dierent numbers of trackers and targets, both

withandwithoutscatteredobstacles. Thefollowingsubsections describethedesignandresultsofthese

(a) (b)

(c)

Fig.10.SimulationresultsshowinginstancewhenA-CMOMMTperformsinferiortoLocal,forverobotsandonetarget,

withtargetsmovingevasively. Therstsnapshotin(a)showsthemostcommonA-CMOMMTbehavior,whichisdesirable,

inwhichonerobotendsuptrackingtheonetarget. Thesecondsnapshotin(b)showstheoccasional,undesirablebehavior

ofA-CMOMMT,inwhichalloftherobotslosethetarget,duetothehigh-levelweightscausingrobotstoresisttracking

thetarget. Thethirdsnapshotin(c)showsthebehavioroftheLocalapproach,inwhichnearlyallrobotsfollowthetarget,

preventingthetargetfromescapingobservation.

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(a) n/m = 1/5, Targets move evasively CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(b) n/m = 1/2, Targets move evasively A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(c) n/m = 1, Targets move evasively A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(d) n/m = 4, Targets move evasively A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

0 0.2 0.4 0.6 0.8 1

0 10,000 20,000 30,000 40,000 50,000

(e) n/m = 10, Targets move evasively A-CMOMMT Local Random Fixed

Normalized Average A

Radius of work area (R)

Fig.11. Simulationresultsof4distributedapproachestocooperativeobservation,forevasivetargetmovements,forvarious

Experimental design TheNomad 200robots arewheeled vehicles withtactile, infrared, ultrasonic,

2D laser range, and indoor global positioning systems. The indoor global positioning system we use

is acustom-designed 2D ConacPositioning System by MTIResearch, Inc. This system usestwolaser

beacons, with asensor onboard each robot that allowsthe robot to use triangulationto determine its

ownx;ycoordinatelocationtowithinabout10centimeters 3

. Inaddition,therobotsareequippedwith

radio ethernetforinter-robotcommunication. Inthe currentphaseofthis research,whichconcentrates

onthe cooperativecontrol issuesofdistributed tracking,weutilizeanindoorglobalpositioning system

as a substitute for vision- or range-sensor-based tracking. Under this implementation, each target to

betrackedis equipped withan indoor global position sensor, andbroadcasts its current(x;y)position

viaradio tothe robotswithin communicationrange. Each robotteam memberis alsoequippedwith a

positioningsensor, and can usethe targets'broadcastinformation to determinetherelativelocationof

nearbytargets.

Figure 12 shows an example of the physical robot experiments. In these experiments, wetypically

designated certainrobotsto be targets, and other robots asobservers(or searchers). Weprogrammed

therobotsactingastargetstomoveinoneoftwoways: movementsbasedonhumanjoystickcommands,

orsimplewanderingthroughtheareaofinterest. Forthedatacollectedandpresentedinthissection,all

thetargets movedusing autonomouswandering. Noevasivemotionswereimplemented. Sincewewere

usingourrobotsasbothobserversandtargets,westudiedvariouscombinationsofobserversandtargets,

upto atotaloffour combinedvehicles. Table5showsthesets ofexperimentsweconducted. For each

ofthe algorithmsshown,ten experimentalruns of 10minuteseach were conducted. Theparametersof

theseexperimentswereasfollows,foraworkspaceof12.8m6.1m(42ft20ft)androbotdiameterof

0.61m(2ft):

do

1

= 1m(40in)

do

2

= 2m(80in)

do

3

= 6:6m(260in)

predictivetracking range = 10:16m(400in)

dr

1

= 1m(40in)

dr

2

= 15:2m(60in)

RecognizingthattheA-CMOMMTapproachdoesnotexplicitlydealwithobstaclesintheenvironment,

weconductedphysicalrobot experimentsbothwithandwithoutscattered,simple,randomobstacles,to

determinetheimpactofsimpleobstaclesonthecontrolapproaches. Weaddedasimpleobstacleavoidance

behaviorto the control approachesto enablerobots to move aroundencountered obstacles. Obstacles

averagedafootprintof :6m 2

, which wasgrown bythe obstacleavoidingbehaviorto afootprintof size

1:3m 2

. Wetested random obstaclecoverage percentages of 7%, 13%, and 20%. These obstacles were

placedrandomlythroughout theenvironment, anddid notfrequently resultin \boxcanyons", orother

diÆcultgeometricalcongurations. Inaddingtheseobstacles,weassumedonlythattherobotnavigation

capabilitieswererestrictedby theobstacles. Wedidnotconsider restrictionsinthe visibilityoftargets

causedbyobstaclesblockingtherobots'views. Underthisscenario,robotswereabletosee overand/or

aroundtheobstacles.

Physical Robot Experimental Results To illustrate the typical behaviorof the physical robots,

we rstpresentanexperiment using humanjoysticking. InFigure 12, the targetsare indicatedby the

triangular ags. Therstframein Figure12showsthearrangementoftheobserversandtargetsat the

verybeginning of experiment. The second frame (top right)showshow the twoobserversmove away

from each other once the experiment is begun, due to the repulsive forces between the observers. In

thethird frame,ahumanjoysticksoneofthetargetsawayfrom theothertargetand theobservers. As

the target ismoved, thetwoobserversalso move in the samedirection, due to theattractiveforces of

1 2 3

1 Fixed, Local Fixed, Local Fixed, Local 2 Fixed, Local, A-CMOMMT Fixed, Local, A-CMOMMT *

3 Fixed, Local, A-CMOMMT

# targets

# observers

Table5. Tabulationofphysicalrobotexperimentsconducted. Allofthecasesindicatedwereaveragedovertenrunsoften

minuteseachwithoutobstacles. Inaddition,the casemarked'*'isthesituationforwhichwealsoranexperimentswith

threedierentpercentagesofscatteredrandomobstacles.

Fig.12. Resultsofrobotteamperformingtaskusingsummationofforcevectors. Therobotswiththetriangular agsare

actingastargets,whiletherobotswithoutthe agsareperformingthedistributedobservation.

fourthframe,thentheobserversarenolongerin uencedbythemovedtarget,andagaindrawnearerto

thestationarytarget,duetoitsattractiveforces. Notethat throughouttheexample,theobserverskeep

awayfrom eachother, duetotherepulsiveforces.

Figures 13 and 14 show typical scenarios of the experimentations both without and with scattered

random obstacles. The results of the experimental runs of the scenarios listed in Table 5 (without

obstacles)areshowninFigures15and16. InFigure15,theresultsshowtherelativeperformancesofthe

Fixed,Local,andA-CMOMMT approaches,measuredin termsofthenormalizedaveragevalueof theA

metric,asafunctionof then=mratio(i.e.,numbersoftargets tonumbersofrobots). Tobeconsistent

withoursimulationresults,wevariedtheeectivesensingrangeoftherobots(throughsoftware)within

a xed experimental area, so as to provide results as a function of the amount of experimental area

the robots canobserve. Figure 15 showsresults for three sensing ranges { 2 meters, 4 meters, and 6

meters. As these results show, the A-CMOMMT approach enabled robots to track all targets for all

theseexperimental scenarios. TheLocal approachalsoworkedwellexceptfortheshort sensingrange(2

meters) with moretargetsthan robots. The Fixed approach waseective onlywhen thesensing range

Fig.13. RobotteamexecutingA-CMOMMTapproachinareawithnoobstacles.

Fig.14.RobotteamexecutingA-CMOMMTapproachinareawithobstaclesintheworkarea.

Figure 16 presentsthese experimental results in termsof the average distance betweena targetand

the closest robot, rather than in terms of the normalizedaverage A metric value. These resultsshow

thesamerelativeperformance ofthethreeapproaches,but showtheactualdistancedataaswellasthe

standarddeviationsin therobotperformanceforeach experimentalscenario. Thesedata areconsistent

withthesimulationresults,showingthenear equivalenceoftheA-CMOMMT andLocalapproachesfor

small ratios of n=m, but the improved performance of A-CMOMMT approach for larger n=m ratios.

Whilewecannot provethatthe simulationresultswould holdfor largernumbersof physical robotand

targetexperiments, itisexpectedthat resultssimilarto thesimulationndingswouldcontinue tohold

trueforlargernumbersofphysical robotsandtargets.

Figure 17 presentsthe resultsof ourexperiments that includedrandomly scattered simpleobstacles

that coveredupto 20%of theexperimentalarea. As theseresultsshow,becausetheobstaclesimpeded

boththerobots andthe targets,and because robotswereenabled tosee overand/oraroundobstacles,

Fig.15. Resultsofphysicalrobot experimentswithnoobstacles. Thethreegraphsshowtheresultsofthe Fixed, Local,

andA-CMOMMTapproachesfordierentsensingrangeswithinaxed-sizeexperimentalarea,asafunctionofn=m.

Fig.16. Resultsofphysicalrobotexperimentswithnoobstacles. Eachdatapointrepresentstheaverageof10experimental

runsoftenminuteseachforagivenratiooftargetstorobots. Standarddeviationsforeach10-runaverageareshown. The

twodatapointsforeachofFixedandLocalatn=m=1representthetwocasesof1)onerobotandonetarget,and2)two

robotsandtwotargets.

Fig.17.Resultsofphysicalrobotexperimentswithscatteredrandomobstacles. Eachdatapointrepresentstheaverageof

10experimentalrunsoftenminuteseachfortworobotsandtwotargets(thusn=minthiscaseequals1).

6. Multi-Robot Learning ofCMOMMT Solutions

WebelievethattheCMOMTapplicationisanexcellentdomainformulti-robotlearningbecauseitisan

inherentlycooperativetask. in these typesof tasks, theutility of theaction ofonerobot isdependent

uponthecurrentactionsoftheotherteammembers. Inherentlycooperativetaskscannotbedecomposed

into independent subtasks to be solved by adistributed robot team. Instead, the success of the team

throughoutitsexecutionismeasuredbythecombinedactionsoftherobotteam,ratherthantheindividual

robotactions. Thistypeoftaskis aparticularchallengeinmulti-robotlearning,duetothediÆcultyof

assigningcredit fortheindividualactionsoftherobotteammembers.

Researchershaverecognizedthatanapproachwithmorepotentialforthedevelopmentofcooperative

control mechanismsisautonomous learning. Hence,muchcurrentworkis ongoingin theeld ofmulti-

agentlearning(e.g.,[38]). Thetypesoflearningapplicationsthataretypicallystudiedformulti-agentand

multi-robotsystemsvaryconsiderablyintheircharacteristics. Someofthelearningapplicationdomains

includeair eetcontrol[34],predator/prey[4],[17],[14],boxpushing[22],foraging[24],andmulti-robot

soccer[35],[23],[16].

Of the previous application domains that have beenstudied in thecontext of multi-robot learning,

only the multi-robot soccer domain addresses inherently cooperative tasks with more than tworobots

while also addressing the real-world complexities of embodied robotics, such as noisy and inaccurate

sensorsand eectorsin adynamic environmentthat ispoorlymodeled. OneaspectofCMOMMT that

isdierentfrommulti-robotsocceristhatitraisestheissueofscalabilitymuchmorethaninmulti-robot

soccer. The issue of dealing with largerand largernumbers of robots and targets must be taken into

accountinthedesignofthecooperativecontrolapproach 4

. Thus,wefeelthatthesecharacteristicsmake

theCMOMMTdomainworthyofstudy formulti-robotcooperation,learning,andadaptation.

We are currently developingmulti-robot learningapproachesto this problem. In [33], wereport on

anapproach thataddressestheCMOMMTproblem withouttheassumptionofanapriori model. This

approachcombinesreinforcementlearning,lazylearning,andapessimisticalgorithmabletocomputefor

eachteammemberalowerboundontheutilityofexecutinganactioninagivensituation. Lazylearning

[1]is used to enablerobot team membersto build amemory ofsituation action pairsthrough random

explorationof theCMOMMT problem. A reinforcementfunction givesthe utilityof agiven situation,

providing positive feedback if new targets are acquired and negative feedback if targets are lost. The

pessimisticalgorithmforeachrobot thenusestheutilityvaluesto selecttheactionthat maximizesthe

lower bound on utility. This selection is performed by nding the lower bounds of the utility value

associated withthe various potentialactions that may beconducted in the currentsituation, andthen

choosingtheactionwiththegreatestutility. TheresultingPessimisticLazyQ-Learningalgorithmisable

toperformconsiderablybetterthantheRandomactionpolicy,althoughitisstillsignicantlyinferiorto

thehuman-engineeredA-CMOMMTalgorithmdescribedin thispaper. Thisreducedperformancelikely

dueto thefact that thisalgorithmonly takesinto accountnearbytargets, ignoring nearbyrobots. Our

ongoingresearchis aimed at improving thislearningapproachto include informationon nearbyrobots

inthereinforcementfunction.

Wehavealso explored asecond approach to learning[11] that is alsobased onreinforcementlearn-

ing. This approachapplies Q-Learningalong withtheVQQL[12] techniqueand theGeneralizedLloyd

Algorithm[21],[20] to addressthegeneralizationissuein reinforcementlearning. This approachallows

a dramatic reductionin the numberof states that would beneeded if a uniform representation of the

environmentoftherobotswereused,thusallowingabetteruseoftheexperienceobtainedfrom theen-

vironment. Thus, usefulinformationcanbestored inthestatespacerepresentationwithoutexcessively

increasing the number of statesneeded. This approach wasshown to be highly successful in reaching

performanceresultsalmostas goodastheA-CMOMMT approachdescribedin thispaper.

Ourongoingresearch isaimed atexploring other approachesto multi-robot learningin thisdomain.

Ideally, we would like to developa learning approach that can discover solutionsthat are better than

researchistodevelopgeneraltechniquesformulti-robot learningthatapply notonlytotheCMOMMT

domain,but alsotoawidevarietyofchallengingapplication domains.

7. Conclusions

Manyreal-worldapplicationsinsecurity,surveillance,andreconnaissancetasksrequiremultipletargetsto

bemonitoredusingmobilesensors. Wehaveexaminedaversionofthisproblem,calledCooperativeMulti-

Robot Observation of Multiple Moving Targets (CMOMMT), and have presented distributed heuristic

approach, called A-CMOMMT, to this problem. This approach is based uponlocal force vectorsthat

causerobotstobeattractedtonearbytargets,andtoberepelledfromnearbyrobots. Theforcevectors

are weighted to cause robots to be less attracted to targets that are also under observation by other

nearbyrobots.

WehavecomparedtheA-CMOMMTapproachtothreeotherpossiblepolicies: Fixed,inwhichrobots

remain at uniformly distributed locations in the area, Random, in which robots move according to a

random/linearmotion, andLocal, which usesnon-weightedlocalforce vectors. Weperformedextensive

studiesusingbothsimulatedandphysicalrobotsandpresentedresultsthattookintoaccountthenumber

ofrobots,thenumberof targets,thesensingrangeoftherobots, andthesize oftheobservationareaS.

Wealso examined twotypes oftarget movements|random/linear motionand evasivemotion. Using

thephysicalrobots,wealsoexplored theimpactofrandomlyscatteredsimpleconvexobstacles.

OurresultsshowthattheA-CMOMMTapproachissignicantlysuperiortotheLocalapproachforthe

diÆcult(buttypicaloftheproblemsweareinterestedin)situationsinwhichtherearemoretargetsthan

robots. Thus,A-CMOMMTissuccessfulinachievingitsgoalofmaximizingtheobservationoftargetsin

themorechallenginginstantiationsoftheCMOMMTproblem. Onthecontrary,forexperimentsinwhich

therearemanymorerobotsthantargets,theweightsonthelocalforcevectorsusedintheA-CMOMMT

approach causerobots to occasionallylose targets,and thusperformworsethanin theLocal approach.

Therefore,practicaluseoftheA-CMOMMTapproachwillrequireadditionalinformationontherelative

numbersoftargetsversusrobots todynamicallychangetheweightsoftheforcevectors.

Wealsointroducedourideasinusing CMOMMTasamulti-robot learningdomainand ourresearch

in developing learning approaches to solve this problem. We are continuing this research, to explore

thepossibilityofautomaticallygenerating solutionstochallengingmulti-robotcontrolproblemssuchas

CMOMMT.

Acknowledgements

TheauthorexpressesherthankstovisitingstudentsBradEmmons,PaulEremenko,andPamelaMurray,

whohelped intheimplementationand/ordatacollectionofthephysicalrobotexperimentations. Many

thanksalsotoClaudeTouzetforhisdiscussionsandsuggestionsduringthedevelopmentofthisresearch.

Theauthorisalsoveryappreciativeofthethreeanonymousreviewerswhoprovidedmanyhelpfulsugges-

tionsforimprovinganearlierdraftofthisarticle. ThisresearchissponsoredbytheEngineeringResearch

ProgramoftheOÆceofBasicEnergySciences,U.S.DepartmentofEnergy. Accordingly,theU.S.Gov-

ernment retains anonexclusive, royalty-freelicense to publish or reproducethe published form of this

contribution, orallowothers todo so,for U. S.Governmentpurposes. Oak RidgeNationalLaboratory

ismanagedbyUT-Battelle,LLCfortheU.S.Dept. ofEnergyunder contractDE-AC05-00OR22725.

Notes

1. Usingthe1.6MbpsProximradioethernetsystemwehaveinourlaboratory,andassumingmessagesoflength10bytes

perrobotpertargetaretransmittedevery2seconds,wendthatnm(forntargetsandmrobots)mustbelessthan

4