Learning to shepherd an unknown flock, while maintaining apparent flock autonomy
© Bram Staps 2011 ICA-3496880
Abstract
Shepherding in flocking models has been done since Craig Reynolds invented flocking using local interaction rules [4]. Since then most shepherding flocking models came with two agents: a herd-able agent and a shepherd. These agent were made for each other in the sense that they exert nice and realistic behavior when used together.
In this thesis a method will be devised that is able to shepherd a wide range of flocks. Part of this method is that shepherds will have to learn how to shepherd a new flock. An extra restriction to this problem will be that the result should look natural.
Table of content
1 Introduction...3
2 Problem description...4
3 Formal framework...7
3.1 Generic Model...7
3.2 Core definitions...11
3.3 Flocking agents & models...14
3.4 Useful external measurable flock properties...26
4 Achieving adaptive shepherding...29
4.1 Shepherding the unknown flock: the basic process...29
4.2 Fundamental problems...32
4.3 Agent and solution structure for usage in evolution...33
4.4 The shadow agent program (IO processing)...35
4.5 Techniques: initial attempt...37
4.6 Techniques: the improvements...43
5 Results...51
5.1 A single EA run...51
5.2 Qualitative EA results...53
5.3 Flocks that could make the EA fail finding an adequate solution...59
6 Future tuning / improvements...62
6.1 Improvements of the EA algorithm...62
6.2 Improvements of the Model...65
6.3 Incorporating more functionality...66
7 Conclusions...68
8 Tools & software...69
1 Introduction
In 1987 Craig Reynolds invented Boids [4]. A Boid is what we now could call a bird like flocking agent. Craig Reynolds was the first to create a flocking model which used local interactions between the flocking agents instead of a centrally controlling algorithm.
Reynolds his model captured the inner workings of real world flocking behaviors more naturally than the central algorithms did: In a flock of birds every bird does makes out for him / herself which way to fly, there is no higher entity telling each bird where to go. This natural representation leads to new possibilities: If some real world flocks work by local interactions, and those flocks can be manipulated (herded) to move in a certain direction, then so can these simulated flocks1.
Shepherding models (example: [4], [7], [9]) are not an exotic sight anymore. These shepherding models usually model the shepherding behavior by modeling two types of agents: The agent that will get steered by the shepherds and the shepherds themselves.
Afterward these two types of agents are put in the same virtual world and their interactions will make for a realistic display of flocking / shepherding behavior.
In this thesis a global shepherding agent will be developed which (after adaptation) should become capable of steering a previously unknown flock. This goes hand in hand with creating a model / framework in which the target flock can be represented2. For this thesis this model / framework will be restricted to agents moving in a two dimensional Euclidean space, but in the future this can easily be adapted to more dimensions and possibly even non Euclidean spaces. If flocks can be persuaded and manipulated in non Euclidean space one could for example model the stock exchanges as flocks to see how to best manipulate the stocks and get most profit out of it. But for now we stick to two dimensional Euclidean space, more details on the goals of this thesis will be outlined in chapter 2.
1 This goes for “modern” flocking models where local interaction is used.
2 Without a standard representation of an agent it is very hard to do anything with said agent. There should always be a standard way the agents can “interface” with each other and the environment.
2 Problem description
The aim of this thesis is to invent a method which trough the use of adaptation can direct most flocks in a certain direction without it being apparent that the flock is being shepherded / manipulated.
The problem with most flocking models is that most shepherds now are “hand-crafted” to fit one particular (type of) flock. An example: in [1] a shepherd is developed to control a flock of ducks, in [7] the shepherd can guide “training ducks” in a very natural way, in [9]
again “training ducks” only then with multiple shepherds. All these solutions work great for the specific agents they have been developed for but are not broadly applicable3.
In this thesis a flock is defined as a number of individuals which stay together and navigate as a coherent group through the use of local interaction rules. Centrally controlled flocks are not covered because they were already outdated in 1987 (see: [4]), these outdated models are not shepherd-able by agents because they model the flock behavior in a very unnatural centralized way.
For a lot of flocks, you do not have access to the internal workings of the agents that need to be controlled / shepherded. Because of this absence of information most agents are black box systems, this means the simulating environment and the other agents within that environment have no knowledge of their internals (there is no knowledge available of what processing takes place inside the system, the only thing that can be seen are the inputs and outputs of that system). This handicap makes that the method should be able to cope with agents which are black box systems.
To be able to steer an agent which is a black box system (from now on all normal flocking agents are presumed to be black box systems) the method to steer them needs at least some level of adaptability. From now on 'the method', refers to the method of steering previously unknown flocks in a natural fashion.
Since all agent’s internals are off limits to the method, the method's only way of steering the agents is by manipulating their inputs. Manipulating inputs of an agent can roughly be done in two ways: direct and indirect. With direct manipulation the inputs of each individual agent are written / set directly by the method that is manipulating the behavior of the agents. Indirect manipulation happens by changing the environment of said agents in the hopes they will react to it in the way you have foreseen. The most widely used indirect method of manipulating agents (which are black box systems) is by using shepherds, and thus falls under the category of indirect manipulation.
The choice between direct and indirect manipulation is made in favor of indirect manipulation, since for direct manipulation there should be direct access to the sensory system of the agents involved. This direct access is a very demanding requirement which severely limits applicability of the method. The indirect method does not have such strict requirements4 and therefore is preferred over the direct approach.
Making the method able to process black box agents and influencing them through
3 That very good sheep shepherding model will for example not work on the bird flock.
4 You only need to be able to change the environment for indirect manipulation, while you need full direct solo access to the sensory inputs for direct manipulation.
external manipulation is the most relaxed setting for this kind of problem. With relaxed it is meant that it puts least constraints on the agents that should be flocking and thus is preferred over other methods. In [10] it is shown that you can exert a lot of control through the use of shepherds using external influencing / manipulation. In the referenced paper it is researched how a flock of cockroaches can be manipulated into choosing a lighter shelter instead of a darker one (which is unnatural behavior for cockroaches).
The method of control needs to be adaptable because of the black box nature of the agents involved, this leads to the conclusion that we need shepherds that can adapt themselves (whether it's online, offline5 or both) to the flock they have to steer.
While online adaptation is a very nice concept to use (because you keep improving while running), it is hard to implement and more time consuming then just using offline adaptation. The focus of this thesis is being able to steer previously unsteerable flocks.
Taking that demand into account online adaptation does not have many advantages over offline adaptation, hence only offline adaptation is used.
The method developed in this thesis will be an algorithm which can adapt / evolve shepherd agents so they can steer a flock, without making it look like the flock is getting influenced. From now on these shepherding agent will be called shadow agents because they need to influence the normal agents in such a way that they can be steered but without giving away their presence, hence the shadow agent name.
Both demands made about the resulting behavior of the shepherded agents (steering and naturalness) make for that finding good tactics for the shadow agents is inherently a multiple-objective optimization problem, because the maximization of one property will lead to the diminishing of of the other:
– If the believability of a flock gets maximized, it will look more and more like no- one is influencing it. This reaches its apex when no control is exerted on the flock at all. Aka: Nothing looks more like P then P itself, and P in this case is is the flock wherein no control is exerted.
– If the steering of a flock is maximized, all flock members will rush to the desired point at maximal speed, completely ignoring each other (best case for steering).
Agents that rush to one point, not following their 'natural' motion doesn’t look natural. Even worse, if all the agents ignore each other its technically not even a flock anymore.
Being a multiple-objective optimization problem is not the only hardship in the undertaking of evolving the shadow agents. The fact is that it is impossible to support all properties a flock can have because there can always be thought of another property not supported by the shadow agents or the default environment (explained in the next chapter). The third problem is that simulating a flock, which obviously is needed if you want to evolve a shepherd to shepherd said flock, is inherently a O(n2) complex problem (n = number of agents), see [4]. While in theory a quadratic complexity does not have to be a problem in the practical case of simulating flocks it is, flocks need to be able to be simulated faster then real-time because the EA uses multiple simulations to determine the
5 Offline adaptation means that something gets adapted a piori to using it, whereas online adaptation means that the adaptation takes place when the adaptee is used, in for example a simulation. Note that online and offline adaptation are not mutually exclusive.
utilities of a given flock.
To support the claim of [4] a test was done: the measured complexity of simulating a mosquito agent (defined later) was O(n1.97) which confirms that the execution time of simulating a flock is quadratic in respect to the number of agents6.
This all makes for a interesting and complex problem which is going to be solved by an algorithm which evolves a group of shadow agents in such a way that they can steer a previous unknown flock of normal agents in a natural way.
Some things that you might expect in a flock simulation / evolution experiment which have not been taken into account are borders and obstacles, The only objects in the environment are the agents themselves, and they move on a infinite plane so their mobility is in no way restricted.
The reason for this absence is the complexity of the problem as it already is without obstacles. Perhaps a second research can incorporate these factors and how agents react on it and how it can be improved, but this is assumed to be future work for now.
6 Please note this only is the complexity of simulating the flock, actually evolving the flocks is even more complex because of the complexity of the EA.
3 Formal framework
Int his chapter a formal framework will be defined that functions as the backbone of yet to be explained and future techniques that can be used to achieve the goal of this thesis, eg:
the agents, the shadow agents, the genetic algorithm, visualization and statistic retrieval (to be able to demonstrate the workings), they all cannot function without the framework.
3.1 Generic Model
This will provide a model which facilitates agents flocking on a infinitely large plane7. The idea is that every agent (and thus flock) that can be expressed in this very non-restrictive model, can be used to train a group of shadow agents to shepherd that flock.
The model should later on meet these demands:
– Can simulate the agents it represents.
– Can visualize the simulation
– Statistical data can be gathered from the simulation.8
– Trough the use of evolving shadow agents, steering capabilities can be be added to control the flock.
The last benefit makes for a crucial point of this thesis: All you need to do to generate a method of control for an existing flock “while it maintains its apparent autonomy” is fitting it in the standard model, the algorithm described later in this thesis takes care of the rest.
3.1.1 Generic environment model
The model (environment) consists of three dimensions:
– A non-discrete Euclidean two dimensional infinitely long and wide plane.
– Discrete time (time slices).
The environment simulates by repeating the same cycle indefinitely (or until a globally set limit in reached). Every repetition of this cycle is called a iteration.
Cycle of the environment
1. Give all the agent in the environment their input . 2. Wait for the output of all agents in the environment.
3. If the output of every agent is received repeat cycle.
Model structure S = <A1, …, An, E>9
7 Defined later in this thesis.
8 There exist tools for benchmarking, measurements and visualization for these agents. The programs are a by-product of the research done in this thesis but valuable nonetheless. These functionalities are needed to later on “prove” that the total solution for adaptive shepherding works.
9 This is and will be referenced as the standard model, from now on.
Where:
Ai : an agent (agents are defined below).
E: The environment.
Please note that currently the only functionality that the environment has is to synchronize the actions of all agents, as has been explained above.
Agents can take a new position and angle in the 2D plane at every time step (tn → tn+1). An agent can move everywhere from anywhere in one go, ergo there are no restrictions on speed, turning cycle or whatsoever. The reason for this is to impose no extra restrictions, because an extra restriction would make for less flocks being able to run on the standard model (and thus decreasing the applicability of the method).
The absence of a global restrictions that prevents agents from having infinite speed and turning cycles does not mean there is no global limit on speed and turning cycle. The global limit just gets “defined“ on a per-simulation basis by the agents that inhabit the world at that time. For every quantifiable property (you can represent this as a number in R) there exists a global maximum and minimum: This simply is the minimal and maximal value of said property of all agents combined. For example: There are 3 agents in a simulation which move with the following speeds: 2 m/s, 1 m/s and 5 m/s. The global minimal speed is then 1 m/s, and the global maximal speed 5 m/s.
3.1.2 Generic agent model
Agents are the core entities the environment works on, and the workhorse of the model.
The cycle of the environment is dependent upon every agent completing its personal cycle.
After all agents completed their cycle the environment can start a new iteration.
Agent structure
A = <x, y ,angle, apparentType, realType, aer>
Where:
x R: current x-coordinate of the agent.∈ y R: current y-coordinate of the agent.∈
angle [0, 2π]: current angle of the agent relative to reference (north).∈ apparentType at: the type that an agent says he is.∈
realType rt: the type that an agent really is.∈
aer aer: Imposed IO restrictions (sensory data, movement).∈ Where:
at = Set of all possible normal types of agents.
rt = {'shadow agent'} ∪ at
aer = Set of all possible agent enforced IO restrictions, these substitute for the lack of restrictions in the environment.
Structure of all items in aer: <inputFilter, moveRestrictions>
Where:
inputFilter if: A function with the following signature (S∈ ** → X) where S** is the input from the environment and X is an arbitrary output which is passed to the agent as sensory input (the S** stands for one of the two sets Sna or Ssa which will be explained shortly). So this function determines what the agent can sense and usually is used as a filter to not make the agent see everything. For example: Say an agent has eyes on the front of his head, then this function discards visual data on the back side of the agent because it cannot see that region of space.
moveRestrictions mr: A function with the following signature (RxRxR →∈ RxRxR) that limits the movement of the agent. This function limits the x, y coordinates as well as the rotation. For example: Say a “human” agent wants to move north in a straight line with 40 km/h then this function could limit that agent to only moving with speed 20 km/h because that is the maximal speed of that agent.
Cycle of an agent
1. Get input from the environment (Environmental input S→ na10).
2. Filter own input (Sna →X (agent specified input)) .
3. Use the input to calculate the next desired move (X new <x,y,angle>).→
4. Filter your desired move so it honors the restrictions you want to abide (<x,y,angle> <x,y,angle>).→
5. Give the output to the environment (execute your move).
Mutable properties An agent tuple:
A = <x, y, angle, apparentType, realType, aer>
Not all parameters of an agent can change, realType and aer are immutable (are inherent to the agent), while x , y and angle can be changed by the agent at every state transition.
The parameter apparentType is special:
– For a normal agent: realType=t apparentType=t , so in principle apparentType = realType, and since realType is immutable the apparentType won't change either.
– For a shadow agent: realType=t apparentType≠t , t=shadow agent , apparentType cannot be a shadow agent (see definition of at). A shadow agent can change apparentType to anything in at at its own discretion. This factor makes it able for the shadow agent to “play: the role of an other agent and thus deceiving / tricking normal agents.
Visible properties
Recall the agent tuple: <x, y, angle, apparentType, realType, aer>.
Normal agents can see the x,y,angle and apparentType of every other agent in the vicinity.
Shadow agents can see those same properties plus the realType of any agent in the vicinity.
Example
10 For a shadow agent these states should be substituted for Ssa.
Suppose the next situation.
Sheep1 and sheep2 are normal agents, shadow is a shadow agent.
S = <sheep1, sheep2, shadow, E>
where:
sheep1 = <0.0, 1.0, π, sheep, sheep, τ>
sheep2 = <0.0, -1.0, π, sheep, sheep, τ>
shadow = <-0.5, 0.0, ½π, shepherd dog, shadow, υ>
sheep1 and sheep 2 would get the following input from the environment:
{<0.0, 1.0, π, sheep>,<0.0, -1.0, π, sheep>,<-0.5, 0.0, π, shepherd dog>}
shadow would get this input from the environment:
{<0.0, 1.0, π, sheep, sheep>, <0.0, -1.0, π, sheep, sheep>, <-0.5, 0.0, ½π, shepherd dog, shadow>}
What should be noticeable is that the shadow agent receives an extra property of every agent in the vicinity. This extra property is the reason why a shadow agent cannot fool another shadow agent, but can fool a normal agent.
3.1.3 Shadow Agent
Shadow agents are the agents that will be adapted to steering a flock.. Shadow agent are basically normal agents with one privilege: Shadow agent can change their own apparentType and have access to the realType of every agent in the environment. These privileges lead to the following properties:
– Shadow agents can distinguish between normal and shadow type agents, where normal agents can't (by looking at the realType a shadow agent can see whether an agent is a normal agent or a shadow agent).
– Shadow agents can “fool” normal agent by posing as another type of agent (normal agents can only see the apparentType of a shadow agent and apparentType is a mutable property for the shadow agent).
– Shadow agents cannot “fool” each other (because shadow agents can see each others real type).
Because of these properties a (group of) shadow agent(s) can guide normal agents by exploiting their natural behavior. Imagine a flock of walking ducks. The shadow agents can now attract the ducks by posing as a duck and standing close enough to attract it.
Alternatively they can chase the ducks in a certain direction by posing as a shepherd dog.
When a group of shadow agents is used to steer the flock of ducks both techniques (attracting, and chasing) can even be combined to guide the ducks in a certain direction.
The fact that ducks are attracted to ducks and walk away from shepherd dogs is something the shadow agents have to “learn”, how they do this is discussed later in this thesis.
3.2 Core definitions
When working with spatial agents forming flocks a few baseline properties / functions are needed to express things about those agents and flocks. The following functions and operators (which in essence are also functions) are used to define those properties. These (mostly) geometric functions will be used extensively in sub-chapter 3.3.
Functions
Agentvector: (agent) vector→
Output of this function is the location of agent supplied to the function (represented as a vector).
Flockvector: (agent) vector→
Same as the agent vector but then for the location of the whole flock (an implementation of this function will be described later in this sub-chapter).
Vector: (A, length) vector→
Returns the corresponding location (as a vector) for a normal vector scaled by length and rotated over one (or multiple) angle(s).
Where A is the tuple of angles.
Vectorxy: (xy →) vector
Gives the vector representation of a tuple of loose coordinates (xy).
Angle: (vector) A→
This function retrieves the angles needed to create the vector given as input.
Angle(Vector ( A , l))=A Minmax: (R,R,R →) R:
Minmax(a,b,c) is defined as: Minmaxa , b , c=Min Maxa ,b , c
Operators and notation
In this thesis (wherever possible 11) scalar operators which are overloaded for vector use are annotated with a vector arrow. This annotation prevents the user form guessing the actual semantics of the operator from the context.
Example:
+ adds 2 vectors together.
- subtracts 2 vectors.
∑
v ∈Vectors
v adds all vectors in Vectors into one big vector.
The following operators are defined for usage on things that can represent a location
11 eg: A division operator written like a dividing line cannot be annotated for obvious reasons.
(vectors, agents, etc):
The ⋮ operator calculates the distance between 2 agents.
a⋮b=∥AgentVector a - AgentVector b∥
∥x∥ is a euclidean norm. It returns the length of a vector.
∥x∥=
d ∈ AxialCoordinates∑
d2
x is a vector normalized to length 1.
x= x
∥x∥
Flock contour
The flock contour, or flock surface as it is called for 2d flocks, is a semantically clear concept for most people but surprisingly hard to define in a strict definition. To strictly define what the flock contour is the semantic definition of flock contour in [7] is formalized and generalized it to a multiple-dimensional definition:
Flock area / contour is defined as the union of all the surfaces in the flock, that could be defined by the flock members locations12.
A =
∪
areas(Pd+1)– P is the set containing all agents, the superscript operator works as follows:
(P2 = PxP, P3 = PxPxP, ... )
– d is the number of dimensions from the universe of that kind of agent.
–
U
is the union operator that unifies all (hyper)planes into one (hyper)plane13.12 It takes only d+1 points to define a limited region in d dimensions, e.g.: it takes 3 points to define a limited surface in 2 dimensions.
13 Plane means the space described by their respective amount of points. A 1d plane is called a line, a 2d plane a surface, a 3d plane a volume, a plane > 3d is a hyperplane.
Figure 1: example of calculation of the swarm area / flock contour in 2d, blue is the flock volume and the black are the agents and borders of the areas.
Flock center
The flocks center is the “location of the flock”. The location of the flock is needed when to quantify anything about the movement of the flock as a whole. In this thesis movement is very important because without a definition for it, one cannot measure how well a flock moves in a certain direction.
There are two general ways of defining the location (center) of a flock:
– Average location of each agent.
– Center of the flock surface (flock contour)
While both methods make sense the former was chosen because in that definition all agent have an equal “say” in the location of the flock. In the latter option this is not the case because in the dense regions agent per saldo have less to say about the flock center. Note that if all agents in a flock are uniformly distributed both methods yield the same result.
The flock center is defined as the average location of all flock members:
flockCenter( flock )=(Flockx , FLocky)=(Fx, Fy)=(
∑
f ∈Flock
fx flockSize,
f ∈Flock
∑
fy flockSize) F* is a flock coordinate, f* a individual's coordinate.
3.3 Flocking agents & models
Here we introduce the agents and the flocks that are to be steered by adapted14 shadow agents. The properties of the agents, and the flocks they form will be discussed. The reason why these agents (and the flocks they form) are representative for a significant part of all possible flocking agents, will also be explained.
This sub-chapter with agent and flock models is conveniently placed here so all readers have a choice. You can read them now to broaden your understanding and make sense of the results later on in the thesis, or you could skip them for now and skip back when needed.
3.3.1 Flocking agents
Here the flocking agents used in the this thesis will be defined. A flocking agent is a agent Which (in combination with some other agents) is capable participating in the process where a group of agents act as a whole (flocking) trough the use of only local interaction rules.
Five different flocking agents, and the reason for choosing them will be described next.
How these agents form flocks will be described in the next paragraph.
Flocking agent properties
Before diving right into the agents and some of their properties, the used properties will be elaborated such that one knows what is meant by those properties. The discussed properties have mixed origins, meaning they do not originate form the same field of science, eg: The distance model is a concept from the “flocking world”, whether an agent is stateless or not is a notion that came form the agent community. The concept determinism has its roots in algorithmic systems, etc.
Distance model:
The distance model is the way an agent processes the locations of other agents surrounding him. Using a metric model the agent processes all agent locations within a certain distance from him. When using a topological model the agent only uses the neighbors directly surrounding him.
14 How shadow agents adapt will be described later in this thesis.
Figure 2: Explanation of the metric vs topological distance model using a school of fish (flock).
Unprovoked movement:
Unprovoked movement is the movement a single agent makes when it (and its flock) is not being influenced by shadows agents.
Determinism:
This property states whether the agent behaves stochastic or not. If the agent does exactly the same thing every time it is put in exactly the same starting situation it is deterministic, if not it is a stochastic agent.
Memory / state-full:
An agent has memory when it stores information for use in future iteration(s). When a agent does not store this information it acts solely on the input the current iteration, in that case the agent can be said to be acting reflexive / on instinct.
3.3.1.1 Duck Agent Origin
In [1] they used a duck agent that is inspired by the ducks used in training sessions for border collies and other herding dogs. These ducks are used in the training when it is desired to train in more confined quarters. In these confined spaces ducks are preferred because they consume less space.
The duck model has been chosen as one of the models because it is (after Reynold's bird model) the most commonly used flocking model used in research. Secondly agent is the only kind of agent in this thesis that will react on two kinds of agents differently: It gets attracted by friendly agents and repelled by unfriendly / shepherding agents.
The duck agent described here is based on the model of the duck agent in [1]. The ducks in this paper calculate their movement by four vectors:
– Repulsion from other ducks.
– Attraction to other ducks.
– Repulsion from the shepherd.
– Repulsion from the walls.
Movement
For using the model in this thesis the repulsion from walls is removed (because there aren't any) and the repulsion from a single shepherd is expanded to facilitate repulsion from multiple shepherds.
After the adaptations to the model the following (quadratically declining) forces are at work:
– Repulsion from other ducks (scaled by K2) – Attraction to other ducks (scaled by K1) – Repulsion from the shepherds (scaled by K3)
These forces translate to this behavior for a duck:
1. Keep some distance between themselves and their neighbors.
2. Approach each other.
3. Steer away from the unfriendly / shepherding agent(s).
This model is implemented the following way:
At each iteration a duck calculates his next move (represented as a vector) in the following fashion:
d=
∑
d ' ∈ Ducks ,d ≠d '
K1
d⋮d ' L2 K2
d⋮d ' 2
∑
s∈ Shepherds
K3
d⋮s2
Formula 1: The movement vector of a duck agent. The value L guarantees that ducks always repulse each other at short distances.
The movement of a duck is also limited by a given value M:
if length d M then length d=M .
This is not in the original paper, it has been introduced to prevent ducks to achieve extraordinary speeds due to a quirk in their original formula15 (see: [1]).
Model settings
K1 = 1.5, K2 = 1.0, K3 = 1.5, L = 0.5, M = 0.5.
Properties
Distance model metric
15 A situation in when this could happen is when ducks get initialized to close to each other, making the repulsive force component astronomically high; catapulting ducks away out of the region where the majority of the ducks are.
Unprovoked movement standing still
Determinism deterministic
Memory / state-full no memory
Since a duck calculates vectors associated with all other ducks, this is a metric model. A flock of ducks will walk to each other on initialization, but when they come together in a clump they all stop moving unless they are disturbed. Since no stochastic variables are used in the ducks they behave deterministic. Ducks react instinctively on the environment and have no memory.
See [18] for a video of a flock of duck agents.
3.3.1.2 Mosquito Agent Origin
The mosquito is an agent I discovered in one of my previous works. The flock behavior when visualized does visually resemble that of the “clouds” of mosquito’s that dance around ponds in the dusk.
The mosquito agent are used in this thesis because at the time of introduction they ware the only agents which were always moving, but in a flock on average never moved anywhere. (The spider agents introduced later also have this quality)
Movement
A mosquito agent is a topological agent which travels at a constant speed and has a limited rotation speed. The agent utilizes its four closest agents from the four quadrants (front, back, left and right) around him to navigate. The agent does this by calculating the vector that would center him between the four agents16. (If no agent is present within a quadrant the agent “sees” an agent at distance C17 in that quadrant).
Simply put the mosquito is controlled as a car with a fixed speed. The only thing it can control is whether it turns to the left or to the right (or keep center), the mosquito always steers to the largest open space it sees (Remember C, if no agent is present, it sees one there so it is not an open space). By all this steering you get nice dynamics where mosquito’s constantly orbit each other in search of the most space around them.
This dynamic is implemented in the following fashion: At each iteration a mosquito calculates his next move (represented as a vector) using the following formula:
m=Vector (Minmax( A , Angle(Vectorxy (d⃗ frontdback, dleftdright)) , A) , S ) Formula 2: The vector the mosquito agent moves by
16 The agent only knows the zone and distance of a agent towards him so he assumes 90 degree angles between the 4 agents he sees.
17 Distance C recursively works to the inside of a flock and is a huge factor in determining the density of the flock.
Figure 3: The four fields of view (quadrants) for a mosquito agent (designated by the red lines). The shortest distances to the other agents in each quadrant are designated by the blue lines. The length of these blue lines are the input value of the corresponding quadrant of the agent.
Model settings A = 0.1 rad, S = 0.1.
Properties
Distance model topological
Unprovoked movement moving (circling)
Determinism deterministic
Memory / state-full no memory
The mosquito agent uses the four closest agents around him, using quadrants, this makes the agent topological in nature. The behavior of a mosquito in a flock can be best described as cycling around other mosquito's. A mosquito uses nothing of stochastic nature and thus is deterministic. The mosquito acts instinctively and has no memory.
See [19] and [20] for a video of a flock of mosquito agents.
3.3.1.3 Bird Agent Origin
A bird agent is the agent that comes closest to the original agent described by Reynolds in [4]. That agent was designed to mimic the behavior of a flock of birds.
The bird agent used in this thesis is based on the standard flocking agent; having parts in common with [2],[5], [6] and based on work in [4] and [3]. With standard it is meant that the model is used extensively as the de facto standard, this is because Reynolds was the first person to use individual agent with local interaction rules to simulate a flock. Before that time centralized algorithms were used.
The reason this agent is used in this thesis is its extensive use in other literature. The agent
or its spin-offs get used in lots of scientific papers, and therefore it is desirable to actually be able to test this agent in the framework set up in this thesis.
Movement
The movement of a bird agent rests on 3 principles:
1. Try to fly in roughly the same directions as the other birds are flying.
2. Try to get closer to the surrounding birds.
3. Try not to collide with the surrounding birds.
This poured into a formula, which calculates the step the bird moves by at every iteration, looks like this:
b=Vector MinmaxA , Angle K1Vattract K2Vrepulse K3Valign
∥ K1Vattract K2Vrepulse K3Valign∥ , A , S Formula 3: The vector the bird agent moves by.
Where:
Vattract={ Agentvectorb '
∥Agentvector b '∥2| b '∈Birds ,∥Agentvectorb ' ∥ Repulseborder } Vrepulse={ Agentvectorb'
∥Agentvector b ' ∥2| b '∈Birds ,∥Agentvector b '∥≤Repulseborder } Valign={Agentvector b ' | b ' ∈Birds }
S designates the constant speed of the bird, K1 to K3 are used to scale the weight of the vectors. Vx stands for the summation of all the vectors in that set (named x in this case):
Vx=
∑
v ∈Vx
v .
Model settings
K1 = 0.5, K2 = 2.0, K3 = 2.0, Repulseborder = 2.0, S = 0.05, A = 0.1.
Properties
Distance model metric
Unprovoked movement moving (directional)
Determinism deterministic
Memory / state-full no memory
A bird is a metric agent because it uses the vectors of all agents within a certain distance.
The movement of a bird when the flock has settled is a straight line in a certain direction.
The absence of stochastic techniques and memory also make this a deterministic reflexive agent.
See [17] for a video of a flock of bird agents.
3.3.1.4 Magnetic bird agent Origin
The magnetic bird is an agent based on the normal bird agent but with a twist. The magnetic bird has an internal compass that guides it in a certain direction. This is inspired by the fact that some type birds can migrate with accurate compass like precision18 (for example some storks, which migrate between the Netherlands and Africa every year).
The main aspect for choosing this agent is that one can tweak the preference for following the magnetic field vs following the other birds. By varying this preference the breaking point of the shepherding capabilities of the shadow agents can be tested. It is this potential for testing use which is desirable (and is used later on in this thesis, see sub-chapter 5.3).
Movement
The magnetic bird agent is a bird agent with a 'sloppy' built in compass. The 'sloppiness' of this compass gets corrected by the presence of other birds in the flock which also have a sloppy compass, but together trough interactions they will find the correct direction. This is called the many-wrongs principle (see: [15]).
Next to the three principles the normal bird agent adheres, the magnetic bird agent also tries to follow his set “magnetic” direction, so a magnetic bird gets steered by four different principles instead of the original three used by a normal bird agent:
The movement of a magnetic bird based on it's four principles19:
1. Try to fly in roughly the same directions as the other birds are flying.
2. Try to get closer to the surrounding birds.
3. Try not to collide with the surrounding birds.
4. Try to fly in the general direction of the magnetic field.
The bird agent model is adapted in the following way, to incorporate this magnetic bias:
⃗b=Vector (Minmax(A , Angle( ( K1Vattract⃗ )(K2Vrepulse⃗ )+(K3V⃗align)+(K4Vmagnetic⃗ )
∥( K1Vattract⃗ )(K2Vrepulse⃗ )+(K3V⃗align)+(K4Vmagnetic⃗ )∥), A), S ) Formula 4: The vector the magnetic bird agent moves by.
Where:
Vmagnetic=Vector , 1
Where:
θ[0, 2π] designates the the compass direction of the magnetic bird.
The 'sloppiness' of a compass in simulated by letting the compass be off by a certain random amount. So = desired deviation Where desired designates the desired direction, and deviation is an random angle by how much the compass is off.
18 Compasses work on magnetic fields hence the name magnetic bird, for the agent.
19 The bold one, is the one that is new compared to the normal bird agent.
Please note that the other “forces” (attraction, repulsion, aligning) are calculated in the same way as in the bird agent.
Model settings
K4 = 0.2 unless otherwise noted, the rest of the used values are the same as those of the normal bird agent.
Properties
Distance model metric
Unprovoked movement moving (directional to the south)
Determinism deterministic
Memory / state-full no memory
The magnetic bird has the same properties as the bird. The only thing that is different is that its movement is not only based on the other agents surrounding it but also on the direction of the internal compass.
3.3.1.5 Spider Agent Origin
The spider agent is an agent which was born out of the need for a state-full stochastic agent. It is not based on any real life spider but instead on spiders that can be seen in some video games running franticly around trying to guard their nests when they think it is under attack.
Movement
The spider agent represents a spider which 'guards' his nest in 3 steps:
1. The spider runs trough the center of the nest, and then continues to run on that path for some more time (stochasticly determined).
2. The spider then stops, plans its new line to walk on (trough the center of the flock) 3. Repeat.
Since the spider is a state-full agent with memory and a mere formula of its movement won't suffice. This is the spider agent described in pseudo code:
If state == Planning:
flock = getVisibleSpiders(RANGE) fc = getFlockCenter(self, flock) target = Vector(self, fc)
steps = length(target) / STEPSIZE steps = steps + random() * EXTRA_STEPS wait = WAIT_TIME * random()
state = Waiting
If state == Waiting:
if wait > 0:
wait()
wait = wait - 1 else:
state = Running If state == Running:
if steps > 0:
step()
steps = steps - 1 else:
state = Planning
Where:
getFlockCenter(a , b)=
(
∑
⃗s ∈b , s≠a
location(s))⃗- location(a) count(spiders)1
getVisibleSpiders={s | s ∈Spiders ,∥location s -locationself ∥≤ RANGE } Model settings
The quantified values used in simulation of this agent are:
RANGE = 20.0, STEPSIZE = 0.25, WAITTIME = 25, EXTRA_STEPS = 50.
Properties
Distance model center point oriented (metric)
Unprovoked movement moving
Determinism stochastic
Memory / state-full memory + states
The distance model for the spider agent is different then the others, but it could be implemented using a metric system. The spiders receive one input only, which is the center of the nest (which is the center of the flock of all spiders). Spiders in a flock run back and forward over a single line crossing the center of the nest. The center of the nest is not static, so the original center of the nest is remembered (memory) to make sure the spider walks on a straight line. How far the spider walks over this straight line is determined stochasticly at the start of the walk.
See [21] for a video of a flock of spider agents.
3.3.2 Benchmark instantiations of flocking agents.
The flocking models that are used throughout this thesis will be stated here. All flocking models used in this thesis are composed of one type of flocking agent, and bear the same name as the used agents.
In nature you see very little mixed flocks, the sheep don't flock with the birds, fish or even ducks. This and the reason that time is better spent improving other parts of this thesis resulted in the fact only homogenous flocks were researched in this thesis. Please do note that after including a few small changes little changes the model and the accompanied software do have support for heterogeneous models.
Properties of flocking models
Before going in to the composition of flocks, some important properties of the flocks will
be explained. These properties will be used to compare flocks, and demonstrate that a lot of categories of flocks are represented and used within this thesis. This information in turn will be used to show that the flocks in this thesis differ enough form each other.
Unprovoked group movement:
Unprovoked group movement is the movement that a group as a whole makes while not being influenced by shadow agents.
Determinism:
Same definition as for an agent. If a flock behaves exactly the same each simulation when the starting situations are identical the flock is deterministic, if not the flock is stochastic.
Note that a flock is deterministic if all its agents are. If one agent in a flock is stochastic, the whole flock will also behave stochastic.
3.3.2.1 Mosquito model
The mosquito model is a model consisting of 2...n mosquito agents. Unless otherwise indicated 25 mosquito’s are used. See [19] and [20] for a video.
Properties
Unprovoked group movement roughly in the same spot (random walk like)
Determinism deterministic (chaotic)
While a single mosquito moves quite a lot, the location of a flock of mosquito’s stays more or less the same. A single mosquito is deterministic so this flock is too.
3.3.2.2 Duck model
The duck model is a model consisting of 2...20 duck agents. The reason the maximum amount of ducks allowed is that above 22-23 the flock starts to oscillate (see [8] for a video). Unless otherwise indicated 15 ducks are used. See [18] for a video.
Properties
Unprovoked group movement standing still
Determinism deterministic (ordered)
A single unprovoked duck does not move, this makes for the flock as a whole to stay still too, after converging into a ball pattern. A duck is deterministic so the flock is too.
3.3.2.3 Bird model
The bird model is a model consisting of 2...n bird agents. Unless otherwise indicated 15 birds are used. See [17] for a video.
Properties
Unprovoked group movement moving in a direction
Determinism deterministic (ordered)
The flock moment of the birds is more or less the same as the movement of a single agent, moving in a single direction. Birds are deterministic, and thus so is their flock.
3.3.2.4 Magnetic bird model
The Magnetic bird model is a model consisting of 2...n bird agents. Unless otherwise indicated 15 birds are used.
The default desired is the same for all birds in the flock, while every single deviation is a angle between [-0.5,0.5]rad unless otherwise noted.
Properties
Unprovoked group movement moving in a direction (always south)
Determinism deterministic (chaotic)
The properties of a magnetic bird are essentially the same as that of a normal bird flock, except for the bias toward the magnetic direction.
3.3.2.5 Spider model
The spider model is a model consisting of 3...n spider agents. Two spiders in a flock don't work well., this is due to the fact that those agents then will move on a single line, and thereafter never can leave this line again. Unless otherwise indicated 15 spiders are used.
See [21] for a video.
Properties
Unprovoked group movement roughly in the same spot (random walk like)
Determinism stochastic
The moment of the spider flock s a whole is best describes as a random walk, with small probability to deviate from the center. The flock itself is not deterministic because the spiders are stochastic.
3.3.3 Why these flocks have been chosen
Where most papers who have found a general solution for problems in a certain field suffice with one “toy” problem, this thesis will present five different ones. Those five problems will be the adequate steering of all models mentioned above20 (section 3.3.2). The reason of choosing five different models is to make an even bigger statement about the capabilities of the algorithm + model.
All specific models mentioned above have been selected to complement each other and represent what is used more commonly in the literature:
The mosquito's are used because of their more fragile model. If tampered with enough the stop swarming around each other anymore and start to show other behavior.
The ducks have in the beginning been chosen because they are the other flock model used in papers (next to the bird model). A nice plus for the ducks is the quirk where they start to oscillate if the density of ducks becomes too high, this makes for an extra challenge for the EA because oscillating ducks behave very unnatural.
The birds have been chosen because it is the first flocking model that did not use a central algorithm but instead used local interaction rules for flocking. This model actually sparked
20 An exception being the magnetic birds with high affinity for the magnetic direction.
the whole area of flocking and it is used in a plethora of flocking papers2 1.
The magnetic birds are an adaptation of the normal bird flock. This flock was designed to actually break the genetic algorithm to see where it breaking point lies.
The last type of flock, the spiders were designed to be able to test the algorithm on a flock with different properties. In this flock individual spiders have memory, have a state, and plan ahead actions. This all made for a flock which is mediocre robust but hard to move.
3.3.4 Why many flocks are small
Recent findings have shown that determining the natural utility is polynomial complex (and nor linear as was assumed). A big flock was planned because it was assumed the complexity lied with the flock and not the utility method. Because a linear time flock will still lead to a polynomial complexity this big flock was canceled.
The other flocks have their current size because they themselves are polynomial in complexity as does the utility function, so they cannot be made much bigger before it becomes unpractical to evolve them.
This late conclusion leads to room for improvement, if the naturalness utility itself can be achieved in a different but equally useable way with linear complexity (or maybe even constant time) it would allow the EA to evolve adequate solutions for bigger and even more complex flocks.
21 In most papers using this model it is used as a basic flock and then altered a little where after it is used in the research.
3.4 Useful external measurable flock properties
This sub-chapter describes three different properties that will be used further on in this thesis to quantify basic behavior of a flock, which in turn will be used by the EA.
The flock properties stated in the previous sub-chapter are theoretical and cannot be measured by merely observing22 and thus are no good for discriminating between different kinds of (black box) agents. The only properties that can be directly observed of a black box flock are the location, rotation of each agent at a given time. With different combinations of these properties one can construct and infinite number of properties, for example: speed, acceleration, average heading of a flock, distribution of speed in a flock, etc. [5] describes some complex properties of a flock that could be useful for discriminating between flocks. These properties vary and can be used to obtain information about the kinds of agents flocking.
In [5] the properties mainly focus on the collective state of the flock and the deviations of individuals to the average of the flock, because it is these properties that are useful for discriminating one kind flock form another using only sensory information (location, rotation, time).
3.4.1 A selection of external measurable flock properties
Here three properties that can be measured directly form a flock will be defined. These three properties have been empirically selected to determine how a group behaves, and can be used to compare two groups and see how much they behave the same (in flocking terms). The properties are calculated on each iteration of the global model. To calculate a final score the average of these properties (average of all iterations) are used:
∑
t =0 xPt t
Formula 5: Averaging out the properties over time
Pt is the value of a property at time t Average minimal distance
The average minimal distance is a measure of how close the agents are together, it is the summed distance to each agent's closest neighbor, scaled to the number of agents.
∑
a∈ Agents
#Agents
∥Agentvector a - Agentvector Nearestneigbor a ∥
# Agents
This method only gives some information (not relevant tot the quality of the flock) about the flock, it can be used to compare flocks but not to give a indication of the quality of the flock (It is as useless asking if red cars are better then green ones).
Density Deviation
The density deviation is a measure of how uniformly the agents in the flock are spread out in the contour of the flock, it is the cumulative density deviation from the average density
22 Without knowledge of the internals of that agent, those properties cannot be stated.
scaled to the number of locations (where density is sampled).
p ∈locations
∑
#locations
% dx Density p , averageDensity
# locations
% dxa , u=∣au∣ u
Notice that Density is a meta function which can be replaced by any function which has the following form: p∈Locations → density∈ℝ. In practice you cannot sample all locations because it would take infinitely long, therefore the Locations set will contain only the locations of all agents. The practical non-meta “density” function used in this research will be the distance to the nearest neighbor, also called minimal distance.
SVL
The svl method indicates how well all agents are facing in the same direction. The svl method stands for “scaled vector length”. This method adds all vectors (length 1) of corresponding angles together and returns the length of the resulting vector, scaled by the number of agents.
This method returns a number between 0 and 1, where 1 being every agent moves in the same direction and 0 where all agents cancel each other out.
∥
a ∈ Agents
∑
#Agents
Anglevectora ∥
# Agents
Anglevector is a function which transforms an agent's angle to a vector by assuming vector length 1, and has to following form: a∈Agents → v ∈Vector.
3.4.2 Example properties (Distance deviation, SVL)
This paragraph will demonstrate how the values of some of the properties describes above will change according to the layout of a group of agents (from ordered to stochastic). The reason for only showing distance deviation and SVL is that the average nearest neighbor property depends on different variations not demonstrated in the following example.
Figure 4: from left to right: perfect flock, semi random flock(0-50%), totally random flock.
Distance deviation
SVL
Perfect flock (0% deviation) 0.0 1.0
Semi random initialized flock (0-25% deviation) ±0.20 ±0.25 Semi random initialized flock (0-50% deviation) ±0.40 ±0.21 Semi random initialized flock (0-75% deviation) ±0.52 ±0.12 Totally random initialized flock (0-100% deviation) ±0.55 ±0.08
Table 1: example of different values of flock and a incoherent group. The values are based on bird-like flocking. In mosquito like flocking the values of SVL and polarity should be the other way around.
As can be seen from table 1 and figure 4, different kinds of characteristics lead to different property values. These values can used to discriminate between different kinds of flocks, and will be used to verify how close a steered flock's behavior is to the original's one. The value that can be calculated form these differences is the naturalness utility which is very important to the EA, and will be explained in detail later on.
4 Achieving adaptive shepherding
In chapter 3 the various models that together model the world and the agents that can reside within it were introduced. In this chapter the the whole process from normal shadow agent to a capable shadow agent shepherd group and the techniques used to achieve this will be explained in detail. After the outline of the whole process used to create shepherding shadow agents and the fundamental problems of of doing so, the representation (genome) of a shadow agent for the differential evolutionary algorithm will be presented, where after the explanation of the genome the techniques used in adapting the it will be discussed (first the basic solution, later the improvements made). These techniques form the hearth of the whole adaptability and quality of the shadow agents, and therefor will be discussed in great detail.
The reason for using an differential evolutionary algorithm is as follows: Because of the lack of knowledge of the target solution domain (the flock that is to be steered is unknown a priori), the fact that its a real valued problem, the shape of the solution space is erratic and unknown make it a prime candidate to use differential evolution on23.
4.1 Shepherding the unknown flock: the basic process
This sub-chapter outlines the steps that need to be taken to come up with a good group of shepherding shadow agents. Every used technique will be explained step by step.
Terms introduced here not immediately explained will be explained in sub-chapter 4.3 to 4.6, where the final details and technical definitions will be given.
Throughout the whole of chapter 4 a kind of chronological algorithmic technical dependency tree view will be used to show where the technology fits in the compete process. This kind of tree shows all things to be done for the end result as a list. The special thing about this list is that some of the list items can be expanded to indicate what needs to be done to to satisfy the parent node.
For example here is a tree that shows the simplified process of building a boat. In this tree it is indicated that we are talking about the technique of chopping down a tree.
Figure 5: Example fully expanded tree of building a boat, with the chopping down a tree technique highlighted.
The expanded tree of the the automated process looks as follows:
23 Most of the properties explained here are indications that differential evolution could lead to satisfactory results (see [22]).
Figure 6: Fully expanded tree of the automated part of the process that creates shepherding shadow agents (left).
The colors in this tree are used for your convenience, they indicate identical techniques (same technique used in multiple places).
Determine desired naturalness level
Before this whole process can start one value needs to be set: The naturalness setting. The naturalness setting in technical terms is the toppling point from where on the naturalness utility of a shadow agent is deemed sufficient and does not contribute to the total utility of a said agent anymore24. It has been empirically determined that for most purposes a value of 10 yields satisfactory results, meaning that most flocks with a value of 10 and up look natural
Determine heuristic use
Optional heuristics that are deemed useful for the problem that is to be solved can be
24 See section 4.6.2 for details on the utility / ranking formula.
selected25.
Get statistic data of flock
Statistical data of the flock that is to be steered is collected in this step. This is done so the natural (meaning normal) behavior of the flock can be quantified. This quantified baseline behavior then can then later be used to see how much an influenced flock behaves like the original (and thus how natural it behaves) .
The three properties used to represent this quantification are:
– Average minimal distance25 – Density deviation25
– Scaled vector length26
The average values of these three properties over multiple simulation steps in from simulations will be used.
A average of N runs will be made to determine the value of these properties. If a property is non-sensible for a flock it will be omitted. For example: In a flock in which the agents don't have a direction it is not sensible to calculate the SVL (nor the Distance deviation) , in that case only Average minimal distance will be used.
Evolve a group of shadow agents
After everything is set the evolutionary algorithm can start. This algorithm is a differential EA which keeps improving a small population of agents until it is terminated after a set number of generations. The technical details, settings and used meta-heuristics are all stated in sub- chapters 4.5 and 4.6. The representation of the problem in terms of genomes is discussed in sub-chapter 4.3.
The EA will assume a symmetrical flock: A symmetrical flock is a flock that will just a happily move in the opposite direction if it gets mirrored27. Because flocks are assumed to be symmetrical the EA only has to learn to persuade the flock to move in one given direction. After this direction is learned the flock can be steered in every direction by just rotating the shadow agent group around the center of the flock.
Use the shadow agents, to steer the flock
After the evolution is done, a group of shadow agents come out. This group of shadow agents now (in most cases) has the capability to steer the flock.
Since most flock models are symmetrical in their local interactions, you can by rotating the group of the shadow agents around the center of the flock, also rotate the heading of the flock that same way.
25 Currently only one heuristic is available, see section 4.6.4 for more information.
26 These properties have already been defined at sub-chapter 3.4.
27 Non symmetrical flocks also can work, but have a far greater chance not to. Example flock: 3.3.2.4.
Figure 7: Shadow agents making a flock (blue fuzzy sphere) moving in two different directions. This demonstrates how symmetrical flocks can be steered when only one direction is learned.
4.2 Fundamental problems
While adapting a few shepherds in such a way that they get a group moving may look straight forward, it is not. Methods for finding a good genetic representation of the shadow-agents and making it work with the differential evolution algorithm have three fundamental problems which make using adaptable techniques hard. There is no 'best solution' for these problems in the classical sense, only solutions that help but still leave the 'real' problem up to the designer. For example: multiple objective optimization could benefit form methods like SPEA (see [13]), but it can never replace your idea of what combination of facets is most beneficial. Another example: you can figure out a Pareto front, but what item in that front is best? There are no set in stone answers to those questions.
Multiple objective optimization
Making an agent move in a certain direction and doing this in a natural way (don’t make a change in behavior too noticeable) is a multiple objective one, because both objectives are in conflict with each other. When you maximize the speed of an agent, it will go straight for its target and this will lead to less natural behavior. On the opposite side: letting an agent behave as natural as can be means you are not influencing the agent at all, and thus it will most probably not move in the direction you want it to.
In this case a good RxR → R function will be be able to rank the solutions. This function provides a single numeric rank, for each combination of utility’s it gets as input.
Techniques like Pareto dominance, SPEA, etc will help,but are to general to be used as the final ranking formula.
Multiple heterogeneous shepherd agents
While flocks are mostly homogenous, a decent group of shadow agents should not be. In a
