CiteSeerX — The Structure of the Optic Flow Field

Mads Nielsen¹ and Ole Fogh Olsen²

1 3D-Lab, School of Dentistry,Nrre Alle 20, DK-2200 Copenhagen N, Denmark

2 Department of Computer Science, University of Copenhagen, Universitetsparken 1, DK-2100, Copenhagen E, Denmark

Abstract. The optic ow eld is dened as preserving the intensity along ow-lines. Due to singularities in the image at xed time, poles are created in the optic ow eld. In this paper we describe the generic types of ow singularities and their generic interaction over time. In a general analytic ow eld, normally the topology is characterised by the points where the ow vanish again subdivided into repellers, attractors, whirls, and combinations hereof. We point out the resemblance, but also the important dierences in the structure of a general analytic ow eld, and the structure of the optic ow eld expressed through its normal ow. Finally, we show examples of detection of these singularities and events detected from non-linear combinations of linear lter outputs.

Keywords:

optic ow, scale-space, singularities, catastrophe theory, equivalence under deformation, transversality, ow structure, ow topology, turbulence, at- tention.

1 Introduction

Most work on optic ow has been devoted to its denition [7,4] and to regu- larization schemes for its robust computation [7,13,2,14,17]. In this paper, we follow the Horn and Schunck denition of the optic ow eld [7]. We do not regularize the solution, but only wish to classify it. The motivation is four fold:

we seek a classication, in mathematical terms, of the ow eld and its temporal changes. We want to emphasize that the events we describe or detect in images must be generic events. We will develop mechanisms for detecting these events, and nally we wish to indicate that this purely academical examination of the optic ow eld may subserve the development of algorithms for many dierent visual task. In this paper we give simple examples using the ow structure for guiding an attention mechanism and for computing the degree of turbulence in a ow eld. The inspiration is mainly from the analysis of autonomous dynamical systems [1] to which we will describe the analogy.

An object moving with respect to a camera induces a motion eld on the image plane. This motion eld is the projection of the motion of physical points xed to the object, and will only under very restricted lightning and re ectance circumstances directly relate to the optic ow eld [8]. We analyse the singu- larities of the data induced optic ow eld while the singularities of the motion

eld have earlier been analysed for recovery of object motion parameters [10, 16]. Since we do not in this work relate the optic ow eld to the motion eld, we can not make similar observations.

The structure of a general analytic ow eld is normally accessed through the singularities of the eld, i.e. points where the ow vanishes[1]. The rst order structure of the ow eld round these points can classify the points as attractors, repellers, or whirls. The optic ow eld is a special ow eld since computed as a simple intensity preservation. Generically the optic ow eld is not everywhere analytical. Furthermore, the tangential component of the ow is not determined by the constraint equation. These dierences cause new ow structures to be created generically and ill-dene the classical ow structures in an optic ow eld. This paper analyses these dierences, show examples, and applications.

A temporarily changing image may be obtained from imaging a dynamic scene. A physical conservation law then denes a spatial vector eld temporarily connecting conserved properties. In computer vision the Horn and Schunck (HS) equation [7] expresses the preservation of intensity over time. The derived spatial vector eld is the optic ow eld. The HS equation only solves locally for one component of the vector eld, giving rise to the so-called aperture problem. A unique representation of the optic ow eld is given by the normal ow: the ow perpendicular to the local isophote.

The normal ow is well dened in all image points with non-vanishing spatial gradient, elsewhere the ow is undened. However, in a neighbourhood round these singular points the ow eld exhibits some typical behaviour. The ow magnitude typically increases towards innity, the direction will be inwards, outwards, or combinations hereof. In Section 4 we analyse the ow around these singularities, categorise it, and see how these poles changes over time. The pur- pose of this exercise is to gain insight in the structure of the optic ow eld.

In general analytic ow elds, vanishing ow points can describe the ow eld structure. In optic ow elds only the normal ow is directly accessible and this will generically vanish at hyper-surfaces of codimension 1, i.e. at curves in 2D images. This means, that the standard classication of the ow eld structure can not directly be applied to the normal ow elds. We can, however, dene whirls in the normal ow eld as second order temporal events and apply the detection of these to the quantization of turbulence.

A proper denition of structure change in the ow eld needs a denition of structure. We do this through the mathematical concept of equivalence of ow elds under deformations. In Section 3 we review this method and its application to structural classication of analytical ow patterns. In Section 4 we dene and derive the generic structure of the optic ow eld.

Normally, the optic ow has been dened and computed directly based on pixel values, so as if they represent the true value of the intensity eld. Recently [4] the optic ow denition and computation have been formulated in a scale- space framework taking the nite extent of pixels and lter-outputs into account.

In Subsection 4.3 we comment on some aspects of the change of structure of the optic ow eld when scale changes.

In Section 5 we describe how changes of ow eld structure can be detected from outputs of linear lters. These events may be useful to detect violations of continuation models: structure emerging or disappearing, and thus guide an attention mechanism. We give examples of computations of the ow structure, and apply this to simple examples from computer vision and turbulent ow.

First, however, we look into the necessary notation and denitions.

2 Optic ow: notation and denitions

In this section we establish notations of what images, optic ow, normal ow, and the spatio-temporal iso-surface are. Furthermore we link the geometric properties of the spatio-temporal iso-surface to the ow and normal ow-eld. We assume that I is suciently dierentiable.

Denition 1 (Image sequence).

An image sequence I(x;t) : IR^DIR^7!IR is a mapping from D spatial dimensions (x = (x¹;x²;:::;x^D)) and a temporal dimension (t) into scalar values, normally denoted intensities.

Denition 2 (Spatio-temporal optic ow eld).

The spatio temporal optic ow eld v : IR^{D+1 7!} IR^D+1 is any vector eld preserving image sequence intensity along ow lines.

The preservation of intensity along ow-lines of v is expressed through the full (or Lie) derivative along the ow:

L

vI =v^x1I^x1+v^x2I^x2+:::+v^xDI^xD+v^tI^t= 0

where upper index denotes component of vector and lower index denotes partial dierentiation. In the following we will often use notation from 2D (x;y) to simplify expressions: ^LvI =v^xI^x+v^yI^y+v^tI^t= 0.

2.1 Temporal gauge

The optic ow equation yields one equation in D+ 1 unknowns. In general, the length of the vector is unimportant as only the actual connection of spatio- temporal points carries information. Often the length of the vectoreld is nor- malised to unit temporal component, i.e.v^t= 1. In this normalised ow eld we denote the spatial componentsu. This reveals the well known Horn and Schunck equation

u^xI^x+u^yI^y+I^t= 0

We call (u^x;u^y) the spatial optic ow eld or simply the optic ow eld. This ow eld answers the typical question asked by the computer vision programmer:

in next frame of my image sequence, where did points move?

Denition 3 (Spatial optic ow eld).

The spatial optic ow eldu: IR^D7!

IR^D is any vector eld preserving image sequence intensity along ow lines of the spatio-temporal vector eldv=

u 1



.

The normalisation of the temporal component is, however, only possible (- nite) whenv^t6= 0. In cases where the temporal component vanishes, the above formulation yields singularities (poles) in the ow eld. In Section 4, we analyse the ow eld round these poles, show that they exists generically, and that they exhibit certain generic behaviours/interactions. In order to do this, even though the spatial ow eld is our main concern, we must stay in the spatio-temporal formulation of the ow eld. In this way, we can derive properties of the spatial ow eld from simple geometric considerations.

2.2 Normal ow

The temporal gauge (or another normalisation) results in one equation inDun- knowns. This shows the intrinsic degree of freedom in the ow, normally denoted the aperture problem. Using the temporal gauge, the spatial optic ow constraint equation reads u^rI =^?I^t where u is the spatial ow eld and ^r denotes the spatial gradient. The component ofualong the spatial image gradient (the normal ow) is uniquely determined by the constraint equation, while any com- ponent in the iso-intensity tangent plane is unresolved. The normal ow will therefore often be considered the solution to the optic ow constraint equation, keeping in mind that any tangential component can be added.

2.3 Spatio-temporal iso-surfaces

When looking at the spatio-temporal ow at a given point in space-time (x⁰;t⁰) it is constrained to preserve intensity.

Denition 4 (Spatio-temporal iso-surface).

In every point (x⁰;t⁰) where I⁶= 0, I(x;t) =I(x⁰;t⁰) denes the corresponding spatio-temporal iso-surface.

Any ow line is though constrained to lie within the spatio-temporal surface.

This surface is only dened for points where the spatio-temporal gradient I does not vanish. Whenever the image is continuous, the spatio-temporal surface is a closed surface dierentiable to the same order as the image.

IfI^t(x⁰;t⁰)⁶= 0 then the spatio-temporal iso-surface can be locally parame- trised by the spatial coordinates. In this situation, the intensity change in the time direction and the iso-surface will not locally be perpendicular to the spatial directions, i.e. the normal ow is not zero.

Denition 5 (Spatio-temporal iso-function).

The functions(x) : IR^D7!IR is dened in an open set round every point(x⁰;t⁰) whereI^t(x⁰;t⁰)⁶= 0 such that I(x;s(x)) =I(x⁰;t⁰).

The graph of the (spatio-temporal) iso-function is simply the iso-surface. The iso-function is linked to the local ow pattern through proposition 1:

Proposition 1 (iso-function normal ow).

The spatial normal ow through a point (x⁰;t⁰) isuⁿ(x⁰;t⁰) =^krs^k?2rs.

Proof. The ow is determined by the equationu^rI+I^t= 0. By spatially dieren- tiation of the denition of the iso-function @^xI(x;s(x)) = 0 we nd^rs=^?rIIt and thereby the optic ow equation reads u(^?I^trs) +I^t = 0. This reduces to u^rs= 1 which is obviously fullled byu=^krskrs2. Since^rsis directed along the image gradient^rI, this is the normal ow.

We are now capable of linking the geometric structure of the iso-function to the local normal ow. In the points where its tangent (hyper)plane coincide with the (hyper)plane spanned by the spatial dimensions, the normal ow is not dened, but in neighbouring points on the iso-surface, we can nd the ow and categorize the undened ow by its limiting structure.

In the following we will analyse the generic shape of the spatio-temporal iso- surfaces in terms of the spatio-temporal iso-function. Especially we will analyse the generic properties and the corresponding generic ow patterns.

3 Structure and genericity

We dene, as in common catastrophe theory [6], structure as equivalence classes under deformations. That is, given a functionf(x;c) : IR^DIR^k7!IR, wherex are theD spatial coordinates andcare thekcontrol parameters of the function, dene equivalence classes from a class of allowable deformations of xand c. In common catastrophe theory, we dene x⁰ = (x;c);c⁰ = (c), where and are dieomorphisms. Now the game is, given a functionf, to choose ; such that f(x⁰;c⁰) takes a special algebraic form. As an example, any point where the spatial gradient off does not vanish, can by a correct choice of; be put on the form f(x⁰;c⁰) = x⁰¹. We call this representation of f the normal form.

Such an analysis ofC¹functions leads to Thom's classication of catastrophes:

regular points, critical points, folds, cusps, swallowtails, etc., each represented by a normal form.

An event is generic if one can not perturb it away with an innitesimal perturbation. Mathematically that is, the event occurs in an open and dense set of all functionsf. We expect to see only generic events in real image sequences, as all other events has measure zero inC¹. Using the transversality theorem[6], one can argue on genericity simply by counting dimensions. In the product space of spatial coordinates and control parameters we expect an event to occur at a manifold of dimension D+k minus the number of linear constraints on the functions jet to be satised for the event to take place.

In the case of ow elds, or dynamical systems, the class of dieomorphisms is constrained since the ow eld represents a connection of physical points. That is, the deformation of the coordinate system is not allowed to change the ow elds connection of physical points, only its coordinate representation. The result of this is that one cannot remove points of vanishing ow, and one cannot alter the eigenvalues of the matrix containing the spatial rst order derivatives of the ow [1]. This leads to a very ne classication of ow elds since the eigenvalues index the equivalence class. Generically we nd in a ow eld at a given time instance points of vanishing ow (xed points), and in 2D we categorize them according to

their eigenvalues: two positive implies an unstable node (repeller), two negative a stable node (attractor or sink), two of opposite sign a bistable node (saddle node), and a pair of complex conjugate eigenvalues yields a spiralling ow called a focus and which may be stable or unstable according to the value of the real part of the eigenvalues.

When time varies the xed points may interact changing the xed point topology of the ow. Transitions which takes place generically when a single parameter (the time) is varied are called codimension 1 events. In case of the ow eld, one generically meets three dierent events at codimension 1: scatter, saddle bifurcation, or Hopf bifurcation.

4 Structure of the optic ow eld

In this section, we apply the general scheme outline above to the analysis of the structure of an optic ow eld. An optic ow is dened through the conservation of image intensity along ow-lines. We dene equivalence of the ow-eld as identical up to a dieomorphism of the image sequence:

Denition 6 (Image isophote equivalence).

^{Two images I(x;t) : IRD}

IR ^7!IR and J(x;t) : IR^DIR ^7!IR are isophote equivalent or I-equivalent if I(x;t) = ~J(~x;^~t), where

J~(x;t) =(J(x;t)); x~= (x;t); ^~t=(t)

where^J >0 and^t>0 since we want to distinguish also the direction of ow on ow-lines.

Notice, that  is not a function of x ort. This is because the optic ow is dependent on the iso-intensity line (isophote) structure, and a varying dieo- morphism would change this structure.  can only change the intensity values, but not change the isophotes. Without further restrictions the classication of ow structure is rather crude, and we make the following smaller equivalence class, which leads to a ner classication.

Denition 7 (Image stationary equivalence).

Two images I(x;t) : IR^D IR^7!IR andJ(x;t) : IR^DIR^7!IR are stationary equivalent or S-equivalent if they are I-equivalent with (0;t) = 0.

This more restrictive equivalence cannot change points of zero ow, unlike I- equivalence. None of them can remove critical points in the iso-functions. We introduce S-equivalence to make the analogy to nodes and foci of analytical ow elds. An even more restrictive equivalence class could be constructed, not allowing the spatial dieomorphism to vary in time. This would be even more analogous to the classication of the analytical ows since the total rst order ow structure would be invariant under the dieomorphisms. This classication is, however, too ne in our taste and the S-equivalence suces for our purposes, so we will not pursue this direction further in this paper.

We dene local I-equivalence (local S-equivalence) inx⁰;t⁰ as being I-equi- valent (S-equivalent) in an open set roundx⁰;t⁰.

4.1 I-equivalent structure of the optic ow eld

The normal ow is uniquely determined by the spatio-temporal iso-surface when- ever the spatio-temporal image gradient does not vanish, and in positions where its tangent plane is not parallel to the time axis, the ow is uniquely determined by the spatio-temporal iso-function. Under I-equivalence the tangent plane of the iso-surface can be tilted away from being parallel to the time axis, and we need not treat this case separately. Hence we only analyse for general analytic iso-functions and vanishing spatio-temporal gradient.

Proposition 2 (I-normal forms of 2D optic ow, codim 0).

At a xed time-slice t=t⁰ the normal ow is generically in any point I-equivalent to one of the following normal forms:

n⁰(x;y) = (1;0)^T n²(x;y) = 1x²+y²



x

y



where the sign combinations inn²: (+;^?) and (^?;+) are equivalent.

Proof. At a xed time slice, the spatio-temporal gradient of the image will not vanish generically (this happens at codimension 1), and thereby the iso-surface is dened in every point. According to the arguments above we need only to analyse analytic iso-functions. Any regular point on the iso-function are I-equivalent to s(x;y) =x, and by using Prop. 1, we nd n⁰. The only generic critical points are Morse critical points, which are I-equivalent tos=^12x^212y². Again using Prop. 1, we ndn².

The normal form n² has respectively identical spatial ow-lines to the stable node (^?;^?), the saddle (^?;+), and the unstable node (+;+) of an analytical ow eld. However, the velocity increases towards plus/minus innity when the point approaches (0;0).

The classication of the ow follows directly from the classication of critical points in analytical functions making the progress simple. The only twist is that for codimension ^1 there exists generically points where the iso-surface is not dened.

Proposition 3 (I-normal forms of 2D optic ow, codim 1).

^{At a xed}

time-slice t = t⁰ the normal ow is generically in any point I-equivalent with codimension 1 to the normal forms of Prop. 2 or one of the following normal forms:

n²⁺¹(x;y) = 1 x²+y²

xt yt



n³(x;y) = 1 (x²+t)²+y²

x²+t y



wherex,y, and t may independently change sign.

Proof. At codimension 0 we nd the normal forms of Prop. 2. In codimension 1 we divide the analysis into two distinct case. Firstly we analyse the case where the spatio-temporal iso-surface is dened (the fold), secondly the case where it is not dened (the spatio-temporal critical point).

When the spatio-temporal surface is dened, we can using I-equivalence trans- form it into the normal form of general analytical functions. We use the the- orem that iso-surfaces behave as generic functions [9], and nd, at codimen- sion 1, the only extra normal form compared to codimension 0 is the fold[6]:

s(x;y) =x³+tx+y². By use of Prop. 1, we nd n³. Since I-equivalence only allows positive Jacobians in the dieomorphisms, we must represent the signs of y andt explicitly.

When the spatio-temporal image gradient vanishes in (x⁰;t⁰), the iso-surface is not dened in this point. We can bring a spatio-temporal critical point on the following normal form by I-equivalence (up to signs of the individual terms):

I(x;y;t) =x²+y²+t². We divide into two cases dependent on the sign of t, and nds=^p(I^0?x^2?y²), whereI⁰ is the intensity of the iso-surface. By Prop. 1 we nd the normal ow:

n=^

pI^0?x^2?y² x²+y²

x y



By substitution of the expression for t =s(x;y) into this, the sign cancels out, and we nd in both casesn²⁺¹.

Codimension one events take generically place in xed time slices in a time sequence; The top of Figure 1 illustrates these events. A stable pole will always meet the saddle pole in its unstable direction while an unstable pole meets a saddle in its stable direction. This is illustrated in Figure 2.

Proposition 4 (I-normal forms of 2D optic ow, codim 2).

^{At a xed}

time-slice t = t⁰ the normal ow is generically in any point I-equivalent with codimension up to 2 to one of the normal forms in Prop. 2 or Prop. 3 or one of the following normal forms:

n²⁺²(x;y) = t²¹+t² x²+y²

x y



n³⁺¹(x;y) = t¹ (x²+t²)²+y²

x²+t² y



n⁴(x;y) = 1

(x³+t²x+t¹)²+y²

x³+t²x+t¹ y



wherex,y, andtⁱ may independently change sign. Any of thetⁱ may correspond to the physical time parameter.

Proof. The proof follows the lines of the proof in Prop. 3. First we divide into cases where the spatio-temporal iso-surface is dened or not. n⁴ follows easily from a cusp in the iso-function: s(x;y) =x⁴+t²x²+t¹x+y².

n

2+1 n

3

n n 4

3+1 n

2+2

Fig.1. Top, the generic events of codimension 1.n²⁺¹is a pole, where the directions of the ow are reversed.n³describes the interaction of two poles, where a saddle pole and a (un)stable pole interact and annihilate or are created.Bottom, the generic events of codimension 2.n²⁺²is a pole that may change direction twice.n³⁺¹is an annihilation in t², butt¹ interchanges the stability of the poles. For instance, a stable pole and a saddle approach like at an annihilation, but they scatter and become a saddle and an unstable pole. n⁴ is a pitchfork bifurcation. An example is a stable pole and two saddles approach, interact and become a single saddle.

In case of a vanishing spatio-temporal gradient, we subdivide into two cases dependent on whether I^xx or I^tt vanishes in the spatio-temporal critical point.

In the rst case we have a spatial fold as in n³, but augmented by a vanishing temporal derivative, yieldingn³⁺¹. In the latter case we have a spatially critical point in which a temporal fold happens, yielding n²⁺². The algebraic derivations are similar to the derivation ofn²⁺¹.

In these normal forms we use two control parameters of which one is the time parameter and the other maybe most easily is visioned as a scale parameter even though these forms have not yet been proven to be the normal forms when the evolution along a control parameter is constrained as in the case of Gaussian scale space. Below we cite a theorem showing that even when the additional control parameter is a scale parameter, these normal forms are valid.

The codimension two events are illustrated schematically at the bottom of Figure 1. Assumet¹is the time parameter, andt²is negative. Then whent²¹>^jt^2j n²⁺² is an unstable pole, and when t²¹<^jt^2j it is a stable pole. Ift²is positive, it is always unstable. Exactly when t² = 0, the pole disappears fort¹ = 0 but reappears with same orientation innitesimally later.

4.2 S-equivalent structure of the optic ow eld

The more restrictive S-equivalence cannot as I-equivalence remove vanishing ow. Hence, points with vanishing ow renes the classication. The intuitive

Fig.2.Then³ normal form fort=^?0:25;0;0:25. A saddle and an unstable pole (left) meet (middle) and annihilate (right). Only the orientation of the ow is shown. The magnitude increases towards innity near the poles.

key to the additional normal forms is the spatio-temporal surface of vanish- ing temporal derivative T = ^fx;y;t^jI^t(x;y;t) = 0^g. Since the image sequence is assumed to be dierentiable, generically T will be a dierentiable non self- intersecting surface.

Proposition 5 (S-normal forms of 2D optic ow, codim 0).

At a xed time-slicet=t⁰the normal ow is generically in an open spatio-temporal neigh- bourhood round any point S-equivalent to one of the following normal forms:

n⁰(x;y) = (1;0)^T m¹(x;y) = (x^?t;0)^T m²(x;y) = (x^?(y+ 1)t;0)^T

n²(x;y) = 1x²+y²



x

y



where the sign combinations inn²: (+;^?) and (^?;+) are equivalent.

Proof. The iso-surface is dened everywhere since at a xed time the spatio- temporal image gradient is not generically zero. For non-vanishing temporal derivative we arrive at n⁰ andn² for regular respectively critical spatial points.

m¹ orm² occurs forI^t= 0. In a spatial coordinate system (v;w) wherewis the image gradient direction, we nd the parameters of the dieomorphism such that the normal ow takes the form of m¹. This form of the dieomorphism is only valid wheneverI^tt6= 0. ForI^tt= 0 we ndm². All these computations have been omitted in this paper due to the space limitations and their algebraic complexity.

The number of linear constraining equations for a particular form determines the dimension of the set with points equivalent to the form. Hence,n⁰,m¹ and n² points group in manifolds of dimension two, one and zero, respectively.

The normal formm¹ shows a line of zero normal ow, denoted a xed line.

Notice that S-equivalence do not distinguish attracting and repelling lines. m² counts for that the xed line in a point does not move. The xed line rotates locally round this point, and we denote this event a \whirl".

Proposition 6 (S-normal forms of 2D optic ow, codim 1).

At a xed time-slice t = t⁰ the normal ow is in a generic one-parameter family in an open spatio-temporal neighbourhood round any point S-equivalent to one of the normal forms of Prop. 5, Prop. 3, or the following:

m¹⁺²(x;y) = (^y^2x²+t;0)^T m²⁺¹(x;y) = (x^(y²+ 1)t;0)^T m³(x;y) = (x^?(y+ 1)t²;0)^T

where the sign combinations inm¹⁺²: (+;^?) and (^?;+) are equivalent.

Proof. m¹⁺² follows from m¹ when also the spatial gradient of I^t vanishes.

m²⁺¹(x;y) follows from m² with the additional constraint that the spatio-tem- poral line of a whirl is locally orthogonal to the temporal dimension.m³ follows fromm² whenI^ttt= 0. Again algebraic derivations have been omitted.

The eventm¹⁺² is, for our purposes, the most important event arising from the S-equivalence next to m¹, if one is interested solely in the xed lines. The latter describes that the normal ow vanishes at lines.m¹⁺² describes topology change of zero ow lines, denoted xed lines. Depending on the signs, it is either a creation event (^?;^?), an annihilation event (+;+) or a xed line saddle event (+;^?) or (^?;+). During an annihilation or creation event a circular zero ow line vanishes/appears. During the xed line saddle event the connectivity of two zero ow lines changes. Four incoming lines meet in a cross exactly during the event. Before and after two dierent pairs of incoming lines are connected.

The eventm²⁺¹ describes the annihilation/creation of a pair of whirls. The two whirls will have opposite rotation directions. In a point they meet and an- nihilate. Even thoughm² does not distinguish the rotation direction, since two whirls of opposite rotation are S-equivalent,m²⁺¹constrains the whirls to having opposite rotation since the dieomorphism can only change direction for both simultaneously. In Figure 3 top arem¹⁺² andm²⁺¹ illustrated.

m³accounts for a locally stationary whirl. This point corresponds to a cusp in the function surfaceI^t= 0. Figure 3, bottom-left illustrates this. Going through the cusp, does not in codimension 1 change the rotation direction of the whirl.

Whirls may change direction, but this event is not singled out since it is S- equivalent to the whirl itself. If one is interested in orientation of whirls another equivalence class must be constructed to subserve this analysis. However, what one can say directly is that in the real spatial plane, there will always be equally many left and right whirls. This holds for any subset of the plane where all xed lines form closed curves, since a curve will after an innitesimal perturbation cross the un-perturbed curve an even number of times, and these crossings will

m

1+2 m

2+1

It=0

Fig.3. Top, Two events of codimension 1:m¹⁺² andm²⁺¹.Bottom, left is the sta- tionary xed line whirl. Time is vertical. It corresponds to the cusp point of the surface of I^t = 0. The dashed lines are the xed line at dierent time instances (horizontal planes cutting the surface). The dots are the corresponding whirls. They move on a parabola open in the direction towards the reader. Right is an illustration of the prin- ciple that a perturbed curve crosses the original curve an even number of times, and equally many times from inside as from outside.

be equally many outwards and inwards crossing. This is illustrated in Figure 3 (right).

The S-equivalence implies that on top of the poles, points of zero motion is the basis of the taxonomy of image sequence structure. The S-equivalence rst picks up lines of zero ow, xed lines. Then points where these lines do not move (whirls) and points where the xed lines changes topology. It does not distinguish attracting and repelling lines.

4.3 A comment on the multi-scale optic ow structure

A scale space is constructed by convolving the image by Gaussians so that the scale-space fullls the Heat equationI^s=⁴I, wheresis the scale parameter and

4denotes the spatial, the temporal, or the spatio-temporal Laplacean dependent on which scale-space one constructs. The analysis of structure in scale-space can not be done by simply using transversality arguments and referring to Thom's classication. The proper analysis has been performed by Damon[3]. In conjunc- tion with ow, however, we have proof, but leave it out here due to the space limitations, that under the heat equation, a spatio-temporal iso-surface is not constrained in its local deformation, only in its topology changes. The idea is that the second derivative across the surface, may make the surface evolve in any direction in its jet space. Similar has been proven by Kergosien and Thom [9] for iso-surfaces of general analytical functions. The importance of these two results are that we can argue of genericity simply by counting constraints on the iso-function: they translate to simple linear bands on the image jet. Thus in all the above normal forms, time may be exchanged with scale. The only limitation is that the time and the scale parameter may not coincide.

resolution. In general, intensities at larger scales will not be preserved since intensities will change weight under the Gaussian aperture functions due to the ow. This is treated in detail in [4]. The above normal forms are though still valid as they deal with the innite resolution ow, but argued from a scale-space point of view, they can never be accessed.

5 Detection of structural changes

The change of topology in the poles or lines of zero normal ow is characterised by the corresponding normal forms. Thus, to detect change in the structure of the ow eld we must detect when and where the normal forms apply. In table below we list the conditions for the events and name the events. In the tablep denotes the direction in which the spatial second order image structure vanishes.

We compute derivatives of digital images as scale space derivatives. That is, we observe the image under a Gaussian aperture dening the spatial and temporal scale (inverse resolution). By dierentiation of this spatio-temporal Gaussian prior to convolution, the computation of image derivatives is well- posed. The side eect is that it is not the image at grid resolution but at a lower resolution which is the object of analysis. We do not in this paper take into account the aspects due to the non-commutation of the Gaussian convolution and the deformation due to ow eld. These eects have been analysed by Florack et al. [4].

n⁰ Regular point ^rI ⁶= 0

n² Pole I^x= 0,I^y= 0

n²⁺¹ Pole stability reversion I^x= 0,I^y= 0 ,I^t= 0

n³ Pole pair creation I^x= 0,I^y= 0,I^xxI^yy?I^xy2 = 0 n²⁺² Pole stability fold I^x= 0,I^y= 0,I^t= 0, I^tt= 0

n³⁺¹ Pole scatter I^x= 0,I^y= 0,I^t= 0, I^xxI^yy?I^xy2 = 0 n⁴ Pole pitchfork bifurcation I^x= 0,I^y= 0,I^xxI^yy?I^xy2 = 0, I^ppp = 0

m¹ Fixed line I^t= 0

m² Fixed line whirl I^t= 0,I^tt= 0

m¹⁺²Fixed line creation I^t= 0,I^xt = 0,I^yt= 0

m²⁺¹Fixed line whirl creation I^t= 0,I^tt= 0, I^tyI^ttx?I^txI^tty = 0 m³ Stationary xed line whirlI^t= 0,I^tt= 0, I^ttt= 0

The zero locus of pre-computed dierential expression is computed using an algorithm similar to the Marching Cubes algorithm [12]. For each dierential expression, the zero locus is computed, and the intersection of loci is computed using an algorithm assembling the Marching Lines algorithm [15]. In this way the normal ow events are detected and their spatio-temporal position simulta- neously computed.

In the following we detect some of these in two dierent image sequences.

First, we detect the poles and their temporal interaction in a sequence of a person walking in a hall way. Secondly, we detect the lines of xed ow and

their interaction in a sequence of turbulent ow, and use the scale interaction for quantifying the amount of turbulence in the sequence.

5.1 Temporal pole evolution

Figure 4 illustrates the detection of ow poles and their temporal interaction in a sequence of a person appearing in a hall way. We see poles due to critical image points at the scale at hand. These are distributed all over the image, and most have close to constant positions. However, in the center region, where the person appears in the hall way we see poles created and annihilated. These points corresponds to points where the topology of the ow pattern changes.

That is, these creation/annihilation points are invariant to any additional ow added to the normal ow. In this way they are not in uenced by quantitative aspects such as speed, orientation etc. We suggest that they may be used for guiding an attention mechanism.

5.2 Fixed line scale-evolution

In turbulent ow, the degree of turbulence can be accessed through the scaling properties of the \eddies". Kolmogorov introduced the cascade models of turbu- lent ow, looking at the energy transport from large scale eddies to small scale eddies [11]. Frisch introduced a variant of these called the-model [5] where the variable of interest is the scaling properties of the space lling of eddies. The so-called structure function characterising the ow is dened in terms of the scaling exponent of the space lling of the eddies. In the following, we sketch how this can be accessed through the multi-scale optic ow structure.

At every scale a number of whirls is present. As an approximation we assume that a whirl corresponds to an eddy and that its space lling corresponds to its area, that is its spatial scale squared. The scaling exponent of the energy as a function of scale may then be estimated from counting whirls at a number of dierent scales.

In Figure 5, smoke induced into a ventilated pigsty is shown. The smoke is illuminated by a laser scanning through a plane such that the smoke in a vertical 2D plane in the 3D pigsty is imaged. In Figure 5 bottom-right the scale evolution of whirls in the Pigsty sequence is shown including annihilation (and the few creation) events. As indicated above the scaling properties of the whirls may be used for accessing the degree of turbulence in the ow. From the number of detected whirls as a function of scale we nd approximately that V ^/ s^0:5. That is, only 70 percent of the energy is transported to whirls at the half length scale¹. This computation is, though, based only on approximately 100 whirls and a scaling interval of a single decade. This is clearly insucient to state that we have proven self similarity or precisely computed the degree of turbulence. We have merely indicated a direction in which the structure of the optic ow eld as suggested in this paper can be used for more practical exercises.

1 Dimensional analysis indicates a scaling ofS⁰, so this is not a trivial result

Fig.4. Top, the rst, middle, and last frame of the hall way sequence.Bottom,The spatio-temporal curves of the poles and their creation points for spatial scale s = 5 (left) ands= 8 (right). Temporal scale is 2. Sequence is 256^256^32 pixels cubed

6 Summary

We have introduced two equivalence classes of optic ow and derived normal forms of codimension 0, 1, and 2 (the latter only in case of I-equivalence). I- equivalence leads to a denition of structure as the poles in the ow eld whereas the S-equivalence leads also to xed points.

The major dierences to normal analytical ow elds as in autonomous dy- namical systems, is the presence of poles and that the tangential component of the optic ow eld is undened. The poles can only be avoided by a regular- ization of the ow eld as is normally done in computer vision algorithms [7].

An arbitrary \gauge condition" could be imposed to x the tangential compo- nent and in this way xed points as in dynamical systems can be introduced.

Restricting the equivalence class of ows could make the analogy to dynamical systems even larger.

Fig.5. Top, the rst, middle, and last frame of the pigsty sequence. The xed lines and the whirls are superimposed on the middle frame. Bottom, left is the temporal evolution of whirls and their creation/annihilation points. Right are the whirls in the middle frame as a function of scale. Points mark annihilations or creations.

We have introduced a concept of whirls. These however have a very dierent nature than the nodes in dynamical systems since the whirl include second order temporal structure, and the analogy to nodes is not clear.

The natural continuation of the research presented in this paper is to look at a gauge xed tangential ow, and to introduce temporarily constant dieomor- phisms for denition of the equivalence of ow. In this way the only dierence to dynamical systems may be the poles.

The theoretical results in this paper has been applied to two simple examples:

computation of spatio-temporal points in which the topology of the ow eld changes, as a mechanism for guiding attention, and computation of the scaling properties of whirls as to characterize turbulent ow.

References

1. Vladimir I. Arnol'd. Ordinary Dierential Equations. Springer Verlag, 3 edition, 1992.

2. J. Arnspang. Notes on local determination of smooth optic ow and the trans- lational property of rst order optic ow. Technical Report 88-1, Institute of Datalogy, University of Copenhagen, Denmark, 1988.

3. J. Damon. Local Morse theory for solutions to the heat equation and Gaussian blurring. Journal of Dierential Equations, 1993.

4. Luc Florack, Wiro Niessen, and Mads Nielsen. The intrinsic structure of optic ow incorporating measurement duality. International Journal on Computer Vision, 1997. (In Press).

5. U. Frisch, P. Sulem, and M. Nelkin. A simple model of intermittent fully developed turbulence. Journal of Fluid Mechanics, 87(4):719{736, 1987.

6. Robert Gilmore. Catastrophe Theory for Scientist and Engineers. Dover, 1981.

7. B. Horn and B. Schunck. Determining optical ow. Articial Intelligence, 23:185{

203, 1981.

8. B. Jahne. Spatio-Temporal Image Processing-Theory and Scientic Applications, volume 751 ofLecture Notes in Computer Science. Springer, 1993.

9. Y. L. Kergosien and R. Thom. Sur les points parabolique des surfaces. Technical Report 290:705{710, C.R. Acad. Sci. Paris t., 1980.

10. J. J. Koenderink and A. J. van Doorn. Second order optic ow. Journal of the Optical Society of America, 8(2):530{538, 1992.

11. A. N. Kolmogorov. The local structure of turbulence in incressible viscious uids for very large reynolds numbers. Technical report, C R Acad. Sci. USSR 30, 301, 1941.

12. William E. Lorenson and Harvey E. Cline. Marching cubes: A high resolution 3d surface reconstruction algorithm. Computer Graphics, 21(4), 1987.

13. H. H. Nagel. Displacement vectors derived from second-order intensity variations in image sequences. Comp. Graph. and Image Proc., 21:85{117, 1983.

14. M. Otte and H. H. Nagel. Optical ow estimation: Advances and comparisons.

In J.-O. Eklundh, editor, Proc. Europ. Conf. on Computer Vision, pages 51{60, Stockholm, Sweden, 1994.

15. Jean Philip Thirion and Alexis Gourdon. The marching lins algorithm: new results and proofs. Technical Report 1881, INRIA, 1993.

16. A. Verri, F. Girosi, and V. Torre. Mathematical properties of the two-dimensional motion eld: from singular points to motion parameters. Journal of the Optical Society of America-A, 6(5):698{712, 1989.

17. P. Werkhoven. Visual Perception of Successive Order. PhD thesis, Utrecht Uni- versity, University of Utrecht, Dept. of Physics of Man, Princetonplein 5, 3508 TA Utrecht, the Netherlands, May 1990.