CAP 5415 Computer Vision Fall 2005

(1)

Alper Yilmaz, Fall 2005 UCF

CAP 5415 Computer Vision Fall 2005

Dr. Alper Yilmaz

Univ. of Central Florida

www.cs.ucf.edu/courses/cap5415/fall2005 Office: CSB 250

Recap Motion

z

Brightness constancy constraint

z

Optical flow equation, brightness constancy equation

– Normal flow can be computed

– Parallel flow cannot

) , ,

( ) , ,

(x y t I x x y y t t

I = +∆ +∆ +∆

=0 + + _y _t

x vI I

uI = + =0 lineequation

y t y x

I I I u I v

(2)

Recap

Computing Optical Flow

z

Define energy function

– Brightness constancy + smooth solution (Horn&Schuck)

– Common flow (Schuck)

– Brightness constancy (Lucas&Kanade)

zLeast squares fitting

Recap

Global Motion

z

Common motion of pixels observed in frame

– Camera motion or rigid object motion

z

Affine Model

– Affine Transformation

– Affine Motion

(3)

Recap

Affine Transformation

z

Direct relation between pixel positions

2 4

3

1 2

1

'' ''

b y a x a y

b y a x a x

+ +

=

+ +

=

image at time t

(x,y)

image at time t+1

(x’’,y’’)

Recap

Affine Motion

image at time t

(x,y)

image at time t+1

(x’’,y’’) (u,v)

x x

u= ''− v= ''y −y

1 2

) 1

,

(x y a x a y b

u = + + v(x,y)=a₃x+a₄y+b₂

1 2

'' x a1x a y b

x − = + +

1 2 1 1) (

'' a x a y b

x = + + + y ''=a₃x+(a₄+1)y+b₂

(4)

Recap

Affine Motion and Transformation

z

Transformation Motion

2 4 3

1 2 1

'' ''

b y a x a y

b y a x a x

+ +

=

+ +

=

2 4

3

1 2 1

) 1 ( ''

b y a

x a y

b y a x a

x

+ + +

=

+ +

+

=

translation rotation

Rigid (rotation and translation) Affine

shear

Recap

Anandan’s Approach (Affine Motion)

2 4 3

1 2 1

) , (

b y a x a y x v

b y a x a y x u

+ +

=

+ +

=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

⎥⎦

⎢ ⎤

⎣

=⎡

⎥⎦

⎢ ⎤

⎣

⎡

2 4 3 1 2 1

1 0

0 0

0 0 0 1 )

, (

) , (

b a a b a a

y x y x y x v

y x u

Xa u =

James R. Bergen, P. Anandan, Keith J. Hanna, Rajesh Hingorani: “Hierarchical Model-Based Motion Estimation," ECCV 1992: 237-252

= 0 +

∆ I u

^T

I

_t

= 0 +

∆ I Xa

^T

I

_t

( )

∑

^∆ ⁺

= I^T I_t ²

E Xa

minimize

0 ) (

) (

2 ∆ +∆ =

∂ =

∂^E_a

∑

Î^T^X ^T Î^t Î^T^Xa

∑

^∆ ^∆ ⁼⁻ ^∆

pixels all pixels

all

t T T

T I I Xa X I I

X

(5)

Algorithm

Block Based Optical Flow

(6)

Block Based

z

Select a patch P image at time t

z

Search for P at frame t+1 in a larger neighborhood

P

Block Search

z Search for patch in overlapping windows

z Compute similarity of intensity values between original patch and search patch

z Select the location with highest similarity

z Distance vector between centroids give optical flow vector

(7)

Computing Similarity

( )

∑∑

⁻

= I_t₊₁ I_t ² SSD

( )|

| ₁

∑∑

⁻

= I_t₊ I_t AD

∑∑

⁺

= I_t ₁I_t CC

( )

∑∑ ∑∑

⁺

=

t t

I I

.

1. NC

Sum of square differences

Absolute difference

Cross correlation

Normalized correlation

( )( )

∑∑

⁻ ⁻

= ₊ ₊

+ t t t t

t t

I

I µ µ

σ

σ ₁ ¹ ¹

64

MC 1 Mutual correlation

µ, σare region mean and stdv

Correlation Surface

z

Using Cross correlation

^CC⁼

∑∑

^I^t⁺¹^I^t

mission reference Correlation surface

Find peak in correlation surface

(8)

Issues With Correlation

z Patch Size

z Search Area

z How many peaks

z Computationally expensive

– Same operations in Fourier domain takes less time

zTake FFT of image patch and search area

zMultiply Fourier coefficients to construct corr. surface

zFind maximum

z Should use pyramids here too for large displacements

Token Based Optical Flow

(9)

Tokens

z

Interest points

– Movarec’s operator

z

Corners

– Harris corner detector

z

Edges

– Any edge detection algorithm

Overview

z

Given two consecutive images

z

Find tokens in both images

z

Find token correspondences

Frame at time t Frame at time t+1

(10)

Representing the Correspondence

z

A graph G(V,E) is a triple consisting of a vertex set V an edge set E and a relation that associates two vertices with an edge.

Representing the Correspondence

z Bipartite graph: A graph G is bipartite if its

vertex set can be partitioned in two subsets

such that no two vertex in same set have a

common edge.

(11)

Finding Correspondence

z Finding matching: Matching is a set of edges such that no two of them have a common vertex

w11

wkl

M M

w11

wkl

M M

Token Based Optical Flow

z

Tokens corresponds to vertices in the bipartite graph

z

Tokens at time instants t and t+1 form partite sets of graph.

z

The cost of corresponding a point at instant t

to a point at instant t+1 is the weight of edge

between the corresponding vertices.

(12)

Defining Weights

Maximum Speed

Consistent Match

Common Motion Minimum Velocity

Model

Weights

i j

ij y x

w = −

( )

₍ ₎

( )

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

−

− −

−

⎥+

⎥

⎦

⎤

⎢⎢

⎣

⎡

−

− −

=

i i

x i j

ij AM y x v

v x y c GM

v x y

v x c y

w

, 1 ,

. 1 1

( ) ( ), 2

, b b a a AM

ab b a GM

= +

= mean

aritmetic mean geometric

( )

∑ ∑

= = = = − + −

−

= − _p

k q

l l p

i j p

k q

l x l k

i j x

ij y x

x y x

y v

x y w v

k i

1 1

Ullman:

Sethi & Jain:

Rangarajan &

Shah:

(Absolute distance between points)

(13)

Algorithm

Find initial correspondences using correlation

Compute costs w_ijfor each pair of points x_i, y_j

z Construct a bipartite graph based on computed costs

z Prune all edges having weights exceeding certain threshold

– Define cost matrix

z Find the minimum matching of constructed graph.

– Hungarian Algorithm

– Greedy search