Motion Estimation. 5LIN0 Video processing. Video course: Motion Estimation. G. de Haan. Schedule lectures 5P530. Picture delay

(1)

technische universiteit eindhoven

5LIN0

Video processing

G. de Haan

Schedule lectures 5P530

2

Week 1

Week 2

Week 3

Week 4

Basics

(Ch 2, 3)

Video Displays

(Ch 9)

Filtering

(Ch 4)

PRC &

De-interlacing (Ch 7,8)

Week 5

Week 6

Week 7

Week 8

Image

Enhancement

(Ch 5)

Motion Estimation

(Ch 10)

Object Detection

(Ch 11)

X

technische universiteit eindhoven 3

Motion

Estimation

Motion Estimation

• Is there any motion?

• How fast?

• Into which direction?

D

x

D

y

5

Application dependency of ME

• Scan rate conversion

(true-motion vectors)

• De-interlacing

• Picture rate conversion

• Video compression

(low prediction error)

• MPEG

•

H.2.63

• True-motion vectors are usually more consistent than

coding vectors. Consistency has some, but no dominant

relevance for coding efficiency

ME

6

Motion estimation and coding

Image compression:

accuracy demands

decrease with increasing

frequency (DCT-transform +

quantization)

Input

Output

+

Picture

delay

-Prediction

error

Motion

compensation

(2)

Gradient ME

methods (optical

flow)

Basic assumptions and consequence

• Constant brightness assumption

• Local linear luminance assumption

8

F(x, n-1)

_{F(x, n)}

F(x, n-1) F(x, n)

Image

n

-1

Image

n

D

Basic assumptions and consequence

• Constant brightness assumption

• Local linear luminance assumption

9

F(x, n-1)

_{F(x, n)}

F(x, n-1) F(x, n)

x-D D

Basic assumptions and consequence

• Constant brightness assumption

• Local linear luminance assumption

10

F(x, n-1)

_{F(x, n)}

F(x, n-1) F(x, n)

x-D’ D’ x-D D 11

DFD

2 Displacement D i I+1 I+2_I+3 dDFD2 dD

Algorithm: Determine

gradient of displaced

frame difference (DFD),

and update vector in

direction of decreasing

DFD.

Iterative optical flow (to deal with non-linear brightness)

12

Iterative optical flow

u

D



_i





_i1





1)

)

,

(

x

D

₁

n

DFD

D

u





_i



d



2)

)

1 ,

(

)

,

(

)

,

(

x

D

₁

n



F

x

n



F

x



D

₁

n



DFD



_i



_i

3)

)

,

(

)

,

(

1

F

x

D

1

n

x

n

D

x

DFD

u





i



i



d



4)

(3)

Time

x

Pel-recursive ME; The use of predictions

Current pixel Spatial causal prediction Temporal prediction

The temporal candidate may also be motion

compensated

14

Not popular for video format conversion

• Initially due to complexity

• Real-time applications:

• Initially coding, later also format conversion

•

For coding one vector per pixel is not attractive

•

There are much simpler block-based methods

• For format conversion true-motion requirement problem

•

Artifacts when assumptions are invalid

Block-matching

ME methods:

Full-search

n -1

Block-matching; find corresponding block in image n-1

Image number n

Current block Search area

Corresponding block

17

Finding block similarity

Search area

x

D

y

D

Current block

18

Formal definitions

)

1 ,

(

x



C

n



F



Luminance value in

previous picture, shifted

over candidate vector C:

And the resulting candidate vector for which the error is minimal

is assumed to be the displacement vector:

)

,

(

x

n

D



A block matcher optimizes a function, Cost, varying C:









) (

))

1 ,

(

),

,

(

)

,

(

X B x

n

C

x

F

n

x

F

Cost

n

X

C

 





(4)



  





) ( ( ) 2 2 ) (

)

1 ,

(

).

,

(

)

1 ,

(

).

,

(

)

,

(

X B x xBX X B x

n

C

x

n

x

n

C

x

F

n

x

F

n

X

C

F

     





Normalised cross-correlation

• favourable performance

• rather high operations count

)

(

2 ) (

)

1 ,

(

)

,

(

)

,

(









X B x

n

C

x

F

n

x

F

n

X

C

 





Summed Square Error

• good performance

• acceptable operations count









) (

)

1 ,

(

)

,

(

)

,

(

X B x

n

C

x

F

n

x

F

n

X

C

 





Summed Absolute Difference

• still good performance

• favourable operations count





















)

(

,

0 )

(

,

1 )

(

:

)

1 ,

(

)

,

(

)

,

(

) (

)

(

threshold

a

threshold

a

T

with

n

C

x

F

n

x

F

T

n

X

C

X B x  





Significantly differently pixels

• Rather poor performance

• Favourable operations count, reduced register size

compared to SAD

23

• Correlation (

NCCF

) of pixels in the two blocks

• Mean Square Error (

MSE

) between pixels in the blocks

• Mean Absolute Difference

(

MAD

) between pixels in the blocks

• Number of significantly different pixels (

NSD

) in the two blocks

Co

m

p

lex

it

y

Alternative match criteria

24

Comparison of match criteria

(5)

25

Operations count of full search block matching

• CCIR signal

• 720x576x50 (pixels/s)

• Search window for realistic velocities

• 64x48 (HxV in pixels) = 3000 possible vectors, assuming

integer vector accuracy

• Matching error (SAD) calculation only:

• approximately: 2x10

11

(ops/s)

• NB: Full HD (1920x1080x50) requires even more than 4

times as many computations!

5LIN0

Video processing

G. de Haan

Block-matching

efficient search

techniques

Finding block similarity

Search

area

x

D

y

D

Current block

29

Sub-sampled search

Search

area

x

D

y

D

Current block

30

Search

area

x

D

y

D

1

2

(6)

Search area

x

D

y

D

3-step search (Koga et al., 1981)

x

D

y

D

One-at-a-time search (Srinivasan & Rao, 1985)

y

1 2 3

x

min

x

0



i

1j

Contour plot of

error plane

x

D

y

D

Successive approximation may become necessary

x

₀ X0(b) X0(a) X0(d) Xmin(d ) Xmin(c ) _X 0(c) Xmin(b)  Xmin(a)

Contour plot of

error plane

x

D

y

D

Prevention of trap in local minimum

35

Intermediate conclusion

• Efficient search techniques can highly reduce the operations

count of a block matching motion estimator,

• but increase the risk of getting trapped in a local minimum

of the error function…

• Methods to prevent the disadvantages of efficient search,

increase complexity again.

(7)

37

And sometimes there is no unique solution…

Comparison of search techniques

FS

LogS

OTS

Pixel

sub-sampling in

match function

40

Pixel sub-sampling of match error criterion

Search

area

x

D

y

D

Current block

41

Pixel sub-sampling in match error criterion

1

4

2

4

42

Block

sub-sampling

(8)

V-position

n-1

n

Picture number H-position

Search area

Candidate vector Current block

Block sub-sampling

44

Interpolate missing motion vectors

Up

Le

current

Ri

Lo

Current D

x

= median{Le

x

, (Up

x

+Lo

x

)/2, Ri

x

}

Current D

_y

= median{Le

_y

, (Up

_y

+Lo

_y

)/2, Ri

_y

}

1:

2:

Use the

vector-median

to prevent new vectors

Vector median: generalization of scalar median

45

Vector that has

smallest (Euclidean)

distance to al other

vectors

Scalar median

Vector median

46

Summary cost reduction block matchers

• Simple match criterion

• Efficient search strategy

• Pixel sub-sampling in match criterion

• a factor of four is usually feasible with little influence on the

performance

• Block sub-sampling

• only valid if motion field is smooth

47

Vectors and

object velocity

(9)

True motion

versus

best match









) (

)

1 ,

(

)

,

(

)

,

(

X B x

n

C

x

F

n

x

F

n

X

C

 





SAD :

C is motion vector, F image grey value B 8x8 block, x pixel position, n picture nr

Number 7 Arm Scarf

1 2 3

Seven: 1 clear minArm: no clear minScarf: multiple min

1

3 2

Poor relation vectors & velocities

Block-matching

true-

motion

estimation

51

What is wrong with block matching?

• Blocks are not unique

• Optimization is ill-posed problem

• Testing for best match gives too many solutions

• Solutions:

• Introduce bias, e.g. towards consistent vectors (

test better

)

• Post-processing, e.g. eliminating outliers (

test again

)

• Pre-selection of likely candidates (

test less

)

Introduce bias

Test better…

Introduce bias – Test better

53 ) ( ) ( | ) 1 , ( ) , ( | ) , , (CXn xB(X)Fxn Fx Cn PsC PtC               



 

An improved criterion takes into account that vectors are consistent

within objects and over time:

Ps and Ps are penalties depending on spatial and temporal

consistency of the candidate vector



     ()| (,) ( , 1)| ) , , (CXn xBX Fxn Fx Cn      

Minimal match error gives no unique solution

PROBLEM:

Consistently only known after completion…

Typically solved using an

iterative approach

54

Post-processing

Test again..

(10)

V-Pos

H-Pos

y-2Y

y-Y

y

y+Y

y+2Y

y+3Y

x-4X x-2X

x

x+2X

x+4X

Post processing

to improve vector consistency

(Reuter, 1988)

ood

Neighbourh

k

X

D

F

X

D

o



p







),

(

)

(

56

The effect of post-filtering (5x3 blocks)

Original

Average

Median

Pre-selection

Test less…

Original picture

Down-sampled picture

at intermediate level

Down-sampled picture

at highest level

Initialise Initialise Coarse estimation Medium size

update vectors Small size

update vectors

Hierarchical block matching (Thoma & Bierling, 1989)

59

Hierarchical block matching

Hierarchical

Full search

Pre-selection in Fourier domain- Phase Plane Correlation

•

PPC is a two-step hierarchical motion estimator

•

1) Selectup to 10 largest correlation peaksin the Fourier domain using blocks larger than 64x64

•

2) Test SAD only for these vectorson small block, here 8x8, in the spatial domain

•

Algorithm originally proposed by Graham Thomas, and applied in professional studio format converters

(11)

+6

+4

+2

0 -2

-4

-6

C

_y

-6

-4

-2

0 +2

+4

+6

C

x

Time recursive block matching (Ninomya, 1982)

Test SAD only for these vectors

centred around

result vector previous picture

ST-recursive

candidate selection

next time

5LIN0

Video processing

G. de Haan

3-D Recursive

Search

block-matching

65

1. Objects are LARGER than blocks

Assumptions:

3-Dimensional Recursive Search (3DRS)

Candidate set

•Spatial candidates

•Temporal candidates

•Updated candidates ?? 2. Objects have INERTIA

66 x

D

y

D

Temporal prediction candidate Spatial prediction candidates Noise vector update

(12)

Chosen candidates

Spatial

Temporal

Update

Performance

technische universiteit eindhoven 69 125 100 75 68 22 10 0 20 40 60 80 100 120 140 PPC 4-St 3-St OTS H2 3-D RS

FS: 2000

H3: 1500

Pel-Rec:1000

Operations Count

Smoothness of vector field

• Compute difference with

all neighbouring vectors

• Average over all blocks in

vector field

• This gives vector

inconsistency

• Smoothness is the inverse

of vector inconsistency

70 71 0.2 0.3 0.3 0.5 0.8 0.9 4.3 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 4-St 3-St FS OTS H2 PPC 3-D RS

Vector field smoothness

72

2 )) , ( ) , ( ( ) (n Fxn F xn MMSE mc x     

_





( (), 1) ( (), 1)



2 1 ) , (xn FxDxn FxDx n Fmc       

ME

picture nr.

n-1

n

n+1

MC

(13)

technische universiteit eindhoven 73 244 196 189 137 120 112 101 106 0 50 100 150 200 250 4-St OTS 3-St H2 FS H3 PPC 3-DRS

M2SE score of ME-methods

Comparison of best vector fields

74

Phase Plane Correlation motion vectors

3-D Recursive Search BM motion vectors

Interpolated images using full search motion vectors

Interpolated image using 3D-RS motion vectors

In contrast with coding, for scan rate conversion true-motion is an absolute must. RATHER SMOOTH THAN ACCURATE!!

MC up-conversion; Relevance of true-motion vectors

Simplifications

1) Reduced candidate set

77

With 8 prediction and 1 update: 9 candidates

H-pos

V-pos

y-Y y y+Y y+2Y x-2X x-X x x+X x+2X

S

_a

T

a

Current block

Block in current field

Block in previous field

S

_c

S

_b

S

d

T

d

T

b

T

c

78

3DRS, 4 candidates are enough (including 1 update)

H-pos

V-pos

S

a

T

Current block

Block in current field

Block in previous field

S

_b

(14)

79

Y-estimator, advantage for pipe-lining

H-pos

V-pos

S

_a

T

Current block

Block in current field

Block in previous field

S

_b

Effect of candidate reduction

M2SE: 21.5

S: 2.8

M2SE: 26.0

S: 1.7

M2SE: 23.3

S: 2.6

technische universiteit eindhoven 81 Update Generator Prediction memory Mod p count Look Up Table Update Best vector selection Current picture Previous picture

0

U(X,n) Nbl D(X,n) D(x,n)

Block diagram of Y-estimator; Simple hardware

Simplifications

1) Reduced resolution for ME

83

D(x,n)

ME with reduced resolution compared to application

Application, like

De-interlacing, PRC, etc.

Down-scale

video signal

Up-scale

motion

vectors

Motion

estimation

on reduced

video

input

output

84

Block-hopping

(15)

Chosen candidates

Spatial

Temporal

Update

• In many cases the

spatial prediction

(SP) is good.

• Save calculations

on the average by

checking the other

candidates only if

SP error is above Th

Block-hopping

Calculate all SADs (grey blocks are skipped)

87

Block hopping; optimal resource usage

Calc. SAD

of SP

compare

Th

MUX

Assign

SP

Assign

best D

Calc. all

SADs

Vector

memory

s

Calculate Resource Usage/field Adapt threshold

5LIN0

Video processing

G. de Haan

89

Sophis-tications

90

Iterating more than once on an image pair

Effect of iterations 0 50 100 150 200 250 300 1 2 3 4 5 6 7 8 9 10 M2SE 100 x smoothness

10 times

Once, 1

st

image

Remark 1: If

estimating in the output domain

(100Hz): 2

iterations on video and 4 iterations on film material!

Remark 2:

Effect mainly shows in 1

st

image after scene change:

• 1 iteration, 10

th

_frame:

_{M2SE: 29, Smoothness: 2.8}

• 10 iterations, 10

th

frame:

M2SE: 28, Smoothness: 3.5

(16)

Block-erosion

technische universiteit eindhoven 92 Update Generator Prediction memory Mod p count Look Up Table Update Best vector

selection _erosionBlock

Current picture Previous picture

0

U(X,n) Nbl D(X,n) D(x,n)

Block diagram of Y-estimator; Simple hardware

Block erosion

V1V2 V3V4 R L D U R L D U C Median V1V2 V3V4 R L D U R L D U C Median V1V2 V3V4 R L D U R L D U C Median V1V2 V3V4 R L D U R L D U C Median

No BE

1 step BE

2 step BE

3 step BE

The effect of block erosion

95

Advanced

scanning

96

3-Dimensional Recursive Search (3DRS)

Reverse scan

(17)

Parametric

motion models

Global motion estimation

• Simple parametric motion model:

•

• p

1

and

p

2

describe pan and tilt

• p

3

and p

4

describe zoom

• p

5

and

p

6

describe rotation

...

)

(

)

(

)

(

)

,

(

...

)

(

)

(

)

(

)

,

(

6 4 2 5 3 1









x

n

p

y

n

p

n

p

n

x

D

y

n

p

x

n

p

n

p

n

x

D

y x



99

Sample vector field to calculate model parameters

Motion model with 4 parameters can be calculated from any 2 independent sample vectors So, in total from these 9 vectors 18 models can be estimated

100

Derive robust background model from sample vectors

p

1

= median{

p

11 ,

p

21 ,

p

31 ,………

p

181

}

p

2

= median{

p

12 ,

p

22 ,

p

32 ,………

p

182

}

p

3

= median{

p

13 ,

p

23 ,

p

33 ,………

p

183

}

p

4

= median{

p

14 ,

p

24 ,

p

34 ,………

p

184

}

Take median of all estimated parameters to eliminate outliers:

101 Update vector generator Prediction memory Mod p counter Look up table update Best vector selection _erosionBlock

Current Current picture picture Previous Previous picture picture

)

,

(

X

n

U



bl

N

)

,

(

x

n

D



)

,

(

X

n

D



>

S am p le ve ct o rs calculate local candidates micro processor calculates parameters

_,

_..

, 2 1

P

0 

Extra candidate from parametric motion model (SAA4992)

102

Effect of extra candidates from parametric model

Clearly, the effect depends on the settings of the candidate’s penalty!

Without parametric candidate

With parametric candidate

(18)

Motion

estimation and

occlusion

104

The basic block matching concept

V-position H-position Search area Candidate vector Reference block 8 x 8 pixels n-1 n Picture number

105

How to estimate

motion estimation in occlusion areas?

n-

1 n

Information not available in

previous picture

Ambiguities due to uncovering

Preference for FG-vector in uncovered areas

Time

n

n-1

P

o

s

it

io

n

?

107

How to estimate

motion estimation in occlusion areas?

n-

1 n

Information not available in

previous picture

Information not available in

next picture

108

Motion estimation problem in occlusion areas

• Observations:

•

Foreground:

• Matches always, i.e. in previous and in next picture

•

Background:

• In case of covering all background will match in previous picture

• In case of uncovering all background will match in next picture

• Conclusion:

•

Switch between “forward” and “backward” motion estimation to prevent ambiguities

(19)

109

Solution:

In covering areas “forward” estimation

V-position H-position Search area Candidate vector Reference block 8 x 8 pixels n-1 n Picture number

110

Solution:

In uncovering areas “backward” estimation

V-position H-position Search area Candidate vector Reference block 8 x 8 pixels n-1 n Picture number

111

Unambiguous motion vectors

for original images

Time

n

n-1

n+1

P

o

s

it

io

n

backward

forward

Look for correspondences in BOTH neighbouring images, select prediction with the highest correlation

112

Comparison 2 frame and 3 frame motion estimation

3 frame ME

2 frame ME

113

Global motion

estimation

114

Projection based global motion estimation

• Algorithm:

• Accumulate luminance over all lines

• Accumulate luminance over all collumns

• Determine global H- and V- motion based on these

projections

Demo

Samsung ME

(20)

115

Projection based global motion estimation

• Global motion: Minimum SAD of projection current

and previous image

i

F(i,k)

i

F(i,k+1)

2v

DEMO

Global ME

116

Success and failure of the projection based global ME

Conclusions

• Motion estimators for scan rate conversion differ from

estimators for coding, due to additional

true-motion

constraint

• True motion results from constraints like

spatial and

temporal consistency

• 3 options: better criterion, post-processing, pre-selection

• Pre-selection options

• Hierarchical approach

(e.g. Phase Plane Correlation.)

• Recursive approach

(3-D RS)

Conclusions

• Picture rate conversion

requires very consistent but not

necessarily very accurate motion vectors (integer

resolution sufficient), the range should be at least +/-16

pixels

• De-interlacing

requires very accurate motion vectors (at

least 1/4 pixel) . For larger vectors the accuracy is less

important

119

Prepare yourself for the exam…

• Last week:

•

Chapter 8

• Today:

•

Chapter 10 (SKIP: Object-based ME)

• I recommend you read the text

•

Book available at Pt9:24

• And try the exercises in the book:

•

Chapter2, 3, 4, 7. 8, 9

•

Chapter 10, skip 10.6

•

You have to download VidProc (www.ics.ele.tue.nl/~dehaan/) •Send me e-mail for password ([email protected])