Image Filtering and Image Frequencies

(1)

CS4501: Introduction to Computer Vision

Image Filtering and Image

Frequencies

(2)

• The Human Eye as a Camera

• Images as Matrices

• Images as Functions

• Image Processing

• Image Filtering

Last Class

(3)

• Image Filtering: Mean Filter

• Image Blurring

• Image Gradients: The Sobel Operator

• Image Frequencies

(4)

(5)

Basic Image Processing

(6)

Basic Image Processing

(7)

Color Images as Tensors

(8)

(9)

(10)

(11)

(12)

(13)

Image filtering: Convolution operator

(14)

(15)

Image filtering: Convolution operator

e.g. mean filter

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

(16)

Image filtering: Convolution operator

e.g. mean filter

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9

(17)

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(18)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(19)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(20)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 10 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(21)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(22)

0 10 20 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(23)

0 10 20 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

?

(24)

0 10 20 30 30 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

?

(25)

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10 0 20 40 60 60 60 40 20 0 30 60 90 90 90 60 30 0 30 50 80 80 90 60 30 0 30 50 80 80 90 60 30 0 20 30 50 50 60 40 20 10 20 30 30 30 30 20 10 10 10 10 0 0 0 0 0

[.,.]

h

[.,.]

f

Image filtering

1 1 1 1 1 1 1 1 1

]

,

[

g

×

(26)

What does it do?

• Replaces each pixel with an average of its

neighborhood

• Achieve smoothing effect (remove sharp features)

1 1 1 1 1 1 1 1 1

]

,

[

g

×

Box Filter

(27)

(28)

Image filtering: Convolution operator

Important filter: gaussian filter (gaussian blur)

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1/16 1/8 1/16 1/8 1/4 1/8 1/16 1/8 1/16

(29)

• Weight contributions of neighboring pixels by nearness 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5, s = 1

(30)

Image filtering: Convolution operator

e.g. gaussian filter (gaussian blur)

(31)

Practical matters

• What about near the edge?

• the filter window falls off the edge of the image • need to extrapolate

• methods:

• clip filter (black) • wrap around • copy edge

(32)

Practice with linear filters

0 0 0 0 1 0 0 0 0 Original

?

(33)

Practice with linear filters

0 0 0 0 1 0 0 0 0 Original _Filtered

(34)

Practice with linear filters

0 0 0 1 0 0 0 0 0 Original

?

(35)

Practice with linear filters

0 0 0 1 0 0 0 0 0

(36)

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 0 0 0 0 2 0 0 0 0

-

_?

(37)

Practice with linear filters

Original 1 1 1 1 1 1 1 1 1 0 0 0 0 2 0 0 0 0

-Sharpening filter

(38)

(39)

Key properties of linear filters

Linearity:

imfilter(I, f1 + f2) =

imfilter(I,f1) + imfilter(I,f2)

Shift invariance: same behavior regardless of pixel

location

(40)

–

Enhance images

• Denoise, resize, increase contrast, etc.

–

Extract information from images

• Texture, edges, distinctive points, etc.

–

Detect patterns

• Template matching

–

Deep Convolutional Networks

(41)

Image filtering: Convolution operator

Important Filter: Sobel operator

𝑘(𝑥, 𝑦)

𝑘(𝑥, 𝑦) =

1 0 -1

2 0 -2

(42)

Other filters

-1 0 1 -2 0 2 -1 0 1 Sobel

(43)

Other filters

-1 -2 -1 0 0 0 1 2 1 Sobel

(44)

Sobel operators are equivalent to 2D partial

derivatives of the image

• Vertical sobel operator – Partial derivative in X (width)

• Horizontal sobel operator – Partial derivative in Y (height)

• Can compute magnitude and phase at each location.

(45)

(46)

Sobel filters are (approximate) partial derivatives

of the image

𝜕𝑓(𝑥, 𝑦) 𝜕𝑥 = lim#→% 𝑓 𝑥 + ℎ, 𝑦 − 𝑓(𝑥, 𝑦) ℎ

𝑓(𝑥, 𝑦) be your input image, then the partial derivative is: Let

(47)

But digital images are not continuous, they are

discrete

Δ_&𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥, 𝑦]

𝑓[𝑥, 𝑦] be your input image, then the partial derivative is: Let

(48)

But digital images are not continuous, they are

discrete

Δ_&𝑓[𝑥, 𝑦] = 𝑓[𝑥 + 1, 𝑦] − 𝑓[𝑥, 𝑦]

𝑓[𝑥, 𝑦] be your input image, then the partial derivative is: Let

-1 1

(49)

Sobel Operators Smooth in Y and then

Differentiate in X

1 0 -1 k(x, y) = 1 2 1 * = 1 0 -1 2 0 -2 1 0 -1 Similarly to differentiate in Y

(50)

Variations

1 0 -1 1 0 -1 1 0 -1 Prewitt -1 -1 -1 0 0 0 1 1 1 Roberts 0 -1 1 0 -1 0 0 1 1 0 -1 -1 -1

(51)

(52)

Any function can be approximated by a

polynomial function

…if you let your polynomial have a high degree

…AND you can compute the derivatives of the original function easily. Taylor Series expansion

(53)

(54)

(55)

Jean Baptiste Joseph Fourier (1768-1830)

had crazy idea (1807):

Any univariate function can be

rewritten as a weighted sum of sines and cosines of different frequencies.

• Don’t believe it?

–Neither did Lagrange, Laplace, Poisson and other big wigs

–Not translated into English until 1878!

• But it’s (mostly) true!

...the manner in which the author arrives at these equations is not exempt of difficulties and...his analysis to integrate them still leaves something to

be desired on the score of generality and even rigour.

(56)

(57)

Discrete Fourier Transform

𝐹 𝑢 = 8 &'% ()* 𝑓 𝑥 cos −2𝜋 𝑥𝑢 𝑁 + 𝑖 sin −2𝜋 𝑥𝑢 𝑁

(58)

Keep in mind Euler’s Equation

𝑒

!"

= cos 𝑥 + 𝑖 sin 𝑥

(59)

Discrete Fourier Transform

𝐹 𝑢 = 8 ()* 𝑓 𝑥 exp −2𝜋𝑖 𝑥𝑢 𝑁 𝐹 𝑢 = 8 &'% ()* 𝑓 𝑥 cos −2𝜋 𝑥𝑢 𝑁 + 𝑖 sin −2𝜋 𝑥𝑢 𝑁

(60)

Inverse

Discrete Fourier

Transform

𝐹 𝑢 = 8 &'% ()* 𝑓 𝑥 exp −2𝜋𝑖 𝑥𝑢 𝑁 𝑓 𝑥 = 1 𝑁 8 "'% ()* 𝐹 𝑢 exp 2𝜋𝑖 𝑥𝑢 𝑁

Discrete

Fourier

Transform

(61)

More generally for images (2D DFT and iDFT)

𝐹 𝑢, 𝑣 = 8 &'% +)* 8 ,'% ()* 𝑓 𝑥, 𝑦 exp −2𝜋𝑖 𝑥𝑢 𝑀 + 𝑦𝑣 𝑁 𝑓 𝑥, 𝑦 = 1 𝑀𝑁 8 +)* 8 ()* 𝐹 𝑢, 𝑣 exp 2𝜋𝑖 𝑥𝑢 𝑀 + 𝑦𝑣 𝑁

(62)

(63)

Fourier Transform

• Fourier transform stores the magnitude and phase at each frequency

• Magnitude encodes how much signal there is at a particular frequency • Phase encodes spatial information (indirectly)

• For mathematical convenience, this is often notated in terms of real and complex numbers

2 2

₍

₎

)

(

w

I

w

R

A

=

±

+

)

(

)

(

tan

1

w

f

R

I

-=

Amplitude: Phase:

(64)

(65)

(66)

(67)

The Convolution Theorem

• The Fourier transform of the convolution of two

functions is the product of their Fourier transforms

• Convolution in spatial domain is equivalent to

multiplication in frequency domain!

]

F[

]

F[

]

F[

g

*

h

=

g

h

]]

F[

]

[F[

F

*

h

1

g

h

g

=

(68)

(69)

(70)

(71)

Why Frequency domain?

• Because the Discrete Fourier Transform can be computed fast using

the Fast Fourier Transform FFT algorithm.

• Because the running time does not depend on the size of the kernel

matrix.

• However rarely used these days because most filters used in

Computer vision are 3x3, 5x5, e.g. relatively small.

(72)

Final Thoughts – JPEG Image Compression

• Small amount of information can recover almost the original image with some loss in resolution.

• Images are dominated by low frequency information. e.g. no need to store repeated pixels. • In practice JPEG uses a simpler

(73)

(74)