Digital Image Processing

(1)

Image Restoration and Reconstruction (Linear Restoration Methods)

Ch i t h Nik

University of Ioannina - Department of Computer Science

Christophoros Nikou [email protected]

2 Contents

In this lecture we will look at linear image restoration techniquesq

– Differentiation of matrices and vectors – Linear space invariant degradation – Restoration in absence of noise

• Inverse filter

• Pseudo inverse filter

C. Nikou – Digital Image Processing (E12)

• Pseudo-inverse filter

– Restoration in presence of noise

• Inverse filter

• Wiener filter

• Constrained least squares filter

(2)

3 Differentiation of Matrices and Vectors

Notation:

A is a MxN matrix with elements a_ij A is a MxN matrix with elements a_ij. x is a Nx1 vector with elements x_i. f(x) is a scalar function of vector x.

g(x) is a Mx1 vector valued function of vector x.

4 Differentiation of Matrices and Vectors (cont...)

Scalar derivative of a matrix.

A is a MxN matrix with elements a_ij A is a MxN matrix with elements a_ij.

1

11 a N

a

t t

∂

⎛ ∂ ⎞

⎜ ∂ ∂ ⎟

⎜ ⎟

∂A = ⎜ ⎟

…

1 MN

M

t a a

t t

⎜ ⎟

∂ ⎜⎜ ∂ ∂ ⎟⎟

∂ ∂

⎝ ⎠

(3)

Vector derivative of a function (gradient).

x is a Nx1 vector with elements x_i x is a Nx1 vector with elements x_i. f(x) is a scalar function of vector x.

f f f T

f ⎛ ⎞

∂ = ∇ = ⎜ ∂ … ∂ ⎟

1 N

f x x

∇ ⎜ ⎟

∂x ^x ⎝ ∂ … ∂ ⎠

Vector derivative of a vector (Jacobian):

x is a Nx1 vector with elements x_i x is a Nx1 vector with elements x_i.

g(x) is a Mx1 vector valued function of vector x.

1 1

1 N

g g

x x

∂ ∂

⎛ ⎞

⎜ ∂ ∂ ⎟

⎜ ⎟

∂g ⎜ ⎟

…

1

M M

N

g g

x x

∂ =⎜ ⎟

∂ ⎜⎜∂ ∂ ⎟⎟

⎜ ∂ ∂ ⎟

⎝ ⎠

g x

(4)

Some useful derivatives.

x is a Nx1 vector with elements x_i x is a Nx1 vector with elements x_i. b is a Nx1 vector with elements b_i.

( )

^T

∂ =

∂ b x b x

It is the derivative of the scalar valued function b^Tx with respect to vector x.

x is a Nx1 vector with elements x_i x is a Nx1 vector with elements x_i. A is a MxN matrix with elements a_ij.

If A is symmetric:

(

^T

) (

^T

)

∂ = +

∂ x Ax A A x

x If A is symmetric:

(

^T

)

²

∂ =

∂ x Ax Ax

x

(5)

x is a Nx1 vector with elements x_i x is a Nx1 vector with elements x_i. b is a Nx1 vector with elements b_i. A is a MxN matrix with elements a_ij.

( )

2 2 ^T

∂ Ax b+ A Ax b+

It may be proved using the previous properties.

( )

+ = 2 +

∂ Ax b A Ax b

x

10 Linear, Position-Invariant Degradation

We now consider a degraded image to be modelled by:y

where h(x, y) is the impulse response of the degradation function ( i.e. point spread function blurring the image)

( , ) ( , ) ( , ) ( , ) g x y = h x y ∗ f x y + η x y

blurring the image).

The convolution implies that the degradation mechanism is linear and position invariant (it depends only on image values and not on location).

(6)

11 Linear, Position-Invariant Degradation (cont...)

In the Fourier domain:

where multiplication is element-wise.

In matrix-vector form:

( , ) ( , ) ( , ) ( , ) G k l = H k l F k l + N k l

where H is a doubly block circulant matrix and f,g, and η are vectors (lexicographic ordering).

g = Hf + η

12 Linear, Position-Invariant Degradation (cont...) ( , ) ( , ) ( , ) ( , )

g x y = h x y ∗ f x y + η x y

If the degradation function is unknown the g = Hf + η

( , ) ( , ) ( , ) ( , ) G k l = H k l F k l + N k l

If the degradation function is unknown the problem of simultaneously recovering f(x,y) and h(x,y) is called blind deconvolution.

(7)

13 Linear Restoration

Using the imaging system g = Hf + η

we want to estimate the true image from the degraded observation with known

degradation H.

A linear method applies an operator (a g = Hf + η

pp p (

matrix) P to the observation g to estimate the unobserved noise-free image f:

ˆf = Pg

14 Restoration in Absence of Noise The Inverse Filter

When there is no noise:

g = Hf

an obvious solution would be to use the inverse filter:

yielding

g = Hf

P = H-1

yielding

ˆ ^-1 ^-1

f = Pg = H g = H Hf = f

(8)

15 Restoration in Absence of Noise The Inverse Filter (cont...)

ˆ ^-1 f = H g

For a NxN image, H is a N²xN² matrix!

To tackle the problem we transform it to the Fourier domain.

H is doubly block circulant and therefore it

H is doubly block circulant and therefore it may be diagonalized by the 2D DFT matrix W:

H = W ΛW-1

where

H = W ΛW-1

where Therefore:

{ (1,1),..., ( ,1), (1, 2),... ( , )}

diag H H N H H N N

= Λ

ˆf = Pg ⇔f = H gˆ ^-1 ⇔ f = (W ΛW) gˆ ^-1 ^-1

⇔ f = W Λ Wgˆ ^-1 ^-1 ⇔ Wf = WW Λ Wgˆ ^-1 ^-1

⇔ F = Λ Gˆ ^-1

(9)

Which is the vectorized form of the DFT of the image:g

Take the inverse DFT and obtain f(m,n).

ˆ ^-1

F = Λ G ˆ ( , ) ( , ) ( , ) G k l F k l

H k l

⇔ =

Take the inverse DFT and obtain f(m,n).

Problem: what happens if H(k,l) has zero values?

Cannot perform inverse filtering!

18 Restoration in Absence of Noise The Pseudo-inverse Filter

A solution is to set:

( , )

, ( , ) 0 ˆ( , ) ( , )

0 , ( , ) 0

G k l

H k l H k l

F k l

H k l

⎧ ≠

⎪⎨

⎪ =

⎩

=

which is a type of pseudo-inversion.

Notice that the signal cannot be restored at locations where H(k,l)=0.

(10)

19 Restoration in Absence of Noise The Pseudo-inverse Filter (cont...)

A pseudo-inverse filter also arises by the unconstrained least squares approach.q pp

Find the image f, that, when it is blurred by H, it will provide an observation as close as possible to g, i.e. It minimizes the distance

between Hf and g.

20 Restoration in Absence of Noise The Pseudo-inverse Filter (cont...)

This distance is expressed by the norm:

( ) 2

J f Hf

{ }

min J( )

f f ^⇔ _∂^∂_f

(

^{Hf - g} ²

)

⁼⁰

( )

T T T

J 0

⇔ ∂ =

∂f

J( )f = Hf - g

( )

2 ^T 0

⇔ H Hf - g = ⇔ 2H Hf^T =2H g^T

(

^T

)

⁻¹ ^T

⇔ =f H H H g

(11)

21 Restoration in Presence of Noise The Inverse Filter

Recall the imaging model with spatially invariant degradation and noiseg

Hf

( , ) ( , ) ( , ) ( , ) g x y = h x y ∗ f x y + η x y

( , ) ( , ) ( , ) ( , ) G k l = H k l F k l + N k l

g = Hf + η

22 Restoration in Presence of Noise The Inverse Filter (cont...)

Applying the inverse filter in the Fourier domain:

( , ) ( , ) ( , ) ( , ) G k l = H k l F k l + N k l

( , ) ˆ ( , ) ( , )

( , ) N k l F k l F k l

H k l

⇔ = +

Even if we know H(k,l) we cannot recover F(k,l) due to the second term.

If H(k,l) has small values the second term dominates (it goes to infinity if H(k,l)=0!).

( , )k l

(12)

One approach to get around the problem is to limit the ratio G(k,l) / H(k,l) to frequencies ( , ) ( , ) q near the origin that have lower probability of being zero.

We know that H(0,0) is usually the highest value of the DFT.

Thus, by limiting the analysis to frequencies near the origin we reduce the probability of encountering zero values.

Blurring degradation

(13)

Inverse filter with cut-off

26 Restoration in Presence of Noise Wiener Filter

So far we assumed nothing about the statistical properties of the image and noise.

We now consider image and noise as random variables and the objective is to find an estimate of the uncorrupted image f such that the mean square error between the estimate and the image is

minimized:

{ }

where E[x] is the expected value of vector x.

{

²

}

ˆ

minf E ⎡⎣(f f−ˆ) ⎤⎦

(14)

27 Restoration in Presence of Noise Wiener Filter (cont...)

Recall also the definition of the correlation matrix between two vectors x and y:y

[ ] [ ] [ ]

1 1 1 2 1

1 2

N T

N N N N

E x y E x y E x y E

E x y E x y E x y

⎡ ⎤

⎢ ⎥

⎡ ⎤

= ⎣ ⎦ ⎢= ⎥

⎢ ⎥

⎣ ⎦

Rxy xy

…

We assume that the image and the noise are uncorrelated:

= =

ηf fη

R R 0

We are looking for the best estimate:

{

^⎡ ^ˆ ²^⎤

}

Let’s confine our estimate to be obtainable by a linear operator on the observation:

{

²

}

minfˆ E ⎡⎣(f f− ) ⎤⎦

ˆf Pg

and the goal is to find the best matrix P.

= f Pg

(15)

2 2

ˆ ˆ 2 ˆ

( ) ( )

J f = E⎡⎣ f f− ⎤⎦ = E⎡⎢⎣ f f− ⎤⎥⎦ = E⎡⎣ f Pg− ⎤⎦ Denoting by the n-th row of P:

( ) ( )

J f = E⎡⎣ f f ⎤⎦ = E⎢⎣ f f ⎥⎦ = E ⎡⎣ f Pg ⎤⎦

( )

²

( )

²

ˆ ⎡

∑

^T ⎤

∑

⎡ ^T ⎤

T

pn

( )

²

( )

²

( ) _n ^T_n _n ^T_n

n n

J ^f = E ⎡⎢⎣

∑

^f −^{p g} ⎤⎥⎦ =

∑

E ⎡⎢⎣ ^f −^{p g} ⎤⎥⎦

( )( )

( )ˆ _n ^T_n _n ^T_n ^T J^{( )}^f =

∑

E ⎡⎢⎣

(

^fⁿ −^{p g f}ⁿ

)(

ⁿ −^{p g}ⁿ

)

⎤⎥⎦

n ⎢⎣ ⎥⎦

∑

^{p g} ^{p g}

T T T T T T

n n n n n n n n

n

E ⎡ ⎤

=

∑

⎣^{f f} −^{p gf} −^{f g p} +^{p gg p} ⎦

T T T T T T

E⎡ ⎤ E⎡ ⎤ E⎡ ⎤ E⎡ ⎤

=∑ ⎣^{f f} ⎦−^p ⎣^gf ⎦− ⎣^{f g p}⎦ +^p ⎣^{gg p}⎦

n n n n n n n n

n

E⎡⎣ ⎤⎦ E⎡⎣ ⎤⎦ E⎡⎣ ⎤⎦ + E⎡⎣ ⎤⎦

∑ ^{f f} ^p ^gf ^{f g p} ^p ^{gg p}

n n 2 n

T T

n n n

n

=∑^R^{f f} − ^{p R}^gf +^{p R p}^gg

(16)

We can now minimize the sum with respect to each term:

(

n n ² n

)

⁰

T T

n n n

n

∂ − + =

∂ R^{f f} p R^gf p R p^gg p

2 2 0

n n

⇔ − R_gf + R p_gg = ¹

n n

⇔p =R R−_gg _gf

C. Nikou – Digital Image Processing (E12) 1

n

T n

⇔p =R R_{f g} −_gg

Assembling the rows together:

P R R 1

We have to compute the two matrices:

−1

= _fg _gg P R R

( )( )

T T

E⎡ ⎤ E⎡ ⎤

= ⎣ ⎦ = ⎣ + + ⎦

Rgg gg Hf η Hf η

T T T T T T

E ⎡ ⎤

= ⎣Hff H +Hfη +ηf H +η η⎦

T T

=HR H_ff +HR_fη +R H_ηf +R_ηη

(17)

Assumming noise is uncorrelated with image:

R HR HT R

Also,

( ) ...

T T T

E⎡ ⎤ E⎡ ⎤

= ⎣ ⎦ = ⎣ + ⎦ = =

fg ff

R fg f Hf η R H

= T +

gg ff ηη

R HR H R

Finally the matrix we are looking for is

( )

¹

1 T T −

= _fg −_gg = _ff _ff + _ηη

P R R R H HR H R

The estimated uncorrupted image is ˆf Pg ^f^ˆ ^{R H HR H}^T

(

^T ^R

)

⁻¹

which may be also expressed as

=

f Pg ^{⇔ =}f R H HR Hff ^T

(

ff ^T ⁺Rηη

)

g

(

¹ ¹

)

¹ ¹

ˆ = ^T ⁻_ηη + _ff⁻ ⁻ ^T ⁻_ηη

f H R H R H R g

This result is known as the Wiener filter or the minimum mean square error (MMSE) filter.

(

ηη

)

ηη

(18)

Special cases:

No blur (H=I g=f+η): ^f^ˆ ⁼ ^R

(

^R ⁺^R

)

⁻¹^g

No blur (H I, g f+η):

No noise (R_ηη=0, g=Ηf):

This is the inverse filter.

( )

= _ff _ff + _ηη

f R R R g

ˆ = ⁻1

f Η g

No blur, no noise (H=I, R_ηη=0):

Do nothing on the observation.

ˆ = f g

The size of the matrix to be inverted poses difficulties and Wiener filter is implemented in p the Fourier domain.

This occurs when H is doubly block circulant (represents convolution) and the image f and noise η are wide-sense stationary (w.s.s).

Definition of a w.s.s. signal:

1) E[f(m,n)]=μ, independent of m,n.

2) E[f(m,n)f(k,l)]=r(m-k,n-l), independent of location.

(19)

Reminder: the inverse DFT complex exponential matrix diagonalizes any circulant matrix:

The columns of W^-1 are the eigenvectors of any circulant matrix H.

The corresponding eigenvalues are the DFT values

−1

= _H

H W Λ W

The corresponding eigenvalues are the DFT values of the signal producing the circulant matrix.

Remember also that and that

T =

W W

( )

^W⁻¹ ^T ⁼ ^W⁻¹

We will employ the following relations:

−1

H W Λ W

If H is real:

= 1 _H

H W Λ W

(

¹

)

^T ¹

T = − _H = _H −

H W Λ W WΛ W

( ) (

^* ¹

)

^* ^* ^*

( )

¹ ^*

T T

( ) (

¹

)

^* ^*

( )

¹

T = T = _H − = _H −

H H WΛ W W Λ W

1 * 1 1 *

(N )

N

− −

= W Λ_H W W Λ W= _H

(20)

The Wiener solution is now transformed to the Fourier domain:

( ) ¹

ˆ = _ff ^T _ff ^T + _ηη ⁻ f R H HR H R g

1 1 * 1 1 1 * 1 1

( ⁻ )( ⁻ ) (⎡ ⁻ )( ⁻ )( ⁻ ) ( ⁻ )⎤⁻

= W Λ W W Λ W^ff ^H ⎣ W Λ W W Λ W W Λ W^H ^ff ^H + W Λ W^ηη ⎦ g

1 * 1 * 1

( ) ⁻

− ⎡ − ⎤

=W Λ Λ W W Λ Λ Λ^ff ^H ⎣ ^H ^ff ^H+Λ^ηη ⎦ g

Notice that the matrices are diagonal.

* * 1

ˆ ( )⁻

⇔Wf =Λ Λ Λ Λ Λ_ff _H _H _ff _H +Λ_ηη Wg

* * 1

ˆ = _ff _H( _H _ff _H + _ηη)⁻

Wf Λ Λ Λ Λ Λ Λ Wg ⇔

S (k l)=DFT(R (m n)) is the power spectrum

* 2

( , ) ( , )

( , ) ( , ) ( , )

ff ff

S k l H k l

F k l G k l

S k l H k l S_ηη k l

= +

S_ff(k,l)=DFT(R_ff(m,n)) is the power spectrum of the image f (m,n).

S_ηη(k,l)=DFT(R_ηη(m,n)) is the power spectrum of the noise η(m,n).

(21)

If S_ff(k,l) is not zero we may define the Signal to Noise Ratio in the frequency domain:q y

and the Wiener filter becomes:

( , ) SNR( , )

( , ) S k lff

k l = S_ηη k l

and the Wiener filter becomes:

*

2 -1

( , )

( , ) ( , )

( , ) SNR ( , ) H k l

F k l G k l

H k l k l

= +

A well known estimate of S_ff(k,l) is the

periodogram (the ML estimate of R_ffwhen f ia

p g ( _ff

assumed Gaussian):

In practice, as F(k,l) is unknown, we use 1 2

( , ) ( , )

S k lff F k l

= MN

p ( )

1 2

ˆ ( , )_ff ( , )

S k l G k l

= MN

(22)

Inverse Wiener

Degraded image

Noise variance one order of magnitude less.

Noise variance ten orders of magnitude less.

(23)

45 Restoration in Presence of Noise Constrained Least Squares Filter

When we do not have information on the power spectra the Wiener filter is not optimal.

Another idea is to introduce a smoothness term in our criterion.

We define smoothness by the quantity

where Q is a high pass filter operator, e.g. the Laplacian, (Q is a doubly block circulant matrix

Qf 2

p , (Q y

representing convolution by the Laplacian).

We look for smooth solutions minimizing Qf ²

46 Restoration in Presence of Noise Constrained Least Squares Filter (cont...) We have the following constrained least

squares (CLS) optimization problem:q ( ) p p Minimize subject to

yielding the Lagrange multiplier minimization of the function:

Qf 2 Hf =g

2 2

( , )

J f λ = Hf g− +λ Qf

Smoothness term Data fidelity term

(24)

47 Restoration in Presence of Noise Constrained Least Squares Filter (cont...)

2H Hf g^T( − +) 2λQ Qf^T =0 ( , ) 0

J λ

∂ =

∂ f

f ⇔

∂f

⇔ ^f ⁼

(

^{H H}^T ⁺^λ^{Q Q}^T

)

⁻¹^{H g}^T

Parameter λ controls the degree of smothness:

( ) ¹

0, ^T ^T

λ= f = H H ⁻ H g pseudo-inverse, ultra rough solution , 0

λ→ ∞ =f ultra smooth solution

48 Restoration in Presence of Noise Constrained Least Squares Filter (cont...) In the Fourier domain, the constrained least squares filter becomes:q

*

2 2

( , )

( , ) ( , )

( , ) ( , ) H k l

F k l G k l

H k l λ Q k l

= +

Keep always in mind to zero-pad the images properly.

(25)

49 Restoration in Presence of Noise Constrained Least Squares Filter (cont...)

Low noise: Wiener and CLS generate equal results.

High noise: CLS outperforms Wiener ifλ is properly selected.

It is easier to select the scalar value for λ than to approximate the SNR which is seldom constant.

50 Restoration Performance Measures

Original image f and restored image ^ˆf

Mean square error (MSE):

1 1 2

0 0

1 ˆ

MSE ( , ) ( , )

M N

m n

f m n f m n MN

− −

= =

⎡ ⎤

= ∑∑⎣ − ⎦

0 0

m n

(26)

51 Restoration Performance Measures (cont...)

The Signal to Noise Ratio (SNR) considers the difference between the two images as noise:

difference between the two images as noise:

1 1

2

0 0

1 1 2

ˆ ( , ) SNR

( ) ˆ( )

M N

m n

M N

f m n

f f

− −

= =

− −

= ⎡ ⎤

∑∑

0 0

( , ) ( , )

m n

f m n f m n

= =

⎡ − ⎤

⎣ ⎦

∑∑