Image Reconstruction Research on Color Filter Array

(1)

Procedia

Engineering

www.elsevier.com/locate/procedia

2012 International Workshop on Information and Electronics Engineering (IWIEE)

Image Reconstruction Research on Color Filter Array

Qichuan Tian

a

*

,b

,Xiaofei Yang

c

, Lanfang Zhang

d

, Yu Yang

e

a,c,e_{College of Electronic and Information Engineering,Taiyuan University of Science and Technology, Taiyuan 030024, China} b_{College of Science, Tianjin Polytechnic University,Tianjin 300387, China}

d_{College of Physical Education,Taiyuan University of Science and Technology, Taiyuan 030024, China}

Abstract

According to the physical structure of color image sensor in camera, cameras acquire images using image sensors overlaid with a color filter array (CFA) from different channels filters, so we can only achieve a single color component at each pixel position. In order to reconstruct a color image, color demosaicing is required to reconstruct the other two color components. General interpolation method may blur the image edge and introduce visible artifacts near edges. An image reconstruction algorithm based on adaptive region demosaicing is put forward in Bayer format to reduce the color artifacts. Experiment results show that the algorithm can improve the image quality and PSNR, sharpen texture and edge of the image and enhance image quality.

Keywords: Color Filter Array; Interpolation Algorithm; Color Reconstruction; Demosaicing

1. Introduction

Usually, a full-color image is composed of three-color planes according to the three-primary-color theory. In order to reduce the cost and the complexity, many cameras use a single sensor covered with a color filter array (CFA). In the CFA-based sensor configuration, only one color is measured at each pixel and the missing two colors are estimated. The estimation process is known as color demosaicing. Different CFA can achieve different information, so there are many reconstruction methods according to different CFA, the Bayer CFA is most popular CFA used nowadays [1].

Here, we compute green components according to the gradient operators and the extent of the texture,

* Corresponding author. Tel.: +86-351-699-8060; fax: +86-351-699-8060.

E-mail address: [email protected].

Open access under CC BY-NC-ND license.

(2)

and then the missing red components and blue components are estimated based on the interpolated green components and the model of color differences. Its experimental performance and comparisons with other methods show our method has good performance for reconstructing high quality image from CFA.

2. Existing image reconstruction algorithms

Human Visual System (HVS) is sensitive to the texture and the edges present in the images and non-adaptive color interpolation algorithm often fail to around the texture and the edges since they are not able to detect the texture and the edges. The earliest proposed techniques were based on well-known interpolation methods for images, but these methods can’t avoid edge blurring and color distortion effect. Based on Mondriaan model, literature [2] proposed a hue-smooth interpolation algorithm transformation, however, the color artifact exists in the abrupt region of green components. In the literature [3][4][5], the edge-based interpolation algorithms are proposed, at first calculated for each pixel along the gradient of different directions, and then gradients can be used to determine the final value of the different interpolation direction. Hamilton and Adams begin by using edge-directed interpolation for the green channel in [6][7]. In [8], an adaptive filter method is proposed under the observations that the high frequencies are similar across three-color components. Literature [9] presents that missing green samples are estimated based on the variances of the color differences along different edge directions firstly and then the missing red and blue components are estimated based on the interpolated green plane. In [10], it is explained that an object of constant color will have a constant color difference even though lighting variations may change the measured values. CFA demosaicing is formulated as a problem of reconstructing correlated signals from their downsampled versions with an opposite phase [11]. A way to reconstruct a full three-color representation of color images by estimating the missing pixel components in each color plane is called a demosaicing algorithm [12].

To protect the interpolation along the edge, at first, we can define two gradients, one in horizontal direction, the other in vertical direction, then determine the appropriate interpolating direction. In Fig. 1, a pixel at location

( )

,i j

in the CFA is represented by either

(

R

_{i j}_,

,

g

_{i j}_,

,

b

_{i j}_,

)

、

(

r G b

_{i j}_,

,

_{i j}_,

,

_{i j}_,

)

or

(

r g

i j,

,

i j,

,

B

i j,

)

，where

R

i j, 、

G

i j, and

B

i j, denote the known red, green and blue components,

and

r

_{i j}_, ,g_{i j}_, and

b

_{i j}_, denote the unknown components in the CFA.

Fig. 1. Four 5×5 regions of Bayer CFA pattern: (a) to (d) correspond to CFA at location (i,j)

A Bayer CFA image consists of 50% green, 25% red, and 25% blue samples. This particular arrangement stems from the fact that the sensitivity of the HVS to the luminance (green) is dominated by the green spectrum. In Fig.1 (a), we can compute the gradients and then estimate the missing pixel’s color at location

( )

i j

,

as formula (1).

(3)

(

)

(

)

(

)

, , 1 , 1 , 1, 1, , , 1 , 1 1, 1, 1 1 1 or or 2 2 4 i j i j i j i j i j i j i j i j i j i j i j g = G − +G + g = G− +G+ g = G − +G + +G− +G+ ( 1 )

After the green channel interpolation is performed, the red/blue channel can be estimated from color difference (the inter-channel correlation). This is based on the assumption that the hue does not change abruptly between neighboring pixels locations. The red channel information at location

( )

i j

,

pixel in Fig.1 (b), (c) and (d) can be estimated as formula (2), (3) and (4).

(

)

, , , , 1 1

1

4

i j i j i m j n i m j n m n

r

g

R

₊ ₊

g

₊ ₊ =± =±

=

+

∑ ∑

−

(2)

(

, 1 , 1 , 1 , 1

)

, ,

2

i j i j i j i j i j i j

R

g

R

g

r

=

G

+

−

+

−

+ (3)

(

1, 1, 1, 1,

)

, ,

2

i j i j i j i j i j i j

R

g

R

g

r

=

G

+

−

+

−

+ (4)

3. Image reconstruction algorithm according to CFA

Having a full-resolution green channel facilitates, we can reconstruct red channel and blue channel. Significant effort has been devoted to improve the accuracy of green channel. First of all, the luminance (green) is interpolated using variable of color gradients, and then the chrominance (red and blue) are estimated according to the recovered green channel from the hue-red difference and hue-blue difference.

3.1. Interpolating missing green components

The Bayer pattern measures the green image on a quincunx grid and the red and blue images on rectangular grids. The green channel is measured at a higher sampling rate than the other two, so the green channel contains more detail information. In Fig.1 (a), the proposed algorithm computes the gradient terms in every direction, and then we can decide how to calculate

g

_{i j}_, . For example, if the direction is horizontal gradients, then ,

(

1, 1,

) (

, 2, 2,

)

2

2 4

i j i j i j i j i j

i j

G G R R R

g = − + + + − − − + ; if the direction is the

vertical gradients, then ,

(

, 1 , 1

) (

, , 2 , 2

)

2 2 4 i j i j i j i j i j i j G G R R R g = − + + + − − − + .

We note that the missing green components are estimated in a raster scan fashion, once the missing green component is interpolated, the same process is performed for estimating the next missing green component in a raster scan manner.

3.2. Interpolating missing red /blue component at green sampling positions

One commonly used assumption in demosaicing is that the hue within object in an image is constant. This perfect inter-channel correlation assumption is formulated such that the color differences within objects are constant. Fig.1(c) and (d) shows the two possible cases where a green CFA sample is located at the center of a 5×5 block. The missing components of the center in Fig. 1(c) can be obtained by formula (3) and (5), the missing components of the center in Fig. 1(d) can be obtained by formula (4) and (6).

(4)

(

1, 1, 1, 1,

)

, , ₂ i j i j i j i j i j i j B g B g b =G + − − − + + − + (5)

(

, 1 , 1 , 1 , 1

)

, , ₂ i j i j i j i j i j i j B g B g b =G + − − − + + − + ( 6 )

3.3. Interpolating missing blue/red components at red/blue sampling positions

The missing blue/red components at the red/blue sampling positions are interpolated according to the assumption of color difference. In Fig.1 (a) and (b), the center missing blue sample is interpolated by formula (7) and (2).

(

)

, , , , 1 1 1 4 i j i j i m j n i m j n m n b g B+ + g+ + =± =± = +

∑ ∑

− ( 7 )

4. Experiments and conclutions

Experiment data comes from Kodak PhotoCD. Firstly, we simulate sampling to achieve CFA from full-color image, and then we reconstruct the Bayer format image using our algorithm and some existing algorithms. We report experimental results of our comparative study among five selective reconstruction algorithms. The PSNR are calculated as the objective measures for comparing the algorithms. Table 1 includes the performance comparison of five reconstruction algorithms. It can be observed that the proposed algorithm achieves the better PSNR performance. From Reconstruction image of Fig.2 (a) shown in Fig.3, it was found that the proposed algorithm could handle fine texture patterns and edge well.

(a) (b) (c) (d) (e) (f) Fig. 2. Original images:(a) to (f) correspond to different test images

(a) (b) (c) (d) (e) (f) Fig. 3. Reconstruction image of the part of Fig.2 (a):(a) is an original image, (b) to (f), left to right correspond to the different methods, they are Bilinear Interpolation, Edge Sensing Interpolation, Smooth Hue Interpolation, Linear Interpolation and Our algorithm.

Experimental results show our image reconstruction algorithm for CFA makes full use of the color gradients variance of the pixels in a local region to estimate the interpolation direction for interpolating the missing green samples, and then the missing red components and blue components are estimated based on the interpolated green components and the model of color differences. A high-resolution image in details and textures can be captured based on CFA and then reconstructed.

(5)

Table 1. The PSNR comparison of five different reconstruction algorithms on six images

Reconstruction Image

Original Image Bilinear

Interpolation Edge Sensing Interpolation InterpolationSmooth Hue InterpolationLinear Our algorithm

Fig.2 (a) 26.17 31.06 28.60 31.84 35.81 Fig.2 (b) 33.20 38.13 35.04 38.89 38.90 Fig.2 (c) 31.23 35.44 32.22 36.10 36.64 Fig.2 (d) 23.61 29.82 26.41 29.88 34.37 Fig.2 (e) 30.50 34.22 32.64 35.26 35.85 Fig.2 (f) 28.54 32.68 30.84 33.75 35.88 Acknowledgment

The Nature Science Foundation of Shanxi Province (No.2008011030), Research Project Supported by Shanxi Scholarship Council of China (No.2011-75), Taiyuan College Innovation and Entrepreneurial Talent Project (No.110148082), the College Students Innovation and Entrepreneurship Project of Shanxi Province (No.2011245), and the UIT Project of Taiyuan University of Science and Technology (No.XJ2010040) support this work.

References

[1] Qi LI, Zhi-hai XU, Hua-jun FENG. Study on new interpolation algorithm for pixel color of CCD in digital camera.

Opto-Eletronic Engineering; 2002,29(3): 68-71.

[2] J. E. Adams and J. F. Hamilton Jr. Adaptive color plane interpolation in single color electronic camera. U. S. Patent; 1996, 5 506 619.

[3] J. E. Adams. Design of practical color filter array interpolation algorithms for digital cameras. Proceedings of SPIE; 1997,3028: 117-125.

[4] E. Chang, S. Cheung and D. Y. Pan. Color filter array recovery using a threshold-based variable number of gradients.

Proceedings of SPIE;1999,3650:36-43.

[5] R. Ramanath and W. E. Snyder. Adaptive demosaicking. Journal of Electronic Imaging; 2003,12(4): 633-642.

[6] Nai-Xiang Lian, Lanlan Chang and Yap-Peng Tan. Adaptive filtering for color filter array demosaicking. IEEE Trans. on

Image Processing; 2007,10:2516-2532.

[7] King-Hong Chung and Yuk-Hee Chan. Color demosaicing using variance of color differences member. IEEE transactions

on image processing; 2006,15(10):2947-2949.

[8] R. Kimmel. Demosaicing: image reconstruction from CCD samples. IEEE Tran. Image Processing; 1999,8(9):1221-1228. [9] B.K. Gunturk, J. Glotzbach, Y. Altunbasak, R.W. Schafer, and R.M. Mersereau. Demosaicking: color filter array interpolation. IEEE Signal Processing magazine; 2005,22:44-54.

[10] W. Lu and Y. p. Tan. Color filter array demosaicing: new method and performance measures. IEEE Transactions on

Signal Processing;2003,12(10):1194-1210.

[11] X. Wu and N. Zhang. Primary-consistent soft-decision color demosaicking for digital cameras. IEEE Transactions on

Signal Processing; 2004,13(9):1263-1274.

[12] K. Hirakawa and T.W. Parks. Adaptive homogeneity-directed demosaicing algorithm. IEEE Transactions on Signal