Brief Analysis of Image Signal Processing for Smart Phone

2016 International Conference on Computer, Mechatronics and Electronic Engineering (CMEE 2016) ISBN: 978-1-60595-406-6

Brief Analysis of Image Signal Processing for Smart Phone

Li-li CHEN, Run-ping HAN

and Yu-xiu BAO

School of Information Engineering, Beijing Institute Of Fashion Technology, Beijing, 100029, China

*Corresponding author

Keywords: ISP, CIS, Color interpolation, White balance, Color correction, Gamma correction.

Abstract. The imaging quality of camera phone is mainly determined by the lens, CMOS image sensor(CIS for short) and image signal processing(ISP for short),the algorithm of ISP is especially important because the thickness of the CIS is greatly restricted as cellphones tend to be thinner and lighter. The main function of ISP is to do the linearization, lens shading, bad pixel correction, color interpolation, AUTO white balance, color correction, gamma correction, de-noising and sharpening to make the digital images closer to the reality scene. In view of the importance of ISP in the imaging system of camera phone, deep research on it has theoretical significance and market value.

Introduction

With the popularity of smart phones and social networks, more and more people use camera phones to capture the scenery. Consequently, the ability of taking photos has become an important indicator to measure the performance of a mobile phone when buying them. Aiming at this requirement, the related software and hardware configuration have been greatly improved.

CIS

CIS and ISP are two important parts in the imaging system of camera phone, the working principle and the processing procedure of ISP are mainly introduced in this paper.

The module of the phone's camera is mainly composed of the lens and CIS. Some CIS chips integrate ISP module, for example, the OV5693 is issued by OmniVision as shown in Figure 1 It is mainly composed of Image Array(Pixel Matrix), Amplifier, AD Conversion, ISP, CSI Interface, etc. The ISP is used to deal with dark current correction, lens shading and spatial transformation. The CIS without integrating ISP outputs raw data which is used as input to the post-ISP directly. The post-ISP has a detailed introduction in the third part of this paper.

The photosensitive elements, photosensitive diode array, are covered with a layer of filter called color filter array(CFA for short) in the CIS module. Bayer pattern is most commonly used type of CFA as shown in Figure 2 Each photosensitive diode corresponds to a pixel that stores raw data, it is the basis for the subsequent ISP analysis. Raw data is the response value to the monochromatic light and retains the original data about the reality scene.

Figure 2. The Bayer CFA pattern.

ISP

Processing ability of ISP is required to be stronger and stronger for the restricted thickness of camera phone and higher imaging quality, therefore, ISP has been separated from the CIS as a separate module called post-ISP which could do more complex operations to make the digital images meet human visual system.

The post-ISP is divided in two types: the Integrated and the Independent. The Integrated refers to that the ISP module is integrated in the CPU, such as Qualcomm Snapdragon series, NVIDIA Tegra series; the Independent is a separate ISP chip which is issued by Fujitsu, Aptina and Altek, etc.

ISP reads raw data from CIS and do the dark current correction, linearization, lens shading, bad pixel correction, color interpolation, AUTO white balance, color correction, gamma correction, de-noising, sharpening and Figure 3 illustrates detailed process. The following is the introduction of ISP modules.

Figure 3. ISP processing flow chart. Dark Current Correction and Linearization

ISP deal with dark current firstly. Dark current refers to weak current output produced by the photosensitive diode without light intensity, this current is interference signal or noise and affects the image quality, hence it must be modified and the average value of non sensitive areas which are the rows and columns at the edge of pixel matrix is the corrected value.

The response of CIS to the external light intensity are not completely linear because of the cameras fabrication process and material limitations. In the nonlinear region color cast appears and affects white balance effect so that the linearization is needed, the corrected method is calibrate the offset of red and blue channels using the green channels as benchmark.

Shading Correction and Bad Pixel Correction

（1） attenuation, which means the middle area is brighter than the surrounding area in digital image, because the light goes into lens at different angles and the reception depends on pixel location. The light near lens center goes into lens at vertically directly and reactions are strongest, whereas the light far from center enters at a greater angle and the pixels only receive light component, which is why lens shading leads to the loss of light and the decrease of response values. Solving this problem is to compensate the brightness of darker areas to closer central areas.

Image sensor produces bad pixels inevitably in the manufacturing process. The damaged diode causes open circuit, 0 is stored and brings dark spots, on the contrary, short circuit caused 255 is stored and brings luminous spots. Bad pixel is judged by ISP when there has a larger distinction among the adjacent pixels and the correction is estimated by the same channels around. A compensation algorithm[5] is introduced in 3 * 5 matrix, as shown in Figure 4.

Figure 4. 3x5 pixel matrix.

In the Figure 4, G33 is judged as a bad pixel, the compensation formula is shown in the formula (1).

6 44 42 35 31 24 22

33 G G G G G G

G      

Color Interpolation

The purpose of color interpolation is to convert raw data into complete RGB. The RGB are the three primary colors which is red color, green color and blue color respectively in the colorimetry. The range of each primary color is 0 to 255 and any color can be mixed in varying proportions. Human can not notice the proportion directly but the brightness, hue and saturation to distinguish things in reality.

The edge-directed interpolation algorithm based on gradient-based is often used in the color interpolation. The algorithm for first-order differential is introduced in this paper, it solves the blurring across edges cased by the interpolation better. The algorithm firstly recover all of the G component in the Pixel Matrix, then modify the missing R and B components by the interpolated green component, take R66 as a example to calculate all of the R,G,B components as shown in Figure 5.

Horizontal gradient value:

67 65

G G

H 



(2) Vertical gradient value:

76 56

G G

V  

 (3)                         V H G G G G V H G G V H G G G , 4 76 56 67 65 , 2 76 56 , 2 67 65 66 (4) The algorithm Interpolate R in three cases:

Interpolate R65 when its left and right two positions of R are known

2 ) 66 66 ( ) 64 64 ( 65

65 G R G R G

R     

Interpolate R76 when its upper and lower two positions of R are known 2 ) 86 86 ( ) 66 66 ( 76

76 G R G R G

R     

(6) Interpolate R77 when the R on the diagonal direction is known

4 ) 68 68 ( ) 88 88 ( 4 ) 86 86 ( ) 66 66 ( 77

77 G R G R G R G R G

R         

(7) The algorithm also Interpolate B in three cases:

Interpolate B65 when its upper and lower two positions of B are known

2 ) 75 75 ( ) 55 55 ( 65

65 G B G B G

B     

(8) Interpolate B76 when its left and right two positions of B are known

2 ) 77 77 ( ) 75 75 ( 76

76 G B G B G

B     

(9) Interpolate B66 when the B on the diagonal direction are known

4 ) 57 57 ( ) 77 77 ( 4 ) 75 75 ( ) 55 55 ( 66

66 G B G B G B G B G

B         

(10) Using the above algorithm can get all the R and B components. Converting raw data into RGB is a prerequisite for white balance and subsequent treatment.

Figure 5. 5x5 pixel matrix. White Balance

For the same scene, the original image received by sensor has a phenomenon of color cast if using the various light sources. Image is partial red in the candle light which is lower chromatic temperature while partial blue in cloudy outdoor which is higher chromatic temperature. The human eye does not appear this phenomenon as human eyes vision system receives constant colors after itself automatic color calibration, accordingly, fixing responses is essential and this correction called white balance.

The white balance algorithm is mostly based on the Von Kries hypothesis and the gray world model assumption [6]. The realization process of Auto white balance can be divided into four steps.

Get Reference Points in the Standard Light Sources

Figure 6. Reference points in the standard light sources. Chromatic Temperature Judgment

Statistical module do weight according to the brightness and the R/G, B/G to judge current light source, there are six small steps.

The image is divided into m*n small areas.

The R/G and B/G of the small areas are collected separately, the result is shown in Figure 7. Obtain the efficient points that fall in the effective rang of standard light sources, Figure 8 illustrates the TL84 effective rang.

According to the reference points to classify the effective points and reckon the distance weight. Count brightness weight of the effective points.

Count the total weight, get current chromatic temperature.

Figure 7. Statistical points of R/G and B/G values.

Figure 8. The effective range of TL84 light source. Gain Calculation

The compensation coefficients Rgain, Ggain and Bgain are obtained according to the current chromatic temperature.

Chromatic Temperature Correction

                                                1 1 1 1 1 1 2 2 2 0 0 0 0 0 0 b g r b g r b g r b g r p p p k k k p p p D p p p (11) In formula (11), the (pr1,pg1,pb1) are the RGB in the E1(λ) light source which can be considered as

the data of the original image and the (pr2,pg2,pb2) in the E2(λ) light source can be considered as the

data of the image after calibration, D is a diagonal matrix, kr kg and kb are diagonal elements, kr is

Rgain, kg is Ggain and kb is Bgain.

Color Correction and Gamma Correction

Visual spectrum curve response between human and CIS is not the same, thus CIS curve needs to be revised in order to consistent with the color of human eye, which is called color correction. It is accomplished by a color adjustment matrix formula (12).

                                B G R CMA CMA CMA CMA CMA CMA CMA CMA CMA B G R 33 32 31 23 22 21 13 12 11 ' ' ' (12) In the formula (12), CMA11-CMA33 are correction factors, they are changed according to the brightness of the image and the light source. R, G and B are color components before the emendation and R’, G’, B’ are after.

The purpose for gamma correction is to compensate the different luminance response between the CIS and the human eye, it is mainly due to the existence of the self-adaptive behavior of human eye. In another words, the sensitivity of different brightness regions is dissimilar and the relationship of luminance response is not a linearity. The gamma correction formula (13) is an exponential operation.



x

y ₍₁₃₎

In the formula (13), γ is gamma coefficient, the X and Y are in the range of [0, 1], the different γ selectively enhance the contrast ratio of the low gray-level region or the high gray-level region, which means that γ determine the gray mapping mode between the input image and the output image.

If γ>1, the contrast ratio of high gray-level region is enhanced; if γ<1, the contrast ratio of low gray-level region is enhanced; if γ=1, the contrast ratio is a linearity and the input image is not changed.

The γ is generally about 0.4 in the ISP process to meet human perception. In the actual operation the gamma curve is vectorized because the exponential operation is too complex. The image after color correction and gamma correction effects as shown in Figure 9, 10, 11.

Figure 10. The image after the color correction.

Figure 11. The image after gamma correction.

The correction experiment above is tested in TL84 light source.

De-noising and Sharpening

Noises appear from image capturing to image decoding and cause the image to blur and to lose a lot of details, therefore image de-noising is a very important module in the image processing technology.

Removing noises and retaining details are contradictory in the filter processing because the aliasing in the frequency band, it become necessary to find a balance among them. Denoising method based on wavelet de-noising algorithm[1] is more effective and retain the boundaries and details better.

The function of sharpening is to enhance the edge and outline of the image, which means to enhance the gray level jump portion. The real reason for edge and outline blurring is average operation or integral operation in the image, accordingly, the inverse operation can be used such as derivative or finite difference and the algorithm of laplacian operator [3] is based on this operation.

The basic method of laplacian operator is to decrease the gray-level of the center pixel if the gray-level lower than the average gray-level around and to increase the gray-level of the center pixel if higher than the average gray-level around to realize sharpening.

Summary

High quality of image dependent on ISP module, ISP algorithms, multi-lens, High-resolution ratio, larger pixel and other factors, which leads the thickness of camera module to becoming bigger and bigger, it is tilted against the tendency to be thinner and lighter.

The emergence of two cameras phone which installs two cameras on the back of the phone to simulate a pair of human eyes not only solves the thickness but also achieves remarkable results in 3D effects, improving image quality at night and optical zoom, etc.

In order to use the two cameras to take snapshots, the captured pictures must be processed at the same time, which requires dual CIS, dual ISP, the assistance of algorithm and ISP's synchronization.

In short, ISP is a relatively complex system and there has a lot of issues need to study and learn.

References

[2] Milan Sonka, Vaclav Hlavac, Roger Boyle. Image Processing, Analysis, and Machine Vision[M]. Beijing: Tsinghua University Press.

[3] Jing Zhang, Hong Xia Ni, Chun Miao Yuan. Proficient in Matlab (Digital Image Processing and Recognition)[M]. Beijing: People's Posts and Telecommunications Publishing House, 2013.

[4] Xiao Tong Li, Zhao Feng Cen. Geometrical Optics, Aberrations and Optical Design[M]. Hangzhou: Zhejiang University Press, 2014:89-93.

[5] Chun Lei Dan. Research on Content Enhancement for Image Signal Processor[D]. Xi'an Electronic and Science University. 2013:8-10.

[6] Jiang Xue. Implementation of Image Signal Processor (ISP) and FPGA verification[D]. Zhejiang University. 2007:29-31.

[7] Wei Wang. Research and Realization of Auto White Balance Algorithm for Digital Camera[D]. Hunan University of Science and Technology. 2014:15-16, 25.