Survey On Multi-Frame Image Super-Resolution

233

Survey On Multi-Frame Image Super-Resolution

Niyanta Panchal, Bhailal Limbasiya, Ankit Prajapati

Abstract: Multi-frame Image Super-Resolution is to generate the high-resolution (HR) image from multiplelow-resolution images perspectives of a same scene and also increase spatial resolution by fusing information. In that first, Image registration which is most important part of multi-frame Super-resolution, they have give accurate alignment using the registration parameter. In this paper, we are going to review the different image registration methods and compare all the methods. Then next using various Super resolution methods which is generate high-resolution(HR) image from one or more low resolution images and lastly different image quality metrics reviewed as measure the original image and reconstructed image.

Index Terms: Multi-frame Image Super-resolution, Super-Resolution, Image Registration, Image Quality, PSNR

————————————————————

1 INTRODUCTION

IMAGE Super-Resolution is the most widely and expensive area of research and they can decade to solve the problem of limited resolution by image acquisition devices and also dependent on sensor. But, high- resolution sensor is very expensive. And for image acquisition could be as simple and it also involves preprocessing such as scaling. Also the available camera may not always sufficient for any given application. So, we need increase the current resolution by two ways, either reducing the pixel size or by increasing the chip size. However it has some limitations which can generate noise and degrade the image quality. Therefore, a new method is required to increase the resolution of the image. Super-resolution can used many application likes Medical imaging, Satellite imaging, Remote imaging, Video surveillance, Enlarging consumer photograph for higher quality. Now, Super-Resolution (SR) is to obtain a high-resolution (HR) image using one or more observed low-resolution (LR) images by down-sampling, de-blurring, and de-noising. Where, Low-Resolution (LR) image represents low pixel quality and it provide less accurate details. High-Resolution (HR) represents high pixel quality and it provides more accurate details. In the Super-resolution techniques, that classifies into two major parts: Frequency domain approach, and Spatial domain approach. Frequency domain approach, which can perform Fourier transform of an image. These methods are simple and computationally cheap, they are extremely sensitive to model error, limiting their use and Spatial domain approach, which can perform directly on pixel and it is also more popular method. These methods are computationally expensive.

Now, for the technical implementation of Super-Resolution in two ways: Single-frame and Multi-frame image Super-Resolution. Single-frame Super-Resolution methods to generate single high-resolution image from single degraded or noisy or blurred image. And Multi-frame Super-Resolution is to generate the high-resolution (HR) image from multiple low-resolution images perspectives of a same scene and also increase spatial resolution by fusing information. In this paper, for multi-frame image super resolution 4 different image registration algorithms that measure movement parameter (i.e. shift and rotation) and align the image and 3 super resolution algorithms that construct high-resolution image different approaches. And last 3 image quality metrics that measure the similarity between the two images. This paper is organized as follows. In Section II, we describe Multi-frame Super-Resolution images. Section III presents Image Registration methods. Section IV represents Super-Resolution Techniques. Section V represents image quality metrics. Section VI represents Experiments Results. Section VII represents conclusions.

2 MULTI-FRAME SUPER-RESOLUTION IMAGES

y = F H D x + n (1)

Where, x is an original HR image, F is linear transformation, H is a image warping, D is the down-sampling operator and n is a noise. In this paper, multi-frame Super-Resolution is to generate the high-resolution (HR) image from multiple low-resolution images perspectives of a same scene and also increase spatial resolution by fusing information. There are three steps for multi-frame SR, 1) Image Registration, which is performed first in order to align the LR images as accurately as possible, and also estimate movement parameters, 2) Image Fusion, which is the process of combing information of interest in two or more LR images into a single high-resolution image, 3) Interpolation, which is done to get high- resolved image. So, Image registration plays accurate role in image reconstruction process.

3 IMAGE REGISTRATION METHODS

Image Registration is the most important method for multi-frame image super resolution. It is used for motion estimation likes shift estimation and rotation estimation. The main goal for image is also aligning the images to decrease the difference between the reference image and the changing image. There are three types of registration parameters: Translation parameter, which calculates horizontal and vertical ____________________________

• Niyanta Panchal is currently pursuing masters degree program in Computer Science & Engineering in Gujarat Technological University, India,

E-mail: [email protected]

• Bhailal Limbasiya is currently working as Assistant Professor in Computer Science & Engineering Department in Parul Institute of Technology, Gujarat, India, E-mail: [email protected]

234 displacement between the reference image and the changing

image; Rotation parameter, which calculates shift angle between two images; Scale parameter, which calculates size variance in the same object. In this section we review the different image registration methods evaluated in this paper.

3.1Principle Component Analysis (PCA) Registration Method

The PCA based Registration method can be obtain approximated shift, rotation and scale variance parameter [2]. In that we firstly segment the focused part from the original image, and then binaries it with special part‟s pixel value as 1 and otherwise 0,where the special part‟s is called as BROI(Binary Region of Interest).Then, the PCA registration method measure the movement parameter and scale variance using the extracted BROI image and PCA. By segmenting human facial BROI from image, the color information can be achieving acceptable face region. In that first translating the RGB color space into YCC space and used the threshold value for using if a pixel belongs to human facial pixels. Then we used the morphological operators (for opening and closing processing) to the initial BROI face image for removing noise or filling the non-facial color. We need to extract the BROI part for all LR images have a same face, and measure a centre (x, y ) of gravity of BROIs to estimate translation parameter between the reference image and the other images the equation (2).

x= 1 N xi

N i=1 y=

1

N yi N

i=1 (2)

And then estimate rotation parameter and measure eigenvector of pixel coordinates covariance matrix C corresponding to gravity center in BROI using PCA. And then, using PCA to calculate the Eigen value λ and eigenvector e of symmetric covariance matrix C. and then the global orientation ө of BROIs in all LR image. Then, lastly we can getting the scale variance parameter between the two BROI images by comparing the total pixel orientation between two BROIs images.

3.2Gradient Based Registration Method

Gradient based registration method can accurately measure translation and rotation parameter with the original image and image for measuring movement parameters. In this method, first to optimize the cost function and second then it can be implemented by the steepest descent method. The correct setting of initial values for movement parameters has an essential role of movement parameters [2].

3.3Optical Flow Registration Method

In Optical Flow Registration method, some applications can benefit from the generalization of SR techniques for facial images to support the imaging of objects that are non-planar, non-rigid, or which are subject to self-occlusion when rotated. One way of the problem of non-rigidity is to allow the image registration to be an arbitrary flow field. However robust estimation techniques can be used to address these problems.

3.4GPOF Registration Method

Gaussian Pyramid Optical Flow (GPOF) registration method, which can reach the sub-pixel precision of super-resolution reconstruction and enable to allow large pixel motions, while keeping the size of the integration window relatively small [3].

In that basic idea of build a Gaussian pyramid for each frame of the low resolution observed Sequence. Then compute the optical flow d which is regarded as the displacement from image g2 to image g1 in each level of the pyramid between two observed images. It starts computing from the upper level till down to the lowest level of the pyramid. The advantage of Gaussian Pyramid Optical Flow (GPOF) registration is that it can reduce the computing complex process to speed up the algorithm implementation because each residual optical flow vector can be kept very small while computing a large overall pixel displacement vector.

4 SUPER-RESOLUTION TECHNIQUES

In this section we review the different super-resolution algorithms evaluated in this paper.

4.1Regularization scheme

SR image reconstruction is normally an ill posed problem. However, it can be stabilized with a regularization procedure. A regularization term compensates the missing measurement information with the desirable HR solution, in the generalized minimization cost function [1]. In this paper, we apply the regularization based SR image reconstruction method on the basis of multi-frame image SR. This method has been demonstrated to be effective. Firstly, a linear observation model is utilized to associate the recorded LR images with the unknown reconstructed HR image estimates, and we apply bilateral total variation (BTV) operator as a regularization term and measured by L1 norm. Now, to analyse the problem of the super-resolution reconstruction, first we should establish the degrade model of the image from a mathematical point-of-view, through the model connect the LR image and the HR image. By imposing the additive noises on the observation model is given by the equation (3).

𝑌𝑘= 𝐷𝑘𝐻𝑘𝐹𝑘 𝑋 + 𝑛𝑘 1 ≤ 𝑘 ≤ 𝑀 (3)

Where, 𝑋 is used to present the HR image,𝐻𝑘 is the blur

matrix, 𝐷𝑘 is the down-sampling matrix, 𝐹𝑘 represents the

motion process of the k th image, 𝑛_𝑘 represents the system additive noise. According to the regularization-based method, a regularization term should be used in this equation, and the SR can be inverted to a generalized minimization cost function the equation (4).

min J 𝑋 ,J 𝑋 = 𝑁 𝜌

𝑘=1 𝑌𝑘, 𝐷𝑘 𝐻𝑘 𝐹𝑘 𝑋 + 𝜆𝛾 𝑋 (4)

Where, this equation controls the stability of the solution and gives a stable HR estimation. Regularization terms, we desire one which can result in HR images with sharp edges and easy to implement using bilateral total variation (BTV) operator the equation (3).

𝛾𝐵𝑇𝑉 𝑋 = 𝑙=−𝑃𝑃 𝑃𝑚 =0𝛼 𝑚 + 𝑙 𝑋 − 𝑆𝑥𝑙𝑆𝑦𝑚𝑋 ₁ (5)

Where l+m≥0, 𝑆𝑥𝑙𝑎𝑛𝑑𝑆𝑦𝑚 are shift matrices to present l and m

235 𝑋 = 𝐴𝑟𝑔𝑀𝑖𝑛

𝑋

𝐷𝑘𝐻𝑘𝐹𝑘𝑋 − 𝑌𝑘 ₁+ 𝜆 𝑝𝑚 =0𝛼 𝑚 + 𝑙 𝑋 − 𝑝

𝑙=−𝑝 𝑁

𝑘=1

𝑆𝑥𝑙𝑆𝑦𝑚𝑋 ₁ (6)

This equation (6) is not closest form solution. Nan Zhao [1] propose and apply the steepest descent method and calculation iteration and give the proper solution in this equation (7).

𝑋

=

𝑋

– 𝛽

𝐹

𝐻

𝐷

𝑠𝑖𝑔𝑛 𝐷

𝐻

𝐹

𝐷

𝑋 – 𝑌

+

𝜆

𝛼

𝑙 − 𝑆

𝑆

𝑠𝑖𝑔𝑛 𝑋

−

𝑆

𝑆

𝑋

Where β is scalar defining the step size in the direction of the gradient, λ controls the regularization term, 𝑆𝑦−𝑚𝑎𝑛𝑑 𝑆𝑥−𝑙

transpose matrices and shifting effect in opposite direction. For, multi-frame SR reconstruction algorithm based on regularization, and this method can enhance the quality of HR image with less blur effect and more details.

4.2 Robust Super-Resolution

Robust Super-Resolution describes an iterative solution for super-resolution algorithm. As we know to generate a reconstructed high-resolution image which is similar to low-resolution frames, we need a minimum cost function which compares with low-resolution and high-resolution frames. We choose L1 norm cost function instead of L2 norm cost function, as including the effects of outliers [5]. In the super-resolution cases (N < 𝑟2_{) in that N is the number of low resolution frames}

and r is the resolution enhancement factor), certain pixel locations will have no measure at all time. For these cases, it is estimate an extra term, it called regularization term to remove outliers. This term calculating missing data. Regularization is a useful tool even in the square and over-determined cases (N = 𝑟2 _{and N > 𝑟}2 _{respectively), it can help algorithm to remove}

artifacts from the final answer. The following formulas as given:

𝑋 = 𝐴𝑟𝑔𝑀𝑖𝑛 𝑋

𝐷𝑘𝐻𝑘𝐹𝑘𝑋 − 𝑌𝑘 ₁+ 𝜆 𝑝𝑚 =0𝛼𝑚 +𝑙 𝑋 − 𝑝

𝑙=0 𝑁

𝑘=1

𝑆𝑥𝑙𝑆𝑦𝑚𝑋 ₁ (8)

Where, λ is a scalar for properly weighting the first term (similarity cost) against the second term (regularization cost). 𝑆𝑥𝑙𝑎𝑛𝑑𝑆𝑦𝑚 are shift matrices to present l and m pixels shift in

horizontal and vertical direction, presenting several scales of derivatives. Scalar weight, 0 <𝛼α < 1, is applied to give a spatially decaying effect to the summation of the regularization term.

4.3 Fast Robust Super-Resolution

Fast robust Super-Resolution algorithm based on the L1 norm, both for the regularization terms and the data fusion terms. Whereas the former is responsible for edge preservation, the seeks robustness with respect to motion error, blur, outliers, and other kinds of errors not modeled in the fused image [5]. In this method, resolution enhancement is broken into two consecutive steps:

1) Non-iterative data fusion 2) Iterative deblurring-interpolation.

Registration followed by median operation results in blurred HR image 𝑍 =𝐻𝑋 .The goal of the deblurring interpolation step is finding the deblurred HR frame X. that for the under-determined cases, not all pixel values can be defined in the data fusion step, and their values should be defined in a separate interpolation step. The following formulas our minimization criterion for obtaining X from:

𝑋 = 𝐴𝑟𝑔𝑀𝑖𝑛 𝑋

𝐻𝑋 − 𝑍 ₁+ 𝜆′ 𝑝 _𝛼 𝑚 + 𝑙 _{𝑋 −} 𝑚 =0

𝑝 𝑙=−𝑝

𝑆𝑥𝑙𝑆𝑦𝑚𝑋 ₁ (9)

Where matrix A is a diagonal matrix with diagonal values equal to the square root of the number of measurements that contributed to make each element of 𝑍 (in the square case A is the identity matrix). So, the undefined pixels of have no effect on the HR estimate 𝑋. Decimation and warping matrices (D and F) and summation of measurements are not present anymore which makes implementation of (9) much faster than (8).

5

IMAGE

QUALITY

METRICS

In this paper, the quality of reconstructed image is how similar to the original high-resolution image. So, there is number of measures for image quality likes,1) Mean-squared Error (MSE), 2) Peak Signal-to-noise Ratio (PSNR), and 3) Structural Similarity Index Measure (SSIM).

5.1 Mean-squared Error (MSE)

Mean-squared error (MSE) is simply the squared error between a Super-Reconstructed image and the original high-resolution image.The MSE can be expressed as,

MSE= 1

mn I i,j −K i,j

2 n−1

j=0

m−1

i=0

Where I and K represent the mxn matrices and also compare the images. They performed for the dimensions „i‟ and „j‟ So, I(i, j) represents the value of pixel (i, j) of original image I. and K(i, j) represents the value of pixel (i, j) of reconstructed image.

5.3 Peak Signal-to-noise Ratio (PSNR)

The Peak Signal-to-Noise Ratio (PSNR) is defined as a measuring of quality of reconstructed image and also comparing with original image. In that MSE is used for two mxn matrices represents with images I and K and compare the images. The PSNR can be expressed as,

𝑃𝑆𝑁𝑅 = 20. log10 𝑀𝐴𝑋_𝐼2 𝑀𝑆𝐸

Here, MAXI perform the maximum possible pixel value of the

image. When the pixels are represented 8 bits per sample, this is 255.The PSNR expressed in decibels.

5.4 Structural Similarity Index Measure (SSIM)

236 𝑆𝑆𝐼𝑀 𝑋, 𝑌 = 2𝜇𝑥𝜇𝑦+𝑐1 2𝜎𝑥𝑦+𝑐2

𝜇_𝑥2_+𝜇 𝑦 2_+𝑐

1 𝜎𝑥2+𝜎𝑦2+𝑐2

Where, x and y are sub images of X and Y, andµx , µy are the average of x, y. σx, σy are standard deviations of x, y. C1 is set to C1=(0.01*255)^2and C2=(0.01*255)^2.

6

EXPERIMENT RESULTS

We use the set of LR images; each includes 5 images, and for validating the efficiency of compare all the methods. The set of LR images for Human face set are shown in Fig 1. The size of LR image of Human face set is 180 * 120 gives the reconstruction of using 4 types of registration methods. So, the results of the registration methods shown in Fig 2(a, b, c, d). And compare all the methods so the results are given in Table 1 PSNR (db) of reconstructed images. We have to using only PSNR image quality measure. Now, the different super-resolution techniques for efficiency of compare all the methods. We choose the image Eia with size of 360*360 as the original HR image using regularized term based on L1 norm. So, the results of the L1 norm shown in Fig 3. and compare all the methods so the results are given in Table 2 PSNR (db) of reconstructed images. Now, We compare the performance of two resolution methods. First, we have created sequence of LR frames using one HR image as shown in Fig.4(a). Then, we shifted this HR image by a pixel in the vertical direction and horizontal directions and they have to produce 16 LR images from the original scene. The LR frames are shown in Fig.4(b).The super-resolved images for Robust and Fast Robust are shown in Fig.4(c) and Fig.4(d) respectively. And compare the methods so the results are given in Table 3 the Robust algorithm gives the highest values for PSNR ,it gives slightly better results than Fast-Robust algorithm

(a) (b)

(c) (d)

(d)

(a) (b)

(c) (d)

Figure 2 For human face set (a) Optical flow, (b) PCA , (c) Gradient method,(d) GPOF Methods

(a) (b)

(c) (d)

(d)

237 (a) (b)

(c) (d)

Figure 4 (a) original image, (b) LR image, (c) Super-resolution using Robust method, (d) Super-Super-resolution using

Fast Robust method

TABLE I. PSNR of reconstructed image

Image Quality Measure

PSNR

Method PCA GPOF Optica l

Gradie nt

Face 28.659 1

29.744 3

22.985 4

23.744 7

TABLE II. PSNR of reconstructed image

Image Quality Measure

PSNR

Algorithm Bi-cubic L1+TV L1+BTV

Eia 18.1355 22.301

5 24.15

TABLE III. PSNR of reconstructed image

Image Quality

Measure PSNR

Algorithm Robust Fast-Robust

Flower 12.164 11.986

7 CONCLUSION

This paper is practical implementation of image registration which is most important part of multi-frame Super-resolution. Now the different image registration methods and compare all the methods they give the accurate results. Then next using various Super resolution methods the regularized term based on L1-norm can enhance the HR quality with less blur effect and more details, and Robust super-resolution method gives better results than the Fast-Robust method. Image similarity measures were used to measure the efficiency of the algorithms. And lastly different image quality metrics reviewed as measure the original image and reconstructed image.

REFERENCES

[1] Nan Zhao; Cuihua Li; Hua Shi; Chen Lin, "Multi-Frame Image Super-Resolution Based on Regularization Scheme," Control, Automation and Systems Engineering (CASE), 2011 International Conference on , vol., no., pp.1,4, 30-31 July 2011

[2] Sasatani, S.; Xian-Hua Han; Yen-wei Chen, "Image registration using PCA and gradient method for super-resolution imaging," Software Engineering and Data Mining (SEDM), 2010 2nd International Conference on , vol., no., pp.631,634, 23-25 June 2010

[3] Lan Zhang; Hua Zhang; Simiao Zhang; YanbingXue, "Multi-frame image super-resolution reconstruction based on GPOF registration and L1-norm," Natural Computation (ICNC), 2010 Sixth International Conference on , vol.7, no., pp.3601,3604, 10-12 Aug. 2010

[4] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust Multi-frame super resolution”, IEEE Trans. Image Process., vol. 13, no.10, pp. 1327-1344, 2004.

[5] Vishal R. Jaiswal, Suhas H. Patil, Girish P. Potdar, ShrishailT.Patil,” Multi-frame Image Super-Resolution-A Comparison”,(IJCSE) International Journal On Computer Science and Engineering, Vol. 02, No. 09, 2010, ISSN 0975-3397

[6] Xuelong Li, Yanting Hu, Xinbo Gao, et.al, “A multi-frame image super-resolution method”. IEEE Transactions on Signal Processing 90, pp.405-414, 2010.

[7] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity”, IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, 2004.

[8] Yusra A. Y. Al-Najjar, Dr. Der Chen Soong,“ Comparison of Image Quality Assessment: PSNR, HVS, SSIM, UIQI ”, International Journal of Scientific & Engineering Research, Volume 3, Issue 8, August-2012 1 ISSN 2229-5518

[9] Turgay Celik, Kai-Kuang Ma, “Fast Object-based Image Registration Using Principal Component Analysis for Super-resolution Imaging”, 5th International Conference on Visual Information Engineering, pp.705-710, 2008.