2016 Joint International Conference on Artificial Intelligence and Computer Engineering (AICE 2016) and International Conference on Network and Communication Security (NCS 2016)
ISBN: 978-1-60595-362-5
Low-Rank Total Variation for
Multi-frame Image Super-resolution
Jun-Bao ZHAO
1,a, Xiao-Yan CHEN
2,b, Sheng-Jin WANG
1,c,*1State Key Laboratory of Intelligent Technology and Systems
Tsinghua National Laboratory for Information Science and Technology Department of Electronic Engineering, Tsinghua University, Beijing, China
2Space Star Technology Co., Ltd., Beijing, China
a[email protected], b[email protected], c[email protected]
*Corresponding author
Keywords: Multi-frame, Super-resolution, Low Rank, ADMM.
Abstract.The methods based on regularization are the mainstream of super resolution research due to the characteristics of natural images. Even most of them are based on the total variation (TV) method, the characteristics of natural images cannot be expressed completely by this way. According to low rank property of natural image statistics, a new algorithm is proposed to restore more high-frequency details by combining both of them. Compare to other methods, experimental results show that the proposed algorithm can achieve a better reconstruction quality.
Introduction
As science advances, the demands of high-resolution images grow fast. Reducing the size of the pixels during manufacturing is the common method to meet the demands. But when the size of photoreceptors become smaller, the incident light intensity of each photoreceptor will be lower, which leads to low signal-to-noise ratio 0. Therefore, researching other ways to super the resolution becomes particularly important.
Common methods of super resolution can be divided into two categories: single-frame methods and multi-frame methods. Single-frame methods process one single degraded image to get the corresponded high-resolution image. On the contrary, multi-frame super resolution algorithms generate high resolution image from multiple images. They can be also divided into two types: spatial domain methods and frequency domain methods.
For spatial methods, some of them are based on non-uniform interpolation, which have short calculation time 0. They suit for real-time application. The methods based on bayesian maximum A-posteriori estimation 0 generate the high resolution image by using the prior information. The super resolution answer is rigorous and theoretical. By combining segmentation, motion estimation, super resolution together, the MAP-based methods 0 is proposed to solve complicated super-resolution problem. The methods based on convex set projections 0 are succinct, and they are easy to use. And the methods 0 based on iterative back projection can construct the high resolution image directly. But they can’t solve the inverse problem well, and their parameters are hard to choose. For frequency methods, the methods based on relative shifts was proposed by Tsai and Huang 0 First. Several algorithms 0 based on these methods were proposed lately. They have three main aspects: the principle of spectral aliasing, Fourier transform shifting, and original high resolution image bandwidth.
the noise in the solution, smoothness constraints were introduced in it. But some details were lost by the constraints. Farsiu’s algorithm 0 is another representative methods. It uses bilateral total variation (BTV) operator to regularize. Compare to Tikhonov method, it retains more detail, and is more stable. But it doesn’t have the local adaptivity and cannot deal the partial smooth area well.
The Proposed Method
In this section, we will first introduce the basic multi-frame super-resolution model and then present our low rank total variation multi-frame super-resolution model and optimization method.
Multi-frame Super-resolution Problem
The observation model of multi-frame super-resolution is formulated as follows:
K
k
n
X
DS
Y
k
k
k,
1
,...,
. (1)Where Ykdenotes the observed kth low resolution image, D is a down sampling matrix, S stands for blurring and warping matrix, X is the HR image that we aim to recover, and n represents the noise. The problem can usually be solved as follows:
) ( || ||
min 2
1 DSkX Yk X
K
k
X . (2)
where the first term is a reconstruction error term. The second term is a regularization term often defined based on prior knowledge or image statistics. Weight is the trade-off parameter to balance the fidelity term and regularization term.
Low Rank Total Variation Multi-frame Super-resolution
Total variation (TV) is a classic image prior in image restoration due to its ability to regularize the gradients of natural image. TV also plays an important role in multi-frame super-resolution. However, due to high variability and complexity of nature image, TV cannot capture all the statistics of images. The super-resolved quality is still limited. So other priors based on the image statistics are needed to improve the result.
In this paper, we propose to use low rank constraint to regularize the targeted high resolution image X.
Recently, low rank property is widely studied by researchers and is applied to many computer vision problems and has achieved a lot of success. More specifically, in the image processing field, nonlocal spectral prior is found and used for image restoration [16]. Furthermore, weighted nuclear norm is proposed for image denoising [17].
Combine the reconstruction error term, TV regularization term and low rank regularizer together, we obtain the final objective function:
) ( ||
|| ||
||
min 2 *
1 DS k X Yk rank X tvTV X
K
k
X . (3)
Model Optimization.
We adopt the alternating direction method of multipliers (ADMM) [18] algorithm to solve the problem. We introduce auxiliary variable M to get the new cost function:
M X X
TV M
Y
DSX k rank tr tv
K
k M
X || || || || ( )subject to
min 2
1
After that, Augmented Lagrangian method of multiplier with multiplies Li is adopted and Eq.(4) can be recast as an unconstrained optimization problem:
) || || || (|| 2 ) ( || || || ||
min 2 2 2
1 ,
, DSkX Yk rank M tr tvTV X X M L L K
k L M
X
. (5)
Next, we split the problem (5) into three subproblems and adopt iterative optimization: Update X: 2 ) ( ) ( 2
1|| || ( ) 2|| ||
min
arg j j
tv k k K
k
X DS X Y TV X X M L
. (6)
This subproblem can be optimized by gradient descent directly. Update M: 2 ) ( ) 1 ( || || 2 || ||
min j j
tr rank
M M X ML
. (7)
This subproblem can be solved according to [20], where SVT( ) is the Singular Value Thresholding
operator:
) ( ( 1) ( ) / j j L X SVT M rank
. (8)
Update L:
) ( ( 1) ( 1) )
( ) 1
(j j j j
M X
L
L . (9)
Parameters are optimized based on a small set of datasets. In this work, we set 0.01
rank
,tv 0.01,and the maximum iteration as 300. The stopping criterion is measured by|| 1||/|| ||1 5 e T X
Xk k
The above optimization procedure can be summarized in Algorithm 1
Experiment
Experiment Settings
To validate the effectiveness and robustness of the proposed algorithm (denoted as “LRTV”) for SR reconstruction, eleven typical 256-gray-level HR images were selected for performance evaluation., it was evaluated subjectively and objectively in comparison with bi-cubic interpolation (denoted as “bicubic”) ,the robust SR method [19] (denoted as “SAR”). We also test the TV Prior method (denoted as “TV”) .The methods are compared quantitatively in terms of Peak Signal-to-Noise Ratio (PSNR) of their HR output images.
In the first place, HR images were downsampled by a given magnification factor with random warp,without blurring or noise. Then, the LR image is bi-cubically interpolated by the same factor to get to an interpolated HR image. Note that the interpolated HR images are also called LR images, as the high-frequency details are still missing.
Experimental Results
[image:4.612.118.503.267.595.2]Ground truth / PSNR Bicubic / 22.81dB LRTV / 23.64dB SAR / 23.68dB TV / 23.58dB Figure 1.
Ground truth / PSNR Bicubic / 25.94dB LRTV / 28.36dB SAR / 28.24dB TV / 28.19dB Figure 2.
Ground truth / PSNR Bicubic / 25.06dB LRTV / 33.08dB SAR / 32.97dB TV / 32.02dB Figure 3.
Summary
In this paper, we proposed a new super resolution algorithm based on regularization. By combining TV method and the low rank characteristics of natural image. The method is able to recover good quality high resolution image and achieves higher PSNR compared to other algorithms.
Acknowledgement
This research was financially supported by the National Science and Technology Support Program under Grant No. 2013BAK02B04. Initiative Scientific Research Program of Ministry of Education under Grant No. 20141081253.
References
[1] Li X, Hu Y, Gao X, et al. A multi-frame image super-resolution method [J]. Signal Processing, 2010, 90(2): 405-414.
[2] Lertrattanapanich S., Bose N.K. High resolution image formation from low resolution frames using Delaunay triangulation. [J]. IEEE Transactions on Image Processing, 2002, 11(12): 1427-1441. [3] Nguyen N., Milanfar P. An efficient wavelet-based algorithm for image superresolution [C]// International Conference on Image Processing. 2000: 351-354 vol. 2.
[4] Elad M., Feuer A., Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. [J]. International Journal of Solids & Structures, 1985, 21(8): 907-926.
[5] Xiao L., Wei Z. A Super-resolution Reconstruction via Local and Contextual Information Driven Partial Differential Equations [C]// fskd. IEEE Computer Society, 2007: 726-730.
[6] Huanfeng S., Liangpei Z., Bo H., et al. A MAP approach for joint motion estimation, segmentation, and super resolution. [J]. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, 2007, 16(2): 479 - 490.
[7] Stark H., Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections. [J]. Journal of the Optical Society of America A Optics & Image Science, 1989, 6(11): 1715-1726.
[8] Patti A.J., Altunbasak Y., Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. [J]. IEEE Transactions on Image Processing, 2001, 10(1): 179-86.
[9] M. Irani, S. Peleg, Improving resolution by image registration, CVGIP: Graphical Models and Image Processing 53(3)(1991)231–239.
[10] Tsai R.Y., Huang T.S., Tsai R.Y., et al. Multiframe image restoration and registration [C]// Advances in computer vision and image processing. 1984.
[11] Tekalp A.M., Ozkan M.K., Sezan M.I. High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration [C]// IEEE International Conference on Acoustics, Speech, & Signal Processing, Icassp. IEEE, 1992: 169-172.
[12] Rhee S, Kang M G. Discrete cosine transform based regularized high-resolution image reconstruction algorithm [J]. Optical Engineering, 1999, 38(8):1348-1356.
[14] Zhang X., Lam E.Y., Wu E.X., et al. Application of Tikhonov Regularization to Super-resolution Reconstruction of Brain MRI Images [J]. Lecture Notes in Computer Science, 2007, 4987 lncs(23): 51-56.
[15] Sina F., M. Dirk R., Michael E., et al. Fast and robust multiframe super resolution. [J]. Image Processing IEEE Transactions on, 2004, 13(10): 1327-1344.
[16] Wang, Shenlong, Lei Zhang, and Yan Liang. “Nonlocal spectral prior model for low-level vision.” Computer Vision–ACCV 2012. Springer Berlin Heidelberg, 2012. 231-244.
[17] Gu, Shuhang, et al. “Weighted nuclear norm minimization with application to image denoising.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014.
[18] Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3, 1–122 (2011).
[19] S. Villena. Superresoluci_on y Reconstrucci_on Bayesiana de Im_agenes a Partir de Im_agenes de Baja Resoluci_on Rotadas y Desplazadas. Combinaci_on de Modelos. PhD thesis, Universidad de Granada, December, 2011.
