High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement

(1)

High Dynamic Range Image

Reconstruction with Spatial Resolution

Enhancement

J

ONGSEONG

C

HOI

, M

IN

K

YU

P

ARK AND

M

OON

G

I

K

ANG*

Institute of TMS Information Technology, Yonsei University, Seoul, Korea *Corresponding author: [email protected]

For the last two decades, two related approaches have been studied independently in conjunction with limitations of image sensors. The one is to reconstruct a high-resolution (HR) image from mul-tiple low-resolution (LR) observations suffering from various degradations such as blur, geometric deformation, aliasing, noise, spatial sampling and so on. The other one is to reconstruct a high dynamic range (HDR) image from differently exposed multiple low dynamic range (LDR) images. LDR is due to the limitation of the capacitance of analogue-to-digital converter and the non-linearity of the imaging system’s response function. In practical situations, since observations suffer from limitations of both spatial resolution and dynamic range, it is reasonable to address them in a unified context. Most super-resolution (SR) image reconstruction methods that enhance the spatial resolution assume that the dynamic ranges of observations are the same or the imaging system’s response function is already known. In this paper, the conventional approaches are overviewed and the SR image reconstruction, which simultaneously enhances spatial resolution and dynamic range, is proposed. The image degradation process including limited spatial resolution and limited dynamic range is modelled. With the observation model, the maximum a posteriori estimates of the response function of the imaging system as well as the single HR image and HDR image are obtained. Experimental results indicate that the proposed algorithm outperforms the conventional approaches that perform the HR and HDR reconstructions sequentially with respect to both

objective and subjective criteria.

Keywords: resolution, dynamic range, sampling, quantization, reconstruction Received 15 May 2006; revised 22 August 2007

1. INTRODUCTION

Recently, digital imaging systems with CCD and CMOS imaging sensors have widely been used for various purposes such as medical tomography, industrial monitoring system, surveillance system, computer vision system, scientific research applications, broadcasting system, consumer appli-ances, etc. Although imaging sensors are widely used and suit-able for most digital imaging applications, they have physical limitations such as limited spatial resolution and dynamic range. Figure 1 shows the various physical limitations of imaging sensors [1].

The spatial resolution that represents the number of pixels per unit area in an image is the principal factor to determine the quality of an image. However, because an imaging sensor is the composition of a number of photodiodes, spatial resolution of the sensor is limited. Besides, in the

process of recording a digital image, there is a natural loss of spatial resolution caused by the optical distortions (out-of-focus, diffraction limit, etc.), motion blur due to limited shutter speed and noise, which occurs within the sensor or during transmission. Thus, the recorded image usually has the limited spatial resolution. Therefore, high-resolution (HR) images should be considered not only to have higher spatial density of pixels but also to be visually sharper and clearer than low-resolution (LR) images. Since an HR image that has more details may be critical in various applications than LR images, the reconstruction of an HR image from multiple LR images is needed.

Low dynamic range (LDR) is also one of the main problems when an image is acquired from imaging sensors. The light intensity of natural scenes has a very wide range, whereas the dynamic range of an imaging sensor is limited. For Permissions, please email: [email protected]

Advance Access publication on October 4, 2007 doi:10.1093/comjnl/bxm080

at Pennsylvania State University on September 11, 2016

http://comjnl.oxfordjournals.org/

(2)

This results in the presence of no response with low exposure and saturation with high exposure. When a scene with a very wide range of light intensity that exceeds the dynamic range of the sensor is captured, it is inevitable that there is a loss of information in either the low-intensity areas, the high-intensity areas or both of them. The limitation of the analogue-to-digital converter (ADC)’s capacitance is one reason for LDR. If the ADC of the imaging system has small capacitance, the quan-tization interval of the system response function becomes wider and it generates a loss of light intensity information. Therefore, the high dynamic range (HDR) imaging system is needed.

Owing to the demand and need for HR and HDR images, many researches have been studied to overcome LR and LDR [2 – 32]. The researches to overcome the spatial resol-ution and dynamic range limitations are categorized into two groups. The first one is based on device physics and circuit technology, and the other one includes signal processing-based approaches. In device-processing-based approaches, the most direct solution to increase spatial resolution is to increase the number of pixels per unit area by sensor manufacturing techniques. According to the decrease of the amount of light per pixel, shot noise that severely degrades the image quality is generated. There are several signal processing tech-niques to obtain an HR image from observed multiple LR images, and it is called super-resolution (SR) image recon-struction in the literature [2 – 17].

To enhance the dynamic range, the new architectures of image sensors have been proposed, and the algorithmic tech-niques to increase the dynamic range of an image using differ-ently exposed multiple frames of the same scene have been researched [19 – 30]. The signal processing approaches to overcome the limitation of dynamic range are also called SR image reconstruction in wide sense. The SR image reconstruc-tion is proven to be useful in many practical cases.

In this paper, the conventional approaches to enhance the spatial resolution and dynamic range are reviewed, and an

SR reconstruction algorithm that simultaneously enhances the spatial resolution and dynamic range is proposed. The image degradation process including limited spatial resolution and limited dynamic range is modelled and the maximum a posteriori (MAP) estimates of both the response function of the imaging system and a single image with HR and HDR are obtained. In the reconstruction process, the response func-tion of the imaging system is updated in every iterafunc-tion step.

The rest of this paper is organized as follows. The previous works to overcome the limitations mentioned above (i.e. the limited spatial resolution and limited dynamic range) are reviewed in Section 2 and a novel SR approach is proposed in Section 3. In Section 4, experimental results are presented and conclusions are drawn in Section 5.

2. PREVIOUS WORK ON SPATIAL RESOLUTION

AND DYNAMIC RANGE ENHANCEMENT

2.1. Spatial resolution enhancement

In most electronic imaging applications, images with HR are desired and often required [3]. HR means that pixel density within an image is high, and, therefore, an HR image can offer more details that may be critical in various applications. For example, HR medical images are very helpful for a doctor to make a correct diagnosis. It may be easy to distinguish an object from similar ones using HR satellite images, and the performance of pattern recognition in computer vision can be improved an HR image is provided. Although imaging sensors are suitable for most imaging applications, the current resolution level and consumer price will not satisfy the future demand. For example, people want an inexpensive HR digital camera/camcorder, and scientists often need a very high-level resolution close to that of an analogue 35 mm, film which has no visible artefacts when an image is magnified. Thus, finding a way to increase the current resol-ution level is needed.

The most direct solution is to increase the number of pixels per unit area by sensor manufacturing techniques. As the pixel size decreases, however, the amount of light available also decreases. It generates shot noise that degrades the image quality severely. Therefore, there is a limitation on the pixel size reduction. Another approach for enhancing the spatial res-olution is to increase the chip size, which leads to an increase in capacitance. Since large capacitance makes it difficult to speed up a charge transfer rate, this approach is not considered effective. The high cost for high precision optics and image sensors is also an important concern in many commercial applications regarding HR imaging. Therefore, a new approach towards increasing spatial resolution is required to overcome these limitations of the sensors and optics manufac-turing technology. One promising approach is to use signal processing techniques to obtain an HR image (or sequence) from the observed multiple LR images. Recently, such a FIGURE 1. Physical limitations of imaging sensors.

T C J

(3)

resolution enhancement approach called as SR image recon-struction has been one of the most active research areas.

The basic premise for increasing the spatial resolution in SR techniques is the availability of multiple LR images captured from the same scene. In SR, LR images typically represent different looks at the same scene. That is, LR images are sub-sampled and aliased as well as shifted with subpixel precision from an HR image. If the LR images are shifted by integer units, each image contains the same information, and thus there is no new information that can be used to reconstruct an HR image. However, if the LR images have different sub-pixel shifts from each other and if aliasing is present, each image cannot be obtained from the others. In this case, the information contained in each LR image can be exploited to obtain an HR image. To obtain different looks at the same scene, some relative scene motions must exist from frame to frame via multiple frames or video sequences. Multiple frames can be obtained from one camera with several captures or from multiple cameras located in different positions. These scene motions can occur due to the controlled motions in imaging systems, e.g. images obtained from orbiting satellites, and the uncontrolled motions, e.g. movement of local objects or vibrating imaging systems. If the scene motions are known or can be estimated within subpixel accuracy, the SR image reconstruction is possible as illustrated in Fig. 2.

There are various approaches for SR reconstruction. Non-uniform interpolation approach [4 – 6] is the most intuitive method and the frequency domain approach [7 – 10] makes explicit use of the aliasing that exists in each LR image. Deter-ministic and stochastic regularization [11 – 13], projection onto convex sets approach [14 – 16] and ML-POCS approach [17] are other methods of SR reconstruction. An example illustrating the improvements of resolution through SR recon-struction appears in Fig. 3. Figure 3a – c shows a frame of LR images (nearest interpolated), a conventional bicubic interp-olated and an SR reconstructed image by MAP estimation

based on deterministic regularization approach [13],

respectively. The SR image reconstruction shows significantly improved image details.

2.2. Dynamic range enhancement

The human visual system exhibits an enormous optical

dynamic range of 200 dB from the scotopic threshold to

the glare limit, as it can adapt to an extremely high level of light intensity [18]. This capability is also required in numer-ous imaging applications. Imaging sensors, however, have a narrow limited dynamic range comparing with the human visual system. When capturing a scene with a very wide inten-sity range of light that exceeds the dynamic range of an imaging sensor, it is inevitable to lose detail in either the dark areas, the bright areas or both. By controlling either exposure time, aperture size or both, we can choose the lumi-nance level of the captured image. For example, with increased exposure time or aperture size, we can obtain a better representation of dark areas at the cost of losing infor-mation in bright areas. However, adjusting the exposure FIGURE 3. Simulation results of SR reconstruction. (a) One frame of LR images (nearest neighbor interpolated). (b) A bicubic inter-polated image. (c) An SR reconstructed image.

FIGURE 2. Basic premise for SR.

(4)

time or aperture size does not help to represent both low- and high-light areas of the scene with HDR. Details will be defi-nitely lost, and varying the exposure time just allows some control over where the loss occurs. Therefore, the novel archi-tectures of image sensors and the algorithmic techniques to increase the dynamic range of images have been proposed [19 – 30].

Fabrication of multiple sensor elements within a pixel is a solution of the new architectures. Each pixel includes two or more photo detectors of different size [19, 20]. When an image is captured, each measurement is made in each photo detector on the same pixel and the HDR pixel values are com-puted on chip. Another approach is a fully new architecture, which is thoroughly different from CCD or CMOS image sensors [21]. Each pixel on the device includes an additional computation element that measures the time to attain full potential well capacity. Since the full potential well capacity is the same, the time to attain full potential well capacity is proportional to the radiance value of the pixel.

The signal processing-based approaches have a common feature of using multiple frames with different exposures. Figure 4 shows an example of multiple frames with different exposures. The simplest method to obtain the frames is sequentially capturing a scene with different exposures [22, 23]. The exposure for each frame can be controlled by varying the lens aperture, exposure time or both. Multiple image detector is proposed as another method for capturing multiple frames by Doi et al. [24] and Aggarwal and Ahuja [25]. They proposed a method to capture multiple images using only one imaging optics. This method generates multiple copies of the optical image using a mirror or a beam-splitter and captures them using multiple image detectors. The advan-tage of this method is that it can be applied for a scene with motion and produce the HDR images in real time. The

reconstruction method proposed by Aggarwal and Ahuja [25] produces the HDR image by transforming the recorded intensities into the actual luminance values. This is possible only when the response function of the sensor is known. Nayar and Mitsunaga [26] proposed spatially varying pixel exposures as an alternative method. The key feature of this method is the simultaneous sampling of the spatial domain and the exposure domain using a pattern masking as shown in Fig. 5. The brightness level associated with each pixel

rep-resents its sensitivity. The pixel e3is the most sensitive pixel,

and the pixel e0is the most insensitive pixel. They proposed

the HDR images reconstruction by aggregation and interp-olation. However, most of the previous approaches using mul-tiple frames have several limitations such as the assumption that the image capture devices have a linear response [23, 27] and there is no saturation even for highest exposure. Debevec and Malik [22] proposed an algorithm considering the nonlinearity of the response function and saturation of the pixel value for the highest exposure. Figure 6 shows the

FIGURE 5. Spatially varying pixel exposures [26].

FIGURE 4. An example of multiple frames with different exposures.

FIGURE 6. Experimental result of HDR reconstruction. (a) An under-exposed image. (b) A properly exposed image. (c) An over-exposed image. (d) The reconstructed HDR image using multiple LDR images.

T C J

(5)

experimental results of dynamic range enhancement. Figure 6a – c shows an under-exposed image, a properly exposed image and an over-exposed image, respectively. Figure 6d is a reconstructed HDR image with Fig. 6a – c. We can see that both dark and bright regions are well represented.

2.3. Simultaneous enhancement of spatial resolution

and dynamic range

Since the image sensors were invented, the enhancements of spatial resolution and dynamic range have been researched independently. However, in many practical situations, since observations suffer from limitations of both spatial resolution and dynamic range, it is necessary to reconstruct the HR and HDR images at once. Recently, the approaches, which simul-taneously enhance the spatial resolution and the dynamic range, have been proposed [31, 32]. On the basis of the spatially varying pixel exposures, the algorithm that enhances the resolution of multisampled images was proposed by Narasimhan and Nayar [31]. Multisampled images mean that a scene is sampled in multiple dimension such as space, time, spectrum and brightness. Gunturk and Gevrekci [32] proposed the MAP approach, which enhances the spatial resolution and the dynamic range with prior knowledge of imaging system’s response function. In the following section, an SR reconstruction algorithm that simultaneously enhances the spatial resolution and dynamic range without any prior knowl-edge about the system response function is proposed.

3. SIMULTANEOUS RECONSTRUCTION

ALGORITHM TO ENHANCE SPATIAL RESOLUTION AND DYNAMIC RANGE

3.1. Problem formulation

The proposed reconstruction algorithm simultaneously recon-structs the response function of imaging system and the HR and HDR images without any prior knowledge about the system response function. For this purpose, the image degra-dation process including limited spatial resolution and limited dynamic range is modelled. Among various obser-vation models to represent the limited dynamic range, we use the reciprocity, which means that the exposure is defined as the product of the luminance r at the surface of an imaging sensor and exposure time Dt [28, 29]. In addition, Hurter – Driffield curve is used as the response function of an imaging system [22], which curve is a graph of the optical density of the imaging sensor against the logarithm of the exposure rDt. The typical shape of Hurter – Driffield curve is shown in Fig. 7.

Consider the desired HR and HDR images of size L1N1

L2N2 written in lexicographical notation as the vector x ¼

[x1, x2, . . . , xN]T, where N ¼ L1N1L2N2. Namely, x is an

ideal undegraded image that is sampled at or above the

Nyquist rate from continuous scene which is assumed to be

bandlimited. Now, let the parameters L1and L2represent the

downsampling factors in the observation model for the hori-zontal and vertical directions, respectively. Thus, each

observed LR image and LDR image is of size N1N2. Let

the kth LR and LDR images of p frames be denoted in

lexico-graphic notation as yk¼ [yk,1, yk,2, . . . , yk,M]T, for k ¼ 1, 2, . . .,

p and M ¼ N1N2. Now, it is assumed that x remains

con-stant during the acquisition of the multiple LR and LDR images, except for any motion and degradation allowed by the model. Then, the observed kth LR and LDR image

cap-tured with the exposure time Dtkresults from warping,

blur-ring, subsampling operators and system response function performed on the HR and HDR images x. Assuming that each LR image and LDR image is corrupted by additive noise and quantization error, we can then represent the obser-vation model as

yk¼f ðDtkWkx þ nakÞ þ n

q

k for k ¼ 1; . . . ; p; ð1Þ

where the matrix Wkof size (N1N2)2L1N1L2N2represents,

via subsampling, blurring and motion, the contribution of

HR and HDR pixels in x to the LR and LDR pixels in yk, nka

is a lexicographically ordered noise vector and nkqrepresents

a lexicographically ordered quantization error vector. A block diagram for the observation model is illustrated in Fig. 8.

FIGURE 7. Typical shape of Hurter – Driffield curve.

FIGURE 8. The block diagram of the observation model.

(6)

The nonlinear function f includes the characteristics of image capture devices and quantization process during the image acquisition. Generally, the function f for an 8-bit system has the form as

f ðzÞ ¼ 0 for z [ ½0; I0; m for z [ ðIm1; Im; 255 for z [ ðI254; 1Þ; 8 < : ð2Þ

where Im21is the lower bound and Imis the upper bound of the

mth quantization interval. To reconstruct the HR and HDR images based on the proposed image formation model, the inverse function of f is required. Even if it is a reasonable assumption that the response function of an imaging system

is monotonically increasing, the inverse function f21 does

not exist because f is a many-to-one mapping due to the quan-tization process. Therefore, the reverse mapping function g is defined as the function, which maps the pixel intensity to the mid-level of the corresponding quantization interval, and

used instead of f21. It is a reasonable assumption that g is

monotonically increasing and differentiable.

With the definition of the reverse mapping function g and Taylor series expansion, Equation (1) can be rewritten as

gðy_kÞ ¼gð f ðDtkWkx þ nakÞ þ n a kÞ ’ DtkWkx þ nakþg 0_ðy k n q kÞ n q k; ð3Þ

where g0is the first-order derivative. The approximate equality

of the second line holds by ignoring the higher-order terms

above the second order. Because, the quantization error nk

q is relatively small compared with yk, it is reasonable that Equation (3) is approximated as gðykÞ’ DtkWkx þ nakþg 0_ðy k n q kÞ n q k; ’ DtkWkx þ nakþg 0_ðy kÞ n q k ¼DtkWkx þ ntk; ð4Þ

where nkt ¼ nkaþg0(yk).nkqis the total noise term including the

additive noise and quantization error. Accurately

characteriz-ing the total noise nkt would be extremely difficult, as it would

require detailed knowledge of the specific image capture device being used. However, with the general assumption

that the additive noise nka and the quantization error nkqare

the independent zero-mean Gaussian random vectors whose

elements have common variances sa2 and sq2, respectively,

the total noise of the ith pixel in the kth frame also has the

zero-mean Gaussian distribution with the variance sk,i2 ¼

sa2þg0(yk,i)2.sq2. Therefore, the total noise nkt is modelled as

zero-mean independent Gaussian random variables with covariance matrix Cn, which is a diagonal matrix, and its

diag-onal elements are equal to sk,i2.

3.2. Reconstruction of the HR and HDR images and the

reverse mapping function

Because the reverse mapping function g is unknown as well as the HR and HDR images x, x and g should be simultaneously reconstructed. To solve this problem, Gauss – Seidel relaxation is used to determine the solution for both x and g. Gauss – Seidel relaxation minimizes an objective function with respect to a single variable, and then uses these values when minimizing with respect to subsequent variables. Using the total noise model and a Gaussian prior for x with mean

vector mxand covariance matrix Cx, the problem is formulated

as finding MAP estimates of the reverse mapping function g and the HR and HDR images x that minimize the objective function given as EðxÞ ¼X p k¼1 ½gðy_kÞ DtkWkxTC1n ½gðykÞ DtkWkx þ ðx mxÞTC1x ðx mxÞ: ð5Þ

With the assumption that there is no spatial correlation among

pixels in x, Cxis a constant diagonal matrix of which diagonal

elements are equal to the local variances of the x.

First, the gradient descent method is used to solve for x minimizing Equation (5). The estimated x at the nth gradient descent iteration is

xðnÞ¼ xðn1ÞarEðxðn1ÞÞ; ð6Þ

where a is the step size, and rE(x) is derived from Equation (5) as rEðxÞ ¼ X p k¼1 DtkWTkC 1 n ðgðykÞ DtkWkxÞ þ C1_x ðx mxÞ: ð7Þ

The step size a of the gradient descent can be fixed or updated adaptively during the iterations.

By Gauss – Seidel relaxation, the reverse mapping function g is reconstructed with the solution x of the gradient descent minimization. Generally, reconstruction of g only requires recovering the finite number of values, since the values of the pixel brightness are finite. For example, the 256 values g(z) for z ¼ 0, . . . , 255 are needed to be recovered for 8-bit system. However, because Hurter – Driffield curve is a simple S-curve and g is the reverse mapping of Hurter – Driffield curve including the quantization, g can be reconstructed using the polynomial fitting method. Let us assume that g is the third-order polynomial function with coefficients a0, a1,

T C J

(7)

a2and a3defined as

gðy_kÞ ¼a0i þ a1ykþa2yð2Þk þa3yð3Þk ; ð8Þ

where i is the M 1 vector, which elements are all 1, yk(2)¼

[yk,12 , yk,22 , . . ., yk,M2 ]T, y(3)k ¼ [yk,13 , yk,23 , . . ., yk,M3 ]T, for k ¼ 1,

2, . . . , p. There are two additional constraints to make g main-tain the shape of typical inverse Hurter – Driffield curve. The one is that g(z) should be a monotonically increasing function within the interval 0 z 255. The other one is that g(128) should be zero. With Equation (8), Equation (5) can be rewritten as EðxÞ ¼X p k¼1 ½ða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞT C1_n ða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞ þ ðx mxÞTC1x ðx mxÞ: ð9Þ

To minimize Equation (9) with respect to a0, the partial

derivative of Equation (9) with respect to a0is taken and set

equal to zero as @EðxÞ @a0 ¼X p k¼1 iTða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞ ¼0: ð10Þ

Similarly, the minimizations with respect to a1, a2and a3are

also calculated as @EðxÞ @a1 ¼X p k¼1 yT_kða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞ ¼0; ð11Þ @EðxÞ @a2 ¼X p k¼1 ðyð2Þ_k ÞTða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞ ¼0 ð12Þ and @EðxÞ @a3 ¼X p k¼1 ðyð3Þ_k ÞTða0i þ a1ykþa2yð2Þk þa3yð3Þk DtkWkxÞ ¼0: ð13Þ

From Equations (10) – (13), the polynomial coefficients are solved as a0 a1 a2 a3 0 B B @ 1 C C A¼ A1 Pp k¼1DtkiTWkx Pp k¼1DtkyTkWkx Pp k¼1Dtkðy2kÞ T_W kx Pp k¼1Dtkðy3kÞ T_W kx 0 B B @ 1 C C A; ð14Þ

where A is a 4 4 matrix of which the ijth element eij is

expressed as

eij¼

Xp

k¼1

ðyðiÞ_kÞTyð_kjÞ for i; j ¼ 1; . . . ; 4; ð15Þ

where yk(0)¼ i and yk(1)¼ yk. The matrix A is calculated only

once before the iteration process. With the calculated poly-nomial coefficients, g is reconstructed, and another gradient descent iteration is started with the reconstructed g by Gauss – Seidel relaxation.

3.3. Algorithm for implementation

In the reconstruction, the other parameters but the HR and HDR images x and the reverse mapping function g should

be either known or estimated in advance. The matrix Wk

requires information about the subsampling factor, the point spread function of blurring and the motion information with

subpixel precision. The exposure time Dtkis another required

parameter. In this paper, the subsampling factor, the point spread function, the motion information and the exposure times of each frames are assumed to be known values. The algorithm starts with initial estimates of x and g. The algor-ithm for implementation is shown below.

(i) Initialization of x(0)with a bilinearly interpolated LR

image and LDR image.

(ii) Calculation of the matrix A21.

(iii) Polynomial fitting as calculating coefficients a0, a1, a2

and a3in g(n)with x(n21).

(iv) Gradient descent iteration for the reconstruction of x(n)

with g(n).

(v) If the stopping criterion is satisfied, terminate iter-ations. If not, go to (iii).

4. SIMULATION RESULTS

In order to demonstrate the performance of the algorithm, several experimental results are presented here. The simu-lations are performed on two approaches. The first experiment is the reconstructing an HR image and HDR image from mul-tiple LR and LDR images which are shifted, blurred and downsampled from the LDR images with different exposure times. In order to measure the quantitative improvement in the reconstructed images, the other experiment is the

(8)

reconstructing an HR image and HDR image from multiple LR and LDR images which are shifted, blurred, downsampled and quantized into 6-bit grey level from one 8-bit gray level image. Since g converges very fast in comparison with the

convergence of x, the criterion kxnþ12 xnk2/kxnk21026

was used for terminating the iteration in both approaches. To acquire 16 LR and LDR images, each frame of four 492 708 ‘memorial’ images as shown in Fig. 9 are shifted with four different subpixel motion information, blurred with 3 3 Gaussian kernel whose variance is 1.0 and down-sampled with a factor of 2 in each dimension. The reverse mapping function g is simultaneously estimated with the reconstruction of a single image with HR and HDR. Figure 10 presents the estimated g by the the conventional method and the proposed method from ‘memorial’ images. Conventional method proposed by Robertson et al. [30] recon-structs the 256 values g(z) for z ¼ 0, . . ., 255 for an 8-bit system in advance. Compared with the result of conventional method, the estimated g by the proposed algorithm well main-tains the typical form of the inverse Hurter – Driffield curve. Since g takes the role of mapping pixel values to HDR radi-ance values, the fluctuation of g can cause the inversion of the local contrast of the reconstructed HR and HDR images. Since the reconstructed g of the proposed algorithm is

smoother than the result of the conventional method, the pro-posed method reconstructs a more accurate HR image and HDR image.

The reconstructed HR image and HDR ‘memorial’ image is shown in Fig. 11. Since the reconstructed values are not inte-gers but floating point numbers that exceed the dynamic range of the traditional image format, it cannot directly be displayed. In this experiment, the dynamic range of the reconstructed HR FIGURE 9. Four LDR “memorial” images.

FIGURE 10. The reconstructed reverse mapping function using the conventional and the proposed method.

FIGURE 11. The reconstructed HR and HDR images using the proposed method.

T C J

(9)

and HDR images was compressed with a simple log-scale mapping. The proposed algorithm is compared with two con-ventional algorithms. The first concon-ventional method sequen-tially enhances the dynamic range and spatial resolution, and the other conventional method simultaneously enhances. First, the four LR and HDR images were reconstructed from four LR and LDR images with the same subpixel motion infor-mation. Then, HR image is reconstructed with a conventional SR image reconstruction, which uses MAP estimates based on deterministic regularization approach [13] (CM1). The other conventional method simultaneously enhances the spatial res-olution and the dynamic range with reconstructed imaging system’s response function in advance [32] (CM2). The par-tially magnified results are shown in Fig. 12. Figure 12a shows a frame of LR images (nearest neighbour interpolated). Figure 12b and c shows the results of CM1 and CM2, respect-ively. Figure 12d is the reconstructed HR and HDR images of which the spatial resolution and the dynamic range are simul-taneously enhanced by the proposed methods. The proposed reconstruction method shows improved image details com-pared to the results of CM1 and CM2.

The other experiment is the reconstruction from multiple LR and LDR images, which are generated with an original 8-bit grey level image. The 36 LR and LDR images were pro-duced from an 8-bit grey level image. Figure 13a and b is the original images of ‘desk’ and ‘sunrise’, respectively. The orig-inal image is shifted with nine different subpixel motion

information, blurred with 3 3 Gaussian kernel, down-sampled with a factor of 4 in each dimension, and re-quantized to 6-bit grey level with four different exposure times. Figure 14 shows the examples of the produced differently exposed LR ‘desk’ images.

The reconstructed HR and HDR ‘desk’ and ‘sunrise’ image are presented in Fig. 15 and the partially magnified results of ‘desk’ image are shown in Fig. 16. Figure 16a shows one nearest interpolated frame of LR images. Figures 16b and c shows the result of CM1 and CM2, respectively. Figure 16d is the simultaneously reconstructed HR and HDR images by the proposed methods without prior knowledge about the

FIGURE 12. Partially magnified results of memorial using the con-ventional and the proposed methods. (a) One frame of LR images (nearest neighbour interpolated). (b) The result of CM1. (c) The result of CM2. (d) The result of the proposed method.

FIGURE 13. The original HR and HDR images. (a) Desk image. (b) Sunrise image.

FIGURE 14. Four examples of LR and LDR desk images.

(10)

imaging system. The proposed method shows improved image details compared to the CM1 and CM2. In order to measure the quantitative improvement in the estimated images, the peak signal-to-noise ratio (PSNR) values for results with desk and sunrise image are measured. The PSNR is defined in decibels as

PSNR ¼ 10 log10

2552_N

k^x xk2 ; ð16Þ

where N is the total number of the pixels in the image, x the original image and xˆ the reconstructed image. In Table 1, the PSNR values calculated between the reconstructed images and the original image are shown in order to compare the performance of the proposed algorithm to the conventional methods. The PSNR of the proposed algorithm

improves 4 and 1.5 dB over the conventional methods,

respectively.

5. CONCLUSION

HR and HDR images can be used in many fields. Since an HR image and HDR image has more detail and light intensity information than an LR and LDR image, it is very useful in the fields such as biometrics, medical imaging and astronom-ical imaging. Spatial resolution and dynamic range enhance-ment based on SR approach has the advantage that it can be applied to the existing imaging devices in relatively low price as compared with the methods to manufacture the new architecture of imaging sensors.

We have proposed an SR-based method for the reconstruc-tion of an HR image and HDR image from multiple LR and LDR frames with different exposures. The image degradation process including limited spatial resolution and limited dynamic range is modelled and the MAP estimates of both the system response function and a single HR image and HDR image are simultaneously obtained. The system response function is simultaneously estimated in every iteration step with the reconstruction of a single HR image and HDR image. With the simulation results, it is shown that the pro-posed algorithm outperforms the the conventional approaches FIGURE 15. The reconstructed HR and HDR images using the

proposed method. (a) Desk image. (b) Sunrise image.

FIGURE 16. Partially magnified results of desk using the conven-tional and the proposed methods. (a) One frame of LR images (nearest neighbour interpolated). (b) The result of CM1. (c) The result of CM2. (d) The result of the proposed method.

TABLE 1. PSNR of the reconstructed images

PSNR (dB) CM1 CM2 Proposed

Desk 35.64 35.56 39.37

Sunrise 37.28 37.31 38.78

T C J

(11)

with respect to both objective and subjective criteria. Simu-lation results verify that the proposed algorithm estimates the reverse mapping function g without any prior knowledge about the system response function as well as an HR image and HDR image.

ACKNOWLEDGEMENTS

This work was supported by the MIC (Ministry of Information and Communication), Korea, under the ITRC (Information Technology Research Center) support program supervised by the IITA (Institute of Information Technology Assessment).

REFERENCES

[1] Kang, M.G. (2003) SPIE Milestone Series Vol. MS177 Selected Papers on CCD and CMOS Imagers. SPIE Press, Washington. [2] Komatsu, T., Aizawa, K., Igarashi, T. and Saito, T. (1993) Signal-processing based method for acquiring very high resolution image with multiple cameras and its theoretical analysis. IEE Proc. Commun. Speech Vision., 140, 19 – 24. [3] Park, S.C., Park, M.K. and Kang, M.G. (2003) Super-resolution

image reconstruction: a technical overview. IEEE Signal Process. Mag., 21 – 36.

[4] Clark, J.J., Palmer, M.R. and Laurence, P.D. (1985) A transformation method for the reconstruction of functions from nonuniformity spaced samples. IEEE Trans. Acoust. Speech Signal Process., 33, 1151 – 1165.

[5] Kim, S.P. and Bose, N.K. (1990) Reconstruction of 2-D bandlimited discrete signals from nonuniform samples. IEE Proc. Radar Signal Process., 137, 197 – 204.

[6] van Viley, L.J. and Hendriks, C.L.L. (1999) Improving spatial resolution in exchange of temporal resolution in aliased image sequences. Proc. 11th Scandinavian Conf. Image Analysis, pp. 493 – 499. Kangerlussnag, Greenland.

[7] Tsai, R.Y. and Huang, T.S. (1984) Multipleframe image restoration and registration. Adv. Comput. Vision Image Process., 1, 317 – 339.

[8] Kim, S.P., Bose, N.K. and Valenzuela, H.M. (1990) Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Trans. Acoust., Speech Signal Process., 38, 1013 – 1027.

[9] Kim, S.P. and Su, W.Y. (1993) Recursive highresolution reconstruction of blurred multiframe images. IEEE Trans. Image Process., 2, 534 – 539.

[10] Rhe, S.H. and Kang, M.G. (1999) Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Opt. Eng., 38, 1348 – 1356.

[11] Hong, M.C., Kang, M.G. and Katsaggelos, A.K. (1997) An iterative weighted regularized algorithm for improving the resolution of video sequences. Proc. Int. Conf. Image Processing, Vol. 2, 474 – 477. Santa Barbara, USA.

[12] Kang, M.G. (1998) Generalized multichannel image deconvolution approach and its applications. Opt. Eng., 37, 2953 – 2964.

[13] Hardie, R.C., Barnard, K.J., Bognar, J.G., Armstrong, E.E. and Watson, E.A. (1998) High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Opt. Eng., 37, 247 – 260.

[14] Stark, H. and Oskoui, P. (1989) High resolution image recovery from image-plane arrays, using convex projections. J. Opt. Soc. Am., 6, 1715 – 1762.

[15] Eren, P.E., Sezan, M.I. and Tekalp, A.M. (1997) Robust, objectbased high-resolution image reconstruction from low-resolution video. IEEE Trans. Image Process., 6, 1446 – 1451.

[16] Patti, A.J. and Altunbasak, Y. (2001) Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Trans. Image Process., 10, 179 – 186.

[17] Schulz, R.R. and Stevenson, R.L. (1996) Extraction of a high-resolution frames from video sequences. IEEE Trans. Image Process., 5, 996 – 1011.

[18] Cornsweet, T.N. (1970) Visual Reception. Academic, New York, NY.

[19] Handy, R.J. (1986) High dynamic range CCD detector/imager. U. S. Patent 4623928.

[20] Street, R.A. (1998) High dynamic range segmented pixel sensor array. U. S. Patent 5789737.

[21] Brajovic, V. and Kande, T. (1993) A sorting image sensor: an example of massively parallel intensity-to-time processing for low-latency computational sensors. Proc. Int. Conf. Robotics and Automation, 1996, September, Vol. 2, 1638 – 1643. [22] Debevec, P.E. and Malik, J. (1997) Recovering high dynamic

range radiance maps from photographs. SIGGRAPH 97 Conf. Proc., August 3 – 8, 369 – 378.

[23] Technical Report MS-CIS-93-96 (1993) Extended Intensity Range Imaging. Grasp Laboratory, University of Pennsylvania. [24] Doi, H., Hara, Y., Kenbo, Y. and Shinba, M. (1986) Image

sensor. Japnese Patent 08-223491.

[25] Aggarwal, M. and Ahuja, N. (2001) Split aperture imaging for high dynamic range. Proc. IEEE Int. Conf. Computer Vision, 2001. (ICCV 2001), July, Vol. 2, pp. 10– 17. Vancouver, Canada. [26] Nayar, S.K. and Mitsunaga, T. (2000) High dynamic range imaging: spatially varying pixel exposures. Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000, June, Vol. 1, 472 – 479. Hilton Head Island, USA.

[27] Yamada, K., Nakada, T. and Yamamoto, S. (1994) Wide Dynamic Range Vision Sensor for Vehicles. Proc. IEEE Conf. Vehicle Navigation and Information Systems Conference, 1994, August, pp. 405 – 408. Yokohama, Japan. [28] Tani, T. (1995) Photography Sensitivity: Theory and

Mechanisms. Oxford University Press.

[29] Kolb, C., Mitchell, D. and Hanrahan, P. (1995) A realistic camera model for computer graphics. Computer Graphics, Proc. SIGGRAPH ’95, pp. 317 – 324. Los Angeles, USA.

(12)

[30] Robertson, M.A., Borman, S. and Stevenson, R.L. (1999) Dynamic range improvement through multiple exposures. Proc. IEEE Int. Conf. Image Processing, 1999. (ICIP 99), October, Vol. 3, pp. 159 – 163. Kobe, Japan.

[31] Narasimhan, S.G. and Nayar, S.K. (2005) Enhancing resolution along multiple imaging dimensions using

assorted pixels. IEEE Trans. Pattern Anal. Mach. Intell., 27, 518 – 530.

[32] Gunturk, B.K. and Gevrekci, M. (2006) High-resolution image reconstruction from multiple differently exposed images IEEE Signal. Process. Lett., 13, 197 – 200.

T C J