Multi-view image coding

We finally evaluate the performance of our correlation estimation algorithms in scenarios with more than 2 correlated images. We use five images from the Tsukuba dataset (center, left, right, bottom and top views), and five frames (frames 3-7) from the Flower Garden sequence [150]. These datasets are down-sampled by a factor 2 and the resolution of these datasets used in our experiments is of 144× 192 and 120 × 180 respectively. In both datasets, the reference image I1 (center view and frame 5 resp.) is encoded with a

quality of approx. 33 dB. The measurements Y_i,∀i ∈ {1, 2, 3, 4} computed from the remaining four images

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 24 25 26 27 28 29 30 31 32 33 Rate (bpp) PSNR (dB) OPT−2:Graph Cuts OPT−2: Local Opt. OPT−1:Graph Cuts OPT−1: Local Opt.

Figure 5.19: Comparison of the RD performances between the global and local optimization methodologies for the Foreman dataset.

is estimated with OPT-2 between the center and left images (in Tsukuba), and frames 5 and 6 (in Flower Garden) respectively. Fig. 5.20 compares the inverse depth error (sum of the labels with error larger than one with respect to groundtruth) between the multi-view and disparity scenarios. In this particular experiment, the parameters α1and α2are selected such that they minimize the error in the depth image with respect to

the groundtruth. It is clear from the plot that the depth error is small for a given measurement rate when all the views are available. It should be noted that the x-axis in Fig. 5.20 represents the measurement rate per view, and hence the total number of measurements used in the multi-view scenario is higher than for the stereo case. However, these experiments show that the proposed multi-view scheme gives a better depth image when more images are available. Fig. 5.21(b) and Fig. 5.21(d) show the depth map estimated using the OPT-3 (multi-view) and OPT-2 (stereo) algorithms respectively from 3% quantized measurements (per view) for the Flower Garden sequence. As the groundtruth for this dataset is not available, we estimate the depth using the original images and it is available in Fig. 5.21(a) for visual comparison. From Fig. 5.21(b) and Fig. 5.21(d) we see that the proposed multi-view scheme estimates a better depth map than the stereo setup, as it is more accurate in the background and tree regions. When this coarse depth map is used for predicting frame 6, we see from Fig. 5.21(b) and Fig. 5.21(e) respectively that the prediction is better when five views are used for depth estimation. We then study the RD performance of the proposed multi-view scheme in the decoding of four images (top, left, right, bottom images in Tsukuba, and frames 3, 4, 6, 7 in Flower Garden). The images are decoded by warping the reference image ˆI1 using the estimated depth

map. Fig. 5.22 compares the overall RD performance (for 4 images) of our multi-view scheme with respect to independent coding performance based on JPEG 2000. As expected, the proposed multi-view scheme outperforms independent coding solutions based on JPEG 2000, as it benefits from the correlation between images. Note that the JPEG 2000 codec is state-of-the-art coding solutions for independent compression though it has not been optimized for low resolution images. Furthermore, as observed in distributed stereo and video coding the proposed multi-view coding scheme saturates at high rates, as the warping operator captures only the geometry and correlation between images and not the detail and texture information.

Finally, we compare our results with the joint encoding approach where the depth image is estimated from the original images and the computed depth image is transmitted to the joint decoder. At the decoder, the views are predicted from the reconstructed reference image ˆI1 and the compressed depth image based

on view prediction. The results are presented in Fig. 5.22 (marked as Joint encoding), where the bit rate is computed only on the depth image encoded using the JPEG 2000 coding solution. The main difference between our scheme and the joint encoding is that the quantized linear measurements are transmitted for a depth estimation in the former scheme, while the depth information is directly transmitted in the latter scheme. Therefore, by comparing these two approaches we can fairly judge the accuracy of the estimated

5.7 Experimental results 89 0 5 10 15 20 25 30 35 10 12 14 16 18 20 22 24 26 28 30

Measurement rate per view

Inverse Depth Error

OPT−2 (Stereo) OPT−3 (Multiview)

Figure 5.20: Inverse depth error at various measurement rates for the Tsukuba multi-view dataset. OPT-2 and OPT-3 are used to estimate the depth in stereo and multi-view scenarios respectively. The measurements are quantized using a 2-bit quantizer.

(a) (b) (c) MSE: 136.99 (d) (e) MSE: 167.85

Figure 5.21: Comparison of the qualities of the depth image and the warped image ˜I2 estimated in multi-view and stereo scenarios in the Garden dataset: (a) depth image Mh estimated using the original images; (b) depth imagemhestimated in the multi-view scenarios using OPT-3; (c) inverted prediction error1 − |˜I2− I2| when depth image in (b) is used for prediction; (d) depth imagemhestimated in the stereo scenario using OPT-2; (e) inverted prediction error1 − |˜I2− I2| when depth image in (c) is used for prediction. The depth image is estimated using a measurement rate of 3% (per image) with 2-bit quantized measurements.

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 19 20 21 22 23 24 25 26 Total Rate (bpp) Avg. PSNR (dB) Tsukuba: OPT−3 Tsukuba: JPEG 2000 Tsukuba: Joint Garden: OPT−3 Garden: JPEG 2000 Garden: Joint Encoding

Figure 5.22: Comparison of the overall RD performances between the proposed scheme and the independent coding scheme based on JPEG 2000. Five images are used in both datasets to estimate a depth image using OPT-3. The bit rate of the reference image I₁ is not included in the total bit budget.

correlation model or equivalently the quality of the predicted view at a given bit rate. From Fig. 5.22 we see that at low rates < 0.2, the proposed scheme estimates a better structural information compared to the joint encoding scheme, thanks to the geometry-based correlation representation. However at rates above 0.2, we see that the proposed scheme becomes comparable with the joint encoding approach, leading to the conclusion that the proposed scheme effectively estimates the depth information from the highly compressed quantized measurements. It should be noted that in joint encoding approach the depth images are estimated at a central encoder. In contrary to this, we estimate the depth images at the central decoder from the independently compressed visual information and therefore our encoder is very simple.

5.8

Conclusions

In this chapter we have contributed a novel framework for the distributed representation of correlated images with quantized linear measurements along with novel algorithms for estimating the geometric correlation between images at the joint receiver. We have proposed a correlation model based on local transformations of geometric patterns that are present in the sparse representation of images. The motion or depth information in distributed video coding and respectively multi-view imaging can then be estimated by matching the corresponding geometric features in different compressed images. We have proposed a new regularized optimization problem in order to identify the geometrical transformations that result in smooth motion or depth fields between a reference image and one or more predicted image(s). Furthermore, we have proposed an improved image warping solution that leads to image prediction that is consistent with the compressed measurements.

Experimental results demonstrate that the proposed methodology provides a good estimation of dense disparity/depth or motion fields in different natural image datasets. We also show that our geometry-based correlation model is more efficient than block-based correlation models. Then, the consistency term proves to offer improved image prediction quality, such that the proposed algorithm outperforms JPEG 2000 and DSC schemes in terms of rate-distortion performance computed on the quantized linear measurements. Finally, we show experimentally that our low complex distributive correlation estimation scheme competes with the solution of the scheme that estimates a correlation model in centralized settings. Therefore, our scheme certainly provides an interesting alternative to distributed image processing applications due to a framework based on effective handling of the geometry that is one of the main component of correlated natural images. Note that the experiments are carried out using the datasets with QCIF resolution images. We left to extend our scheme to large image resolution, which is one of the interesting future perspectives

5.8 Conclusions 91

of this thesis. In Chapter 7, we propose a joint reconstruction methodology that estimates the missing high frequency texture information of the reconstructed image from the quantized linear measurements. In the next chapter, we extend the scenario to the symmetric case, where we propose to estimate the correlation model directly from the quantized linear measurements.

Chapter 6

Correlation Estimation from

Compressed Linear Measurements

6.1

Introduction

In the previous chapter, we have proposed an algorithm to estimate the geometrical relationship between correlated images from a reference image and compressed linear measurements. In this chapter, we propose to estimate the correlation between images in a symmetric framework, where all the sensors transmit com- pressed images that have been obtained by a small number of linear projections of the original images, as illustrated in Fig. 6.1. Such linear projections typically represent simple measurements in low complexity sensing systems [15, 16].

We propose a novel solution for the correlation estimation at the joint decoder1, where the analysis is performed directly in the compressed domain to avoid expensive image reconstruction tasks. Recall that state-of-the-art distributed schemes estimate the correlation model from the reconstructed images, which are typically obtained by solving an l2-l1 or l2-T V optimization problem in the compressed sensing

framework. Unfortunately, reconstructing the reference images based on solving optimization problems are highly complex. Also, in image analysis applications (e.g., object detection) it is sufficient to estimate only the correlation model and the explicit reconstruction of images is often not required. Hence, we propose to estimate the correlation model directly from the quantized linear measurements by building a pixel-wise geometrical model between images.

We assume that the correlation between images corresponds to camera or object motion, which can be efficiently represented by a dense disparity or motion field model. We show that such a correlation model can be described by a linear operator and we further analyze in detail the effect of such an operator in the compressed domain. Later, we cast the correlation estimation problem as a regularized energy minimization problem with constraints on data consistency and on consistency of the motion field. In particular, we regularize the correlation model such that the motion values in the neighboring pixels are similar except at image discontinuities. Such an optimization problem can be solved using a Graph Cut algorithm. Finally, we propose robust solutions to the correlation estimation by effectively considering the non-linearities due to quantization of measurements.

We then analyze in details the performance of our novel correlation estimation framework. In particular, we study the penalty in the correlation estimation that is due to working in the compressed domain as opposed to the original image domain as in traditional correlation estimation problems. We show that the penalty decreases when the number of measurements increases and that our algorithm tends to find the 1_{Part of this work has been accepted to: V. Thirumalai and P. Frossard, “Correlation Estimation from Compressed Images,”}

ENCODER LINEAR MEASUREMENTS I₁ Y1 LINEAR MEASUREMENTS I2 Y2 QUANTIZER QUANTIZER Yˆ1 ˆ Y₂ CORRELATION ESTIMATION DECODER (mh_,mv₎ ˆ Y₂ ˆ Y₁

Figure 6.1: Schematic representation of the proposed scheme. The images I1 and I2 are correlated through displacement of the scene objects due to view point change or motion of scene objects. The correlation model is directly estimated in the compressed domain without any intermediate image reconstruction steps.

optimal correlation solution at high measurement rates. Extensive simulations in distributed multi-view and video imaging applications confirm the effective estimation performance of our algorithm. Finally, we show experimentally that our correlation estimation algorithm properly handles the quantization noise in the measurements.

The rest of this chapter is organized as follows. In Section 6.2, we describe the proposed framework and show how the correlation estimation problem carries out to the compressed domain. Section 6.3 describes the proposed correlation estimation algorithm; its performance are analyzed in details in Section 6.4. Section 6.5 extends our correlation framework to quantization scenarios. Finally, in Section 6.6 we describe extensions to multi-view imaging frameworks.