In this chapter, we have proposed a novel radiometric calibration method for estimating the radiometric response function, exposures, and vignetting. By decoupling vignetting from the response function, we can approach each problem with a robust estimation method. The robustness of our method was validated synthetically and also with real examples. Our method accurately estimates the parameters even in the presence of large noise and mismatches including matches from stereo sequence whereas other existing methods were not effective against noise and outliers. We applied the estimation results to radiometrically align images for seamless mosaics and 3D model textures. We also used our method for creating the HDR mosaic which is more representative of the scene
Figure 3.15: HDR mosaic. (First) Original mosaic, (Second) A zoomed-in image added to the mosaic. The exposure and the vignetting function for this image is computed
than the normal mosaic.
Some may question the accuracy of our response function estimation process which uses only corresponding pixels between images that are of equal distance (with some tolerance) from the image center. Even though it may seem like we are using fewer samples than other methods, we are actually using more samples than the existing methods which rely on random sampling of points. For the method of Goldman and Chen (2005), only 1000-2000 samples were used for each sequence shown in this chapter. It is very difficult to use more samples because it will slow down the computation which is already very slow. While more samples can be used for the method of Litvinov and Schechner (2005a), it is still limited due to memory constraints when solving the linear equation. It is important to note that we do not use the pixel values directly to compute the response function as in other methods but rather use them to compute the brightness transfer function by dynamic programming. Given the robustness of this process along with the power of the model (EMoR) we use, the number of points we use in our method is not much of a problem. One case that would be problematic for our method is when the distribution of pixel values is very limited such as when the regions we use are of an uniform color. But this is usually not the case in practice especially since we can easily expand our method to include correspondences from more images, not just the next image in the sequence.
Chapter 4
Joint Feature Tracking and Radiometric
Calibration from Auto-Exposure Video
4.1
Introduction
Extracting and tracking features is a fundamental step in many computer vision sys- tems since it provides means to relate one image to another spatially. One of the most commonly used feature tracker, especially for processing videos, is the KLT (Kanade- Lucas-Tomasi) tracker (Lucas and Kanade (1981); Shi and Tomasi (1994)) due to its robustness and efficiency. However, there are cases that pose problems for the KLT tracker, mainly when images of a high dynamic range scene are captured. In order to capture the full brightness range of natural scenes, where parts are in the shadow and others are in bright sunlight for example, the camera has to adjust the exposure accord- ingly. As a result, the appearance of the same scene point varies throughout the video sequence, breaking the basic assumption for the KLT tracker that the brightness of the scene points stays constant. Hence, we need methods to find radiometric relationships between image features in addition to the spatial relationships.
In this chapter, I introduce a new method that models the changes in image bright- ness between images globally and nonlinearly rather than treating the variation locally and linearly by comparing local regions independently. The brightness change can be
explained by the radiometric response function which defines the mapping from the im- age irradiance to the image brightness. We first introduce a method for tracking features and estimating the exposure changes between frames when the camera’s radiometric re- sponse function is known. In many cases the radiometric response function is not known a priori, so I also present a method for joint feature tracking and radiometric calibration by formulating the estimation of the response function within a linear feature tracking scheme that can deal with varying intensity values of features due to exposure changes. The novel framework presented here performs an integrated radiometric calibration in contrast to previous radiometric calibration techniques (including the method presented in Chapter 3) which require the correspondences as an input to the system which leads to a chicken-and-egg problem as precise tracking requires accurate radiometric calibration. By combining both into an integrated approach we solve this chicken-and-egg problem. The remainder of the chapter is organized as follows. In the next section, a review of the related work will be presented. In Section 4.3, the KLT tracker algorithm will be reviewed. In Section 4.4, a method for tracking features when the response function is known will be introduced first, and then a method for simultaneous tracking and the re- sponse function estimation will be explained. We evaluate our method with experiments in Section 4.5, including an application of the method for an adaptive stereo system, and conclude with discussion about our algorithm in Section 4.6.
This work was originally presented in Kim et al. (2007) and the extended version of the original work is currently under review (Kim et al., 2008a).