Dynamic Foreground Cases

In this section, we test the proposed method in the dynamic foreground cases, where the task is that given two frames of input images I andI0_{, recover four component} layers L1,L01,L2,L02, and two dense motion fieldsU, V. In the problem of reflection removal, both the background scene and the reflection can be dynamic, which can give rise to such a situation.

We use two pairs of dynamic reflection scenes from [Li and Brown, 2013] to test the proposed method on the double-layer optical flow problem. In previous single- flow experiments, we initialize the method with foreground layers being all zero. However, this simple strategy did not work for the double-flow case. No reasonably good flow field could be obtained with this strategy for the background or reflection layer, especially for the reflection layer as its signal is weak. Indeed, the fact that the background layer is much more prominent has been taken advantage of by some layer separation methods Li and Brown [2013]; Guo et al. [2014] which align the input images with respect to the background layer. To obtain proper initialization, we first ran method of [Li and Brown, 2013] for initial layer separations3_{, then computed} initial optical flows on them.

3_{The method of [Li and Brown, 2013] takes multiple images as input, with one of them being the} reference on which the reflection is to be removed. We apply this method on two images, and run it twice with each image as reference.

InputI InitialU FinalU

InputI0 InitialV FinalV

InputI InitialU FinalU

InputI0 InitialV FinalV

Figure 6.7: Double-layer optical flow estimation results on real reflection images. Visually inspected, the final optical flow fields are smoother and more consistent (see e.g., the results on the back wall in the first example, and results on the floor in the second example). The corresponding warping errors are presented in Table 6.2.

(Best viewed on screen)

Table 6.2: Mean image warping errors (in gray levels) from the double-flow estima- tion results.

Image pair Initial results Our final results

#1 6.27 2.55

§6.6 Conclusion 121

The initial and final results are presented in Figure 6.7. Visually inspected, the final optical flow fields are smoother and more consistent (see e.g. the results on the back wall in the first example, and results on the floor in the second example). As no ground truth optical flow is available, we use image warping error to quan- titatively evaluate the estimated flows. The warping error for a pixel x in L1 or L2 is kL1(x+U(x))−L01(x)k2 or kL2(x+V(x))−L02(x)k2, respectively. We compute the mean warping errors for all pixels onL1 andL2. As shown in Table 6.2, our method has significantly reduced the warping error upon the initializations. Figure 6.8 and 6.9 show the improvements of the reflection removal upon the initial estimates. Discussion. The dynamic foreground case with double-layer flow estimation is generally much harder than the single-flow case. This is not only because the former has more unknown variables to be solved for, but also due to the difficulties in ob- taining a good initialization. Nevertheless, our experiments show that the proposed method consistently improved the reasonable initializations given to it, for both the single-flow and double-flow cases.

Limitation. The proposed method is better suited for scenarios where the correla- tion between latent layers and their flow fields are relatively small. It will fail if both the two layers are textureless (as infinite numbers of possible motions exist satisfying the BCC constraints), or they undergo the same motion (thus the original BCC holds and only a single motion field can be extracted).

6.6

Conclusion

This chapter has defined the problem of robust optical flow estimation in the presence of possibly moving transparent or reflective layers. To our knowledge, the problem goes beyond the scope of conventional optical flow methods and was not properly investigated before.

We have presented a generalized double-layer brightness constancy condition as well as an optimization framework to solve this problem. The double-layer brightness constancy condition couples the flow fields and the brightness layers. Encouraging experimental results of optical flow estimation and layer separation on challenging data have been obtained, even though we are using simple priors for them.

The current framework is based on a generative model, which is applied uni- formly to both the foreground and background layers. In future, we plan to leverage discriminative models to exploit the differences between the two layers for better layer separation. We also would like to explore some other optical flow priors. One possible strategy is to apply piecewise parametric motion model proposed in Chap- ter 5, which provides stronger constraints than general smoothness regularizers such as a TV, and is demonstrated to have advanced performances. Some other issues such as occlusion handling could also be considered.

InitialL1 FinalL1

Close-up of initialL1 Close-up of finalL1

InitialL0 1 FinalL01 Close-up of initialL0 1 Close-up of finalL01 InitialL2 FinalL2 InitialL0 2 FinalL02

Figure 6.8: Layer separation results on real reflection images (the 1st pair). The initial layer separations are estimated by running method of [Li and Brown, 2013] on the two input images. The corresponding warping errors are presented in Table 6.2. The close-up images show the improvements of the reflection removal results upon the initial estimates. (Best viewed on screen)

§6.6 Conclusion 123

InitialL1 FinalL1

Close-up of initialL1 Close-up of finalL1

InitialL0 1 FinalL01 Close-up of initialL0 1 Close-up of finalL01 InitialL2 FinalL2 InitialL0 2 FinalL02

Figure 6.9: Layer separation results on real reflection images (the 2nd pair). The initial layer separations are estimated by running method of [Li and Brown, 2013] on the two input images. The corresponding warping errors are presented in Table 6.2. The close-up images show the improvements of the reflection removal results upon the initial estimates. (Best viewed on screen)

Chapter7

Summary and Future Work

Motion estimation is one of the fundamental problems in computer vision which has broad application. The studies of camera and image motion have started since the emergence of the computer vision field. However, motion estimation remains an active topic nowadays with many challenging problems yet to be solved, as we have shown in the previous chapters.

7.1

Summary and Contributions

This dissertation has been devoted to analyzing the current challenges and push the limits of the state-of-the-art in various aspects, such as optimality, robustness, accuracy, and flexibility. A summary of the contributions is given below.

Optimality for 3D point cloud registration and 3D camera motion estimation (Chapter 2) and 2D color camera relative motion estimation (Chapter 3). We have proposed the first globally optimal algorithm for the ICP-style 3D point cloud regis- tration problem and applied it to the motion estimation of 3D imaging devices. The idea is to analyze the structure of the SE(3)geometry and derive the error bounds for

Branch-and-Bound (BnB) optimization. Similarly, we also have proposed a globally optimal inlier-set maximization algorithm for color camera relative motion estima- tion. We achieve this by analyzing the structure of the 5-D essential manifold, and presenting a new parameterization which enables efficient BnB search. The two BnB- based methods are actually highly related to each other with the similar insights in bound derivation.

Robustness for 2D color camera relative motion estimation (Chapter 3) and im- age motion estimation in the presence of transparency or reflection (Chapter 6). To deal with outliers/feature mismatches in 2D camera motion estimation, we have for- mulated an inlier-set maximization problem as in the popular RANSAC algorithm, but solved it optimally via BnB. Experiments have shown that our method always finds more inliers than RANSAC, and can work under high outlier ratio especially for the wide-FOV cases. To achieve robust image motion estimation under trans- parency or reflection, we have proposed an algorithm which performs both optical flow estimation and image layer separation. It exploits a generalized double-layer

brightness consistency constraint connecting these two tasks and utilizes the priors for both of them. In this way, not only the robustness is achieved as shown in the experiments, but also clean background images are restored which are appealing for other vision tasks.

Accuracy for classical image motion estimation (Chapter 5). We have proposed a highly-accurate optical flow estimation algorithm based on a piecewise parametric motion model. A key innovation is that we fit a flow field piecewise to a variety of parametric models where the domain of each piece (i.e., shape, position and size) and adaptively determine model parameters, while at the same time maintaining a global inter-piece flow continuity constraint. The proposed algorithm has archived top-tier performances on three public optical flow benchmarks (KITTI, MPI Sintel, and Middlebury).

Flexibilityfor 2D color camera and 3D camera relative motion estimation (Chap- ter 4). Existing works for the 2D color camera and 3D camera relative motion esti- mation often involve cumbersome human intervention and lack flexibility (e.g., for on-site estimation). In this dissertation, we have developed a single-shot method and provided a corresponds-free solution in order to minimize human intervention. We make use of known geometric constraints from the scene, and formulate relative pose estimation as a 2D-3D registration problem minimizing the geometric errors from scene constraints. The experiments have shown that the method is both flexible and accurate.