Flat Refractive Stereo Systems

FRSS s are one example of refractive systems that are a specialization of the FRS . In

contrast to the approaches from the previous section that are able to calibrate the relative pose between several FRS s additionally to the single FRS itself, it is now mandatory that two FRS s exist. A single FRS can not be calibrated with these approaches. Similarly to Section 3.3.5, calibration approaches that are based on an underlying model for underwater image formation that is restricted occur in this category as well. Kang et al. [KWY12b] propose a calibration approach for a FRSS with both image planes being perpendicular to the respective interface normal. Additionally, the layer thickness of the housing is neglected. The intrinsic camera parameters and the refractive indices of the participating media need to be known in advance. The authors use the concept of the refrax (See Section 3.3.1) and develop two new concepts, namely the ellipse of refrax and the refractive depth. These are used in an algorithm for the estimation of the relative translation of the stereo camera, air layer thicknesses and 3D structure. This algorithm is incorporated into a differential evolution approach to search for the best rotation between the two cameras. Afterward, the parameters and the 3D structure are refined by bundle adjustment to minimize the reprojection error. In a secondary version of their work, Kang et al. [Kan+17] extend their bundle adjustment approach by a local sliding procedure for an improvement of the convergence behavior of their optimization framework.

The authors in [Brä+15a;BKN15;Brä+15b] develop an underwater 3D-scanner, which is composed of a projection unit and a FRSS . The authors propose an approach for the calibration of the air layer thickness for each camera. A requirement is that the interfaces are perpendicular to the image planes and that the refractive indices of the participating media are known. The layer thicknesses of the housing need to be known as well and the stereo camera system needs to be calibrated for its intrinsic and extrinsic parameters in advance. The calibration of the air layer thicknesses is realized with the aid of measurements of a plane and an object of known length. Therefore, a systematic search is performed in the range of minimally and maximally possible air layer thicknesses with a certain step width in order to calculate 3D points for the objects to be measured. The measurement error is minimized.

Dependencies on Calibration Objects

Underlying models for underwater image formation that are not restricted by the above described perpendicularity are the more flexible ones. Figure 3.6 shows schematic representations of possible setups for FRSS s. These are characterized by two single

FRS s (FRS₁ and FRS₂) that are based on general models for refractive image formation,

3 Related Works

Figure 3.6: Schematic representation: FRSS composed of two single FRS s (FRS1and FRS2).

Left: Linking by 3D reconstruction of object points O_i (with i ∈ {0, .., n}) in the

scene. Right: Calibration object (CO) and the linking transformations (R_i, t_i, with

i ∈ {0, .., m}).

object points in the scene with respect to the coordinate system of the master camera. This is also the case for FRSS s (See left side of Figure 3.6). The advantage is that this relaxes the dependency on calibration objects with known feature coordinates (See right side of Figure 3.6). This in turn avoids the need to know or to determine the transformation between the coordinate systems of the calibration object and of the FRSS . As will be seen in the following, not many approaches make use of this possibility. Known Coordinates of Features on Calibration Objects. Li et al. [Li+97] propose an underwater photogrammetric model based on a 3D ray tracing technique. The authors formulate a set of equations for the intersection of corresponding left and right rays in water. The stereo camera is calibrated for its intrinsic and relative extrinsic parameters in advance. Each glass layer thickness and the refractive indices of the participating media are known. The unknown parameters are estimated by linearization of the formulated equations, which then are solved using the least squares principle [Li95]. For calibration, a 3D frame with known 3D coordinates is necessary and a set of stereo images of it needs to be acquired under water.

No Calibration Object. In the approach of Sedlazeck and Koch [SK11] the intrinsic parameters of both cameras are calibrated in advance and the glass layer thicknesses need to be known. The estimated parameters are the interface poses and the absolute as well as the relative extrinsic parameters of the stereo camera. There is no need for a calibration object, since stereo correspondences, which are matched by using SIFT features, are sufficient for the estimation. Initial values for the camera poses and 3D scene points are necessary. An approximate intrinsic camera calibration by the method

3.3 Explicit Modeling of Refractive Effects

of [FF86], [LRL00; LRL03] (approximation of refractive effects by focal length and radial

distortion) is used to refine the initial camera poses and the 3D scene points by bundle adjustment. This results in an approximation of the scene, which is used to find the interface poses by a further bundle adjustment step. Together with the true intrinsic parameters of the cameras, this optimization is done in nested loops. An inner loop minimizes the reprojection error to estimate the air layer thickness and an outer loop minimizes a virtual error function for estimation/refinement of all remaining parameters. This virtual error function is derived from the authors virtual camera model described in Section3.3.1, which is based on a caustic surface.

Chen and Yang [CY14] propose an approach for the calibration of a FRSS with an arbitrary number of layers. The stereo camera is calibrated for its intrinsic and relative extrinsic parameters in a pre-processing step and the number of layers as well as their refractive indices need to be known. Estimated are the interface orientations and the layer thicknesses. No calibration object is needed, since stereo correspondences generated by a projector, which projects gray code structured light patterns onto the scene, are utilized. The important finding of the authors is that the layer thicknesses can be estimated if the orientations of the system axes are known. Therefore, they formulate a constraint, which makes the determination of the layer thicknesses possible by solving a linear set of equations. To determine the axes, a binary search over a defined search space is performed for both cameras. For every combination of axes during this search, the layer thicknesses need to be determined, which is followed by a refractive back-projection to compute 3D points for every stereo correspondence. These 3D points are used to compute the reprojection error as a measure of parameter accuracy. Therefore, by referring to

[Agr+12], an equation of degree four for a single refraction and of degree twelve for

two refractions needs to be solved. Although the approach is said to be able to handle an arbitrary number of layers, it is not mentioned what needs to be done if there are more than two interfaces. All the determined parameters are refined by sparse bundle adjustment. After experiments with this approach, the authors remark that one should try to avoid the cases when both the axes and the glass layer thicknesses are unknown. They recommend that the glass layer thicknesses should be measured instead.

Findings

As within the calibration approaches for FRS s, a provided perpendicularity between the interface normals and the image plane simplifies the calibration problem, but restricts the setup severely. If the underlying image formation model is not restricted in this way and if a stereo system is available, which allows a refractive calibration from stereo correspondences, suitable calibration approaches amount to a powerful tool. The disadvantage is that the calibration of a single FRS is missing. On the contrary, they have the advantage that no calibration object with known feature coordinates is necessary. This advantage outweighs in particular if otherwise a 3D calibration object has to be handled underwater, which can be quite cumbersome.

3 Related Works

Figure 3.7: Schematic representation: SFRS composed of single cameras (Ci with i ∈ {1, .., n})

and calibration objects (CO) on the refractive interface Φ1 or in water and the

linking transformations (Ri, ti with i ∈ {1, .., m}).