Figure 2.9: Relation between the location of the VOP ˇO and the real object point O in the case
of a single refractive interface.
Note that some conversions of Equation 2.26 are possible. Therefore, the Pythagorean trigonometric identity [BS79]:
sin2α + cos2α = 1 (2.27)
can be used. If the normalized direction vectors ¯a and ¯s of the ray a in air and the
interface normal s are known, the angle of incidence α can be expressed by Equation
2.23. With these substitutions, Equation2.26 can be converted to the form: ˇ x x = ¯ a · ¯s r n w na 2 − 1 + (¯a · ¯s)2 . (2.28)
Thereby, the need to compute trigonometric functions can be eliminated completely.
2.4 Implications of Refraction
As already indicated in the previous section, the physical phenomenon of refraction of light has some major consequences on the applicability of the conventional computer vision approaches made for aerial environments. The approaches that rely on the pinhole camera model are based on collinearity. Due to refraction, an object point in water, the center of projection of a camera and an image point are in general not collinear any more. Hence, the pinhole camera model can not be applied as it has been in aerial environments. In the following, it will be shown why the approximation of underwater image formation by the pinhole camera model is invalid and that the distortion due to refraction is distance-dependent. This distance-dependence makes clear that refractive distortions are not image space distortions, such as lens distortion. A direct consequence of the invalidity of the pinhole camera model is the invalidity of the epipolar geometry.
2 Basics
Figure 2.10: Left: Image formation based on the pinhole camera model in air. Right: Invalidity
of the pinhole camera model for underwater image formation.
Invalidity of the Pinhole Camera Model. The left side of Figure 2.10 shows the image formation based on the pinhole camera model in air. Let us consider a number of real object point Oi, with i ∈ {1, .., n}, which are projected to image space. The imaged object points Oi on an arbitrary surface are the starting point of the rays ai. All these rays intersect at a single point, namely the center of projection C, and end up on the image plane at the respective image points Ii. A real object point Oi, the center of projection C and an image point Ii lie on a single ray and are therefore collinear. The right side of Figure2.10 shows the image formation in the case of a flat refractive interface Φ, which separates air and water. This time, the object points Oi on the imaged surface are located in water. They are the starting points of the rays wi. These rays change their direction at the interface Φ due to refraction, intersect at the center of projection C and end up on the image plane at the respective image points Ii. The difference to image formation in air is that a real object point Oi, the center of projection
C and an image point Ii are not collinear any more. As can be seen the on right side of
Figure2.10, the extension of the rays wi into air (dashed blue lines) and the pairwise intersection of rays with equal incidence angles leads to several intersection points. These represent multiple virtual centers of projection ( ˇC1, ˇC2, ˇC3), which are in this case located on the principal axis p of the real camera. Hence, the pinhole camera model is not valid for the approximation of the image formation in underwater imaging setups [Tre+12] . This occurrence of multiple virtual centers of projection makes it impossible to infer a single focal length adjustment of a camera that can represent the image formation in terms of the pinhole camera model correctly.
2.4 Implications of Refraction
Figure 2.11: Distance-dependence of refraction.
Distance-Dependent Distortion due to Refraction. The distortions in an image that stem from refraction are dependent on the distance of the imaged object point to the refractive interface. Let us assume that refraction can be modeled as an image space distortion, such as in the case of radial and tangential distortion of a real imaging system. This would mean that the pinhole camera model with some additional non-linear terms for image distortion can be used for the approximation of the underwater image formation. Figure2.11 shows that this is theoretically impossible. The image point I0 is the result of the mapping of all possible object points Oi, with i ∈ {1, .., n}, on a ray w in water. This ray gets refracted at the interface Φ, which results in the ray a. Two of the infinitely many possible locations of the object points Oi on the ray w are illustrated by
O1 and O2. In reality, both end up in image point I0. As can be seen, the linear mapping of the point O1 and O2 by the pinhole camera model would result in the two different image points I1 and I2. Hence, refractive distortion is clearly dependent on the 3D coordinates of the mapped object points and thus not an image space distortion, which would be independent of the scene. Therefore, refractive distortion can theoretically not be compensated by additional non-linear terms for image distortion. This is also shown by Treibitz et al. [TSS08;Tre+12].
Invalidity of the Epipolar Geometry. The epipolar geometry makes it possible to transform both views of a stereo camera into a common plane to reduce the search space of corresponding pixels in both views. Let us assume that this transformation is unnecessary due to the already perfect orientation of the stereo camera. Figure2.12shows such a perfectly oriented stereo camera from three orthographic views. Both cameras are have identical intrinsic parameters and their principal axes p and p0 are parallel.
2 Basics
Figure 2.12: Epipolar lines an their invalidity in underwater stereo vision, illustrated by three
orthographic views.
The back view, the side view and the top view of this setup can then be stitched in the presented way. The back view is essentially constructed by the pairwise intersections of parallel lines from the side and the top view. Note that the representation of the cameras with a flipped and therefore virtual image plane is necessary for this kind of visualization (See Section2.2.1).
Let us consider the mapping of a real object point O by this stereo camera. This is illustrated once based on the pinhole camera model (dashed black) and once with explicit modeling of refraction at an interface Φ (solid blue turning solid black). The dashed black rays and the image points I1 and I10 represent the mapping if water would not exist. The horizontal line through the points I1 and I10 in the back view represents the epipolar line, which can be used as the search space for stereo matching of these two points. In an aerial environment, the assumption of epipolar geometry is that the rays connecting an object point O with the two centers of projection C and C0 form a plane [HZ04]. This assumption does not hold if refraction occurs [Tre+12]. If the epipolar geometry would be valid in the case of the mapping with explicit modeling of refraction, the dashed horizontal lines in red and green would coincide. The illustration clearly shows that the two image points I0 and I00 do not share an epipolar line in general. The only exception are object points that are located in the plane that is formed by the interface normal and the two centers of projectionC and C0. In the illustrated case, this would mean that the object point O needs to lie in the plane that is formed by the two centers of projection
C and C0 and the principal axes p and p0 to get a valid epipolar line. However, in the