Lens Distortion Correction

Lens distortion correction usually involves the use of features that are invariant under projective transformation. One popular feature choice is straight lines, which gives rise to the so-called plumb-line methods. Due to lens distortion, straight lines will appear curved in

Figure 2.4: Calibration harp used for high-precision lens distortion correction via plumb-line method [144]

the distorted image, as shown in Fig. 2.3. Plumb-line methods compensate for the curvature of extracted line segments in the distorted image [45,115,144]. These methods are a popular form of non-metric camera calibration due to their accuracy, as well as the relative ease of obtaining images of straight lines compared to more involved calibration targets [144].

Plumb-line methods extract sets of image points that are known (or are supposed to be) lying on straight lines. Let x^l,(i) = (x^l,(i), y^l,(i)) be the coordinates of the i-th point on the l-th line segment, then under a distortion-free camera we should have

x^l,(i)cos(✓_l) + y^l,(i)sin(✓_l) = ⇢_l, i = 1, 2, ..., N_l, (2.30)

where Nl is the number of points on the l-th line segment. However, in the presence of lens distortion, the image points will not satisfy (2.30) exactly; for each point, there may be some offset from the common best-fit regression line [115].

Conventional plumb-line methods approximate the inverse transformation ˜f ¹(·, ·) by the same functional form as the forward transformation f(·, ·) [45, 115, 154]. For example, for the lens distortion model in (2.20)-(2.21), the inverse transformation would be assumed

to follow the same equations, but with the roles of (xu, y_u) and (xd, y_d) reversed, and with a different set of model parameters k⁰,

x_u= ˜f ¹(x_d, k⁰). (2.31)

Then the following cost function composed of perpendicular offsets from the common regression line is minimized,

N_l

X

i=1

k ˜f_x¹(x^l,(i)_d , k⁰) cos(✓_l) + ˜f_y¹(x^l,(i)_d , k⁰) sin(✓_l) ⇢_lk²2. (2.32) where the subscripts denote the coordinates of the undistorted image point - ˜f ¹(x_d, k⁰) = ( ˜f_x¹(x_d, k⁰), ˜f_y¹(x_d, k⁰)).

Plumb-line methods have preferentially been applied to urban environments, where straight line features are more prevalent [45]. Recently, the work of Tang et al. [144] re-visited the use of calibration tools, proposing the use of a calibration harp for high-precision lens distortion measurements. The calibration harp (Fig. 2.4) is a frame with carefully cho-sen wires stretched tautly across its face; thereby, providing very straight lines. Multiple images are taken of the calibration harp under various rotations, each image providing a group of straight lines. If L is the total number of line segments from all images, the cost function becomes [144]: Grompone et al. use a calibration harp and a very high-order polynomial inverse-transform model for lens distortion correction via the plumb-line method [152]. The high-order polynomial model is chosen for its self-consistency and universality [151]. Self-consistency of a lens distortion correction model refers to the ability to correct distortion produced by the same model. Universality refers to the ability of the model to correct distortion produced by other models. The 11^th-order polynomial model used has 156 parameters; hence, requires a large number of samples (i.e. control points) as well as careful handling of the optimization routine. The optimization is carried out by incrementally solving for successively high-order

polynomial models and performing refinements once all parameters have been incorporated.

In turn, the optimization is susceptible to overfitting.

Wang et al. derive a new lens distortion forward-transform model based on explicitly modeling the tilt of the real image sensor plane from the ideal image plane [154]. The inverse of the model is then used to transform the observed distorted coordinates xdinto estimates of the undistorted coordinates xu, so that an optimization identical to the conventional plumb-line method can be carried out. The new lens distortion model is shown to achieve comparable correction performance as the conventional model with fewer parameters.

Multiple view lens distortion correction methods are a popular alternative to plumb-line methods, since theoritically the methods do not have any requirements on the content of the views - such as the presence of straight lines for plumb-line methods. The methods rely on epipolar constraints, similar to the fundamental matrix to be discussed later in the context of triangulation in Chapter 3, between the different views.

Claus et al. devise a lifting strategy with a rational function model (2.27) to solve for the lens distortion parameters using a single calibration grid image. The method is shown to generalize to two-view geometry; whereby, lifted image points correspondences are used to solve for a lifted fundamental matrix along with the distortion parameters.

Fitzgibbon et al. devise a single-parameter radial distortion model (2.24) and use image point correspondences across two views to formulate a quadratic eigenvalue problem for solving for the parameter [50].

Algebraic methods have also been proposed to correct for lens distortion. Alvarez et al.

reformulate the conventional plumb-line method as a 4-degree polynomial using a new cost function [4]. The cost function can be solved via algebraic methods.

Lastly, full calibration methods have been proposed that use 3D calibration targets to get correspondences between 3D world points and 2D image points. The intrinsic parame-ters of the camera are solved simultaneously with the lens distortion parameparame-ters [68, 148].

Unfortunately, the methods are shown to be prone to errors as coupling between the intrinsic and lens distortion parameters may result in erroneous values as one compensates for the other [155].

f˜ ¹(·, k⁰) Projection +

x_d x_u ˆx_u

Optimization

e k⁰

Figure 2.5: Conventional plumb-line lens distortion correction