Co-registration

Image co-registration is a fundamental step in image processing that has applications in computer vision and pattern recognition, medical image analysis and remote sensing data processing and fusion [161–163]. Image co-registration deals with a variety of problems, such as (1) registration of images over the same scene from different sensors (multimodal registration), (2) registration of images over the same scene from different viewpoints (mul- tiviewpoint registration), (3) registration of images over the same scene at different times or under different conditions (temporal registration), and (4) find pattern or object in the image that matches a desired pattern (template registration) [161]. For multi-sensor fusion purposes, the co-registration step usually refers to the first three scenarios.

Most multi-sensor image co-registration methods are based on geocoding, parametric methods such as similarity measures, or non-parametric methods such as optical flow [18].

Geocoding

doppler approach, interpolation and nearest-neighbor resampling, among others [18, 164,

165].

The geocoding co-registration process plays an essential role for fusion, but the accu- racy of geocoding highly depends on the availability and accuracy of auxilary geographic data such as a dital elevation model (DEM) [18, 166]. However, many Global Position- ing System (GPS) device are often only accurate to the level of several meters [24, 167]. Ground control points can be selected to enhance the accuracy of geocoding [166] but the process still requires geometrically corrected reference systems and it requires extra expense for picking control points.

Similarity measures

Similarity measures methods, described here, are feature-based co-registration methods [168]. In the literature, features such as edges, edge orientations, points, regions and line features have been used to perform image co-registration [169, 170]. The scale-invariant feature transform (SIFT), for example, has been used as a popular feature descriptor [171]. Image intensity was also used in classical area-based methods such as cross-correlation [172] or mean square difference of image intensity values [173]. Feature-based methods are useful if the images contain enough detectable and distinctive details and/or objects in the scene [168].

Once the features were selected, a similarity measure or function is used to evaluate and optimize the similarity between images. The choice of similarity measures plays a significant role in image co-registration [18]. Mutual Information (MI) [174] is widely used as a similarity measure to co-register images, especially in applications such as medical imaging and remote sensing [175–182]. The MI between two random variables A and B is

defined as [174,183]:

I(A, B) = H(A) +H(B)−H(A, B), (2.44)

where H(A) and H(B) are the Shannon entropies [184] of A and B, respectively and H(A, B)is the joint entropy ofAandB.

Suppose AandB are two sensor images to be co-registered and assume the intensity values in imageAranges from[0, M−1]and the intensity values in imageB ranges from [0, N−1]. Define the joint histogram matrix,h, as:

h =          h(0,0) h(0,1) · · · h(0, N −1) h(1,0) h(1,1) · · · h(1, N −1) .. . ... · · · ... h(M −1,0) h(M −1,1) · · · h(M −1, N −1)          , (2.45)

whereh(a, b)is the number of pairs having intensity valueain imageAand intensity value b in image B. The joint histogram matrixh, thus, describes the relationship between the image pairsAandB. The joint probability mass functionpA,B(a, b)is then defined as:

pA,B(a, b) =

h(a, b)

P

a,bh(a, b)

. (2.46)

The marginal probability mass functionspA(a)andpB(b)are computed as:

pA(a) = X b pA,B(a, b), (2.47) and pB(b) = X a pA,B(a, b). (2.48)

The entropiesH(A)andH(B)and joint entropyH(A, B)can be computed as: H(A) = X a −pA(a)logpA(a), (2.49) H(B) = X b −pB(b)logpB(b), (2.50) H(A, B) = X a,b

−pA,B(a, b)logpA,B(a, b). (2.51)

Here the intensity values are used as suggested in [174,181] but other features can be used as well. Other interpolation algorithms, such as nearest neighbor [185], can also be used to estimate the joint histogram matrix.

Similarity measures methods such as the MI-based co-registration face challenges such as high computation time due to large data volume and wide differences of sensor geometry and radiometry due to increased spatial resolution of sensor data [181]. Traditional features tend not to work well with multiangle remote sensing images as the resolution changes in images with large view angles may affect the accuracy of key points selection [186]. Other properties, such as the low-rank constraint [186], has been explored for image co- registration. The RANdomSAmple Consensus (RANSAC) algorithm [187–191] was also widely used in feature-based image registration.

Transformation, Interpolation and Resampling

Transformation, interpolation and resampling methods are studied in the literature for im- age co-registration [169, 192]. To co-register two images, one image is fixed to be the “master image” and the other image is projected and resampled to the same reference sys- tem as the master image [18,193]. A variety of transformations can be used to co-register images, such as similarity transform, affine transform, perspective projection, and elastic

transform [169]. Popular nterpolation methods include the nearest neighbor [194, 195], cubic convolution [196], bilinear/tri-linear interpolation [197–199], quadratic interpolation [200], polynomial interpolation [192], and spline interpolation [201,202]. A survey of in- terpolation methods in medical imaging applications can be found in [203]. Interpolation methods can reduce the computational expense of conventional co-registration methods [192]. The efficiency and performance, however, will depend on the selection of the opti- mal interpolation kernel [192].

Additionally, the spectral diversity technique is used to perform co-registration, more specifically for SAR images, based on the spectral properties of the comple SAR signal [204, 205].The spectral diversity co-registration does not need any interpolation or cross- correlation and can be “at least as accurate as” conventional co-registration methods [204].

Conflation

The term “conflation” has been used, in some cases, interchangably in the literature as image (co-)registration [206–211]. Conflation is in particular associated with applications relating to map compilation and geospatial data, such as overlaying a vector road map to a geospatial imagery [212–214]. Conflation integrates and combines (in particular) geo- graphic information from multiple sources to “retain accurate data, minimize redundancy, and reconcile data conflicts.” [211,215] Just like image co-registration, features and sim- ilarity measures can be extracted to match images or to overlay different geospatial infor- mation sources [211].