be the images of atlases i and j, then the joint probability can be written as:
Mx(i, j) ∝ [|Pt(x) − Pi(x0)||Pt(x) − Pj(x00)|] s
(2.27)
with Pt(.), Pi(.) and Pj(.) being patch extraction operators for image It, Ii and Ij respectively,
and s being a parameter for sensitivity. The proportionality does not affect the choice of weights since multiplying by a positive constant does not change the solution w. x0 is selected from a search window Nx around voxel x by:
x0 = arg min
y∈Nx
||Pt(x) − Pi(y)||2 (2.28)
and similarly for x00 with atlas j. This presents the biggest difference, ideologically, to ap- proaches derived from [47] based on nonlocal means in that only a single patch is evaluated from each atlas at each voxel x, whereas the nonlocal means compares multiple patches from each atlas. Prior to joint label fusion, the author also proposed a regression based label fusion approach using a similar principle [178].
For performance reasons and to provide context for patch comparisons, the window size for Nx
is often limited in the most patch-based methods (typically less than 113 = 1331 voxels). This
means that images must be aligned within this margin of error in order for the patch selection and label fusion strategy to work well.
2.4
Image Similarity and Distance Metrics
It is often desirable to quantify the similarity, or conversely the distance, between two images in order to compare them. This is fundamental for many methods in machine learning as well as from a data analysis and retrieval perspective. For example, to rank search results, or to compare alignments for registration. There are many ways to compare images, however the meaningfulness of the comparison is very much dependent on the context. In general,
the range of methods can be broadly split into two main categories: distance metrics and similarity measures. They can thought of as being inverse to each other, the former is derived from mathematics and geometry, whilst the latter is largely rooted in signal processing and information retrieval.
2.4.1
Distance Metrics
Commonly, when images are vectorised, distance metrics are applied to them using vector norms - for example the 2-norm which is also referred to as the Euclidean distance:
v u u t n X i=1 (xi − yi)2 (2.29)
As a generalised form, this is also known as the p-norm, Lp-distance or Minkowski distance
with order p: n X i=1 |xi− yi|p !1/p (2.30)
When p = 1 this is also known as the Manhattan distance and is sometimes referred to as the sum of absolute differences (SAD).
For p ≥ 1, p-norms satisfy the mathematical definition of a distance metric. This is defined as a function, d, on a set X which returns a real number, where given x, y, z ∈ X , the following constraints hold true:
1. Non-negativity:, d(x, y) ≥ 0 2. Identity: d(x, y) = 0 ⇐⇒ x = y 3. Symmetry: d(x, y) = d(y, x)
2.4. Image Similarity and Distance Metrics 73
The triangle inequality is particularly important for several algorithms, such as metric trees for nearest neighbour search, which exploit it to avoid exhaustive search of the entire space. However, not all distance functions satisfy this constraint. Outside the pure mathematical context, distance functions and metrics are often used interchangeably without regard to the mathematical definition. A commonly used example is the sum of squared differences (SSD). This is essentially the squared Euclidean distance and is defined as:
n
X
i=1
(xi− yi)2 (2.31)
SSD does not satisfy the triangle inequality in a case where x = (0, 0), y = (0, 2), z = (0, 4): here, d(x, z) = 16, d(x, y) = 4 and d(y, z) = 4 but 16 4 + 4. However SSD is still often used for comparisons as it penalises large differences and is more sensitive than the SAD or Euclidean distance.
The choice of distance metrics are not particularly well studied for image comparison applica- tions, although [2] provides a comparative review of Minkowski distances in high dimensional spaces, as well as suggesting the use of Lp-distances with p < 1. This is contrary to the majority
of applications using distance metrics, where the Euclidean distance is chosen as a default.
Other relevant distances used include the Hamming distance [77] and the Mahalanobis distance [112], although they are generally not used directly for comparison in the image space. The Hamming distance is only valid between binary vectors, whilst the Mahalanobis distance is a statistical measure which requires the calculation of a covariance matrix first.
2.4.2
Similarity Metrics
Similarity can be considered the inverse of distances, although similarity measures in general can be defined more loosely than distance metrics. Recently, [40] presented a formal definition of a similarity metric as a counter to the established definition of a distance metric: for x, y, z in a set X , a similarity metric is a function s which satisfies the following:
1. s(x, x) ≥ 0 2. s(x, x) ≥ s(x, y) 3. s(x, y) = s(y, x)2
4. s(x, y) + s(y, z) ≤ s(x, y) + s(y, y) 5. s(x, x) = s(y, y) = s(x, y) ⇐⇒ x = y
Popular similarity measures, particularly for registration, includes cross-correlation, mutual information and normalised mutual information. [131] provides a review of these as well as some additional measures used in registration.
Cross-correlation (CC) measures similarities in intensities at corresponding points between images by their dot product, and can be considered analogous to the convolution (or interfer- ence) of two signals. For two images x and y, CC is defined as:
scc= n
X
i=1
xi· yi (2.32)
Normalised cross-correlation (NCC) is a form of CC which reduces sensitivity to the differences in brightness and contrast between images x and y:
sncc = n X i=1 (xi− µx) · (yi− µy) p(xi− µx)2· (yi− µy)2 (2.33)
where µx and µy are the mean intensities of x and y respectively.
In contrast to cross-correlation, mutual information and normalised mutual information provide entropy-based measures. These measures examine the joint probability distribution of the two images after binning the intensities at each voxel. Entropy [151] is defined by the probability
2This case is not always true - for example the correlation ratio which is sometimes used as a similarity
2.4. Image Similarity and Distance Metrics 75
p of (binned) intensity values i within an image x as:
H(x) = −X
i
p(i) log p(i) (2.34)
and the joint entropy, which itself can be considered a similarity measure, for two images x and y with intensities ix in x and iy in y, and joint probability function p is given by:
H(x, y) = −X
ix,iy
p(ix, iy) log p(ix, iy) (2.35)
Mutual information (MI) [176] is defined using entropy as:
sM I = H(x) + H(y) − H(x, y) = X ix,iy p(ix, iy) log p(ix, iy) p(ix) · p(iy) (2.36)
A normalised form of this also exist - normalised mutual information (NMI) [165] is defined as
sN M I =
H(x) + H(y)
H(x, y) (2.37)
For image alignment, NMI is usually favoured since it is invariant to the amount of overlap of the background and low intensity regions between the images, whereas MI and joint entropy are both sensitive to this. NMI also allows comparison between images of different modalities, whereas intensity-based metrics like cross-correlation assumes that there is a linear relationship between intensities when comparing images.
Similarity measures are also used in evaluating segmentation accuracy. One commonly used measure is the Dice coefficient, which is defined for two segmentations, A and B as:
sDice =
2|A ∩ B|
where the intersection of two segmentations are calculated for each label c on a voxel-to-voxel basis, i.e. for each voxel x A(x) ∩ B(x) ⇐⇒ A(x) = B(x) = c
This defines a function in terms of overlap between the two segmentations, which is one way in which the similarity of two segmentations can be measured. A related function which is also used is the Jaccard index, which measures the similarity between segmentation A and B by:
sJ accard =
|A ∩ B|
|A ∪ B| (2.39)
where the union and intersection of the two segmentations are calculated as before, on a voxel- to-voxel basis.
2.4.3
Learned Similarities and Distances
In addition to predefined similarity measures, it is also possible to learn a task-specific similarity or dissimilarity measure. Approaches have been proposed for registration [86], segmentation [164], as well as general classification [186]. Recently, this line of thought has also led to an approximate nearest neighbour data structure using random forests [97]. However, the use of learned similarity or distance metrics remain an open research topic and is not yet widely adopted for practical applications.