Fitting the APM to image data

In this section we explore how well the model, constructed from the most significant modes as discussed above, fits or describes image data. In principle the patch model should be scanned across an entire image and the goodness of fit recorded for each position of the centre (say) of the patch. However, as described by Buxton and Zografos [22] this would result in a very large, complicated error surface so, in order to focus in detail on what happens, the centre of the APM was scanned over a 31 × 31 pixel square resulting in a search window 111 × 111 pixels centred on a RBC or some other selected part of an image. The sum of squared error of fit of the APM to the image data within this window was calculated according to:

²_{P CA}= [x − hxi − and displayed as a surface over the 31×31 pixel search square using Matlab (figure 3.6). In 3.41 we note that x, hxi and the modes p(k) are long vectors of dimension d. The scalar products thus automatically imply a sum over all pixels in the patch and over the colour channels. The mean hxi is estimated as in 3.20.

A RBC used in training was chosen so that there were no others or artefacts nearby in the image. The squared error (SE) surface shown in figure 3.6 thus has the expected bowl-shape with a minimum at the centre of the search square when the model template lies precisely over

the cell in the image. However, this minimum is rather broad so setting a threshold on the error would lead to many ‘hits’ for the possible location of the cell as shown in figure 3.6 (d). The cell selected and the model fit to it at the correct location is shown in figure 3.7 (a). Two other RBCs not used in training were similarly chosen, one from an image that contained other cells which were used in constructing the training set and one from an image not used at all in training. The error surfaces for these cells are shown in figure 3.6 (b) and (c) with the corresponding cells and the model fits to them in figure 3.7 (b) and (c). These images and the quantitative results in figure 3.6 illustrate that the model fits the training data and other healthy RBCs with little error.

In particular, the results in (b) and (c) in figures 3.6 and 3.7 are similar to those obtained with an isolated cell used in training (see (a) in the figures) though the SE is roughly twice as large for an isolated cell in an image that was not used for training (figure 3.6 (c)).

The results in figure 3.6(d) indicate that non-maximal suppression would be needed to locate the cell to pixel accuracy whilst interpolation could be used to obtain the location to sub-pixel accuracy. However, before an APM could be used in this way to detect and locate healthy RBCs in an image, we need to know how it performs when located over other regions of an image, in particular the background. Figure 3.8 (a) and (b) show respectively the error surfaces as the APM is scanned over windows placed in background (plasma) regions of an image used in training and one not used in training whilst (c) shows the error surface when the model was scanned over a window centred on a part of a cell.

The results are qualitatively different. None of the error surfaces in figure 3.8 have well-defined, bowl-shaped minima like those in figure 3.6. For the background regions they are also quantitatively different with a range of errors from ∼ 80 − 85 for the background region from a training image and ∼ 130 − 135, for a background region from an image not used in training.

It was because of the former that the threshold in figure 3.6 (d) was chosen as ∼ 80 but it is apparent that this threshold is below the minimum in figure 3.6 (c) obtained for the best fit of the APM to a healthy RBC in an image that was not used in construction of the training set. The APMused in this manner would thus not be able reliably to detect RBCs and would generate many false negatives (i.e. many cells would be missed). Raising the threshold to reduce the false negative rate would generate many false positives.

It is an open question whether introducing a colour alignment step in the model building and in application of the active model would improve matters. Aspects of the colouration of each image would then become extrinsic to the flexible model and fitting the active model would have additional degrees of freedom. The flexible model itself would thus become more specific – making it more likely that a suitably discriminating threshold might be found – but the

(a) (b)

(c) (d)

Figure 3.6: (a) The SE surface obtained when the FPM is scanned as a flexible template or APM across a window centred on a RBC selected for training; (b) the surface obtained when the APM was scanned across an isolated cell in an image used in training but where the cell itself was not selected as a member of the training set; and (c) the surface obtained when the APM was scanned across an isolated cell from an image not used in training. (d), the same error surface as in (a) showing the many ‘hits’ for the possible location of the cell resulting when a threshold

∼ 80 is chosen.

(a) (b) (c)

Figure 3.7: (a): Left: the cell from the training set used to construct the error surface shown in figure 3.6(a) and, right, the model fit to it. (b): As in (a) but with a cell not selected in the training (left) and the model fit to it (right) corresponding to figure 3.6(b). (c): Similarly for the cell used in figure 3.6(c) and the model fit to it. Note the lighter background.

(a)

(b)

(c)

Figure 3.8: The SE surfaces obtained when the APM is scanned across a window centred on:

(a) a background region of an image used in training; (b) on a background region of an image not used in training, and (c) on a region containing a part of a cell from an image not used in training.

(a) (b) (c)

Figure 3.9: (a): Left: A background, plasma region taken from an image used in training used to construct the error surface shown in figure 3.8(a) and, right, the model fit to it when the patch was accurately located over this region. (b): As in (a) but from an image not used in training corresponding to the error surface in 3.8(b). (c): Similarly for a region that contained part of a cell from an image not used in training corresponding to the error surface in 3.8(c).

danger of over-fitting the model and extrinsic parameters in the active modus operandi would be increased which might reduce the discriminating power and increase the false positive rate.

It can be seen from figure 3.9 that the active model ‘hallucinates’ the appearance of a RBC when there is really no cell present in the input region. This may be expected to happen whenever the mean colour levels within a patch ¯x are different from the average colour over the patch of the model mean ¯hxi because, as noted in section 3.2.3, all components of the first mode p(1) are of the same sign and thus

b(1, x) = p(1)^T(x − hxi) (3.42)

will be non-zero. It can be seen from figure 3.10 that extending the FPM to include a locally adaptive additive colour bias with the principal components given by:

b(k, x) = p(k)^T[x − hxi − (¯x − ¯hxi)] (3.43) may slightly reduce the tendency of the active model to ‘hallucinate’ RBCs in a background region but does not eliminate the effect entirely. In the above ¯x denotes the colour averaged over the patch and ¯hxi denotes it averaged also over the training set.

The ‘bottom-line’ is that the active model is adaptive and may be expected to share the tendency of such models to adapt and fit well to background regions of an image as noted by Buxton and Zografos [22]. In fact, according to Buxton and Zografos an adaptive template model will only grossly fail to fit well to an image patch if the patch contains some features notcorresponding to the object of interest. They point out this is most likely when the patch in the input image includes part of an object of interest and parts of other things, background or other objects, and will generate a ‘rim’ in the error surface where the model fits poorly around the correct location of an object. This effect is consistent with the results above. The largest errors in the background regions (∼ 85 or ∼ 135 depending on whether they are from images

(a) (b)

Figure 3.10: (a): The model fit to the plasma region shown in figure 3.9(a) when a locally adaptive additive colour bias was included. (b): As in (a) but with the background plasma region from figure 3.9(b). It can be seen that the colours are slightly different in the two cases.

used in training or not) are much smaller than the largest errors (∼ 400 − 450 and ∼ 700 respectively) in figure 3.6 (a)– (c) whereas these latter errors are comparable to those in figure 3.8(c). This is borne out by the error surfaces obtained when the APM is scanned over larger regions surrounding healthy RBCs as shown in figure 3.11.