Speed

One particularly unfavourable aspect of the proposed framework is the speed and computational requirement for each segmentation. This is primarily due to performing global kNN search for the patches in each atlas and is particularly a problem for larger images since each voxel is labelled individually. This imposes a restriction on both the number of atlases and the dimensional size of the images that can be used. The two are related, and it may be for larger images, fewer atlases can be used, as the computational time requirement could be halved by simply halving the number of atlases used. However, using fewer atlases reduces the anatomical variability that can be expressed through the atlases and thus the robustness of the segmentation performance.

6.3. Limitations and Future Work 169

Implementation

Without altering the algorithm of the proposed methods, one area that would yield improve- ments in speed is in the implementation, particularly in exploiting parallelism. Although the independent labelling for each voxel enables parallelism to be exploited, this has so far been with CPU-based architecture, which is limited and more costly for massively parallel tasks. Using a GPU-based implementation for kNN search could provide a significant boost in perfor- mance, and recent works such as [69] and [16] demonstrate the possibility for such an approach. However, the memory limitation of the GPU can still present a significant engineering challenge to overcome.

Alternative Data Structures

In the software implementations, the kNN datastructure used was a ball tree from scikit-learn, an open-source module for Python. This is a generalised kNN data structure but may not be the most optimal, in terms of speed and memory usage, for patch search and comparison. The proposed segmentation framework is suitably modular such that the ball tree could be replaced with alternatives, and a wide range of these were reviewed in chapter 2. Data structures that provide approximate nearest neighbours could also be used instead of exact kNN. Both the PatchMatch algorithm and the neighbourhood approximation forests (NAFs) are potential candidates that could be integrated into the proposed framework, although the training times for NAFs would be prohibitive for on-the-fly spatial context extraction and would limit the flexibility of the framework.

One aspect of performing kNN search that is not often taken into account in many data struc- tures is the similarity of the queries. In general, applications of kNN search rarely only perform a singular kNN query, but performs them more like a batch process which are repeated many times. This has been termed as general N-body problems by Alexander Gray et al [73]. It would be plausible to exploit the fact that similar queries should return similar results, and this is proposed by works such as [73] and [137] which propose the use of dual-trees, where a tree

is constructed for the query items as well the data library. This is related to the all nearest neighbour problem which has been studied in computational geometry [43], [173].

Alternative Classification Frameworks

A more drastic change to improve the speed would be to alter the classification framework from relying on using the kNN at run time, to one which does not use such a dense and potentially redundant representation of the training data. The approach of using random forests in [196] provides a potential candidate for which the spatial context methods developed in this thesis could be easily adopted. Another alternative could be using random ferns [126] and the extremely random variant [71], which could allow similar on-the-fly functionality for incorporating spatial context as the proposed framework. Additionally, dictionary learning could be applied to learn the range of patches and how they can be expressed in order to reduce the redundancy between them. A recent approach [170] uses such a principle for segmentation of the hippocampus, and could provide a template for future work.

6.3.2

Atlas Selection

One aspect of multi-atlas segmentation approaches which could benefit from further investiga- tion is the atlas selection process, particularly for large datasets and where it is desirable to select atlases without registration first. This was briefly visited in chapter 4, where an approach using histogram of 3D oriented gradients was proposed, but was not explored to substantial depth. Methods from content-based image retrieval could be used in alternative atlas selection schemes, but this has received relatively little attention in the medical imaging field.

Additionally, traditional atlas selection strategies of selecting the most similar atlases for label propagation should be reconsidered. Although selecting the most similar atlases would reduce the errors, it does not guarantee the atlases can account for the anatomical differences between them. This may have been the most optimal approach for multi-atlas approaches which are reliant on registration accuracy, but it may not hold for patch-based frameworks where there is

6.3. Limitations and Future Work 171

less dependency on voxel-wise image correspondence. Perhaps an alternative strategy should be to select a set of nearby atlases which are diverse enough to account for a wide range of local differences instead.

Furthermore, there is a trade-off in speed and accuracy with regards to the number of atlases to use. An increase in the number of atlases used yields logarithmic returns in the improvement to segmentation performance, as both the experimental results in chapter 3 and the results from [47] suggest. An atlas selection strategy could also be optimised to produce the maximum local diversity with the minimum number of atlases, which would be highly beneficial to the computational performance.