description o f im age in terms
o f e.g. boundaries
retrieval o f im ages w ith interpretations provides decision su pport
Query Image User Reference Image Image Description Clinical Interpretation
Figure 11: P c , a system f o r indexing an image database on the basis o f image content. Each image is associated with stored information, which includes details o f the clinical interpretation and an description o f image features derived from image processing. A semi-automatic interpretation o f the query image can
Orphanoudakis et al. [Orphanoudakis 1994] have developed a system in which images are classified using criteria based on clinical interpretation but also on semi automated analysis of image features. This analysis recovers a symbolic image description which is in terms of properties of the image data. The authors state that the system is designed to be independent of any particular approach to image description but their account includes a description of a technique in which a segmentation algorithm identifies chains of edge segments which define the boundaries of regions of interest. The properties of these symbolically represented descriptions can then be compared with descriptions derived from the query image. The mapping from the query image to the symbolic representation is semi-automated.
3.4.1.2
From the query image to the reference images
In the Collins et al. [Collins 1995] system the query image is matched with an image from an atlas of reference images and the labelling associated with the reference image is applied to the query image. The mapping from the signal data in the query image to the symbolic labelling is achieved by associating each stored symbol with a description in terms of actual image data which can be matched with the query image. The same pattern-matching approach is used in systems where particular anatomical features or abnormalities are represented using templates or idealised images which can be matched with the query image. For example, in a system described by Matsumoto et al. [Matsumoto 1992] the correspondence between the reference images - which are templates of idealised lesions - and the query image is the basis for the decision support (in this system there is no symbolic knowledge). Where pattern-matching is used to support the segmentation of a query image, as in the Collins system, the final solution may represent some kind of best fit between the query and the set o f reference images. Where it is used, as in the Matsumoto system, to assist in the detection of abnormalities, a threshold must be set on the function which matches image data and template to determine when an adequate match has been found.
Reference Image Image Labelling retrieval o f im age labelling p rovides decision support
query im age m atched with reference im age fro m a tlas
User
Query Image
Figure 12: in the Collins et al. system the association between the sym bolic labelling o f reference images and the data in the images is used to assist the user by providing anatomical labels f o r a segmentation o f the query image.
3.4.1.3 From the query image to the symbolic knowledge
The phrase ‘image interpretation’ is used here to refer to the recovery from an image of a symbolic description of the scene depicted in the image. The symbolic description consists of statements about the objects in the scene and how they relate to each other. For example, a symbolic model of the anatomy of the abdomen would consist of statements about the different organs and how they are arranged relative to each other. Mapping from an image of the abdomen, for example on a CT scan, involves using image processing to identify the different organs. This means that each organ, or rather the symbol representing each organ, must be associated with a description of the organ, a description derived from the output of the image processing. Three approaches used in the systems described above are to describe anatomical features in terms of properties of the axes, the boundaries or regions associated with their appearance in image data. Robinson [Robinson 1992] uses an axial description,
descriptions. The commonest technique is to use region-based descriptions: [D hawanl990, Li 1993, Li 1994, Stewart 1994, Kobashil995, K ulikowskil995].
Radiological Knowledge
autom ated interpretation recovers
description o f regions
know ledge abou t properties o f regions provides decision
support
Query Image
User
Figure 13: in AutoMEX [S tew a rtl9 9 4 ] processing o f the query image results in a set o f measurements which describe the image regions corresponding to abnormalities. These measurements are com bined with the sym bolic knowledge to provide inferences about the kind o f abnormality likely to be present in the image.
To recover from an image a description which can be matched to a description stored with the image, it is generally necessary to segment the image into distinct regions and derive a description of each region. It is important to note that the selection of a particular segmentation technique - the detection of boundaries between regions or the separation of the image into regions of uniform brightness - does not necessarily imply a commitment to a particular technique for representing the object depicted in the region. A region-based segmentation technique could be used to identify regions on an image, but the stored description of the image doesn’t also have to be in terms of properties of regions. Once a region is identified, its boundary and its axis can be recovered. On the other hand, an edge-based segmentation algorithm could recover enough information about the boundaries in an image to allow objects to be described
as a collection of boundary segments, but which could be too incomplete to allow the region to be identified.
The three representation formalisms mentioned above are the only ones used in the systems described in this chapter. There are, however, other techniques not described in the literature on systems integrating signal and symbolic representations. These include Active Shape Models which contain an explicit model of the variability of objects of the kind which are found in medical images [Cootesl994] and sc ale-space approaches which search for variation or stability in an object’s appearance as image resolution varies [Fritschl994] or the use of symmetries to describe an object’s shape [W alker1990].
3.4.2
The representation of signal data
In each of the systems which provide a mapping between symbolic knowledge and signal data, symbolic terms are associated with a stored representation which can be matched with signal data, or with a representation which can be recovered from signal data. The stored representations associated with symbols in the work described above include real images and idealised images, region-based descriptions, boundary- based descriptions and axial descriptions.
A description of a symbol in terms of image data simply means associating the symbol with an image, or a region of an image. In the same way, mapping from a piece of signal data to a symbol means matching the query image with the stored image. A description of a symbol in terms of regions means associating the symbol with a vector of values (or ranges of values) for properties taken by the region: size, shape average brightness, relative position and so on. Mapping from signal to symbol means dividing the image up into its component regions, measuring the properties of these regions and matching them against the values stored in the feature vectors. In the case o f boundary
and axial-based descriptions the same process of image segmentation is required, but the matching uses vectors which are values of boundaries or axes.
In choosing the appropriate representation for signal data, it is necessary to consider four questions. The first two concern the applicability of the description method to the domain covered by the symbolic representation: coverage, that is, can all the required symbols be described, and uniqueness, that is, do all the different symbols have different descriptions? The other two concern the applicability of the method to the signal data: recoverability, that is, how easily can the description of an image be obtained, and sensitivity, that is, how reliably can the recovered description be matched with the stored descriptions associated with symbols?
Consider how the different categories of representation fare when judged against these criteria.