7.3 Training data labeling
Collecting appropriately labeled data for training and optimization of a supervised pat- tern recognizer may be a difficult task. This is especially the case of CAD, for which the common approach is to have human experts manually outlining the diseased regions of training images. This approach is not only time consuming but also prone to errors, as it is not always possible to accurately determine the actual extension of the lesions or if suspicious regions are truly affected by disease. In the worst-case scenario, the experts needed for outlining are not available, and the labeling required by supervised algorithms cannot be obtained. Thus, in order to facilitate continued and proper CAD development, a different labeling/learning strategy should be devised.
To address this issue, Chapter 4 explores multiple instance learning (MIL) as an alternative to supervised learning for CAD applications. The main advantage of MIL in the context of CAD is that no lesion outlines are needed, and instead, labeling at a higher level, e.g., per image, suffices for training. Moreover, the bag/instance mechanism that is core to MIL can be straightforwardly extended to CAD applications by relating, for example, bags with images and instances with image pixels. As a result, and depending on the utilized MIL algorithm, the same types of output information obtained from a supervised CAD system can be obtained from a MIL-based one.
Taking into account the expedient features of MIL indicated above, it is possible to conceive a number of circumstances where a MIL-based CAD approach could prove superior to a conventional supervised one. Perhaps, the most immediate situation is the worst-case scenario described at the beginning of this section. Typically, information for CAD training is gathered from medical or research centers in which diagnoses have already been made. These diagnoses, however, are usually made at the case level. While that information is enough to train a MIL-based CAD system, further details, in the form of lesion annotations, are needed by a supervised approach. This poses the prob- lem of finding suitable experts to carry out the annotations, which may be difficult even under privileged circumstances. For instance, in a research department at a medical center, experts may have several other duties and priorities, or depending on the cen- ter’s resources, they may simply be scarce. Moving away from a medical environment, for example, to a computer vision or machine learning research facility, finding qualified medical personnel may not be feasible at all. As a consequence, the potentially helpful methods developed by researchers on these other fields may not be appropriately ex- ploited. Another scenario to consider is when experts are available but only for a limited time. Since it would be much easier to indicate if an image is abnormal than outlining its abnormal zones, it would be possible to accumulate much more labeled data for a MIL- based strategy than for a supervised approach. Given that one of the requirements for
112 Concluding remarks
automated learning is to have enough training data, MIL could lead to a more accurate solution than supervised learning under such restrictive conditions.
One disadvantage of MIL, which is related to its low labeling requirements for train- ing, is that the labels at the instance level for a large number of instances are not reliably known. As a result, training is not as accurate as within supervised learning, and pre- dictions at the instance level may deviate from the optimal ones. Although the MIL algorithms themselves are responsible of dealing with and ideally eliminating the afore- mentioned uncertainty, the complexity of the classification problem at hand may prevent the devised mechanisms from fully accomplishing this goal. Acknowledging this issue, Chapter 5 proposes to externally compensate for the deficiencies of those mechanisms and explores active learning (AL) as a means of reducing the uncertainty remaining af- ter the MIL process. The contribution of AL in this context is two-fold. First, it allows replacing several of the uncertain labels with reliable information provided by an expert. Second, it allows carrying out that task while minimizing the expert’s effort, which is of high interest considering the limited expert availability pointed out before.
Besides the crucial effect of reducing uncertainty in the training process, AL may also enable refinements in terms of parameter selection that could further improve MIL prediction at the instance level. As it is well known, in addition to a properly labeled training set, proper parameter optimization is key to take advantage of a given classifier. This is especially true for complex methods, such as SVMs, for which a strategy com- bining grid search and cross-validation is often followed for parameter tuning. However, although that strategy works well for supervised learners, in the case of a pure MIL approach, such as the one described in Chapter 4, only a partial benefit at the instance level is observed. The reason is that the optimal parameters are selected based on the available bag labels, and that due to uncertainty, correspondence between bag labels and instance labels does not necessarily exist. A solution to this problem is introduced by the AL method proposed in Chapter 5, as correct instance labels, as provided by the expert, are readily available for part of the training set. By taking the performance on these correctly labeled instances into account, a more meaningful parameter selection criterion, in line with the one used in supervised learning, could be defined.
Another important contribution of Chapter 5 has been to exploit the MIL problem definition and use one-class classification to select valuable instances for relabeling within AL. Given the success of one-class classification in discriminating the normal from the abnormal regions in the abnormal images, it seems logical to attempt its application directly to the MIL process. One way to do so would be, for example, to utilize a high- performance one-class classification technique as part of or as the main mechanism to impute the uncertain labels based on the certain ones. If the problem were too complex for this approach to reach a reasonable accuracy, a less critical target would be the