In many supervised classification tasks training examples need to be labeled by human annotators before they can be used for learning models. The labeling process may be very costly and tedious, especially in domains where data are complicated and a high level of expertise is required. An example of such domains is medical diagnosis. In this dissertation, I studied and proposed solutions to address the cost-efficient learning problem, where the objective is to learn better classification models while reducing and efficiently distributing the cost of labeling. The main contributions of this work are summarized as follows.
• We presented a learning from soft-label framework, where we ask the human annotator to provide us with, in addition to class label, also soft-label reflecting his/her belief in the class label, and incorporate this information into the learning process. In general, the soft-label can be presented in terms of probability, e.g. 0.85, or qualitative categories, e.g. weak/strong belief. Our framework was motivated by the observation that the cost of labeling is mostly in the example review. Once an example was reviewed and class label was given, the human annotator can give us the auxiliary soft-label at an insignificant cost. We have done a study in medical domain to confirm this conjecture.
• We showed that the soft-label information can help to significantly increase the per- formance of classification models. However, we pointed out that the soft-label given by human may be noisy and negatively influence the learning process. To address this prob- lem, we proposed a ranking-based method that is very robust to noise and outperforms standard binary classifiers even with a strong level of noise in the soft-label. The idea
of this method is to model the pair-wise ranking relations between training examples, and combine that with the class information, instead of relying on the exact values of soft-labels or class label alone.
• We showed that, while the above ranking method learns high-quality classifiers, it does
not scale up well with large training data because of theO(N2) number of ranking con-
straints in the optimization. To address this issue, we proposed a ranking method that
is able to reduce the number of constraints to O(N) while retaining the high classifica-
tion performance. The idea is to distribute training examples to a constant number of discrete bins based on the soft-label information, then enforce optimization constraints on examples and bin boundaries. Another advantage of this method, besides the linear number of constraints, is that it can be naturally applied in the case when soft-labels are presented in qualitative categories. In this case the discrete categories/bins are given directly by the human annotators.
• We presented a novel multi-annotator learning framework that takes into account dif- ferent aspects of the labeling process and efficiently combines information from different annotators to learn better classification models. This framework addresses a problem of the traditional supervised learning framework: it assumes that, either there is a single annotator labeling all examples, or the labeling process is uniform and labels from dif- ferent annotators are treated similarly. This is often not the case in practice. First, the labeling process may be tedious, so we cannot expect that a single human annotator will be able to work on the whole training data. Second, in some practical applications, there are naturally many annotators, e.g. a group of physicians examining the same patient, or many people annotating the same news article. Lastly, it is important to accept the fact that the labeling process is not uniform - people often have different opinions on the same subject.
• Our multi-annotator learning approach learns better classification models than state-of- the art methods. The current methods mainly come from the learning-from-crowds field, which has been motivated by the growing number of crowd-sourcing services. These methods typically learn a voting model that weights the labels collected from the crowds of online workers. The methods often rely on a large number of annotators and labels
which can be had at low costs. However, in many practical tasks, e.g. disease diagnosis, it is not feasible to acquire a large number of annotators because the labeling process requires people with high levels of expertise, which is an expensive and scarce resource. Our approach does not rely on a large number of labels, instead, it takes into account and learns different characteristics of the (expert) annotators. Briefly, it models the domain knowledge, the bias and the consistency of different annotators. This approach results in successful learning of a consensus model that represents the collective wisdom of the experts.
• Another advantage of our multi-annotator learning approach is that it also learns the model and other characteristics of each annotator, such as consistency, reliability and bias. This is useful in practice because we can use the individual models to predict labeling patterns of the experts. Moreover, this gives us more insights into the labeling process, which helps us to tune the models and the labeling process itself for better predictive performance and higher quality of data in the future (e.g., we may prefer to collaborate with more reliable annotators).