Methods
The algorithms we have represented in this chapter specifies what the model selection methods do, and they are similar to the actual MATLAB code we used for the Monte Carlo simulation results. This code however, has some issues. Since the early works of this thesis (see section 1.2), we have attempted to use different classifiers, hyperparameter values, and features. Gradually we refined upon the code, attempting to reuse code such as cross-validation methods for different classifiers. However, adding new capabilities such as feature selection of groups, deciding where to use cross-validation, or having classifiers use the source information (GMM classifier), were tedious and required alterations in many parts of the code. To address these issues, we have made a new general framework for model selection methods that can easily be extended. We used the new code implementation for the results in 3.6.3. We will discuss the most important concepts of this framework here.
50
For clarity, we will explain the framework using an example model-selection configuration. We have a k-NN classifier with the hyperparameter search method described in section 3.3.3. Furthermore, we use greedy forward selection for feature selection, using the same cross-validation error that we use for hyperparameter selection. Lastly, we obtain an unbiased error estimate by wrapping this in an outer cross-validation loop. Figure 3-10 illustrates this configuration.
Figure 3-10 Example Setup of a Model Selection Method Algorithm
Minimalistic illustration of the model-selection method framework, training a classifier with hyperparameter selection, feature selection, and an outer cross-validation loop for unbiased error estimates. The MSM prefix of class names is an abbreviation for model selection method.
All subclasses of MSelMethod inherits the prediction function, and has the same interface for training. The superclass also contains other functionality not necessarily specific to a single subclass. Every MSelMethod object contains a sub MSelMethod object, except the MSMHyperparam object, which contains a classifier object. A, MSelMethod object is trained by training the sub MSelMethod object.
Subclass: MSMUnbiased Subclass: MSMFeatureSel Subclass: MSMHyperparam Class: Classifier Function Handle: Feature Selection Function Handle:
Predict Data Function Handle:Train Classifier Hyperparameter SelectionFunction Handle: Class: Mapping Abstract Class:
51 The train function of an MSelMethod serves three functions. The first is learning the optimal features or hyperparameters. The second is to train a model using the full dataset. The third function is to provide an estimate of the error, via either cross - validation, or inheriting the cross-validation error of the sub MSelMethod object. The MSMUnbiased object has no feature or hyperparameters to optimize. It simply performs cross-validation on the sub MSelMethod object, obtaining an unbiased error estimate. Therefore, the MSMUnbiased object is always configured to provide an error estimate via cross-validation, rather than inheriting the error from the sub MSelMethod object. It only makes sense to use the MSMUnbiased object as the outer object, to provide an unbiased error estimate of the entire model-selection method training.
The MSMFeatureSel uses the feature selection function handle to select a subset of the feature groups. The feature subset is learned as a mapping stored in a Mapping object. The Mapping object has a function that transforms input data. In this case the function returns the data with the selected feature groups. To reduce the computational complexity, we have chosen to inherit the error from the MSMHyperparam object. Had we added a cross-validation loop here, we would get feature selection based on an unbiased estimate of the mode with optimal hyperparameters. Instead, we select feature using a biased estimate, because it were also use to select hyperparameters.
The MSMHyperparam object has some small alterations from other MSelMethod objects, as it contains a Classifier object, rather than a sub MSelMethod object. For hyperparameter selection, it extracts the hyperparameter search function stored in the classifier.
Normally when calling train on the MSMHyperparam and MSMFeatureSel objects, it initiates a feature search. However, we can configure the object to be pre - learned, in which case they will skip the hyperparameter selection and feature selection. We exploit this concept in the selection functions by taking the calling object as an argument, configuring them to be pre-learned with the hyperparameter or features we want to test, and then calling their training function. Using this, the calling object decides how to compute the error estimate, making the design of the selection functions very simple.
After training the entire model selection method, it is capable of predicting new data. The MSelMethod superclass has a predict function that takes input data, transforms it by passing it through a mapping function (if a mapping exists), calls the
52
predict function of the sub object, and then returns the predictions obtained by the sub object. As such, calling predict on the model of Figure 3-10 will first call the prediction function of the MSMUnbiased object. It will call the MSMFeatureSel predict function, which will transform the data to the subset of the selected features before calling the MSMHyperparam predict function. The MSMHyperparam predict function calls the predict function handle of the classifier. In this case the predict function normalizes the in accordance with the mean and standard deviation of the training set before calling the knnclassify function in MATLAB. Normalizing data is a functionality we should have as an option integrated in the Classifier class, which would be easy to implement.
To simplify the setup of an entire model selection method, each class has a default configuration. The configuration is a class in itself, and has an overwrite fields. As such, we can overwrite the default configuration whenever we want to make a change deviating from the default.
We only used the bucket of models, selection method for the synthetic data model. Its excessive computational time discouraged further use. We did not use the bucket of models for the wound dataset due to excessive computations. Therefore, we skipped supporting it in the new model-selection method framework; however, it would be easy to implement.