Rotation forest

Rotation forest is based on the technique of feature extraction [192]. In order to create the training data, the predictors are divided into K subsets [58]. Sub-sequently, principal component analysis (PCA) is used on each of the K subset.

Next for each subset a decision tree is built including all the principal components.

The final prediction is created by taking the average of each of the K predictions.

There are two important parameters for rotation forest: the number of variables to select for each subset (K) and the number of base classifiers (L). We follow the recommendations of Rodriguez et al. [192] and set the number of variables to 3 and the number of base classifiers to 10. In order to implement rotation forest we used the R-package rotationForest [22].

3.3.5 Performance evaluation

In order to assess the performance of our prediction model, we use the following common measures: accuracy, G-mean, F-measure and the area under the receiver operating characteristic curve (AUC or AUROC) [67]. Consider a binary classi-fication problem, where {Po,No} stands for the observed positive and negative instances and {Pp,Np} for the predicted positive and negative instances. We can then easily represent the classification performance by means of a confusion matrix (or contingency table) given in Figure 3.1 [110]. In Figure 3.1 TP stands for true positives, FP for false positives, FN for false negatives, and TN for true negatives.

In our case the positive class is defined as being an alter’s significant other.

TN FP

FN TP

P_p N_p Predicted class

N_o P_o

Observed class

Figure 3.1: Confusion matrix

Accuracy, or the percentage correctly classified, is one the most used perfor-mance measures and can be defined as [110]:

Accuracy =T P + T N Po+ No

. (3.1)

Since accuracy is sensitive to the distribution of the data, it is not fit as a performance measure for imbalanced settings [110]. To overcome this problem precision, recall, F-measure and G-mean can be used [110]. These measures can be defined as follows [67]:

P recision = T P

T P + F P =T P

P_p , (3.2)

Recall = T P

T P + F N = T P

P_o , (3.3)

F − measure = (1 + β)²· Recall · P recision

β²· Recall + P recision , (3.4) where β determines the relative weight given to precision in comparison to recall (we set β = 1), and

G − mean =√

Recall · P recision = s

(T P P_o )(T P

P_p). (3.5) We note that we only allow for probabilistic output since a ranking is re-quired of people who are most likely to be one’s significant other. For accuracy, F-measure and G-mean a specific threshold (i.e., operating condition) needs to be chosen that classifies an instance in the positive or negative class [113]. This threshold has to be proportional to the percentage of people that we want to tar-get [19]. Since we are not interested in recommending a large proportion of an ego’s network as a significant other, we will select the top 10% users most likely to be a significant other. Hence we set our threshold to the value that results in a proportion of 10%.

However, often the future operating conditions in which a classifier has to be deployed are unknown. A solution in that case is to use aggregate measures which aggregate over a distribution of all possible cutoff values [113]. The most-well known example of portmanteau measures is the area under the receiver operating characteristic curve (AUC or AUROC). AUC is considered to be a more appro-priate and objective performance measure when the cut-off value is unknown at the time of model evaluation. The receiver operating characteristic curve (ROC) graphically depicts the relationship between the sensitivity and one minus the specificity for the entire range of cut-off values [106, 206]. Intuitively, AUC is the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance [50]. AUC ranges between the values of 0.5 and 1. The former indicates that the model performance is not better than ran-dom, the latter implies that the prediction is perfect [106]. The AUC is defined as follows:

In order to ensure the robustness of our results, we use five times two-fold cross-validation (5x2cv)[1]. First, 5x2cv randomly splits the data into two folds. Next, each fold is used once as a training and once as a test set. If tuning of the hyper-parameters is required, the training data is split again into two equal parts, namely a training and a validation set. After parameter tuning on the validation set, the initial training set was used to build the model and the test set to evaluate perfor-mance. Finally, the procedure is iterated five times until there are ten different performance measures [64]. As a measure of overall performance, we report the median accuracy, G-mean, F-measure and AUC of the ten different models. In addition, we also include the interquartile range (IQR) as a measure of dispersion.

In order to test for significant differences between the different modeling tech-niques, we use the non-parametric Friedman test [96] with Bonferroni-Dunn post hoc test [64, 77]. Within each fold the classifiers are ranked. Rank 1 is assigned to the best algorithm, rank 2 is assigned to the second best algorithm, . . . , and the rank equal to the number of algorithms (in our case 8) is assigned to the worst performing algorithm. In case of ties the algorithms receive the average rank. The Friedman statistic is defined as [64]:

χ²_F = 12N

with N the number of folds (in our case 10), k the number of different classifiers

(in our case 8), and Rj=PN

i r_i^j. Rjis the average rank of the j^thalgorithm and r_i^jstands for the rank of the j^thof k algorithms on the i^thof the N folds.

The null hypothesis of the Friedman test states that there is no difference be-tween the different algorithms [96]. If the null hypothesis of the Friedman test is rejected, we calculate the Bonferroni-Dunn post hoc test to compare the different classifiers to a control classifier [77]. The Bonferroni-Dunn test is preferred over the Nemenyi post hoc test [169] since it has greater power when all classifiers are compared to a control classifier and not between each other [64]. Two classifiers are defined as statistically different when their average ranks differ by at least the critical difference (CD):

CD = q_α

rk(k + 1)

6N = 2.690

r8(8 + 1)

6.10 = 2.9469 (3.8)

with qαthe critical value for a given significance level α (in our case 0.05), k the number of different algorithms (in our case 8) and N the number of folds (in our case 10). In our study two classifiers are statistically different if their mean ranks differ by more than 2.9469.