Model enhancement through parameter tuning

Despite the extensive use of RF classification for detection of spam/malicious content in OSNs, there is a lack of detailed information about how this method is used in terms of parameter settings [86]. Not giving clear information regarding the hyper- parameters used in building/training models could limit the reproducibility of the reported results and validation. This practice also makes it difficult to understand the impact of parameters on the performance of RF classification applications for OSN spam detection. Although RF does not require high effort in fine-tuning, setting improper RF hyper-parameters could lead to an over-fitted/under-fitted model, which could give low detection performance when tested on new data. To explain the process of tuning the highest-ranked algorithm in the first experiment, which is RF, first, the author needed to identify the top hyper-parameters that have a high impact on the RF model. Based on the literature review, the most important performance-affecting RF hyper-parameters [73][74] are tree number, maximum depth and minimum leaf size, which are mostly specified as the depth of trees could grow (stopping criteria) (as shown in Table 4.2).

Table 4.2 Random forest main hyper-parameters

Parameter Description

Tree number Number of trees in building the RF classifier Max depth The maximum depth that the tree can grow Min leaf size The minimum number of leaves a branch can have

As there is only two free hyper-parameters, maximum depth and minimum leaf size, trees are numbered logically, and the more trees in RF the better. Tofind the best parameter values for the model, the Scikit-learn grid search method was used. Using this approach, the parameters could be varied based on a range of pre-specified values. All options for all parameters cannot be considered, especially those that can be infinite, such as tree number and maximum depth. Therefore, in the following tuning experiments, high, medium and low numbers were assigned for such infinite hyper- parameters.

In this experiment the following hyper-parameters of RF classification considered: the number of trees, the maximum depth of trees, and the minimum size of leaf nodes (i.e. of the data subset associated with such nodes). The number of data features (i.e. data dimensionality) was kept unchanged, features selection/reduction will be discussed later in section 4.4. For each parameter setting, 20 experiments were run with randomly generated samples (stratified by target) for training and test datasets. To analyse the impact of parameter settings on the classification performance, the average performance and the standard deviation of the performance metric across the 20 experiments were calculated. The performance of the classifiers was measured in terms of recall, precision, and F-measure. The performance results were compared using the t- test to determine whether the difference in mean performance is statistically significant at the significance level of p=0.05. For the comparison of standard deviations (in fact, variances), the F-test was used with the significance level at p=0.05.

The number of trees

Generally, it is expected that the greater the number of trees, the better the performance [140]. This was confirmed by the results (see Figure 4.1). The results also show that the standard deviation of the performance values decreases as the number of trees increases. These results are valid for all settings of maximum tree depth and leaf node size.

Figure 4.1 The effect of the number of trees parameter on the performance of the spam classification

The leaf size and the maximum depth of trees are different constant values in the four panels: A) max depth = 10, leaf size = 10; B) max depth = 44, leaf size = 10; C) max depth = 10, leaf size = 300; D) max depth = 44, leaf size = 300.

The effect of the increase in the number of trees is most prominent in terms of mean performance for random forests with large maximum tree depth. For small maximum tree depth, adding more trees to the random forest improves the performance in terms of reducing the standard deviation, although there is not much improvement in terms of mean performance. In all considered cases, the mean performance does not improve statistically significantly beyond nine trees in the random forest.

The standard deviation of the performance improves more for random forests with smaller maximum tree depth than for those with larger maximum tree depth. The improvement is statistically significant (at p = 0.05) up to 25 trees and becomes insignificant beyond that.

Maximum tree depth

Maximum tree depth is one of the variables that determine the complexity of the RF classifier. Trees can be built without any depth limit; however, in general, it is recommended to control the tree depth to avoid overfitting [141]. Here, the effect of varying maximum tree depth was analysed, while considering a range of fixed combinations of the number of trees and minimum leaf size.

The results show that the maximum depth of trees has a major effect on classification performance in the context of the spam detection problem in this research for all considered combinations of number of trees and minimum leaf size values (see Figure 4.2). It was found that the mean performance of the classifiers improves with the increase in the maximum depth of the trees. This improvement is statistically significant (at p = 0.05) up to maximum depth 16 for random forests with large minimum leaf size and up to maximum depth 24 for random forests with small maximum leaf size. It was

also found that the standard deviation of performances gets smaller as the maximum depth is increased and that this effect is the strongest for RF classifiers with large minimum leaf size and few trees.

Figure 4.2 The effect of the tree max depth parameter on the performance of the spam classification

The leaf size and the number of trees are different constant values in the four panels: A) number of trees = 3, leaf size = 10; B) number of trees = 41, leaf size = 10; C) number of trees = 3, leaf size = 300; D) number of trees = 41, leaf size = 300.

These results show that setting the maximum tree depth too low leads to low classification performance irrespective of the minimum leaf size and the number of trees (see Figure 4.2 for maximum tree depth below 8 in all four panels). Setting the maximum tree depth high may not lead to trees with that depth due to the limit on the minimum

leaf size; however, setting low leaf size and high maximum depth could lead to overly deep and complex trees that could show high performance on the training dataset but poorly when tested on unseen examples.

Minimum leaf size

Figure 4.3 The effect of the leaf size parameter on the performance of the spam classification

The maximum tree depth and the number of trees are different constant values in the four panels: A) number of trees = 3, max depth = 8; B) number of trees = 3, max depth = 44; C) number of trees = 41, max depth = 8; D) number of trees = 41, max depth = 44.

The minimum leaf size controls the complexity of the decision trees by setting a size limit for the data subsets associated with leaf nodes, consequently preventing the adding of further decision nodes to the tree after the nodes reach this limit. The effect of

changing the minimum leaf node size have been investigated while keeping the number of trees constant and the maximum tree depth for RF classifiers applied to the spam detection task.

The results show that the increase in the minimum leaf size reduces the performance of the classifier in all cases (see Figure 4.3). This effect is much more pronounced in the case of classifiers with high maximum tree depth than in the case of classifiers with low maximum tree depth. This because the similar effect of both parameter, short trees have large leaf size and vice versa. The effect is similar for different numbers of trees in the classifiers, the only difference being that for a large number of trees, the standard deviation of performance is lower than for a small number of trees.

For RF classifiers with low maximum tree depth, the minimum leaf size has a statistically significant effect on the performance if it is larger than 30 (number of trees) or 50 (few trees). This indicates that in these cases, the limited depth of the trees implies the limited performance of the classifiers for smaller minimum leaf sizes. Conversely, for classifiers with high maximum tree depth, the effect of the leaf size is statistically significant for all values of this (i.e. larger minimum leaf size implies significantly reduced performance).