This chapter investigated a method of adapting the permutation method for use with temporally or spatially dependent data. The permutation method was first described in a general setting, before introducing the blocked-permutation
method. Next, the validity of the blocked-permutation method was argued
under assumptions of locally dependent sequences.
Three different blocking strategies were applied in an empirical study of the
blocked-permutation method. Thestaticblocking strategy [75] was extended to
mitigate issues with periodicity in the data, in the form ofdynamicandsingle-
value blocking. Also, the static and dynamic strategies were investigated both with and without applying a cyclic shift to the data before each permutation [2]. The blocked-permutation method was performed for each of the blocking strategies over a wide range of block lengths on two datasets of vehicle telemetry. In these experiments we found similar patterns for each of the RCD, the CoventryDMD and the WarwickDMD over the block sizes investigated. When the block size was small, the permutation method did not provide reasonable re- sults. When the block size was large enough to introduce independence between blocks, we found that the permutation method provides stable significance mea- sures. Finally, as the block size approached that of the sample size, the number of permutations which would be produced using the static and dynamic strate- gies decreased, and it again did not provide reasonable results. The application of the cyclic shift, however, extended the range of suitable block sizes substan- tially.
Two non-parametric ranking metrics were proposed for performing feature
selection, namely M RM I (Equation 4.18) and M DM I (Equation 4.19). These
were then compared, ranking by the significance value [65], rejecting features with a MI below a significance threshold [37] and normalising MI by a param- eterised permutation distribution (Equation 4.17) [118, 155]. We found that
4. Temporal permutation feature relevancy
these datasets, as manyp-values are zero or close to zero. TheZM I andM DM I
ranking metrics produced similar feature rankings as those that would be ex-
pected by a human expert. The M RM I metric failed when there were zeros in
the permutation distribution, which was the case for all of the vehicle teleme- try datasets. In classification experiments, we found that the performance of a classification algorithm can be affected by the bias features selected using the MI and SU rankings.
We have also shown that, because the permutation-based rankings do not select such bias features, the AUC performances in classification evaluations was higher for the RCD and the CoventryDMD. The performances of HSIC rank- ing was also higher, even though some non-generalisable features were selected. The HSIC ranking would therefore be expected to have poorer performance on new data, if it was collected in a different location. It may be possible to com- bine these approaches in a feature selection framework to increase performance further. For the WarwickDMD we found that AUC performance was no better than a random classifier when evaluated using data from all drivers. This im- plied that good models cannot be built for all the drivers, and so we investigate models built for subsets of drivers and individuals in Chapter 6
This chapter considered the relevancy of features and their ability to gener- alise to new data recorded in a different situation or at a different time or loca- tion. For a successful feature selection process, however, redundancy between features should also be considered [76] with the permutation method. In Chap- ter 5, efficient redundancy computation using the permutation method is inves- tigated and applied in the minimal Redundancy Maximal Relevance (mRMR) framework [113].
CHAPTER
5
Redundant permutation feature selection
In Chapter 4 we presented permutation normalised Mutual Information (MI) for temporal data, such as vehicle telemetry, and ranked features solely by their relevance to the target variable. In general, however, redundancy between fea- tures also affects the performance of models built using them. Filters for feature selection, such as minimal Redundancy Maximal Relevance (mRMR), typically consider both relevancy and redundancy [76] via the same measure, such as MI or Symmetrical Uncertainty (SU). Permutation methods as presented in Chapter 4 are, however, computationally very expensive. Each individual per- mutation method, for instance, consists of thousands of permutations. Using a
permutation statistic such asZM I for both relevancy and redundancy is there-
fore infeasible. Computing normalised MI relevancies and redundancies between
mfeatures requiresm+m2permutation methods, which is prohibitive for large
feature sets. In this chapter the P CCor redundancy measure is introduced,
which is the Pearson correlation between permutation distributions produced
from a common target during the relevancy calculations. The P CCor measure
shown to approximate all m2 redundancies while performing onlym permuta-
tion methods for the relevancies, overcoming the problems with usingZM I for
both relevancy and redundancy.
Autocorrelation and temporal artefacts are not considered in this chapter, and vehicle telemetry data is not used. This is both for simplicity and generality, as the aim of this chapter is to propose a permutation redundancy measure that reflects the properties ofZM Iand is feasible to compute with even large datasets. The techniques developed in this chapter and in Chapter 4 are combined in Chapter 6, where temporal artefacts are again considered. Here, simulated data
5. Redundant permutation feature selection
and non-temporal datasets outlined in Section 3.4 are used. Simulated data is
used to show that P CCor holds similar properties to normalised MI, and then
the UCI and Tuned IT datasets are used in classification evaluations.
5.1
Introduction
Supervised feature selection aims to choose a subset of features that will pro-
vide high performance when used in a learning algorithm. As discussed in
Section 2.1.7 there are several approaches to feature selection, and in this thesis filter methods are considered as they are efficient for large datasets. Filter meth- ods in general aim to select features that are relevant to the target while being unrelated to each other. Feature clustering, for example, clusters features using their correlation as a distance measure, and the feature with highest relevancy in each cluster is selected [67]. Where the number of features required is known
k-means can be applied. If the number of clusters is unknown, an iterative ap-
proach where new clusters are generated if a feature is sufficiently different to existing clusters [67], or through computing the minimum spanning tree of the redundancy graph [138], can be used. Other approaches employ genetic algo- rithms and use fitness functions based on the total relevancy of selected features combined with their redundancy [16].
Another approach, introduced by Koller and Sahami [77], uses the concept of Markov blankets from Bayesian networks to describe the optimal feature set. The Markov blanket of a target variable is the smallest set of features that maximally describe the target variable [16, 64]. It can be computed by iteratively eliminating the feature that least changes the probability distribution of the target, conditioned on the remaining features [77], although this is an expensive procedure computationally.
In this chapter we study the commonly used mRMR filter for feature se- lection, as proposed by Peng et al. [113]. In this framework, the relevancy,
5. Redundant permutation feature selection
Rel(F, Y), of a feature setFof size |F|, is defined as
Rel(F, Y) = 1 |F|
X
Xi∈F
Cor(Xi, Y), (5.1)
whereCor(Xi, Y), is a measure of the relationship between the feature,Xi, and
the target,Y. The redundancy is defined as
Red(F) = 1
|F|2 X
Xi,Xj∈F
Cor(Xi, Xj). (5.2)
The most common form of mRMR aims to select the feature set, F⊆X, that
maximises the difference between the relevancy and redundancy of the features,
mRM R(F, Y) =Rel(F, Y)−Red(F), (5.3)
although several other variations exist [58]. Finding the optimal feature subset is infeasible, and so in practice a forward greedy search is used to iteratively select the feature that satisfies,
max Xi∈X\F
Rel({Xi}, Y)−Red(F∪ {Xi}), (5.4)
whereFis the set of currently selected features at each step.
In most applications of mRMR, Cor(·) is given by MI [58], and this is re-
ferred to as M ImRM Rin this thesis. MI is biased as discussed in Chapter 4,
however, and increases with the number of values a variable has, which harms the selection process. One way to reduce this bias is to normalise MI by the entropy of the variables and target, as in SU [144, 160]. Where SU is used in
place of Cor(·) for mRMR, it is referred to as SU mRM R. This approach is
also imperfect since it does not account for other potential biases in the data or feature selection process [65]. Another approach to mitigating these biases is to use the permutation method [65].
5. Redundant permutation feature selection
to combining a permutation method with redundant feature selection are dis- cussed using mRMR as an example, and the permutation redundancy measure is introduced in Section 5.2. In Section 5.3, we apply this redundant feature selection method to data available from the UCI repository, with redundancy being artificially introduced to the data. Finally, we present our conclusions in Section 5.4