Related Work on Combining Different Ensemble Methods

ods

Following the predictive advantage shown by ensemble methods, further gains have been reported by approaches that combine different ensemble learning methods. Webb (2000) developed Multiboosting, which combines AdaBoost

(Freund and Schapire, 1997) with Wagging (Bauer and Kohavi, 1999) (a variant of bagging (Breiman, 1996)). The idea is to use AdaBoost as the individual learning algorithm for Wagging. In a posterior work, Webb and Zheng (2004) claim that Multiboosting and other similar approaches provide a better trade- off between the error of the individual members of the ensemble and the di- versity among them. Yu et al. (2007) presented an approach for combining a number of regression ensembles, dubbed Cocktail Ensemble, using the ambi- guity decomposition. Wu et al. (2001) propose E-GASEN (Genetic Algorithm based Selective Ensemble method), a neural network ensemble. Essentially, this approach combines several GASEN ensembles using a simple average. A GASEN ensemble works by initially building a set of neural network models; afterwards, a genetic algorithm is run to prune the ensemble. In 2000, Pfahringer (2000) won the well-known KDD Cup competition with a combination of bagging and boosting (“bagged boosting”). The EasyEnsemble and BalanceCascade (Liu et al., 2009) are another two approaches that combine bagging with boosting, focusing on imbalanced learning problems.

A key distinguishing feature of this type of approaches is that they typically combine different ensemble methods during the learning process. We hypothe- sise that similar effects can be obtained from a portfolio of pre-trained heteroge- neous models, like the ones used in the previous chapter. This is the main goal for the work in this chapter. This setting can be advantageous because experts in a portfolio are typically independent and thus easily parallelised (Caruana et al., 2004).

5.1.2

Our Approach: Constructive Aggregation

In this chapter, we propose an aggregation framework for a set of diverse and independently created models M , following the basic principles of constructive induction (Wnek and Michalski, 1994). Constructive induction refers to proce- dures that modify a set of original attributes, where some of the attributes are removed, others are added, and some of the existing ones are aggregated (Wnek and Michalski, 1994). This leads to a new set of attributes which hopefully provides an overall better description for approximating a regression or classifi- cation function f relative to the original set. We follow a similar approach, but

in our work, the attributes refer to predictive models.

The proposed aggregation framework works by rearranging a set of diverse

models M into different, overlapping subsets (denoted as CM). The elements

in each of these subsets are then aggregated, leading to a new set of models

M0 _{(or sub-ensembles, as denoted by Webb and Zheng (2004)). Similarly to}

constructive induction approaches, the search for CM is done by analysing the

individual predictive performance of the original models M in observations not used in the learning process (e.g. a validation set). Essentially, the new models

m0 ∈ M0_{correspond to aggregated subsets of consistently top performing models}

from M . We refer to this approach as Constructive Aggregation (CA). CA can be thought of a hierarchical approach for aggregating a set of predictive models. Our working hypothesis is that similarly to approaches for combining different ensemble methods (Webb and Zheng, 2004), CA leads to a decrease in the individual error of ensemble members, without overly decreasing the diversity among them.

To illustrate our idea, the workflow of CA is presented in Figure 5.2 with

M = {m1_{, m}2_{, m}3_{}. After analysing the performance of each model in unseen}

observations, the set of committees CM = {{m1}, {m1, m2}, {m2, m3}} is cre-

ated; then the models mi _{within each committee are aggregated into models}

M0. Finally, the new set of combined models M0 is aggregated into a final

approximation ˆyM0_{. Both of these aggregations are done according to a linear}

combination (Equation 5.1). The construction of the committee set CM is car-

ried out by applying the concept of out-performance contiguity (OC), which is also formalised in this chapter.

Although we have not formalised our approach yet, we present a motivating example similar to the one shown in Figure 5.1. As a preliminary analysis, we applied CA to the time series used in the experiment shown in the introduction of this chapter. We used the same portfolio of 50 models, which lead to the creation of 79 different subsets (we will explain how these are obtained in the next section). These subsets were aggregated and evaluated according to their rank in each testing point, whose distribution is illustrated in Figure 5.3. Similarly to the individual 50 predictive models, different groups of models show a varying relative performance in the testing observations of the time series.

m1 m2 m3

ŷ

Figure 5.2: Workflow of constructive aggregation: the set of the available models

M is rearranged into different subsets CM. Each subset is aggregated into M0,

and become hypotheses for approximating f . The models in M0_{are aggregated}

into the final decision ˆyM0_.

In experiments using the 62 time series from several domains (c.f. ap-

pendix A.1), aggregating a number of forecasting models using CA provides a consistent advantage in terms of predictive performance. That is, applying

state of the art aggregation methods to M0 leads to a better predictive per-

formance relative to applying them to M . Moreover, we provide results that demonstrate that the constructive aggregation process entails a small execution time overhead. In summary, the contributions of this work are the following:

• Constructive Aggregation (CA), a new concept which consists in rearrang-

ing a portfolio of experts M into different subsets CM, aggregating them,

and using them as hypotheses M0_;

• Out-performance Contiguity (OC), an approach for building CM in time-

dependent forecasting problems;

• An extensive set of experiments including: paired comparisons that quan- tify the percentual difference in error between using CA and aggregating M directly (non-CA); execution time analysis of CA using OC; sensitivity analysis of the main parameters behind the approach; and a study of CA in terms of bias-variance-covariance trade-off.

0 20 40 60 80 m'6 m'1 m'9 m'4 m'2 m'12 m'10 m'22 m'23 m'25 m'8 m'26 m'27 m'29 m'30 m'14 m'13 m'15 m'28 m'32 m'34 m'33 m'36 m'17 m'18 m'20 m'35 m'24 m'5 m'37 m'19 m'21 m'31 m'38 m'7 m'39 m'40 m'16 m'41 m'11 m'42 m'45 m'44 m'43 m'46 m'47 m'48 m'3 m'49 m'50 m'57 m'51 m'68 m'52 m'54 m'62 m'61 m'53 m'56 m'66 m'72 m'55 m'71 m'77 m'58 m'69 m'60 m'70 m'78 m'59 m'76 m'65 m'63 m'74 m'64 m'73 m'67 m'75 Rank

Figure 5.3: Distribution of rank of 79 aggregated groups of forecasting models across the testing observations of a time series

problems via OC. In Section 5.3, we provide an extensive set of experiments to validate the proposal. Afterwards, in Section 5.4, we discuss the results and limitations of our work. Finally, we summarise this chapter in Section 5.5. The

proposed method is publicly available as an R software package2.