In modelling of large regression data sets, where the number of predictors p far exceeds the number of observations n, identifying important predictors is a cru- cial yet a complex task. In previous chapter, variable selection was introduced to deal with such high dimensional data and reduce the dimension by selecting important variables. We also discussed a branch of variable selection methods called regularisation methods. These methods perform variable selection and parameter estimation simultaneously by imposing a sparsity-inducing penalty on the residual sum of squares function. Although regularisation methods fa- cilitate the analysis of data with p≫ n, they may not be practically efficient in high dimension settings where the number of predictors is as large as a few thousand. For example, in many modern applications, where the data are collected from areas such as genomics, microarrays, finance and brain images, these methods often suffer from computational deficiency. Moreover, condi- tions that are necessary to hold for selection to be consistent may not hold due to the significant difference between p and n (Wang et al., 2015); (Fan
et al., 2009). To address these challenges, variable screening can be applied to reduce the dimension. In the variable screening, a measure is defined to evalu- ate and rank the importance of each variable. This is followed by thresholding the ranked variables to end up with a reasonable size of the variables. In the linear regression context with high dimension in predictors, the importance of predictors is specified according to the influence of the predictors on the response variable. Therefore, predictors with a weak impact on the response variables are removed.
Variable screening procedures fall into two main categories: model-free and model-based procedures. Unlike model-based approaches, in model-free screening procedures, imposing a specific model structure on regression func- tions is not necessary. The recent literature in screening procedures is very rich and covers a broad variety of models such as linear regression models, generalised linear models, parametric and non-parametric regression models and even nonlinear models. Liu et al. (2015) provide an excellent overview of these feature screening methods for high dimensional data.
Here, our focus is on model based screening procedures for linear regres- sions. We review a pioneering work in this field called sure screening. The concept of sure screening was first introduced by Fan and Lv (2008) where the aim is to reduce the dimension of variables to a moderate size as small as sample size n, while maintaining the informative part of the variables in the model. A variable screening procedure has sure screening property if the survival probability of important variables after screening tends to one. The Sure Independence Screening (SIS) proposed by Fan and Lv (2008), is a selec- tion technique based on the marginal Pearson correlations of predictors with the response variable. Through this technique, the importance of predictors are evaluated based on their marginal correlations with the response variable, and as a result, predictors that have a weak correlation with the response are discarded. Consider the following univariate multiple regression model
where each column of the matrix Xn×pdenote n observations on each predictor
xk; k = 1,· · · , p and the vector y contains n observations on the response variable Y and ǫ = (e1,· · · , en) is the error term and β = (β1,· · · , βp) is the
regression coefficient vector. If we denote the estimated coefficient vector by ˆ
β = ( ˆβ1,· · · , ˆβp), by applying (SIS) the regression coefficients corresponding
to each predictor k are ranked and thresholded. Then, the predictors with the highest regression coefficients are selected. The following reduced model is obtained through the SIS technique. For any given γ ∈ (0, 1),
Mγ ={1 ≤ j ≤ p : | ˆβj| is among the first [γn] largest of all}, (3.2.2)
where ˆβj = XTjy; j = 1,· · · , p and [γn] denotes the integer part of γn. Here,
the assumption is that both X and y are standardised which implies that ˆβj
is actually the Pearson correlation between the jth predictor and the response variable. As a result of this screening, the full model M = {1, · · · , p; p ≫ n} is shrunk to the submodel Mγ of size d = [γn] < n.
Since SIS procedure is based on the marginal correlations, it may not be perfectly efficient in practice. The reason is that, an important predictor which is marginally uncorrelated but jointly correlated with the response is neglected by SIS; whereas, an unimportant predictor that is highly correlated with important predictors are more likely to be selected than other impor- tant predictors with weak correlation with the response variable. To address this issue and enhance the screening accuracy, Fan and Lv (2008) proposed an iterative sure independent screening process (ISIS). The first step of this iterative process starts by selecting a subset of important variables using a variable selection method, say Lasso, then a regression model is fitted to this subset and the fitted residuals are obtained. In the next step, these residuals are treated as response variables. Thus, a model is fitted to these residuals and the remainder of unimportant predictors in the previous step. In such an iterative procedure, the unselected important variables in previous steps can survive. Although the aim of screening is to reduce the high dimension of variables to a dimension as small as sample size n, a model with size d ≥ n
can also be shosen. In fact by choosing large d, the probability that the true model is included in the submodel Mγ is increased. The possible drawback
of such choices is the computational cost. Fan and Lv (2008) chose d = n− 1 and d = [n/ log n] in implementing SIS. According to their numerical results, choosing a submodel of size [n/ log n] is consistent with the sure screening property of SIS.
SIS is designed in linear regression framework and Pearson correlation cap- tures the linear dependancy, so to extend this correlation to a nonlinear case, Hall and Miller (2009) proposed a generalised correlation that captures both linear and nonlinear correlations.
Despite the fast growing research in variable screening for univariate re- gression models wherein one response variable is regressed against a set of predictors, less progress has been made in screening methods that are suitable for multivariate regressions where multiple response variables are regressed against a set of predictors. In this thesis, modelling multi-response data is a centre of attention hence, in the next section we revise the SIS procedure to become applicable to the regression models with multivariate (multiple) response variables.