Penalized regression procedures aim to discover useful sparse patterns in high dimen- sional regression. Our numerical results demonstrated again that these methods can sometimes be quite unstable. To provide more information on reliability of a selected sparse pattern, we have introduced the concept of variable selection deviation (VSD) to measure the uncertainty of the selection in terms of inclusion of predictors in the model.
The Stability Selection improves the existing model selection method and gives a stable set of variables, which does not depend on the size of penalization parameters. Based on our study, by setting the threshold value to 0.9, the selected set is very stable, except the grouping structure case. Furthermore, the selection size pattern of
the Stability Selection is very similar to the forward selection with modified BIC. The VSD measures based on the ARM and BIC weighting with the non-uniform prior can be very helpful in pointing out how many important variables are possibly missed and how many are unnecessary in the selected model. Clearly, VSD measures are relative to the weighting assignment, and they can only quantify deviations from the models supported by the weights. Fortunately, we have seen that under the ARM weighting, for noise level not too high, the VSD values are very close or reasonably close to the actual deviation size between the true model and selected model and hence provide quite useful information on reliability of the selected model. When the noise level is high, without additional information, it is not possible to reliably find the true model. Any sensible weighting on the models needs to necessarily concentrate on models of small sizes that only keep the most important terms (or even none). In such a case, the VSD measures would be small, which are addressing the selection of the best model for prediction instead of the true model.
For ARM weighting, we chose the half-half data splitting. As shown in Yang (2001), this gives the best rate of convergence offered by the candidate regression procedures (not necessarily parametric) in terms of estimating the regression function. When comparing a fixed list of models/methods with the worse ones converging at a rate slower than 1/n under the squared L2 loss for estimating the regression function,
the results of Yang (2007a) (see the proof of Theorem 1 there) suggest that the half- half splitting would lead to a consistent weighting. A rigorous theoretical investigation is needed to understand when the ARM method yields a weakly consistent weighting for high-dimensional regression.
Another comment is on the use of CV for tuning parameter selection. Based on our simulations, 5-fold CV sometimes performs much better than 10-fold CV for the purpose of model identification (which is perhaps expected based on the work of Shao (1993) and Yang (2007a)).
Model selection diagnostics are severely missing both in research and application. Suitable model selection diagnostics measures can much improve quality of decisions based on statistical data analysis. For instance, if a biologist is to decide which genes to investigate in expensive and time consuming confirmatory study based on an exploratory data analysis, the VSD measures may honestly tell that the majority of the genes recommended by a method may not be as promising as the selected model suggests.
VSD in Generalized Linear Model
3.1
Introduction
The Generalized linear model (GLM) is a standard framework for modeling the asso- ciation between a continuous/discrete response and a set of independent variables. In modern research and application areas, often a large number of predictors are used to classify the response variable into several categories. For example, the target of the genome wide association studies is to identify a subset of single-nucleotide polymor- phisms (SNPs) that are associated with human diseases over thousands of SNPs (Wu et al. (2009)). In such cases, it is challenging and difficult in the variable selection and the coefficient estimation to use the traditional methods, such as AIC (Akaike (1973)), BIC (Schwarz (1978)) and Cp (Mallows (1973)), due to heavy computation
demand.
Assume that a random sample of n subjects {(yi, xi), i = 1, · · · , n} is observed. Let
Yi be a response variable following a distribution in the exponential family f (yi; θi) =
exp[yiθi− A(θi) + B(yi)], where θi is the parameter and µi = E(Yi) = A0(θi). Let
Xj ∈ Rn, j = 1, . . . , p be p predictors. The usual GLM framework models the mean
µi of Yi via the link function transformation
g(µi) = β0+ βiXi1+ · · · + βpXip.
A variety of exponential distributions (i.e. in McCullagh and Nelder (1989)) can be modeled with different link functions, such as the identity link for Gaussian regression, the logit link log1−prpr = g(µ) for logistic regression where the binary response follows a binomial distribution Bin(n, pr), and the log link for Poisson regression where the response is from a Poisson distribution.
In recent years, the penalization method, an effective and computationally feasible approach, has been well developed to perform the variable selection and parameter estimation. The penalized likelihood estimator is solved by following objective func- tion: ˆ βλ = arg min ( `n(β) + p X j=1 p(|βj|; λ) ) ,
where `n=Pni=1[yiθ(Xi, β) − A(θ(Xi, β))] and p(|βj|; λ) is the penalty function with
a tuning parameter λ. Tibshirani (1996) introduced an L1 penalty for variables called
Lasso. Zhao and Yu (2006) and Zou (2006) proved the inconsistency of Lasso for model selection in certain scenarios. Fan and Li (2001) proposed SCAD, which is a non-concave penalized likelihood method. Zou (2006) proposed the adaptive Lasso, which modified Lasso penalty to guarantee the selection consistency. Zhang (2010) proposed a minimax concave penalty (MCP) and developed a fast penalized linear unbiased selection algorithm. The results of penalization methods heavily depend on the amount of regularization. Therefore, choosing a proper amount of regularization is critical. On the other hand, the theoretical results of penalization methods are usually derived under the assumption of sparsity. Sometimes, the assumption is not easy to justify. In particular, when the sample size is much smaller than the number predictors, sensible variable selection methods tend to select a sparse model. There- fore, we may question the reliability of the results of a penalization method. How much certainty do we have about the set of variables selected by the regularization method on the current data? Also, is the sparse pattern discovered by these variable
selection methods real?
In literature, the uncertainty of model selection methods has been well studied, such as Breiman (1996b), Yuan and Yang (2005), and Chen et al. (2007). When the dimensionality is large, we expect the instability of a model selection method to be high. Nan and Yang (2014) tested three instability measures for three penalization methods: Lasso, SCAD, and MCP in the context of linear regression. The results showed that these methods sometimes could be quite unstable. They also proposed a model selection diagnostic measure called variable selection deviation (VSD), which provides a proper sense on how many predictors in the selected set are likely trustwor- thy in certain aspects. Rather than focusing on the sense of instabilities of a model selection method, the VSD evaluates the reliability of a set of selected variables and captures the difference from the underlying true set of predictors.
In the generalized linear model context, the VSD measures also uses a weighting mechanism to help measure the deviation of the selected variables by a generalized linear model method from the true model, so that users will have a reasonable sense about the reliability of the set of selected variables. In this chapter, we extend the VSD measures from the linear regression setting to the generalized linear model, in particular, logistic regression. We also propose the weighting function for poisson regression, in order to demonstrate the implementation of the VSD measures to other generalized linear models.
The chapter is organized as follows. In Section 2, the theoretical results of the VSD for generalized linear model are proposed, and we also introduce the weight function for logistic regression and the corresponding algorithm. The numerical results of simulation studies and the studies of four real data examples are shown in Section 3.