Massive Data Example

This example compares the SaM approach with several popular penalized likeli- hood approaches for massive data sets. We simulated 100 data sets with σ∗2 = 0.25. Each data set was generated from a multivariate normal distribution with n = 100 observations andpn = 500,000 predictors, among which 2,000 predictors are equally

correlated with a correlation coefficient of ρ = 0.25 and the rest predictors are un- correlated. Among those 2,000 correlated predictors, 5 of them were chosen as the true predictors with the regression coefficients (0.2, 0.3, 0.4, 0.5, 0.6).

Each data set was randomly split into 1,000 subsets with s = 500. SAMC was then run for each subset data and the aggregated subset data with the same setting as for Example 2 of Section 2.3.1. In simulations, we set ¯s= 45 as the upper bound of model size for the subset data and ¯rn=35 for the aggregated subset data, and

set the FDR level α1 = 0.15 for variable selection in stage I and α2 = 0.01 in stage

II. On average (over the 100 data sets), SaM reduced the dimension of the data to 18,057.7 in stage I and then selected 5.3 predictors in stage II.

For comparison, several popular penalized likelihood approaches, including Lasso, elastic net, SIS and ISIS, were also applied to this example. Lasso was imple- mented for this example using the R package glmnet [24] with the tuning param- eter λ determined through a 10-fold cross-validation procedure. Elastic net was also implemented using the R package glmnet, for which we set the penalty function pλ =λ(||β||L1+||β||

2

L2) and determined the value ofλ via a 10-fold cross-validation

procedure. We used the R package SIS to implement SIS-SCAD and ISIS-SCAD, which first reduce the number of predictors to n/logn by marginal utility, and then apply SCAD to refine the model. The results are summarized in Table 2.2.

Methods SaM Lasso Elastic net SIS ISIS

|ξ|a 5.3(0.22) 44.62(0.98) 52.70(0.21) 11.47(0.29) 13.57(0.59) |ξ∩t|b 3.53(0.073) 4.24(0.070) 4.07(0.074) 3.31(0.083) 3.37(0.086) fsr(%) 25.5(2.30) 89.7(0.38) 92.3(0.15) 67.7(1.58) 65.8(2.50) nsr(%) 29.4(1.46) 15.2(1.40) 18.6(1.48) 33.8(1.65) 32.6(1.72) MSEc _0.146(.013)d _{0.263(.011) 0.488(.011) 0.246(.016) 0.342(.022)}

Table 2.2: Simulation results for half-million-predictor data sets. The table compares the SaM result with Lasso, elastic net, SIS and ISIS for the massive data example: a average number of predictors selected by different methods; b _{average number of true}

predictors selected by different methods; c_{mean squared error ofβ}

ξ, i.e.,kβξ−β ∗_k2_,

produced by different methods;dthe posterior mean ofβby model average is used as ˆ

β for SaM method; The numbers in the parentheses are the corresponding standard deviations.

To measure the performance of different methods for this example, we calculated the false selection rate (fsr) and negative selection rate (nsr). Lett denote the set of

true predictors, and let ξ denote the model selected by a method. Then, fsr and nsr can be defined as fsr = |ξ\t| |ξ| , nsr = |t\ξ| |t| ,

where | · | denotes the size of a model. The smaller values of fsr and nsr are, the better the performance of the method is. From Table 2.2, it is easy to see that SaM yields a much smaller value of fsr than all other methods, and about the same value of nsr as SIS and ISIS. Lasso and elastic net yield a smaller nsr but at the expense of a high fsr. We have also compared the mean squared error of the estimator ofβ∗. For SaM approach, we use the posterior mean of the second Bayesian analysis stage as the estimator of true coefficients. Strictly speaking, this estimator is not a sparse estimation, but it, as shown in Table 2.2, does indicate that SaM is more accurate than those selected by other methods.

A final remark for this massive data simulation is the computation cost. The same setting of Markov chain burn-in period and total iteration numbers are used as in section 2.3.1. A serial analysis of this big data set cost approximated 16 hours for Markov chain simulation on a single core of IntelR _XeonR _{CPU E5-2690(2.90Ghz).}

The computation time should be significant reduced if implemented in a parallel ar- chitecture. Since our approach has an embarrassingly parallel structure, the imple- mentation of parallel computing does require communication among different nodes except collecting the selected variables’ index at the end of first stage. Thus the bandwidth limit of the connection between nodes should not influence the computa- tion time very much. Recently, there has been a development of scalable continuous shrinkage prior based on mixture of normal distribution (see e.g. [26], [59]), which aim at avoiding reversible jumps and reducing computation time. Gibbs sampler is always implemented, where the full condition of β follows multivariate normal

distribution, with computation complexity of order O(p3

n). In case of pn equals half

million, this is not feasible at all. The authors experience shows that it cost one day to update only 20 more iterations. In contrast, the computation complexity of proposed posterior is at most of the order ofO(¯r3

n). [3] show that such shrinkage pri-

ors lead to posterior consistency in the case of pn =o(n), and [59] also demonstrate

its prediction outperformance compared to frequentist methods. However, to the authors’ best knowledge, a general study of estimation consistency under shrinkage prior for the high dimensional regression is not established yet. In Appendix C, we present one simulation study that horseshoe prior fails for high dimensional variable selection.

2.4 Real Data Examples