Type I Error Rate Control and Choice of Null Distribution

This section describes how to create a null distribution of the test statistics through the bootstrap, instead of the permutation approach, such that the FWER error rate is preserved while using the maxT or minP procedures (Dudoit et al. (2004), van der Laan et al. (2004), Pollard et al. (2005)). In this section, the error rate of inter- est will be the FWER and it will be denoted byF W ER(Vn)to denote its dependence on the unknown number of falsely rejected null hypotheses Vn.

General Test Statistics Null Distribution

In a multiple testing setting, a rejection region is dened such that the type I error rate is controlled at a given level α, i.e., such that

F W ER(Vn) ≤α (2.6)

lim sup

n→∞

F W ER(Vn) ≤α. (2.7)

Inequality (2.6) and (2.7) are called nite sample control and asymptotic control, re- spectively. The type I error rate F W ER(Vn) is dened under the true distribution Qn(P) of the test statistics T_n, which in turn are functions of the underlying data generating distribution P. However, P is usually unknown to the statistician or the researcher and therefore is replaced by an assumed null distributionQ0, or an estimate of it, denoted by Q0n. The null distribution Q0 has be chosen in such a way that the type I error rate is controlled under the true distribution Qn(P) and not only Q0. In order for this to happen, the error rate under the assumed null distribution Q0 must be such that it dominates the rate under the true distributionQn(P). In other words,

the following null domination condition must be satised:

F W ER(Vn) ≤F W ER(V₀), lim sup

n→∞

F W ER(V_n) ≤F W ER(V0).

Here V0 denotes the number of Type I errors under Q0, i.e., for Tn ∼ Q0. Dudoit, Pollard and van der Laan, in the papers mentioned before, for controlling the type I error rate, propose a null distribution Q0(P) which is the asymptotic distribution of the M-vector Zn of null value shifted and scaled test statistics

Zn(m) ≡ ¿ Á Á Á Àmin ⎛ ⎝ 1, τ0(m) V ar[Tn(m)] ⎞ ⎠ (Tn(m) +λ0(m) −E[Tn(m)]), m=1, ..., M.

For the test of single-parameter null hypotheses using the t-statistics, the null values areλ0(m) =0andτ0(m) =1. For testing the equality ofK population means using the F-statistics, the null values areλ0(m) =1andτ0(m) =2/(K−1), under the assumption of equal variances in the dierent populations. Stepwise procedures based on such a null distribution do indeed provide the desired asymptotic control of the Type I error rate F W ER(V_n), for general data generating distributions, null hypotheses, and test statistics.

Bootstrap-based multiple testing procedures

The test statistics null distribution Q0 =Q₀(P) depends on the true data generating distributionP and is therefore typically unknown. It can be estimated with the boot- strap as explained below.

Bootstrap estimation of the test statistics null distribution Q0

Xn = {Xi ∶ i = 1, ..., n}. For the model-based bootstrap, Pn∗ belongs to a model M

for the data generating distribution P, such as a family of multivariate Gaussian dis- tributions. One then proceeds as follows to generate the bootstrap test statistics null distribution.

1.- Obtain thebth bootstrap dataset,Xb

n = {Xib ∶i=1, ..., n},b=1, ..., B, by generating

n i.d.d. random variables Xb

i with distribution Pn∗.

2.- For each bootstrap dataset Xb

n, compute the M-vector of test statistics, Tn(⋅, b) =

(T_n(m, b) ∶ m = 1, ..., M), which can be arranged in an M ×B matrix, Tn ≡ (Tn(m, b)), with rows corresponding to the M null hypotheses and columns to

the B bootstrap samples.

3.- For each null hypothesis H0(m), compute empirical means E[Tn(m,⋅)] ≡ ∑bTn(m, b)/B and variances

V ar[Tn(m,⋅)] ≡ ∑b(Tn(m, b) −E[Tn(m,⋅)])2/B of the B bootstrap test statistics Tn(m, b) (i.e., row means and variances of the matrix Tn), to yield estimates of

E[Tn(m)] and V ar[Tn(m)], respectively, m=1, ...M.

4.- Obtain anM XBmatrix,Zn≡ (Zn(m, b)), of null value shifted and scaled bootstrap

statisticsZn(m, b), by row-shifting and scaling the matrixTnusing the bootstrap

estimates ofE[Tn(m)]and V ar(Tn(m))and the user-supplied null values λ0(m) and τ0(m). That is,

Zn(m, b) ≡ ¿ Á Á Á Àmin ⎛ ⎝ 1, τ0(m) V ar[Tn(m,⋅)] ⎞ ⎠ (Tn(m, b) +λ0(m) −E[Tn(m,⋅)]).

5.- The bootstrap estimateQ0nof the null distributionQ0 is the empirical distribution of the B columnsZn(⋅, b) of the matrix Zn.

Bootstrap estimation of common cut-os and adjusted p-values for the single-step maxT procedure:

0.- Apply the previous bootstrap to generate an M ×B matrix, Z_n = (Z_n(m, b)), of null value shifted and scaled bootstrap statisticsZn(m, b).

1.- Compute the maximum statistic, maxmZn(m, b), b = 1, ..., B, for each bootstrap datasetXb

n, i.e., each column of the matrix Zn.

2.- For controlling the FWER at nominal level α ∈ [0,1], the bootstrap single-step maxT common cut-oc(Q0n, α) is the(1−α)-quantile of the empirical distribu- tion of the B maxima {max_mZ_n(m, b) ∶b=1, ..., B}.

3.- The bootstrap single-stepmaxT adjusted p-value for null hypothesis H0(m)is the proportion of maxima {max_nZ_n(m, b) ∶b =1, ..., B} exceeding the corresponding observed test statistic Tn(m),

˜ P0n(m) ≡ 1 B B ∑ b=1 I(max_mZ_n(m, b) ≥T_n(m)), m=1, ..., M.