Bootstrapping in non-regular smooth function models

Contents lists available atSciVerse ScienceDirect

Journal of Multivariate Analysis

journal homepage:www.elsevier.com/locate/jmva

Bootstrapping in non-regular smooth function models

Mihai C. Giurcanu

University of Florida, United States

a r t i c l e i n f o Article history:

Received 23 April 2011 Available online 9 May 2012 AMS subject classifications: 62G09

62G15 62G20 Keywords:

Smooth function model Non-regular estimators Standard bootstrap m-out-of-nbootstrap Oracle bootstrap Confidence intervals

a b s t r a c t

We study the large sample behavior of the standard bootstrap, them-out-of-nbootstrap, and the oracle bootstrap (Giurcanu and Presnell, 2009) [14] percentile confidence intervals in non-regular smooth function models. We show that the oracle bootstrap percentile confidence intervals are consistent while the standard bootstrap and the m

-out-of-nbootstrap confidence intervals are inconsistent. Further analysis of coverage probabilities reveals that, for large samples, the iterated oracle bootstrap percentile confidence intervals are more accurate than their non-iterated versions. We also describe the large sample local behavior of the bootstrap confidence intervals for parameter values near the points of inconsistency of the standard bootstrap. In a simulation study, we describe the finite sample local behavior of various bootstrap confidence intervals.

©2012 Elsevier Inc. All rights reserved.

1. Introduction

Babu [1] shows that, in non-regular smooth function models, the standard bootstrap estimator of thedistributionof a smooth function of the sample mean is inconsistent for some values of the mean parameter vector, while Shao [25] shows that them-out-of-nbootstrap estimator is consistent over the entire parameter space. In this paper, we show that both the standard bootstrap and them-out-of-nbootstrap percentileconfidence intervalsare inconsistent for some values of the mean parameter vector, while the test inversion and them-out-of-nbootstrap test inversion confidence regions are consistent over the entire parameter space. Moreover, asymptotic expansions of coverage probabilities reveal that, for large samples, the test inversion confidence regions are more accurate than them-out-of-nbootstrap test inversion confidence regions.

Giurcanu and Presnell [14] propose the oracle bootstrap as an alternative to them-out-of-nbootstrap for consistent estimation of the distributions of many non-regular estimators, including the square of the sample mean, the Hodges and Stein estimators, and sparse estimators such as the LASSO. In this paper, we focus on the consistency and the higher order properties of oracle-bootstrap percentile confidence intervals and their iterated versions in non-regular smooth function models. We show that the oracle-bootstrap percentile confidence intervals are consistent over the entire parameter space, and that iteration increases their accuracy by an order of magnitude for large sample sizes.

It is well known that fixed parameter consistency may not adequately describe the large sample behavior of bootstrap estimators. For example, while the standard parametric bootstrap inconsistently estimates the distribution of both the Hodges and Stein estimators over a subset of the parameter space, it nevertheless performs better than the m -out-of-nbootstrap and a parametric version of the oracle bootstrap near the points of inconsistency [24]. More generally, a pointwise-consistent bootstrap estimator of the distribution of a non-locally asymptotically equivariant estimator is typically not locally uniform convergent at the points of inconsistency of the standard bootstrap [2,21]. In our local asymptotics analysis, we show that if the local parameter is close to a point of inconsistency, then the oracle bootstrap percentile confidence intervals are consistent, and that both the standard bootstrap and them-out-of-nbootstrap percentile

E-mail address:[email protected].

0047-259X/$ – see front matter©2012 Elsevier Inc. All rights reserved.

confidence intervals are inconsistent. Furthermore, if the local parameter is moderately close to a point of inconsistency, then all bootstrap confidence intervals are inconsistent. Finally, if the local parameter is moderately far from a point of inconsistency, then all bootstrap confidence intervals are consistent.

We conclude this section with an outline. In Section2, we present the asymptotic behavior of the standard bootstrap percentile confidence intervals in non-regular models. In Section3, we present the large sample results for them

-out-of-nbootstrap confidence intervals and for them-out-of-nbootstrap test inversion confidence regions. In Section4, we present the asymptotic behavior of the oracle bootstrap percentile confidence intervals and their iterated versions and in Section5, we study the large sample local behavior of the bootstrap confidence intervals. In Section6, we provide some computational details and the results of an empirical study which describe the finite sample local behavior of the bootstrap confidence intervals. The proofs of the theoretical results can be found in theAppendix.

2. Standard bootstrap estimation

Consider thesmooth function model(see, e.g., [4,15]) given by a random sampleX1:n

=

X1

, . . . ,

Xnof i.i.d. observations

from a distribution_Pon_Rd_{such thatE}

₍

_X

₎

₌

_µ

_{and var}

₍

_X

₎

₌

_{Σ, whereX}

_∼

PandΣis positive definite. Let

θ

=

f

(µ)

be the parameter of interest, wheref

:

_Rd

_→

Ris asmoothfunction, and let

θ

ˆ

n

=

f

(

X

¯

n

)

be itsnaturalestimator, where

¯

Xnis the sample mean. The smooth function model is calledregular if

∇

f

(µ)

̸=

0 for all

µ

∈

Rdandnon-regularif there exists a

µ

∈

_Rd_{such that}

_∇

_f

_(µ)

₌

_{0, where}

_∇

_f

_(µ)

_{is the gradient off}

_(µ)

_{. LetΘ}

₌



f

(µ)

:

µ

∈

_Rd

_,

_∇

2_f

_(µ)

_̸=

₀



, S

=



µ

∈

_Rd

_{: ∇}

_f

_(µ)

₌

₀



, andΘ0

=



f

(µ)

:

µ

∈

S



be the image set ofSunderf, where

∇

2_f

_(µ)

_{is the Hessian matrix} off

(µ)

. By a two-term Taylor expansion off

(

X

¯

n

)

about

µ

, we have

f

(

_X

¯

n

)

=

f

(µ)

+

γ

T

(

X

¯

n

−

µ)

+

(

X

¯

n

−

µ)

TΓ

(

X

¯

n

−

µ)

+

oP

(

n

−1

_),

where

γ

= ∇

f

(µ)

andΓ

=

(

1

/

2

)

∇

2_f

_(µ)

_{. If}

_θ

_∈

_Θ

_\

_Θ

0, thenn1/2

(

θ

ˆ

n

−

θ)

N

(

0

, γ

TΣ

γ )

, where denotes convergence

in distribution. If

θ

∈

Θ₀, thenn

(

θ

ˆ

n

−

θ)



d

j=1

λ

jUj2, where

λ

1

, . . . , λ

dare the eigenvalues ofΣ1/2Γ Σ1/2andU1

, . . . ,

Ud

are i.i.d.

∼

N

(

0

,

1

)

. LetF

(

x

)

=

Pr

(γ

TZ

≤

x

)

andG

(

x

)

=

Pr

(

ZTΓZ

≤

x

)

, whereZ

∼

N

(

0

,

Σ

)

.

The standard bootstrap confidence intervals were introduced by Efron [13] and their higher order properties were characterized by Hall [16]. LetX∗

1:n

=

X

∗

1

, . . . ,

X

∗

n be a standard bootstrap resample, i.e.,X

∗

1:nis a with replacement random

sample of sizenfromX1:n. Let

θ

ˆ

n∗

=

f

(

X

¯

∗

n

)

be the bootstrap version of

θ

ˆ

n, and letL

ˆ

_θˆ∗

n

(

x

)

=

Pr



_ˆ

θ

∗ n

− ˆ

θ

n

≤

x

|

X1:n



,F

ˆ

_θˆ∗ n

(

x

)

=

ˆ

L_θˆ∗ n

(

x

/

n 1/2

₎

_{, andG}

ˆ

ˆ θ∗

n

(

x

)

= ˆ

Lθˆn∗

(

x

/

n

)

be the bootstrap estimators ofLθˆn

(

x

)

=

Pr

(

ˆ

θ

n

−

θ

≤

x

)

,F_θˆ_n

(

x

)

=

Pr



n1/2

₍

_θ

ˆ

n

−

θ)

≤

x



, and G_θˆ_n

(

x

)

=

Pr



n

(

θ

ˆ

n

−

θ)

≤

x



, respectively. Note that if

θ

∈

Θ

\

Θ₀andL_θˆ_n

(

x

)

is continuous atL−_θˆ1

n

(

1

−

α)

, whereL−_ˆ1 θn

(

1

−

α)

is the

(

1

−

α)

quantile ofL_θˆ_n

(

x

)

, an exact

α

level upper confidence limit for

θ

is

θ

ˆ

n

−

L

−1 ˆ θn

(

1

−

α)

. Let

ˆ

L−_θˆ∗1 n

(

1

−

α)

be the

(

1

−

α)

quantile of

ˆ

L_θˆ∗

n

(

x

)

, then the

α

level bootstrap percentile upper confidence interval for

θ

is

ˆ

In

(α)

=



−∞

,

θ

ˆ

_n

− ˆ

L−_ˆ1 θ∗ n

(

1

−

α)



. Similarly, the bootstrap percentile lower, equal-tailed, and symmetric confidence intervals for

θ

are_I

ˆ

L

n

(α)

=



_ˆ

θ

n

− ˆ

L−_θˆ∗1 n

(α),

∞



,

ˆ

InET

(α)

=



_ˆ

θ

n

−ˆ

L −1 ˆ θ∗ n

((

1

+

α)/

2

),

_θ

ˆ

n

−ˆ

L −1 ˆ θ∗ n

((

1

−

α)/

2

)



, and_I

ˆ

S n

(α)

=



_ˆ

θ

n

−ˆ

L ′₋₁ ˆ θ∗ n

(α),

ˆ

θ

n

+ˆ

L ′₋₁ ˆ θ∗ n

(α)



, where

ˆ

_L′ ˆ θ∗ n

(

x

)

=

Pr



| ˆ

θ

∗ n

− ˆ

θ

n

| ≤

x

|

X1:n



(see, e.g., [15]).

Recall that an

α

level confidence intervalIn

(α)

is consistent if its asymptotic coverage probability equals the nominal

level, i.e., limn→∞Pr

(θ

∈

In

(α))

=

α

, and is inconsistent otherwise (see, e.g., [26, p. 329]). The following theorem shows

thatI

ˆ

n

(α)

is consistent for

θ

∈

Θ

\

Θ0and is inconsistent for

θ

∈

Θ0. Part (ii) of this theorem gives the asymptotic coverage probability of_I

ˆ

_n

_(α)

_for

_θ

∈

Θ₀. Similar results can be readily derived for the other types of confidence intervals.

Theorem 1.Suppose f is two times continuously differentiable and E

(

∥

X

∥

2

) <

∞

.

(i) If

θ

∈

Θ

\

Θ₀, thenlimn→∞Pr

(θ

∈ ˆ

In

(α))

=

α

.

(ii)If

θ

∈

Θ₀, thenlimn→∞Pr

(θ

∈ ˆ

In

(α))

=

1

−

Pr



ZTΓZ

−

F_N−₍1_0,Σ)

(

1

−

α

;

Z

) <

0



, where F_N−₍1_0,Σ)

(

1

−

α

;

y

)

is the

(

1

−

α)

quantile of the distribution of 2yT_ΓZ

₊

_ZT_{ΓZ .}

3. Them-out-of-nbootstrap estimation

Them-out-of-nbootstrap estimation was introduced by Bickel and Freedman [5] and its large sample and higher order properties were analyzed by Bickel et al. [6]. Cheung et al. describe them-out-of-nbootstrap confidence intervals for

θ

under the constraint that

∇

f

(µ)

=

0. However, solving

∇

f

(µ)

=

0 for

µ

, one may be able to determine the parameter

θ

, and thus, the confidence interval estimation may not be suitable under such constraints on the parameters.

LetX∗

1:m

=

X

∗

1

, . . . ,

X

∗

mbe anm-out-of-nbootstrap resample, i.e.,X

∗

1

, . . . ,

X

∗

mis a with replacement random sample

of sizemfromX1:n, withm

=

o

(

n

)

and m

→ ∞

. Let

θ

ˆ

m∗

=

f

(

X

¯

∗

m

)

be them-out-of-n bootstrap version of

θ

ˆ

n, and

letL

ˆ

_θˆ∗ m

(

x

)

=

Pr



_ˆ

θ

∗ m

− ˆ

θ

n

≤

x

|

X1:n



,F

ˆ

_θˆ∗ m

(

x

)

= ˆ

Lθˆm∗

(

x

/

m 1/2

₎

_{, andG}

ˆ

ˆ θ∗

percentile upper confidence interval for

θ

isI

ˆ

m

(α)

=



−∞

,

θ

ˆ

_n

−

(

m

/

n

)

1/2L

ˆ

−_θˆ∗1

m

(

1

−

α)



. Similarly, them-out-of-nbootstrap percentile lower, equal-tailed, and symmetric confidence intervals for

θ

are_I

ˆ

L

m

(α)

=



_ˆ

θ

n

−

(

m

/

n

)

1/2

ˆ

L−_θˆ∗1 m

(α),

∞



,_I

ˆ

ET m

(α)

=



_ˆ

θ

n

−

(

m

/

n

)

1/2L

ˆ

−_θˆ∗1 m

((

1

+

α)/

2

),

θ

ˆ

n

−

(

m

/

n

)

1/2L

ˆ

−_θˆ∗1 m

((

1

−

α)/

2

)



, and_I

ˆ

S m

(α)

=



_ˆ

θ

n

−

(

m

/

n

)

1/2

ˆ

L ′₋₁ ˆ θ∗ m

(α),

ˆ

θ

n

+

(

m

/

n

)

1/2

ˆ

L ′₋₁ ˆ θ∗ m

(α)



, where_L

ˆ

′ ˆ θ∗ m

(

x

)

=

Pr



| ˆ

θ

∗ m

− ˆ

θ

n

| ≤

x

|

X1:n



. The following theorem shows that them-out-of-nbootstrap percentile upper confidence interval is consistent for

θ

∈

Θ

\

Θ₀and is inconsistent for

θ

∈

Θ₀.

Theorem 2. Suppose f is two times continuously differentiable, E

(

∥

X

∥

2

) <

∞

,

λ

j

≥

0for all j

=

1

, . . . ,

d, m

=

o

(

n

)

, and

m

→ ∞

. Thenlimn→∞Pr

(θ

∈ ˆ

Im

(α))

=

α

if

θ

∈

Θ

\

Θ0andlimn→∞Pr

(θ

∈ ˆ

Im

(α))

=

0if

θ

∈

Θ0.

Bootstrap confidence regions can be derived by inverting bootstrap test statistics in analogy with the classical theory (see, e.g., [9, Section 9.2.1]). Consider testingH0

:

θ

=

θ

0versusHa

:

θ > θ

0using the test statisticTn

(

X1:n

, θ

0

)

=

n1/2

(

θ

ˆ

n

−

θ

0

)

if

θ

0

_∈

_Θ

_\

_Θ

0andTn

(

X1:n

, θ

0

)

=

n

(

θ

ˆ

n

−

θ

0

)

if

θ

0

∈

Θ0. UnderH0,Tn

(

X1:n

, θ

0

)

γ

TZif

θ

0

∈

Θ

\

Θ0andTn

(

X1:n

, θ

0

)

ZTΓZ

if

θ

0

∈

Θ₀. An

α

level test inversion confidence region for

θ

is defined as In

(α)

=



θ

∈

Θ

\

Θ₀

: ˆ

F_n−1

(

1

−

α)

≤

n1/2

(

θ

ˆ

n

−

θ)



∪



θ

∈

Θ₀

: ˆ

G−_n1

(

1

−

α)

≤

n

(

θ

ˆ

n

−

θ)



,

whereF

ˆ

n

(

x

)

=

Pr

(

γ

ˆ

nTΣ

ˆ

1/2 n U

≤

x

|

X1:n

)

andG

ˆ

n

(

x

)

=

Pr

(

UTΣ

ˆ

1/2 n Γ

ˆ

nΣ

ˆ

1/2

n U

≤

x

|

X1:n

)

are conditional distribution functions

givenX1:n,U

∼

N

(

0

,

I

)

is independent ofX1:n,

γ

ˆ

n

= ∇

f

(

X

¯

n

)

,Γ

ˆ

n

=

(

1

/

2

)

∇

2f

(

X

¯

n

)

, andΣ

ˆ

n

=

n−1



ni=1

(

Xi

− ¯

Xn

)(

Xi

− ¯

Xn

)

T.

The correspondingm-out-of-nbootstrap test inversion confidence region for

θ

is

ˆ

Jm

(α)

=



θ

∈

Θ

\

Θ₀

: ˆ

F−_ˆ1 θ∗ m

(

1

−

α)

≤

n1/2

(

_θ

ˆ

_n

−

θ)



∪



θ

∈

Θ₀

: ˆ

G−_ˆ1 θ∗ m

(

1

−

α)

≤

n

(

_θ

ˆ

_n

−

θ)



.

The following corollary shows that the test inversion confidence regions and the m-out-of-n bootstrap test inversion confidence regions are consistent over the entire parameter space.

Corollary 1. Suppose f is two times continuously differentiable, E

(

∥

X

∥

2

_{) <}

_∞

_{, m}

₌

_o

₍

_n

₎

_{, and m}

_{→ ∞}

_{. Then, for all}

_θ

_∈

_Θ, limn→∞Pr

(θ

∈

In

(α))

=

α

andlimn→∞Pr

(θ

∈ ˆ

Jm

(α))

=

α

.

SupposeE



∥

X

∥

5



<

∞

and that the characteristic function ofXsatisfies the Cramer condition lim sup∥t∥→∞

|

Eexp

(

itTX

)

|

<

1, and thus, the Edgeworth expansion of the distribution functionFZn

(

x

)

ofZn

=

(

Z

(1)

n

, . . . ,

Zn(d)

)

T

=

n1/2

(

X

¯

n

−

µ)

holds (see, e.g., [4, Theorem 2, p. 436]):

FZn

(

x

)

=



I

(

z

≤

x

)



1

+

n−1/2p1

(

z

)

+

n−1p2

(

z

)



φ

Σ

(

z

)

dz

+

O

(

n−3/2

),

(1)

where

φ

_Σ

(

z

)

is the density function of N

(

0

,

Σ

)

,pi

(

z

)

are odd/even polynomials inz for odd/eveni

=

1

,

2 (see, e.g.,

[15, pp. 162–167]), and I

(

A

)

is the indicator function of an event A. Let H

(

x

)

=

E



p1

(

Z

)

I

(γ

TZ

≤

x

)



and J

(

x

)

=

E



ZTΓZI

(γ

TZ

≤

x

)



. Furthermore, letf(i1···ik)

₌



∂

k

/∂µ

i1

· · ·

∂µ

ik



f

(µ)

fork

≥

2,Ξi

=



(

1

/

3

!

)

f(ijk)

:

j

,

k

=

1

, . . . ,

d



∈

Rd×d,Ξ

=

(Ξ

1

, . . . ,

Ξd

)

∈

Rd×d 2 ,Π

=

(Π

₁

, . . . ,

Π_d

)

∈

_Rd×d3,Πi

=

(Π

i1

, . . . ,

Πid

)

∈

Rd×d 2 ,Πij

=



(

1

/

4

!

)

f(ijkl)

:

k

,

l

=

1

, . . . ,

d



∈

_Rd×d_,K

₍

_x

₎

₌

_E



p2

(

Z

)

I

(

ZTΓZ

≤

x

)



,M

(

x

)

=

E



p1

(

Z

)

ZTΞ(Z

⊗

Id

)

ZI

(

ZTΓZ

≤

x

)



,N

(

x

)

=

E



ZTΠ(Z

⊗

I_d2

)(

Z

⊗

Id

)

ZI

(

ZTΓZ

≤

x

)



,P

(

x

)

=

(

1

/

2

)

E



(

ZT_Ξ(Z

_⊗

_I d

)

Z

)

2I

(

ZTΓZ

≤

x

)



, where

⊗

denotes the Kronecker product. The following lemma provides the Edgeworth and the Cornish–Fisher expansions ofF_θˆ_n

(

x

)

,F

−1 ˆ θn

(α)

,Gθˆn

(

x

)

, andG −1 ˆ θn

(α)

, respectively. Similar expansions have been obtained by Cheung et al. [10] for

θ

∈

Θ₀.

Lemma 1. Assume that f is five times continuously differentiable, E



∥

X

∥

5



<

∞

, andlim sup∥t∥→∞

|

Eexp

(

itTX

)

|

<

1.

(i) If

θ

∈

Θ

\

Θ₀, then F_θˆ_n

(

x

)

=

F

(

x

)

+

n−1/2



H

(

x

)

−

J′

(

x

)



+

O

(

n−1

)

and F_ˆ−1 θn

(α)

=

F−1

(α)

−

n−1/2



F′

(

F−1

(α))



−1



H

(

F−1

(α))

−

J′

(

F−1

(α))



+

O

(

n−1

)

; (ii) If

θ

∈

Θ₀, then G_θˆ_n

(

x

)

=

G

(

x

)

+

n−1



K

(

x

)

−

M′

₍

_x

₎

₋

_N′

₍

_x

₎

₊

_P′′

₍

_x

₎



+

O

(

n−3/2

₎

_{and G}−1 ˆ θn

(α)

=

G−1

_(α)

₋

n−1



G′

₍

_G−1

_(α))



−1



K

(

G−1

_(α))

₋

_M′

₍

_G−1

_(α))

₋

_N′

₍

_G−1

_(α))

₊

_P′′

₍

_G−1

_(α))



+

O

(

n−3/2

₎

_. LetZ∗ m

=

(

Z ∗ m( 1)

_{, . . . ,}

_Z∗ m( d)

₎

T

₌

_m1/2

₍

_X

¯

∗

m

− ¯

Xn

)

, then the Edgeworth expansion ofF

ˆ

Z∗

m

(

x

)

, the conditional distribution function ofZ_m∗givenX1:n, holds:

ˆ

F_Z∗ m

(

x

)

=



I

(

z

≤

x

)



1

+

m−1/2p

ˆ

1n

(

z

)

+

m−1p

ˆ

2n

(

z

)



φ

_Σˆ_n

(

z

)

dz

+

OP

(

m−3/2

),

(2)

where

φ

_Σˆ_n

(

z

)

is the conditional density function ofN

(

0

,

Σ

ˆ

n

)

givenX1:n,p

ˆ

in

(

z

)

are odd/even polynomials inzfor odd/even

i

=

1

,

2, the sample versions of p1

(

z

)

andp2

(

z

)

defined by(1). LetH

ˆ

n

(

x

)

=

E



ˆ

p1n

(

Z∗

)

I

(

γ

ˆ

nTZ ∗

_≤

x

)

|

X1:n



,

ˆ

Jn

(

x

)

=

E



Z∗TΓ

ˆ

nZ∗I

(

γ

ˆ

nTZ ∗

_≤

x

)

|

X1:n



, andQ

ˆ

n

(

x

)

=

E



ˆ

p1n

(

Z∗

)

n1/2

γ

ˆ

nTZ ∗ I

(

Z∗TΓ

ˆ

nZ∗

≤

x

)

|

X1:n



lemma provides the empirical Edgeworth and the empirical Cornish–Fisher expansions of the m-out-of-n bootstrap estimatorsF

ˆ

_θˆ∗ m

(

x

)

,

ˆ

F−_ˆ1 θ∗ m

(α)

,G

ˆ

_θˆ∗ m

(

x

)

, and

ˆ

G−_ˆ1 θ∗ m

(α)

, respectively.

Lemma 2. Suppose f is five times continuously differentiable, m

=

o

(

n

)

, m

→ ∞

, E



∥

X

∥

5



<

∞

, andlim sup∥t∥→∞

|

Eexp

(

itTX

)

|

<

1. (i) If

θ

∈

Θ

\

Θ₀, thenF

ˆ

_θˆ∗ m

(

x

)

=

ˆ

Fn

(

x

)

+

m−1/2



_ˆ

Hn

(

x

)

− ˆ

Jn′

(

x

)



+

OP

(

m−1

)

and F

ˆ

_θ−ˆ∗1 m

(α)

=

F

ˆ

−1 n

(α)

−

m −1/2



_ˆ

F′ n

(

F

ˆ

−1 n

(α))



−1



_ˆ

Hn

(

F

ˆ

n−1

(α))

− ˆ

J ′ n

(

F

ˆ

−1 n

(α))



+

OP

(

m−1

)

. (ii)If

θ

∈

Θ₀, then G

ˆ

_θˆ∗ m

(

x

)

=

ˆ

Gn

(

x

)

−

n−1/2Q

ˆ

n′

(

x

)

+

OP



m−1

₊

_mn−1



and G

ˆ

−_ˆ1 θ∗ m

(α)

=

G

ˆ

−1 n

(α)

+

n −1/2



_ˆ

G′ n

(

G

ˆ

−1 n

(α))



−1

_ˆ

Q′ n

(

G

ˆ

−1 n

(α))

+

OP



m−1

₊

_mn−1



.

Next theorem shows that the test inversion confidence regions are more accurate than them-out-of-nbootstrap test inversion confidence regions.

Theorem 3.Suppose f is five times continuously differentiable, m

=

o

(

n

)

, m

→ ∞

, E



∥

X

∥

5



<

∞

, andlim sup∥t∥→∞

|

Eexp

(

itT_X

₎

_|

_<

_1. (i) If

θ

∈

Θ

\

Θ₀, thenPr



θ

∈

In

(α)



=

α

+

O

(

n−1/2

₎

_andPr



θ

∈ ˆ

Jm

(α)



=

α

+

O

(

m−1/2

₎

_. (ii)If

θ

∈

Θ₀, thenPr



θ

∈

In

(α)



=

α

+

O

(

n−1

₎

_andPr



θ

∈ ˆ

Jm

(α)



=

α

+

O



m−1

₊

_mn−1



.

4. Oracle bootstrap estimation

The oracle bootstrap is a particular type of the empirical likelihood bootstrap [8] and of the intentionally-biased bootstrap [18] that adapts a parametric bootstrap proposed by Putter and van Zwet [22] to nonparametric setting. First, the empirical distribution is embedded in a parametric family of weighted empirical distributions of the sample and then Putter and van Zwet’s parametric bootstrap procedure is applied to this family. To construct the parametric family

{ ˆ

_Pυ

:

υ

∈

_Rd

}

associated with the smooth function model, we choose a vector of weights

(w

1

, . . . , w

n

)

to minimize

−

n−1



n

i=1log

(

n

w

i

)

,

the Kullback–Leibler divergence of

(w

1

, . . . , w

n

)

to the vector of uniform weights

(

n−1

, . . . ,

n−1

)

, subject to the constraints



n

i=1

w

iXi

=

υ

,



n

i=1

w

i

=

1, and

w

i

≥

0 for alli

=

1

, . . . ,

n. The solution of this constraint optimization problem coincides

with the empirical likelihood weights for testing the null hypothesisH0

:

µ

=

υ

[20]. If

υ

is not in the convex hull ofX1:n,

then we arbitrarily takeP

ˆ

υ

= ˆ

Pn, whereP

ˆ

nis the empirical distribution of the sample.

Let

µ

˜

_nbe a sequence ofn1/2_{-consistent estimators of}

_µ

_{such that Pr}



˜

µ

n

∈

S



→

1 whenever

µ

∈

S, i.e.,

µ

˜

_nhas the ‘‘oracle property’’ for the setS. A typical oracle bootstrap resampleX₁Ď_:n

=

X₁Ď

, . . . ,

XnĎ is a sequence ofnconditionally

i.i.d. random draws from_P

ˆ

_µ˜

n and

µ

˜

nplays the role of

µ

in the oracle bootstrap world. Let

θ

ˆ

Ď

n

=

f

(

X

¯

nĎ

)

and

θ

˜

n

=

f

(

µ

˜

n

)

be the oracle bootstrap versions of

θ

ˆ

nand

θ

, respectively, whereX

¯

nĎis the sample mean of the oracle bootstrap resample.

Then,

˜

L_θˆ_nĎ

(

x

)

=

Pr



_ˆ

θ

Ď

n

− ˜

θ

n

≤

x

|

X1:n



,F

˜

_θˆ_nĎ

(

x

)

= ˜

L_θˆ_nĎ

(

x

/

n1/2

)

, andG

˜

_θˆ_nĎ

(

x

)

= ˜

L_θˆ_nĎ

(

x

/

n

)

are the oracle bootstrap versions

ofL_θˆ_n

(

x

)

,F_θˆ_n

(

x

)

, andG_θˆ_n

(

x

)

, respectively. Thus, the

α

level oracle bootstrap percentile upper confidence interval for

θ

is

˜

In

(α)

=



−∞

,

θ

ˆ

_n

− ˜

L−_ˆ1 θĎ n

(

1

−

α)



. Similarly, the oracle bootstrap percentile lower, equal-tailed, and symmetric confidence intervals for

θ

are defined asI

˜

nL

(α)

=



_ˆ

θ

n

− ˜

L −1 ˆ θĎ n

(α),

∞



,I

˜

ETn

(α)

=



_ˆ

θ

n

− ˜

L −1 ˆ θĎ n

((

1

+

α)/

2

),

θ

ˆ

n

− ˜

L −1 ˆ θĎ n

((

1

−

α)/

2

)



, and

˜

InS

(α)

=



_ˆ

θ

n

− ˜

L ′₋₁ ˆ θĎ n

(α),

ˆ

θ

n

+ ˜

L ′₋₁ ˆ θĎ n

(α)



, respectively, where

˜

L′_ˆ θĎ n

(

x

)

=

Pr



| ˆ

θ

_nĎ

− ˜

θ

_n

| ≤

x

|

X1:n



.

Note that there are many estimators

µ

˜

_n which have the oracle property for the setS. For example, ifS

= {

µ

₀

}

is a singleton, let

µ

˜

_n

=

µ

₀

+

(

X

¯

n

−

µ

0

)

◦

I

(

|∇

f

(

X

¯

n

)

| ≥

an

)

, where ‘‘

◦

’’ is the component-wise vector to vector multiplication

(Hadamard product),I

(

|∇

f

(

X

¯

n

)

| ≥

an

)

=



I

(

|∇

f

(

X

¯

n

)

(j)

| ≥

an

)

:

j

=

1

, . . . ,

d



,an

→

0 andn1/2an

→ ∞

, e.g.,an

=

n−1/2_log

₍

_log

₍

_n

₎₎

_{. Another option is to use a LASSO-type estimator which has the oracle property for the setS, such as an} adaptive LASSO-type estimator (see, e.g., [28]):

˜

µ

n

=

argmin b



_n



i=1

(

Xi

−

b

)

T

(

Xi

−

b

)

+

λ

n d



j=1

| ¯

X_n(j)

−

µ

(₀j)

|

−1

|

b(j)

−

µ

(₀j)

|



,

wheren−1/2

λ

n

→

0 and

λ

n

→ ∞

. If the non-empty setSis not a singleton, then for all

ν

∈

S, let

µ

˜

n,ν

=

ν

+

(

X

¯

n

−

ν)

◦

I



|∇

f

(

X

¯

n

)

| ≥

an



. In the case of the adaptive LASSO estimator, let

˜

µ

n,ν

=

argmin b



_n



i=1

(

Xi

−

b

)

T

(

Xi

−

b

)

+

λ

n d



j=1

| ¯

X_n(j)

−

ν

(j)

|

−1

|

b(j)

−

ν

(j)

|



.

Then, set

µ

˜

_n

= ˜

µ

_n,νˆn, where

ν

ˆ

_n

=

argmin_ν∈S

∥ ˜

µ

n,ν

− ¯

Xn

∥

2and

∥ · ∥

is the Euclidean norm. Next theorem shows that the

Theorem 4. Assume that f is two times continuously differentiable and E

(

∥

X

∥

2

) <

∞

. Thenlimn→∞Pr

(θ

∈ ˜

In

(α))

=

α

for all

θ

∈

Θ. LetAn

= ∩

dj=1



|∇

f

(

X

¯

n

)

(j)

| ≥

n−r



, wherer

∈

(

0

,

1

/

2

)

. Next lemma shows that ifan

=

n−r and

∇

f

(µ)

(j)

̸=

0 for

allj

=

1

, . . . ,

d, then there exists a constantc

>

0 such that Pr

(

An

)

=

1

−

O



exp

(

−

cn1−2r

₎



. Since

µ

˜

_n

= ¯

XnonAn, it

follows that the oracle bootstrap and the standard bootstrap are equivalent except on an event of probability exponentially small. If

θ

∈

Θ₀, next lemma shows that there exists a constantc

>

0 such that Pr

(

Bn

)

=

1

−

O



exp

(

−

cn1−2r

₎



, where Bn

= {∥∇

f

(

X

¯

n

)

∥

∞

≤

n−r

}

and

∥ · ∥

∞ is the infinity norm. Therefore,

∇

f

(

µ

˜

n

)

=

0 except on an event of probability

exponentially small.

Lemma 3. Suppose f is twice continuously differentiable, E



∥

X

∥

2



<

∞

, andlogE



exp

(

tT_X

₎



<

∞

in a neighborhood of

0

∈

_Rd_.

1. If

∇

f

(µ)

(j)

_̸=

_{0for all j}

₌

₁

_{, . . . ,}

_{d, there exists c}

_>

_{0such thatPr}

₍

_A

n

)

=

1

−

O



exp

(

−

cn1−2r

₎



.

2. If

θ

∈

Θ₀, there exists a constant c

>

0such thatPr

(B

n

)

=

1

−

O



exp

(

−

cn1−2r

)



.

Suppose

θ

∈

Θ

\

Θ₀. Similarly to them-out-of-nbootstrap, under the conditions ofLemma 4, the following empirical Edgeworth expansion ofF

˜

_ZĎ n

(

x

)

holds:

˜

F_ZĎ n

(

x

)

=



I

(

z

≤

x

)



1

+

n−1/2p

˜

1n

(

z

)

+

n−1p

˜

2n

(

z

)



φ

_Σ˜_n

(

z

)

dz

+

OP

(

n−3/2

),

(3)

whereZnĎ

=

n1/2

(

X

¯

nĎ

− ˜

µ

n

)

, and givenX1:n,

φ

_Σ˜_n

(

z

)

is the conditional density ofZĎ

∼

N

(

0

,

Σ

˜

n

)

,Σ

˜

n

=



ni=1

w

˜

i



Xi

− ˜

µ

n

)(

Xi

−

˜

µ

n

)

T,

w

˜

i,i

=

1

, . . . ,

n, are the oracle bootstrap weights corresponding toP

ˆ

µ˜n,p

˜

in

(

z

)

are odd/even polynomials inzfor odd/eveni

=

1

,

2, the oracle bootstrap versions ofp1

(

z

)

andp2

(

z

)

defined by(1). Let

γ

˜

n

= ∇

f

(

µ

˜

n

)

,Γ

˜

n

=

(

1

/

2

)

∇

2f

(

µ

˜

n

)

∈

Rd×d,Ξ

˜

n

=

(

Ξ

˜

1n

, . . . ,

Ξ

˜

dn

)

∈

Rd×d 2 ,Ξ

˜

in

=



(

1

/

3

!

)

f

˜

n(ijk)

:

j

,

k

=

1

, . . . ,

d



∈

_Rd×d,Π

˜

n

=

(

Π

˜

1n

, . . . ,

Π

˜

dn

)

∈

Rd×d 3 ,

˜

Πin

=

(

Π

˜

i1n

, . . . ,

Π

˜

idn

)

∈

Rd×d 2 ,Π

˜

ijn

=



(

1

/

4

!

)

f

˜

n(ijkl)

:

k

,

l

=

1

, . . . ,

d



∈

_Rd×d_{, andf}

˜

(i1···ik) n

=

f(i1···ik)

(

µ

˜

n

)

, wherek

≥

2. Let furtherF

˜

n

(

x

)

=

Pr

(

γ

˜

nTZĎ

≤

x

|

X1:n

)

,H

˜

n

(

x

)

=

E

(

p

˜

1n

(

ZĎ

)

I

(

γ

˜

nTZĎ

≤

x

)

|

X1:n

)

,

˜

Jn

(

x

)

=

E

(

ZĎTΓ

˜

nZĎI

(

γ

˜

nTZĎ

≤

x

)

|

X1:n

)

,G

˜

n

(

x

)

=

Pr



ZĎT_Γ

˜

_nZĎ

_≤

_x

_|

_X 1:n



,K

˜

n

(

x

)

=

E



˜

p2n

(

ZĎ

)

I

(

ZĎTΓ

˜

nZĎ

≤

x

)

|

X1:n



,M

˜

n

(

x

)

=

E



˜

p1n

(

ZĎ

)

ZĎTΞ

˜

n

(

ZĎ

⊗

Id

)

ZĎI

(

ZĎTΓ

˜

nZĎ

≤

x

)

|

X1:n



,

˜

Nn

(

x

)

=

E



ZĎT_Π

˜

_n

₍

_Z

˜

Ď

_⊗

_I d2

)(

ZĎ

⊗

Id

)

ZĎI

(

ZĎTΓ

˜

nZĎ

≤

x

)

|

X1:n



, andP

˜

n

(

x

)

=

(

1

/

2

)

E



(

ZĎT_Ξ

˜

_n

₍

_ZĎ

_⊗

_I d

)

ZĎ

)

2I

(

ZĎTΓ

˜

nZĎ

≤

x

)

|

X1:n



. Next lemma gives the empirical Edgeworth and the empirical Cornish–Fisher expansions ofF

ˆ

_θˆĎ

n

(

x

)

,

ˆ

F−1 ˆ θnĎ

(α)

,G

ˆ

_θˆĎ n

(

x

)

, and

ˆ

G−_ˆ1 θĎ n

(α)

, respectively.

Lemma 4. Assume that f is five times continuously differentiable, E



∥

X

∥

5



<

∞

, andlim sup∥t∥→∞

|

Eexp

(

itTX

)

|

<

1.

(i) If

θ

∈

Θ

\

Θ₀, then F

˜

_θˆ_nĎ

(

x

)

=

F

˜

n

(

x

)

+

n−1/2



_˜

Hn

(

x

)

− ˜

Jn′

(

x

)



+

OP

(

n−1

)

and F

˜

_θˆ−Ď1 n

(α)

=

F

˜

_n−1

(α)

−

n−1/2



_˜

F′ n

(

F

˜

−1 n

(α))



−1



_˜

Hn

(

F

˜

n−1

(α))

− ˜

J ′ n

(

F

˜

−1 n

(α))



+

OP

(

n−1

)

; (ii) If

θ

∈

Θ₀, then G

˜

_θˆ_nĎ

(

x

)

= ˜

Gn

(

x

)

+

n−1



_˜

Kn

(

x

)

− ˜

Mn′

(

x

)

− ˜

N ′ n

(

x

)

+ ˜

P ′′ n

(

x

)



+

OP

(

n−3/2

)

andG

˜

−_θˆĎ1 n

(α)

= ˜

G−_n1

(α)

−

n−1



_˜

G′ n

(

G

˜

−1 n

(α))



−1



_˜

Kn

(

G

˜

−n1

(α))

− ˜

M ′ n

(

G

˜

−1 n

(α))

− ˜

N ′ n

(

G

˜

−1 n

(α))

+ ˜

P ′′ n

(

G

˜

−1 n

(α))



+

OP

(

n−3/2

)

.

Bootstrap iteration is known to further reduce the size distortion of a bootstrap test, to further reduce the bias of an estimator, and to further improve the coverage probability of a bootstrap confidence interval [3,17,10]. Bootstrap iteration is usually used to estimate a parameter of the distribution of a statistical quantity defined in terms of a bootstrap estimator. For example, to obtain a (coverage calibrated) iterated bootstrap confidence interval, one first estimates the coverage probability of the bootstrap confidence interval using the iterated bootstrap, and then, the nominal level of the bootstrap confidence interval is re-adjusted based on this estimate. Specifically, letX₁Ď_:n

=

X₁Ď

, . . . ,

XnĎbe an oracle bootstrap resample, and let

ˆ

θ

n

− ˜

L −1 ˆ θĎ n

(

1

−

α)

be the

α

level oracle bootstrap percentile upper confidence limit, and let

π

n

(α)

=

Pr



θ

≤ ˆ

θ

_n

− ˜

L−_ˆ1 θĎ

n

(

1

−

α)



be the coverage probability of the confidence interval. A typical iterated oracle bootstrap corresponding toX₁Ď_:nis a collection

X₁ĎĎ_:n

=

X₁ĎĎ

, . . . ,

XnĎĎofnconditionally i.i.d. draws fromP

ˆ

µ˜Ďn. Let

ˆ

θ

ĎĎ

n be the iterated oracle bootstrap version of

θ

ˆ

n. The iterated

oracle bootstrap estimate of

π

n

(α)

is

π

˜

n

(α)

=

Pr



_˜

θ

n

≤ ˆ

θ

nĎ

− ˜

LĎ −1 ˆ θnĎĎ

(

1

−

α)

|

X1:n



, where

˜

LĎ ˆ θnĎĎ

(

x

)

=

Pr



_ˆ

θ

ĎĎ n

− ˜

θ

nĎ

≤

x

|

X1Ď:n



. Thus, the

α

level coverage calibrated confidence interval is_I

˜

Ď_n

_(α)

=



−∞

,

_θ

ˆ

_n

− ˜

L−1 ˆ θnĎ

(

1

− ˜

π

−1 n

(α))



. Similarly, the iterated standard bootstrap and them-out-of-nbootstrap confidence intervals areI

ˆ

∗n

(α)

=



−∞

,

θ

ˆ

_n

− ˆ

L−_ˆ1 θ∗ n

(

1

− ˆ

π

_n−1

(α))



and

ˆ

I∗ m

(α)

=



−∞

,

θ

ˆ

_n

−

(

m

/

n

)

1/2

ˆ

_L−1 ˆ θ∗ m

(

1

− ˆ

π

−1 m

(α))



, respectively, where

π

ˆ

_n

(α)

=

Pr



θ

ˆ

n

≤

θ

ˆ

n∗

− ˆ

L ∗−1 ˆ θ∗∗ n

(

1

−

α)

|

X1:n



,

ˆ

π

m

(α)

=

Pr



_ˆ

θ

n

≤ ˆ

θ

m∗

−

(

l

/

m

)

1/2

ˆ

L ∗−1 ˆ θ∗∗ l

(

1

−

α)

|

X1:n



Next theorem shows that iterated oracle bootstrap percentile confidence intervals are more accurate than their non-iterated versions by an order of magnitude for large sample sizes.

Theorem 5.Assume that f is five times continuously differentiable, E



∥

X

∥

5



<

∞

, andlim sup∥t∥→∞

|

exp

(

itTX

)

|

<

1.

(i) If

θ

∈

Θ

\

Θ₀, thenPr



θ

∈ ˜

In

(α)



=

α

+

O

(

n−1/2

₎

_andPr



θ

∈ ˜

IĎn

(α)



=

α

+

O

(

n−1

₎

_. (ii)If

θ

∈

Θ₀, thenPr



θ

∈ ˜

In

(α)



=

α

+

O

(

n−1

₎

_andPr



θ

∈ ˜

IĎn

(α)



=

α

+

O

(

n−3/2

₎

_.

In practice, the bootstrap confidence bounds do not have closed form expressions, and thus, we usually approximate them by Monte-Carlo methods. Specifically, letX₁Ď_:1_n

, . . . ,

X₁Ď_:B_nbeBoracle bootstrap resamples, and let

ℓ

˜

n,B

(

1

−

α)

denote

the

(

1

−

α)

Bth order statistic of

(

θ

ˆ

nĎb

− ˜

θ

n

)

, for b

=

1

, . . . ,

B, where

θ

ˆ

nĎb is the bootstrap version of

θ

ˆ

n onX1Ď:bn. For

eachb

=

1

, . . . ,

B, letX₁ĎĎ:nb1

, . . . ,

XĎĎ bC

1:n beC iterated oracle bootstrap re-resamples and let

ℓ

˜

nĎ,b,C

(

1

−

α)

denote the

(

1

−

α)

Cth order statistic of

(

θ

ˆ

nĎĎbc

− ˜

θ

nĎb

)

, forc

=

1

, . . . ,

C, where

θ

ˆ

nĎĎbc is the bootstrap version of

θ

ˆ

nonX1ĎĎ:nbc. Since

˜

LĎ_ˆ−1 θĎĎ n

(

1

−

α)

≤

(

θ

ˆ

_nĎ

− ˜

θ

_n

)

⇐⇒

(

1

−

α)

≤ ˜

LĎ_ˆ θĎĎ n

(

ˆ

θ

Ď n

− ˜

θ

n

)

,

π

˜

n

(α)

=

1

−

Pr



_˜

LĎ_ˆ θĎĎ n

(

ˆ

θ

Ď n

− ˜

θ

n

) < (

1

−

α)

|

X1:n



. The Monte-Carlo approximation of

π

˜

_n

(α)

is (see also, e.g., [11, p. 224])

˜

π

BC n

(α)

=

1

−

B −1 B



b=1 I



C−1 C



c=1 I

(

θ

ˆ

_nĎĎbc

− ˜

θ

_nĎb

≤ ˆ

θ

_nĎb

− ˜

θ

_n

) < (

1

−

α)



.

Due to the parametric structure of the oracle bootstrap, we can use a bootstrap recycling algorithm to approximate

π

˜

_n

(α)

. Similar algorithms have been proposed by Newton and Geyer [19] for the parametric bootstrap. To this end, we generateC

independent oracle bootstrap resamplesX₁Ď_:1_n

, . . . ,

X₁Ď_:C_n, and use the importance resampling identity to reweight the terms of the Monte-Carlo approximations of the iterated bootstrap expectations as follows:

Pr



ˆ

θ

ĎĎ n

− ˜

θ

Ď b n

≤ ˆ

θ

Ď b n

− ˜

θ

n

|

X1Ď:bn



≃

C−1 C



c=1 I



ˆ

θ

Ďc n

− ˜

θ

Ď b n

≤ ˆ

θ

Ď b n

− ˜

θ

n



dP

ˆ

_µ˜Ďb n

(

X Ďc 1:n

)

d_P

ˆ

_µ˜ n

(

X Ďc 1:n

)

=

C−1 C



c=1 I



ˆ

θ

Ďc n

− ˜

θ

nĎb

≤ ˆ

θ

nĎb

− ˜

θ

n



˜

w

bc

,

where

˜

w

bc

=

n



i=1



˜

w

Ďb i

˜

w

i



m(_ic)

,

and m(_ic)

=

#

{

j

:

X_cjĎ

=

Xi

}

,

#Adenotes the number of elements of a setA,

w

˜

_iĎband

w

˜

_iare the oracle bootstrap weights corresponding to_P

ˆ

˜ µĎb

n and

ˆ

Pµ˜n, respectively. Note thatC−1



C

c=1

w

˜

bc

Pr

−

→

1 asC

→ ∞

. For finiteC, (C−1

_w

bc

:

c

=

1

, . . . ,

C) are not valid weights since some

bootstrap recycling estimates of

π

˜

_n

(α)

may be greater than 1. A remedy to this problem is to use the following approximation instead: Pr



ˆ

θ

ĎĎ n

− ˜

θ

nĎb

≤ ˆ

θ

nĎb

− ˜

θ

n

|

X1Ď:bn



≃

C



c=1 I



ˆ

θ

Ďc n

− ˜

θ

nĎb

≤ ˆ

θ

nĎb

− ˜

θ

n



¯

w

bc

,

where

w

¯

_bc

=

w

˜

_bc

/



C

c=1

w

˜

bc. In fact, this version of the oracle bootstrap recycling algorithm is implemented in the

simulation study carried out in Section6. 5. Local asymptotics

In this section, we study the large sample behavior of the bootstrap percentile confidence intervals when the mean parameter vector is in a local neighborhood of the set of points of inconsistency of the standard bootstrap. Specifically, assume thatX1:nn

=

X1n

, . . . ,

Xnnis a sample ofni.i.d. observations from a distributionPnonRd, such thatE

(

X1n

)

=

µ

n

and var

(

X1n

)

=

Σ, where thelocal parameter

µ

nis defined as

µ

n

=

µ

+

hn−δwith

δ >

0,

µ

∈

S, andh

∈

Rd

\ {

0

}

is a nonzero constant vector. The behavior of the bootstrap confidence intervals depends on how far is the local parameter from S; specifically, we consider three cases for

δ

, namely

δ <

1

/

2,

δ

=

1

/

2, and

δ >

1

/

2.

Local parameters ‘‘close’’ toS. Consider first the case when

δ >

1

/

2, that is, when the local parameter is ‘‘close’’ toS. By a Taylor expansion off

(

X

¯

n

)

about

µ

n, since

∇

f

(µ

n

)

=

O

(

n−δ

)

, it follows thatn

(

θ

ˆ

n

−

θ

n

)

=

ZnTΓZn

+

oP

(

1

)

and

n1/2

∇

f

(

X

¯

n

)

=

2ΓZn

+

oP

(

1

)

, where

θ

n

=

f

(µ

n

)

andZn

=

n1/2

(

X

¯

n

−

µ

n

)

. Similarly to the proof ofTheorem 1, we obtain

lim n→∞Pr



θ

n

∈ ˆ

In

(α)



=

1

−

Pr



ZTΓZ

−

F_N−₍1_0,Σ)

(

1

−

α

;

Z

) <

0



,