ABC Methods for Bayesian Model Choice

(1)

Christian P. Robert

Universit´e Paris-Dauphine, IuF, & CREST

http://www.ceremade.dauphine.fr/~xian

(2)

ABC Methods for Bayesian Model Choice

ABC for model choice

Model choice

Gibbs random fields

(3)

ABC Methods for Bayesian Model Choice Model choice

Bayesian model choice

Principle

Several modelsM1, M2, . . . are considered simultaneously for

datasetyand model indexMcentral to inference.

Use of a priorπ(M=m), plus a prior distribution on the

parameter conditional on the valuem of the model index,πm(θm)

Goal is to derive the posterior distribution ofM, a challenging computational target when models are complex.

(4)

Generic ABC for model choice

Algorithm 1Likelihood-free model choice sampler (ABC-MC)

for t= 1 to T do repeat

Generatemfrom the prior π(M=m)

Generateθm from the priorπm(θm)

Generatezfrom the modelfm(z|θm)

until ρ{η(z), η(y)}<

Set m(t)₌_m _and_θ(t)₌_θ m

(5)

ABC estimates

Posterior probabilityπ(M=m|y) approximated by the frequency

of acceptances from modelm

1 T T X t=1 Im(t)₌_m.

Early issues with implementation:

I should tolerances be the same for all models?

I should summary statistics vary across models? incl. their dimension?

I should the distance measure ρvary across models?

Extension to a weighted polychotomous logistic regression estimate ofπ(M=m|y), with non-parametric kernel weights

(6)

ABC estimates

Posterior probabilityπ(M=m|y) approximated by the frequency

of acceptances from modelm

1 T T X t=1 Im(t)₌_m.

Early issues with implementation:

I then needs to become part of the model

I Varying statistics incompatible with Bayesian model choice

proper

(7)

The great ABC controversy

On-going controvery in phylogeographic genetics about the validity of using ABC for testing

Against: Templeton, 2008, 2009, 2010a, 2010b, 2010c, &tc

argues that nested hypotheses cannot have higher probabilities than nesting hypotheses (!)

(8)

The great ABC controversy

On-going controvery in phylogeographic genetics about the validity of using ABC for testing

Against: Templeton, 2008, 2009, 2010a, 2010b, 2010c, &tc

argues that nested hypotheses cannot have higher probabilities than nesting hypotheses (!)

Replies: Fagundes et al., 2008, Beaumont et al., 2010, Berger et

al., 2010, Csill`ery et al., 2010

point out that the criticisms are addressed at [Bayesian]

model-based inference and have nothing to do with ABC...

(9)

ABC Methods for Bayesian Model Choice Gibbs random fields

Gibbs random fields

Gibbs distribution

The rvy= (y1, . . . , yn) is a Gibbs random fieldassociated with

the graphGif f(y) = 1 Zexp ( −X c∈_C Vc(yc) ) ,

whereZis the normalising constant, C is the set of cliques of G

andVc is any function also called potential

U(y) =P

c∈_C Vc(yc) is the energy function

c

(10)

Gibbs random fields

Gibbs distribution

The rvy= (y1, . . . , yn) is a Gibbs random fieldassociated with

the graphGif f(y) = 1 Zexp ( −X c∈_C Vc(yc) ) ,

whereZis the normalising constant, C is the set of cliques of G

andVc is any function also called potential

U(y) =P

(11)

Potts model

Potts model P c∈CVc(y)is of the form X c∈C Vc(y) =θS(y) =θ X l∼i δyl=yi

wherel∼idenotes a neighbourhood structure

In most realistic settings, summation

Z_θ = X x∈X

exp{θTS(x)}

involves too many terms to be manageable and numerical approximations cannot always be trusted

(12)

Potts model

Potts model P c∈CVc(y)is of the form X c∈C Vc(y) =θS(y) =θ X l∼i δyl=yi

wherel∼idenotes a neighbourhood structure In most realistic settings, summation

Z_θ = X x∈X

(13)

Bayesian Model Choice

Comparing a model with energyS0 taking values inRp0 versus a

model with energyS1 taking values inRp1 _{can be done through}

theBayes factorcorresponding to the priors π0 andπ1 on each

parameter space B_m₀_/m₁(x) = R exp{θT₀S0(x)}/Z_θ₀_,₀π0(dθ0) R exp{θT₁S1(x)}/Z_θ₁_,₁π1(dθ1)

(14)

Neighbourhood relations

Setup

Choice to be made betweenM neighbourhood relations

i∼mi0 (0≤m≤M −1) with Sm(x) = X im∼i0 I{xi=xi0}

(15)

Model index

Computational target: P(M=m|x)∝ Z Θm fm(x|θm)πm(θm)dθmπ(M=m)

IfS(x) sufficient statisticfor the joint parameters

(M, θ0, . . . , θM−1),

(16)

Model index

Computational target: P(M=m|x)∝ Z Θm fm(x|θm)πm(θm)dθmπ(M=m)

IfS(x) sufficient statisticfor the joint parameters

(M, θ0, . . . , θM−1),

(17)

Sufficient statistics in Gibbs random fields

Each modelm has its own sufficient statisticSm(·) and

S(·) = (S0(·), . . . , SM−1(·))is also (model-)sufficient.

Explanation: For Gibbs random fields,

x|M=m∼fm(x|θm) = fm1(x|S(x))fm2(S(x)|θm) = 1 n(S(x))f 2 m(S(x)|θm) where n(S(x)) =]{˜x∈ X :S(˜x) =S(x)} c

(18)

Sufficient statistics in Gibbs random fields

(19)

Sufficient statistics in Gibbs random fields

(20)

ABC model choice Algorithm

ABC-MC

I Generatem∗ from the prior π(M=m).

I Generateθ∗_m∗ from the priorπ_m∗(·).

I Generatex∗ from the modelfm∗(·|θ∗_m∗).

I Compute the distanceρ(S(x0), S(x∗)). I Accept (θ∗_m∗, m∗) ifρ(S(x0), S(x∗))< .

(21)

Toy example

iid Bernoulli model versus two-state first-order Markov chain, i.e.

f0(x|θ0) = exp θ0 n X i=1 I{xi=1} ! {1 + exp(θ0)}n, versus f1(x|θ1) = 1 2exp θ1 n X i=2 I{xi=xi−1} ! {1 + exp(θ1)}n−1,

with priorsθ0∼ U(−5,5)andθ1 ∼ U(0,6)(inspired by “phase

(22)

Toy example (2)

− 40 − 20 0 10 −5 0 5 BF 01 BF^01 − 40 − 20 0 10 −10 −5 0 5 10 BF 01 BF^01

(23)

ABC Methods for Bayesian Model Choice Generic ABC model choice

Back to sufficiency

‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’

[Scott Sisson, Jan. 31, 2011, ’Og]

Ifη1(x) sufficient statistic for modelm= 1 and parameterθ1 and

η2(x) sufficient statistic for modelm= 2 and parameterθ2,

(η1(x), η2(x))is not always sufficient for (m, θm)

c

(24)

Back to sufficiency

c

(25)

Back to sufficiency

c

(26)

Limiting behaviour of

B

12

(

T

→ ∞)

ABC approximation d B12(y) = PT t=1Imt₌₁_I_ρ_{_η₍_zt₎_,η₍_y_)}≤ PT t=1Imt₌₂_I_ρ_{_η₍_zt₎_,η₍_y_)}≤ ,

where the(mt_{, z}t₎_{’s are simulated from the (joint) prior}

AsT go to infinity, limit B₁₂ (y) = R Iρ{η(z),η(y)}≤π1(θ1)f1(z|θ1)dzdθ1 R Iρ{η(z),η(y)}≤π2(θ2)f2(z|θ2)dzdθ2 = R Iρ{η,η(y)}≤π1(θ1)f1η(η|θ1)dηdθ1 R Iρ{η,η(y)}≤π2(θ2)f2η(η|θ2)dηdθ2 ,

(27)

Limiting behaviour of

B

12

(

T

→ ∞)

ABC approximation d B12(y) = PT t=1Imt₌₁_I_ρ_{_η₍_zt₎_,η₍_y_)}≤ PT t=1Imt₌₂_I_ρ_{_η₍_zt₎_,η₍_y_)}≤ ,

where the(mt_{, z}t₎_{’s are simulated from the (joint) prior}

AsT go to infinity, limit B₁₂ (y) = R Iρ{η(z),η(y)}≤π1(θ1)f1(z|θ1)dzdθ1 R Iρ{η(z),η(y)}≤π2(θ2)f2(z|θ2)dzdθ2 = R Iρ{η,η(y)}≤π1(θ1)f1η(η|θ1)dηdθ1 R Iρ{η,η(y)}≤π2(θ2)f2η(η|θ2)dηdθ2 ,

(28)

B

12

(

→

0

)

Whengoes to zero,

B₁₂η (y) =

R

π1(θ1)f1η(η(y)|θ1)dθ1

R

π2(θ2)f₂η(η(y)|θ2)dθ2

(29)

Limiting behaviour of

B

12

(

→

0

)

Whengoes to zero,

B₁₂η (y) =

R

π1(θ1)f1η(η(y)|θ1)dθ1

R

π2(θ2)f₂η(η(y)|θ2)dθ2

(30)

B

12

(under sufficiency)

Ifη(y) sufficient statistic in both models,

[Didelot, Everitt, Johansen & Lawson, 2011]

c

(31)

B

12

(under sufficiency)

Ifη(y) sufficient statistic in both models,

[Didelot, Everitt, Johansen & Lawson, 2011] c

(32)

Poisson/geometric example

Sample

x= (x1, . . . , xn)

from either a PoissonP(λ) or from a geometricG(p)

Sum S= n X i=1 xi =η(x)

sufficient statistic for either modelbut not simultaneously

Discrepancy ratio g1(x) = S!n −S_/Q ixi! −1

(33)

Poisson/geometric discrepancy

Range ofB12(x) versus B12η (x): The values produced have

(34)

Formal recovery

Creating an encompassing exponential family

f(x|θ1, θ2, α1, α2)∝exp{θT1η1(x) +θT1η1(x) +α1t1(x) +α2t2(x)}

leads to a sufficient statistic(η1(x), η2(x), t1(x), t2(x))

(35)

Formal recovery

In the Poisson/geometric case, ifQ

ixi!is added toS, no

(36)

Formal recovery

Only applies in genuine sufficiency settings...

c

(37)

The Pitman–Koopman lemma

Efficient sufficiency is not such a common occurrence:

Lemma

A necessary and sufficient condition for the existence of a sufficient statistic with fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family.

[Pitman, 1933; Koopman, 1933]

(38)

The Pitman–Koopman lemma

Efficient sufficiency is not such a common occurrence:

Lemma

A necessary and sufficient condition for the existence of a sufficient statistic with fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family.

[Pitman, 1933; Koopman, 1933]

(39)

Meaning of the ABC-Bayes factor

‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’

In the Poisson/geometric case, ifE[yi] =θ0 >0,

lim n→∞B η 12(y) = (θ0+ 1)2 θ0 e−θ0

(40)

Meaning of the ABC-Bayes factor

‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’

In the Poisson/geometric case, ifE[yi] =θ0 >0,

lim n→∞B η 12(y) = (θ0+ 1)2 θ e −θ0

(41)

MA example

Consider the MA(q) model

xt=t+ q

X

i=1

ϑit−i

Simple prior: uniform prior over the identifiability zone, e.g. triangle for MA(2)

(42)

MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle 2. generating an iid sequence(t)−q<t≤T 3. producing a simulated series (x0_t)1≤t≤T

Distance: basic distance between the series

ρ((x0_t)1≤t≤T,(xt)1≤t≤T) = T

X

t=1

(xt−x0t)2

or between summary statistics like the firstq autocorrelations

τj = T

X

t=j+1

(43)

MA example (2)

ABC algorithm thus made of

1. picking a new value (ϑ1, ϑ2) in the triangle 2. generating an iid sequence(t)−q<t≤T 3. producing a simulated series (x0_t)1≤t≤T

Distance: basic distance between the series

ρ((x0_t)1≤t≤T,(xt)1≤t≤T) = T

X

t=1

(xt−x0t)2

or between summary statistics like the firstq autocorrelations

τj = T

X

(44)

Comparison of distance impact

(45)

Comparison of distance impact

0.0 0.2 0.4 0.6 0.8 0 1 2 3 4 θ1 −2.0 −1.0 0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 θ2

Evaluation of the tolerance on the ABC sample against both distances (= 100%,10%,1%,0.1%) for an MA(2) model

(46)

Comparison of distance impact

0.0 0.2 0.4 0.6 0.8 0 1 2 3 4 θ1 −2.0 −1.0 0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 θ2

(47)

MA

(

q

)

divergence

1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0

Evolution [against] of ABC Bayes factor, in terms of frequencies of visits to models MA(1)(left) and MA(2)(right) whenequal to

10,1, .1, .01%quantiles on insufficient autocovariance distances. Sample of50points from a MA(2)withθ1= 0.6,θ2= 0.2. True Bayes factor

(48)

MA

(

q

)

divergence

1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0 1 2 0.0 0.2 0.4 0.6 0.8 1.0

Evolution [against] of ABC Bayes factor, in terms of frequencies of visits to models MA(1)(left) and MA(2)(right) whenequal to

(49)

Further comments

‘There should be the possibility that for the same model, but different (non-minimal) [summary]statistics (so different η’s: η1 andη1∗) the ratio of evidences may no

longer be equal to one.’

[Michael Stumpf, Jan. 28, 2011, ’Og]

Using different summary statistics [on different models] may indicate the loss of information brought by each set but agreement does not lead to trustworthy approximations.

(50)

A population genetics evaluation

Population genetics example with

I 3 populations

I 2 scenari

I 15 individuals I 5 loci

I single mutation parameter

I 24 summary statistics

I 2 million ABC proposal

(51)

A population genetics evaluation

Population genetics example with

I 3 populations

I 2 scenari

I 15 individuals I 5 loci

I single mutation parameter

I 24 summary statistics

I 2 million ABC proposal

(52)

Stability of importance sampling

● 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 ● 0.2 0.4 0.6 0.8 1.0 ● 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

(53)

Comparison with ABC

Use of 24 summary statistics and DIY-ABC logistic correction

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

ABC direct and logistic

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

(54)

Comparison with ABC

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● −4 −2 0 2 4 6 ABC direct

(55)

Comparison with ABC

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● −4 −2 0 2 4 6 −4 −2 0 2 4 6

ABC direct and logistic

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

(56)

Comparison with ABC

●● ●● ●●● ● ●● ●● ● ●●● ●● ● ● ● ●● ●● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●●●● ● ● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● −1 0 1 2 log−r atio ● ● ● ●●●● ● ●●●● ● ● ●●●● ● ● ● ● ● ● ● ●●● ● ● ●●●●● ● ●●●●●●●●● ● ● ●● ● ● ● ● ●●●● ● ● ●●●● ● ● ● ● ●● ● ● ● ●● ● ●●●●●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●●●● ● ●●●● ● ● ●●●● ● ● ● ● ● ● ● ●●● ● ● ●●●●● ● ●●●●●●●●● ● ● ●● ● ● ● ● ●●●● ● ● ●●●● ● ● ● ● ●● ● ● ● ●● ● ●●●●●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ●● ●● ●●● ● ●● ●● ● ●●● ●● ● ● ● ●● ●● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●●●● ● ● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●

(57)

A second population genetics experiment

I three populations, two divergent 100 gen. ago

I two scenarios [third pop. recent admixture between first two pop. / diverging from pop. 1 5 gen. ago]

I In scenario 1, admixture rate0.7 from pop. 1

I 100 datasets with 100 diploid individuals per population, 50 independent microsatellite loci.

I Effective population size of 1000 and mutation rates of

0.0005.

I 6 parameters: admixture rate (U[0.1,0.9]), three effective population sizes (U[200,2000]), the time of admixture/second divergence (U[1,10]) and time of first divergence (U[50,500]).

(58)

Results

IS algorithm performed with 100 coalescent trees per particle and 50,000 particles [12 calendar days using 376 processors]

Using ten times as many loci and seven times as many individuals degrades the confidence in the importance sampling approximation because of an increased variability in the likelihood.

● ● ● ● ●●●● ● ● ● ●● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ● ● ● ● ● ●●●●● ● ● ●●● ● ● ●●●●● ● ● ●●●●●●●● ● ● ● ● ● ● ● ● ●●●● ● ● ● ● ● ● ●●●●●●●●●●●●●●●●●● ● ● ● 0.4 0.6 0.8 1.0 Impor

tance Sampling estimates of

P(M=1|y) with 1,000 par

(59)

Results

Blurs potential divergence between ABC and genuine posterior probabilities because both are overwhelmingly close to one, due to the high information content of the data.

● ● ● ● ●●●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●●●●● ● ● ●●●● ● ●●● ● ●● ●● ●●●●●●●●● ● ● ●●●●●●●● ● ●● ● ●● ●●●●●●●●●●●●● ● ● ● ● ● ● ●● 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Impor

tance Sampling estimates of P(M=1|y) with 50,000 par

(60)

The only safe cases

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...

[Toni & Stumpf, 2010;Sousa et al., 2009]

...and so does the use of more informal model fitting measures

(61)

The only safe cases

Besides specific models like Gibbs random fields,

using distances over the data itself escapes the discrepancy...

[Toni & Stumpf, 2010;Sousa et al., 2009]

...and so does the use of more informal model fitting measures

http://www.ceremade.dauphine.fr/~xian