Bayesian Penalized Methods for High Dimensional Data

Bayesian

Penalized

Methods

for

High

Dimensional

Data

Joseph

G.

Ibrahim

Joint

with

Hongtu

Zhu

•

Motivation

•

GLRR:

Bayesian

Generalized

Low

Rank

Regression

•

L2R2:

Bayesian

Longitudinal

Low

Rank

Regression

•

ADNI

data

analysis

Alzheimer’s

Disease

•

genetically complex, progressive, and fatal neurodegenetive 

disease. 

• The incidence of AD doubles every five years after the age of 65 and the number of AD patients has recently dramatically 

increased, which has caused a heavy socioeconomic burden.  

• AD is the sixth‐leading  cause of death  in the United States, and  

ADNI

Database

• The Alzheimer's Disease Neuroimaging Initiative (ADNI) is the first 

"Big Data" project for AD and is collecting imaging, genetic, 

clinical, and cognitive data  for measuring the progress of AD or 

the effects of treatment.

• ADNI began 2004 and has three phases including ADNI 1, ADNI 

Go, and ADNI 2. 

• Efficiently integrating big ADNI data  may  lead to 

(AD1)  detecting AD at the earliest stage possible and marking its progress    

through  biomarkers;

(AD2)  developing new diagnostic methods for AD intervention, prevention and 

ADNI

Database

•

ADNI

1.

genetic and environmental contributions to  brain baseline data 

and brain development trajectories. 

•

Model:

Brain

volume

=

f(SNP,

age,

gender,

…)

•

Data:

•

•

•

Magnetic Resonance Imaging (MRI)

•

•

a voxel,  stores in 3‐D array

•

•

blood flow

•

Voxel:

n

subjects

will

yield

n

x6 million

matrix

•

ROIs

reduce

dimension

to

93

ROIs

Single

‐

Nucleotide

Polymorphism

(SNP)

•

Normal

(not

rare)

different

nucleotides

in

the

same

location

•

SNPs

may

affect

gene

function

•

ADNI:

600,000

SNPs

→

n=750

<<

600,000

SNPs

Select SNPs only on top 40 genes reported by

Bayesian

Shrinkage

and

Selection

•

Prior

:

•

̶

_log(prior)

₌

_penalty

_function

₌

•

Posterior:

•

Frequentist

penalized

estimation

≡

Maximum

aposteriori

(MAP)

estimation

•

MLE

sets

penalty

to

0

(MAP

with

non

‐

Bayesian

Shrinkage

and

Selection

•

Popular

choice

•

α ≤

1

 →

shrinkage

and

selection:

creates

singularity

at

0

and

a

 black

hole

,

to

pull

smaller

elements

to

0

•

Bridge

regression:

  α

<

1

•

L

priors

(lasso,

adaptive

lasso):

 α

=

1

•

α

>

1

 →

No

selection,

shrinkage

only

Black

Hole

Priors:

α ≤

1

Want

huge

spike

(gravity)

at

the

origin;

Gravity

should

pull

the

smaller

coefficients

to

0

Huge spike/gravity implies smaller

coefficients shrink more

Smaller spike/gravity implies smaller

coefficients shrink less

Distributional

Perspective

Singularity/Discontinuity at the origin

Want

heavy

tails/minimum

gravity

/

flat

density

far

from

origin;

Gravity

should

not

affect

the

larger

coefficients

Steeper slope/stronger gravity implies larger coefficients shrink more Flatter tail/weaker gravity

implies larger

coefficients shrink less

Commonly

Used

Priors

Larger spike at the origin and heavier tails

•

Do

SNPs

act

alone

or

work

together?

•

Do

the

ROIs

also

act

together?

•

Do

ROIs

and

SNPs

acting

together

support

some

underlying

structure

in

the

regression

coefficients.

•

We

try

and

exploit

this

structure

to

reduce

dimension

GLRR:

Low

Rank

Regression

p

=

r*(p+d)

<<

p*d,

5*(1K+1K)

=

10K

<<

1K*1K

=

1

million

•

U

and

V

need

not

be

unitary

(orthonormal)

–

otherwise

need

matrix

VMF

and

metropolis

•

No

ordering

restriction

on

elements

of

 Δ

–

otherwise

need

truncated

normal

and

metropolis

•

Many

Bayesian

applications

do

not

require

identifiability

•

Allows

closed

form

full

conditionals

to

apply

Gibbs

sampler

•

•

GLRR:

Model

and

Priors

Cov(Y_i) =

GLRR:

Why

L

Priors

If

covariates

are

correlated

‐

•

L

tends

to

push

them

towards

each

other

¾ more correlated estimates (Ridge), reason for 

our choice 

•

_L

_tends

_to

_pick

_one,

_force

_the

_rest

_to

₀

least absolute subset selection operator (lasso)

True β 1 1 1 1 1 1 1 1 1 1

OLS _{2.95 1.09 ‐1.11 1.24 0.98 0.98 1.57 1.14 1.33 0.66} Ridge _{1.13 1.02 0.75 1.19 0.86 0.99 1.46 1.03 1.21 0.62} Lasso _{0 0 0 2.95 0 0.07 0.97 0 0.23 0}

GLRR:

Comparison

Criteria

for

Determining

the

Rank

of

B

GLRR:

Simulated

ROC

GLRR:

Simulated

Image

Recovery

GLRR better for low rank, lasso and

GLRR are similar for high rank

•

volumes, p = 1,072 SNPs on top  40 genes from 

AlzGene database. 

•

•

•

thresholding

•

correspond to ROI

•

bin (ROI), Bbin BbinT (SNP)

GLRR:

ADNI

Application

Largest Diagonals Largest Column Sum

Top ROI: highest # of 

significant SNPs

Top ROI: highest # sig. SNPs and 

highest # sig. of SNPs that also  

affect other ROIs

7.1 g protein/ounce 0.81 g  protein/ounce 0.10 g protein/calorie 0.12 g  protein/calorie

GLRR:

Using

B

_B

GLRR:

ADNI

Results

-log₁₀(p) of B -log₁₀ (p) of U -log₁₀ (p) of V

B B_binT_B

GLRR:

ADNI

ROI

Network

Top 20 ROIs based on B_binT_B

bin and 3 layers of V

ROIs most highly correlated with rs10792821(PICALM),

rs9791189(NEDD9), rs9376660(LOC651924), rs17310467(PRNP), respectively.

•

q*

=

number

of

random

effects

•

Covariance

estimation

same

as

GLRR

•

Can

apply

Gibbs

sampler

L2R2

:

Simulated

ROC

same for prognostic 

factors

L2R2  better than G‐SMuRFS for 

L2R2:

Simulated

Image

Recovery

•

GLRR

outperforms

LASSO,

BLASSO,

and

G

‐

SMuRFS

in

a

great

many

settings.

•

Gibbs:

Scale

to

larger

dimensions

•

only

feasible

choice

for

HD

data

•

Metropolis:

Don’t

scale

•

Single try: works on small dimensions

•

Multiple

try:

only

on

tiny

dimensions

•

Selection

with

p >> n

is

unstable

•

Computer

code

written

in

MATLAB

•

For

r=3

in

GLRR,

30

minutes

for

10K

samples

(1500

parameters).

•

For

r=5

in

GLRR,

40

minutes

for

10K

samples

(2500

parameters)

•

BLASSO

takes

3

hours

(40K

parameters).