R 2 -type Curves for Dynamic Predictions from Joint Longitudinal-Survival Models

(1)

u n i v e r s i t y o f c o p e n h a g e n d e p a r t m e n t o f b i o s t a t i s t i c s

Faculty of Health Sciences

R

2

-type Curves for Dynamic Predictions from

Joint Longitudinal-Survival Models

Inference & application to prediction of kidney graft failure

Paul Blanche joint work with

M-C. Fournier & E. Dantan (Nantes, France)

July 2015 Slide 1/29

(2)

Context & Motivation

• Medical researchers hope to improve patient management

using earlier diagnoses

• Statisticians can help by tting prediction models

• The making of so-called "dynamic" predictions has recently

received a lot of attention

• In order to be useful for medical practice, predictions should

be "accurate"

(3)

Data & Clinical goal

I Data:

DIVAT cohort data

of kidney transplant recipients (subsample, n = 4, 119)

I Clinical goal:

• Dynamic prediction of risk of kidney

graft failure(death or return to dialysis)

• Using repeated measurements of

(4)

DIVAT data sample

(n=4,119)

• French cohort • Adult recipients

• Transplanted after 2000 • Creatinine measured every year

• 6 centers

(5)

DIVAT data sample

(n=4,119)

• French cohort • Adult recipients

• Transplanted after 2000 • Creatinine measured every year • 6 centers

(6)

Statistical challenges discussed

How to evaluate and/or compare dynamic predictions?

I Using concepts of:

• Discrimination • Calibration

I Accounting for:

• Dynamic setting • Censoring issue

(7)

Basic idea & concepts for evaluating predictions

Basic idea: comparing predictions and observations

(simple!)

Concepts: I Discrimination:

A model has high discriminative power if the range of predicted risks is wide and subjects with low (high) predicted risk are more (less) likely to experience the event.

I Calibration:

A model is calibrated if we can expect that x subjects out of 100 experience the event among all subjects that receive a predicted risk of x% (weak denition).

(8)

Basic idea & concepts for evaluating predictions

Basic idea: comparing predictions and observations (simple!)

Concepts: I Discrimination:

I Calibration:

(9)

Basic idea & concepts for evaluating predictions

Basic idea: comparing predictions and observations (simple!) Concepts:

I Discrimination:

I Calibration:

(10)

Basic idea & concepts for evaluating predictions

I Discrimination:

I Calibration:

(11)

Basic idea & concepts for evaluating predictions

I Discrimination:

I Calibration:

(12)

Dynamic prediction

• s: Landmark time at which predictions are made(varies)

• t: prediction horizon(xed)

Creatinine (mol/L) 100 150 200 250 300 0

(13)

Dynamic prediction

follow−up time (years)

Creatinine (mol/L) 100 150 200 250 300 0

(14)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0

(15)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0

(16)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0 s = 4 yearss = 4 years Landmark time ''s''

(17)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0 s = 4 years ● s = 4 years Landmark time ''s''

(18)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0 s = 4 years ● s = 4 years s+t = 9 years Landmark time ''s'' Horizon ''t'' Horizon ''t''

(19)

Dynamic prediction

Creatinine (mol/L) 100 150 200 250 300 0 s = 4 years s+t = 9 years ● 0 % 100 % 1−Pr(Gr aft F ailure in (s ,s+t]) Landmark time ''s'' Horizon ''t'' 1− πi(s, t)=64%

(20)

Right censoring issue

Landmark time s

(when predictions are made) (end of prediction window)Time s + t

time

: uncensored : censored

For subject i censored within (s, s + t] thestatus

Di(s, t) =11{event occurs in(s, s + t]}

(21)

Right censoring issue

Landmark time s

time : uncensored

: censored

(22)

Right censoring issue

Landmark time s

time : uncensored : censored

(23)

Right censoring issue

Landmark time s

For subject i censored within (s, s + t] thestatus Di(s, t) =11{event occurs in(s, s + t]}

(24)

Notations for population parameters

I Indicator of event in (s, s + t]:

Di(s, t) =11{s < Ti ≤ s + t} where Ti is the time-to-event.

I Dynamic predictions: πi(s, t) = bPbξ Di(s, t) =1 Ti> s, Yi(s),Xi

• bξ: previously estimated parameters(from independent training data) • Yi(s): marker measurements observed before time s

(25)

(26)

Notations for population parameters

• bξ: previously estimated parameters(from independent training data)

• Yi(s): marker measurements observed before time s

(27)

• bξ: previously estimated parameters(from independent training data)

• Yi(s): marker measurements observed before time s

(28)

Notations for population parameters

I Dynamic predictions: πi(s, t) = bPbξ Di(s, t) =1 Ti> s, Yi(s), Xi

(29)

(30)

Predictive accuracy

How close are the predicted risks πi(s, t)to the true underlying risk P event occurs in (s,s+t]

information at s? I Prediction Error: PEπ(s, t) = E _n D(s, t) − π(s, t)o2 T > s

• the lower the better • PE ≈ Bias2 + Variance

• evaluates both Calibration and Discrimination • depends on P event occurs in (s,s+t]

at risk at s

(31)

Predictive accuracy

at risk at s

(32)

Predictive accuracy

at risk at s

(33)

How does the PE relate to calibration and

discrimination?

PEπ(s, t) =E hn D(s, t) − π(s, t)o2 T > s i E hn D(s, t)+ ED(s, t) Hπ(s) | {z }

true underlying risk

−π(s, t) T > s

i

Hπ_{(s) = {X}π_{(s), T > s}}_{denotes the subject-specic history at} time s.

I the more discriminating Hπ(s)the smaller VarD(s, t)Hπ (s) I ED(s, t)Hπ(s) − π(s, t) ≡0 denes "strong" calibration.

(34)

discrimination?

PEπ(s, t) =E hn D(s, t) −_ED(s, t) Hπ(s)o2 E hn D(s, t)+ ED(s, t)Hπ(s) | {z }

− π(s, t)o2 T > s

i

Hπ_{(s) = {X}π_{(s), T > s}}_{denotes the subject-specic history at} time s.

(35)

How does the PE relate to calibration and

discrimination?

PEπ(s, t) =E hn D(s, t) −_ED(s, t) Hπ(s)o2 T > s i + Ehn ED(s, t)Hπ(s) | {z }

− π(s, t)o2 T > s

i

Hπ

(s) = {Xπ(s), T > s}denotes the subject-specic history at time s.

(36)

discrimination?

PEπ(s, t) = E hn D(s, t) −_ED(s, t) Hπ(s)o2 T > s i | {z } Inseparability + EhnED(s, t)Hπ(s)− π(s, t) o2 T > s i | {z } Bias/Calibration Hπ

(37)

How does the PE relate to calibration and

discrimination?

PEπ(s, t) = E h VarD(s, t)Hπ(s) T > s i | {z } Discrimination + EhnED(s, t) Hπ_(s) − π(s, t)o2 T > s i | {z } Calibration Hπ

(38)

discrimination?

I the more discriminating Hπ(s)the smaller VarD(s, t)Hπ (s)

(39)

discrimination?

(40)

discrimination?

PEπ(s, t) = EhVarD(s, t)Hπ(s) T > s i | {z }

Does NOT depend onπ(s,t)

+ EhnED(s, t)Hπ(s) − π(s, t) o2 T > s i | {z } Depends onπ(s,t) Hπ

I the more discriminating Hπ(s)the smaller VarD(s, t)Hπ (s) D(s, t) π_{(s) − π(s, t) ≡}_{0 denes "strong" calibration.}

(41)

R

_π2

-type criterion

I Benchmark PE0

The best "null" prediction tool, which gives the same (marginal) predicted risk

S (s + t|s) = ED(s, t)

H0(s), H0(s) = {T > s} to all subjects leads to

PE₀(s, t) =Var{D(s, t)|T > s} = S(s + t|s)n1 − S(s + t|s)o.

I Simple idea

R_π2(s, t) =1 − PEπ(s, t) PE0(s, t)

(42)

R

_π2

-type criterion

I Benchmark PE0

The best "null" prediction tool, which gives the same (marginal) predicted risk

S (s + t|s) = ED(s, t)

H0(s), H0(s) = {T > s} to all subjects leads to

PE₀(s, t) =Var{D(s, t)|T > s} = S(s + t|s)n1 − S(s + t|s)o.

I Simple idea

R_π2(s, t) =1 − PEπ(s, t) PE0(s, t)

(43)

Why bother?

I Rπ2(s, t)aims to circumvent the dicult interpretation of:

• the scale on which PE(s, t) is measured • interpretation for trend of PE(s, t) vs s

I Because the meaning of the scale on which PE(s,t) is measured changes with s, an increasing/decreasing trend can be due to changes in:

• the quality of the predictions and/or • the at risk population

(44)

Why bother?

I Rπ2(s, t)aims to circumvent the dicult interpretation of:

• the scale on which PE(s, t) is measured • interpretation for trend of PE(s, t) vs s

I Because the meaning of the scale on which PE(s,t) is measured changes with s, an increasing/decreasing trend can be due to changes in:

• the quality of the predictions and/or • the at risk population

(45)

Changes in the quality of the predictions

"Essentially, all models are wrong, but some are useful.", G. Box

I The prediction model from which we have obtained the predictions can be more wrong for some s than for some others.

• Calibration term of PE(s, t) changes with s • We can work on it!

(46)

Changes in the quality of the predictions

"Essentially, all models are wrong, but some are useful.", G. Box

I The prediction model from which we have obtained the predictions can be more wrong for some s than for some others.

• Calibration term of PE(s, t) changes with s • We can work on it!

(47)

Changes in the at risk population

An example:

• Patients with cardiovascular history (CV) all die early. • Only those without CV remain at risk for late s.

• Then:

• the earlier s the more homogeneous the at risk population • CV is useful for prediction for early s but useless for late s.

I The available information can be more informative for some s than for some others.

• Discriminating term of PE(s, t) changes with s • This is just how it is, there is nothing we can do!

(48)

Changes in the at risk population

An example:

• Patients with cardiovascular history (CV) all die early. • Only those without CV remain at risk for late s.

• Then:

• the earlier s the more homogeneous the at risk population • CV is useful for prediction for early s but useless for late s.

I The available information can be more informative for some s than for some others.

• Discriminating term of PE(s, t) changes with s • This is just how it is, there is nothing we can do!

(49)

R

_π2

(s, t)

interpretation

I Always true:

Measure of how the prediction tool π(s, t) performs compared to the benchmark null prediction tool, which gives the same predicted risk to all subjects (marginal risk).

I When predictions are calibrated(strongly): R_π2(s, t) = Var{π(s, t)|T > s}

Var{D(s, t)|T > s} explained variation

=Corr2D(s, t), π(s, t)T > s correlation = Enπ(s, t) D(s, t) =1, T > s o

mean risk dierence − Enπ(s, t) D(s, t) =0, T > s o .

(50)

R

_π2

(s, t)

interpretation

I Always true:

=Corr2D(s, t), π(s, t)T > s correlation = Enπ(s, t) D(s, t) =1, T > s o

(51)

R

_π2

(s, t)

interpretation

I Always true:

=Corr2D(s, t), π(s, t) T > s _correlation = Enπ(s, t) D(s, t) =1, T > s o

(52)

R

_π2

(s, t)

interpretation

I Always true:

=Corr2D(s, t), π(s, t) T > s _correlation = Enπ(s, t) D(s, t) =1, T > s o

(53)

Observations & IPCW PE estimator

I Observations (i.i.d.) n e Ti, ∆i, πi(·, ·), i =1, . . . , n o where eTi = Ti∧ Ci, ∆i=11{Ti ≤ Ci}

I Indicator of observed event occurrence in (s, s + t]: e Di(s, t) =11{s < eTi ≤ s + t, ∆i=1} =    1 : event occurred 0 : event did not occur

or censored obs. I Inverse Probability of Censoring Weighting (IPCW) estimator:

c PEπ(s, t) =1 n n X i =1 c Wi(s, t) n e Di(s, t) − πi(s, t) o2 and b Rπ2(s, t) =1 − c PEπ(s, t) c PE0(s, t)

(54)

Observations & IPCW PE estimator

I Observations (i.i.d.) n e Ti, ∆i, πi(·, ·), i =1, . . . , n o where eTi = Ti∧ Ci, ∆i=11{Ti ≤ Ci} I Indicator of observed event occurrence in (s, s + t]:

e Di(s, t) =11{s < eTi ≤ s + t, ∆i=1} =    1 : event occurred 0 : event did not occur

or censored obs.

I Inverse Probability of Censoring Weighting (IPCW) estimator: c PEπ(s, t) =1 n n X i =1 c Wi(s, t) n e Di(s, t) − πi(s, t) o2 and b Rπ2(s, t) =1 − c PEπ(s, t) c PE0(s, t)

(55)

Observations & IPCW PE estimator

c PEπ(s, t) =1 n n X i =1 c Wi(s, t) n e Di(s, t) − πi(s, t) o2 and b Rπ2(s, t) =1 − c PEπ(s, t) c PE0(s, t)

(56)

Observations & IPCW PE estimator

c PEπ(s, t) =1 n n X i =1 c Wi(s, t)nDei(s, t) − πi(s, t) o2 and b Rπ2(s, t) =1 − c PEπ(s, t) c PE0(s, t)

(57)

Observations & IPCW PE estimator

c PEπ(s, t) =1 n n X i =1 c Wi(s, t)nDei(s, t) − πi(s, t) o2 and b Rπ2(s, t) =1 − c PEπ(s, t) c PE0(s, t)

(58)

Inverse Probability of Censoring Weights

c Wi(s, t) = 11{s < eTi ≤ s + t}∆i b G ( eTi|s) + 11{ eTi> s + t} b G (s + t|s) + 0

with bG (u|s)the Kaplan-Meier estimator of P(C > u|C > s).

Landmark time s Time s + t

time

(59)

c Wi(s, t) = 11{s < e Ti≤ s + t}∆i b G ( eTi|s) + 11{ eTi> s + t} b G (s + t|s) + 0

(60)

Inverse Probability of Censoring Weights

c Wi(s, t) = 11{s < e Ti≤ s + t}∆i b G ( eTi|s) +11{ eTi> s + t} b G (s + t|s) + 0

time

(61)

c Wi(s, t) = 11{s < e Ti≤ s + t}∆i b G ( eTi|s) +11{ eTi> s + t} b G (s + t|s) +0

time

(62)

Asymptotic i.i.d. representation

Lemma: Assume that the censoring time C is independent of (T , η, π(·, ·))and let θ denote either PEπ, Rπ2 or a dierence in PE or R2 π, then √ nθ(s, t) − θ(s, t)b = √1 n n X i =₁ IFθ( eTi, ∆i, πi(s, t), s, t) + op(1) where IFθ( eTi, ∆i, πi(s, t), s, t)being :

I zero-mean i.i.d. terms

(63)

Pointwise condence interval (xed s)

• Asymptotic normality: √ nθ(s, t) − θ(s, t)b _D −→ N 0, σs,t2 • 95% condence interval: b θ(s, t) ± z_1−α/2σb_√s,t n

where z1−α/2is the 1 − α/2 quantile of N (0, 1). • Variance estimator: b σ_s,t2 = 1 n n X i =1 n b IFθ( eTi, ∆i, πi(s, t), s, t) o2

(64)

Simultaneous condence band over s ∈ S

b θ(s, t) ±q_b(S,t)_1−ασb_√s,t n , s ∈ S Computation ofqb (S,t)

1−α by thesimulation algorithm(≈ Wild Bootstrap):

1 For b = 1, . . . , B, say B = 4000, do: 1 Generate {ωb

1, . . . , ωbn}from n i.i.d. N (0, 1).

2 Using the plug-in estimator bIFθ(·), compute:

Υb=sup s∈S 2 Computebq (S,t) 1−α as the 100(1 − α)th percentile of Υ1, . . . , ΥB (Conditional multiplier central limit theorem)

(65)

1−α by thesimulation algorithm

(≈ Wild Bootstrap):

1 For b = 1, . . . , B, say B = 4000, do:

1 Generate {ωb

1, . . . , ωbn}from n i.i.d. N (0, 1).

Υb=sup s∈S 2 Computebq (S,t) 1−α as the 100(1 − α)th percentile of Υ1, . . . , ΥB (Conditional multiplier central limit theorem)

(66)

Simultaneous condence band over s ∈ S

1 For b = 1, . . . , B, say B = 4000, do:

1 Generate {ωb

1, . . . , ωbn}from n i.i.d. N (0, 1).

Υb=sup s∈S 2 Computebq (S,t) 1−α as the 100(1 − α)th percentile of Υ1, . . . , ΥB

(67)

1, . . . , ωbn}from n i.i.d. N (0, 1).

Υb=sup s∈S 2 Computebq (S,t) 1−α as the 100(1 − α)th percentile of Υ1, . . . , ΥB

(68)

Simultaneous condence band over s ∈ S

b θ(s, t) ±qb (S,t) 1−αb σs,t √ n , s ∈ S Computation ofqb (S,t)

1, . . . , ωbn}from n i.i.d. N (0, 1).

Υb=sup s∈S 1 √ n n X i =1 ωbi b IFθ( eTi, ∆i, πi(s, t), s, t) b σs,t 2 Computebq (S,t) 1−α as the 100(1 − α)th percentile of Υ1, . . . , ΥB

(69)

1, . . . , ωbn}from n i.i.d. N (0, 1).

(70)

Simultaneous condence band over s ∈ S

1, . . . , ωbn}from n i.i.d. N (0, 1).

(71)

1, . . . , ωbn}from n i.i.d. N (0, 1).

(72)

DIVAT sample

• Population based study of kidney recipients (n=4,119) • Split the data into training (2/3) and validation (1/3) samples

• T: time from 1-year after transplantation to graft failure which is:

Death Return to dialysis

OR

• Censoring due to:

• delayed entries: 2000-2013 • end of follow-up: 2014

• Baseline covariates: age, sex, cardiovascular history • Longitudinal biomarker (yearly): serum creatinine

(73)

DIVAT sample

• Population based study of kidney recipients (n=4,119) • Split the data into training (2/3) and validation (1/3) samples • T: time from 1-year after transplantation to graft failure which is:

OR

• Baseline covariates: age, sex, cardiovascular history • Longitudinal biomarker (yearly): serum creatinine

(74)

DIVAT sample

• Population based study of kidney recipients (n=4,119) • Split the data into training (2/3) and validation (1/3) samples • T: time from 1-year after transplantation to graft failure which is:

OR

(75)

Descriptive statistics & censoring issue

• s ∈ S = {0, 0.5, . . . , 5} • t =5 years s=0 s=1 s=2 s=3 s=4 s=5 Censored in (s,s+5] Known as event−free at s+5 Observed failure in (s,s+5]

landmark time (year)

No . of subjects 0 200 600 1000 1400

(76)

Joint model

I Longitudinal loghYi(tij)

i

= (β₀+b_0i) + β_0,ageAGEi+ β_0,sexSEXi + β₁+b_1i + β_1,ageAGEi × tij+ ij =mi(t)+ εij I Survival (hazard) hi(t) = h0(t)exp γ_ageAGEi+ γCVCVi + α₁mi(t)+ α2 dmi(t) dt

(77)

Joint model

I Longitudinal loghYi(tij)

i

= (β₀+b_0i) + β_0,ageAGEi+ β_0,sexSEXi + β₁+b_1i + β_1,ageAGEi × tij+ ij =mi(t)+ εij I Survival (hazard) hi(t) = h0(t)exp γ_ageAGEi+ γCVCVi + α₁mi(t)+ α2 dmi(t) dt

(78)

R

_π2

(s, t)

vs s

(t=5 years)

Landmark time s (years)

Rπ 2( s , t ) 0 1 2 3 4 5 0 % 10 % 20 % 30 % ● ● ● ● ● ● ● ● ● ● ● 95% pointwise CI 95% simultaneous CB

(79)

Comparing R

2

π

(s, t)

vs s for dierent π(s, t)

Rπ 2( s , t ) 0 1 2 3 4 5 0 % 10 % 20 % 30 % ● ● ● ● ● ● ● ● ● ● ● T ~ Age + CV + m(t) + m'(t) (JM)

(80)

Comparing R

2

π

(s, t)

Rπ 2( s , t ) 0 1 2 3 4 5 0 % 10 % 20 % 30 % ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● T ~ Age + CV + m(t) + m'(t) (JM) T ~ Age + CV

(81)

Comparing R

2

π

(s, t)

Rπ 2( s , t ) 0 1 2 3 4 5 0 % 10 % 20 % 30 % ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● T ~ Age + CV + m(t) + m'(t) (JM) T ~ Age + CV + Y(t=0) T ~ Age + CV

(82)

Comparing R

2

π

(s, t)

Rπ 2( s , t ) 0 1 2 3 4 5 0 % 10 % 20 % 30 % ● ● ● ● ● ● ● ● ● ● ● ● ● ● T ~ Age + CV + m(t) + m'(t) (JM) T ~ Age + CV + Y(t=0)

(83)

Comparing R

2 π

(s, t)

Landmark time s Diff erence in Rπ 2(s ,t) ● ● ● ● ● ● ● −5 % 5 % 10 % 20 % 0 % 0 1 2 3 4 5

(84)

Comparing R

2 π

(s, t)

Landmark time s Diff erence in Rπ 2(s ,t) ● ● ● ● ● ● ● ● ● ● ● −5 % 5 % 10 % 20 % 0 % 0 1 2 3 4 5

(85)

Calibration plot

(example for s = 3 years)

(0,10) (0;3.5] (10,20) (3.5;5.3] (20,30) (5.3;8] (30,40) (8;10.4] (40,50) (10.4;13.1] (50,60) (13.1;16.5] (60,70) (16.5;22.3] (70,80) (22.3;33] (80,90) (33;55.1] (90,100) (55.1;100] Predicted Observed

Risk groups (quantile groups, in %)

5−y ear r isk (%) 0 20 40 60 80 100 2% 8(4)% 4% 15(6)% 7% 15(5)% 9% 13(5)% 12% 16(6)% 15% 21(7)% 19% 26(6)% 27%29(7)% 42% 36(9)% 77%_74(8)%

(86)

Area under the ROC(s, t) curve vs s

A U Cπ ( s , t ) 0 1 2 3 4 5 50% 60% 70% 80% 100% ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● T ~ Age + CV + m(t) + m'(t) (JM) T ~ Age + CV + Y(t=0) T ~ Age + CV

(87)

Summing up

I R2-type curve

• summarizes calibration and discriminating simultaneously • has an understandable trend

I Simple model free inference

• predictions can be obtained from any model • we do not assume any model to hold

• allows fair comparisons of dierent predictions

I The method accounts for:

• Censoring

(88)

Discussion

I The strong calibration assumption allows dierent interesting

interpretations:

• Explained variation • Correlation

• Mean risk dierence

I Unfortunately

• the strong calibration cannot be checked (curse of dimensionality)

I However

• weak and strong denitions are closely related: • _{strong calibration implies weak calibration}

• _{weak calibration can often be seen as a reasonable}

approximation of strong calibration in practice

• weak calibration can be assessed (plots)

(89)

Discussion

I The strong calibration assumption allows dierent interesting

interpretations:

• Explained variation • Correlation

• Mean risk dierence

I Unfortunately

• the strong calibration cannot be checked (curse of dimensionality)

I However

• weak and strong denitions are closely related: • _{strong calibration implies weak calibration}

• _{weak calibration can often be seen as a reasonable}

approximation of strong calibration in practice

• weak calibration can be assessed (plots)

(www.divat.fr)