• No results found

Modelling and added value

N/A
N/A
Protected

Academic year: 2021

Share "Modelling and added value"

Copied!
53
0
0

Loading.... (view fulltext now)

Full text

(1)

Modelling and added value

Course: Statistical Evaluation of Diagnostic and Predictive Models

Thomas Alexander Gerds (University of Copenhagen) Summer School, Barcelona, June 30, 2015

(2)

Multiple regression

Multiple regression can be used to exploit the joint predictive power of several or many variables, and also to assess the added value of new markers in the presence of conventional risk factors.

Commonly used modelling techniques:

I logistic regression for binary outcome

I Cox regression for time-to-event (survival) outcome

P-values testing the null hypothesis of no association are not a good measure of predictive power.

(3)

Example: epo study

1

Anaemia is a deciency of red blood cells and/or hemoglobin and an additional risk factor for cancer patients.

Randomized placebo controlled trial: does treatment with epoetin beta epo (300 U/kg) enhance hemoglobin concentration level and improve survival chances?

Henke et al. 2006 identied the c20 expression (erythropoietin receptor status) as a new biomarker for the prognosis of locoregional progression-free survival.

1Henke et al. Do erythropoietin receptors on cancer cells explain

unexpected clinical ndings? J Clin Oncol, 24(29):4708-4713, 2006.

(4)

Treatment

The study includes 1492 head and neck cancer patients with a

tumor located in the oropharynx (36%), the oral cavity (27%), the larynx (14%) or in the hypopharynx (23%).

One of the treatments was radiotherapy following Resection

Complete Incomplete No

Placebo 35 14 25

Epo 36 14 25

2with non-missing blood values

(5)

Outcome

Blood hemoglobin levels were measured weekly during radiotherapy (7 weeks).

Treatment with epoetin beta was denedsuccessfulwhen the hemoglobin level increased suciently. For patienti set

Yi = (

1 treatment successful 0 treatment failed

(6)

Target

Patient no. Treatment successful Predicted probability

1 0 P1 2 0 P2 3 1 P3 4 1 P4 5 0 P5 6 1 P6 7 1 P7 · · · · · · 6 / 53

(7)

Predictors

Age min: 41 y, median: 59 y, max: 80 y

Gender male: 85%, female: 15%

Baseline hemoglobin mean: 12.03 g/dl, std: 1.45

Treatment epo: 50%, placebo 50%

Resection complete: 48%, incomplete: 19%,

no resection: 34% Epo

receptor status neg: 32%, pos: 68%

(8)

Logistic regression

Response: treatment successful yes/no

Factor OddsRatio StandardError CI.95 pValue

(Intercept) 0.00 4.01 <0.0001 Age 0.97 0.03 [0.91;1.03] 0.2807 Sex:female 4.71 0.84 [0.91;26.02] 0.0657 HbBase 3.25 0.27 [1.99;5.91] <0.0001 Treatment:Epo 90.92 0.76 [23.9;493.41] <0.0001 Resection:Incompl 1.75 0.81 [0.36;9.03] 0.4924 Resection:Compl 4.14 0.69 [1.13;17.36] 0.0395 Receptor:positive 5.81 0.66 [1.72;23.39] 0.0076 8 / 53

(9)

The model provides general information

Treatment with epo increases the chance (odds) of reaching the target hemoglobin level signicantly by a factor of

90.92 (CI95%: [23.9;493.4],p<0.0001)

in the overall study population.

Does that mean everyone should be treated?

(10)

The model provides information for a single patient

For example: the predicted probability that a 51 year old man with complete tumor resection and baseline hemoglobin level 12.6g/dl

reaches the target hemoglobin level (Yi=1) is

[Epo group: ] 97.4% [ Placebo: ] 29.2 %

If a similar patient has baseline hemoglobin level 14.8g/dl then the

model predicts: [Epo group: ] 99.8% [Placebo: ] 84.7 %

(11)

Predictions and Brier score for logistic regression

Patient Treatment Predicted Brier no. successful probability (%) Residual score

Yi Pi Yi−Pi (Yi−Pi)2 · · · · · 142 0 84.09 -84.09 0.7071 143 0 93.47 -93.47 0.8737 144 0 18.73 -18.73 0.0351 145 0 1.81 -1.81 0.0003 146 0 3.86 -3.86 0.0015 147 1 96.64 3.36 0.0011 148 0 0.5 -0.5 <0.0001 149 0 11.93 -11.93 0.0142 Σ0.0869 11 / 53

(12)

The model behind the table

log Pi 1−Pi =β01x1,i +· · ·+βkxk,i ⇔ Pi = 1 1 +exp{β01x1,i+· · ·+βkxk,i}

I Pi the probability of successful treatment

I x1,i rst predictor for subject i: (e.g. age = 50)

I x2,i second predictor for subjecti: (e.g. gender = male)

I · · ·

I xk,i k'th predictor for subjecti: (e.g. epoReceptor = pos)

I β0, . . . , βk are regression coecients that are estimated based

on the epo study

(13)

Predicted treatment success probability (logistic regression)

For a treated man with no resection possible and negative epo receptor status. Predicted risk Age (years) Baseline hemoglobin (g/dl) 9 10 11 12 13 14 50 60 70 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 13 / 53

(14)

Nomogram

Points 0 10 20 30 40 50 60 70 80 90 100 age 80 55 sex male female HbBase 8 9 10 11 12 13 14 15 16 17 Treat Placebo Epo Resection No Compl Incompl epoRec 0 1 Total Points 0 20 40 60 80 100 120 140 160 180 Linear Predictor −8 −6 −4 −2 0 2 4 6 8 Chance of treatment success

0.001 0.01 0.05 0.25 0.75 0.95 0.99 0.999 Points 0 10 20 30 40 50 60 70 80 90 100 age 80 55 sex male female HbBase 8 9 10 11 12 13 14 15 16 17 Treat Placebo Epo Resection No Compl Incompl epoRec 0 1 Total Points 0 20 40 60 80 100 120 140 160 180 Linear Predictor −8 −6 −4 −2 0 2 4 6 8

Chance of treatment success

0.001 0.01 0.05 0.25 0.75 0.95 0.99 0.999

Figure: Nomogram showing the predictions of the logistic regression

(15)

Nomogram: R-code

library(rms)

f7 <- lrm(Y~age+sex+HbBase+Treat+Resection+epoRec,data= Epo,x=TRUE,y=TRUE)

dd <- datadist(Epo)

options(datadist = "dd")

nom7 <- nomogram(f7, fun=function(x)1/(1+exp(-x)),

fun.at=c(.001,.01,.05,0.25,0.75,.95,.99,.999),

funlabel="Chance of treatment success")

plot(nom7)

library(DynNom)

f7 <- glm(Y~age+sex+HbBase+Treat+Resection+epoRec,data= Epo,family=binomial())

DynNom(f7,Epo,clevel=0.95)

(16)

Tools for evaluating prediction accuracy

For each subject we have a predicted risk based on multiple predictors. To evaluate the prediction performance of the logistic regression model we consider the following tools:

I Prediction accuracy: Brier score (lack of calibration and lack of spread of predictions)

I Discrimination: Roc curve, c-index = AUC (lack of spread of predictions)

I Calibration plot: (lack of calibration)

I Re-classication scatterplot/table: (changes of risk predictions)

Brier score: The squared dierence between the observed status and the predicted risk.

AUC: The fraction of randomly selected pairs of patients where the predicted risk was higher for the diseased subject compared to the non-diseased subject.

(17)

Brier score for null model in the Epo study

Patient Treatment Predicted Brier no. successful probability (%) Residual score

Yi Pi Yi−Pi (Yi−Pi)2 · · · · · 142 0 44.3 -44.3 0.1962 143 0 44.3 -44.3 0.1962 144 0 44.3 -44.3 0.1962 145 0 44.3 -44.3 0.1962 146 0 44.3 -44.3 0.1962 147 1 44.3 55.7 0.3103 148 0 44.3 -44.3 0.1962 149 0 44.3 -44.3 0.1962 Σ0.247

The predicted probability is the prevalence of patients with treatment success in the data set.

(18)

Prevalence model

Calibration plot

Predicted probability of treatment success

Obser v ed propor tion 0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % ●

Performance null model Brier=24.7 AUC=50.0

(19)

Univariate logistic regression models

Categorical predictors library(rms)

resecModel <- lrm(Y~Resection,data=Epo,x=TRUE,y=TRUE) sexModel <- lrm(Y~sex,data=Epo,x=TRUE,y=TRUE)

treatModel <- lrm(Y~Treat,data=Epo,x=TRUE,y=TRUE)

## or via glm

treatModel <- glm(Y~Treat,data=Epo,family="binomial") Continuous predictors

library(rms)

baseHbModel <- lrm(Y~HbBase,data=Epo,x=TRUE,y=TRUE) ageModel <- glm(Y~age,data=Epo,family="binomial")

(20)

Categorical predictors

Resection status

Treatment success 0 Treatment success 1

No 33 17

Incompl 16 12

Compl 34 37

Gender

Treatment success 0 Treatment success 1

male 71 56

female 12 10

Treatment

Treatment success 0 Treatment success 1

Placebo 66 8

Epo 17 58

(21)

Categorical predictors: Resection status, gender, treatment

Calibration plot

Predicted probability of treatment success

Obser v ed propor tion 0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % ●● ● ● ● ● ● Null model Brier=24.7 AUC=50.0 ● Gender model Brier=24.7 AUC=50.3 ● Resection model Brier=24.0 AUC=58.7 ● Treatment model Brier=13.6 AUC=83.7 21 / 53

(22)

Continuous predictors: Baseline hemoglobin, Age

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 40 50 60 70 80 8 10 12 14 16 Scatter plot Age (years) Baseline hemoglobin (g/dl) ● ● Treatment success Treatment failed 22 / 53
(23)

Continuous predictors: Baseline hemoglobin, Age

Calibration plot

Predicted probability of treatment success

0 % 25 % 50 % 75 % 100 % Obser v ed propor tion 0 % 25 % 50 % 75 % 100 % Null model Brier=24.7 AUC=50.0 ● Age model Brier=24.7 AUC=51.2 ●

Baseline hemoglobin model Brier=19.3 AUC=77.2

(24)

Continuous predictors: Baseline hemoglobin, Age

Roc curves 1−Specificity Sensitivity 0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % Null model Brier=24.7 AUC=50.0 ● Age model Brier=24.7 AUC=51.2 ● Baseline hemoglobin Brier=19.3 AUC=77.2 24 / 53
(25)

Continuous predictors: Baseline hemoglobin, Age

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Re−classification plot

Predicted chance (Age model)

Predicted chance (Hemoglobin model)

0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % ● ● Treatment success Treatment failed 25 / 53

(26)

Multiple logistic regression

Model excluding epo receptor status

add <- lrm(Y~age+sex+HbBase+Treat+Resection,data=Epo,x= TRUE,y=TRUE)

Model including epo receptor status

add.epoR <- lrm(Y~age+sex+HbBase+Treat+Resection+epoRec

,data=Epo,x=TRUE,y=TRUE)

(27)

Multiple logistic regression

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● Re−classification plot

Predicted chance (excluding receptor status)

Predicted chance (including receptor status)

0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % ● ● Treatment success Treatment failed 27 / 53

(28)

Multiple logistic regression

Calibration plot

Predicted event probability

0 % 25 % 50 % 75 % 100 % Obser v ed propor tion 0 % 25 % 50 % 75 % 100 % Null model Brier=24.7 AUC=50.0 ● All variables Brier= 9.6 AUC=93.3 ●

All + receptor status Brier= 8.7 AUC=94.7

(29)

Multiple logistic regression

Roc curves 1−Specificity Sensitivity 0 % 25 % 50 % 75 % 100 % 0 % 25 % 50 % 75 % 100 % ● All variables Brier= 9.6 AUC=93.3 ●

All + receptor status Brier= 8.7 AUC=94.7

(30)

Exercises 2.1

1. Do the tutorial 'Added value of new marker'

2. Split the IVF data (see link on course homepage) at random into two parts (60% for learning, 40% for evaluation). Then, build a multiple logistic regression model to predict response. Include the following covariates: antfoll, smoking, fsh, ovolume, bmi.

3. Produce a table which shows the odds ratios with condence limits (hint: Publish::publish.glm(t)) and write a caption which explains the table.

4. Produce a calibration plot and write a caption. (hint: ModelGood::calPlot2)

5. Produce a Roc curve, add the Brier score and AUC as a legend, and write a caption.

6. Build a second logistic regression model where you include the above variables and add the variable cyclelen.

7. Evaluate the added value of cyclelen: re-classication table and plot (hint: ModelGood::reclass), dierence in Brier scores and AUC with appropriate tests. Describe the underlying null hypotheses.

8. For each subject in the test data compute the dierence of the predictions between the model which excludes cyclelen and the model that includes cyclelen. Consider this dierence as a new continuous marker and produce the corresponding ROC curve and AUC. Describe the interpretation of AUC for this specic ROC curve in words and comment.

(31)

Model selection

Very many dierent 'logistic regression models' can be constructed by selecting subsets of variables and transformations/groupings of variables.

Standard multiple (logistic) regression works if

I the number of predictors is not too large, and substantially

smaller than the sample size

I the decision maker has a-priory knowledge about which

variables to put into the model

Ad-hoc model selection algorithms, like automated backward elimination, do not lead to reproducible prediction models.

(32)

A Conversation of Richard Olshen with Leo Breiman3

. . .

Olshen: What about arcing, bagging and boosting?

Breiman: Okay. Yeah. This is fascinating stu, Richard. In the last ve years, there have been some really big breakthroughs in prediction. And I think combining predictors is one of the two big breakthroughs. And the idea of this was, okay, that suppose you take CART, which is a pretty good classier, but not a great classier. I mean, for instance, neural nets do a much better job.

Olshen: Well, suitably trained? Breiman: Suitably trained. Olshen: Against an untrained CART?

Breiman: Right. Exactly. And I think I was thinking about this. I had written an article on subset selection in linear regression. I had realized then that subset selection in linear regression is really a very unstable procedure. If you tamper with the data just a little bit, the rst best ve variable regression may change to another set of ve variables. And so I thought, Okay. We can stabilize this by just perturbing the data a little and get the best ve variable predictor. Perturb it again. Get the best ve variable predictor and then average all these ve variable predictors. And sure enough, that worked out beautifully. This was published in an article in the Annals (Breiman, 1996b).

. . .

(33)
(34)

Backward elimination

On full data (n=149): library(rms)

data(Epo)

f7 <-lrm(Y~age+sex+HbBase+Treat+Resection+epoRec,data=Epo,x=TRUE,y=TRUE) fastbw(f7)

Deleted Chi-Sq d.f. P Residual d.f. P AIC age 1.16 1 0.2807 1.16 1 0.2807 -0.84 Resection 3.75 2 0.1532 4.92 3 0.1781 -1.08 Approximate Estimates after Deleting Factors

Coef S.E. Wald Z P

Intercept -16.772 3.6854 -4.551 0.00000534015 sex=female 1.672 0.8221 2.034 0.04195853231 HbBase 1.099 0.2719 4.043 0.00005279348 Treat=Epo 3.843 0.6992 5.496 0.00000003887 epoRec 1.413 0.6355 2.224 0.02615849462 Factors in Final Model

[1] sex HbBase Treat epoRec

(35)

Backward elimination

On reduced data (n=130): library(rms)

data(Epo)

set.seed(1731)

f7a <-lrm(Y~age+sex+HbBase+Treat+Resection+epoRec,data=Epo[sample(1:149, replace=FALSE,size=130),],x=TRUE,y=TRUE)

fastbw(f7a)

Deleted Chi-Sq d.f. P Residual d.f. P AIC age 0.49 1 0.4850 0.49 1 0.4850 -1.51 sex 1.34 1 0.2462 1.83 2 0.4001 -2.17 Resection 4.58 2 0.1012 6.41 4 0.1704 -1.59 Approximate Estimates after Deleting Factors

Coef S.E. Wald Z P

Intercept -14.0968 3.4516 -4.084 0.0000442475 HbBase 0.9241 0.2629 3.516 0.0004388287 Treat=Epo 3.4380 0.6807 5.050 0.0000004411 epoRec 1.3169 0.6744 1.953 0.0508560568 Factors in Final Model

[1] HbBase Treat epoRec

(36)

Guided model selection

The hope of conventional regression modelling is that the better the model ts the better it predicts. But, the model should predict new patients.

Prostate Cancer Risk Calculator: We used multivariable logistic regression to model the risk of prostate cancer by considering all possible combinations of main eects and interactions.

The models chosen were those that minimized the Bayesian information criterion (BIC) and maximized the average

out-of-sample area under the receiver operating characteristic curve (via 4-fold cross-validation).

(37)

The two cultures

4

4L. Breiman. Statistical modeling: The two cultures. Statistical Science, 16

(3):199-215, 2001.

(38)

The two cultures

(39)

Classication trees

A tree model is a form of recursive partitioning.

It lets the data decide which variables are important and where to place cut-os in continuous variables.

In general terms, the purpose of the analyzes via tree-building algorithms is to determine a set of splits that permit accurate prediction or classication of cases.

In other words: a tree is a combination of many medical tests.

(40)

Epo study

arm p < 0.001 1 Placebo Epo Resection p = 0.043 2

{No, Incomplete} Complete Node 3 (n = 39) 0 1 0 0.2 0.4 0.6 0.8 1 Node 4 (n = 35) 0 1 0 0.2 0.4 0.6 0.8 1 HbBase p < 0.001 5 ≤11.3 >11.3 Node 6 (n = 19) 0 1 0 0.2 0.4 0.6 0.8 1 Node 7 (n = 56) 0 1 0 0.2 0.4 0.6 0.8 1 40 / 53

(41)

Roughly, the algorithm works as follows:

1. Find the predictor so that the best possible split on that predictor optimizes some statistical criterion over all possible splits on the other predictors.

2. For ordinal and continuous predictors, the split is of the form

X <c versus X ≥c.

3. Repeat step 1 within each previously formed subset.

4. Proceed until fewer than k observations remain to be split, or

until nothing is gained from further splitting, i.e. the tree is fully grown.

5. The tree is pruned according to some criterion.

(42)

Characters of classication trees

I Trees are specically designed for accurate

classication/prediction

I Results have a graphical representation and are easy to

interpret

I No model assumptions

I Recursive partitioning can identify complex interactions I One can introduce dierent costs of miss-classication in the

three But:

I Trees are not robust against even small perturbations of the

data.

I It is quite easy to over-t the data.

(43)

More complex tree (overtting?)

arm p < 0.001 1 Placebo Epo Resection p = 0.043 2 {No, Incomplete}Complete Node 3 (n = 39) 0 1 0 0.2 0.4 0.6 0.8 1 HbBase p = 0.309 4 ≤12.1 >12.1 Node 5 (n = 25) 0 1 0 0.2 0.4 0.6 0.8 1 Node 6 (n = 10) 0 1 0 0.2 0.4 0.6 0.8 1 HbBase p < 0.001 7 ≤11.3 >11.3 Node 8 (n = 19) 0 1 0 0.2 0.4 0.6 0.8 1 Resection p = 0.641 9 No{Incomplete, Complete} Node 10 (n = 18) 0 1 0 0.2 0.4 0.6 0.8 1 epoRec p = 0.527 11 positive negative Node 12 (n = 27) 0 1 0 0.2 0.4 0.6 0.8 1 Node 13 (n = 11) 0 1 0 0.2 0.4 0.6 0.8 1 43 / 53
(44)

Comparing the dierent predictions

Predicted probability (%) Patient

no. Treatmentsuccessful LRM Simpletree Complextree

1 0 2.31 22.86 12 2 0 1.91 0 0 3 1 98.11 92.86 100 · · · · · · · · · · 142 0 84.09 92.86 90.91 143 0 93.47 92.86 83.33 144 0 18.73 0 0 145 0 1.81 0 0 146 0 3.86 22.86 12 147 1 96.64 92.86 100 148 0 0.5 22.86 12 149 0 11.93 31.58 31.58 44 / 53

(45)

Comparing the dierent predictions

Model Brier score AUC

Simple tree 0.094 0.925

Logistic regression 0.087 0.947

Complex tree 0.085 0.951

Random forest 0.03 0.998

Note: These numbers are estimated by using the same data that were used to construct the models.

(46)

Dilemma:

Both, logistic regression with automated variable selection, e.g., backward elimination, and also decision trees are notoriously unstable (overt).

How shall we proceed?

(47)

In search of a solution

Genuine algorithms to obtain a useful prediction model:

Xi →

Neural Nets

Support Vector Machines Bump hunting and LASSO

Rigde regression and boosting

RandomForests Logic regression

→ Fˆ(y|Xi)

All these algorithms can be applied in high dimensional settings, i.e., when there are more candidate predictor variables than subjects.

(48)

Penalized likelihood regression (works for logistic and Cox

partial likelihood)

Ridge regression:

ˆ

βridge =argmax{likelihood(β)−λ X j βj2} Shrinks LASSO regression: ˆ

βLASSO =argmax{likelihood(β)−λ

X

j |βj|}

Shrinks and selects

Elastic net: combines L1 and L2 norm

(49)

Package glmnet

library(ModelGood) library(glmnet)

g1a <-glmnet(as.numeric(Epo$Y)-1,x=model.matrix(~-1 +age +

HbBase + Treat + Resection + epoRec+sex,data=Epo),alpha=0.1)

g1 <- ElasticNet(Y~age + HbBase + Treat + Resection + epoRec+sex

,data=Epo,alpha=0.1)

plot(g1a)

print(g1) $call

ElasticNet(formula = Y ~ age + HbBase + Treat + Resection + epoRec + sex, data = Epo, alpha = 0.1)

$enet

Call: glmnet(x = covariates, y = response, alpha = 0.1, lambda = optlambda) Df %Dev Lambda [1,] 7 0.5824 0.01655 $Lambda [1] 0.01655384 attr(,"class") [1] "ElasticNet" 49 / 53

(50)

Shrinked regression coecients

0.0 0.2 0.4 0.6 0.8 1.0 −0.3 −0.2 −0.1 0.0 0.1 0.2 0.3 L1 Norm Coefficients 0 3 5 5 7 8 50 / 53
(51)

A function of the penalization parameter

λ

−5 −4 −3 −2 −1 0 1 −0.3 −0.2 −0.1 0.0 0.1 0.2 0.3 Log Lambda Coefficients 8 8 8 7 5 3 2 51 / 53
(52)

Summary

I Predicted probabilities for the unknown current or future event

status of a subject can be obtained from a penalized or unpenalized logistic regression models.

I Predictions can also be obtained from a decision tree or

random forest.

I Re-classication plots, calibration plots, ROC curves, Brier

score and AUC can be used to assess and compare the performance of dierent models.

I The apparent comparison using the same data that were used

to select and t the models is not fair and may be grossly misleading.

I Advanced algorithmic methods have tuning parameters which

are optimized for obtaining accurate predictions.

(53)

Exercise 2.2

I Consider the results of Exercise 2.1.

I Change the seed several times used to split the IVF data and

repeat the analysis. Report the Monte carlo error in the AUC of the two models.

I Introduce a random normal noise variable into the IVF data set

and analyse its added value. Repeat with 10 such variables to see if any of these random noise variable has higher added value than cyclelen.

References

Related documents

The vari- able ordering chosen allows for contemporaneous e ff ects of all variables on the monetary policy instrument, while the fiscal policy indicator is assumed not to react

Chart 1: Credit/GDP and real credit growth for the Baltic countries, observed value, long-term trend and interval of 1.75 times the standard deviation around the trend, in %..

Because of the importance and sensitivity of CA1 area of hippocampus and the pathophysiological mechanisms of cerebral ischemia, this study provided the novel thera- peutic window

An independent assessment of the rate of attack by termites, and the associated costs of the ensuing damage on domestic housing in Sydney was undertaken by surveying home owners

The decisive variables that have a negative impact on the attractiveness and popularity of a natural tourism destination are Tourism environmental impact as a corollary of the

If your proposed research is likely to generate significant outputs - data, software, materials and/or intellectual property - that will hold clear value as a resource for others

MDFT recognizes the central role of the family in the development and treatment of adolescents ’ drug use problems ( Liddle et al., 2001 ; Muck et al., 2001 ), and its