The Assessment of Fit in the Class of Logistic Regression Models: A Pathway out of the Jungle of Pseudo-R²s

(1)

The Assessment of Fit in the

Class of Logistic Regression

Models:

A Pathway out of the Jungle

of Pseudo-R²s

Dr. Wolfgang Langer

Martin-Luther-University of Halle-Wittenberg

(2)

1. What is the problem ?

Situation now:

G An increasing number of people uses logistic

models for qualitative dependent variables.

G There is no general agreement on how to assess

the fit corresponding to practical significance.

G Where is the pathway out of the jungle of the

(4)

Amemiya (1981):

Recommendations

G 1. There is not any universial pseudo

coefficient of determination.

G 2. Most of the Pseudo-R²s have values outside

the range of [0;1]. They are not completely standardized.

G 3. Assessing the model fit you have to use 2

or 3 Pseudo-R²s basing on different construction criteria.

(5)

What is the situation now ?

J.Scott Long: (1997):

- Regression Models for Categorical and Limited Dependent Variables. SAGE

- (2000): Scalar Measures of Fit for Regression Models. (http://www.indiana.edu/~jsl650/ )

1.He presents a lot of Pseudo-R2s discussed in

econometric literature.

2.He recommends the McKelvey-Zavoina

Pseudo-R2 as the best fit measure for binary and ordinal

(6)

2.Summary of the econometric Monte-Carlo

studies for Pseudo-R

2

s

G

Econometricians have made a lot of

Monte-Carlo studies in the early 90s:

- Hagle & Mitchell 1992

- Veall & Zimmermann 1992, 1993, 1994

- Windmeijer 1995

G

They have systematically tested the most

common Pseudo-R² for binary and ordinal

probit / logit models

(7)

How does a Monte-Carlo simulation study

work?

G 1. You have to generate 100 data sets with a

multinormal density and a prescribed multiple correlation. Then you have to

manipulate systematically the sample size and the multiple correlation.

G 2. You have to estimate the multiple linear

regression models for each cell of your design.

G 3. You have to dichotomize your dependent

variable using systematically different cutpoints.

(8)

G 4. You have to estimate your binary probit or logit models.

G 5. You have to calculate your Pseudo-R²s for

each probit or logit model.

G 6. You have to compare the Pseudo-R²s of the

probit / logit model with the “true R²” of the linear regression model.

(9)

Which Pseudo-R²s have been tested?

G Likelihood-based measures:

- Maddala / Cox-Snell Pseudo-R²

- Cragg-Uhler / Nagelkerke Pseudo-R²

G Log-Likelihood-based measures:

- McFadden Pseudo-R²

- Aldrich-Nelson Pseudo-R²

- Aldrich-Nelson Pseudo-R² with the Veall-Zimmermann correction

G Basing on the estimated probabilities:

- Efron / Lave Pseudo-R²

G Basing on the variance decomposition of the

estimated Probits / Logits:

(10)

Maddala Pseudo R 2 1 L0 L_A 2 n Theoretical range: 0 Maddala Pseudo R 2 1 ( L_A ) 2 n Notation :

L₀: Likelihood of the zero model

(with a regression constant only)

L_A: Likelihood of the alternative model

n: Sample size

(11)

Cragg Uhler Pseudo R 2 LA 2 n _L 0 2 n 1 L₀ 2 n Theoretical range:

0 Cragg Uhler Pseudo R 2 1

Standardization of the Maddala Pseudo R 2

(12)

McFadden Pseudo R 2 1 ln LA ln L₀

Theoretical range:

0 McFadden Pseudo R 2 1

It does not reach its maximum of 1 !

Rule of Thumb: 0.20 McFadden Pseudo R 2 0.40

The model has an excellent fit ! (McFadden 1979, 307) Notation :

ln L_A: Log Likelihood of the alternative model

ln L₀: Log Likelihood of the zero model

(13)

For Probits:

Aldrich Nelson Pseudo R2 LR

2 (M_A) LR 2_(M A) n_(M A) 2(lnLA lnL0) 2(lnL_A lnL₀) n_(M A) For Logits:

Aldrich Nelson Pseudo R2 LR

2 (M_A) LR 2_(M A) (n(MA)3.29) 2(lnLA lnL0) 2(lnL_A lnL₀) (n_(M A)3.29 )

(14)

T heo re tica l rang e :

Probits : 0 Aldrich N elson Pseu do R 2 2 lnL0 n 2 lnL₀ Lo gits : 0 Aldrich N elson Pseu do R 2 2 lnL0

3 .29 n 2 lnL₀ T he A N Pseu do R 2 de pen ds on the nu m ber an d the

prop ortio ns of the cate gories of the de pen dent variables! You can ea sily see this at the form u la of the log like liho od of the zero m o del:

ln L₀ K k1 n_k ln nk n N otation :

n_k: N umbe r of cases of cate gory k of the de pen dent variable

n: Sam ple size

(15)

Veall Zimmermann correction of the Aldrich Nelson Pseudo R2:

It is standardized by its own reachable maximum depending on k categories and their proportions range [0 ;1 ]

For Probits:

Aldrich Nelson Pseudo R2_V&Z

2(lnL_A lnL₀) 2(lnL_A lnL₀) n

2lnL₀ n 2lnL₀ For Logits :

Aldrich Nelson Pseudo R2_V&Z

2(lnL_A lnL₀)

2(lnL_A lnL₀) 3.29n

2lnL₀

(16)

La ve / Efron Ps eudo R 2 1 n i1 ( Y_i pˆ_i)2 n i1 ( Y_i Y )¯ 2 , w ith Y¯ 1 n n i1 Y_i N otation:

Y_i: Value of the de pe ndent v ariab le in c as e i

ˆ

p _i: Es tim ate d pro bab ility for Y1 in c as e i ¯

Y: O bserved pro portio n of c ate go ry "1 " of the de pe ndent v ariab le

(17)

McKelvey Zavoina Pseudo R2 SSRegression SS_Errors SS_Regression (For PROBITs) n i1 ( ˆy yˆ)2 (n1.0) n i1 ( ˆy yˆ)2

G Basing on the variance decomposition of the estimated Probits / Logits

(18)

McKelvey Zavoina Pseudo R2 SSRegression SS_Errors SS_Regression (For LOGITs) n i1 ( ˆy_i yˆ)2 (n3.29) n i1 ( ˆy_i yˆ)2 N o t a t i o n : ˆ y _i : E s t i m a t e d L o g i t / P r o b i t o f c a s e i ( t h e l i n e a r p r e d i c t o r ) 1 . 0 : 2 o f t h e n o r m a l P D F 3 . 2 9 : 2 o f t h e l o g i s t i c P D F Theoretical range: [ 0 ; 1]

(19)

Results of the Monte-Carlo-Studies

Corresponding to binary and ordinal probits / logits:

G The McKelvey-Zavoina Pseudo-R² is the best

estimator for the “true R²s” of the OLS regression.

G The Aldrich-Nelson Pseudo-R² with the

Veall-Zimmermann correction is the best

approximation of the McKelvey-Zavoina Pseudo-R².

G Lave / Efron, Aldrich-Nelson, McFadden and

Cragg-Uhler Pseudo-R² severely underestimate the “true R²”.

(20)

(1) ln P2i P_1i ß021 K k1 ß_k 21xki 21 (2) ln P3i P_1i ß031 K k1 ß_k 31xki 31

Assumptions for the error terms _k1:

1. ₂₁ and ₃₁ are stochastically independent.

2. ₂₁ and ₃₁ are identically distributed.

(I I D Assumption : Hensher &Johnson 1981, 154)

Estimation equations of a multinomial logit model with 3 alternatives:

3. The generalization of the McKelvey&Zavoina Pseudo-R²s for multinomial logits

(21)

What is really a multinomial logit (MNL)?

G It is a nonlinear multi-equation model.

G The error terms ₂₁ and ₃₁ are independent and

identically distributed.

G Therefore you can separately calculate the

McKelvey-Zavoina Pseudo-R² for each equation / comparison of alternatives !

G Resulting options:

- .1 Simultaneous estimation with the MNL - .2 Separate estimation of binary logistic

regression models for each comparison (Begg & Gray 1984)

(22)

4.An application in an election study of East German students (Sachsen-Anhalt 1998)

Social Democratic Party (SPD)

Christian Democratic Union (CDU)

Party of Democratic Socialism (PDS, Post-Communist Party)

Free Democratic Party (FDP)

The Green Party / Alliance 90 (B90)

Right extremist parties (DVU, REP, NPD)

Dependent variable:

Which party do you prefer to vote for ? Alternatives: (one vote only)

(23)

• AGE in years

• GENDER: boys vs. girls

• SCHOOL TYPE: Secondary school, GRAMMAR school, VOCATIONAL school

• Internal and external political EFFICACY

• Perceived influence of the peers on the vote (PEERS)

• Perceived influence of the parents on the vote (PARENTS)

• Perceived influence of the media on the vote (MEDIA)

• Perceived influence of the teachers on the vote (TEACHERS)

• Countryside vs. City (LOCATION)

(24)

SPD 46,9% CDU 19,5% PDS 12,6% FDP 3,1% B90 6,9% DVU/REP/NPD 11,1% ( n = 1894 )

(25)

Table 1: Party preference of pupils in Sachsen-Anhalt 1998. Multinomial Logit Model (LIMDEP 7 NT) Independent variables: Comparison: CDU vs. SPD Comparison: PDS vs. SPD Comparison: FDP vs. SPD Comparison: B90 vs. SPD Comparison: DVU,REP,NPD vs. SPD

Logit S.E. Logit S.E. Logit S.E. Logit S.E. Logit S.E.

Constant - 0,577 0,605 - 3,922** 0,755 - 6,387** 1,436 +0,265 0,906 - 6,562** 0,871 AGE - 0,043 0,035 +0,118** 0,042 +0,178* 0,077 - 0,052 0,053 +0,206** 0,047 GENDER +0,511** 0,131 +0,383* 0,154 +0,520 0,285 - 0,0002 0,200 +1,275** 0,188 GRAMMER VOCATIONAL +0,870** +0,756** 0,210 0,296 +0,932** +0,166 0,254 0,356 +0,898 - 0,214 0,467 0,655 +1,082** - 0,388 0,292 0,546 - 0,628 - 0,327 0,346 0,371 EFFICACY - 0,011 0,021 +0,049 0,025 +0,087 0,048 - 0,084** 0,032 +0,109** 0,029 PEERS - 0,031 0,084 +0,024 0,100 +0,060 0,183 +0,062 0,131 +0,838** 0,097 PARENTS +0,026 0,070 +0,062 0,082 - 0,034 0,154 - 0,164 0,111 - 0,488** 0,102 MEDIA - 0,146* 0,066 - 0,117 0,079 - 0,247 0,143 - 0,300** 0,102 - 0,219* 0,086 TEACHERS - 0,072 0,084 - 0,302** 0,108 - 0,226 0,201 - 0,063 0,137 - 0,032 0,108 LOCATION +0,296 0,187 +0,359 0,236 +0,232 0,447 -0,616* 0,304 +0,699** 0,246

(26)

Tab le 2: Co mp aris o n: C DU vs. SPD Co mp aris on : PD S vs . SPD Co mp aris on : FD P vs . SPD C o mp ariso n: B 9 0 v s. SP D Co mp aris on : D VU,R EP,N PD vs. SPD Ov erall M cKelvey& Zavo in a Pseudo R ² 5 ,64 2 % 10 ,0 48 % 1 5,0 96 % 1 8, 77 9 % 34 ,99 8 % In d ep en dent v ariab les û-M& Z Ps eu do R ² in % û-M & Z Pseu do R² in % û-M & Z Pseu do R ² in % û-M& Z Pseudo R ² in % û-M & Z Pseud o R² in % A GE 0,2 26 3 ,4 60 5,0 82 1, 47 3 2 ,91 3 GENDER 2,0 53 0 ,5 95 0,8 94 0, 05 2 8 ,70 9 S CH OOL TYP E 2,0 71 2 ,2 14 2,6 61 1 1, 08 4 8 ,10 5 EFF ICA C Y 0,0 41 0 ,6 71 2,2 05 1, 69 9 2 ,07 7 P EER S 0,0 78 0 ,0 41 0,0 03 0, 02 7 7 ,46 6 PA RTENTS 0,0 26 0 ,0 00 0,2 77 0, 62 4 3 ,85 0 M EDIA 0,7 01 0 ,7 06 1,8 44 2, 11 0 0 ,77 3 TEA C HER S 0,0 70 1 ,4 70 0,7 74 0, 04 8 0 ,09 3 LOCA TI ON 0,3 08 0 ,4 46 0,1 66 1, 50 9 0 ,68 0 M od el f it:

Likelih o od- Ratio ²-Test = 4 52 ,29 F. G.= 5 0 . = 0 ,00 n = 1 89 4 ln L M0 = - 27 82, 79 ln L MA = -25 56 ,64

M cFad den -Pseud o R ² = 8 ,1 27 %

(27)

0 5 10 15 20 25 30 35 D e l t a -M c K e l v e y & Z a v o i n a P s e u d o -R 2

CDU PDS FDP B90 DVU / REP / NPD

Vote for party in comparison with alternative SPD

Party preferences of pupils in Sachsen-Anhalt 1998: Overall fits and relative effect sizes of the

independent variables LOCATION TEACHERS MEDIA PARENTS PEERS EFFICACY SCHOOLTYPE GENDER AGE

(28)

(29)

Comparision the overall fit of the multinomial and binary logit models by their McKelvey-Zavoina Pseudo-R²s

y = 1,117x - 1,229 R2 = 0,995 5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 Multinomial Logit B ina ry Lo g it

(30)

Comparison of the partial McKelvey-Zavoina Pseudo-R²s of the

multinomial and the binary logit models _{y = 1,025x + 0,037} R2 = 0,979 0 2 4 6 8 10 12 0 1 2 3 4 5 6 7 8 9 10 11 12 Multinomial Logits Bi nar y Logi ts

(31)

5.What’s to do in future methodological studies?

G Researchers have to present the LR-X², the

log-likelihoods of their alternative and zero binary or ordinal logit models and the sample size.

G ==> You can easily calculate the corrected

Aldrich-Nelson Pseudo-R² with these scalars. Then you can realistically assess the fit of their binary or ordinal logit / probit model.

G Comparing different models you have to adjust

McKelvey-Zavoina Pseudo-R²s for the sample size and the number of estimated parameters without the constant (under investigation).

G Bootstrap studies might be helpful to analyse the

distribution of the M-Z Pseudo-R² and to construct its confidence intervals (under investigation).

(32)

References:

GAldrich, J.H.& Nelson, F.D. (1984):

Linear probability, logit, and probit models. Newbury Park: SAGE (Quantitative Applications in the Social Sciences, 45)

GAmemiya, T. (1981):

Qualitative response models: a survey. Journal of Economic Literature, 21, pp.1483-1536

GBegg, C.B. & Gray, R. (1984):

Calculation of polychotomous logistic regression parameters using individualized regression. Biometrika, 71, pp.11-18

GCragg, S.G.& Uhler, R. (1970):

The demand for automobiles. Canadian Journal of Economics, 3, pp. 386-406

GEfron, B. (1978):

Regression and Anova with zero-one data. Measures of residual variation. Journal of American Statistical Association, 73, pp. 113-121

GHagle, T.M. & Mitchell II,G.E. (1992):

Goodness of fit measures for probit and Logit. American Journal of Political

Science, 36, 3, pp. 762-784

GHensher, D.A.& Johnson, L.W. (1981):

Applied discrete choice modelling. London: Croom Helm

GMaddala, G.S. (1983):

Limited-dependent and qualitative variables in econometrics. Cambridge: Cambridge University Press

(33)

GMcFadden, D. (1979):

Quantitative methods for analysing travel behaviour of individuals: some recent developments. In: Hensher, D.A.& Stopher, P.R.: (eds):Behavioural travel

modelling. London: Croom Helm, pp. 279-318

GMcKelvey, R. & Zavoina, W. (1975):

A statistical model for the analysis of ordinal level dependent variables. Journal of Mathematical Sociology, 4, pp. 103-20

GNagelkerke, N.J.D. (1991):

A note on a general definition of the coefficient of determination. Biometrika, 78, 3, pp.691-693

GRonning, G. (1991):

Mikro-Ökonometrie. Berlin: Springer

GVeall, M.R. & Zimmermann, K.F. (1992):

Pseudo-R2 in the ordinal probit model. Journal of Mathematical Sociology, 16, 4, pp. 333-342

G Veall, M.R. & Zimmermann, K.F. (1994):

Evaluating Pseudo-R2's for binary probit models. Quality&Quantity, 28, pp. 151- 164

GWindmeijer, F.A.G. (1995):

Goodness-of-fit measures in binary choice models. Econometric Reviews, 14, 1, pp. 101-116

GZimmermann, K.F. (1993):

Goodness of fit in qualitative choice models: review and evaluation. In: Schneeweiß, H. & Zimmermann, K. (eds):

The Assessment of Fit in the Class of Logistic Regression Models: A Pathway out of the Jungle of Pseudo-R²s

Full text