Stephen P. Klein. Description of the LSAT. Predicting First Year GPA (FYGPA) in Law School

(1)

Law School Admi ions,

LSATs, and the Bar

Stephen P. Klein

N

early 200 accredited law schools in the U n i t e d States a n d Canada require that their applicants take the Law School Admission Test (LSAT). Most o f these schools place a great deal o f weight o n LSAT scores in the admissions process. However, c o n c e r n s have b e e n raised a b o u t this practice because minority students (and particularly African A m e r i c a n applicants h e r e i n a f t e r re- f e r r e d to as blacks) g e n e r a l l y e a r n m u c h lower LSAT scores t h a n t h e i r non-Hispanic white classmates.~ T h e r e also are g e n d e r a n d ethnic differences. Because o f these disparities a n d a desire to have a culturally diverse s t u d e n t body, m a n y law schools have e n d e a v o r e d to increase minority e n r o l l m e n t s by e m p l o y i n g various affirmative action policies.

This p a p e r describes certain salient features o f the LSAT a n d reviews the empirical evidence r e g a r d i n g the appropriateness o f using this test in the admissions process. It also examines w h e t h e r LSAT scores s h o u l d be i n t e r p r e t e d differently for different groups. I c o n c l u d e with some r e c o m m e n d a t i o n s re- g a r d i n g law school admissions a n d r e t e n t i o n policies.

Description of the LSAT

T h e LSAT is developed, administered, a n d scored by the Law School Ad- mission Council (LSAC), a private, not-for-profit organization. It is taken by over 100,000 examinees annually. T h e LSAT measures logical reasoning, read- ing c o m p r e h e n s i o n , a n d analytic reasoning. A l t h o u g h the exam has five 35- m i n u t e sections, only f o u r o f t h e m are used to c o m p u t e an applicant's score. Various versions o f the fifth section are used for pre-testing new questions. Consequently, a candidate's score is based o n just the 101 multiple choice items f r o m the f o u r operational sections. T h e r e also is a 30-minute essay ques- tion, b u t the candidate's answer is n o t scored by LSAC a n d thus does n o t figure into the calculation o f an applicant's LSAT score. LSAT scores are re- p o r t e d o n a 120 to 180 p o i n t scale ( m e a n o f 150, standard deviation o f 10). T h e scores o n this scale are adjusted for possible differences in average item difficulty across administrations o f the exam.

Predicting First Year GPA (FYGPA) in Law School

T h e primary p u r p o s e o f the LSAT is to predict an applicant's first year grades in law school. T h e LSAT is n o t d e s i g n e d to p r e d i c t w h o will be a g o o d (or Stephen E Klein is senior research scientist at the RAND Corporation in Santa Monica, California. Please address c o r r e s p o n d e n c e to Academic Questions / NAS, 221 Witherspoon Street, Second Floor, Princeton, NJ 08542-3215; [email protected].

(2)

otherwise "successful") lawyer or even who will graduate from law school. Pre- dictive accuracy is generally m e a s u r e d in terms of the "correlation" between LSAT a n d first year law grade point average. Correlation coefficients are re- p o r t e d on a -1.00 to 1.00 scale. A value o f 0.00 indicates there is no relationship between LSAT a n d FYGPA whereas a 1.00 indicates there is a perfect relationship between them; i.e., the rank ordering of students on the LSAT would correspond exactly with their rank ordering on FYGPA. A negative correlation would indicate an inverse relationship between these measures.

A r e c e n t study of 167 law schools f o u n d m e d i a n validity coefficients in predicting FYGPA o f about .25 for U n d e r g r a d u a t e Grade Point Average (UGPA), .40 for LSAT scores, a n d almost .50 for the combination o f UGPA a n d LSAT. 2 Thus, although both measures should be used to maximize the prediction of FYGPA, generally m o r e weight should be given to the LSAT.

T h e validity coefficients above u n d e r e s t i m a t e the ability of UGPA and LSAT to predict FYGPAs. This occurs because o f restrictions in the range of values on these measures at a given law school. Specifically, the students who apply to and attend a particular law school are likely to be m o r e h o m o g e n e o u s with respect to their admissions credentials (as reflected by the standard deviation o f their LSAT scores) than is the total pool o f all bar applicants nationally. Similarly, the applicants to a school who have relatively low UGPAs and LSATs are less likely to be admitted to that school a n d therefore do not have a FYGPA. These a n d related factors bias the validity coefficients downward.

T h e candidates who have high LSAT scores also tend to have high UGPAs. This is true nationally and in the applicant pool for most law schools. How- ever, the correlation between these measures is close to zero a n d often negative a m o n g the candidates who attend a given school. This occurs because most schools use a compensatory admissions policy, i.e., they allow a high LSAT score to offset a low UGPA and vice versa.

Test Bias

Simple differences in average scores between groups are n o t e n o u g h to establish that a test is biased because these differences may just reflect differences in ability between the groups. In o r d e r for bias to be present, there must be some form of over or u n d e r prediction. U n d e r prediction would oc- cur o n the LSAT if a group did better in law school than would be e x p e c t e d given that group's average LSAT score. If that h a p p e n e d , the LSAT would be considered to be biased against the group.

(3)

Klein 35

they d o slightly less well in law school than would be e x p e c t e d o n the basis o f the c o m b i n a t i o n o f their UGPAs a n d LSAT scores.

UGPA, a n d to a lesser degree LSAT, favors minority candidates. T h e m e a n FYGPA for blacks is a b o u t a half o f a standard deviation lower than what would be e x p e c t e d o n the basis o f their UGPAs a n d a quarter o f a standard deviation lower t h a n what would be e x p e c t e d o n the basis o f their LSATs. Thus, a l t h o u g h blacks e a r n lower UGPAs a n d LSAT scores t h a n their classmates, these scores still yield an inflated estimate o f their chances o f e a r n i n g g o o d grades in law school. T h e c o m b i n a t i o n o f UGPA a n d LSAT provides a m o r e accurate (less biased) prediction, b u t it still favors black candidates.

T h e r e is a similar p a t t e r n for Hispanic a n d Asian students, b u t with consid- erably less over-prediction. 4 In short, if there is any bias in the LSAT, it is in the direction of favoring minority applicants by over predicting their law school grades. Affirmative Action's F o o t p r i n t

Many if n o t most law schools engage in a variety o f activities a n d practices that are i n t e n d e d to increase minority s t u d e n t e n r o l l m e n t a n d thereby achieve a m o r e diverse s t u d e n t body. In some instances, this involves formally or infor- mally setting different admissions standards for different groups. Either way, such practices leave b e h i n d a tell-tale statistical f o o t p r i n t in the form o f a statistical relationship between a candidate's admission scores a n d probability o f b e i n g accepted.

Figure 1 illustrates this p a t t e r n using a few years' worth o f data from a large a n d highly r e g a r d e d public law school. T h e horizontal (x-axis) shows the "in- dex" score. This score is derived f r o m the weighted c o m b i n a t i o n o f LSAT scores a n d UGPAs that optimizes the p r e d i c t i o n o f first year grades at the

Figure 1

Statistical F o o t p r i n t o f Affirmative Action

Probability of

adnllaaioln

1.0 0.8 0.6 0.4 0.2

//

_-

..-,, ~ ;]...1!~#

" I

"" I"

"1'

s

"1 " " I

70

75

80

85 ill

I/:" I I

!

I.!

(4)

school. T h e vertical (y-axis) shows a n applicant's probability o f b e i n g admit- t e d to the school. T h e line for e a c h r a c i a l / e t h n i c g r o u p shows t h e relationship b e t w e e n that g r o u p ' s i n d e x score a n d probability o f b e i n g a c c e p t e d .

For e x a m p l e , t h e left m o s t line is for black applicants. This line shows that blacks with i n d e x scores below 79 have almost n o c h a n c e o f b e i n g a d m i t t e d w h e r e a s all o f those with i n d e x scores o f 80 o r h i g h e r were a c c e p t e d . T h e curve for white applicants parallels the o n e for blacks, b u t it is shifted m u c h f u r t h e r to t h e right. Specifically, whites with i n d e x scores below 85 h a d almost n o c h a n c e o f b e i n g a d m i t t e d w h e r e a s those with i n d e x scores over 88 were virtually g u a r a n t e e d admission. It is e v i d e n t f r o m these data that i n d e x scores drive admission decisions, b u t the school u s e d substantially d i f f e r e n t c u t o f f scores for d i f f e r e n t groups. T h e flatter lines for Hispanics a n d especially for Native A m e r i c a n s m a y stem f r o m the g r a p h b e i n g based o n

self-reported

r a c i a l / e t h n i c g r o u p w h e r e a s t h e school's actual admissions decisions w e r e b a s e d o n

verified

r a c i a l / e t h n i c group.

Prediction o f Bar E x a m Scores

A l t h o u g h t h e p u r p o s e o f the LSAT is to p r e d i c t FYGPA, it is instructive to see h o w well LSAT scores correlate with b a r e x a m results. Table 1 uses data f r o m a typical California law school to illustrate this relationship. This table shows that o f t h e 20 p e r c e n t o f the s t u d e n t s with t h e h i g h e s t LSAT scores at this school, 81 p e r c e n t passed t h e b a r e x a m o n t h e i r first try. In contrast, o f t h e 20 p e r c e n t with t h e lowest LSAT scores, only 61 p e r c e n t passed. Thus, h i g h e r LSAT scores were generally associated with h i g h e r bar passage rates. However, Law School G r a d e P o i n t Average (LGPA) was a b o u t a t h r e e times m o r e a c c u r a t e forecaster o f w h o was g o i n g to pass t h a n was LSAT. A l t h o u g h t h e same p a t t e r n is r e p e a t e d at m o s t schools, the p e r c e n t passing at a given quintile at a particular school will vary as a f u n c t i o n o f that school's admissions standards a n d o t h e r factors. In c o n t r a s t to these trends, UGPA has only a very weak relationship with bar e x a m passing rates.

Table 1

P e r c e n t Passing at O n e Law S c h o o l as a F u n c t i o n o f Quintile o f LSAT Score a n d LGPA at that S c h o o l

Quintile LSAT LGPA

Highest 20% 81 97

Next highest 20% 71 89

Middle 20% 63 63

Next lowest 20% 65 53

Lowest 20% 61 40

(5)

Klein 37

T h e r e is a n e x t r e m e l y s t r o n g relationship b e t w e e n a school's m e a n LSAT score a n d the p e r c e n t a g e o f its g r a d u a t e s w h o pass the b a r e x a m o n t h e i r first try ( c o r r e l a t i o n is over .90). Figure 2 illustrates this r e l a t i o n s h i p with d a t a f r o m t h e J u l y 2000 California b a r exam. E a c h d o t o n this c h a r t c o r r e s p o n d s to a law s c h o o l with 25 o r m o r e first t i m e takers (i.e., i n c l u d i n g out-of-state schools). This c h a r t shows that a school's first timer bar passage rate can be predicted with a very high d e g r e e o f accuracy f r o m its m e a n LSAT score. How- ever, there are exceptions, such as the school with a m e a n LSAT o f 153, but only a 20 p e r c e n t passing rate (which is about half the rate t h a t w o u l d b e e x p e c t e d given t h e usual r e l a t i o n s h i p b e t w e e n m e a n LSAT score a n d passing rate). S u m m a r y

T h e LSAT is a generally m o r e a c c u r a t e p r e d i c t o r o f a student's first y e a r grades in law school t h a n is UGPA. However, t h e c o m b i n a t i o n o f LSAT a n d UGPA usually provides a m o r e a c c u r a t e forecast t h a n does e i t h e r o n e o f these variables alone. A l t h o u g h m i n o r i t y c a n d i d a t e s t e n d to e a r n lower ( a n d some- times substantially lower) UGPAs a n d LSATs t h a n t h e i r n o n - m i n o r i t y classmates, b o t h o f these m e a s u r e s (but especially UGPAs) t e n d to over p r e d i c t t h e FYGPAs o f m i n o r i t y candidates. In this sense, b o t h m e a s u r e s provide an inflated estimate o f a m i n o r i t y c a n d i d a t e ' s likelihood o f e a r n i n g g o o d grades in law school.

T h e r e is a m o d e r a t e relationship b e t w e e n LSAT a n d b a r e x a m scores even t h o u g h t h e LSAT was n o t d e s i g n e d to p r e d i c t such scores. A l t h o u g h this relationship is m u c h s t r o n g e r t h a n the o n e b e t w e e n UGPA a n d b a r scores, it is

Figure 2 S c h o o l ' s Passing Rate P r e d i c t e d by L S A T 100 Percent passing 8 0 R 6 0 - - 4 0 - - 2 0 - -

t.

|

9 1 I I I 140,0 142.5 145.0 147.5 150.0 I I I I I I I 152.5 1,55.0 157.5 160.0 162.5 165.5 167.5 170.0

(6)

still substantially less than the relationship between LGPA and bar scores. However, when data are aggregated to the school level, there is an extremely strong correlation between a school's m e a n LSAT score a n d the p e r c e n t a g e o f its graduates who pass the bar exam on their first try.

Conclusions and Implications

Given the findings above, law schools should c o n t i n u e to use the LSAT in conjunction with UGPA in their admissions process. W h e n used together, they provide a reasonably accurate forecast o f a candidate's law school grades. Admission officers also should tend to give the LSAT m o r e weight than UGPA in this process. If schools engage in affirmative action programs, they will n e e d to justify t h e m on the benefits derived from having a diverse s t u d e n t body because the empirical evidence clearly contradicts any claims of the LSAT or UGPA being biased against minority groups.

T h e fact that bar exam scores are m u c h m o r e highly related to LGPAs than they are to LSATs suggests that some schools may want to modify their admission and academic policies. Specifically, schools with relatively low bar passage rates could address this p r o b l e m by doing two things. First, they should o p e n their doors wider at time of admission to increase the size o f their first year classes. The only m o d e s t accuracy of LSAT and UGPA in predicting grades shows that there are m a n y candidates who could be successful in law school despite having relatively low admission scores. Second, these schools should use first year grades to place a m u c h larger percentage of their students on academic probation. In this way, the total n u m b e r of graduates will be about the same, but they will have a h i g h e r probability of passing. T h e second of these two r e c o m m e n d a t i o n s will certainly n o t be popular with students, fac- ulty, or even ABA accreditation teams. Nevertheless, the schools that have a d o p t e d this and related policies have f o u n d that they very quickly contribute to a substantial i m p r o v e m e n t in bar passage rates. This is not a policy I would r e c o m m e n d for all schools, but just for the ones that are c o n c e r n e d about their bar passage rates. It the long run, it is fairer to applicants.

Notes

1. Stephen Klein and Roger Bolus, "The size and source of differences in bar exam passing rates among racial and ethnic groups," The Bar Examineg, 66, No. 4 (1997): 8-16; and

Linda E Wightman, LSAC National Longitudinal Bar Passage Study (Newtown, PA: Law

School Admission Council, 1998).

2. Lisa C. Anthony, Vincent E Harris, and Peter J. Pashley, Predictive validity of the LSAT." A national summary of the 1995-1996 correlation studies--LSA C Technical Report 97-01 (Newtown,

PA: Law School Admission Council, 1999).

3. Jennifer R. Duffy and Louis A. Roussos, Analysis of differential prediction of law school performance by gender subgroups: 1996-1998 entering law school classes--LSA C Technical Report

TRO0-03 (Newtown, PA: Law School Admission Council, 2000).

4. Lisa C. Anthony and Mei Liu, Analysis of differential prediction of law school performance by ethnic subgroups: 1996-1998 entering law school classes--LSAC Technical Report TRO0-02