Defining and Measuring College Success
Wayne Camara Krista Mattern
Andrew Wiley The College Board
Research and Development February 24, 2009
Remediation and Post Secondary Success
• 41.4% of post secondary students take at least one remedial class.
This percent is even higher among African Americans (61.7%) and Hispanics (63.2%).
• For all post secondary students taking remedial classes, the
graduation (or completion for certificate students) rates were between 30% and 57%, depending on the type and number of remediation
courses taken. For those not taking remedial classes, the graduation rate was 69%
• For Bachelors candidates taking remedial classes, between 17% and 39% of students graduated as compared to 58% for those not taking remedial classes.
College Readiness Index
• Long term goal: To provide states, districts and schools with an aggregate data that estimates the percent of
students who are college ready, or on track to be college ready, upon graduation from high school.
• It is not intended to make high stakes decisions at the student level. It will not be used for admissions or
placement decisions.
• To provide diagnostic information that will identify steps necessary to increase the percentage of students ready for college.
The College Readiness Index will have 3 components
HS Performance (GPA)
Academic rigor index
National Benchmark
Index of
college readiness
Creation of the College Readiness Index
• SAT Validity
– SAT was revised in March of 2005, so the first class entering college with the new SAT is the entering class of 2006.
– Tracked data from 100 colleges and approximately 150,000 students.
– Released data showing the predictive and differential validity results
– Upcoming research investigates the validity of the SAT for predicting individual course grades and retention to 2nd year in college.
SAT Validity Study — Correlations for Predictors (N=151,316)
Predictors R*
HSGPA 0.54
SAT-CR + M + W 0.53
SAT-CR 0.48
SAT-M 0.47
SAT-W 0.51
SAT-CR + M 0.51
SAT Validity Study — Incremental Validity (N=151,316)
Predictors R1 R2 ΔR
HSGPA (Add SAT-CR + SAT-M) 0.54 0.61 0.07
HSGPA (Add SAT-CR + SAT-M + SAT-W) 0.54 0.62 0.08
SAT-CR + M (Add SAT-W) 0.51 0.53 0.02
HSGPA + SAT-CR + SAT-M (add SAT-W) 0.61 0.62 0.01
* Correlations corrected for restriction of range, pooled within-institution
SAT Validity Study — Incremental Validity (N=151,316)
N SAT HSGPA SAT+HSGPA
CONTROL
Private 45,786 0.57 0.55 0.65
Public 105,530 0.52 0.53 0.61
SELECTIVE
Under 50% 27,272 0.58 0.55 0.65
50-75% 84,433 0.53 0.54 0.62
>75% 39,611 0.51 0.54 0.60
Correlation of SAT scores and HSGPA with FYGPA by Ethnicity
Subgroup
Race/Ethnicity American
Indian Asian
African-
American Hispanic White
k 16 82 83 86 109
n 384 14,109 10,096 10,486 104,017
SAT-CR 0.41 0.41 0.40 0.43 0.48
SAT-M 0.41 0.43 0.40 0.41 0.46
SAT-W 0.42 0.44 0.43 0.46 0.51
SAT 0.54 0.48 0.47 0.50 0.53
HSGPA 0.49 0.47 0.44 0.46 0.56
SAT, HSGPA 0.63 0.56 0.54 0.57 0.63
Average Overprediction (-) and Underprediction (+) of FYGPA for SAT Scores and HSGPA by Ethnicity
Subgroup
Race/Ethnicity American
Indian Asian
African-
American Hispanic White
k 103 109 108 110 110
n 798 14,296 10,304 10,659 104,024
SAT-CR -0.26 0.05 -0.30 -0.17 0.04
SAT-M -0.25 -0.07 -0.26 -0.16 0.05
SAT-W -0.22 0.04 -0.26 -0.16 0.04
SAT -0.22 0.01 -0.20 -0.11 0.03
HSGPA -0.25 0.02 -0.32 -0.27 0.06
Creation of the College Readiness Index
• Data has been collected from a sample of approximately 109 schools for the 2007 entering class.
• During registration, SAT students provide extensive high school coursework information as well as HSGPA.
• College data includes grade point average, course taking and retention. Students will be tracked until graduation.
• College performance data will be used to identify benchmarks linked to college success.
• Students will be considered college ready if they meet or exceed benchmarks in all three indicators.
• Future cohorts will also be used to validate the benchmarks.
College Readiness Reports
• Data on the percent of students deemed College Ready disaggregated by key demographic variables.
• Comparison of college readiness to a peer group (i.e.
compare state to nation).
• Data on the percent of students attaining college readiness according to each indicator.
• Information on where students behavior could be altered to move more students towards college readiness.
Hypothetical Chart Showing Longitudinal data on College Readiness
5 y e ar re port on colle g e re adin e s s for a School Dis trict
17.5 18.3 20.8 21.7
33.3 36.7 40.0 41.7 45.0
50.0 45.8
41.7 37.5
33.3
16.7
0 10 20 30 40 50 60
2009 2010 2011 2012 2013
Colle g e Re ady Plus Colle g e Re ady Not Colle g e Re ady
College Readiness Reports
• Reports will be linked to other sources of
information which will provide feedback on steps that can be followed to move more students
towards college readiness.
• SAT Skills Insight
– Provides descriptive information on the knowledge,
skills and abilities for students across SAT score bands.
– Developed by SMEs based upon a systematic review of items and student performance on each item.
Teacher Use Examples Writing Section
SAT Skills Insight
Writing Section
WRITING – Vague Pronouns
Although students use pronouns in their speaking and writing, some students may not be able to easily identify a pronoun or what is meant by a ―vague pronoun.‖
This skill is tested in the SAT® writing section. Students who score in the 300-390 band generally have this skill.
This sample SAT question, which is an Identifying
Recognizing vague pronouns is also described in the 400-490 score band.
The example illustrating this skill at this level is an Improving Sentences type: the student must find the correction rather than just spot the error.
Related skills involving pronouns appear in score band 500-590 (―Recognize the
antecedent of a pronoun despite multiple distracters that change the number and/or the subject of the referent within the
sentence) and score band 200-290
(―Recognize simple pronoun references‖).
Look to the Suggestions for Improvement for ideas for reinforcement exercises - e.g., ask students to underline the pronouns and antecedents in the next writing assignment).
Academic Rigor Index
Academic Rigor Index (0-25)
Math (0-5)
English (0-5)
Science (0-5)
Social Science
(0-5)
Foreign Language
(0-5)
Proposed Academic Intensity Index (AII)
• Empirically based – awards points for either coursework, honors, or AP.
• The Academic Intensity Index will have 5 sections:
– English, Math, Science, Social Science/History, and Foreign/Classical Language
• A preliminary version of an Academic Intensity Index has been created:
– College level data will be will be used to choose and validate a scale
– Thus far, preliminary scales have been created and tested using SAT scores as outcome data
AII – Proposed Math scale
• Identify courses, by grade, that are associated with college success
• Each year is reviewed and the student is assigned a value of 1 if they have taken an appropriate class during that year.
• Class level values are summed to reach a scale of 0 to 5.
• If a student has taken Calculus, they are assigned a 5 automatically
• Some examples:
Grade Level 8th 9th 10th 11th 12th
Algebra 2 1 1 1 0 0
Geometry 1 1 0 0 0
Hypothetical Report for a School District:
Student Summary for Math Academic Intensity
Percent
identified as:
Asian students
Black students
Hispanic students
White students
College Ready 60% 42% 46% 55%
Algebra by 10th grade 85% 67% 69% 78%
Trigonometry by 11th grade 79% 59% 64% 74%
Pre-Calculus by 12th grade 74% 54% 59% 69%
1 Honors or AP class 45% 25% 30% 35%
2 or more Honors or AP class
22% 5% 9% 14%
AII – Proposed English scale
• A student is awarded 1 point for having taken 3 years (excluding courses taken concurrently) of English (0/1)
• A student is awarded between 0 and 4 points depending on honors and AP participation: (0/4)
– No honors, no AP = 0
– 1 honors or dual enrollment course, no AP = 1
– 2 or more honors or dual enrollment course, no AP = 2 – 1 honors or dual enrollment course, 1 AP = 3
– 2 or more honors or dual enrollment course, 1 AP = 4
AII – Proposed Science scale
• Coursework
– A student is awarded 1 point for having taken an 8th grade science class (0/1)
– A student is awarded 1 point for having taken biology, chemistry, and physics (0/1)
– A student is awarded 1 point for having taken 4 years of science (one or more science course in 9th, 10th, 11th, and 12th grade) (0/1)
• AP participation
– 1 point for having taken any science honors or dual enrollment course, Earth Science AP or Biology AP
– 2 points for having taken a Chemistry or Physics AP (0/2)
AII – Proposed Social Science scale
• A student is awarded 1 point for 3 or more years of Social Science (excluding courses taken concurrently) (0/1)
• A student is awarded 1 point for having 3 or more years of history (excluding courses taken concurrently) (0/1)
• Students are awarded points for honors, dual enrollment, and AP participation (0/3):
– 1 point for having 1 or more honors or dual enrollment classes but no AP
– 2 points for having 1 AP class
– 3 points for having 2 or more AP classes
AII – Proposed Foreign Language Scale
• Number of Years of Foreign Language (0/3)
– A student is awarded 1 point for having taken 2 years of language
– A student is awarded 2 points (0/1) for having taken 3 years of language
– A student is awarded 3 points for having taken 4 or more years of language)
• A student is awarded up to 2 points depending on his or her AP, honors, or dual enrollment
participation (0/2):
– 1 point for each class taken within language honors, dual
AII - Correlations Among the Sub-Scales
1 2 3 4 5
1. English --
2. Mathematics 0.38 --
3. Science 0.39 0.47 --
4. Social Science 0.56 0.39 0.43 --
5. Foreign/World 0.33 0.32 0.35 0.36 --
A preliminary version of AII
Mean Scores by ethnicity
Ethnicity Total Math English
Natural Science
Social Science
Foreign Language
Total group 12.5 2.7 2.3 2.5 2.8 2.2
American Indian 11.5 2.0 2.6 2.3 2.5 1.9
Asian, Asian-American,
Pacific Islander 13.4 2.9 2.7 2.8 2.8 2.2
Black or African-
American 10.7 1.9 2.4 2.2 2.4 1.9
Hispanic 11.6 2.1 2.6 2.3 2.6 2.1
White 13.0 2.4 2.8 2.6 2.9 2.3
A preliminary version of AII
Mean scores by family income
Income Total Math English Science
Social Science
Foreign Language
less than 30k 11.0 2.0 2.4 2.3 2.4 1.9
30k to 50k 11.8 2.1 2.6 2.4 2.6 2.0
50k to 70k 12.3 2.2 2.7 2.5 2.7 2.1
70k to 100k 12.7 2.4 2.8 2.6 2.8 2.2
A preliminary version of AII
Mean scores by family income and ethnicity Asian
students
Black students
Hispanic students
White students
< $30K 12.2 10.0 10.9 11.5
> $100K 14.5 11.9 12.9 13.5
Moving Forward for the CRI
• Data verification with future cohorts of students.
• Other definitions of college success.
• Link to PSAT/NMSQT
Noncognitive Assessments: Expanding
the definition of college readiness
Background
• Choosing students: Higher education admissions tools for the 21st century (Camara & Kimmel, 2005)
• Purpose:
– Identify additional predictors of college success
– Expand the definition of what constitutes successful performance in college beyond freshman GPA
• College Board has initiated several projects to address this research area
• Identify a broader domain of college student performance:
– Review university mission statements and department objectives
– Interview with university staff responsible for student life at Michigan State University – Review of the education literature on student
outcomes
• Our systematic search resulted in 12 dimensions of student performance…
Research collaboration with
Michigan State University
12 Dimensions of Student Performance
Broadening the Performance Domain in the Prediction of Academic Success (Schmitt, Oswald, & Gillespie, 2004)
1. Knowledge, learning, mastery of general principles 2. Continuous learning, intellectual interest and curiosity 3. Artistic and cultural appreciation
4. Multicultural appreciation 5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement 8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills 11. Perseverance
Two ―Noncognitive‖ Measures
• Situational judgment inventory
– A situation is presented along with several alternative courses of action.
– The respondent is asked to indicate what she/he would be most likely and least likely to do.
• Biodata
– Short, multiple choice reports of past
experience/background and interests/preferences.
Study 1: Psychometric adequacy
& scale refinement
• 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year.
• Identical empirical-keying procedures were conducted on both
instruments at the item level (double-cross validated using randomly split samples).
• Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing
measures of personality in predicting college GPA.
• The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students’ self ratings, and peer-ratings of
Study 1: Standardized Differences Compared with White group…
Non-cognitive Dimension Black Hispanic Asian
Knowledge -0.08 -0.20 -0.25
Learning 0.01 0 .63* -0.19
Artistic -0.19 0 .73* 0.15
Multicultural -0.11 0 .63* 0.02
Leadership -0.18 0.08 -0.30
Interpersonal -0.18 0.33 -0.38*
SJI composite -0.05 -0.14 -0.21
Citizenship 0.05 0.23 -0.14
Health -0.31* 0.06 -0.67*
Career 0 .34* 0 .56* 0.14
Adaptability 0.03 0.09 -0.41*
Perseverance 0.13 0 .55* -0.18
• Positive values indicate that minorities perform better than White students.
• The d values for biodata and SJI measures across ethnic and gender subgroups were
consistently smaller than those found on cognitive predictors.
• * p <.05
Study 2: Predicting FYGPA: Total Sample across
10 Institutions (N = 2443)
Predicting Self-Rated Performance:
Total Sample across 10 Institutions (N = 900)
Predicting Class Absenteeism: Total Sample across 10
Institutions (N = 899)
Representative Subgroup Differences in
Standardized Units
Percent of Students Selected:
Two Composites and Three Selection Strategies
Top 85% Top 50% Top 15%
Group AB AB+ AB AB+ AB AB+
Hispanic 4.4 4.6 4.1 4.9 3.9 5.5
(+.2) (+.8) (+1.6)
Asian 7.6 7.7 9.9 9.5 17.5 12.9
(+.1) (-.4) (-4.6)
African-American 17.9 19.8 9.6 13.6 1.3 7.2
(+1.9) (+4.0) (+5.9)
White 70.2 67.9 76.4 71.9 77.2 74.4
(-2.3) (-4.5) (-2.8)
Limitations & Future Research
• Public relations and acceptance of these measures by consumers (i.e., admissions officers, parents, students). Need to collection
reactions to new admissions measures along a variety of dimensions (e.g., fairness, face validity).
• Fakability in high-stakes situation especially relevant for biodata, less so for SJI. However, note that essays can be coached and edited, and self-reported activities can also be inflated.
• More research and evaluation efforts need to be conducted when these measures are used operationally in college settings.
Study 3: Purpose & Research Questions
• Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible
– Administer new measures to college applicants rather than college freshmen.
– On an annual basis, collect class absenteeism, self rated performance of the
noncognitve dimensions, and commitment to the university from enrolled students;
institutions will provide course grades and retention information.
• Research Questions:
– The incremental validity of the biodata and the situational judgment measures will be assessed after controlling for high school GPA and SAT/ACT scores.
– Differential prediction will also be assessed to see whether each measure-outcome relationship differs across various subgroups (e.g., gender and race).
– The relationship between scores on these noncognitive measures and holistic file review will be examined to test whether these measures could be substituted for the more subjective file review.
Thank You
Thanks to ATP and
Thanks to you
Questions , Comments, Suggestions
• Researchers are encouraged to freely express their professional judgment. Therefore, points of view or opinions stated in College Board presentations do not necessarily represent official College Board position or policy.
• Please forward any questions, comments, and suggestions to:
Andrew Wiley at: [email protected]
