Multiple Measures Assessment Project

(1)

Fall 2015

Pilot College Examples

September 23, 2015

(2)

Mallory Newell The RP Group [email protected] Terrence Willett The RP Group [email protected] Craig Hayward The RP Group [email protected] Loris Fagioli The RP Group [email protected]] John Hetts

Educational Results Partnership [email protected]

Ken Sorey

Dan Lamoree

(3)

Cathy Hasson, Director, Institutional Research and Planning, San Diego Community College District

[email protected]

Nathan Pellegrin, Director of Institutional Research, Peralta Community College District

[email protected]

Janet Fulks, Dean of Precollegiate and Student Success, Bakersfield College

[email protected]

(4)

MULTIPLE MEASURES ASSESSMENT PROJECT (MMAP) P R E S E N T E D B Y : C A T H Y H A S S O N , E D . D . D I R E C T O R I N S T I T U T I O N A L R E S E A R C H & P L A N N I N G S A N D I E G O C O M M U N I T Y C O L L E G E D I S T R I C T

(5)

MMAP PILOT PROCESS

• Established a districtwide MMAP Work Group in Spring 2015:

1. Discussed the concept, and adoption of MMAP model

2. Reviewed the retrospective analysis, and validated rule sets

3. Provided input for piloting MMAP

4. Drafted an assessment plan that includes initial implementation strategies

5. Continually share knowledge and inform campus constituents of MMAP & CAI

• SDCCD IRP conducted retrospective analyses in Spring 2015 using SDCCD student data

to test the local predictive validity of the MMAP model.

• Worked with CalPASS to implement the pilot in Fall 2015. Students that were

identified as eligible to be ‘bumped’ to transfer level English and/or math were contacted via email prior to Fall 2015 registration.

(6)

MATH TRANSFER LEVEL PLACEMENTS

FALL 2015

The MMAP pilot cohort consists of students who applied, but may not have registered, Jan. 2015 and June 2015, and took the Accuplacer test between . Approximately 941 students are in this cohort.

Accuplacer Placement at Transfer MMAP Placement at Transfer Combined Accuplacer and MMAP Placement at

Transfer DifferencePercent

City College/ECC 67 138 205 206% Mesa College 94 159 253 169% Miramar College 100 80 180 80% All Colleges 261 377 638 144%

(7)

ENGLISH TRANSFER LEVEL PLACEMENTS

FALL 2015

The MMAP pilot cohort consists of students who applied, may not have registered, between Jan. 2015 and June 2015 and took the Accuplacer test . Approximately 941 students are in this cohort.

English Writing Accuplacer Placement at Transfer MMAP Placement at Transfer Combined Accuplacer and MMAP Placement at

Transfer DifferencePercent

City College/ECC 48 116 164 242% Mesa College 113 120 233 106% Miramar College 66 83 149 126% All Colleges 227 319 546 140%

(8)

48% 20% 7% _6% 36% 11% 5% 8%

Latino Asian/PI Filipino African Amer

(9)

• Engagement Strategies

 Engaging key stakeholders

 Professional development

 _{Engaging special program}

populations • Assessment Strategies  Research  Non-cognitive measures  Contingency plans • Enrollment Strategies  Class offerings  Implementation process • Communication Strategies  Informing stakeholders  Informational materials • Evaluation Strategies  Snapshot analysis  Longitudinal analysis  Process analysis SDCCD ASSESSMENT PLAN

(10)

• Share the Assessment Plan broadly and

collect feedback to refine strategies. (Fall 2015)

• Continue pilot in Spring 2016 using

revised implementation logistics. (November 2015)

• Evaluate outcomes of fall and spring

cohorts, and refine placement rule sets if needed. (Spring and Summer 2016)

• Review, select, and pilot non-cognitive

measures. (Fall 2016)

(11)

Peralta Community College District

Overview of a District-wide Approach Berkeley City College

College of Alameda Laney College

Merritt College

Nathan Pellegrin, Director of Institutional Research [email protected]

(12)

Peralta CCD MMAP

• Official launch of pilot in Spring 2016

• Data sharing agreements with two districts

(additional MOUs in progress)

• Secure web portal for collection, validation and

exchange of student data

• Automated the processing of electronic

transcripts

• Created a tool for applying MMAP criteria to

(13)

(14)

(15)

Multiple Measures Assessment

Project

Janet Fulks

Dean of Precollegiate and Student Success Bakersfield College

(16)

• ASCCC Senate representation and collaboration in

basic skills and multiple measures

• Grew from concern that 80-84% of students were

underprepared

– Stemmed from interest by the presidential leadership

and administration who were concerned about student success

• Basic skills faculty were focused on success and

participation in acceleration

• Data included scorecard and progression of

first-time students through development sequence

(17)

Philosophy behind MMs

• Tests aren’t always the best measures

• Tests alone are TERRIBLE measures • The goal is to predict success

• More information provides better placement

(18)

Beginning the Process

• Academic Senate

• STEPS (Student Transcript-Enhanced

Placement Study)

• Create Faculty and Expert Committee – Data coaches, Math and English faculty,

Counselors, Administrators

• Implement BC MM’s with 500 student HS

transcripts

(19)

Multiple Measures Process

Electronic File from Kern High School District Included:

– GPA through fall 2014

– Highest Writing class and grade – Highest math class and grade

Moved students up if their score was in the upper half of range, and:

– High school GPA of at least 3.0, or

– Four or more years of English or math in high school,

and

(20)

What we learned from the data…

4 different treatment groups: 1. Bumped up a level

2. Accelerated 3. Compressed

4. Left with placement score validated by GPA and grades The worst performance was from:

– Students left where they placed, and – Students bumped up more than once

(21)

21

What we learned from the data……

MIH Group Math

(college-wide) English (college-wide) Reading (college-wide) Total students enrolled from each cohort 2011 64% (50.5%) 57% (57.4%) 62% (59.3%) 73 2012 59% (52.7%) 64% (61%) 75% (60.8%) 92 2013 64% (53.1%) 61% (61.8%) 59% (61.6%) 99 2014 47% 60% 62% 326

(22)

What we learned from high school testing..

• Some shocking information:

– Students test better at the high schools than in a foreign

location – a lot better

– This is a challenge since we have 41 feeder high schools

• Previous testing was not web-based, we changed to

Accuplacer – a web-based (more easily delivered) test which promised to automatically apply multiple measures and branched (smart) testing.

• Accuplacer also provided writing exams versus

multiple choice for English

(23)

(24)

Paradigm shifts with high schools

-Mostly ineffective -Testing cumbersome -Multiple measures not applied

-Test scores/placement low -Students unsuccessful -Academic Development & English Curricular Redesign & Math Lab

-Counseling impacted -Classes close early; HS students unable to get necessary classes

-One-day orientation and registration events

-Testing at high schools: student performance higher -Multiple Measures applied to 500 HS students; higher

placements -MIH Mentoring

-Continued curricular redesign -Emphasis on Habits of

Mind/Classroom Interventions

-Orientation at 21 HS; 2,508 attendees

-Assessment – new exam Accuplacer at 29 HS -47 proctors trained

-5548 assessment tests given -Counseling at 40 HS

-2,130 students complete ASEPs

-Registration at 12 HS -1,421 scheduled to attend -Follow-up counseling and registration New students must matriculate at BC Matriculation streamlined at BC and high schools

Matriculation provided at most high schools

(25)

• Negative

Positive

• Correction of institutional barriers

• A learning institution (president &

classified)

• Analysis of this group compared

to overall

• Equity insights

• Improvements to bridge and

summer

• Improvements to other support

services and instruction

• Partners – high schools,

community groups, CalPASS+, CalSOAP

• Inability to use many college

services

• Problems with communicating

through email

– Email address

– Not using email – texting

– No computers

• Messaging and culture (financial

aid & early alert)

• Overall Math + English + Reading +

12 + units were too much all together

• Learned about unit ceiling for this

cohort

Lessons Learned and Unintended Consequences

(26)

What we learned from the process…

• Colleges will need support on the algorithm • Colleges will need a direct line to high school

grades

• Colleges will need dedicated champions and

expertise

• If you use the senior grade, then currently

CalPASS+ cannot provide transcript data

• Devise a plan that can get grades and upload • High schools are ecstatic and will want to do all

they can (it was a real image booster as we reached out)

(27)

Mallory Newell The RP Group [email protected] Terrence Willett The RP Group [email protected] Craig Hayward The RP Group [email protected] Loris Fagioli The RP Group [email protected]] John Hetts

Ken Sorey

Dan Lamoree