PREDICTIVE ANALYTICS FOR STUDENT SUCCESS: Developing Data-Driven Predictive Models of Student Success

(1)

PREDICTIVE ANALYTICS

FOR

STUDENT SUCCESS:

Developing Data-Driven Predictive Models of Student

Success

Final Report

University of Maryland University College

January 6, 2015

(2)

The purpose of the Predictive Analytics for Student Success (PASS) project was to: (a) aggregate data across multiple institutions to track the academic progress and completion of community college transfer students, (b) identify factors associated with success, and (c) implement interventions that promote student success. In completing the PASS project research and interventions, University of Maryland University College (UMUC) partnered with two community colleges, Montgomery College (MC) and Prince George’s Community College (PGCC). This work was funded by a grant from the Kresge Foundation -- Developing Data-Driven Predictive Models of Student Success.

The purpose of the grant was:

 To build an integrated database tracking students across institutions from community college to UMUC.

 To use predictive statistical models and data mining techniques to track and model students’ progress across institutions.

 To identify factors predictive of students’ success at UMUC

 To inform the development of interventions aimed to improve outcomes for undergraduate students transferring from community colleges to UMUC or to other four-year institutions. This report will summarize the data development, research, intervention implementation and evaluation, dissemination, and application creation completed through the PASS project. Phase 1

During the first 24 months (Phase 1) of the grant, UMUC and the partner institutions developed and signed a Memorandum of Understanding (MOU) to ensure data security and establish parameters for data use. The MOU allowed the PASS project team to conduct research using individual student data while protecting student information and confidentiality. Once the MOU was in place, researchers identified the population of interest, conducted an initial literature review to identify variables of interest, and began data collection and exploratory analyses. The research team identified over 250,000 students enrolled at UMUC between 2005 and 2012. Of those, over 30,000 students transferred from MC and PGCC. Student demographics,

academic performance at the three institutions, behavior in the online classroom at UMUC, and advising data were combined into an integrated, multi-institutional database: the Kresge Data Mart (KDM).

The literature review covered student performance in online courses, successful course completion, factors associated with re-enrollment and retention, and the use of data mining techniques in higher education. Existing research showed that factors such as the number of schools attended, the number of credits transferred, and community college GPA influenced student success. Key measures of success included successful course completion and retention.

(5)

In Phase 1, initial data mining was conducted to identify variables that were associated with success. Specific courses were identified as having predictive value in relation to success at UMUC. Regression analyses determined that student online classroom activities prior to the start of a class (i.e., entering the online classroom prior to the first day) and during the early weeks of the course were predictive of successful course completion.

Phase 2

Phase 2 of the PASS project was completed in months 25 to 36 of the grant. The initial plan for Phase 2 was to:

 Secure external evaluators

 Further develop collaboration with the community colleges  Identify the scope of the project

 Clarify the research plan and conduct associated analyses  Begin initial dissemination of research findings

UMUC began meeting regularly with the community colleges to develop the Phase 2 research plan and evaluate research findings and grant progress. Two external evaluators were selected to conduct an independent evaluation of the research project. These collaborations proved to be highly beneficial in developing the research program and designing interventions. As a result of the collaborations, new data were identified for collection, and a full scope of the research was outlined in the form of a research plan.

A research plan was developed to model students’ progress and performance from the

community college to graduation from a four-year institution. The research plan created a model addressing the relation between students’ prior academic work and performance at UMUC to include graduation. The full path model of students’ academic trajectory from community college to UMUC is below.

The plan identified the following research goals for Phase 2: 1. To develop profiles of transfer students at UMUC

2. To identify factors from students’ community college academic backgrounds that predict success at UMUC

3. To develop predictive models of student success based on demographic information, community college course taking behaviors, and first-term factors.

4. To develop interventions designed to improve the success of students transferring from community colleges to UMUC.

Community College

Data

UMUC First

Term GPA Retention Graduation

Re-enrollment

(6)

Phase 2 considered two primary outcomes of interest in predictive models: 1) earning a first-term GPA of 2.0 or above at UMUC, termed successful first-term GPA, and 2) students’ re-enrollment at UMUC within 12 months following their first academic term, termed retention.

Key findings from Phase 2 include:

 Across studies, age and marital status were associated with success at UMUC. Older, married students were found to be more likely to succeed.

 Four profiles of student success at UMUC were identified based on students’ GPAs and retention rates. The profiles differed in terms of community college course taking preferences and course load and in the change in GPA when transferring to UMUC. These results suggest that the degree of student preparedness, particularly in specific target areas (e.g., accounting, economics), is predictive of success at UMUC.

 Course efficiency, or the ratio of credits earned to credits attempted, in the community college was determined to be a predictor of success at UMUC. The higher the course efficiency, the more likely a student was to succeed.

 A new factor, delta GPA, was introduced in these analyses, corresponding to the

difference between students’ GPA at the community college and at UMUC. While most students experienced a decreased GPA when transferring to UMUC, the magnitude of this decrease was predictive of students’ continued enrollment at UMUC beyond the first term.

 Students who took math or honors courses in community college were more likely to succeed at UMUC, suggesting that rigor of community college courses may prepare students to succeed at a four-year university.

 Student behaviors in the online classroom indicated high variability in the extent to which they engage in course content and course-related activities. A substantial percentage of students accessed course content and course materials to a limited extent, thus impacting successful course completion.

Phase 3

Phase 3, the final year of the Kresge Grant, focused on four goals: 1. Data enhancement

2. Extended research

3. Implementation and evaluation of the interventions

4. Continued dissemination of research findings and intervention results

As a result of continued collaboration with the community colleges, the KDM was expanded to include additional variables from the community colleges as well as updated data from UMUC to allow for expanded analyses of retention and graduation.

With the inclusion of new data, Phase 3 analyses focused on re-enrollment, retention, and graduation from UMUC.

(7)

Key Findings from Phase 3 include:

 Demographic Factors. Gender and marital status were associated with both performance (earning a successful first-term GPA) and persistence (re-enrollment). These

characteristics may indicate students’ maturity and commitment to pursuing academic goals. Interestingly, while African American status was negatively associated with

earning a successful first-term GPA, it was positively associated with persistence metrics. This suggests that while not always successful in their first semester, African American students are nonetheless committed to their educational goals.

 Math at the Community College. Across models examining both persistence and

performance, variables associated with taking math at the community college were found to be significant predictors. Within our sample, taking math at the community college reflects academic abilities and may also reflect students’ commitment to meeting the requirements necessary for transfer and graduation.

 Community College Success and Completion. In models predicting first-term GPA, re-enrollment, and graduation, students’ community college GPA was a significant

predictor. This suggests that, overall, performance at the community college matters for success and persistence at a four-year institution.

 First-Term Performance. As in findings from Phase 2, students’ performance in their first semester at UMUC remains crucial in predicting re-enrollment, retention, and graduation. In fact, across models, it was the strongest individual predictor of performance. First-term GPA may be an indicator of factors contributing to students’ success, beyond academic abilities. Specifically, students who are better at acclimating to an online university and the demands associated with a four year institution may have a higher first-term GPA and may be more able to persist.

 Online Classroom Engagement. A particularly rich finding from Phase 3 analyses is the association between student online classroom engagement as measured in the learning management system (LMS) and course performance. The general pattern was that students earning higher grades in a particular course were also significantly more engaged in the online classroom. Further, online course engagement, in combination with students’ community college GPA, was predictive of overall course performance; such a model linked students’ community college backgrounds with four-year

institutional experience.

Phase 3 also included the examination of the efficacy of four interventions aimed at promoting community college transfer student success at UMUC. In addition, two interventions at the community college were used to better prepare students for transfer.

Interventions undertaken at UMUC were:

 Checklist. New student orientation checklist administered to community college transfer students to aid them in navigating online resources at UMUC. Although no significant differences were found, students responding to an evaluation survey found the checklist to be a useful tool.

(8)

 Mentoring. Eight week structured mentoring program, where new UMUC community college transfer students were paired with a peer mentor -- a successful student at UMUC who had transferred from the same community college. Each week, mentors emailed mentees with study tips and information to support adjustment to UMUC. Although no statistically significant improvements in semester performance were found for mentees, unexpectedly, students serving as mentors had a significantly higher cumulative GPA and a significantly higher rate of successful course completion when compared to the control group of students who were invited to be mentors and elected not to participate. This phenomenon may be due to the bias inherent in the self-selection process.

 Jumpstart Summer. A program that paired mentoring with Jumpstart, a four-week

onboarding course, designed to support students’ goal setting and academic planning. Four experimental conditions were examined: (a) a control group, (b) students only completing the Jumpstart course, (c) students only participating in the mentoring program, and (d) a Jumpstart Summer group, receiving both mentoring and enrolled in the Jumpstart course. No significant differences in performance were found; however, students successfully completing the Jumpstart course had a higher rate of successful course completion and re-enrolled at a higher rate.

 Accounting 220/221: The online tutoring intervention was developed by faculty for students taking Accounting 220 and Accounting 221 -- courses with a disproportionally high failure rate both at UMUC and nationally. Students who participated in the online tutoring had a significantly higher GPA at the end of the semester and a significantly higher rate of successful course completion, when compared to students not participating in online tutoring.

Interventions developed at the community colleges were:

 Diverse Male Student Initiative (DMS-I). DMS-I is a two-year program at Prince

George’s Community College that provides minority male students with role models and academic and career mentoring. DMS-I held a two-day summer institute at PGCC that featured keynote speakers and awarded book and tuition vouchers for early registration to participants with the aim of improving academic planning and persistence. PGCC and UMUC will track and evaluate the success and persistence of students who participated in the program and who transfer to UMUC.

 Women’s Mentoring, Boys to Men, TriO: Women’s Mentoring, Boys to Men, and TRiO are comprehensive mentoring programs, developed at Montgomery College, that provide minority students with comprehensive academic and social support throughout their transfer pathways from high school to MC, and ultimately to a four-year institution. MC and UMUC will identify students participating in these programs who transfer to UMUC and will track them to evaluate their performance. UMUC will provide similar mentoring and support if these students transfer to UMUC.

Findings from research and interventions were disseminated through ten conference

presentations and manuscripts. In addition, a website, http://www.umuc.edu/PASS, was created to share project goals and results with a broad audience of stakeholders.

(9)

This report was produced by the UMUC Office of Institutional Research and Accountability and contains 12 sections:

Section 1: Introduction Section 2: Literature Review

Section 3: Research Scope and Design Section 4: Data Sources

Section 5: Survival Analysis: Registration and Withdrawal for Online Courses Section 6: Mining of Community College Data

Section 7: Predicting Student Success Section 8: Graduation Rates

Section 9: Data Mining of Online Learner Behavior Section 10: Students’ Motivation and Self-Regulation Section 11: Implementation and Evaluation of Interventions Section 12: Dissemination

Section 13: Financial Statement Section 14: Conclusions

(10)

SECTION 1: INTRODUCTION

The purpose of this report is to present the results of research conducted by the University of Maryland University College (UMUC), in partnership with two community colleges,

Montgomery College (MC) and Prince George’s Community College (PGCC) as part of the PASS project. The scope of work was undertaken as part of a grant from the Kresge foundation and includes: (a) data development, (b) research using data mining and predictive modeling to examine community college transfer student success, and (c) intervention development, implementation, and evaluation to provide academic, social, and institutional support to community college students both prior to and after transfer. The project was broken out into three phases. Each phase was built upon results from the previous phase, resulting in continued research development and comprehensive analyses. A research plan identifying the scope of the project began in Phase 1. The final research design and methods were finalized in Phase 3. The research plan was developed to conceptualize students’ academic pathways from the community college, to transfer to a four-year institution, to graduation from UMUC. In

developing the research plan, specific milestones in students’ academic pathways were modeled. These included: (a) earning a successful first-term GPA, (b) re-enrolling in the immediate next semester after transfer, (c) retention (re-enrollment within a 12-month window), and (d) graduation within an eight-year period. Each of these milestones was predicted based on data aggregated from the community college and UMUC. Specifically, students’ demographic information, community college course taking behaviors, indicators of first-semester

performance at UMUC, and behaviors in the online classroom were used in predictive modeling and in data mining. A model presenting students’ academic trajectories from transfer to

graduation that guided the research was developed. PASS project goals included:

 To develop a data sharing partnership and create a memorandum of understanding between partner community colleges and UMUC

 To build an integrated database tracking students across institutions, from community college to UMUC

 To integrate data from students’ community college backgrounds with UMUC performance data for use in research

 To use data mining to develop profiles of transfer student success at UMUC

 To identify factors from students’ community college academic backgrounds that predict success at UMUC

 Develop predictive models of UMUC first-term GPA, re-enrollment, retention, and graduation based on community college data

 Examine graduation rates of community college transfer students at UMUC  Examine online classroom engagement as associated with course performance  Profile students’ motivational and self-regulatory attributes

 Develop, implement, and evaluate interventions aimed at promoting community college transfer students’ success

(11)

Grant Partnership

UMUC is a four-year public university that offers online degree programs to a diverse population of working adults. With support from the Kresge Foundation, UMUC established partnerships with two Maryland community colleges that also serve large and diverse student populations. Montgomery College (MC), established in 1946, enrolls over 60,000 students annually. Prince George’s Community College (PGCC) enrolls more than 40,000 students from approximately 128 different countries. Both institutions serve the metro-D.C. area, but differ in that PGCC serves more low-income students. Both institutions have endorsed the goals of this project and are committed to working with UMUC to find ways to promote student success throughout their academic careers.

Objectives and Milestones

Specific objectives and milestones were identified for each phase of the research project. These objectives and milestones have been modified throughout the course of the project, but are consistent with grant requirements. Table 1 presents the objectives and milestones for each phase.

Table 1. Project objectives and milestones

Objectives Milestones Status

Phase 1 April 2011 – October 2012

Develop a Project Action Plan

Develop a project action and collaboration plan with the partnering agencies.

Complete

Data Collection and Preparation

Prepare a data ―universe‖ (integrated database system) on CC transfer students in the UMUC population (KDM)

Complete

Understand variables; define student characteristics and retention data; develop data dictionary.

Complete Data Analysis Conduct initial predictive analyses and employ data

mining techniques to identify factors contributing to students’ success

Complete

Project Evaluation

Conduct ongoing project evaluation. Take action on identified areas for improvement.

Complete

Phase 2 November 2012 – October 2013

Develop and Validate

Analytic Models of Student Success

Analyze data and identify factors that predict success/failure.

Complete Validate predictive analyses and models developed

through data mining techniques to predict students’ success and retention at UMUC.

Complete

Build student profiles based on analyses. Complete Disseminate Key

Findings

Discuss results with Kresge Workgroup and share with advisory board.

Complete Discuss results with Project Partners and obtain

feedback.

(12)

Objectives Milestones Status Present key findings at national conferences on higher

education

Complete Develop

Interventions

Work with stakeholders at UMUC and CC partners to develop a list of potential interventions.

Complete Project

Evaluation

Conduct ongoing project evaluation. Take action on identified areas for improvement.

Complete Research Plan 3 Design and develop KDM2 to update and improve

data related to student success

Complete Plan Phase 3 analyses on expanded integrated data. Complete

Phase 3 November 2013 – December 2014

Develop Interventions

Review relevant literature on interventions that promote student success in online learning.

Complete Develop an implementation plan and timeline for

piloting of interventions.

Complete Implement Pilot

Interventions

Implement and evaluate pilot interventions. Complete Disseminate

Results on Interventions

Develop and disseminate report on the pilot interventions

Complete

Phase 3 Analyses Develop and execute Phase 3 research plan Complete Report Findings Present key findings from Phase 3 analyses at national

conferences; publish research in journals

Complete Prepare written report of both Phase 3 analyses and

full scope of Kresge grant work.

Complete Dissemination of

Results and Resources

Develop website and repository for educational data mining and student success.

December 2014 Host a national convening on data mining and learner

analytics.

Complete Project

Evaluation

Deliver final project evaluation. December 2014

(13)

SECTION 2: LITERATURE REVIEW

The literature review was conducted over the course of the four-year project. This section presents a review of the literature in the following areas:

 Theoretical models of community college transfer student performance  Educational data mining

 Predicting transfer student first-term GPA  Predicting transfer student re-enrollment  Literature guiding interventions

 Community college student transitioning  Literature to support specific interventions

Theoretical Models of Community College Transfer Student Performance

Two theoretical models of community college transfer student performance and persistence have guided work in the field, as well as in the PASS project analyses. The first is Tinto’s (1975, 1987) Student Integration Model, which applies a psychological lens to understanding student attrition. The Student Integration Model identifies four aspects of student-institution interactions that have an effect on persistence. Specifically, these are the background characteristics and academic goal commitments that students bring to a university setting, and in turn, their effects on students’ academic and social integration at the transfer institution. Background

characteristics include students’ demographic attributes, family backgrounds, and experiences prior to college (Tinto, 1975). Goal commitments include learners’ motivation for degree pursuit and educational expectations as well as institutional commitment to a particular university. Academic and social integration is based on students’ interactions with a variety of institutional features over time. Tinto (1975) suggests that these interactions may be evaluated based on both

structural and normative considerations. Structural considerations refer to objective and explicit

social and academic standards that students may have to meet (e.g., a minimum GPA, meeting with an advisor), whereas normative components of integration relate to students’ identifications with these standards (e.g., earning a high GPA). Tinto emphasized the central importance of students’ institutional integration, both academic and social, by saying, ―we learned that involvement matters and that it matters most during the first critical year of college,‖ (Tinto, 2006, p. 3; Upcraft, Gardner, & Barefoot, 2005).

At the same time, academic and social integration into a transfer institution are not givens for many students. Building on Tinto’s earlier work (1975), Bean and Metzner (1985) developed a model of attrition, reflecting the experiences of non-traditional undergraduate students, termed the Conceptual Model of Non-Traditional Student Attrition. In their definition, non-traditional undergraduate students may be defined as those who are older (i.e., 25 and above, Stewart & Rue, 1983), enrolled part-time, non-residential, commuting to campus, or representing some combination of these characteristics (Bean & Metzner, 1985). Understandably, this population of students is considered to undergo a different socialization process from that of traditional students conceptualized in Tinto’s model (1975). Non-traditional students may have different experiences with and potential for institutional commitment and social integration. Bean and Metzner (1985) suggest that this may be because older students exhibit greater characteristics

(14)

associated with maturity and therefore may be less open to the socialization process and because these students have more limited contact with socializing agents (e.g., faculty, peers, Chickering, 1974). More generally, non-traditional students may be less interested in institutions’ social culture, and rather more concerned with academic offerings and credentials.

Juxtaposing the experiences of traditional and non-traditional students, for non-traditional learners there is (a) more limited interaction with faculty and peers as well as with college services (i.e., more limited social integration, as per Tinto, 1975), (b) similarity in academic focus and experience (i.e., parallel classroom experience), and (c) much greater interaction with the external, non-institutional environment (Bean & Metzner, 1985).

Based on the differences identified between traditional and non-traditional students, Bean and Metzner (1985) conceptualize students’ decisions to drop out as predicated on four general types of variables. The first of these are background factors, including students’ demographics, past academic performance, and educational goals and expectations. The second group of

considerations is students’ academic performance, or factors reflecting learners’ grades, study habits, and pursuit of major at the transfer institution. The third group of factors are students’

intent to leave, considered to be more psychological; these include students’ goal commitment,

perceived utility of a given degree, and institutional satisfaction. Finally, unique to this model, the fourth group of factors are external factors that may have a direct effect on students’ decisions to drop-out; these include finances, out-of-school work, and family commitments (Bean & Metzner, 1985).

There are two compensatory relationships between variables identified. First, if students’ academic outcomes are low, they may nonetheless persist, compensating with high levels of psychological commitment. Further, when academic performance is low, students will persist if environmental factors support their continued enrollment. Conversely, when environmental factors do not support persistence, for non-traditional students, even high academic performance may not be sufficiently compensatory. More generally, Bean and Metzner (1985) suggest that for non-traditional students, environmental factors may have a much more pronounced effect on attrition decisions than do academic factors, as non-traditional students are much more closely affiliated with the non-institutional environments than are traditional students residing on university campuses (Bean & Metzner, 1985; Metzner, 1984).

As such, understanding non-traditional student persistence may be particularly challenging at the institutional level as, in large part, it may be attributed to environmental factors that the

institution may not be aware of or able to control. Indeed, for non-traditional students, the primary point of institutional interaction has to do with academic factors; as such these areas represent targets for intervention (Bean & Metzner, 1985).

As part of the PASS project, we were interested in gaining insight into community college transfer students’ persistence at a four-year institution by looking longitudinally to consider which background factors, including learner characteristics and community college experiences, may impact student re-enrollment and continued pursuit of educational goals.

(15)

Educational Data Mining

Current literature on student success focuses on outcomes such as course success, course

withdrawal and retention. For example, variables such as student characteristics, previous course work, grades, and time spent in course discussions and activities may be useful in predicting course success (Aragon & Johnson, 2008; Morris & Finnegan, 2009; Morris, Finnegan & Lee 2009; Park & Choi, 2009). Course-level variables acquired from student login data from the LMS may have predictive value in measuring course withdrawal rates (Willging & Johnson, 2008; Nistor & Neubauer, 2010). Variables such as student characteristics, number of transfer credits, final grade in any given course, experience in online environments, and course load may be useful in predicting re-enrollment and retention (Aragon & Johnson, 2008; Morris &

Finnegan, 2009; Boston, Diaz, Gibson, Ice, Richardson & Swan, 2011).

Although these studies showcase a variety of findings related to student success, the majority of studies of retention in online learning environments use traditional statistical or qualitative methods. Park and Choi (2009) point out that expansion of methods such as data mining may have utility when student, course, program, and institutional level variables are well defined and institutionally meaningful. Literature related to educational data mining focuses on exploratory research.

Data mining is a method of discovering new and potentially useful information from large amounts of data (Baker & Yacef, 2009; Luan, 2001). Educational data mining is a subset of the field of data mining that draws on a wide variety of literatures such as statistics, psychometrics, and computational modeling to examine relationships that may predict student outcomes (Romano & Ventura, 2007; Baker & Yacef, 2009). In educational data mining, data mining algorithms are used to create and improve models of student behavior in order to better understand student learning (Luan, 2002).

Data mining methods are most helpful for finding patterns already present in data, not

necessarily in testing hypotheses (Luan, 2001). Baker and Yucef (2009) suggest that research in higher education should use a variety of algorithms, such as classification, clustering or

association algorithms in determining relationships between variables. Although many

definitions of these techniques exist in data mining literature, Han and Kamber (2001) offer the following definitions. Classification is the process of finding a set of models or functions that describe and distinguish data classes or concepts to predict a class of objects whose class label is unknown. Clustering analyzes data objects that are related to similar outcomes without

consulting a class label. Association is the discovery of rules showing attribute value conditions that occur frequently together in a given set of data (Han & Kamber, 2001).

Recent research suggests that these data mining algorithms can be used to examine variables related to student success. Yu, DiGangi, Jannach-Pennell, Lo, and Kaprolet (2010) used a

classification algorithm to explore potential predictors related to student retention in a traditional undergraduate institution. In this study, the authors used a decision tree to explore demographic, academic performance, and enrollment variables as they related to student retention. This study revealed a predictable relationship between earned hours and retention, but also found that at this institution, retention was closely related to state of residence (in-state/out of state) and living

(16)

location (on campus/off campus). The authors speculate that this finding points to the potential utility of online courses in improving retention for out-of-state or off-campus students.

Despite these recent developments in exploring variables related to student success in traditional higher education settings, research using data mining techniques to uncover relationships among variables in online courses is limited in scope. The PASS project is designed to fill this gap in the extant literature by utilizing data on online students who attended multiple institutions.

Predicting Transfer Students’ First-Term GPA

Generally, the transition from community college to a four-year university has been considered to be a stressful period for students. In early examinations of this transitional period, Hills (1965) determined that when students from junior college transfer to a four-year university they might experience an ―appreciable loss in their level of grades‖ (p.209), termed transfer shock. Transfer shock has been defined as a decrease in academic performance (i.e., GPA) experienced by students in their first semesters at a four-year university, due to difficulties with adjustment (Keeley & House, 1993). Since Hill’s (1965) initial exploration, a wealth of studies have emerged examining transfer shock and students’ decreases in GPA when transitioning from community college to a four-year university (e.g., Best & Gehring, 1993; Keeley & House, 1993; Preston, 1993; Soltz, 1992).

However, recent research has painted a more nuanced picture of transfer shock. Cejda, Kaylor, & Rewey (1998) determined that transfer shock is discipline specific. For instance, while students transferring into mathematics and science majors did experience a drop, those majoring in the fine arts and humanities actually experience an increase in GPA. Further, in a meta-analysis of 62 studies examining transfer shock, Diaz (1992) determined that while the majority of studies did find that community college students experience a transfer shock, it was slight (i.e., one half of a grade point or less); also, the majority of studies reviewed found that students recovered from transfer shock over the course of their university careers. Nickens (1972) skeptical of transfer ―shock‖ and ―recovery‖ suggests that transfer students’ GPAs cannot be distinguished from the GPAs of their native counterparts. Specific decreases in GPA may be explained by difference in institutional practices and any subsequent increases in GPA may be explained by regression to the mean and the attrition of weaker students (Nickens, 1972). Regardless of findings, across studies examining community college students’ performance when transferring to a four-year university, first-term GPA has been a key outcome of study (Carlan & Byxbe, 2000; Driscoll, 2007; Glass & Harrington, 2010; Hughes & Graham, 1992; Townsend, McNerny, & Arnold, 1993). This may be because first-term GPA has been considered to be a barometer of transfer students’ success and adjustment to a four-year

institution (e.g., Knoell & Medsker, 1965) as well as level of preparedness to meet the academic demands of a four-year university (Carlan & Byxbe, 2000; Roksa & Calcagno, 2008). Further, first-term GPA has been considered to be strongly associated with persistence or students’ retention and graduation from a four-year university (Gao, Hughes, O’Rear, & Fendley, 2002). Indeed, there have been a number of studies examining predictors of first-term GPA for

community college transfer students (e.g., Graham & Hughes, 1994; Townsend et al., 1993). Most commonly, demographic factors have been examined as potentially impacting community

(17)

college students’ transfer success. For example, Durio, Helmick, and Slover (1982) found that demographic factors (i.e., gender and ethnicity) impacted transfer students’ success. Examining an expanded pallet of variables predicting first-term GPA, Keeley and House (1993) considered students’ age, gender, ethnicity, college major, residence status, as well as class standing (e.g., sophomore) as predictive of first-term GPA. In particular, age (i.e., being older) and gender (i.e., being female) were found to the associated with higher first-term GPA for transfer students, as was having earned an associate degree prior to transferring. In addition to the focus on students’ demographic factors, GPA at the community college level has been found to be a key

determinant of first-term GPA when students transfer to a four-year institution (Baldwin, 1994; Towsend, McNerny, & Arnold, 1993). However, more research is needed to identify predictors of transfer students’ success at a four-year university (Johnson, 1987).

Course Taking Behavior at the Community College

In examinations of community college transfer students’ performance at four-year institutions, at the forefront have been considerations of students’ preparedness to handle the challenges

associated with university-level course work (e.g., Berger & Malaney, 2003; Keeley & House, 1993; Townsend, 1995; Townsend et al., 1993). Despite concerns over community college transfer students’ preparedness, limited research has examined the nature of community college students’ course taking backgrounds to determine predictors of university success. Some studies provide initial insights. For example, Phlegar, Andrew, and McLaughlin (1981) determined that students fundamentally prepared in key subject areas (i.e, math, science, and English) at the community college level performed better upon transferring. Deng (2006) determined that students attending career-focused community college programs outperformed those attending liberal-arts community college programs, when transferring to a four-year university. Rather than considering specific courses of study, Pennington (2006) determined that students’

enrollment in developmental course work in community college was associated with a decreased GPA upon transfer to a four-year institution.

Carlan and Byxbe (2000) found community college major to be significantly associated with first-term GPA; for instance, students majoring in education and psychology had a higher GPA after transfer than did students majoring in business and the sciences. However, it is unclear whether these major-specific differences in GPA drop were associated with different levels of students’ preparedness or with cross-institutional differences in the academic demands required by these various programs of study.

Rather than examining community college majors, Cejda et al. (1998) found students’ first-term GPA to be related to university major. Parallel to prior findings (i.e., Carlan & Byxbe, 2000) students in the sciences, indeed, experienced a drop in first-term GPA, while students in the fine arts, humanities, and social sciences experienced a GPA increase. This replicated findings that students majoring in the sciences and mathematics (i.e., biology, chemistry, math, physics, accounting, and economics) had a lower GPA than their fellow community college transfer students (James Madison University, 1989). However, the nature of students’ preparedness for a four-year university and the types of community college academic experiences that may support transfer success have yet to be fully examined.

(18)

Studies examining community college students’ preparedness have primarily examined students’ academic backgrounds at the level of the major (e.g., Carlan & Byxbe, 2000). Institutional data sharing as part of the PASS project, allowed the specific courses of community college students to be examined as predictors of performance at the four-year institution.

Predicting Transfer Student Re-Enrollment

Historically, research on student retention largely focused on the experiences of traditional students, until Tinto (1993) expanded on extant models of retention to consider which factors may impact the retention of non-traditional students. For both traditional and non-traditional students, retention was thought to be a consequence of students’ academic and social integration (Tinto, 1993). Other research has echoed the central role of social factors in predicting retention for non-traditional students, online, and distance learners (Boston, Diaz, Gibson, Ice, Richardson, & Swan, 2009). At the same time, a number of demographic and community college factors have been considered as predictive of community college transfer students’ persistence at a four-year university.

Based on a comprehensive review of the persistence literature, Peltier, Laden, and Matranga (1999) determined that gender, race and ethnicity, socioeconomic status, high school GPA, college GPA and interaction variables are all related to persistence. In particular race/ethnicity and prior academic achievement have been robust predictors of persistence (e.g., Astin, 1997; Tross, Harper, Osher, & Knwidinger, 2000; Levitz, Noel, & Richter, 1999), whereas findings with regard to gender have been more mixed, Reason, 2009; St. John et al., 2001). Wetzel, O’Toole, and Peterson (1999) used logistic regression, with a dichotomous outcome variable, retained or not. Retention was significantly predicted primarily based on academic factors, including GPA, earning a low GPA which places students at low academic risk, and the ratio of credit hours earned to the credits attempted.

Murtaugh, Burns and Schuster (1999) used survival analysis to examine the retention of undergraduate students, enrolling in a university between 1991 and 1996; 25% to 35% of the cohort examined had interrupted enrollment within this period. Specifically, 13.5% stopped out for a single term, 10.8% had stopped out for two terms, and 1.8% had stopped out for three terms, after which they were required to undergo a readmission process. Predictors of stopping out were referred to as hazards. Hazards were examined for one year, two year, and four year retention. Minority status had a higher rate of withdrawal than did white students; also associated with withdrawal was age, high school GPA, first quarter GPA, area of study, and participation in freshman orientation. In particular, Murtaugh et al. (1999) highlight the importance of pre-college characteristics in predicting persistence.

Looking at a sample of traditional, first time freshman, Cabrera, Nora, and Castaneda (1993) used structural equation modeling to analyze predictors from both Tinto’s (1975) and Bean and Metzner’s (1985) models to predict student persistence. Cabrera et al. (1993) ranked variables predicting persistence; the most important factor was psychological goal commitment, or intent to persist, followed by GPA, institutional commitment, and encouragement from family and friends. In turn, intent to persist was predicted by institutional commitment, encouragement from family and friends, academic goal commitment, and academic integration – these factors having an indirect effect on persistence.

(19)

Whereas the aforementioned studies focused on individual student factors predicting retention, Moore and Fetzner (2009) addressed the institutional characteristics that fostered commitment in non-traditional students. These factors included having a leadership culture that fosters

commitment to student success and institutional policies and practices that incorporate student support services and technological support. For online learners, access to services and support that meet their needs was found to be crucial (Moore & Fetzner, 2009). Further, student

satisfaction, defined as students happy with their progress and with support received for learning, and with a perception that the knowledge they were learning was valuable, was predictive of retention. Faculty satisfaction, stemming from involvement in curricular design and training in the use of online technologies supporting learning, were found to be key to engagement and contributors to retention (Moore & Fetzner, 2009).

Predicting Re-Enrollment for Non-Traditional Students

Based on theoretical work (Astin, 1975; Bean & Metzner, 1985; Tinto, 1975), we may expect that community college transfer students’ persistence may be affected by different factors. First, given that much of the literature examining community college students performance has

focused on the degree of student preparedness (Carlan & Byzbe, 2000), learners’ prior academic experiences may be particularly important to examine, especially as they include not only high school work, as for traditional students, but college-level course work at a two-year institution as well. Further, to the extent that transfer students enter more connected to external factors beyond their experiences at the transfer institution, it may be particularly important to examine learner background characteristics and how these are related to academic factors at the transfer

university.

Wang (2008), using logistic regression, found that the probability of graduating with a bachelor’s degree for students starting at community college was predicted by gender, socio-economic status, high school curricula, educational expectations, community college GPA, college involvement, and math remediation; while persistence, prior to graduation, was predicted by community college GPA and locus of control. Just as in the Wang (2008) study, in the PASS project, researchers looked to students’ demographics and community college factors, including course taking behaviors as well as overall performance, to predict next-semester re-enrollment. Kreig (2010) examined students at Western Washington University, an institution with a substantial population of community college transfer students comprising each education level, and found that native students were more likely to graduate, even after demographic

characteristics and prior academic performance were controlled. Krieg (2010) compares the experience of community college students to that of freshmen at a four-year university. For new community college students, there may be a difficult adjustment to a new learning context, which may result in early attrition if students consider themselves to be incompatible with the new environment. As such, first year retention is a particularly important factor to consider in understanding students’ persistence and ultimate graduation.

While this tension in fit has most commonly been examined by considering the drop in

performance (i.e., GPA) that community college students experience upon transfer to a four-year university, alternately termed transfer shock (Cejda, Kaylor, & Rewey, 199; Townsend,

(20)

McNemy, & Arnold, 1993), Krieg (2010) suggests that this may more profoundly manifest in rapid attrition from the four-year institution. More generally, there may be an interaction between transfer student status, first-term GPA, and drop-out rates (Spady, 1970). Specially, those transfer students scoring a low GPA in the first quarter were twice as likely to drop out as were native students (Krieg, 2010). Pascarella and Terenzini (1980) likewise conclude that the majority of attrition occurs in the freshman year, when students are new to the university setting, and further indicate that this marks a misalignment between theory and evidence. For instance, Tinto’s model of academic attrition is better suited to modeling student attrition beyond the first year.

The difference Krieg (2010) documents, is not specific to low performing students. Even high performing community college transfer students are more likely to drop-out than are their native counterparts. This may be because transfer students have less immediate affiliation and

integration into the transfer institution or because these transfer students are required to take prerequisite courses before entering into a major (Krieg, 2010). This points to the importance of looking beyond community college students’ prior academic performance, to look at specific course taking behaviors at the community college as well as to consider first-term GPA at the transfer institution – these factors were examined as part of the PASS analyses. More broadly, these findings are aligned with the interactional relationships identified in Bean and Metzner’s model (1985) that suggest that for non-traditional learners, academic success may not be a sufficient factor to promote persistence.

Literature Guiding Interventions

Intervention with specific populations (e.g., community college transfer students) and in specific contexts (e.g., online universities) are needed, as the majority of interventions have focused on finding solutions that will have a general effect on a broad population of students (Pascarella & Terenzini (1998).

Two prominent models of student retention have been proposed, however, both of these models speak primarily to the needs of traditional students. Tinto proposed the Student Integration Model, which identified attrition as resulting from a lack of congruency between students’ needs and institutional offerings (1987). Specifically, Tinto points to the need for students’ academic abilities and motivational orientations to match an institution’s academic and social

characteristics. In determining whether or not students will persist in post-secondary education, Tinto (1987) suggests that two forms of commitments must be in place. The first is students’ commitment to educational goals and the second is students’ commitment to remain within a particular institution.

From Tinto’s model of student retention, conclusions may be drawn regarding the types of factors that interventions to promote retention ought to foster in students; specifically Tinto’s model suggest the need for interventions that target students’ (a) academic abilities, (b)

motivational orientations, specifically with regard to the types of educations goals students adopt in pursuing higher education, and (c) institutional connections. Intervention designs should emphasize the correspondence between students’ abilities or goals and institutional offerings.

(21)

Yet more work is needed to understand how to adapt the Student Integration Model (Tinto, 1987), to reflect the experiences of non-traditional students, transfer students, and students enrolled in online universities, such as UMUC. In particular, factors affecting students’ retention may deviate from the proposed model based on differences in the type of institution students are a part of as well as students’ gender and ethnicity (Pascarella & Terezini, 1997). To this end, proposed interventions are geared not only with general UMUC student populations, but also speak to the specific needs of female students (e.g., Girls to Women) and diverse learners (e.g. mentors in the Community College Mentor and College Writing interventions are matched with mentees according to demographic characteristics, including ethnicity.

Tinto’s (1987) model has further been critiqued for being limited in considering the role that external factors, or considerations independent of students and institutions, may have on

retention (Pascarella & Terezini, 1997). These external factors, including financial and familiar considerations, may be particularly important to consider when modeling retention of non-traditional students. Studies have found that often times these students do not persist in post-secondary education because of finances, employment demands, and taxing family

responsibilities (Bean & Metzner, 1985).

To expand Tinto’s model and to consider the needs of non-traditional students – those classified as part-time, older, and non-residential (e.g., online learners at UMUC) – Bean and Metzner proposed a Student Attrition Model (1985). This model suggests that students’ persistence and academic outcomes can be understood as a result of four factors, namely: (a) background variables, (b) academic variables, (c) psychological factors, and (d) environmental variables. Background variables refer to students’ characteristics that may put them at a risk or deficit relative to their peers. These factors include age, high-school performance, gender, and

ethnicity. Academic variables include students’ study habits, the role of advising, and students’ certainty in their major. Psychological factors reflect students’ motivation for engaging in post-graduate education – these include students’ goal commitment, the expected utility or value of a degree, and psychological stress. Finally, environmental factors introduced in the model address students’ responsibilities outside of the university and may represent constraints on students’ pursuit of educational goals. The role of social interaction is featured in this model, as previous models have identified the importance of social integration in predicting students’ persistence (e.g., Tinto, 1975; Pascarella & Chapman, 1983), however, the nature of social interactions may differ for traditional versus non-traditional students.

In designing interventions, all four of the factors described in Bean and Metzner’s model were considered. In particular, mentoring programs targeted students and matched mentees with mentors according to background variables. Further, mentoring programs were intended to help students in mitigating the effects of environmental variables; the intention of these interventions was to provide students with role-models who have successfully persisted, despite limiting external factors. In providing students with an Introductory Check-List and academic tutoring, the interventions were designed to impact academic variables. Finally, psychological outcomes were targeted by encouraging students to take advantage of the advising available through UMUC and providing students’ with the opportunity to discuss their long term and professional goals with mentors.

(22)

One of the unique challenges for non-traditional students is the identified lack of social integration and social interaction (Bean & Metzner, 1985). A number of interventions were geared toward connecting students with social resources. For instance, the checklist intervention encouraged students to be involved with available student organizations. Further, to the extent that persistence is marked by a match between a student and an institution (Cabrera, Nora, & Castaneda, 1993), a number of the interventions were aimed specifically at helping students recognize others like them as members of the UMUC community.

Community College Transfer Students’ Transitioning

Transferring from community college has been identified as a high-stress time for students, presenting academic, psychological, and environmental challenges (e.g., Laanan, 2001). Flaga (2006) identified five dimensions of transitioning. These are, learning resources, connecting,

familiarity, negotiating, and integration. The first two dimensions deal with the knowledge and

skills that students need in order to be successful, whereas the last three dimensions address how these skills may develop over time.

Learning resources refer to the tools students may use to gain information about the university. Three types of learning resources were specified; these were: (a) formal resources provided by the university (e.g., orientation information), (b) informal resources provided by individuals knowledgeable about the university but not officially affiliated (e.g. information from alumni), and (c) initiative-based resources that students gather independently (Flaga, 2006). The second dimension, connecting, refers to the relationships that students are required to form when transferring to a new institution; including (a) academic connections (e.g., with faculty), social connections (e.g., with other students), and physical connects (e.g., with the university

environment) (Flaga, 2006).

The third dimension, familiarity, emerges when students become more comfortable with their new environment. The fourth dimension, negotiating, occurs when students adjust their

behaviors to better fit their new environment. Finally, the fifth dimension, integrating, does not always happen, but involves students shifting their identities to reflect their new institution (Flaga, 2006).

Literature to Support Specific Interventions

Checklist

The Checklist targeted the first two dimensions identified by Flaga (2006) as supporting students’ transitioning. Specifically, through the checklist, students received support

encouraging them to connect with both formal and informal information resources with the intent of forming academic, social, and physical connections. In completing the activities specified in the checklist, students had the opportunity to exercise initiative in connecting with resources and develop familiarity with UMUC as an institution and an academic community.

In a qualitative study of community college transfer students, one of the recommendations transfer students proposed as a resource to help their transition was the creation of a transfer

(23)

checklist (Owens, 2007). Indeed, 27% of students expressed a desire for the introduction of a

checklist or guide to aid them in the transfer process (Owens, 2007). In describing the features

that would make checklists appeal to them, students expressed desires for ease-of-use, online availability, and comprehensiveness (with information ranging from where to park to how to register for classes); as well as checklists that break down complex processes in a step-by-step manner and include necessary contact information (Owens, 2007).

This is a particularly important initiative given that surveys of community college students have determined that students have a need for more information (e.g., Harbin, 1997; Andres, 2001) and more assistance (Townsend & Wilson, 2006) as they move to their new institutions.

Community College Mentor

Peer-mentoring for community college students transferring to four-year schools has been under-examined in the literature. However, mentoring interventions have been broadly used as an avenue to promote students’ retention (Good, Halpring, & Halprin, 2000; Hoyt, 2000). Flaga (2006) suggests that benefits associated with peer mentoring are not only academic; through mentoring, students gain access to informal learning resources and have the opportunity to socially connect with their peers. Likewise, Good et al. (2000) found mentoring to confer psychological and academic benefits to both mentors and mentees.

The mentoring relationship has been identified as supporting three types of outcomes, namely

psychosocial, vocational, and role-modeling (Ensher, Heun, & Blanchard, 2003). Psychosocial

support refers to mentors providing counseling, friendship, and, encouragement to their mentees (Enscher et al., 2003). Vocational support is considered to be support that enhances the

professional lives of mentees (Enscher et al., 2003) and can be extended to include the academic support provided by mentors to new students. Finally, role-modeling refers to mentors

demonstrating appropriate behaviors or expectations, either implicitly or explicitly (Enscher et al., 2003). For example, role-models can offer examples of effective study strategies or describe appropriate standards of communication when conferring with professors. Tinto (2001) further suggests that peer mentor relationships can address both specific classwork and general skills associated with successful college completion. Moreover, these benefits can affect mentees as well as mentors (Good et al., 2000; Snowden & Hardy, 2012).

Mentoring has been found to be particularly beneficial for minority students (Good et al., 2000; Redmond, 2000). Redmond (2000) suggests that mentoring programs must adopt the following goals to meet the needs of diverse students: (a) promote greater student contact, (b) promote students’ use of services for support with non-academic problems, (c) intervene quickly when students encounter academic difficulties, and (d) develop culturally-sensitive psychosocial environments.

A case study for mentorship in diverse communities is the ALANA (Asian, Latin, Africa, and Native American) mentoring program, targeting minority community college students (Mueller, 1993). The stated goals of the ALANA program were to (a) provide social and academic support for minority students, (b) engage in role-play to help students critically think through

(24)

challenging situations, and (c) assist students in making time-sensitive decisions (e.g., course add/withdrawal). Mentors in the ALANA program seek to maximize social interaction with their mentees as a mechanism for relieving students’ anxieties (Mueller, 1993).

Peer mentoring has been shown to benefit students transitioning from two- to four-year

institution and those in distance education programs. For instance, Lenaburg, Auirre, Goodchild, and Kuhn (2012) reported on the impact of a program that oriented community college students to a four-year institution. As part of the program, students were provided with peer mentors. At the conclusion of the program, participants rated their peer mentor experience very highly, commenting that peer mentors were instrumental in explaining the transfer process, providing social support and helping them maintain interest in a four-year institution. Most recent results suggest that peer mentors were instrumental in helping students transition from community colleges to a four-year university. Peer mentoring has also benefitted students new to online learning contexts (Boyle et al., 2012; Brown, 2011). A study of peer mentoring programs in three distance education universities, for example, found evidence of improvement in mentees’ course passage rates, retention, and sense of belonging (Boyle et al., 2012).

Though not evaluated in the empirical literature, the University of California at Berkeley has a mentoring program for transferring community college students: the Starting Point Mentorship Program. Through this program, transferring students are paired with mentors who offer: (a) guidance, (b) motivation, and (c) access to campus and community resources. Specifically, the benefits to mentees are outlined as: advice on study skills, time management and goal-setting, information about the differences in academic and social culture between community college and a four-year institution, encouragement to set and pursue academic goals, and the point-of-view of a current student.

Despite the likelihood that peer mentoring can mitigate the shock of student transfer—either from community college to a four-year institution or from face-to-face to online environments— there have been few experimental studies directly assessing peer mentoring programs’ impact on key student indicators (Boyle et al., 2010). Further, to date, there have been no such studies of peer mentoring for students experiencing the double shock of transferring from a largely face-to-face community college to an online, four-year institution.

(25)

SECTION 3: RESEARCH SCOPE AND DESIGN

In Phase 3, research was undertaken to expand on and validate initial findings from Phases 1 and 2. In particular, variables previously identified as potentially predictive of performance and persistence at UMUC, as well as newly introduced factors, were used to predict key outcomes throughout the path model of students’ academic trajectories. The path model identifies the academic milestones along the path to completion for community college students. (See figure 1.)

Figure 1. Path model of students’ academic trajectory from community college to UMUC.

Research Questions

Predictive modeling was used to answer the following research questions related to students’ performance, persistence (re-enrollment and retention), and ultimate achievement of a credential (graduation)

Performance

1. To what extent do demographic characteristics, community college course taking behaviors, and community college performance metrics predict earning a successful first-term GPA (2.0 or above) at UMUC?

Persistence

2. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term GPA predict re-enrolling at UMUC in a semester immediately following the first semester of transfer?

3. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term GPA predict retention at UMUC, or re-enrollment within a 12-month window following the first semester of transfer?

Graduation

4. To what extent do demographic characteristics, community college course taking behaviors, community college performance metrics, and UMUC first-term metrics predict graduation from UMUC?

5. What are the graduation rates of community college transfer students at UMUC? Community

College Data

UMUC First

Term GPA Retention Graduation

Re-enrollment

(26)

Beyond building predictive models of key milestones along students’ academic trajectories, students’ experiences while enrolled at UMUC were examined. In particular, two aspects shaping students’ academic trajectories were examined.

First was an examination of students’ motivational and self-regulatory profiles as they relate to socio-demographic characteristics (e.g., employment status, family structure). Examining motivation and self-regulation as well as probing aspects of students’ background introduced a deeper examination of students’ backgrounds that may shape their experiences at both the community college and the transfer institution.

Further, data mining analyses were used to examine whether students’ engagement in the online classroom, as measured by UMUC’s LMS, was associated with performance at UMUC. This examined the extent to which students’ interactions within the online classroom was potentially facilitative of meeting academic milestones (e.g., earning a successful GPA).

These in-depth, learner-focused analyses introduced two additional research questions: Learner-Focused

6. What is the association between student motivational and self-regulatory characteristics, socio-demographic factors (e.g., employment status, family structure) and performance at UMUC?

7. What is the nature of students’ engagement in the online classroom environment and its association with successful course completion?

Student Population

The population of interest for analyses was defined as first-term undergraduate students, whose first semester of transfer to UMUC, from MC or PGCC, was between Spring 2005 and Spring 2012.

In this report, a number of outcomes reflecting student success and corresponding to key academic milestones were examined. These are defined as:

Successful first-term GPA–earning a GPA of 2.0 or above in the first semester at UMUC Re-enrollment–enrollment in the immediate next semester after initial enrollment

Retention–re-enrollment at UMUC within 12 months after initial enrollment

Graduation–earning a first bachelor’s degree from UMUC within a specified time period, specifically within 4, 6, or 8 years of transfer

Models predicting each of the target outcome variables were developed, with results presented in Sections 7 and 8. Further, learner-focused analyses were undertaken examining the relation between online course engagement and performance and motivational and self-regulatory profiles, socio-demographic characteristics and performance with results presented in Sections 9 and 10.

PREDICTIVE ANALYTICS FOR STUDENT SUCCESS: Developing Data-Driven Predictive Models of Student Success