BIG DATA AND
ACCREDITATION
CHEA 2014 Summer Workshop
Positioning Accreditation: Role, Responsibilities and Expectations
June 25, 2014
Implementing analytics and applying it to
make data driven decisions is a major
differentiator between high performing and
low performing organizations.
Big Data: The Next Frontier for Innovation, Competition and Productivity McKinsey Global Institute 2012
Analytics: The Game
Changer for Higher
“No More Excuses”
Goals of Arizona State University 2002
• Increasing graduate numbers
• Graduation rates
• Freshman retention rates
• Expand ethnic and economic diversity
Outcomes in 2011
• Increased enrollment 30% in 10 years
• Increased minority enrollment as % of total population by 52%
• Increased degrees awarded by 52%
• Increased 6-year grad rate by 19%
• Increased freshman persistence to 84% up 9%
Solutions
• Comprehensive use of analytics
If you use these analytical tools, you will know • where you are,
• what you’re doing,
• if what you are doing is working or not
• whether or not you need to be doing new things
customized to fit your particular school or demographic
• infinitely more information to help students be successful
Why Study Data Analytics?
“…to address societal imperatives, higher education must begin by transforming its own culture, which is reflected in the questions we ask (and those we don’t), the
achievements we measure and highlight (and those we ignore), and the initiatives we support (or don’t support).”
-- Dr. Freeman Hrabowski President of University of Maryland, Baltimore County
• Expectations for accreditation and external
accountability are increasing, so that it is no longer sufficient for institutions to have assessment plans.
Instead, institutions strive to build a culture of evidence with examples of how assessment results are used to improve student learning.
• Very seldom do institutions now complete their
reaccreditation without including language about work that needs to be done regarding the collection of
student learning outcomes assessment evidence and using that evidence to improve.
Baker, Gianina R., Natasha A. Jankowski, Staci Provezis, and Jillian Kinzie. Using Assessment Results: Promising Practices of Institutions That Do It Well. NILOA July 2012
QUESTIONS
•
What is big data?
•
Why is it important?
•
How is big data tied to accreditation?
•
What does big data have to do with
•
Big Data has captured public attention as a
critical tool to advance data analytics,
visualization and customized services to
consumers. This trend is affecting higher
education with more data available on student
readiness, progression and success. While
much is known about what works to improve
student persistence, completion, and
interventions that increase the likelihood of
success, it takes local will to make sense of
and act on this information.
What is Big Data?
• Big data is a blanket term for any collection of data
sets so large and complex that it becomes difficult to
process using on-hand database management tools or traditional data processing applications.
• The challenges include capture, curation, storage,
search, sharing, transfer, analysis and visualization.
• More data may lead to more accurate analyses.
• More accurate analyses may lead to more confident
decision making
What Accreditors Need to
Know about Big Data
• Most institutions are collecting evidence of student
learning, but it is not clear how those results are being used to improve student outcomes
• “Mountains of data, very little action.”
• Institutions are at many levels of capacity in the use of
data and analytics to improve decision making
• Institutions need to link data with action and targeted
While Big Data raise expectations,
Connecting the Dots
Student
Success
Assessment Accountability Analytics AccreditationConnecting the Dots
•
Assessment
•
Accountability
•
Accreditation
•
Analytics
•
Concrete Measures
•
Concrete Targets
•
Concrete Quality Assurances
•
Concrete Practices
Accreditation
• Advanced academic quality
• Demonstrate accountability through consistent, reliable
information about academic quality and student
achievement that fosters continuing public confidence and investment
• Encourage where appropriate self scrutiny and planning for
change and needed improvement
• Employ appropriate and fair procedures in decision making
• Demonstrate ongoing review of accreditation practices
• Possess sufficient resources that are predictable and
stable
Accreditation
Assessment
•
Institutional performance
•
Program performance
•
Student performance
Accountability
•
Proven metrics
•
Progress over time
•
Culture of inquiry
•
Internal and external drivers
Analytics
Definition: Analytics is the use of data,
statistical analysis and explanatory
and predictive models to gain insights
and act on complex issues.
http://www.educause.edu/library/resources/2012-ecar-study-analytics-higher-education
STRATEGIC QUESTION DATA ANALYSIS AND PREDICTION INSIGHT AND ACTION
ANALYTICS
http://www.educause.edu/library/resources/2012-ecar-study-analytics-higher-educationStrategic Intelligence for Higher Education
How many, how often, where?
Where exactly is the problem? What actions are needed? Why is this happening?
What if these trends continue?
What will happen next?
Key Questions Addressed by
Analytics
Past Present Future
Information Insight What happened? (Reporting) What’s happening Now? (Alerts)
What will happen?
(Extrapolation)
What’s the
best/worst that can happen?
(Prediction, Optimization, Simulation) What’s the next
best action?
(Recommendation) How and why
did it happen?
(Modeling, Experimental design)
Obligation of Knowing
We now have models for improving student
success. What is our obligation to do
something about it?
–
Maximize the availability and use of data
–
Unlock the silos
–
Involve faculty in meaningful ways
–
Change approach to the facilitation of
learning
–
Empower students to know how they are
doing and what they can do to improve
Linking Analytics to
Student Success
Critical is the issue of “closing the loop” of using
assessment evidence to improve student
learning and inform curricular decisions. After
scouring the literature and discussing the
question, Banta and Blaech 2011 found that only
six percent of cases actually could link to student
learning improvement.
Implications for
Accreditation
Higher Learning Commission
4.C. The institution demonstrates a commitment to educational improvement through ongoing attention to retention,
persistence, and completion rates in its degree and certificate programs.
1. The institution has defined goals for student retention,
persistence, and completion that are ambitious but attainable and appropriate to its mission, student populations, and
educational offerings.
2. The institution collects and analyzes information on
student retention, persistence, and completion of its programs. 3. The institution uses information on student retention,
persistence, and completion of programs to make improvements as warranted by the data.
http://www.higherlearningcommission.org/Information-for-Institutions/criteria-and-core-components.html?highlight=WyJwZXJzaXN0ZW5jZSIsImNvbXBsZXRpb24iXQ==
Example of a Retention, Persistence and Completion
Checklist
CRITERIA
• The institution has defined goals for student retention, persistence, and
completion.
• The institution collects and analyzes information on student retention,
persistence, and completion of its programs.
• The institution uses information on student
retention, persistence, and completion of programs to make improvements as warranted by the data.
EVIDENCE
Measures, metrics, targets
Data warehouse, mining, analysis, assessment, accessibility, action
Improvement activities, interventions, resource allocation
Professional Accreditation
and Analytics
• Beyond reporting to action
• Competencies and skills
• Progress in programs
• How to improve student achievement
• How to continue to link to national professional
WHAT ARE INSTITUTIONS
DOING WITH BIG DATA?
Areas Where Analytics Can Improve
Student Success
• Early predictions of “at-risk” students
• Likelihood of success in courses, programs of study
• Course sequencing
• Likelihood of attending class next week
• “Students like you…” scenarios
• Relevant, readily available information about where students
are in a class for students, advisors and faculty
• Targeted interventions to match student needs
• Early alerts and kudos
• Improved alignment of student’s interests, skills, and
P
REDICTIVE
A
NALYTIC
R
EPORTING
F
RAMEWORK
(PAR)
predictors
learner characteristics
learner behaviors
academic integration
social-psychological integration
other learner support
course/program characteristics
instructors behaviors
D
ATA
I
NPUTS
Student Course Information Course Location Subject Course Number Section Start/End Dates Initial/Final Grade Delivery Mode Instructor Status Course Credit Student Financial Information
FAFSA on File – Date Pell Received/Awarded – Date Student Academic Progress Current Major/CIP Earned Credential/CIP Course Catalog Subject Course Number Subject Long Course Title Course Description Credit Range Lookup Tables
Credential Types Offered Course Enrollment Periods
Student Types Instructor Status Delivery Modes Grade Codes Institution Characteristics Possible Additional ** Placement Tests NSC Information SES Information Satisfaction Surveys College Readiness Surveys
Intervention Measures ** Future Student Demographics & Descriptive Gender Race Prior Credits Perm Res Zip Code
HS Information Transfer GPA
Action Behind Analytics
Know What Works
Customize Interventions
• Bridging Programs
• Learning Communities
• Active Learning
• First Year Experiences
• Course redesign
• Capstone courses
• Supplemental Instruction
• Intrusive Advising
• Longitudinal data system
to track course patterns
• Multiple intervention
strategies – academic and student support
• Build from learning
management systems
• Act soon enough to make
a difference
Analytics Maturity Index
http://www.educause.edu/ecar/research-publications/ecar-analytics-maturity-index-higher-education
QUESTIONS FOR ACCREDITORS
•
How does big data affect the assurance of
quality?
•
What do accreditors need to know?
•
What are institutions doing with big data that is
important to accreditors as they assess
academic quality?
•
As an accreditation reviewer, what evidence do
you need from the institution?
•
Should there be a template for guiding
QUESTIONS?
References
• AACU High Impact Practices http://www.aacu.org/leap/index.cfmC&U
• Association of Community College Trustees. 2013. Student Success Toolkit. http://governance-institute.org/toolkit
• Baer, Linda and John Campbell. 2012. From Metrics to Analytics, Reporting to Action: Analytics’ Role in Changing the Learning Environment. In Game Changers, edited by Diana Oblinger. http://www.educause.edu/library/resources/chapter-4-metrics-analytics-reporting-action-analytics%E2%80%99-role-changing-learning-environment
• Bean, John P. and Barbara Metzner 1985 A Conceptual Model of Nontraditional Undergraduate Student Attrition in Educational Research Winter, 1985, Vol.55, No 4, 485-540.
• Compete College America www.completecollegeamerica.org
• Crow, Michael. No More Excuses in EDUCAUSE Review, vol. 47, no. 4 July/August 2012
http://net.educause.edu/ir/library/pdf/ERM1241P.pdf
• Davenport, Thomas. http://www.slideshare.net/sasindia/keynote-thomas-davenportanalyticsatwork Retrieved November 23, 2013
• Gilbert, C., M. Eyring, and R. N. Foster. “Two Routes to Resilience.” Harvard Business Review, December, 2012, 65–73.
http://hbr.org/2012/12/two-routes-to-resilience/ar/1.
• Graduate School Metrics. Association of American Universities http://www.aau.edu
• Jones. Dennis. 2013. Outcomes-Based Funding: The Wave of Implementation in September 2013. National Center for Higher Education Management Systems
• Kamenetz Anya. 2012 Fast Company 2012 Most Innovative Companies 2012 Southern New Hampshire University
http://www.fastcompany.com/3017340/most-innovative-companies-2012/12southern-new-hampshire-university
• University of Illinois Graduate Model. http://www.grad.illinois.edu/sites/default/files/researchschemagraphic.png
• Kuh. George and Jillian Kinzie, John H. Schuh, Elizabeth J. Whitt and Associates. 2010. Student Success in College: Creating Conditions that Matter
• National Commission on Higher Education Attainment. 2013. An Open Letter to College and University Leaders: College Completion Must Be Our Priority. American Council on Education. Washington D.C.
• Norris, Donald and Robert Brodnick, Paul LeFrere, Joseph Gilmour, Linda Baer, Ann Hill Duin and Stephen Norris. 2013. Transforming in an Age of Disruptive Change. Strategic Initiatives, Inc. and the Society for College and University Planning • Predictive Analytics Reporting Framework. [email protected]
• Sowell, Bell and Kirby Ph.D. Completion and Attrition: Policies and Practices to • Promote Student Success. Council of Graduate Schools. 2010
• Student Success Matrix (SSMX ) A model classifying influences on student success within the PAR Project WCET Annual Meeting
Presentation Mindy Sloan, Ashford University, Karen Swan, University of Illinois Springfield, Michelle Keim, Bridgepoint Education, Heidi Hiemstra, Predictive Analytics Reporting (PAR) Framework. November 15, 2013
• Shugart, S. M. 2012. The Challenge to Deep Change: A Brief Cultural History of Higher Education. Planning
for Higher Education, December 28. Retrieved January 15, 2013, from the World Wide Web:
http://mojo.scup.org/forum/topics/the-challenge-to-deep-change-a-brief-cultural-history-of-higher.
• Tinto, Vincent. 2012a. Leaving College: Rethinking the Causes and Cures of Student Attrition. University of
Chicago Press.
