A Quality Scorecard
for the Administration
of Online Education
Programs
Kaye Shelton, Ph.D.
The Sloan Consortium
Focus
Introduce the Quality Scorecard
Explain the history and rationale
Why use the Scorecard?
How to use the interactive Scorecard
Handbook as a guide for completing
To develop a quality
scorecard to evaluate
the administration of
online education
programs.
Purpose
The Need For This Study
Quality Assurance (Bates & Poole, 2003; Meyer, 2004; Sallis, 1996).
From the beginning, the quality of online education
was questioned (Benson, 2007).
Standards must be clearly defined (Institute for Higher
Education Policy, 1998, 2000; Khan, 2005; Lee & Dziuban, 2002; Leh & Jobin, 2002; Meyer, 2002; Onay, 2002; Shelton & Saltsman, 2005; Stella & Gnanam, 2004; Suryanarayanaravu, Srinivasacharyulu, & Mohanraj, 1995).
Tool for evaluation, benchmarking, and strategic
Quality Assessment in
Higher Education
Rankings in US News and World Report
National Survey of Student Engagement (NSSE) Regional and discipline specific accreditation
Need for continuous improvement strategy
Public accountability
Total Quality Management Balanced Scorecard
Malcolm Baldrige National Quality Award
From the Literature…
Several recommended groups of standards (14
different articles and studies)
WCET, CHEA, IHEP 24 Quality Standards , Bates’
ACTION model, Sloan Consortium Five Pillars
The IHEP 24 standards was highly referenced in
the literature and therefore used as a beginning list of indicators.
Methodology
Delphi Method
Structured flow of information using a series of systematic
surveys and reciprocal feedback
Used to gain consensus from a panel of experts
(informed judgments)
• The experts were widely located throughout the United States
Results were fed back to the panel of experts in
Sample (Expert Panel)
Study Population: Online Education Administrators
in Higher Education
Sampling Frame: Identified experts in the field by
the Sloan Consortium (gatekeeper)
76 were invited; 43 participants completed first
round
83% of the panel members had nine or more
years of experience in the administration of online education
Panel Member Experience
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 5 years or less 7 years or less 9 years or less 10 or more years 9.5% 7.1% 14.3% 69.0%Sample Distribution
Institutional Classification Type Size Total
Public (4 year) Non-profit Large 24
Public Community College (2 year) Non-profit Large 2
Private (4 year) Non-profit Large 4
Private (4 year) For-profit Large 1
Private Faith-Based (4 year) Non-profit Large 1
Public (4 year) Non-profit Medium 2
Private (4 year) Non-profit Medium 3
Private Faith-based (4 year) Non-profit Medium 3
Public (4 year) Non-profit Small 1
Panel Participation
Delphi Round Total Experts Enlisted Total Experts Who Completedthe Survey Response Rate
I 44 43 97.7% II 43 41 95.5% III 38 33 86.8% IV 33 30 90.9% V 30 28 93.3% VI 28 26 92.9%
Research Questions
1. Are the standards identified in the IHEP/NEA
study in 2000 still relevant today for indicating quality in online education programs in higher education?
2. What additional standards should be included
that address the current industry in 2010?
3. If additional standards are suggested, will they
fall into the already identified themes or will new themes emerge?
Research Questions
4. What values will be assigned to the
recommended standards that will ultimately yield a numeric scorecard for measuring quality online education programs from an online education
administrator’s perspective that could also support strategic planning and program
Instrumentation
Combination of open and closed questions
The 24 quality standards identified by IHEP for Round I Open-ended for additional standards and categories. Survey Monkey - Web-based survey tool
Five point Likert Scale:
1=Definitely Not Relevant 2=Not Relevant
3=Slightly Relevant 4=Relevant
Delphi Survey
Each survey round was open for two weeks
After one week, a follow-up email was sent
Data analyzed to develop next round survey
Mean scores/suggestions for additional
quality standards and standards revisions
were fed back in the next survey round.
Data Analysis
Descriptive Statistics
Mean Scores and Standard Deviation
Mean score of 4 and above with 70% or more agreement
were included in subsequent surveys
Those with mean scores of below 4 were fed back one
additional time for final disapproval or during the first few rounds.
If consensus was not achieved, 70% of response
Results
Are the standards identified in the IHEP/NEA study
in 2000 still relevant in 2010 for indicating quality in online education programs in higher education?
A form of all 24 indicators were included in the
quality scorecard (23 relevant, 1 completely revised)
Results
What additional standards should be included that
address the current industry in 2010?
The panel suggested a total of 80 potential
quality indicators.
• 45 quality indicators were approved and included in the scorecard.
Adding these 45 indicators to the 25 indicators
stemming from the IHEP study yielded a total of 70 quality indicators.
Results
If additional standards are suggested, will they fall
into the already identified themes or will new themes emerge?
Twenty were suggested: Three categories achieved
consensus;
• Only two were added to the scorecard: Technology Support (divided from the original IHEP Institutional Support) and Student and Social Engagement.
The Instructional Design category that achieved panel
Indicators by Category
and Consensus
Category Total Number of Suggested Quality Indicators
Total Number Approved by the Panel of Experts
Percent Achieving Consensus
Institutional Support 10 4 40%
Technology Support 4 3 75%
Course Development and Instructional Design
11 8 72%
Teaching and Learning 6 2 33%
Course Structure 12 5 42% Student Support 16 11 69% Faculty Support 6 3 50% Evaluation and Assessment 14 8 57%
Social and Student Engagement
Results
What values will be assigned to the recommended
standards that will ultimately yield a numeric
scorecard for measuring quality online education programs from an online education administrator’s perspective that could also support strategic
Guidelines
for Scoring
0 points = Not Observed. The administrator does not
observe any indications of the quality standard in place.
1 point = Insufficiently Observed. The administrator has
found a slight existence of the quality standard in place. Much improvement is still needed in this area.
2 points = Moderate Use. The administrator has found there
to be moderate use of the quality standard. Some improvement is still needed in this area.
3 points = Meets Criteria Completely. The administrator has
found that the quality standard is being fully implemented and there is no need for improvement in this area.
Guidelines for Scoring
A perfect score = 210 points.
90-99% = 189-209 - Exemplary (little improvement is
needed)
80-89% = 168-188 - Acceptable (some improvement is
recommended)
70-79% = 147-167 - Marginal (significant improvement is
needed in multiple areas)
60-69% = 126-146 - Inadequate (many areas of
improvement are needed throughout the program)
Course Development and
Instructional Design
Teaching and Learning
Interactive Scorecard
On the Sloan-C website
Must have an institutional membership
Justification notes
Upload artifacts for support
Organizational chart
Faculty training materials Student survey results
Sloan-C Five Pillars of Quality
Learning Effectiveness
Scale (Cost Effectiveness and
Commitment)
Access
Faculty Satisfaction
Why Bother?
Self-study in quality
Could be reviewed by accrediting
agencies
Strategic planning
Next Steps
Developed additional
support materials that
further explain each
indicator
Process for benchmarking
Determine minimum
scores for each category
Community of practice
Next Steps
Community of Practice
Ongoing discussion
Opportunity for scorecard refinement
Benchmarking process
Anonymous
Expert Panel
Shirley Adams Matt Baker Arnold Bateman Jennifer Baysden Rhonda Blackburn Kathleen Burke Tom Cavanagh Nan Chico Nancy Coleman Sharon Davis Mary Fehr Katherine Galaitsis Gwendolyn Godard Carmen Gonzalez Jo Hamilton Kenneth Hartman Margaret Haughey Mary Hawkins Cheryl Herdklotz Lisa Holstrom Janet Ross Kendall Karen King Jon Lenrow Frank McCluskey Darrell Naylor-Johnson Loyce Pailen Latonia Pernell Alexandra M. Pickett Patrick Pluscht Maria Puzziferro Rob Robinson George Saltsman Jodi Servatius Scott Shiffer Mike Simmons Wayne Smutz James Stenerson Helen Torres Ken Udas
Marvin Van Kekerix Karen Vignare Roger Von Holzen Ginger Zillges
Institutions Represented
Abilene Christian University American Public University System Athabasca University
Bellevue University Boston University
California State University East Bay Central Texas College
Charter Oak State College
Dallas Baptist University Drexel University
Online
Drexel University
East Carolina University
East Tennessee State University Franklin University
Michigan State University New Mexico State University
North Carolina Agricultural & Technical State
University
Northwest Missouri State University Pace University
Peirce College
Pennsylvania State World Campus
Regent University
Rochester Institute of Technology
Rocky Mountain College of Art and Design San Antonio College
Savannah College of Art and Design State University of New York
Texas Tech University Texas Woman's University
The George Washington University The University of Texas at Dallas UMassOnline
University of Central Florida University of Cincinnati
University of Maryland University College University of Massachusetts, Boston University of Nebraska-Lincoln
University of North Texas
University of Texas Telecampus University of Wisconsin-Madison University Wisconsin-Milwaukee Washington State University
Questions
How is this different from Quality Matters?
QM is a course rubric, this examines the entire program.
Why are these quality indicators important?
Because a panel of experts were brought to consensus
through the Delphi process. (43 very seasoned administrators)
Can I use the scorecard freely without constraints?
The scorecard is free to use.
To use the interactive scorecard on the website, you must
Questions
Will Sloan-C be convening special interest groups
around this? Commons site
Why should Sloan-C endorse this scorecard since
it doesn’t have everybody’s input?
Panel of experts with most 9 or more years
Inviting you to join our commons site and participate in
ongoing discussions.
Why is the scorecard input based and not output?
The scorecard offers broad goals for outputs but focusing
Questions
Can I give the scorecard to others?
Yes, share the link
• http://tinyurl.com/qualitysc
Will my information be private or public?
Private, we will not share the information you provide.
Will we be comparing scores?
We are working on a process that would compare them
anonymously by groups of peer institutions.
How can we use the results from others to help?
To see where other programs’ strengths are.
Why isn’t the score normalized (100 points)?
The expert panel felt that the graduated scale of 0-3 points per
How can I obtain a consultation on my program?
We are developing a business model now and have some
of the expert panelist that are using the scorecard and could help with evaluation.
Send requests to kshelton@sloanconsortium.org
What if I already have a list of standards that I
use?
The scorecard should mesh with other standards easily The Sloan-C pillars of quality aligned easily.