Online Course Evaluations

(1)

Stanford University

Office of the University Registrar

Online Course

Evaluations

Collecting, Consolidating, and Leveraging Data

Jackie Charonis

(2)

Stanford University

Private research University founded in 1891

Our Student Body

6,689 Undergraduates 8,201 Graduates

Our Faculty

1,807 Tenure-line faculty

(3)

Student Systems at Stanford

PeopleSoft SA 8.0 – student records database

Resource 25/Schedule 25 3.3 – scheduling software OnBase - document imaging

(4)

Developing the System: Our Requirements

Vendor-delivered ASP solution: CollegeNET

Integrated approach: “Stanford” presence and security

via our portal

Self-service application collects and displays results Paper-free process including email reminders &

announcements

(5)

Our Design Considerations

No changes to existing evaluation forms One approach for all participating schools Ease of use (self-service model)

Ease of management (paper-free, hassle-free) Confidentiality

at least 3 students enrolled in the course or section combined courses receive one summary of aggregate

data

Validity - only one evaluation per enrolled student per

course

Accuracy - data validated by third-party statistical

(6)

Our Implementation Timeline

Winter 2005-06 Pilot

2 volunteer departments

85 classes and no discussion sections Spring 2005-06 Early Adopters

25 volunteer departments

659 classes and 241 discussion sections Summer 2005-06 No Paper

Autumn 2006-07 All departments (excl. professional schools) 1758 courses and 721 discussion sections

(7)

Collecting Data: The Forms

Two forms currently used Course Form

Section Form

Only officially scheduled courses and enrolled students

(8)

Collecting Data: Communication

Email notifications to instructors and students when

evaluation period opens

Email reminders

Email notifications to instructors when data is available

(9)

Collecting Data: Timing & Incentives

Evaluation Period

Evaluations open two weeks at end of term incl. finals Results available following grading deadline

Grade Withholding

Grades are always viewable in system to staff and

available on official transcripts

Grades are released daily for students’ view who

complete all their evaluations

Grades available for those who do not complete

(10)

Our Response Rates

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Autumn Winter Spring

(11)

Consolidating Data: Basic Reporting

Means by Department, School, and Areas

Histograms by Department, School, and Areas Individual Course Summaries

(12)

(13)

(14)

(15)

Leveraging Data: Aspects of Courses

Class size and course level

Class type (e.g., lecture vs. seminar) Effects of team teaching

(16)

Sample Analysis: Single vs. Multiple Instructors

3.5 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 5.0 Q01-02 Q03-07 Q08-09 Q10-12 Q13-17 Q18 Qu e s ti o n #

(17)

Sample Analysis: Single vs. Crosslisted Courses

3.5 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 5.0 Q01-02 Q03-07 Q08-09 Q10-12 Q13-17 Q18 Qu e s ti o n #

Mean Course Evaluation Rating

(18)

Leveraging Data: Aspects of Instructors

Same course but different instructors Tenured vs. untenured instructors

(19)

Sample Analysis: Faculty vs. Visiting Instructors

3.5 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 5.0 Q01-02 Q03-07 Q08-09 Q10-12 Q13-17 Q18 Qu e s ti o n #

Mean Course Evaluation Rating

(20)

(21)

Leveraging Data: Aspects of Students

(22)

Evaluating the Evaluation Process

Did the change in medium effect the results?

Did the change in timing of the evaluation period

adversely effect the results?

Is there meaning behind blank submissions

(23)

Examples of Analysis: Paper vs. Electronic

"Difference" between 2006-07 (online) and 2005-06 (paper) area means (mean=-0.020, stdev=0.090, n=306) 0 0 0 0 0 0 0 0 0 0 0 0 1 2 3 1 1 11 16 59 89 68 36 14 2 ₁ ₁ 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 40 50 60 70 80 90 100 -1.00 -0.95 -0.90 -0.85 -0.80 -0.75 -0.70 -0.65 -0.60 -0.55 -0.50 -0.45 -0.40 -0.35 -0.30 -0.25 -0.20 -0.15 -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

Difference Score (2006-07 minus 2005-06 for areas with 10 or more evals)

F

req

uenc

y

(24)

The Future of Course Evaluations

Benchmarking

Evaluating learning rather than teaching

Evaluating the impact of instructor training programs Developing systems for reviewing qualitative data

(25)

(26)

Questions & Answers

Jackie Charonis

Assistant Vice Provost for Student Affairs & Associate University Registrar

[email protected] Paul Casey

Associate Vice President - Sales