Analysis of student evaluation of teaching
2.2 Validity of course evaluations
Although, most of the researchers may agree that student evaluations of teach- ing are reliable tools, there is somewhat less consensus regarding their overall validity. Validity determines if what is supposed to be measured by SETs is be- ing measured in reality. Researchers have tried different approaches, collecting data that either support or contest the conclusion that student ratings reflect the effectiveness of teaching.
A historical overview of the research by Greenwald (1997) notes that the major- ity of publications produced during the period 1975-1995 indicate that course evaluations are valid. McKeachie (1997) also concluded that the student course evaluations are the “single most valid source on teaching effectiveness”.
Researchers have tried to compare the results of the student evaluations to dif- ferent measures of student learning such as students actual or expected grades (Feldman, 1989a; Cohen, 1981) or to other evaluations of teacher effectiveness such as instructor self-evaluation (Feldman, 1989a; Marsh and Dunkin, 1992), ratings by administrators (Kulik and McKeachie, 1975; Feldman, 1989b), peer evaluations (Kulik and McKeachie, 1975; Feldman, 1989b), and alumni evalua- tions (Overall and Marsh, 1980; Braskamp and Ory, 1994; Hobson and Talbot, 2001).
There are various ways to asses student learning. The obtained grade is one of the most popular measure of students learning used in literature. There are a number of studies which tried to compare multiple-section courses, the courses where different parts of the course are performed by different teachers using the same syllabus and textbook and external exam. Cohen (1981) and Feldman (1989a) made a review of these studies. The reported relationships confirmed the validity of SET, the classes where teachers get higher SET scorer students also scored higher on the external exam (Cashin, 1995). More discussion on the relationship between the student ratings and grades is presented in section ??. Instructor self-evaluation is another tool used to establish validity of SETs. Feld- man (1989b) provides a list of 19 papers that investigated the correlation be- tween student evaluations and instructor self-evaluations. The average correla- tion is 0.29.
Feldman (1989b) provided a review of the research comparing the SETs made by current and former students, colleagues, administrators, external (neutral) observers, and the teachers themselves. The author reports the average corre- lation between student ratings and ratings by administration to be 0.39 (based on results of 11 studies) and between students ratings and peer-ratings to be
2.3 Online-based vs paper-based course evaluations 21
0.55 (based on results of 14 papers).
To address the issue that the current students may not adequately judge the long-term effects of instruction some studies checked how the student ratings are related to the retrospective ratings of the same instructor provided by the same students several years later. Different studies report such a correlation to be quite high. Feldman (1989b) reports an average correlation to be 0.69, based on results of 6 cross-sectional studies. The study by Overall and Marsh (1980) showed large and statistically significant correlations between end-of-term and retrospective ratings. The authors analyzed the data of more than 1,000 under- graduate and graduate business administration students from different classes that provided feedback at the end of each class, and again at least one year after program completion.
Some studies used external observers, who were trained to make classroom ob- servations. Feldman (1989b) in his meta-study reported an average correlation of 0.50 between the student ratings and ratings of trained observers. Kulik (2001) in his review mentions a careful study by Murray (1983), where 6 to 8 trained observers visited classes taught by 54 university lecturers receiving ei- ther low, medium, or high student ratings in an earlier semester. Highly rated teachers tend to receive high scores and low-rated teachers tend to receive low scores from observers, especially in such teaching qualities as clarity, enthusiasm, and rapport.
2.3
Online-based vs paper-based course evalua-
tions
The traditional way to obtain student evaluations of the course and the teacher is to distribute printed questionnaires and survey forms among students at the end of the course, while more modern techniques are based on on-line questionnaires. The in-class paper-based method of conducting evaluations is less likely to suffer from the effects of non-response, because most of the active students are assumed to be in class when evaluations are conducted. However many universities have switched to the online-based student evaluations, that offer several advantages over paper-and-pencil evaluations.
For the web-based course questionnaires students can respond outside of class at their convenience (Dommeyer et al., 2002; Layne et al., 1999), therefore it requires less class time. Additionally, web-based questionnaires provides a less
expensive method of collecting course evaluation results and can provide results immediately
A literature review by Anderson et al. (2005) suggest that there is some evidence that the Web-based evaluation methods lead to lower response rates. Hoverer, study by (Avery et al., 2006), that analysed SET results conducted on-line or on paper of 29 courses taught between 1998 and 2001, and study by Fike et al. (2010), that employs the sample of student evaluations of 247 courses, showed that there is no evidence that evaluation scores change when evaluations are completed online rather than on paper. Lower response rates may occur for several reasons: students concern about anonymity, computer technical difficul- ties, and the time required to respond outside of class.
Dommeyer et al. (2002) analysed settings where sixteen professors who taught two sections of the same class and were randomly assigned to have one of their sections evaluated by the in-class method and the other by the on-line method. The traditional paper based method had a higher response rate than the web- based method. During the post-evaluation survey the on-line responders com- plained that web-based evaluation process may not be anonymous and that the log-on system was time-consuming. Analysis of almost 2500 students, who were randomly assigned to either the traditional or the electronic evaluation by Layne et al. (1999) also showed that students were more likely to evaluate their teachers when the evaluations were conducted in class.
Additionally, Layne et al. (1999) found that average ratings did not differ be- tween the two methods of conducting SET. This fact is also confirmed by further investigation by Dommeyer et al. (2004) and Donovan et al. (2006), who anal- ysed 11 courses with settings similar to Dommeyer et al. (2002).
Donovan et al. (2006) found differences between the two methods of conducting SETs in number and length of comments. Students completing faculty evalua- tions online wrote more comments, and the comments were more often formative in nature. Layne et al. (1999) also found that the response rates to open-ended questions posted on-line tend to be higher. Hardy (2003) examined 26 classes in which the same instructor taught the same class multiple times. The classes using paper rating forms which were compared with the classes evaluated online. The overall response rate in the classes evaluated online was lower. However, students wrote many more comments and students who respond online write more detailed comments. Similar findings are also presented in works by An- derson et al. (2005); Johnson (2003a); Kasiar et al. (2002); Ballantyne (2000). Crews and Curtis (2011) analysed the data gathered from university faculty, who transitioned from traditional paper to online course evaluations. Authors provides some suggestions for universities transitioning from traditional paper-
2.4 Faculty, Administrator and Student Perceptions of Course Evaluations23
based course evaluations to web-based course evaluations. One of the most important suggestions is providing adequate training for faculty members on the on-line course evaluation system, ensuring that when students withdraw from a course that they are dropped from the evaluation system, sharing with faculty strategies to increase the student response rate.
Norris and Conn (2005) suggested that usage of reminder e-mails from instruc- tors or university administration and messages posted on on-line class discussion boards and forums can increase response rates. Dommeyer et al. (2004) showed that when a grade incentive (one quarter of one percent for any student who had completed the on-line evaluation) was used to encourage response to the online survey, a response rate was achieved that was comparable with that to the in-class survey.
Introduction of the web-based evaluation systems had a positive effect on the way the results are summarized and used. It enabled an easy comparison of the courses within each department or university/college. Overall, theoretical and practical considerations, seem to lead to the conclusion that the online- based SETs bring remarkable advantages. The cost reductions, time savings and the ease of calculation of results, overcome the disadvantages of web-based evaluation surveys.