Vol. 32, No. 6, December 2007, pp. 667–679
ISSN 0260-2938 (print)/ISSN 1469-297X (online)/07/060667–13 © 2007 Taylor & Francis
DOI: 10.1080/02602930601117191
Student evaluation of teaching quality
in higher education: development of an
instrument based on 10 Likert-scales
P. Spooren
*
, D. Mortelmans and J. Denekens
University of Antwerp, Belgium
Taylor and Francis Ltd CAEH_A_211654.sgm 10.1080/02602930601117191 Assessment & Evaluation in Higher Education 0260-2938 (print)/1469-297X (online) Original Article 2007 Taylor & Francis 32 6 000000December 2007 PieterSpooren [email protected]
Students’ evaluation of teaching skills has been an important yet controversial tool in the improve-ment of teaching quality during the last few decades. When searching for an apt student question-naire to measure instructional skills, it appeared that most existing questionquestion-naires the authors were able to collect are based on a single-item type of evaluation. Additionally, most of these instruments lack a theoretical foundation and hardly any instrument was tested with modern tests for reliability and validity. The authors managed to create a 31-item instrument which comprises 10 Likert scales and is based on both the educational theory and empirical data. In this article, they present the different steps in constructing the instrument and discuss its reliability and validity. The results of this study underline the value of the use of a scaling technique in students’ evaluation of teacher performance.
Introduction
Since the early 1970s, educational sciences and instructional psychology have paid a great deal of attention to the research on student ratings of instruction in higher education (Marsh, 1987; Chonko et al., 2002). Obviously, this theme can be called rather ‘controversial’, yet it faces us with a very interesting discussion that fell silent in scientific debates for more than a decade. Supporters state that evaluative judge-ments on a regular basis have a strong positive impact on the improvement of instruc-tional skills. According to them, these evaluations can only be useful if students, who are being educated and instructed, are involved in this form of quality care (Theall & Franklin, 2001). Opponents do not trust this type of evaluation and have their doubts as to the validity of students’ perceptions of teaching (Sproule, 2002). With regard to this matter, Marsh (1984) states that ‘opinions about the role of students’ evaluations
*Corresponding author. University of Antwerp, Faculty of Political and Social Sciences, Sint-Jacobstraat 2, B-2000 Antwerpen, Belgium. Email: [email protected]
vary from “reliable, valid and useful” to “unreliable, invalid and useless”. How can opinions vary so drastically in an area which has been the subject of thousands of studies?’ (p. 708).
When going through literature on this subject (for an extensive overview see among others Marsh, 1987; Chonko et al., 2002; Spencer & Schmelkin, 2002) one comes to the conclusion that many valid and reliable evaluation instruments that can be used by students have been developed. However, research showed the existence of factors influencing the results of students’ evaluations on the side of both teachers and students (e.g. class size, grade expectations, gender, age). According to some authors, this calls into question the validity of student perceptions of teaching (Neath, 1996; Haskell, 1997; Sproule, 2002). They consider student ratings as ‘meaningless quan-tification’ and leading to ‘personality contests’ (see Kulik, 2001), instead of being an effective measure of teaching effectiveness. On top of that, the link between student evaluations and noticeable improvement in teaching quality has been questioned. According to Marsh (1987) one of the key purposes of student evaluations of teaching is providing feedback that might lead to better teaching skills. However, Kember et al. (2002) found no evidence for the hypothesis that the use of their Student Feedback Questionnaire improved teaching performance, at least as perceived by the students. Therefore, Ballantyne et al. (2000) state that when student evaluations are used to improve teaching quality, much attention must be given to personal follow-up and correct information on the use, procedure and results of these evaluations.
The evaluation of teachers by their students has in fact been widespread on college campuses for many years. Despite the intensive use of evaluation questionnaires, assessing teaching qualities is far from evident in many cases: ‘Student ratings of university teachers have been common for at least thirty years, but it is a rare campus where they are accepted with equanimity’ (Knapper, 2001, p. 3).
As seems to be the case for several institutions of higher education, one of the key purposes of the education policy plan of the University of Antwerp (Belgium) is the gradual development of more evaluations of teachers and instructional quality, in which the students, amongst others, play an important role. In order to achieve this goal successfully, we developed an evaluation questionnaire that should allow students to share their experiences and appreciation concerning the lessons they took from the professors and their assistants. The results of these evaluations will be used for the improvement of teaching quality and tenure decisions at the University of Antwerp. The final instrument contrasts with existing instruments in its elaborateness of dimensions, its measurement of teacher quality in terms of Likert scales and its extensive testing of reliability and (construct) validity. In this article, we present the development of the instrument as well as the different tests that were undertaken to validate the instrument.
Methods
Since many evaluation instruments have already been developed, we felt no need to invent a new evaluation tool. Nevertheless, when searching for an apt instrument to
measure a teacher’s quality in a broad sense, it appeared that most existing question-naires are based on a single-item type of evaluation. In addition, the single items often reside in a large ‘stock’ of evaluation questions from which an evaluee can choose. Additionally, most of the instruments we were able to collect lack a theoreti-cal foundation or do not provide any theoretitheoreti-cal background for their users. A last consideration is the scarceness of information on the validation procedures used for these instruments. Hardly any instrument was tested with modern tests for validity and reliability.
The shortcomings of the existing evaluation batteries resulted in a research project aimed at developing a theory-based and thoroughly validated evaluation instrument that could be used in a wide variety of contexts (different stages in the curriculum, several faculties, various types of students). From the beginning of the project, a Likert-based item questionnaire was aimed at. The widely used single-item approach was abandoned due to severe methodological shortcomings. This starting point needs some further explanation and justification. Teaching skills must be considered as latent dimensions, i.e. a teaching quality is something that is not directly and imme-diately observable. A single-item approach assumes that all aspects of teaching quality can be unequivocally observed. According to this view, a response of a student on a single question makes it possible to capture a particular quality. We are convinced that measuring instructional skills on a single-item level poses serious methodological problems:
● a higher sensitivity to social desirability;
● a more ambiguous interpretation of the answers given; ● a higher sensitivity to accidental fluctuations;
● a more difficult distinction between groups in the student population (e.g.
some-what satisfied – somesome-what dissatisfied);
● a problematic way of testing both the reliability and the validity of the answers
given.
In other words, if an aspect of teaching quality (for instance presentation skills) is measured with a single question, it will be almost impossible to gain full insight into students’ feelings concerning that aspect. Students with the same attitude towards their instructor’s presentation skills will never give an identical answer to that specific question, for instance because they perceive the question in a different way or because they base their judgement of the question on other grounds. If one asks several ques-tions on (different aspects of) presentation skills, however, the chances of obtaining a truthful insight into the students’ attitude will increase.
We agree with McIver and Carmines (1981) when they state that ‘the most funda-mental problem with single-item measures is not merely that they tend to be less valid, less accurate, and less reliable than their multi-item equivalents. It is, rather, that because they provide only a single measurement, the social scientist rarely has sufficient information to estimate their measurement properties. Thus, their degree of validity, accuracy, and reliability is often unknowable’ (p. 15). Research also shows that single-item scales are less stable, less reliable, less valid and less representative
than multi-item scales (Rushton et al., Marsh et al. in Marsh, 1987). Although this seems a rather obvious observation, it is surprising that many evaluation instruments still use the single-item approach.
Even though different scaling techniques are available, we have chosen to develop a Likert scale-based instrument that consists of item sets relevant to measuring students’ attitudes towards several latent concepts (e.g. presentation skills, value of the course, clarity of objectives). The items can be scored on a scale ranging from ‘strongly disagree’ to ‘strongly agree’. The main reason for choosing the Likert-type scale is its ease of use and its straightforward quality check. Likert scales are not diffi-cult to use because the end result is a scale mean, which is easily converted to a percentage. The teaching quality component is easy to communicate, even to a non-statistically trained target public (e.g. philosophy students). Another advantage of Likert scales is the ability to test each scale for reliability by means of the Cronbach’s alpha statistic. Again, the interpretation of this statistic is far from complicated (an alpha value higher than 0.70 indicates a reliable scale) and as a consequence is easily communicated to a broader community of users.
In summary, we are convinced that when using scaling techniques (1) the validity increases, (2) the reliability of the data increases, (3) the measurement level rises (ordinal instead of categorical) and (4) the communication of the results remains comprehensible (see amongst others Swanborn, 1982; Bourque & Clarke, 1994; Spector, 1994). In other words, measuring teaching skills by using a scaling technique is more reliable, more valid and hence more useful.
Results
The evaluation instrument was constructed in a three-phased project. First, the theoretical foundations of the instrument and the empirical translation of the theo-retical concepts were established. Based on the theotheo-retical development of the qual-ities that ought to be measured by the instrument, a large set of potential scale items was constructed. In the second phase, this large battery of items was presented to a student population. The data from this survey were used to select the most valid and reliable scale items. During the third phase the selection was tested by resubmitting the short list of items to a second group of students. This phase was carried out in two steps in order to test the scale stability over time. In the following paragraphs we will go into further detail on the different stages of the instrument’s construction.
Phase 1: Construction of an extensive questionnaire based on theoretical grounds
A first step in measuring qualities of education is determining the minimal require-ments of sound educational practice. Even though it seems rather obvious, it was surprising to find that only a few instruments have a theoretical basis. During the first phase, we reviewed the literature to distinguish between and define the various characteristics of teaching in relation to the assessment of the education quality of
university courses. We do not make a strict distinction between course evaluation and teacher evaluation. This is, as Marsh (1987) argues, because student ratings of teach-ing effectiveness ‘are primarily a function of the instructor who teaches the course rather than the course that is being taught, and thus provide little information that is specific to the course’ (p. 259). Thus, a teacher and his or her course cannot be considered or rated independently.
In total, eight main dimensions were distinguished and these were further divided into 22 sub-dimensions (see Table 1). Each (sub-)dimension had a theoretical foun-dation. This was the basis for translation into empirical (measurable) terms. Based on the empirical translation of the theory, we formulated at least six Likert items that could measure each sub-dimension. For this purpose, we constructed new Likert items and complemented them with usable items from existing questionnaires. Phase 1 resulted in an extensive test questionnaire with 165 items representing all 22 sub-dimensions. As mentioned before, all items were measured on a six-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. The main reason for opting for an even scale is that answers to questions concerning teacher skills are either positive or negative, and never neutral (e.g. you liked the instructor’s lectures or you did not).
Table 1. Theoretical construction of main and sub-dimensions of teaching quality
Main dimension Sub-dimension
1. Course objectives (1) Clarity of objectives (2) Quality of objectives 2. Subject matter (3) Value of subject matter
(4) Attractiveness of subject matter (5) Build-up of the subject matter 3. Course structure (6) Linking up with advance knowledge
(7) Harmony with other courses in the programme (8) Linking up with social reality and future profession 4. Teaching activities (9) Presentation skills
(10) Harmony between objectives and organization of the course (11) Harmony between organization of the course and learning
process of the students
5. Course materials (12) Contribution to understanding the subject matter (13) Contribution to preparing for examination(s) (14) Link–up with organization of the course 6. Course feasibility (15) Course difficulty
7. Coaching (16) Help of the teacher during the learning process
(17) Contribution of the teacher to preparing for examination(s) (18) Stimulation of the teacher in order to learn to be
self-responsible
8. Evaluation (19) Transparency of the examination(s) (20) Authenticity of the examination(s) (21) Content validity of the examination(s) (22) Formative examination(s)
Phase 2: From extensive questionnaire to a 31-item test instrument
It is only in the second phase that we made a first appeal to our students. The students were asked to rate a course they attended during the previous academic year using the elaborate test questionnaire. They were instructed that the questionnaire was constructed for empirical purposes and that several items might measure similar aspects. In total, seven courses were evaluated by 433 students from six different faculties.
Using different statistical tools (Cronbach’s alpha tests, exploratory factor analysis, confirmatory factor analysis) we checked how well the items measured each of the 22 sub-dimensions. Items that did not comply with the minimal criteria of reliability and validity were deleted. The Cronbach’s alpha test and an exploratory factor analysis were used to cut back the extensive questionnaire to 31 items. These items repre-sented 10 of the 22 original sub-dimensions. With the Cronbach’s alpha testing we checked the consistency of all items in a scale for each course, using the ‘alpha if item deleted’ criterion to increase the internal consistency of the scale. Exploratory factor analysis was used in order to detect items that – according to the empirical data – did not belong to their sub-dimension or had high loadings on several sub-dimensions. These techniques allowed us to delete 134 items from the extensive questionnaire.
Subsequently, we introduced the 31 remaining items in a confirmatory factor anal-ysis using the Lisrel program. This technique is the best qualified to extensively test the validity and reliability of the different scales. Table 2 summarizes the results of this analysis.
The confirmatory factor model shown in Table 2 provides a reasonable fit to the data. The χ2 test of exact fit is significant, whereas the objective is to achieve a non-significant p-value. However, Hatcher (1994, p. 289) indicates that a non-significant χ2 does not make a confirmatory factor analysis model inadequate. The χ2 ratio shows that the ratio of the χ2 value and the degree of freedom is lower than two (1.57). This indicates that the χ2 test lies within acceptable limits (Marsh, et al. 1988). In terms of validity of the constructs, convergent validity is evidenced by the large and significant loadings of the items on their posited indicators. Further evidence of convergent validity is shown in Table 3. None of the correlations between the latent constructs is too high to challenge the convergent validity of the constructs.
Discriminant validity of the construct is shown because the confidence interval (± two standard errors) around the correlation estimate between any two latent constructs never equals 1.0 (Anderson & Gerbing, 1988, p. 416). The Variance Extracted Test also demonstrates the discriminant validity of our constructs. This test compares the variance extracted from two latent constructs with the square of the correlation between these two constructs (Fornell & Larcker, 1981). Discriminant validity is shown whenever the explained variance is greater than the squared corre-lation. All pairs of factors have been compared and they all showed an acceptable vari-ance extracted.
At the conclusion of phase two, the end result was a smaller instrument with 10 reliable and valid scales. However, these items were not tested as an independent
Table 2. Remaining scales in the confirmatory factor analysis Standardized
factor loadings t-test
Probability t-test R2 Variance extracted F 1: clarity of objectives (ρc = 0.96) 0.88 Item 11 0.92 55.32 0.001 0.84
Item 12 (‘Throughout the course, it was made clear what I should learn and accomplish as part of this course’)
0.98 79.12 0.001 0.97
Item 13 0.92 52.40 0.001 0.84
F 2: value of subject matter (ρc = 0.90) 0.75
Item 21 0.91 53.17 0.001 0.82
Item 22 (‘Some topics covered in the course are, in my opinion, completely superfluous’)
0.82 36.90 0.001 0.67
Item 23 0.87 40.07 0.001 0.76
F 3: build-up of subject matter (ρc = 0.91) 0.77
Item 31 0.86 41.51 0.001 0.74
Item 32 0.95 53.10 0.001 0.91
Item 33 (‘The congruence of the themes presented in this course was clear to me’)
0.82 36.37 0.001 0.67
F 4: presentation skills (ρc = 0.94) 0.85
Item 41 0.94 75.66 0.001 0.89
Item 42 0.91 55.03 0.001 0.83
Item 43 (‘The lectures of the instructor were clearly presented’)
0.92 62.60 0.001 0.84
F 5: harmony organization course – learning process (ρc = 0.85)
0.58 Item 51 (‘The instructor invited
discussion’)
0.77 31.48 0.001 0.59
Item 52 0.85 38.44 0.001 0.72
Item 53 0.67 23.62 0.001 0.45
F 6: (course materials) contribution to understanding the subject matter (ρc = 0.91)
0.72
Item 61 0.90 45.48 0.001 0.81
Item 62 (‘The study material presented by this instructor was not organized at all’)
0.84 38.42 0.001 0.70
Item 63 0.90 47.31 0.001 0.81
Item 64 0.75 31.60 0.001 0.56
F 7: course difficulty (ρc = 0.86) 0.68
Item 71 (‘The expectations of the instructor with regard to what we should know and accomplish were realistic’)
0.94 47.68 0.001 0.88
Item 72 0.79 31.47 0.001 0.63
Table 2. (continued) Standardized factor loadings t-test
Probability t-test R2
Variance extracted F 8: help of the teacher during the learning
process (ρc = 0.91)
0.77
Item 81 0.86 44.37 0.001 0.73
Item 82 (‘The instructor helped me with questions and problems which arose during this course’
0.89 42.38 0.001 0.79
Item 83 0.88 44.87 0.001
F 9: authenticity of the examination(s) (ρc = 0.83)
0.63
Item 91 0.68 22.79 0.001 0.47
Item 92 (‘During the examination the instructor expected me to have thought critically about the subject material, rather than having memorized everything’)
0.84 33.79 0.001 0.71
Item 93 0.84 34.53 0.001
F 10: formative examination(s) (ρc = 0.83) 0.61
Item 101 0.80 28.94 0.001 0.63
Item 102 (‘The feedback I received throughout the course was very helpful with regard to further studying the material’)
0.86 30.40 0.001 0.73
Item 103 0.69 21.32 0.001 0.48
Notes: Scale composite reliability: (Bagozzi & Yi, 1988, p. 80). Fit statistics for confirmatory factor analysis of 31 indicators for 10 constructs: χ2(701) = 1102.58, p = 0.00; GFI = 0.96; CFI = 0.97; PNFI = 0.79; NNFI = 0.97; RMSEA = 0.039.
ρc=
(
λi) ( )
ξ λi ξ θii
∑
(
∑
) ( )
+∑
2 2
var / var
Table 3. Construct correlation matrix
Construct F1 F2 F3 F4 F5 F6 F7 F8 F9 F2 0.68 F3 0.79 0.81 F4 0.75 0.65 0.73 F5 0.68 0.59 0.67 0.65 F6 0.73 0.66 0.84 0.72 0.57 F7 0.79 0.62 0.68 0.61 0.56 0.65 F8 0.83 0.63 0.71 0.68 0.78 0.67 0.80 F9 0.43 0.38 0.42 0.48 0.68 0.29 0.23 0.36 F10 0.40 0.18 0.40 0.29 0.37 0.36 0.31 0.41 0.08
instrument. Phase two was no more and also no less than a test case in which 165 items were reduced to 31 items. In order to demonstrate the quality of the selected items, we presented them to a new sample of students in phase three.
Phase 3: Testing the instrument
The confirmatory factor analysis model allowed us to develop a valid 31-item evalu-ation instrument that can be used for students’ evaluevalu-ations of university teaching. In the third phase, we put the instrument to the test and made a new appeal to our students. As in the second phase, we asked them to evaluate a particular course they attended during the previous academic year. This time the participants used the new instrument. The students were asked to complete the questionnaire two times (at an interval of one or two weeks). They were also asked to provide some identification details, which were used to link the questionnaires from both moments. The evalua-tion results for eight courses (completed by 566 students) allowed us to execute a number of reliability and stability tests.
An easy way to investigate the (retest) reliability of an instrument is the construc-tion of a transiconstruc-tion matrix and the use of this matrix in reliability tests. To be more specific, for each item we set up a frequency table in which the results for this item at moment 1 were crossed with the results for this item at moment 2. For each table we executed a Cohen’s Kappa consistency test and calculated the Kendall’s Tau B and Spearman correlation coefficient. Cohen’s Kappa is a statistical measure indi-cating to what extent two respondents (in our case student X who evaluated course Y twice with the same instrument) match concerning their opinion on a certain issue (in our case item Z in the instrument). Kappa is expressed in a value between 0, standing for coincidence, and 1, suggesting perfect agreement (Liebetrau, 1994; Ho Yu, 2005). Kendall’s Tau B indicates to what extent rankings made up by a number of people correspond (e.g. from ‘very negative’ to ‘very positive’). In our case, this meant examining whether or not a student gave two identical answers to an item at both test moments (e.g. his/her answer to the question was twice ‘strongly agree’). The Kendall’s Tau value is expressed in a number between 1 (perfect agreement) and – 1 (opposite appraisal) (Liebetrau, 1994; Ho Yu, 2005). Spearman’s correlation coefficient is probably the best known and most widely used way to calculate the relation between two series of rankings. Like Kendall’s Tau, a value of 1 indicates a perfect agreement and –1 denotes the opposite views (Liebetrau, 1994).
The results of these tests (Table 4) show that the 31 items included in our test instrument are reliable. The only deviation is the recurrence of rather low Kappa values. This might be related to the use of the six-point scales on which the items are scored. When the six categories are reduced to three categories (‘disagree’, ‘neutral’ and ‘agree’), the Kappa values are remarkably higher. In other words, working with fewer scale values increases the reliability of an instrument. Conversely, working with more scale values increases the sensitivity of the instrument. Because the Kappa values are only affected by the number of response categories, we can conclude that
Table 4. Validity and reliability tests of the evaluation instrument Internal consistency (Cronbach’s alpha) t 1 t 2 Cohen’s Kappa (t 1 – t 2) (p = .00) Kendall’s Tau (t 1 – t 2) (p = .00) Spearman’s Corr. (t 1 – t 2) (p = .00) F 1: clarity of objectives 0.781 0.782 Item 11 0.337 0.601 0.682 Item 12 0.390 0.516 0.613 Item 13 0.366 0.556 0.652
F 2: value of subject matter 0.714 0.758
Item 21 0.373 0.510 0.688
Item 22 0.404 0.454 0.510
Item 23 0.389 0.559 0.638
F 3: build-up of subject matter 0.668 0.729
Item 31 0.334 0.283 0.487 Item 32 0.286 0.502 0.548 Item 33 0.397 0.573 0.635 F 4: presentation skills 0.898 0.875 Item 41 0.550 0.793 0.861 Item 42 0.567 0.756 0.814 Item 43 0.401 0.593 0.673
F 5: harmony organization course – learning process 0.663 0.760 Item 51 0.438 0.549 0.608 Item 52 0.279 0.497 0.571 Item 53 0.251 0.514 0.606 F 6: (course materials) contribution to
understanding the subject matter 0.875 0.856 Item 61 0.548 0.738 0.797 Item 62 0.356 0.629 0.745 Item 63 0.453 0.602 0.713 Item 64 0.440 0.512 0.567 F 7: course difficulty 0.835 0.858 Item 71 0.473 0.698 0.761 Item 72 0.376 0.601 0.666 Item 73 0.435 0.692 0.756
F 8: help of the teacher during the learning process
0.765 0.730
Item 81 0.408 0.509 0.556
Item 82 0.228 0.558 0.636
the retest reliability is not compromised. All 10 latent dimensions remained stable in the third phase.
Summary
Students’ evaluation of teaching skills in higher education has been an important tool in the improvement of teaching quality during the last few decades. In the education policy of the University of Antwerp it is stated that the university will develop more systematic evaluations of the instructional quality. These evaluations will be done on a regular basis in which students will play an important role. This being the case, this article presents the extensive development process of a 31-item student questionnaire based on a theoretical framework and empirical research. In three subsequent research phases, 10 Likert scales were constructed allowing students to evaluate and rate the courses they attended as well as their teachers’ instructional skills. According to us, the use of scaling techniques in questionnaires concerning teacher evaluation makes these evaluations more valid, more reliable and thus more useful.
A valid and reliable instrument has been developed, based on empirical data collected by asking students to evaluate a course they attended. Both an extensive questionnaire (phase 2) and a test instrument (phase 3) were used. This test instru-ment underwent a number of reliability and validity tests and is now ready for implementation.
The results of this study underline the value of the use of scaling techniques in students’ evaluations of teacher performance. In comparison with the single-item approach, scale-type evaluations measure instructional skills (which must be seen as latent constructs) better since they are less sensitive to social desirability, ambiguous
Table 4. (continued) Internal consistency (Cronbach’s alpha) t 1 t 2 Cohen’s Kappa (t 1 – t 2) (p = .00) Kendall’s Tau (t 1 – t 2) (p = .00) Spearman’s Corr. (t 1 – t 2) (p = .00) F 9: authenticity of the examination(s) 0.858 0.833 Item 91 0.351 0.543 0.609 Item 92 0.336 0.552 0.611 Item 93 0.281 0.497 0.581 F 10: formative examination(s) 0.779 0.833 Item 101 0.468 0.705 0.793 Item 102 0.403 0.528 0.589 Item 103 0.381 0.613 0.709
interpretations and accidental fluctuations of the answers given. In addition, the internal consistency of each scale is easily tested by calculating the Cronbach’s alpha statistic.
The construction of valid and reliable scales requires systematic research, in which both the literature and empirical data should play an important role. This type of preliminary research does not yet seem to take place in all institutions of higher education. This strengthens the arguments of those who are concerned about the reli-ability, validity and thus the usefulness of students’ perceptions of teaching.
Notes on contributors
Pieter Spooren holds a master’s in educational sciences. Since 2004 he has been affiliated as a staff member in the Innovation and Quality of Education Centre at the Faculty of Political and Social Sciences of the University of Antwerp (Belgium). His particular activities are educational innovation and evaluation of the educational process and of educators.
Dimitri Mortelmans, PhD, is Assistant Professor in Sociology at the Faculty of Political and Social Sciences of the University of Antwerp (Belgium). He teaches qualitative research techniques and advanced statistics. He is head of the Panel Survey of Belgian Households (PSBH). His principal research interests lie in the sociology of family and youth. His main research topics cover divorce, work–life balance and leisure time of youngsters. He is also head of the Innovation and Quality of Education Centre.
Joke Denekens, MD PhD, is a general practitioner at Mechelen in a group practice with four general practitioners and one trainee. She is full professor in General Practice and head of the Department of General Practice at the University of Antwerp. In this function she is responsible for the undergraduate, graduate and postgraduate education of general practitioners. She is vice-rector of the University of Antwerp. Her particular areas of activity are: curriculum innova-tion in the bachelor/master context, organizainnova-tion and implementainnova-tion of inno-vative actions, and evaluation of the educational process and of educators.
References
Anderson, J. C. & Gerbing, D. W. (1988) Structural equation modeling in practice: a review and recommended two–step approach. Psychological Bulletin, 103, 411–423.
Bagozzi, R. P. & Yi, Y. (1988) On the evaluation of structural equation models. Academy of Marketing Science, 16, 74–94.
Ballantyne, R., Borthwick, J. & Packer, J. (2000) Beyond student evaluation of teaching: identify-ing and addressidentify-ing academic staff development needs. Assessment & Evaluation in Higher Education, 25(3), 221–236.
Bourque, B.B. & Clarke, V.A. (1994) Processing data: the survey example, in: M. S. Lewis-Beck (Ed.) Research practice: international handbook of quantitative applications in the social sciences (Vol. 6) (London, Sage Publications).
Chonko, L. B., Tanner, J. F. & Davis, R. (2002) What are they thinking ? Students’ expectations and self-assessments, Journal of Education for Business (May/June), 271–281.
Fornell, C. & Larcker, D. F. (1981) Evaluating structural equation models with unobservable vari-ables and measurement error, Journal of Marketing Research, 18, 39–50.
Haskell, R. (1997) Academic freedom, tenure, and student evaluation of faculty: galloping polls in the 21st century, Education Policy Analysis Archives, 5(6). Available online at http://epaa.asu. edu/epaa/v5n6.html (accessed 10 July 2006).
Hatcher, L. (1994) A step-by-step approach to using the SAS system for factor analysis and struc-tural equation modelling (Cary, CN, SAS Institute).
Ho Yu Chang (2005) Test retest reliability, in: K. Kempf-Leonard (Ed.) Encyclopedia of social measurement (Vol. 3) (Oxford, Elsevier).
Kember, D., Leung, D. & Kwan, K. (2002) Does the use of student feedback questionnaires improve the overall quality of teaching?, Assessment & Evaluation in Higher Education, 27(5), 412–425.
Knapper, C. (2001) Broadening our approach to teaching evaluation, New Directions for Teaching and Learning (88, Winter), 3–9.
Kulik, J. A. (2001) Student ratings: validity, utility and controversy, New Directions for institutional Research, 27(5), 9–25.
Liebetrau, A. M. (1994) Measures of association, in: M.S. Lewis-Beck (Ed.) Basic statistics: inter-national handbook of quantitative applications in the social sciences (Vol. 1) (London, Sage Publications).
Marsh, H. W. (1984) Students’ evaluations of university teaching: dimensionality, reliability, validity, potential biases and utility, Journal of Educational Psychology, 76(5), 707–754. Marsh, H. W. (1987) Students’ evaluations of university teaching: research findings,
methodologi-cal issues, and directions for further research, International Journal of Educational Research, 11(3), 253–388.
Marsh, H. W., Balla, J. R. & McDonald, R. P. (1988) Goodness-of-fit indexes in confirmatory factor analysis: the effect of sample size, Psychological Bulletin, 103, 391–410.
McIver, J. P. & Carmines, E. G. (1981) Unidimensional scaling. Sage University Paper series on Quantitative Applications in the Social Sciences, 24–001. (Newbury Park, CA, Sage Publications).
Neath, I. (1996) How to improve your teaching evaluations without improving your teaching, Psychological Reports, 78, 1363–1372.
Spencer, K. J. & Schmelkin, L. P. (2002) Student perspectives on teaching and its evaluation, Assessment & Evaluation in Higher Education, 27(5), 397–409.
Spector, P.E. (1994) Summated rating scale construction, in: M. S. Lewis-Beck (Ed.) Basic measure-ment: international handbook of quantitative applications in the social sciences (Vol. 4) (London, Sage Publications).
Sproule, R. (2002) The underdetermination of instructor performance by data from the student evaluation of teaching, Economics of Education Review, 21, 287–294.
Swanborn, P.G. (1982) Schaaltechnieken: theorie en praktijk van acht eenvoudige procedures [Scaling Techniques: Theory and Practice of eight elementary procedures] (Meppel, Boom).
Theall, M. & Franklin, J. (2001) Looking for bias in all the wrong places: a search for truth or a witch hunt in student ratings of instruction?, New Directions for institutional Research, 27(5), 45–56.
