PLEASE SCROLL DOWN FOR ARTICLE

(1)

Access Details: [subscription number 777157963] Publisher: Routledge

Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Studies in Higher Education

Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713445574

Students' perceptions of teaching quality in higher

education: the perspective of currently enrolled students

Paul Ginnsa; Michael Prosserb; Simon Barriea a_{University of Sydney, Sydney, Australia} b_{University of Hong Kong, Hong Kong} Online Publication Date: 01 October 2007

To cite this Article: Ginns, Paul, Prosser, Michael and Barrie, Simon (2007) 'Students' perceptions of teaching quality in higher education: the perspective of currently enrolled students', Studies in Higher Education, 32:5, 603 - 615 To link to this article: DOI: 10.1080/03075070701573773

URL:http://dx.doi.org/10.1080/03075070701573773

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use:http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

(2)

Downloaded By: [University of Sydney] At: 04:10 5 March 2008

Vol. 32, No. 5, October 2007, pp. 603–615

DOI: 10.1080/03075070701573773

Students’ perceptions of teaching

quality in higher education:

the perspective of currently enrolled

students

Paul Ginns

a

*

, Michael Prosser

b

and Simon Barrie

a

a_{University of Sydney, Sydney, Australia;}b_{University of Hong Kong, Hong Kong}

Taylor and Francis Ltd CSHE_A_257233.sgm 10.1080/03075070701573773 Studies in Higher Education 0307-5079 (print)/1470-174X (online) Original Article 2007 Society for Research into Higher Education 32 5 000000October 2007 Dr PaulGinns [email protected]

The psychometric properties of a version of the Course Experience Questionnaire revised for students currently enrolled at the University of Sydney, the Student Course Experience Question-naire (SCEQ), were assessed, gathering students’ perceptions on a number of scales, including Good Teaching, Clear Goals and Standards, Appropriate Assessment, Appropriate Workload, and an outcome scale measuring Generic Skills development. Confirmatory factor analyses supported the hypothesised factor structure, and estimates of inter-rater agreement on SCEQ scales indicated student ratings of degrees can be meaningfully aggregated up to the faculty level. The authors discuss the SCEQ’s usage as an integral part of a broader quality assurance programme at the University of Sydney, including benchmarking relationships with other universities.

Student Evaluations of Teaching (SET) represents one of the most voluminous research literatures in applied psychology. In his review of this corpus, Marsh (1984) concluded that subject-average ratings (e.g. ratings of a semester-long subject in educational psychology taught by a single lecturer) are: (1) multidimensional; (2) reli-able and streli-able; (3) more a function of the instructor who teaches a course than the course that is taught; (4) relatively valid against various indicators of effective teaching; (5) relatively unaffected by variables often purported to be biases; and (6) considered useful by faculty as feedback on teaching, by students in course selection, and by administrators for personnel decisions. While the bulk of this research has focused on ratings of individual classes, particularly ratings of individual teachers and their prac-tices, other researchers have focused on students’ perceptions of the learning environ-ment across their entire degree, and how these perceptions are related to approaches

*Corresponding author: Institute for Teaching and Learning, 3rd Floor Carslaw Bldg F07,

(3)

to study and subsequent learning outcomes (e.g. Ramdsen & Entwistle, 1981; Entwistle & Ramsden, 1983; Crawford et al., 1998; Lizzio et al., 2002).

Universities around the world are exploring the use of teaching performance indicators for performance-based funding and for benchmarking purposes (Marsh

et al., 2002; Prosser & Barrie, 2003; Barrie et al., 2005). Since 1993, the Graduate Careers Council of Australia has included the Course Experience Questionnaire (CEQ: Ramsden, 1991; Wilson et al., 1997) as part of its annual Graduate Destination Survey (see www.avcc.edu.au/students/gradlink/GCCA/index.htm), and the UK has recently developed a similar national survey (see www.hefce.ac.uk/learning/nss/). The CEQ is designed as a performance indicator of teaching and learning at the degree level, as opposed to the level of individual subjects. It is derived from the student learn-ing framework, which holds that students’ approaches to study are contingent upon both their prior experiences of teaching and learning and their perceptions of current learning contexts, in turn affecting learning outcomes (e.g. Marton & Säljö, 1976; Biggs, 1999; Prosser & Trigwell, 1999; Ramsden, 2003). Crawford et al. (1998) demonstrated the links between such perceptions and outcomes using cluster analysis. They analysed first-year university mathematics students’ responses on modified forms of the CEQ and the Study Process Questionnaire (Biggs, 1987), and found students who took a deep approach to study also perceived that the teaching was good, the goals and standards were clear, and that there was some emphasis on indepen-dence. In contrast, students who adopted a surface approach perceived that workload was too high, and that assessment was inappropriate (focusing on rote recall). Students in the former group outperformed those in the latter group on achievement tests. For a recent meta-analysis of the correlates of approaches to learning, including perceptions of the learning environment and learning outcomes, see Watkins (2001). The CEQ has a substantial literature addressing its reliability and validity in a variety of settings (e.g. Ramsden, 1991; Trigwell & Prosser, 1991; Richardson, 1994, 2005a, b; Sadlo, 1997; Wilson et al., 1997; Richardson & Woodley, 2001; Lawless & Richardson, 2002; Lizzio et al., 2002; Byrne & Flood, 2003; Espeland & Indrehus, 2003). Accompanying Australia’s annual Graduate Destination Survey of undergrad-uate and postgradundergrad-uate coursework students since 1993, this questionnaire assesses a range of student perceptions related to teaching and learning, as well as overall satisfaction as a validity check on the other scales. The form of the CEQ used between 1993 and 2001 had five scales and one item evaluating overall satisfaction with degree quality; Table 1 gives these scales and an exemplar item for each scale.

The results of the CEQ are used by a wide range of stakeholders, including the Australian Commonwealth Government, tertiary institutions, researchers in higher education; and prospective students, through the Good Universities Guide annual series (e.g. Ashenden & Milligan, 2002).

As a performance indicator, the CEQ is somewhat limited by its long ‘lag’. The survey is conducted the year after students graduate. In some cases, several years may pass before students exposed to an institutional programme (e.g. for improving clarity of goals and standards in the first year of a given degree) graduate and rate their perceptions of this aspect of their university experience on the CEQ. To address this

(4)

shortfall, the University of Sydney modified the CEQ for use with currently enrolled students. The modified CEQ was called the Student Course Experience Question-naire (SCEQ). By drawing upon a reliable and valid instrument for measuring the quality of the student experience, credible to teaching staff, researchers and the Australian Commonwealth Government, the university aimed to retain the focus on the student experience that the CEQ supports, and benefit from the established psychometric properties of the CEQ, while allowing additional analyses that the CEQ does not. Specifically, by surveying students across entire degree programmes using the SCEQ, results can be disaggregated to specific years of study, allowing more focused consideration of results without the time lag delay than was possible with graduate data from the CEQ. An example of such disaggregation is given by Barrie

et al. (2005). They presented evidence, based on SCEQ responses from commencing first-year students, that an extended institution-wide First Year Experience initiative to improve the experiences of such students was associated with concurrent improve-ments in SCEQ responses between 1999 and 2003. Moreover, the First Year Expe-rience initiative’s planning at the inception of the project in 2000 was based on SCEQ data in relation to the experiences of these particular students. Had the CEQ data been used to inform the planning of the project, there could be no certainty that the situation had not changed in the four or five years since graduates were in their first

Table 1. Course experience questionnaire and student course experience questionnaire scales

and exemplar items

Exemplar item

Scale

Course Experience Questionnaire

Student Course Experience Questionnaire

Good Teaching (6 items) The teaching staff normally gave

me helpful feedback on how I was going.

The teaching staff normally give me helpful feedback on how I am going.

Clear Goals & Standards (4 items)

The staff made it clear right from the start what they expected from students.

Appropriate Assessment (3 items)

The staff seemed more interested in testing what I had memorised than what I had understood. (reversed item)

Appropriate Workload (4 items)

The sheer volume of work to be got through in this course meant it couldn’t all be thoroughly comprehended. (reversed item)

The sheer volume of work to be got through in this degree means it can’t all be thoroughly comprehended. (reversed item)

Generic Skills (6 items) My course helped me to develop

the ability to plan my own work.

My degree course has helped me to develop the ability to plan my own work.

Overall Satisfaction with degree quality (1 item)

Overall, I was satisfied with the quality of this course.

Overall, I was satisfied with the quality of this degree course.

(5)

year. Moreover, there would have been a considerable disjunction in terms of the evidence for the impact of this institutional programme, as CEQ data on impact would not have been available until 2005 at the earliest, with a considerable likelihood that the impact of the programme might have been ameliorated in subsequent years. As a survey of enrolled students, then, the SCEQ provides academic staff and academic managers with a timelier indicator of teaching and learning quality. Given that the SCEQ is being used by the university both for performance-based funding and for national and international benchmarking purposes, it is important that its psychometric properties are confirmed for its use with enrolled undergraduate students. As noted above, Marsh (1984) argued that subject-average SET results are the appropriate level of analysis; hence, the validity and reliability of SET responses at the subject level should be demonstrated, not simply at the level of individual responses. Furthermore, Marsh et al. (2002, p. 317) noted, ‘[a]lthough it may be possible to construct an alternative set of items that would capture the quality of a course or program that was reasonably independent of the effects of specific teachers, there is little empirical support for this possibility in the student evaluation literature’. The CEQ and SCEQ differ from the types of student evaluation instruments Marsh (1984) discussed, in that they refer to students’ experiences of teaching and learning at the degree level, rather than the individual subject level. Previously published research on the factor structure of the CEQ (e.g. Ramsden, 1991; Wilson et al., 1997) reported statistically significant department-level mean differences as evidence of discriminant validity. If the SCEQ scales are to be used as performance indicators, then the appropriateness of students’ responses on such teaching performance indi-cators for creating faculty-level scores needs to be demonstrated. In this article, the first set of analyses examines the psychometric properties of the SCEQ through confir-matory factor analysis and reliability analysis, using the combined data set from the 2001 and 2002 surveys. The second set of analyses estimates the degree of inter-rater agreement (Chan, 1998; Burke & Dunlap, 2002; Dunlap et al., 2003) within faculties for each of the SCEQ scales using the above data set. Such analyses are commonly used in organisational research as evidence that agreement at a lower level (e.g. indi-vidual ratings of organisational climate) justifies aggregating lower-level scores to represent scores at the higher level (see Chan [1998] for a typology of composition models). Thus, the overall goal of the analyses is to examine the fitness for purpose of the SCEQ within our university, where that purpose is to use the SCEQ scales as performance indicators of undergraduate teaching quality at the faculty level.

Method Participants

Stratified random sampling of the student population was used to reduce the cost of data collection associated with a census of all students. The sampling frame used was designed to ensure that the sample was of sufficient size and of a composition that permitted meaningful conclusions to be made on the basis of the data. The total

(6)

population was first divided into the key strata of interest in the final analysis. For the SCEQ the strata of particular interest are: (1) faculties, (2) aggregated degree courses and (3) commencing first-year students (aggregated degrees are groupings of similar degrees, which are used by the university’s Office of Planning and Statistics for administrative purposes).

The size of the top-level stratum sample was determined based on the nature of the characteristics of the data gathered from the 1999 census survey of undergraduate students. To ensure that all aggregated degrees were represented in the faculty sample, the target number of students from each degree was calculated based on the proportion of students from each degree present in the population of the faculty. In addition, the university is particularly interested in the experiences of first-year students. To ensure that meaningful conclusions could be drawn from the data in relation to the experiences of these students, the random sample was further stratified to ensure that a representative number of first-year students from each faculty was included. As students are surveyed late in the second semester of each year, there is sufficient time for commencing first-year students to form meaningful impressions of core aspects of the teaching environment across their entire degree (e.g. regarding the quality of teaching, assessment, etc.)

The present study examines the psychometric properties of responses to the SCEQ scales using the combined 2001 and 2002 undergraduate data sets. The overall response rates for these surveys were 54% and 53% respectively. The combined 2001 and 2002 data set consisted of 7632 respondents, who provided a complete set of responses to all items. The combination of the two years of data reflects the usage of the data within the university for performance-based funding of teaching: funding decisions are made on the basis of a two-year rolling average of teaching performance indicators, with SCEQ data forming a core part of this funding system.

Materials

The SCEQ is an annual survey of a random stratified sample of University of Sydney students. The SCEQ asks students about their perceptions of a variety of factors related to how they experience their course, using a five-point Likert scale marked ‘Strongly Disagree’, ‘Disagree’, ‘Neutral’, ‘Agree’ and ‘Strongly Agree’. Scales and sample items are given in Table 1. Other items and scales in this section of the survey, developed as indicators of University of Sydney-specific strategic priorities, measure intellectual motivation, learning community, learning resources, graduate attributes, supportiveness of the degree course administration, and benefits of contact with active researchers. The following analyses will not include these items and scales, as they do not form the core of the SCEQ.

Analyses

Because the SCEQ has a clear theoretical lineage, confirmatory rather than explor-atory factor analysis, using LISREL 8.54 (Jöreskog & Sörbom, 2001) were conducted

(7)

to test whether students’ responses could be clearly differentiated into the expected underlying dimensions, using data from the 2001 and 2002 University of Sydney SCEQ surveys. While the SCEQ is based on the CEQ, it is nonetheless distinctive in two major ways. Firstly, items are written in the present, rather than the past tense, to reflect the experience of currently enrolled students rather than graduates of degree programmes. Secondly, the CEQ Likert scale used by Australian universities is anchored only by two descriptors, ‘Strongly Disagree’ and ‘Strongly Agree’, whereas, as mentioned above, the SCEQ uses anchors for each point on the five-point scale. Together, these differences from the CEQ mean the psychometric functioning of the SCEQ should not be taken for granted, and that such evidence should be provided.

Prior to these analyses, univariate and multivariate normality statistics were calculated using PRELIS 2.54; these statistics indicated the presence of considerable non-normality (univariate and multivariate skewness and kurtosis) in the data, violat-ing a key assumption of structural equation modellviolat-ing (Raykov & Marcoulides, 2000; Boomsma & Hoogland, 2001). To take account of this violation of assumptions, the confirmatory factor analysis used robust maximum likelihood estimation, a method less sensitive to violations of the normality assumption with large models than other estimation methods (Boomsma & Hoogland, 2001).

The analysis was conducted in two parts. First, a highly restricted five-factor model, with no error covariances, was fitted to the combined 2001 and 2002 data set using confirmatory factor analysis. The measurement properties of each scale were assessed with reliability analysis, using Cronbach’s (1951) estimate of the reliability coefficient, including 95% confidence intervals for each estimate of coefficient alpha (Fan & Thompson, 2001). Fit indices reported for the five-factor model include the comparative fit index (CFI), and the Relative Fit Index (RFI), for which values greater than 0.90 indicate good fit; the root mean square error of approximation (RMSEA) and its 90% confidence interval, for which values below 0.05 indicate good fit, and values up to 0.08 indicate moderately good fit; and the standardised root mean square residual (SRMR), for which values below 0.05 indicate good fit (Spec-tor, 2001). The CFI and RFI measures of fit compare the proposed model to the independence model, in which all variables are uncorrelated. The RMSEA gives an estimate per degree of freedom of the discrepancy between the model and population covariance matrices. The SRMR is the mean difference, based on standardised resid-uals, between the predicted and observed variances and covariances in the model.

Following the above analyses, we calculated inter-rater agreement for each SCEQ scale, for each faculty. These analyses assess the extent to which multiple raters, using typical Likert scale formats (e.g. 1 = ‘Strongly Disagree’ to 5 = ‘Strongly Agree’), rate aspects of an organisation’s culture or climate similarly (Chan, 1998; Lindell, 2001). Evidence of suitable levels of inter-rater agreement supports arguments for the aggre-gation of ratings at a lower level (e.g. quality of teaching at the degree level) to repre-sent levels of a phenomenon at a higher level (e.g. quality of teaching at the faculty level). As SCEQ scales are comprised of multiple items, we calculated a form of inter-rater agreement, AD_M(j), using Burke and Dunlap’s (2002) recommended method; a version of this measure suitable for the Overall Satisfaction with Degree Quality

(8)

item was calculated using methods described in Dunlap et al. (2003). The AD (aver-age deviation) index is calculated by ‘determining the extent to which each item rating differs from the mean (or median) rating, summing the absolute values of these deviations … and dividing the sum by the number of deviations’ (Burke & Dunlap, 2002, p.160). Burke and Dunlap provide a rationale for setting the upper cut-off limit for acceptable inter-rater agreement as the number of response options divided by 6; in the present study, the Likert scale used had five options, making the recommended cut-off level equal to 0.83 for aggregation of individual results to the faculty level.

Results

Table 2 gives the completely standardised factor loadings from the five-factor measurement model, and the correlations between latent factors. The hypothesised measurement model had good fit to the data by conventional criteria (Spector, 2001), CFI = 0.97, RFI = 0.96, RMSEA = 0.049 (90% confidence interval 0.048–0.051), SRMR = 0.049.

Table 2. Completely standardised factor loadings for SCEQ items

Factor loading GTS CGS AAS AWS GSS

Good Teaching Scale (GTS)

3. The teaching staff normally give me helpful feedback on how I am going.

.70 0 0 0 0

8. The teaching staff of this degree course motivate me to do my best work.

.75 0 0 0 0

16. The staff make a real effort to understand difficulties I may be having with my work.

.69 0 0 0 0

19. My lecturers are extremely good at explaining things. .62 0 0 0 0

21. The teaching staff work hard to make their subjects interesting. .62 0 0 0 0

27. The staff put a lot of time into commenting on my work. .65 0 0 0 0

Clear Goals and Standards Scale (CGS)

6. I have usually had a clear idea of where I am going and what is expected of me in this degree course.

0 .70 0 0 0

12. It is always easy to know the standard of work expected. 0 .70 0 0 0

25. The staff made it clear right from the start what they expected from students.

0 .72 0 0 0

29. It has often been hard to discover what is expected of me in this degree course.*

0 .71 0 0 0

Appropriate Assessment Scale (AAS)

13. The staff seem more interested in testing what I have memorised than what I have understood.*

0 0 .73 0 0

(9)

Table 3 gives estimates of Cronbach’s alpha and 95% confidence intervals around these estimates. These results indicate, for all scales, a range of plausible estimates of scale reliability that are clearly acceptable by typical psychometric conventions (e.g. Schmitt, 1996).

Inter-rater agreement analyses

Table 4 gives average deviation (AD) estimates of inter-rater agreement across scales and faculties. Estimates range from 0.49 to 0.87, with the majority of estimates falling under Burke and Dunlap’s (2002) recommended cut-off of 0.83. Amongst the scales, six faculties had AD estimates of inter-rater agreement on the Clear Goals and

Table 2. (Continued.)

Factor loading GTS CGS AAS AWS GSS 26. To do well in this degree all you really need is a good

memory.*

0 0 .66 0 0

Appropriate Workload Scale (AWS)

2. There is a lot of pressure on me as a student in this degree course.*

0 0 0 .63 0

4. The workload is too heavy.* 0 0 0 .73 0

15. I am generally given enough time to understand the things I have to learn.

0 0 0 .63 0

24. The sheer volume of work to be got through in this degree means it can’t all be thoroughly comprehended.*

0 0 0 .70 0

Generic Skills Scale (GSS)

5. The degree course has helped me develop my ability to work as a team member.

0 0 0 0 .43

9. The degree course has sharpened my analytic skills. 0 0 0 0 .72

10. As a result of my degree course, I feel confident about tackling unfamiliar problems.

0 0 0 0 .72

18. The degree course has developed my problem-solving skills. 0 0 0 0 .73

22. The degree course has improved my skills in written communication.

0 0 0 0 .47

23. My degree course has helped me to develop the ability to plan my own work.

0 0 0 0 .56

Factor correlations

Good Teaching Scale 1

Clear Goals and Standards Scale .61 1

Appropriate Assessment Scale .42 .24 1

Appropriate Workload Scale .35 .35 .43 1

Generic Skills Scale .57 .38 .30 .11 1

(10)

Standards Scale that were higher than the recommended cut-off, but these differ-ences from the recommended cut-off were not overly large in magnitude. In general, these results support the aggregation of students’ ratings of their degrees up to the level of faculty, and the use of such aggregated results for internal benchmarking within our university.

Discussion

The present study builds upon previous research on the Course Experience Question-naire (CEQ) in several ways. Firstly, it demonstrated that the factor structure of the CEQ is replicated when currently enrolled students, rather than graduates, are surveyed as part of a large-scale survey of teaching quality extending across all facul-ties within our university. The results indicate that the SCEQ has a clear, interpret-able factor structure for undergraduate students, using a highly restricted hypothetical model of the covariation between items. While some improvement in model fit may have been achieved by investigating covariances between error terms, the fit of the hypothesised model had good levels of fit by conventional criteria, such that any post hoc adjustments would violate the principle of parsimony of explana-tion. Reliability analyses indicated that the hypothesised scales have acceptable psychometric properties for undergraduate students. The appropriateness of aggre-gating SCEQ responses at the faculty level was also demonstrated using estimates of inter-rater agreement. This result supports the use of SCEQ scores for purposes such as competitive performance-based funding of teaching at the faculty level, as well as other uses such as Academic Board faculty reviews (for a description of the different components of the university’s system of performance-based funding of teaching, see www.usyd.edu.au/learning/quality/td.shtml).

The public reporting of such results, then, as well as the system of faculty-based working groups formed around major strategic projects, supports the communication of alternative policies and practices between faculties towards the ultimate goal of improving the student experience.

It is important to recognise that the SCEQ, like the CEQ, is not designed for use as a tool for gathering specific diagnostic feedback about particular subjects or teach-ers; as Wilson et al. (1997, p. 48) note, ‘such specific data are best obtained through customised surveys and qualitative processes’. While low scores on particular scales

Table 3. Reliability estimates of SCEQ scales (Cronbach’s α and 95% confidence intervals),

combined 2001 and 2002 data set

Scale Cronbach’s α

Good Teaching Scale 0.83 (95% CI 0.826–0.837)

Clear Goals and Standards Scale 0.80 (95% CI 0.781–0.805)

Appropriate Assessment Scale 0.72 (95% CI 0.703–0.725)

Appropriate Workload Scale 0.76 (95% CI 0.753–0.770)

(11)

Downloaded By: [University of Sydney] At: 04:10 5 March 2008 T able 4. Inter-Ra ter ag reement f aculty estima

tes for SCEQ scales using a

v erage devia tion metr ic Faculty Good Teaching Clear Goals and Standards Appropriate Assessment Scale Appropriate Workload Scale Generic Skills Scale Overall Satisfaction Item Agriculture 0.81 0.80 0.79 0.79 0.70 0.70 Architecture 0.80 0.86 0.61 0.78 0.70 0.78 Arts 0.76 0.84 0.74 0.76 0.66 0.57 Conservatorium of Music 0.81 0.85 0.72 0.80 0.80 0.75 Dentistry 0.71 0.77 0.82 0.75 0.62 0.67

Economics and Business

0.74 0.77 0.78 0.79 0.69 0.70

Education and Social Work

0.80 0.84 0.74 0.78 0.72 0.65 Engineering 0.72 0.73 0.80 0.78 0.70 0.74 Health Sciences 0.72 0.77 0.80 0.83 0.67 0.61 Law 0.79 0.80 0.73 0.81 0.62 0.58 Medicine 0.73 0.87 0.73 0.79 0.56 0.48 Nursing 0.77 0.80 0.81 0.83 0.71 0.80 Pharmacy 0.68 0.75 0.80 0.74 0.67 0.72 Rural Management 0.79 0.83 0.71 0.75 0.64 0.49 Science 0.75 0.79 0.79 0.82 0.71 0.66

Sydney College of the Arts

0.84 0.87 0.62 0.77 0.76 0.81 Veterinary Science 0.75 0.79 0.85 0.73 0.72 0.52

(12)

may be addressed by certain general strategies (e.g. low ‘Clear Goals and Standards’ scale scores suggest that course handbooks might need to be revised; Ramsden & Dodds, 1989), SCEQ scores may be used more fruitfully by degree coordinators and faculty managers for initiating discussions and more focused investigations of the issues facing specific programmes and/or faculties. An example of this approach is given by Lizzio et al. (2002), who developed focus group protocols for investigating issues of appropriate workload and assessment in specific degree programmes suggested by CEQ scores.

The SCEQ is currently being used to benchmark the University of Sydney student academic units with comparable units at the University of Queensland, Monash University and the University of Oxford. Each of these institutions is a large, research-intensive university catering for a broad range of degrees, with the Australian universities being members of the Group of Eight organisation (see www.go8.edu.au/). The mission of the Group of Eight is to build the intellectual, social, cultural and economic excellence of Australia’s future. These benchmarking relationships should assist schools, departments and faculties in identifying teaching and learning ‘best practice’ in research-intensive universities, supporting dialogue between equivalent units regarding approaches to teaching and learning, with the ultimate goal of improving the student experience across institutions. While these universities would be considered to be strongly research-intensive, the Australian experience with the CEQ is that it is an appropriate performance indicator across the full range of universities, and fields of study, and it is reasonable to assume that the SCEQ would be similarly useful. We note that the University of Wollongong, for instance, has also adapted the CEQ for currently enrolled students.

The SCEQ has become an integral part of the University of Sydney’s goal of enhanc-ing the student experience (Prosser & Barrie, 2003; Barrie et al., 2005). Derived from a substantial research base, linking the student experience to approaches to study and learning outcomes, its goal is to support both quality assurance and improvement processes within the university, at both the degree level and faculty level. The analyses described above indicate that the SCEQ is appropriate for these purposes. While the CEQ will continue to be an important source of data on the quality of teaching in the Australian higher education sector, we contend that its long lag acts as a limiting factor in its usefulness for improving the quality of student learning; for instance, several years might pass before the effects of a programme to improve the experience of commencing first-year students could be seen in CEQ responses, with some risk that experiences in subsequent years might vitiate the effects of such a programme on CEQ responses. Supplementing CEQ data with similar data collected from currently enrolled students using the SCEQ has allowed our university to make more focused, timely inferences and decisions regarding the student experience in our degrees and faculties.

Note

An earlier version of this article was presented at the Higher Education Research and Develop-ment Society of Australasia Annual Conference, Christchurch, New Zealand, 2003.

(13)

References

Ashenden, D. & Milligan, S. (2002) The good universities guide. 2002 edition (Melbourne,

Hobsons).

Barrie, S., Ginns, P. & Prosser, M. (2005) Early impact and outcomes of an institutionally aligned,

student focused learning perspective on teaching quality assurance, Assessment and Evaluation

in Higher Education, 30(6), 641–656.

Biggs, J. (1987) The Study Process Questionnaire (SPQ) users’ manual (Hawthorne, Victoria,

ACER).

Biggs, J. (1999) Teaching for quality learning at university (Camberwell, Victoria, ACER).

Boomsma, A. & Hoogland, J. J. (2001) The robustness of LISREL modeling revisited, in: R.

Cudeck, S. du Toit & D. Sörbom (Eds) Structural equation modeling: present and future

(Lincolnwood, IL, Scientific Software International).

Burke, M. J. & Dunlap, W. P. (2002) Estimating interrater agreement with the average deviation

index: a user’s guide, Organizational Research Methods, 5(2), 159–172.

Byrne, M. & Flood, B. (2003) Assessing the teaching quality of accounting programmes: an

evaluation of the Course Experience Questionnaire, Assessment and Evaluation in Higher

Education, 28(2), 135–145.

Chan, D. (1998) Functional relations among constructs in the same content domain at different

levels of analysis: a typology of composition models, Journal of Applied Psychology, 83(2),

234–246.

Crawford, K., Gordon, S., Nicholas, J. & Prosser, M. (1998) Qualitatively different experiences of

learning mathematics at university, Learning and Instruction, 4(5), 331–345.

Cronbach, L. J. (1951) Coefficient alpha and the internal structure of tests, Psychometrika, 35(3),

297–334.

Dunlap, W. P., Burke, M. J. & Smith-Crowe, K. (2003) Accurate tests of statistical significance

for r_wg and average deviation interrater agreement indices, Journal of Applied Psychology,

88(2), 356–362.

Entwistle, N. J. & Ramsden, P. (1983) Understanding student learning (London, Croom Helm).

Espeland, V. & Indrehus, O. (2003) Evaluation of students’ satisfaction with nursing education in

Norway, Journal of Advanced Nursing, 42(3), 226–236.

Fan, X. & Thompson, B. (2001) Confidence intervals around score reliability coefficients, please:

an EPM guidelines editorial, Educational & Psychological Measurement, 61(4), 517–531.

Jöreskog, K.G. & Sörbom, D. (2001) LISREL 8 user’s reference guide (Lincolnwood, IL, Scientific

Software International).

Lawless, C. J. & Richardson, J. T. E. (2002) Approaches to studying and perceptions of academic

quality in distance education, Higher Education, 44(2), 257–282.

Lindell, M. K. (2001) Assessing and testing interrater agreement on a single target using

multi-item rating scales, Applied Psychological Measurement, 25(1), 89–99.

Lizzio, A., Wilson, K. & Simons, R. (2002) University students’ perceptions of the learning

environment and academic outcomes: implications for theory and practice, Studies in Higher

Education, 27(1), 27–51.

Marsh, H. W. (1984) Students’ evaluations of university teaching: dimensionality, reliability,

validity, potential biases and utility, Journal of Educational Psychology, 76(5), 707–754.

Marsh, H. W., Rowe, K. J. & Martin, A. (2002) PhD students’ evaluations of research supervision,

Journal of Higher Education, 73(3), 313–348.

Marton, F. & Säljö, R. (1976) On qualitative differences in learning. II—Outcome as a function of

the learner’s conception of the task, British Journal of Educational Psychology, 46(2), 115–127.

Prosser, M. & Barrie, S. (2003) Using a student-focussed learning perspective to strategically align academic development with institutional quality assurance, in: R. Blackwell & P. Blackmore

(Eds) Towards strategic staff development in higher education (Buckingham, Open University

(14)

Prosser, M. & Trigwell, K. (1999) Understanding learning and teaching: the experience in higher

education (Buckingham, Open University Press).

Ramsden, P. (1991) A performance indicator of teaching quality in higher education: the Course

Experience Questionnaire, Studies in Higher Education, 16(2), 129–150.

Ramsden, P. (2003) Learning to teach in higher education (2nd edn) (London, Routledge).

Ramsden, P. & Dodds, A. (1989) Improving teaching and courses: a guide to evaluation (Melbourne,

University of Melbourne).

Ramsden, P. & Entwistle, N. J. (1981) Effects of academic departments on students’ approaches

to studying, British Journal of Educational Psychology, 51(3), 368–383.

Raykov, T. & Marcoulides, G. A. (2000) A first course in structural equation modelling (Mahwah, NJ,

Lawrence Erlbaum Associates).

Richardson, J. T. E. (1994) A British evaluation of the Course Experience Questionnaire, Studies

in Higher Education, 19(1), 59–68.

Richardson, J. T. E. (2005a) Students’ perceptions of academic quality and approaches to

studying in distance education, British Educational Research Journal, 31(1), 7–27.

Richardson, J. T. E. (2005b) Instruments for obtaining student feedback: a review of the literature,

Assessment & Evaluation in Higher Education, 30(4), 387–415.

Richardson, J. T. E. & Woodley, A. (2001) Perceptions of academic quality among students with a

hearing loss in distance education, Journal of Educational Psychology, 93(3), 563–570.

Sadlo, G. (1997) Problem-based learning enhances educational experiences of occupational

therapy students, Education for Health, 10(1), 101–114.

Schmitt, N. (1996) Uses and abuses of coefficient alpha, Psychological Assessment, 8(4), 350–353.

Spector, P. (2001) Research methods in industrial and organisational psychology: data collection and data analysis with special consideration to international issues, in: N. Anderson, D. S.

Ones, H. K. Sinangil & C. Viswesvaran (Eds) Handbook of industrial, work and organizational

psychology (London, Sage).

Trigwell, K. & Prosser, M. (1991) Improving the quality of student learning: the influence of

learning context and student approaches to learning on learning outcomes, Higher Education,

22(3), 251–266.

Watkins, D. (2001) Correlates of approaches to learning: a cross-cultural meta-analysis, in: R.

Sternberg & L. Zhang (Eds) Perspectives on thinking, learning, and cognitive styles (Mahwah, NJ,

Lawrence Erlbaum).

Wilson, K. L., Lizzio, A. & Ramsden, P. (1997) The development, validation and application of