Chapter 4 CASE STUDY: REDUCING STUDENT NON-CONTINUATION RATES
4.7 The student experience
This chapter has so far concentrated on quantitative aspects of student retention. The presentation of the case study is now developed by including the students‟ perceptions of their higher education experiences in an attempt to discover new insights or correlations with previous data. Analysis will be presented for the institution, school and programme as far as the data permits.
This section focuses on the „student experience‟ as determined by three surveys: The National Student Survey (NSS)47 ; The Programme Experience Questionnaire (PEQ) 48 and the Student Barometer Survey (SBS)49 and provides a degree of qualitative analysis of the „student voice‟ into what hitherto has been a systems, performance driven analysis. The three surveys cover programme experience, teaching and learning and, experience of student support and the campus
environment. All were deemed crucial dimensions of student retention and widely considered in the literature.
National student survey (teaching, learning and assessment)
The NSS was introduced into the higher education sector in 2005 and captures feedback from final year completing students on their experience at the institution: the results are published on http://www.unistats.com and supports comparisons across institutions and subjects. The case institution considers the data each year at SQC and sends reports to Academic Board. In 2007/08, additionally each school was required to consider the results and make a formal response. This was received by Academic Board, November 2008. The information is captured from completing
47
The National Student Survey forms part of the revised quality assurance framework (QAF) for higher education. The aim of the survey is to gather feedback on the quality of students' courses in order to contribute to public accountability as well as to help inform the choices of future applicants to higher education. Downloaded on 14 April 2009 from http://www.hefce.ac.uk/learning/nss/
48
The Course Experience Survey is directed at final year students on undergraduate degree courses in Hospitality, Leisure, Sport and Tourism. It aims to uncover information about their perceptions and attitudes towards a whole programme of study, rather than a single year or module/unit. In 2001 a pilot study was conducted to find out if the Ramsden Course Experience Questionnaire (widely used in Australian HE institutions) would be suitable for measuring student satisfaction in Hospitality, Leisure, Sport and Tourism courses. Following the pilot study a slightly modified version of the questionnaire was used to conduct nationwide surveys in 2002 and 2003. Downloaded on 14th April 2009 from http://www.heacademy.ac.uk/hlst/resources/detail/ourwork/OP_sceq_2004
49
students, and does not therefore include previously withdrawn students or those not progressed into the final year; it is therefore a selective sample. The 2008 and 2009, NSS results, presented alongside the „questions‟ and „scale‟ are shown in Table 28 and evidences that „Organisation and management‟ and „Assessment and feedback‟ remains a challenge. This was recognised by AB.
From the survey‟s introduction in 2005, the case institution evidenced steady and consistent improvement in student‟s overall satisfaction: 70% [2005]; 73% [2006] to 77% in 2008; it remained at this level in 2009. Institutions are also ranked against each other, including for overall student satisfaction. The case institution‟s ranking was: 121 out of 127 [2005]; 111/127 [2006]; 130/145 [2007]; 145/194 [2008] and 147/210 in 2009.
Across 2008 and 2009, a number of questions achieved 80% or above, these were: the course is intellectually stimulating [80%]; staff are enthusiastic about what they are teaching [82%]; staff are good at explaining things [82%]; assessment
arrangements and marking had been fair [81%]; I have been able to contact staff when I needed to [81%]; I have been able to access general IT resources when I needed to [80%]; my communication skills have improved [80%] and the course has helped me present myself with confidence [80%].
The questions relating to „Organisation and management‟ was the lowest performing group and remained so for 2009. The performances ranged from 57% to 75% in 2008 and 60% to 73% in 2009. This group included practical matters such as timetabling changes being communicated effectively, as well how well the timetable works for individuals. With a large number of part-time students and full-time
commuting students both these would be weighted heavily.
In general, the NSS 2009 performances showed some improvements in teaching, academic support and personal development but small gains and losses in most other categories. There was a marked improvement in the results for Q8 [5%], referring to detailed comments on student work. The institution however remained in the lower percentile of UK ranked universities. This provides a „select‟ student voice and insight into learning, teaching and assessment as well as programme related organisation and management which may be influencing factors behind the levels of student withdrawals and non-continuations.
Table 28 National student survey results, 2008 (2009)
Question number
% Agree Actual
value Scale Question
Q22 77 (77) 'Overall, I am satisfied with the quality of the course'
Overall, I am satisfied with the quality of the course.
Q4 77 (80) The teaching on my course The course is intellectually stimulating. Q3 79 (82) Staff are enthusiastic about what they are
teaching.
Q1 82 (86) Staff are good at explaining things. Q2 77 (77) Staff have made the subject interesting. Q9 68 (68)
Assessment and feedback
Feedback on my work has helped me clarify things I did not understand.
Q8 69 (74) I have received detailed comments on my work.
Q6 81 (80) Assessment arrangements and marking have been fair.
Q5 79 (77)
The criteria used in marking have been clear in advance.
Q7 62 (67) Feedback on my work has been prompt.
Q11 76 (81) Academic support
I have been able to contact staff when I needed to.
Q12 73 (74) Good advice was available when I needed to make study choices.
Q10 72 (78) I have received sufficient advice and support with my studies.
Q15 59 (60) Organisation and management The course is well organised and is running smoothly. Q13 75 (73) The timetable works efficiently as far as my
activities are concerned.
Q14 57 (62) Any changes in the course or teaching have been communicated effectively.
Q18 66 (73)
Learning resources
I have been able to access specialised equipment, facilities or room when I needed to. Q17 80 (79) I have been able to access general IT
resources when I needed to.
Q16 71 (73) The library resources and services are good enough for my needs. Q21 78 (79)
Personal development
As a result of the course, I feel confident in tackling unfamiliar problems.
Q20 78 (80) My communication skills have improved. Q19 77 (80) The course has helped me present myself with
confidence.
Adapted from DOC 63 and DOC 66
Thus far the presentation of the case has concentrated on institutional level
performance. The following section describes subject based performances which do not necessary correlate across to the schools due to the definitions adopted in the survey. The external data sets dictate the levels of data interrogation. The
information presented draws on a report from SQC to AB (Doc 63) but notes that the School of Business was omitted from the analysis.
A number of questions scored less than 60% across a number of subjects: questions 14, 15 and 7 were evident across 6 subjects [almost half] and questions16, 8 and 9 were evident across 4 subjects. The lowest performing
question groups were „Organisation and management‟ and „Assessment and feedback‟ and additionally there was evidence of concern relating to learning resources. Two other questions, 12 and 18 relate to academic support and access to specialist facilities occurring across three subjects.
The Subjects of Communications Technology and Design Communication had performances less than 60% across all three „Organisation and management‟ questions with the latter scoring 20% and 27% across two of them. The Subject of Fine and Applied Art, scored 50% and 46% in two of the three questions, whilst the Subject of Social Care scored 39% in one question. For questions associated with „Assessment and feedback‟, the Subject of Computing scored between 45% and 52% across three questions, the Subject of Design Communications scored 54% to 56 % across three questions, and the Subject of Science scored 27% on one question. A third area, „Academic support‟, was particularly prominent in the responses from the Subject of Design Communications with the associated three questions receiving between 52%-57%; Science also had two questions below 60% [47% and 53%].
The above summary of the poorer performing questions suggests that the students in the Schools of C&CT, A&D (Subjects of Design Communications, Fine and Applied Art) and S&T (Subject of Science) are less satisfied than in other subjects; this presents a risk to retaining students. The Subject of Sports and Exercise Sciences within the School of HSCSES, on the other hand received no result less than 60%; this was not however matched by other Subjects in the School that were critical of resources.
The results are obtained from self selecting samples of students (by definition, as they completed the survey) and in some cases the samples were small. The
responses, never-the-less provide valuable insights to students‟ perceptions of their experiences. Furthermore the results are available for public scrutiny to inform institutional choice for UCAS applicants. The importance of measuring the NSS „overall student satisfaction‟ was acknowledged in 2007, when it was incorporated as a new Board of Governors level KPI and monitored annually.
The following section describes the results of a parallel internal survey to assert completed students‟ perceptions of their programme of study.
Programme experience survey
Student retention has been a priority since 2001. Concern about student retention echoed across all senior level committees, including Academic Board. During 2006/07 concern escalated following the publication of the NSS results in 2006 and in response, the institution commissioned an internal survey; AB agreed to adopt the Course Experience Questionnaire (P. Ramsden, 1991). It expressed the survey as the Programme Experience Questionnaire (PEQ) to concur with internal
terminologies. It was sent to all students that had completed their studies in 2006/07 [1,293 students] in the autumn of 2007.
The PEQ 2007, institutional level results are presented in Table 29 in descending order of concurrence with „% agree/strongly agree‟. There were 25 separate statements relating to the programme of study. Questionnaires from 22% [284 students] of eligible students were used to inform the analysis and of this 75.5% indicated „Overall, I was satisfied with the quality of the course‟. The highest scoring questions were those relating to developing graduate level transferable skills: written communications [84.6%], analytical [87.8%], problem solving [83.4%], planning [86.8%], team membership [76.1%] and developing confidence about tackling unfamiliar problems [80.4%]. Scoring less highly were areas relating to student feedback [in the range 62.3%-72.4%] and clarity of expectations [in the range 66.2%-69.6%] both considered important aspects of retaining students. A high percentage of students [69.7%- 92.3%] „agreed or strongly agreed‟ that the „Workload and assessments not being reliant on facts and a good memory‟ was appropriate.
Inhabiting the range 62.3%-77.7% were questions relating to teaching. In this group 67% of respondents considered staff made it clear from the start what they expected from students; 70.4% considered the lecturers were extremely good at explaining things and 74.5% considered the teaching staff worked hard to make their subjects interesting.
Table 29 Programme experience questionnaire 2007: institution level summary Question % agree/ strongly agree % disagree/ strongly disagree
Q5 The course sharpened my analytical skills 87.80% 12.20%
Q22 My programme helped me to develop the ability to plan my own work 86.80% 13.20%
Q11 The programme improved my skills in written communication 84.60% 15.40%
Q14 I was generally given enough time to understand the things that I had
to learn
83.90% 16.20%
Q2 The programme developed my problem-solving skills 83.40% 16.60%
Q10 As a result of my programme, I feel confident about tackling unfamiliar
problems
80.40% 19.60%
Q3 The teaching staff on this programme motivated me to do my best work 77.70% 22.40%
Q9 The programme helped me develop my ability as a team member 76.10% 23.90%
Q15 The staff made a real effort to understand any difficulties I had with
my work
76.00% 24.00%
Q25 Overall, I was satisfied with the quality of the course 75.50% 24.50%
Q20 The teaching staff worked hard to make their subjects interesting 74.50% 25.50%
Q17 The teaching staff normally gave me helpful feedback on how I was
doing
72.40% 27.60%
Q1 It was always easy to know the standard of work expected 72.40% 27.60%
Q18 My lecturers were extremely good at explaining things 70.40% 29.50%
Q6 I usually had a clear idea of where I was going and what was expected
of me on this programme
69.60% 30.40%
Q24 The staff made it clear, right from the start, what they expected from
students
67.00% 33.00%
Q7 The staff put a lot of time into commenting on my work 62.30% 37.80%
Q21 There was a lot of pressure for me to do well on this programme 39.20% 60.80%
Q13 It was often hard to discover what was expected of me on this
programme
33.80% 66.20%
Q16 Feedback on my work was usually provided only in the form of marks
or grades
33.30% 66.70%
Q23 The sheer volume of work to be got through on this programme meant
it couldn‟t all be thoroughly comprehended 30.30% 69.70%
Q4 The workload was too heavy 20.50% 79.60%
Q8 To do well on this programme all you needed was a good memory 10.40% 89.50%
Q12 The staff seemed more interested in testing what I had memorised
than what I had understood
9.80% 90.30%
Q19 Too many staff asked me questions just about facts 7.70% 92.30%
Sort based on ranking of Agree/Strongly Agree. Adapted from Appendix 2 in Doc 62
The paper provided to SQC presented subject level data, the home of specific programmes; this means the sample sizes for some were small. The report highlighted:
„It is difficult to draw firm conclusions on the quality of the student experience from the questionnaire, since the numbers of responses received in some subject areas are very small, but it can be noted that key themes for development appear to relate to the communication of expectations and consistency of feedback on performance.‟
(Doc 62 p.1)
Although the student returns were low at the level of the subject the following comparisons between institution and subject level data were notable:
Q3.The teaching staff on this programme motivated me to do my best work. Graduates strongly disagreed with the statement in Engineering [31%], Design Communication [35.3%] and Business [28%] in contrast with the institution average [16.6%]. Science was the highest at 40% although this amounted to only 5 returns.
Q17.The teaching staff normally gave me helpful feedback on how I was doing. Several subjects strongly disagreed with this statement more strongly than the institution average [27.6%]: Engineering [65.5%]; Fine Art [57.1%] and Science [60%].
Q18. My lecturers were extremely good at explaining things.
A number of graduates more strongly disagreed with this statement than the institution average [29.5%]: Fine Art [71.4%]; Engineering [40.7%] and Science [80%].
Q20.The teaching staff worked hard to make their subjects interesting.
Graduates strongly disagreed with this statement in Engineering [48.3%], Business [44%], Design Communication [40%] and Science [60%] compared with the
institution average [25.5%].
The responses for PEQ 2007 highlight subjects where students are less satisfied than the institutional average. It emphasises particular themes relating to teaching, learning and assessment such as engagement and motivation as well as student feedback. There appears to be some correlation between the graduate feedback on programmes and those schools and subjects which have previously been identified as having high levels of withdrawals and referrals, namely Engineering and
levels of withdrawals. Since the PEQ applies to graduated students (and therefore the successful students) it is likely to present the most optimistic view.
In developing a model and performance improvement framework for student retention, the above data suggests that having performance indicators relating to student satisfaction could have a meaningful role in focusing interventions to drive up student retention. If adopted at institution level, and providing robust sample sizes could be determined at subject or school level, the indicator could be cascaded down through the institution.
The PEQ provided an additional internal „moderation‟ for the externally administered NSS and provides for more immediacy of interventions to realise performance improvements.
Student support and non academic interfaces with the institution are other aspects which influences student experience. In the same year the PEQ was being
administered with recent graduates, the institution also took part in the on-line Student Barometer Survey (SBS) considering non academic student support.
Student barometer survey (non academic student support)
There was a degree of investigation overlap in the areas of student satisfaction (teaching, learning, assessment, academic support and programme satisfaction) undertaken through the NSS 2007 (2008 and 2009) and PEQ 2007 surveys. However, neither survey addressed non-academic support.
In autumn 2007, the institution contributed to the on-line SBS. The report to Operational Managers Group (OMG) and Senior Executive identified the lead department should work with internal support teams and the conducting company, i- graduate, to strengthen the applicability of the questionnaire for all students. The on- line questionnaire addressed areas of learning, living, support and arrival with some questions targeted towards international students. The following results were
highlighted in an internal report to OMG:
„Students at eight higher education institutions across Wales were surveyed in the first annual Wales Student Barometer on 16 aspects of student support, ranging from student finance to accommodation, counselling and disability, with [the] „Case Institution‟ scoring the highest overall rating out of the institutions taking part...Out of the four categories under consideration
(Learning, Living, Support, and Arrival), [the] „Case Institution‟ scored best on the latter... We ranked reasonably well in support and learning overall, but less well in the “living” categories. Areas in which it is statistically significant that [the] „Case Institution‟ ranked first in Wales...Employability; Work experience; Careers advice; Internet access; Good contacts; Financial support and finance office; IT support.‟
(Doc 65 pp.2-3)
The Institution was placed last in the ranking against other Welsh institutions for university clubs and societies. This is seen by potential students as an important aspect of the student experience but is not one the institution has a management responsibility for50. However, reputational positioning for non-academic support is important for applicants, and is exploited by some external student guides.
The following sections focus on a series of non-data led interventions to improve student retention that the institution has made since 2001.
Interventions
The institution responded to the key issues highlighted in the NSS surveys, the PEQ 2007, and on a number of quantitative reports highlighting the performance of student withdrawals and non-continuation of students by applying a range of tools and interventions. A number of these are described in this section.
Strengthening student led engagement
Strengthening the Student Union has been a focus of attention (and resources) by the Board of Governors, Core Executive and Academic Board since 2001. In 2007, the Student Union agreed to reform into a Student Guild and focus activities on the provision of student support, clubs and societies whilst the case institution assumed responsibility for the operational management of bars and shops. Subsequently, the Student Guild, reporting to the Board of Governors, evidenced a greater
engagement with enrichment activities, clubs and societies for all students and a
50
The Student Union/Student Guild has separate management and governance arrangements but has reporting responsibilities to the Board of Governors and Academic Board. They receive an annual grant from the institution for
move away from the drinking club that had dominated previously. Notably in 2008, there was an increase in voting engagement by students and attendance at key committees. The potential for academic and social integration was increasing.
Audit and review
The next section describes how the university‟s audit and review quality enhancement process was applied to improve „Programme organisation and management‟. This has been consistently highlighted in the NSS since 2005 and echoed in the PEQ and SBS in 2007.
The institution applies its audit and review process to any aspect of business:
academic, operational and cross institution themes. For example a themed review of recruitment and admissions was undertaken during 2004/05 and resulted in
widespread recommendations within schools and central administration (Doc 67). The impact included: a reorganisation of student recruitment and admissions across the institution; strengthened procedures and a new physical and virtual, single, initial point of contact for enquiries, applicants, students and alumni, called the Service Information Desk (SID), located close to student support services and a new „learning zone‟ in the library. This drew students together and gave a greater sense