Measuring effectiveness

There was general agreement that measuring effectiveness and providing robust evidence is challenging, as it is often impossible to identify whether the intervention or other external factors have bought about a bettering of the students’ situation.

However, there was also agreement that measures need to be put in place to try to assess what works best.

Overall, the measuring, monitoring and reporting of the effectiveness of the service appeared to be a weak area. Measurements of success were often ad hoc and heavily reliant on feedback from students. HEIs made reference to a range of ways in which they measured their effectiveness, including some or all of the following: forums (student feedback); annual surveys and surveys used by the students union; external consultants undertaking reviews; student satisfaction forms completed when students had used any of the services; or informal feedback from students. In most cases, the use of feedback forms was erratic at best, mainly due to admin overload, and at worst provided no useful way to improve the service.

‘Forms, when they are used, go back to Head of Service but there’s no feedback loop to staff on the sharp end’.

None of these methods suggested a systematic assessment of what was working well and what was failing to achieve appropriate outcomes, or indeed, the identification of what those outcomes might be.

In one instance very few staff outside of the senior team knew if or how monitoring was being undertaken, or received any feedback about their own work or that happening in the department. Academic staff were equally unaware of what

monitoring was taking place and were not asked to comment on a student’s progress during or following an intervention, even when students had agreed to disclose their condition to the course leader or where a personal learning plan existed.

Staff in another HEI, however, described the importance of responding to student feedback. There was an advisory group of students who helped to develop the service and discuss what the service might offer, for example, the Wellbeing Centre opening until 7pm once a week. Staff then used the website and posters to complete the feedback ‘loop’, and were able to report:

‘“You said, we did”. Also ‘drop-in’ sessions for all services had been bought into the same time frame at the students’ suggestion, which has proved very effective in making the service more accessible’.

Another HEI involved students in welfare issues and provision through regular scheduled meetings between the Student Union sabbatical team and the Wellbeing team.

Counselling service staff more commonly used the Clinical Outcomes Routine

Evaluation (CORE) form to measure the effectiveness of their interventions. This form was used to assess changes in the student and how they felt. One form was used at the outset of the counselling, and another in the last session. One Senior Counsellor noted:

‘This has posed some challenges as not all students return for their final session as they are feeling better, so there is discussion about whether it could be used in the penultimate session instead. But when both are completed, 80 per cent of students show improvement’.

Another HEI measured whether the services provided have been effective in helping the student to reach her/ his potential. In this instance, the focus of mental health and disability support was on making sure students are able to properly engage with their education, so one system of measurement of success is based on grades:

‘[I’ve] just done the analysis of our graduates last year: 76 per cent of students across the piece got a 2:1 or a first, but actually, from the cohort that are declared as specific learning difficulty, 80 per cent got 2:1s or firsts. So, we feel that the support that we’ve put in place is actually delivering outcomes we’re looking for, which is high-achieving students.’

There was work to do on tracking students, particularly those with intermittent mental health problems, those on modular courses, and those on courses that use sessional tutors (external specialists who come in to teach one or a few classes) and so lack the continuity of a regular tutor. In these instances, staff collected feedback from service users through Survey Monkey, or were able to retrieve information from databases to track students and measure the effectiveness of interventions more effectively. Some HEIs were making good use of electronic registers to monitor students’ attendance and, where problems were spotted, to intervene and offer further support.

Others were keen to do more, but time was proving a challenge:

‘We're a small team, although we sound big … We've got a large number of students, and I think one of the big things for us will be we'd like to do more monitoring. We try to do as much as we possibly can, but it would be much more in our interests, I think, if we monitor students. We were talking this morning about attendance monitoring, and that's often where we find students with complex disabilities in particular; one of the benefits of having the [liaison officers in departments] is they're out there and in faculties, so they can spot if students aren't attending. If a student’s not getting into lectures at the time they're supposed to, or not engaging with the learning environment.’

Only one senior interviewee addressed the question of how effectiveness was measured by referring to Key Performance Indicators (KPIs). These were: National Student Survey (NSS) scores and variation between disabled and non-disabled student scores, and comments offered as part of the NSS; retention; attainment; appeals and complaints ie learning from what goes wrong (see Good Practice Example below).

These KPIs were reviewed annually, and fed into the annual report. Results were also fed back through Boards of Study. Patterns were also sought in looking at complaints and appeals, so these could be addressed with particular departments or staff. In addition, quantitative data was obtained from the CRM system and qualitative data from specific feedback when it was sought.

Feedback of such information to appropriate committees was also seen as important, as it could make a difference to management decisions regarding spending or staffing, and therefore have a direct impact on how the service is developed and delivered:

‘There are three main university committees: Learning Teaching and Enhancement; Quality and Standards; and Academic Board. Reports on attainment, retention and destination go to the Learning, Teaching and

Enhancement Committee, whilst general quality and diversity reports go to the Academic Board, the Executive Board and the Governors’.