Chapter One
1.10 Evaluating Services
In the late 1990s, a number of UK-based organisations for older people, including Age Concern, Help the Aged and the National Pensioners’ Convention, participated in a major programme on outcomes in community care practice from the
perspective of users and carers. Underlying values, partnership, quality issues and organisational culture were central concerns for these groups.
Nocon et al.(1997), in a review of this programme, outlined four questions central to the process of service evaluation and which are of relevance to this study as follows:
is the service providing what people really need?
what effect does the service have on the lives of users and their carers?
is it possible to tell whether the service has made any difference to what users are able to do and the way they feel?
are there any easy, straightforward ways in which funding bodies, service providers, users and carers can regularly check on the impact of the service?
Nocon et al. conclude by saying:
‘Just providing services is not enough. We need to know whether they do in fact meet people’s needs. We need a way of routinely monitoring the impact of what is provided so that any mismatch between what people need and what is on offer can be put right ... [users and their carers] must have a say in assessing the effectiveness of the services. After all, users are best placed to say what difference services have made to their lives.’
The last few decades have seen a substantial increase in both the range
and extent of community and social services, as well as a growing realisation of the importance of evaluating the quality of these services. Shaping a Healthier Future: A Strategy for Effective Health Care in the 1990s(Department of Health, 1994) set the challenge of measuring the quality of services as one of its targets. This challenge was to be met by constantly monitoring and evaluating quality through various means, including clinical audit and customer surveys, and by monitoring service outcomes to ensure that they were as effective and efficient as possible.
While acknowledging that Irish health services, including those specifically provided for older people, were of a high standard and were managed by committed and caring staff, Shaping a Healthier Futuredid state that:
‘Many of the services are not sufficiently focused towards specific goals or targets and it is therefore difficult to assess their effectiveness. The information which would support this focusing is frequently unavailable 50
The 1994 Health Strategy placed an explicit onus on health providers to monitor, evaluate and re-orientate if necessary, services for older people, to ensure that optimum service is provided for those who might require it.
The Eastern Health Board (now the ERHA) was also conscious that evaluation and measurement of quality have become more important. In its Ten Year Action Plan for Services for Older Persons 1999-2008(Eastern Health Board, 1998), the Working Group recommended that all new service developments for older people should be evaluated as soon as possible after they have been put into place. As part of its mandate to put in place ‘systems, procedures and practices to enable it to monitor and evaluate services’ (Health(Eastern Region Health Authority) Act, 1999) the ERHA is currently drawing up standards for residential services for older people.
Following on from this theme, Brenner and Shelley (1998), points out:
‘All programmes require evaluation, initially of feasibility and later of process and outcome. Older people themselves should be encouraged to become involved in research projects. Programmes which are positively evaluated need to be sustained and used as models elsewhere.’
Two major interrelated factors contributing to the increasing realisation of the importance of evaluation have been the growing voice of the consumer and the growing demand for quality community services. However, the HeSSOP report found that, in Ireland, efforts to consult with consumers in the health and social system have been limited. It describes how, when there is a consumerist approach to consultation, people are only given limited opportunities for
involvement and participation. They are being asked to evaluate output without an explicit explanation of the reasons for and results of that evaluation.
The National Council on Ageing and Older People (Garavan et al., 2001; Delaney et al., 2001; Ruddle et al., 1997) has argued that democratic models of consultation be adopted and that evaluation programmes focus on, among other things, quality of life outcomes for the client.
This model proposes that service users should take an active role in the decision- making process, including the way in which services are developed, structured and provided. In evaluation programmes that have at their centre quality of life outcomes, evaluations must focus on how well the needs of the client are met,
and service users must be consulted during the development and planning of those aspects of the evaluation.
Regular evaluation of services at all levels is essential if services are to become more effective and more appropriate to individual older people. In The Years Ahead Report: A Review of the Implementation of its Recommendations(Ruddle et al., 1997) the authors stressed the importance of evaluating the effectiveness of day services in order to determine standards and principles of good practice.
In the past, and even today, the evaluation process has often been regarded by service planners and providers as an enormous burden. Evaluation was often seen as a unique and complex process occurring at a certain time in a certain way, and almost always involving outside experts. It was often perceived to be about proving the complete success or total failure of a programme.
Programme evaluation methods were often chosen largely on the basis of
achieving complete scientific accuracy, reliability and validity. As McNamara (1999) and many others have described, this approach often generated extensive data from which very carefully chosen conclusions were drawn. Generalisations and recommendations were avoided. As a result, evaluation reports tended to reiterate the obvious and left service planners and providers disappointed by the process.
Evaluation can be confused with other related activities such as assessment, monitoring, reassessment or quality assurance. This difficulty is referred to in some detail in Care and Case Management for Older People in Ireland (Delaney et al., 2001). As the authors point out, all these activities involve examining aspects of a service or a programme and are linked to the evaluation process (e.g. quality assurance relates to actions followingservice evaluation), but are not substitutes for evaluation per se.