As part of the PhOEBE programme, we developed candidate indicators to assess the performance of emergency ambulance services. The process of development has included wide consultation with patients, ambulance service, commissioners, policy-makers and researchers. We have also calculated the indicators and examined their statistical reliability and whether or not case-mix adjustment makes them a potentially more reliable way of assessing performance or quality. To decide whether or not to recommend these indicators for inclusion in a national assessment framework, we have also considered wider questions about what characteristics a good indicator should exhibit.
There is a wide literature on performance indicators, some of which is focused on what makes a good indicator and how to choose and develop them. We have examined some of the publications issued by UK organisations, as well as a comprehensive review, to try to identify a‘checklist’ of the characteristics of good indicators. This literature is not comprehensive or based on a systematic search as we are only trying to achieve‘saturation’ of the ideas about the criteria that good indicators should possess.
Sets of indicators
Because services have many components, and many objectives, assessment of the quality of services will ideally be based on sets of, rather than individual, indicators. Before looking at the quality of individual indicators it is worth asking what properties this set should have, but there is little literature on this question. Based on previous research undertaken by PhOEBE programme study investigators,40we consider that the set should ideally be:
l Inclusive– the set of indicators should ensure that service or system performance is relevant to all patient groups that might use the service. If some patients are excluded, then there is the possibility of distorting the service or system to focus on only those groups included. This is related to equity.
l Comprehensive– addressing all the important dimensions of performance quality, such as effectiveness of services and care, efficiency, cost, equity and safety. A comprehensive indicator set should also address outcomes, processes and structures.
DOI: 10.3310/pgfar07030 PROGRAMME GRANTS FOR APPLIED RESEARCH 2019 VOL. 7 NO. 3
© Queen’s Printer and Controller of HMSO 2019. This work was produced by Turner et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
l Co-ordinated– indicators should work independently or with each other, not against each other. For example, including both the proportion of traumatic brain injury patients treated in a neurosurgery centre and the total number of transfers could cause conflict.
l Parsimonious– equally, a good set of indicators should avoid unnecessary duplication (indicators that are measuring or indicating performance in the same area). An over-riding principle identified by the Royal Statistical Society working group on indicators is the need for parsimony.54
Types of indicator Indicators or measures
One issue to consider in selecting types of indicator is the question of whether they are‘indicators’ or ‘measures’. Indicators do just that, as they are said to ‘resonate’ with performance and quality but are not direct measures of it. For example, in emergency and urgent care, one indicator could be based on the number of attendances at an ED on a Monday morning compared with the average number on other weekday mornings. There is known to be a Monday morning blip, which in itself does not matter, nor is it a measure of anything; however, when it is high it might indicate poor access to the emergency and urgent-care system over the weekend. Therefore, this could be a measure that resonates with the quality and performance of the emergency and urgent-care system over the weekend.
We take the view that measures are generally preferable and that the set of indicators we choose for assessing ambulance service performance should, as far as possible, actually measure aspects of the service performance that are important. Individually, if these measures change it could indicate a better performance overall and, taken together as a set, are an indicator of the quality of the service. Nevertheless, we have also considered some service-specific measures that were considered to be indicators of performance.
Characteristics of good performance indicators
Several lists of the attributes of a good performance indicator have been published. We have synthesised the lists published by Pringle et al.54(12 items), The Audit Commission55(13 items), the Royal Statistical Society56(14 items), the Institute for Innovation and Improvement (11 items relevant to individual indicators and two items focused on the set of indicators)57and the literature review and expert assessment carried out by Jones et al.,58which although focused on health care generally has considered services, organisations and policy as well as frontline‘care’ (15 items). There are many other published lists but a review of this sort seeks saturation rather than completeness and we have found that adding other lists is simply repetitive. Broadly speaking, these checklists have identified six key criteria for good indicators. Criteria should: 1. be important to users
2. be valid and evidence based 3. use reliable data
4. be statistically robust 5. be simple to understand 6. be remediable.
We have taken the view that the ambulance service indicators should be chosen principally with either performance monitoring within a system in mind, to answer the question‘are things getting better?’, or performance assessment to determine whether or not a change in the system has improved performance. Between-service comparisons and the construction of league tables are fraught with difficulties and should not be a priority in choosing indicator sets for ambulance services, but some of our indicators would make this possible. This was illustrated in two of the indicators that we developed, for which the effects of individual hospitals on outcome was very apparent. With this in mind, we have placed emphasis not on issues such as comparability or consistency between places, nor on the question of whether or not they are ‘context free’, but rather on their value as signals of change in individual services or the ambulance service
We have used these six main criteria to assess all our candidate indicators to ensure that we have made an all-round assessment of their quality. Within each of the six main criteria there are several characteristics and those that have been explicitly identified in the lists we have examined are presented in the results. We have used these characteristics as a guideline of the issues to consider in making the assessment of each criterion.