Methodology for the design and development of the
new indicators for QOF+
Methodology for the creation of new indicator areas long list
To inform development of new QOF+ indicators, a member of the group conducted an initial review of the existing literature on quality improvement in health care settings, and on development of quality indicators. The methodology chosen for development of the new clinical indicators drew on the methodology for reviewing the Quality and Outcomes Framework –the QOF Review (Lester, 2006) and for developing indicators for the quality of health promotion, prevention and primary care at the health systems level in OECD countries (Marshall et al., 2004). The QOF Review is an independent academic process which advises the NHS Employees and GP negotiators on a range of evidence based clinical indicators. This Review is a collaboration between the Universities of Birmingham and Manchester, the Society of Academic Primary Care and the Royal College of General Practitioners.
Potential local, national, and international sources for new indicator areas were identified and considered. Local data sources included the NHS Hammersmith and Fulham public health report (Zeuner, 2008), PCT Operating Plan 2008/9 (Hammersmith and Fulham PCT, 2008) and local enhanced service schemes (Hammersmith and Fulham PCT, 2008). National data sources included the Healthcare Commission Key Performance Indicators (Healthcare Commission, 2008), NHS Vital Signs indicators (DoH, 2008) and recent national guidance, including that produced by the National Institute for Health and Clinical Excellence (NICE), as well as considering topics addressed by other Local Enhanced Services schemes nationally. The OECD indicators developed for use in a primary care setting were also considered (Marshall et al., 2004). Selected local stakeholders including the Local Director of Public Health were consulted to seek their views on potential approaches for creation of the long list. The members of the QOF+ Development Group, which included GPs and Consultants in Public Health, contributed additional ideas for potential new indicator areas. Documentation from NHS Hammersmith and Fulham’s Patient and Public Involvement Forum was also reviewed to gain a perspective on the views of local patients in order to help inform the process (Hammersmith and Fulham PCT, 2007). A full formal consultation with local stakeholders including primary care and patients was not performed at this stage however, due to the time restraints and subsequent tight deadlines that the group was working within.
Following discussion within the QOF+ development group, it was decided that areas would initially be included on the long list if they were identified as either a local or national priority area. The group agreed that an area would be defined as a local priority if it was highlighted in the NHS Hammersmith and Fulham Annual Public Health Report (Zeuner, 2008). A national priority area would be defined by its inclusion as a Key Performance Indicator by the Healthcare Commission (Healthcare Commission, 2008). It was agreed that a key theme of QOF+ should be the focus on reducing health inequalities locally. The PCT public health report focuses on areas relevant to health inequalities locally, and these were therefore used to help identify potential new indicator areas for QOF+ (Zeuner, 2008). Following discussion within the QOF+ development group, with the PCT and with national experts in quality improvement, it was felt that due to the
tight timescale of this project, it was beyond the scope of QOF+ to incorporate all the LES schemes currently offered by the PCT. However, it was felt that it would be feasible to consider inclusion of selected existing LES schemes which were more likely to fit into the QOF+ framework. The LES which were considered for inclusion within QOF+ for year 1 included CVD Primary Prevention, Smoking Cessation, Immunisations and Choose and Book (which was under development by the PCT).
Following further discussion, it was decided to:
consider adapting the existing CVD Primary Prevention and immunisations LES for QOF+, stop development of new indicators for Choose and Book through QOF+ as there was
already a proposed LES for this and there were practical issues around indicator development (e.g. most GPs do not read-code referrals so it would be difficult to use an indicator to evaluate what percentage of referrals were made through Choose and Book), and to
consider development of indicators for smoking cessation in parallel to the existing LES on smoking cessation.
Following review of the long list, it was agreed that areas that were already incorporated into the national QOF should not be included as new indicators for the clinical domains of QOF+, but could instead be considered in terms of raising thresholds of the existing indicators. These areas include heart failure, CVD, hypertension, stroke, diabetes, COPD, CKD, mental health, dementia, palliative care (BMA, 2006).
Remaining indicator areas were classified using the OECD framework of Health Promotion, Preventative Care, and Diagnosis and Treatment (Marshall et al., 2004), as well as the following domains of quality: Patient-centredness/Empowerment, Organisational and Safety. A decision was initially taken not to include organisational indicators or indicators related to primary care access within QOF+, in view of the presence of this area within the national QOF, and in view of the limited timescale for roll-out of the scheme. In addition, the QOF+ Development Group was aware of the possibility that incentive arrangements for organisational quality through the national QOF may change as a result of the roll-out of practice accreditation schemes including a scheme developed by the Royal College of General Practitioners (National Primary Care Research and Development Centre, 2008).
However, additional funding for QOF+ became available at a later stage of the project, and it was then decided to incorporate aspects of the Imperial College Data Quality project into QOF+. Additional domains relating to patient experience, patient safety, patient information and patient registration (new patient screening for Tuberculosis) were developed further by NHS Hammersmith and Fulham using a separate methodology.
As a result of the prioritisation process for new clinical indicators, a long list of twenty potential priority areas was identified. For the purposes of further evaluation of these areas through a structured consultation with local stakeholders, four areas (including chronic disease case finding, reduction of healthcare-associated infections (eg MRSA), data quality on ethnic group and quality of medical records) were removed from the long list as they were identified as fundamental components of QOF+, and would therefore be automatically incorporated within QOF+. This resulted in a final long list of 16 areas.
Alcohol Adult smoking
Breastfeeding initiation and maintenance Childhood immunisation
Childhood obesity CVD primary prevention Drug misuse
Increasing attendance for breast screening Increasing attendance for cervical screening Influenza immunisation
Measurement of patient experience Osteoarthritis
Self-management of long-term conditions
Sexual health, including STI screening and prevention Smoking in pregnancy
Tuberculosis screening
Consultation with local stakeholders to select priority areas for the
development of QOF+ indicators
The final long list of potential indicator areas was developed into a Delphi questionnaire. This questionnaire was piloted among the group, and wording and structure modified as a result. The questionnaire was then circulated to a local stakeholder panel for consultation to seek their views through participation in a consensus building exercise.
The local stakeholder panel was drawn from NHS Hammersmith and Fulham and from local primary care services and included the Director of Public Health, Head of Primary Care Development, Professional Executive Committee (PEC) Chair, a local GP and a local Practice Nurse. The merits of including a patient representative on the panel were debated, but in view of the fact that the group had been unable to make contact with the PCT’s Patient and Public Involvement Forum, it was decided that a patient representative would not be included in view of the tight timescale for indicator development. It was also proposed that the panel included a practice manager. However, attempts to identify a practice manager for this purpose were unsuccessful, and therefore there was no practice manager representation for the first phase of the structured consultation process. The local stakeholder panel’s views were used to help select
which additional priority areas would be included in QOF+ through a structured consultation to prioritise areas on the basis of their importance.
The consultation method used incorporates aspects of both the Delphi Technique and the RAND Appropriateness Method. These are robust research methodologies with substantial literature to support them. The approach involves identifying experts and obtaining their views anonymously. This provides qualitative and quantitative information on expert views.
The Delphi technique is a postal method which involvestwo or more rounds of questionnaires. Researchers clarify a problem,develop questionnaire statements to rate, select panellists torate them, conduct anonymous postal questionnaires, and feed backresults (statistical, qualitative, or both) between rounds (Campbell et al., 2003).
The RAND appropriateness method requires a systematic literaturereview for the condition to be assessed, generation of indicators based on this literature review, and the selection of expert panels.A postal survey then takes place, in which panellists are askedto read the evidence and rate the preliminary indicators, and a face to face panel meeting (not conducted in the development of the QOF+ indicators), in which panellists discuss andre-rate each indicator. It incorporates a rating of the feasibility of collectingdata (Campbell et al., 2003). It has been described as the only systematic method of combining expert opinion and evidence (Naylor, 1998).
The rating scale used was based on the RAND appropriateness method. Indicators with an overall median rating of 7, 8, or 9 without disagreement for each criterion of importance, scientific soundness, feasibility and clarity were retained; indicators ratedwith an overall median of 1- 3 were removed; indicators rated with an overall median of 4-6 were rated as equivocal, and these were further discussed by the group and retained if thought to be particularly important to include within QOF+. Disagreement was defined as 30% or morescores in both the bottom (1-3) and top (6-9) tertile.
After obtaining questionnaire responses from the local stakeholder panel in the first round, a second round of structured consultation was used to achieve consensus among respondents. Respondents were provided with a summary of the panel’s responses from the first round and asked to consider their own responses in light of this, rating each indicator again using the same method as in the first round.
The results of this consultation showed consensus for the following indicator areas: CVD Primary Prevention, Alcohol, Smoking, Smoking in Pregnancy and Breastfeeding. In addition, the areas of cervical screening and immunisations were rated as equivocal and were retained by the group in view of their local importance. Following further discussion, it was felt that since cervical screening indicators were already incorporated into national QOF, they could be incentivised in QOF+ by raising thresholds instead of developing new indicators. It was decided that the Directed Enhanced Service (DES) currently in place for Immunisations was comprehensive and that it would not be developed further through QOF+. The remaining potential indicator areas were not developed further at this stage as they did not achieve consensus on ratings of importance. Indicators relating to measurement of patient experience, self management of long-term conditions and new entrant screening for TB were however developed at a later stage by the PCT through the methodology for the non-clinical indicators, and included within the “patient experience”, “patient information” and “new patient screening” domains.
Methodology for the development of new indicators
In addition to the indicator areas selected through the structured local consultation, the group developed new indicators for ethnicity recording (incorporating recording of first language) and data quality (records). In addition, funding was made available by the PCT in the later stages of the project for the incentivisation of indicators relating to additional non-clinical domains. To inform development of new QOF+ non-clinical indicators, a member of the group conducted an initial brief review to assess the importance of each indicator area (on the basis of national guidance and the local context), which Healthcare Commission core standards would be met by inclusion of the proposed indicators, and the anticipated training and development needs. For each selected new indicator area, a literature review of the evidence base was undertaken, and this was used to guide development of indicators, ensuring that new indicators would reflect best available evidence. National policy and relevant national guidelines, as well as evidence of local performance and evaluations of relevant Local Enhanced Services (LES) schemes, including the Hammersmith and Fulham CVD Primary Prevention LES and the Lewisham Alcohol LES (Mookherjee, 2007), were used to inform the process. A database of national and London-based LES schemes was created by the group which acted as an additional resource for indicator development. Consultation also took place with recognised national and international experts in selected indicator areas including alcohol and data quality (records), and with relevant local services including the Hammersmith and Fulham Drug and Alcohol Team. The process of indicator development was also informed by the QOF evidence-based reports produced by the National Primary Care Research and Development Centre (National Primary Care Research and Development Centre, 2008). These reports were generated by the nationwide call for evidence to health professionals and patient groups in Spring 2007 and published evidence base. Each report and its associated set of indicators was “commented on by members of the Royal College of General Practitioners Patient Participation Group and IT experts to ensure that proposed indicators made sense to patients and would work within primary care IT systems” (National Primary Care Research and Development Centre, 2008).
Proposed new QOF+ indicators were discussed at regular face-to-face QOF+ development group meetings and through email between group members, in order to further revise and refine them. An evidence-based “QOF+ Report” on each indicator was produced. This was modelled on the evidence-based reports produced as part of the national QOF process, and incorporated similar headings (where appropriate) including proposed indicators, background, priority and relevance to national policy, prevalence of condition, associated morbidity and mortality, review of evidence, degree of perceived professional consensus, degree of perceived support from patients and carers, health impact, and workload and implications for primary care. Additional categories relating to local context, impact on health inequalities and training implications for primary care were also included.
During the final few months prior to launch of the QOF+ scheme, the Department of Health announced a number of new Directed Enhanced Services (DES), which include Alcohol and Ethnicity (NHS Employers, 2008). In light of this, the alcohol and ethnicity indicators for QOF Plus were reviewed and revised to ensure they were in line with the DES, and this process was facilitated by discussions with national and international experts in these areas.
Consultation with local practices
Local practices were consulted on the proposed QOF+ Scheme including the proposed new QOF+ indicators. This consultation process took place through a face-to face meeting with primary care teams (GPs, practice managers and practice nurses) and through consultation conducted through email.
Assessment of new indicators
Campbell et al. (2003) comment that “although it may never be possible to produce an error- free measure of quality, measures should be tested during their developmentand application for acceptability, feasibility, reliability, sensitivity to change, and validity. This will optimise their effectiveness in quality improvement strategies. Marshall et al. (2002) highlight the role of consensustechniques in facilitating quality improvement.
Proposed new indicators were assessed using the OECD (Organisation for Economic Co-operation and Development) criteria of importance, scientific soundness and feasibility as defined below (Marshall et al., 2004). Each indicator was also assessed for clarity.
Scientific Soundness refers to the extent to which you believe that each indicator makes sense
logically and clinically, and captures meaningful aspects of the quality of care.
Importance refers to the extent to which you believe that each indicator is important for
measuring the quality of primary care. The following dimensions were taken into account when evaluating each indicator:
Impact on health
Does the indicator address areas in which there is a clear gap between the actual and potential levels of health?
Policy importance. Are policymakers and the general public concerned about this area? Susceptibility to being influenced by primary care. Can the primary health care system
meaningfully address the aspect or problem being measured by the indicator?
Feasibility refers to the extent to which you believe that each indicator is feasible in primary care,
and to what extent you believe that the value of the information contained in each indicator outweighs the cost of data collection and reporting.
Clarity refers to the extent to which you believe that each indicator is expressed in clear, precise
and unambiguous language.
Members of the QOF+ Development Group assessed the indicators in terms of importance. The assessment of feasibility and clarity of indicators was also informed by consultation with local and national experts in the proposed indicator areas, including the local Professional and Executive Committee, Clinical lead for Child Protection, the Hammersmith and Fulham TB Action Group, the local Drug and Alcohol Team, Professor Richard Baker and Professor Colin Drummond. Changes were made to the wording of some indicators as a result.
Assessment of new clinical and records indicators
Structured consultation with local stakeholders drawn from the PCT and from primary care was used further inform indicator development. As part of this consultation, local stakeholders were asked to assess each proposed new indicator in the clinical and records domains using the OECD (Organisation for Economic Co-operation and Development) criteria of importance, scientific soundness and feasibility as defined above (Marshall et al., 2004), and also to assess each indicator for clarity. Each indicator was rated using a 9-point Likert scale. Local stakeholders were also asked to comment on any aspect of the indicator – including wording of the indicator and proposed thresholds.
The consultation method used incorporated aspects of both the Delphi Technique and the RAND Appropriateness Method as described above. After obtaining questionnaire responses from the local stakeholder panel in the first round, a second round of structured consultation was used to achieve consensus among respondents. Respondents were provided with a summary of the panel’s responses from the first round and asked to consider their own responses in light of this, rating each indicator again using the same method as in the first round.
As a result, all proposed new indicators were rated by the panel as scientifically sound, relevant, feasible and clear, and consensus was achieved for all indicators.
Assessment of new indicators in non-clinical domains
The process of assessment was informed by consultation with practices on the group’s QOF+ proposals for non-clinical indicators. Feedback received from practices was used by the QOF+ Development Group to help assess the indicators further in terms of acceptability, importance, soundness (face-validity), feasibility and clarity of the proposed indicators.
Response to feedback on proposed new indicators
A number of proposed indicators were assessed by the group as being either not acceptable to practices, not sound in terms of face validity or not feasible, following internal review and taking account of feedback from practices and from local and national experts. These included