The aim of the pilot was to examine how practical and useful the GPET is, by testing it in 50 practices. The plan was that the 50 practices should be spread across 8–10 CCGs, some of which would also have participated in stage 1 of the study.
In total, 51 practices agreed to participate. After an initial opening invitation, follow-up requests for participation were targeted towards areas and types of practice that had lower representation in the sample, to ensure that the overall set of practices had a good spread in terms of location, size and characteristics of the local population. Each of these 51 practices was visited by a member of the research team or a trained CRN representative, and at least one member of staff in the practice was trained in using the online tool, including the details of the data collection required and retrieving and interpreting monthly reports.
Each practice would then use the tool for a period of 6 months: collating and entering data on a monthly basis, and getting a report to indicate how indicators, objectives and overall effectiveness were changing over time. This would be followed by the evaluation. Following completion, practices were each paid £500 in compensation for the time spent piloting the tool.
Recruitment of practices
Initially, all CCGs and practices who had participated in stage 1 were invited; the offer of involvement in the pilot had been part of the incentive for participation in stage 1. As expected, a reasonable subset of
those that had been involved in stage 1 (13 practices) also wished to participate in stage 2. To recruit the other practices, a three-part strategy was used. First, CCGs involved in stage 1 were asked to send an invitation to other practices in their region. Second, CCGs that had previously expressed an interest in involvement, but had not actually participated in stage 1, were invited to ask their practices. Third, the areas of the country that were already represented, and the balance between urban and rural settings, were examined and other areas/CCGs were purposively targeted via CRNs.
In total, 55 practices agreed to participate. Four withdrew before the pilot commenced as a result of illness or changes in practice staff, leaving these practices unable to participate. The 51 remaining pilot sites were spread across 18 CCGs in total and distributed across six NHS regions (as measured by the former Strategic Health Authority areas) as follows:
1. Yorkshire and the Humber,n= 21 2. South East,n= 17
3. East Midlands,n= 5 4. London,n= 4 5. South Central,n= 3 6. West Midlands,n= 1.
The two regions with the highest representation were spread across several CCGs in different areas. For example, in Yorkshire and the Humber, the 21 practices were spread across four CCGs: one primarily urban, one primarily rural and the other two mixed urban/rural.
The 51 practices were diverse in terms of levels of deprivation: they were spread across all 10 deciles of the Index of Multiple Deprivation as calculated in 2015.143Twenty-eight practices were classed as urban, 14 as rural and nine as mixed urban/rural; 31 practices used EMIS as their clinical information system and the remaining 20 used SystmOne.
Training of practices
Each practice was visited by a member of the research team, or a clinical studies officer who had been trained by the research team, for training in using the GPET. The training session would normally take
≈2 hours and comprised an overview of the study and the GPET, an introduction to the online software (including being given individual log-in details) and instructions on how to enter data on a monthly basis, as well as how to extract and interpret reports about practice performance.
Most practices had one representative being trained (often, but not always, a practice manager or assistant practice manager), although some had more than one to enable sharing of responsibility more easily. Each practice was given a manual that fully explained the process, including details of each indicator that was to be collected. The collection of these indicators was discussed (often at length) during the training and practice representatives were given the opportunity to ask as many questions about this process as they wanted. For the data to be collected via questionnaires completed by others (i.e. three questions for patients using the FFT method and five questions for practice team members), practices were given templates of data collection tools that could be either used directly or amended to fit with a practice’s existing communications.
Practices were offered support from the research team throughout the process via telephone and e-mail. Many practices contacted the research team at least once; sometimes this was to ask questions that could have been answered from the manual, but there was either a desire to speak to a human or an inability to find the right answer in the right place. Some queries were related to asking for help in accessing the system, entering data or resetting passwords. It appeared that the most important support given was to provide help in understanding what data needed to be collected for certain indicators.
Once a practice was trained in using the GPET, it was considered to be in the study.
STAGE 2: PILOTING AND EVALUATING THE MEASURE
NIHR Journals Library www.journalslibrary.nihr.ac.uk 70
Gathering and entry of data
As described inChapter 4, the software had been specifically tailored for the use of the GPET. One feature of this amended software was that the performance areas, objectives and indicators within the tool were set centrally (as a result of stage 1 of the study). Therefore, the only requirements for a practice using the software were as follows:
l Set up accounts and activate the tool (this was typically done during the training visit).
l Receive and respond to monthly e-mail alerts. A named individual within each practice was allocated to each indicator (this could be the same individual for every indicator or divided up among staff so that multiple individuals each had responsibility for some indicators). The online tool would send an e-mail to this named person 5 days before the end of the month and required data input within 10 days (i.e. before 5 days into the next month). The e-mail included a direct link to the correct page of the online tool for the individual to enter the data directly. The data, once entered, would be visible to that practice and to the members of the central research team only.
l Gather data. As noted inChapter 4,Final tool for piloting, there were four main sources of data: (1) data automatically extracted on a monthly basis from the clinical systems, (2) data from practice records, (3) checklists and (4) questionnaires for patients and staff. Clinical system data would be produced in a form that allowed relatively simple entry into the online tool. Practices would set up their own systems for ensuring practice record data were in an appropriate form. Checklists would be allocated to individuals who should know the answers to the questions (or be able to achieve those answers very easily). Practices would devise their own procedures for administering and entering questionnaire data, usually based on the templates provided by the research team.
l Read, extract and/or print feedback reports. Each month, after data entry was completed, the online tool would provide an updated feedback report. This was largely graphical in nature and provided feedback on overall performance, as well as effectiveness in each performance area, objective and indicator. Feedback was provided in terms of effectiveness points and as a percentage of available points. Although the primary presentation was graphical, the data were also available in tabular form and could be downloaded as a Microsoft Excel®(Microsoft Corporation, Redmond, WA, USA) spreadsheet. Practices could then decide how to use the feedback; they were strongly encouraged during the training to discuss it in team meetings.
The original plan was to start training practices in September 2016. Owing to the delays in stage 1, described inChapter 4, training could not actually begin until February 2017, with some practices not trained until May 2017. Thus, the timescale for the pilot, and its evaluation, was considerably tighter than initially planned. In consequence, not all practices were able to complete their 6 months’worth of data collection before participating in the evaluation.
Analysis of data
There were several key questions that were answered by examining the raw data:
1. To what extent were practices able to complete all data fields every month? 2. Did practices enter data for the whole 6-month period, or for part of it?
3. Was the completeness of data entry within a given month, and/or completion of all 6 months, related to practice characteristics such as size, region, deprivation of population, clinical system used and stage of recruitment to the study?
4. Was there evidence that effectiveness scores, as measured by the GPET, improved over time? 5. If so, was this improvement in specific areas or across all areas?
6. Was the extent of change shown by practices related to practice characteristics such as size, region, deprivation of population, clinical system used and stage of recruitment to the study?
Questions 1 and 2 were examined using descriptive statistics. Question 3 was examined via non-parametric correlations.
Questions 4 and 5 were answered using multilevel growth models (with the month of data collection predicting the number of effectiveness points within practices); this way, practices that did not enter data for all 6 months could still be included. Question 6 was answered using similar models, but with practice characteristics as cross-level predictors of change over time.
Interviews with practice staff and members of the public
The interviews with practice staff had three main aims. The first was to discover the perceived usefulness of the GPET. In particular, this was concerned with whether or not the feedback provided was useful; whether or not practices discussed this feedback in team meetings and what was done with the information; and whether or not practices would be interested in continuing to use the tool beyond the pilot. The second aim was to get views on the content of the tool using the following questions:‘did it cover the areas that were perceived as important, useful and relevant?’;‘was there anything missing or superfluous?’; and‘were the indicators easy/feasible to collect?’. The third aim was to examine the usability of the tool, in terms of how easy the online platform was to use.
In total, 38 semistructured interviews were conducted with practice staff. Each practice was asked to nominate a member of staff who had been involved in the use of the tool; this included 26 practice managers, nine GPs, two practice or research nurses and one data manager. The interviews, typically of about 40 minutes’duration, were audio-recorded, anonymised and transcribed. The interview schedule for these interviews is shown inAppendix 5.
It was also planned to interview patient representatives from 10–15 practices. The aim of these interviews was to assess the content of the tool in a similar way to the second aim of the interviews with practice staff. The interview schedule for these interviews is shown inAppendix 6.
It proved difficult to interview patients because, at the time of the interviews, the majority of participating practices had not yet shared information about the study with their PPG and practices felt that their patients would therefore be unable to answer the questions; consequently, only four interviews were arranged via this route. Therefore, to add to the four interviews, two focus groups were also arranged. These were held in London and Sheffield and were attended by seven members of the public. Each focus group lasted 2 hours; the session plan is shown inAppendix 7.
Focus group participants were sent a Microsoft Word (Microsoft Corporation, Redmond, WA, USA) version of the GPET at least 7 days prior to the focus group (seeAppendix 8). They were asked to think about the following questions:
1. Do these objectives and indicators reflect what is important to the patient population?
2. Do you think the data held by the practice will give a true reflection of how the practice is performing? 3. Are there any other indicators that should be included?
4. Would the patient population understand this information (e.g. PPG/Healthwatch)?
Practice manager questionnaire
In addition to the interviews, an online survey of all practice managers was conducted at practices that had participated in the pilot (including those that had not submitted data independently). This survey offered an opportunity to cover some of the same ground as the interviews, but with a different emphasis. In particular, there was the opportunity for respondents to comment on each performance area, objective and indicator; it also asked for quantitative responses about Likert-type scales about views on ease of use, usefulness of the tool and usefulness of feedback.
In addition, there was a section in the survey on practice expenditure, which was designed to measure the practice‘inputs’for comparison with the outputs (the effectiveness measured by the GPET) to form a productivity measure. The main question in this section asked practices to identify expenditure on four categories (staff, premises, bills and other) for each of the 6 months of the pilot phase. They were also
STAGE 2: PILOTING AND EVALUATING THE MEASURE
NIHR Journals Library www.journalslibrary.nihr.ac.uk 72
asked whether or not anything had changed during the pilot that may have had an impact on expenditure or productivity.
The full questionnaire used is shown inAppendix 9.
Practice managers were e-mailed with a link to the survey [which was hosted by Qualtrics (Provo UT and Seattle, WA, USA)] towards the end of the 6-month period. Up to three reminders were sent to non-responders; ultimately, responses were received from 41 practice managers.