We developed our toolkit using feedback from stakeholders and examples in the existing data. In terms of stakeholders, we held two workshops to elicit feedback from individuals external to the research team. The first workshop took place on 1 March 2016, in which a small group of SSAP, including patient representatives, participated. We made a short presentation outlining the aims and objectives of the toolkit, including a set of questions targeted at different toolkit users. Stakeholders suggested we should make the following changes:
l remove the word‘diagnostic’from the toolkit, as this may have misleading connotations
l target different types of users within the toolkit (e.g. director-level CCG managers and redesign project leads) because these individuals will have different interactions and understandings of evidence and capabilities
l simplify and reword the language used in some questions to increase usability
l include the criterion‘assessing local applicability and relevance of evidence’to make clear the purpose of the toolkit.
On the basis of this feedback, other offline comments and emergent findings regarding the capabilities, we revamped the toolkit questionnaire. In addition to using stakeholder feedback, we looked at examples of toolkits developed by academics for practitioner audiences, including the normalisation process toolkit (http://normalizationprocess.org). This toolkit includes a number of user-friendly features, such as a radar plot, which visualises answers and enables users to understand potential strengths and weaknesses (in that context, implementation of new technology). The radar plot can also be used to visualise‘performance’ against specific capabilities (e.g. showing high‘expert engagement’vs. low‘effective framing’). We decided to include a radar plot in our toolkit to enable users to get a sense of how well prepared their organisations and teams are to use evidence. A consequence of this decision is that item wording needs to elicit answers that are not simply‘yes’/’no’or open-ended, but reflect different levels of capability (e.g. Likert-scale types of questions). We therefore took particular care about how items were phrased to make the production of a radar plot possible.
Against this background, we presented an updated toolkit at our second national workshop (18 May 2016). This was a much larger workshop, comprising 46 participants with backgrounds in a number of NHS organisations, including CLAHRCs, Academic Health Science Networks, NHS commissioning policy units, CSUs and foundation trusts. We presented the findings of our research and then used interactive exercises to elicit participants’feedback about the toolkit. After we made a short presentation describing the toolkit, the workshop participants were split into five groups. Each group was asked to examine three or four questions from the toolkit for approximately 20 minutes and offer their feedback on these. We also supplied each group with sticky notes and asked them to give comments on question sets. For example, one group was asked to comment on the set of questions concerning‘managing expert collaboration’, while another was asked to debate and discuss the appropriateness and practical usefulness of the set of questions concerning‘engaging the expert’.
Owing to the interactive nature of this workshop, the feedback we gathered was very rich and immensely helpful. The workshop participants made specific and constructive comments on all aspects of the toolkit. The participant comments and suggested questions per capability are summarised inTable 8and were used to inform the final toolkit development. Feedback informed significant changes, and the resulting toolkit is presented inAppendix 5. This toolkit would be accompanied by a summary table that defines different types of evidence and different capabilities with concrete examples.
Chapter summary
In this chapter, we described our toolkit for improving evidence use in commissioning. The development of this toolkit was shaped by our research findings, by the literature and by feedback from our SSAP and workshop participants. We have described the importance of appraising capabilities for evidence use, both within the team and at the organisational level. We have selected a radar plot tool to enable easy interpretation among users. It is important to note that this is just an overview of our toolkit. We also secured an Economic and Social Research Council Impact Acceleration Grant to refine the design and develop this toolkit (e.g. wording) into a user-friendly online learning resource. Work (6 months) began on this resource immediately after the completion of the current NIHR project. The resource ensures that our‘actionable findings’have the best chance of having an impact on practice. The work was being carried out in collaboration with Health Education England, the West Midlands CLAHRC and local AHNs. The resource is available at www2.warwick.ac.uk/fac/ soc/wbs/research/ikon/commissioning/.
TABLE 8 Stakeholder feedback on toolkit
Capability General comments/considerations Question selection
Sourcing and evaluating evidence
Make explicit reference to NHS Academic Health Science Network, library and knowledge services
Do you have access to evidence?
Are there explicit criteria to assess evidence? Engaging experts Consider:
l Expert categories (e.g. clinicians, patients)
l When and how experts evaluate evidence
l How experts respond across the redesign process
Include questions about identifying, not only engaging, the right experts
Effective framing Is it about framing or scoping or defining? Is the problem a symptom of another, deeper problem?
Can we solve the problem by adapting what we do now, copying what others do, or starting from scratch?
Managing roles and expectations
Some projects may not have clearly defined roles so a flexible approach is adopted
Think about project life cycle
Too advanced for commissioners!
–
Managing expert collaboration
What does‘balancing’and‘preparing’for divergent interests mean?
Is it managing or facilitating/accommodating?
How do you measure divergence of interests?
Chapter 8
Exploratory study of evidence
production
I
n the previous chapters of this report, evidence use by managers was considered as our main response to the commissioning brief (i.e. research on‘pull’). A secondary aim was to consider the ways in which the production of evidence shapes its use in health-care organisations (research on‘push’). In this chapter, we consider the design and production of evidence-based products in one large guidance-producing organisation. We use the lens of‘inscribed meanings’to conduct this part of the analysis.60Inscribed meaningsconstitute, in effect, the‘instruction manual’for evidence, whereby the view that designers take of how target audiences are meant to use evidence become incorporated into the design of their products.63,64 Accordingly, we set out to explore what sort of imagined and projected users are inscribed in evidence products aimed at improving design and commissioning of health-care services. From our interviews and visits, we saw three main discourses, which we label as‘the discourse of production’,‘the discourse of audience feedback’and‘the discourse of implementation consultancy and direct marketing’. These are described next.