Questionnaire Development

The questionnaire was developed following the completion of the qualitative phase of the research. Preliminary qualitative results, prior research and existing theoretical work informed the development of the questions and refinement was achieved during team meetings that included all members of the research team.

The entire project team was involved in two full-day workshops in order to compare field notes and narratives, and engage in inductive thematic analysis –i.e. identifying themes from the data (rather than from theory) that had particular bearing on our research objectives and on the design of the survey. This involved examining individual case narratives in great detail as well as comparing narratives, i.e. across research settings (e.g. redesign projects) and sites (PCTs). This enabled us to identify common themes across narratives and enhanced our understanding of how these manifested in decisions made. For example, a common theme was that practitioners relied upon a plurality of forms of evidence (clinical and non- clinical) at various stages of commissioning and that the transition across commissioning stages (e.g. designing a clinical model and then

commissioning it) posed significant translational challenges. As a result of this intensive joint work we were able to ensure that these issues were fully reflected in the survey design.

The questionnaire also drew on two particular sources of previous work in this area. Weatherly et al (103) and Gallego et al (104). In a descriptive survey of use of evidence in local health policies Weatherly and colleagues investigated the balance of influences of different evidence types in

decision-making in the NHS. The study aimed to explore the use of evidence (economic evidence in particular) in the development of local health policy. It focussed on exploring the consumption or use of evidence in the decision making process as this was perceived to be little understood or

investigated. The study consisted of an in-depth questionnaire sent to each of the 102 English Health Authorities, semi-structured interviews with 10 questionnaire participants and a review of 26 documents relating to local health policy.

The authors proposed two different types of evidence commonly used in decision making of this type;

Internal evidence – experiential evidence gained from professional experience, tacit knowledge etc.

External evidence – based on research from studies, published papers, guidelines etc.

The research by Weatherly et al (103) aimed in part to discover what types and sources of evidence were commonly used, what improvements were needed and what barriers to evidence use existed. The survey received a 67% response rate and concluded that 66% of evidence used in decision making came from external sources and 34% internal. Key external sources included government reports and guidelines such as NICE guidelines rather than academic research reports, partly explained by government requirements to adhere to such guidelines as targets or

examples of excellence and partly due to the perceived quality of this type of resource.

Key barriers to evidence usage were much as previously reported by others, including lack of time, lack of availability of resources and difficulties

synthesising evidence to apply at a local level. The authors conclude that there is a need for more national guidance such as that provided by NICE but also the need for more accessible and locally applicable summary documents for use in decision making situations.

Gallego et al (104) investigated resource allocation and health technology assessment in Australia in a local health department serving just over a million people “to gauge healthcare providers’ and managers’ perceptions of local level decision making”. Using previous research and pilot interviews they identified factors important in the introduction and approval of health technologies, including ‘evidence on safety/quality’, ‘evidence on

effectiveness,’ ‘evidence on cost-effectiveness (i.e., the cost per quality life- year gained)’, ‘total costs impact to the Area/hospital/department,’ ‘burden of disease (i.e., the number of people affected),’ ‘disease severity,’ ‘lack of alternative treatment,’ ‘equity’ and ‘patient preferences.’ Although their survey was not representative they found that ‘less than a third of participants agreed with the statements that local decision-making

processes were appropriate, easy to understand, evidence-based, fair, or consistently applied’ and that ‘decisions were reportedly largely influenced by total cost considerations’ and policy directives. They concluded that policy makers engaged in renewed initiatives in HTA should take heed of the decision-making contexts within which HTA can successfully be

implemented and that ‘any HTA initiative should be accompanied by efforts to improve decision-making processes’.

Potential areas of inquiry were tested with the whole research group and exact research questions to be answered by the survey were refined at these meetings. Particular areas included the development of a list of current likely organisational constraints and practical impediments to co- production and to decision making, and also the practical experiential

evidence sources most likely to be being used in current PCT commissioning practice. As a group we were also able to establish methods to identify individual decisions and to identify the type and size of decisions being made. We were also able to consider methods of incorporating the influence of different interested parties on the decision making process and the co- production environment.