Consensus methods: exploration of consensus methods

When there is little, insufficient or too much information regarding a particular research topic, consensus methods are a means of dealing with conflicting scientific evidence or understudied topics (222). Consensus methods determine the degree to which experts or the general public concur in regards to a specified issue (222). Consensus methods are often concerned with measures of agreement. This takes two forms, firstly the extent to which the individual agrees with the issue specified. Secondly, the extent to which each individual agrees with one-another (222).

I decided that a consensus method would be used to develop a core outcome set, as this has been used successfully in that way for numerous past empirical studies (211,223–225). As described previously, COMET suggests this is the ‘gold standard’ method for development of a core outcome set (210). This was important, as the sources of data were not judged for quality, so agreement was needed in the two respects described above, agreement with the proposed core outcomes and agreement within the group. To reiterate, stakeholder agreement (consensus) regarding the proposed PPD outcomes was essential, as the proposed outcomes were extracted from numerous sources that may not be valid, true or representative of all placements. The stakeholder consensus provided a filter to remove anything that is not relevant or agreed upon.

95

4.4.1. Rationale for choosing Delphi methodology

I made a decision to use the Delphi method over the available alternatives. The first alternatives were traditional data collection techniques such as questionnaires or

interviews. However, whilst a questionnaire or interview provides a range of opinions or judgement on an issue, none of these attempt to gain consensus. A small number of consensus methods exist, namely Delphi, focus groups, round tables or nominal group technique (222,226). Nominal group technique generally involves one or two questions sent to the group in advance of a face-to-face meeting (227). They then spend time

independently reflecting on their own ideas before a facilitator asks each individual to state a single idea in a round robin fashion. It has been recommended that there are no more than 7 participants in a group (228). Whilst this method is more suitable than a

questionnaire or interview, as there is a consensus focus, it did not meet the needs of my project. Firstly the small group size of 7 was not feasible for the multiple stakeholder groups I wanted to include. Secondly, the face-to-face element was difficult considering the most significant stakeholders would likely be engaged in international work at the time of the research, so face-to-face meetings may exclude important stakeholders.

I chose the Delphi technique, as it met the needs my research needs in terms of group size and online dissemination. Delphi methodology was first used by the Rand Corporation in USA in the 1950s; in defense research (229). It has since been used in numerous fields, such as business, health and education. The Delphi is a widely used and accepted method of gaining consensus of judgments on a particular issue. It is a controlled process involving a series of questionnaires often named ‘rounds’ to collect opinions until consensus is reached (211). Researchers use Delphi to explore levels of agreement and disagreement amongst experts. Each stage is focused on improving the results gathered in the last stage based on the comments and median score of the previous ‘round’ (230). The Delphi method is an iterative method that uses numerous rounds to collect data and condense individual opinions into a group consensus (231). It involves a series of questionnaires that record participant agreement with statements concerning a particular topic. It has become a way of generating consensus amongst key stakeholders regarding core outcomes in

healthcare (211).

It is argued that Delphi has greater advantages in terms of generating a core outcome set than round table discussions or focus groups, (230). For example, participants do not

96

actually interact with each other, so it is less likely that group situations may be dominated by one individual. This method aims to overcome some of the problems associated with decision making in groups, i.e. the dominance of one or two individuals or those with vested interests (222). Also the decision to change an opinion is not dominated by the social variables, such as the desire to be seen to agree (225). This is particularly important in regards to my research, as whilst lots of literature exists regarding the outcomes and benefits of international placements, I needed to ensure that those people with knowledge and experience of international placements agreed with them. As there is no agreed outcome set, seeking the opinions of those in the field provided a way of ensuring the proposed set is valid and agreed upon.

The method works well for a research question that is not well answered by precise analytical techniques, but could benefit from collective subjective judgements (232). The question of learning on international placements does not well lend itself to precise analytical techniques; collecting subjective judgment’s seemed to be the most logical way to gather this data in the timescale of this thesis.

Since its introduction in the 1950’s there have been various types of Delphi studies. The Delphi technique has evolved through time and with technology (229). The classic Delphi study involved a questionnaire sent by post to Delphi participants (229). Since, an E- Delphi has been developed to incorporate the same methodology into modern technology, so the survey is administered via email or an online web survey (233). There are many variations in the way a Delphi is conducted. The ‘modified’ Delphi replaces the first round with a focus group, face-to –face interviews or literature review (233). There is also a real- time Delphi where experts are physically in the same room. Delphi is a methodology particularly used with policy makers; hence a policy Delphi is a way of reaching consensus on a future policy.

I chose to use a modified E-Delphi. It was modified as the first stage was a systematic review, stakeholder workshop and meta-synthesis. I chose to administer it in an online manner (E-Delphi), as opposed to postal, as it was easier, quicker and allowed me to include participants that were not currently in the UK. Choosing to use online Delphi software allowed for more efficient collation of responses and reporting back to

97

however the benefits of this method outweighed the negatives in this instance (cost, administration to those overseas, time). I decided to use a modified Delphi as if I had not included a literature review ahead of the Delphi, the data provided by stakeholders alone may have been too general for this purpose and benefits/costs may have been missed. Throughout the literature, stakeholders describe learning using generalised broad terms (13). I needed concrete, specific outcomes to use in measurement tool, so chose to search for these within the literature. A modified Delphi process (a systematic review in the first round) is appropriate and common if basic information concerning the target issue is usable and available (234).