DATA COLLECTION TECHNIQUES AND ANALYSIS Comparing existing frameworks

The key goals of the comparison of the frameworks are to identify the similarities and differences. Similarities will be carried over to the universal framework as these would facilitate the adoption of the new framework by the owners of the reference frameworks. The similarities will also be reviewed for their functionality and how these can be harmonised across the frameworks. The differences may potentially challenge the use of a universal framework and these will be assessed for the contribution to the overall decision-making process. Differences that are deemed relevant to benefit-risk assessment will be considered for the universal framework, while those differences found to be related for the purpose of fulfilling specific jurisdiction requirements may be omitted. Beyond the content of the framework, the flow of processes will also be compared. The ideal flow should correlate closely to the processes undertaken by a reviewer.

35

Validating the proposed universal framework and templates

To carry out the systematic collection of opinions and comments, study tools will be developed. Questionnaires, surveys and decision conferencing are common tools employed for such purpose. One established approach to develop a survey is the use of the Delphi method for structuring group communication process to ensure the effectiveness in allowing a group of individuals to solve a complex problem (Linstone et al., 2002). This will be further explored here.

Delphi Technique

Linstone et al expounded on the application of the Delphi process, which can be carried out either using the traditional "Delphi Exercise" or the newer “Delphi Conference” manner. The traditional approach requires the draft questionnaire to be sent via hardcopy documents to the respondent group for feedback on the proposed contents. With the inputs returned from the respondents, the questionnaire is revised and the group is again sought to review their original answers based on the new questionnaire. This approach is similar to a combination of a poll and a process to shift the need for a large communication to the smaller team developing the questionnaire. The newer "Delphi Conference" replaces the hardcopy exchanges with real-time communications afforded by the current technology and thus reduces the time to obtain the responses. Regardless of the approaches, there are four distinct phases. The first phase determines the subject for discussion and provides the initial content deemed relevant for the questionnaire. The second phase aims to understand of how and where the group agrees or disagrees on the contents.

Disagreements are then explored in the third phase to find out the underlying reasons for the differences and review them. The final phase includes the final review by the group when all previous responses are reviewed and the outcomes have been fed back for consideration. Okoli et al (2004) showed an alternative but similar way for executing the Delphi method (Figure 2.2) and also further explained on the process of selecting the panel of experts forming the respondent group. Simple statistical analysis of the responses can be carried out to assist in the analysis of the outcomes.

36

The use of the Delphi method is frequently employed for postgraduate and higher learning degrees (Skulmoski et al., 2007). It has been utilised widely in social sciences (Landeta, 2006) as well as in healthcare systems, such as the identification of characteristics for injury surveillance and long term prevention (Mitchell et al., 2009) and a consensus statement among respiratory specialists on the health effect of asbestos (Banks et al., 2009). It is also considered a versatile tool and can be used in selecting and defining a further research topic (Okoli et al., 2004).

Figure 2.2 One approach of carrying out the Delphi method*

*adapted from Okoli, 2004

Comparison of the Delphi Technique with other questionnaire techniques

Okali et al (2004) also provided differences between a traditional survey against a questionnaire constructed via the Delphi method. Some surveys may require statistical tools to power the findings and thus require an appropriate sample size, the Delphi method does not require a statistical number of participants. However, the ideal number of members in a panel has been recommended to be 10 to 18. While a survey tends to extrapolate a conclusion based on a select group of individuals, the Delphi method can draw out expert opinions that are superior to the views of the individuals. As per the Delphi method, there is a follow-up to the data collected during

37

the process, leading to a richer amount of relevant data. This however is limited when conducting a traditional survey.

Linstone et al (2002) also shared on the limitations of the Delphi method. One of these is discounting the future, since the subjective nature of the inputs tends to change over time and the applicability of the questionnaire would be affected. With the multiple of contributing opinions, there is a tendency to select a few and fit them into a familiar context. This behaviour is called the “simplification urge”. As such, the final questionnaire may not also represent holistically or entirely the actual situations.

Another limitation is the illusory expertise, where the group, typically a panel of experts in the field, may not be the best at predicting the relevance of the contents.

This may be due to the panel members being too specialised in a niche area, leading to a failure to understand the interactions of the entire system at large. Therefore, it is important that the panel selection is carried out effectively to best optimise the Delphi method, as the entire process is dependent on their inputs. Bolger et al (2011) investigated the impact of various factors related to the panel, including degree of confidence, expertise and majority positioning. It was found that majority opinion is the strongest influence and the conduct of Delphi method should aim to reduce this along the process. A recent more scientific method of weighing and pooling scientific advice, the Cooke method may be considered (Aspinall, 2010). Its goal is to quantify uncertainty and not eliminate this unavoidable concern from the decision-making process.

Validity of questionnaire techniques

The validity of the questionnaire will determine the robustness of the outcomes. A basic way of looking at validity would be the content validity, which is how well the item on the questionnaire can measure what it is intended to measure and possesses the appropriate level of emphasis and focus (Nunnally and Bernstein, 1994). The importance of content validity should be emphasised as it forms the foundation of accurate measurement of the outcomes (Yaghmaie, 2003). It should be noted that in order to achieve content validity, there must be face validity. Face validity has been defined as the appropriateness of the items in relating to the goals of the questionnaire (Nunnally and Bernstein, 1994; Anastasi, 1988; Nevo, 1985). As for most research, the conclusions are generalised and extrapolated beyond the

38

original research. It is important that such claims are supported by causal relationships between the observations i.e. internal validity (Johnson, 1997). Indeed, the ultimate aim of a questionnaire is to achieve construct validity, where the logical relationship between the outcomes and the outcomes with the system is being established (Guyatt et al. 1993).

Design conferencing

Another method of systemically reviewing a group’s input is to conduct a decision conference (Phillips, 2006). The process starts with a discussion on the objectives (Figure 2.3). To achieve these objectives, the model that captures the key elements is required to resolve the issues. Discussions would involve personal judgments, intuitive opinions and feelings of unease. Exploring the observed difference may identify new insights that feed into improving the model. With the new inputs, the process is repeated again until the model reflects the new perspectives. Decision conferences help to generate a shared understanding of the issues, without requiring consensus about all issues. It can also develop a sense of common purpose, and find the best way forward in the midst of disagreements. Decision conferencing can be frequently employed during workshops in which many new initiatives can be generated.

Figure 2.3 A decision conference process*

*adopted from Phillips, 2006

39

The choice of methods for carrying out a study would be dependent on the availability of both time and experts. It is expected that both questionnaire techniques and decision conferencing will be the main tools employed for this study to achieve the objectives.