Consensus Development Process – Expert Measures of Integration

As noted in Section 2.2.3 there are few validated measures of health system integration. The use of

expert opinion is an established methodology in health research (Kuzel, 2010) , particularly when

developing indicators where there is no data or consensus available, or where there is limited

evidence (Campbell & Cantrill, 2001). Consensus development processes such as the nominal group

technique have a long-standing history of use in the healthcare setting, however the methodology is

dependent upon the credibility of the participant experts and the methodological rigour of the process

(Campbell & Cantrill, 2001). For these reasons a methodology that engaged a broad selection of

experts in consensus development around appropriate measures of system integration was used.

4.2.1 Purpose

A modified, iterative consensus development process, called a nominal group technique, was used to

gather an expert advisory panel’s input into how system integration might be measured through

electronic information exchange. The nominal group technique or process (NGT) is preferred over

other consensus techniques as it allows for face-to-face discussion, and facilitates more equitable

contributions from groups of participants due to its structured format and anonymous ranking process

(Campbell & Cantrill, 2001). The process has been widely used in exploratory health service research

and for preliminary investigations prior to more formal research methods such as surveys. In one

example, valid, important and useful financial indicators in the health care sector led Pink and

colleagues (2007) to use an expert panel, literature review, and survey methodology to select

indicators and methods of calculation. This combined methodology produced indicators which were

4.2.2 Procedures

Experts were identified by “social acclamation”, a reasonable strategy according to Shanteau and

colleagures (2002). Their experience and training gave them the ability to understand complex

systems due to their relational and causal knowledge (Abernethy, Horne, Lillis, Malina, & Selto,

2005); it is this comprehensive understanding of systems which was required in this conceptually

challenging performance measurement task. The participants were purposively sampled to include

recognized experts from a variety of sectors and disciplines, such as primary care and public health,

with a sound knowledge of the regional health system, and an understanding of electronic information

systems (Van de Ven & Delbecq, 1972). Participation was also determined by availability and

willingness to participate. Recommended group size for nominal group process is no larger than eight

to twelve individuals (Fink, Kosecoff, Chassin, & Brook, 1984; Van de Ven & Delbecq, 1972).

Figure 8 presents the four phases of the nominal group technique through which the panel of

Figure 8. Nominal Group Technique Phases for Expert Panel

Ideally, consensus processes should be informed by a concise summary of existing empirically-

derived data (Fink et al., 1984). Introductory materials identifying indicator selection criteria, as well

as interoperability frameworks, were distributed ahead of the meeting and introduced participants to

both the domain of interest through a literature review, and the procedures for the nominal group

technique (see Appendix A). Participants were asked to read the introductory material ahead of time

and consider the question: “What metrics satisfy the indicator selection criteria (scientifically sound, Phase 1 - Introduction to

Literature & Nominal Group Technique - Literature review &

frameworks sent in advance of the meeting

- Participants instructed to note their views on prompt question & come prepared to discuss

Phase 2 - Brainstorming & Group Discourse

- Discussion of both the NGP and literature

- Individual reflections - Round Robin of metric suggestions until exhustion - Group discussion of each metric for clarity

Phase 3 - Ranking

- Experts anonymously rank all

metrics on a 9-point Likert scale using selection criteria - Results collated in real time and fed back to panel for brief discussion

Phase 4 - Refining - Summary report of

proceedings & rankings emailed to all participants

Conference call to further define the indicators, their relevance and feasibility Refinement of indicators based on feasbility & availability of data

relevant, feasible and communicable), and might contribute to our goal of measuring system

integration using between-provider electronic health information exchange?”

While there are no accepted benchmarks to identify when consensus is reached, consensus

must be defined in advance and the more demanding the criteria the better (Fink, Kosecoff, et al.,

1984). Consensus was achieved in this process, when the mean score of all participants’ rankings for

an indicator was greater than or equal to 7, and 70 per cent or more of the participants ranks the

indicator at 7 or higher. Despite its acceptance as an established practice in performance

measurement, the risk remains that expert panels in a nominal group process may reach consensus but

agree on indicators that do not satisfy the measurement objective. In order to mitigate this risk the

literature review provided an empirical foundation for methodological decisions.

However, we know that healthcare system measurement constructs must be clearly defined

and consistently understood in order to populate performance measures with reliable data (The

National Quality Forum, 2008). This is particularly important in nascent domains such as that

addressed by this study, where it is necessary to decompose concepts into smaller foundational

elements. In order to share this knowledge with a wider community for future validation and use, an

explicit conceptualization of regional electronic health information exchange and system integration

was proposed through the development of an ontology to formalize the Conceptual Model and

Measurement Framework (see Figure 5). Gruber (as cited inUschold & Gruninger, 2004) defines an

ontology as “a formal, explicit specification of a shared conceptualization”. Gandon's (2010)

definition is a little more accessible; it is as a “hierarchical organization of the relevant concepts and

relevant relationships between these concepts, as well as rules and axioms that constrain these