Volume 37
Article 2
8-2015
The Delphi Method Research Strategy in Studies of
Information Systems
Richard Skinner
C.T. Bauer College of Business, University of Houston, [email protected]
R. Ryan Nelson
McIntire School of Commerce, University of Virginia
Wynne W. Chin
C.T. Bauer College of Business, University of Houston
Lesley Land
UNSW Business School, The University of New South Wales
Follow this and additional works at:
https://aisel.aisnet.org/cais
This material is brought to you by the AIS Journals at AIS Electronic Library (AISeL). It has been accepted for inclusion in Communications of the Association for Information Systems by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contact
Recommended Citation
Skinner, Richard; Nelson, R. Ryan; Chin, Wynne W.; and Land, Lesley (2015) "The Delphi Method Research Strategy in Studies of Information Systems," Communications of the Association for Information Systems: Vol. 37 , Article 2.
DOI: 10.17705/1CAIS.03702
C
A
I
S
ssociation for
nformation
ystems
Research Paper ISSN: 1529-3181
The Delphi Method Research Strategy in Studies of
Information Systems
Richard Skinner
C.T. Bauer College of Business, University of Houston, USA
R. Ryan Nelson
McIntire School of Commerce, University of Virginia, USA
Wynne W. Chin
C.T. Bauer College of Business, University of Houston, USA
Lesley Land
UNSW Business School, The University of New South Wales, Australia
Abstract:
In this paper, we discuss the nature and use of the Delphi methodology in information systems research. More specifically, we explore how and why it may be used. We discuss criteria for evaluating Delphi research and define characteristics useful for categorizing the studies. We review Delphi application use in IS research over the last 23 years, summarize lessons learned from prior studies, offer suggestions for improvement, and present guidelines for employing this distinctly useful qualitative method in future information systems research studies.
Keywords: Delphi Method, Experts, Panel, Anonymity, Qualitative, Iteration, Feedback, Bias.
1 Introduction
Even though qualitative research techniques have been used in information systems (IS) studies for many years, disproportionally low numbers of qualitative papers have been published in top-tier IS journals (Galliers & Huang, 2012). Conboy, Fitzgerald, and Mathiassen (2012) identify this dearth of qualitative publications as being a result of inadequate numbers of qualitative courses in universities, inequity with quantitative content in general research method courses, negative bias perceptions against qualitative approaches from editors and reviewers in leading journals, and a dwindling number of qualitative experts that include leaders, champions, supervisors, and reviewers of qualitative material.
However, this perspective may be changing. As Sarker, Xiao and Beaulieu’s (2013) informative MISQ guest editorial illustrates, qualitative publication numbers increased from 2001–2012 across four of the seven journals included in the Association for Information System (AIS) Senior Scholars’ basket of journals. This growth suggests the increasing viability of qualitative method use in IS research. As a result, qualitative techniques previously neglected by the IS field have gained in relevance, which, in turn, has made strategic qualitative investigations increasingly significant. Consequently, as qualitative research’s importance has grown, so has the requirement for clear qualitative method guidelines.
In this paper, we partially address this need by providing a guide to one of these methods—the Delphi—in the IS field. Developed by the Rand Corporation in the 1950’s, the Delphi method is a methodical and interactive research procedure for obtaining the opinion of a panel of independent experts concerning a specific subject. Using previous IS papers that employ the Delphi method as examples, we provide recommendations for assessing and applying the Delphi method when undertaking IS research. This approach is necessary due to the majority of IS papers concentrating on reporting the Delphi study result (i.e., using the Delphi technique for IS theory generation rather than for reflection and evaluation of the method itself (Holsapple & Joshi, 2002; Day & Bobeva, 2005).
Note that, when it comes to selecting an appropriate method for a qualitative research study, we do not suggest that Delphi method selection will be suitable in every scenario. As Benbasat, Goldstein, and Mead (1987) highlight, researchers’ goals and the nature of their research topic influence what research strategy they select. As a result, certain research conditions are non-conducive to using the Delphi methodology. However, Rowe and Wright (2001) suggest that the Delphi method is effective when statistical method use is unsuitable, several experts are available, the alternative is simply to average the forecasts of several individuals, or the alternative is using a traditional group. We propose that the Delphi method is particularly appropriate for acquiring expert recommendations when addressing an IS research issue. Due to these specialist authorities having extensive knowledge of specific areas of IS interest, using the Delphi method confirms Powell’s (2003, p. 376) observation that “the method... is exceptionally useful where the judgments of individuals are needed to address a lack of agreement or incomplete state of knowledge... the Delphi is particularly valued for its ability to structure and organize group communication”.
This paper is organized as follows: in Section 2, we review the method’s characteristics and appraise how to undertake the technique as part of IS research. In Section 3, we examine previous IS papers’ adoption of the Delphi method, review IS Delphi methodology use, and summarize lessons learned. In Section 4, we conclude the paper and note observations about the method and its potential for future use in the IS field.
2 Delphi Research
2.1 Characteristics of the Delphi Method
The Delphi method first came into being in the early 1950s. Subsequently, over the next 60 years, its reputation as an effective approach to technological forecasting grew, waned, and grew again1. Notwithstanding these changes in popularity, previous studies have sought to define and characterize the
1
method. From these reports, we suggest that studies using the Delphi method should possess the following generic characteristics:
• Use of experts: Lilja, Laakso, and Palomaki (2011) suggest that an expert fit for a Delphi
panel requires the individual to be at the top of their field of technical knowledge, interested in a wide range of knowledge not only in their own field but everything around it, able to see connections between national and international and present and future development, able to see connections between different fields of science, able to disregard traditional viewpoints, able to regard problems from not only known and safe angles but also unconventional ones, and interested in creating something new. Rowe and Wright (2001) suggest using heterogeneous experts. We describe the requirements that experts should have in more detail in Section 2.
• Panel: the panel should consist of a group of selected experts with no size limitations.
However, because the main task is to include experts who have the greatest knowledge and experience in the field under review, group size often remains fairly small. Delbecq, Van de Ven, and Gustafson (1975) suggest a panel as little as four experts under ideal circumstances. However, under typical circumstances, the panel is usually between 10 and 30 experts (Baldwin-Morgan, 1993; Doke & Swanson, 1995; Keil, Tiwana, & Bush, 2002; Akkermans, Bogerd, Yucesan, & van Wassenhove, 2003; Daniel & White, 2005; Kasi, Keil, Mathiassen, & Pedersen, 2008; De Haes & Van Grembergen, 2009; Baldwin & Trinkle, 2011; Worrell, Di Gangi, & Bush, 2013). Insofar as research studies have not found a consistent relationship between panel size and decision making effectiveness (Brockhoff, 1975; Boje & Murnighan, 1982), it is highly unlikely that another equally expert group will produce radically different results from a panel of 15 experts (Martino, 1985).
• Anonymity: this characteristic supports panelist independence by avoiding the official position
status of a panelist potentially affecting others' opinion, expression problems, fear of losing face, or fear of attitudes that might be inappropriate to express in public (Lilja et al., 2011). It also removes the potential for mimicking others and provides a safety net for panelists from having to act as competitors. This guarantees more-objective answers and results. We evaluate anonymity's central role in countering judgment bias in more detail in Section 2. • Rounds: the Delphi method is executed in a series of rounds (Von der Gracht, 2012). Insofar
as two rounds are considered the minimum (Bradley & Stewart, 2003), between three and six rounds are required to facilitate realistic findings (Linstone & Turoff, 1975; Custer, Scarcella, & Stewart, 1999). Up to 10 rounds have been suggested as necessary for achieving consensus (Lang, 1994). However, Rowe and Wright (2001) suggest that three structured rounds are generally sufficient.
• Iteration and feedback: opinions are collected for analysis and information on the answers is
fed back to the panelists for comments and/or as a basis for the next round. In using this feedback, the panelists are obliged to justify their choices, with the build of information proceeding round by round so that the previous phase becomes the source for the next.
2.2 Deciding to Use the Delphi Method
Given a particular research subject, researchers must consider whether the Delphi method is the most productive technique for acquiring maximum insight. This consideration obliges researchers to appreciate the method’s advantages/strengths versus its limitations/weaknesses. Hung, Altschuld, and Lee (2008) identify papers reviewing these characteristics in detail (e.g., Eggers & Jones, 1998; Franklin & Hart, 2007; Gordon, 1994; Hartman, 1981; Hsu & Sandford, 2007; Lang, 1994; Linstone & Turoff, 1975; Mitchell 1991; Powell 2003; Price, 2005; Williams & Webb, 1994; Yousuf, 2007). Table 1 summarizes their report outlining the method’s respective strengths, advantages, weaknesses, and limitations:
Table 1. Comparison of Advantages / Strengths versus Limitations / Weaknesses of the Delphi Method (Hung et al., 2008)
Advantages / strengths Limitations / weaknesses
Consensus building Group pressure for consensus—may
not be true consensus
Future forecasting Feedback mechanism may lead to
conformity rather than consensus Bring geographically dispersed panel experts
together
No accepted guidelines for determining consensus, sample size, and sampling techniques
Anonymity and confidentiality of responses Outcomes are perceptual at best
Limited time required for respondents to complete surveys
Requires time/participant commitment
Quiet, thoughtful consideration Possible problems in developing initial
questionnaire to start the process Avoids direct confrontation of experts with one
another (encourages honest opinion, free from group pressure)
May lead to hasty, ill-considered judgments
Structured/organized group communication process Requires skill in written communication Decreasing somewhat a tendency to follow the
leader
Potential danger of bias—surveys are open to researchers’ manipulation Focused, avoids unnecessary side-tracking for
panelists
Selection criteria for panel composition
Ties together the collective wisdom of participants Time delays between rounds in data
collection process
Cost effective and flexible/adaptable May force a middle-of-the-road
consensus
Validity, as the content is driven by panelists Concerns about the reliability of the
technique
Fairly simple to use Drop-outs, response rates
Beneficial for long-range educational planning and short-term decision making
Applicable where there is uncertainty or imperfect knowledge, providing data where little exists before Effectively used to establish the basis for future studies
Accommodates a moderately large group
Note that solely evaluating the method’s relative merits and limitations allows the researcher only partially to determine its potential because reviewing the method’s merits through an absolute mode of information processing provides limited information regarding its overall appropriateness (Mussweiler & Epstude, 2009). Evaluating a Method also requires one to compare it with contrasting qualitative procedures to provide a more objective technique appraisal. Techniques that may be considered include action research (AR), which “immerses” the researcher in the research approach by simultaneously assisting in the practical problem solving of a problem and enhancing the competencies of organizational actors (Simon, 2000), and action design research (ADR), which is recommended when considering the design of ensemble technology artifacts (Sein, Henfridsson, Purao, Rossi, & Lindgren, 2011).
However, one of the most frequently used alternative methods to the Delphi method is the common survey. While the Delphi method is frequently considered a type of survey, albeit a more complex version, there are key differences between the two techniques. Common surveys seek to identify “what is”,
whereas the Delphi method attempts to address “what could / should be” (Miller, 2006)2. A further key difference between these techniques is the traditional survey’s dependence on a representative sample size. This dependency exists because surveys need to enhance the sample population’s external validity to the theoretical population of interest. Surveys must also identify statistically significant effects in the sample population. Although the Delphi method’s reliance on expert opinion removes these dependencies, generalizing the opinions and estimations of a non-representative group to a larger population may become problematic (Worrell et al., 2013). However, Worrell et al. (2013) further identify the expert panel having insights above and beyond a representative group, with panel results producing potentially fruitful benefits for both research and practice.
When assessing the feasibility of using the Delphi method, researchers should consider the following questions:
• Is the phenomenon of interest able to be evaluated by a panel of experts? • Are a sufficient (minimum) number of experts available to make up the panel? • Is anonymous feedback from the panel of experts feasible?
• Are experts able to dedicate sufficient time to assess the phenomenon over multiple iterations of feedback and evaluation?
• In the absence of precise analytical techniques, is gathering subjective judgments moderated through group consensus the only approach possible (Linstone, 1978)?
• Is personal contact not possible due to time and cost constraints or is it not desirable due to concerns about the difficulty of ensuring democratic participation (Linstone, 1978)?
The Delphi method is, therefore, most effective when research method alternatives are not viable or when constraints exist that cannot easily be overcome when attempting to gather impartial data. However, in the absence of a recognized group of experts or where more effective analytical techniques exist, the Delphi method may not be appropriate. Day and Bobeva (2005) suggest that, since Delphi method inquiries are anchored in aggregations of opinion, they are not helpful for investigating an individual’s psychosocial conditions3. As a result, Delphi method research is not recommended for research where nuances and experiences of human behaviors must be studied in situ. Veltri (1985) further suggests that Delphi method use is unsuitable when:
• The method is applied for a purpose other than achieving expert consensus • The Delphi process ideals are violated
• Objective data are available
• Experts are unavailable or reluctant to participate, and
• When the facilitator does not have the appropriate knowledge, education, or time to direct the study or interpret outcomes.
2.3 Guidelines for using the Delphi method
Even though researchers may decide that the Delphi method is suitable for their specific research needs, they may still be unsure as how best to proceed. As such, in this section, we provide practical assistance in understanding and implementing the Delphi method for IS research.
Linstone and Turoff (1975) suggest four broad, distinct phases to using the Delphi method:
• Phase 1: characterized by exploring the subject under discussion. Each individual contributes additional information felt to be pertinent.
2 For a detailed evaluation criteria comparison between the Delphi method and the traditional survey, see Okoli and Pawlowski (2004).
3
• Phase 2: reaching an understanding of how the group views the issue (i.e., where the members agree or disagree, what it meant by relative terms such as importance, desirability, feasibility, etc.).
• Phase 3: if there is significant disagreement, then it is explored to identify the underlying reasons for the differences and to evaluate them.
• Phase 4: final evaluation occurs after all previously gathered information has been analyzed and evaluations fed back to panelists.
These broad phases may be broken down into a step process categorized by three main stages: the exploratory stage, the distillation stage, and the utilization stage (see Figure 1).
Figure 1. Delphi Process (Adapted from Hallowell & Gambatese, 2010; Day & Bobeva, 2005)
2.4 The Exploratory Stage
The first stage of the Delphi method involves study preparation. It focuses on having a clear understanding of the research question, piloting the study to remove potential methodological obstacles, identifying and validating potential panelists, and selecting the final expert panel.
2.4.1 Identify Research Question
Research must be guided by clear and feasible research questions (Fraenkel & Wallen, 2000). If the questions are not clear, then the research may not result in useful evidence. Furthermore, readers may be
unable to evaluate the investigator’s efforts adequately (Fink, 1998). Skulmoski et al. (2007) suggest a continuum representing the degree of focus or openness of the questionnaire questions, which defines how broad or narrow the questions should be. For instance, the preliminary questions may be expansive and open-ended to evaluate a wider research area (Adler & Ziglio, 1996; Delbeq et al., 1975; Linstone & Turoff, 1975), or they may be more defined and structured to direct the expert panelists toward a pre-determined objective. The former approach provides a wider response range compared to the latter, which focuses on the panel’s collective intelligence (Skulmoski et al., 2007). The tradeoff with adopting a broad approach is that, with more data collected, subsequent data analysis becomes more time consuming.
2.4.2 Undertake Pilot study
The research team pilots the research questions’ effectiveness and proposed approach to ensure that the level of detail is appropriate, the panelists’ role is defined, and the instructions are easy to follow (Hallowell & Gambatese, 2010). Undertaking a pilot study also allows one to test for wording difficulties and gives one an opportunity to refine administration tasks (Jairath & Weinstein, 1994). Prescott and Soeken (1989) identify Delphi method pilot tests as providing researchers with an opportunity to refine the research instrument and test data analysis techniques. Piloting the questionnaire also allows for one to discover ambiguities (Gordon, 1994).
2.4.3 Identify Potential Experts, Select Experts, Validate Status, and Inform Panelists of
Study Requirements
The research team can use several approaches to identify expert panelists for the study. An expert convenience sample may be selected premised on the researchers’ knowledge of experts in the area of interest (e.g., Brancheau & Wetherbe, 1987; Baldwin-Morgan, 1993; McCubbrey, 1999; Schmidt, Lyytinen, Keil, & Chule, 2001; Keil et al., 2002). Conversely, Delbecq et al. (1975) recommend a procedure for expert canvassing for nominal group technique (NGT) research that supersedes any one individuals’ knowledge. Using the NGT procedure may be considered in the Delphi process because the differences between the Delphi and the NGT (Appendix 1) only occur once both studies are under way (i.e., Delbecq et al.’s (1975) NGT approach can be used at the outset of using the Delphi method) (Okoli & Pawlowski, 2004). We present an IS-adapted version of Delbecq et al.’s (1975) steps for panel expert identification, selection, and validation below:
• Prepare a knowledge resource nomination worksheet (KRNW). Creating a KRNW provides the research team with a template for categorizing panelists’ disciplinary background and/or skills. It includes a literature review of both academic and practitioner journals to support the fields and skills needed for membership (at this stage, focusing on the expertise of members rather than the experts themselves). The research team must also appreciate that IS practitioner experts may not have publications to their name that facilitate their identification as a result of undertaking a literature review. This being the case, using an expert convenience sample may provide a more effective approach to identifying practitioner experts.
• Review and select desired experts. Analyzing the relevant IS literature / individual contacts will provide an initial “wish list” of the most desirable experts to comprise the expert panel.
• Contact experts and request them to nominate further experts in the field. At this stage, there is no formal expert solicitation for joining the Delphi panel. Given that the initially identified experts are best positioned to recommend further experts in the field, the research team can request the original experts to nominate others. This approach provides the research team with the largest overall selection from which to choose the final panelists. This step acquires greater importance when considering subsequent attrition; should those experts who have provisionally agreed to panel membership retrospectively judge the perceived participation overheads as too high, panelist member numbers may drop below the minimum quorum needed. By having the largest selection of experts from which to choose, subsequent withdrawals are contingently addressed. The research team, therefore, reaches out to the initial list of experts and requests their nomination for other experts in the field under research. • Ranking experts. As a result of the nomination process, it is also conceivable that the overall
list of experts will exceed the desired maximum panel number. Should this occur, priority ranking of experts takes place to reduce numbers to the desired panel size. This requires each
research team member to rank all experts independently premised on their perceived “expertness” in the subject under review. Each ranking is subsequently reconciled into an initial list of experts invited for study participation.
• Inviting experts to become members of the Delphi expert panel. Each expert panelist is contacted and requested to join the panel. At this stage, each provisional panelist is provided with the study subject, the method procedures, ethical consent requirements, and expectations of commitment necessary for effective panel membership. Although engaging, concise, and well-written questions may often entice expert participation (Skulmoski et al., 2007), experts are often very busy and unable to participate. However, the research team should be cognizant of the incentives for the experts to participate in the panel. These include being chosen to be a member of a diverse but select group, increasing knowledge as a result of consensus building, and enhancing visibility to the field as a result of panel membership.
2.5 The Distillation Stage
The second broad phase of the study develops the questionnaire, transmits it to the panelists, and collects / analyzes round responses.
2.5.1 Develop Questionnaire Using Methods to Minimize Bias
Having achieved participation agreement from the desired number of panelists, the research team provides a questionnaire that allows the experts to begin investigating the issue. Previously identified as a potential method limitation (Table 1), questionnaire development must be focused on removing the potential for biased feedback because cognitive shortcuts that misrepresent opinion or observation potentially may lead to judgment imprecision (Heath & Tindale, 1994). Contrary to objective techniques where the researcher is most likely to establish bias, should bias occur in a Delphi process, it is more likely to arise as part of the expert judgment process. In this eventuality, eight bias forms are suggested as most likely to impact the studies’ result quality negatively (Hallowell & Gambatese, 2010). To facilitate their recognition in the IS Delphi process, we overview these forms here:
• Collective unconscious: Durkheim (1982) suggests that individual beliefs are limitless unless constrained or directed by social forces such as peer pressure or dominance. The collective unconscious theory suggests that a “bandwagon” effect occurs when social forces compel an individual to conform to a majority position. Because bias may occur when a decision maker accedes to popular opinion without giving due consideration towards the merits of any one position, this effect must be considered in a Delphi method study.
• Contrast effect: Bjarnason and Jonsson (2005) propose that, when an individual is evaluating a criterion, the individual may be directly influenced by previous exposure to a similar criterion of substantially higher or lower value. The contrast effect occurs when a subject’s perception is enhanced or diminished by the value of the immediately preceding subject. In a Delphi method study, where panelists are required to identify differences among various factors, contrast effect bias may occur when ratings are given against factors that have substantially different values.
• Neglect of probability: this bias considers the scenario where individuals focus on the potential consequences of an outcome without considering the probability that that outcome will occur (e.g., if individuals underestimate the role of probability in a subjective quantification of risk). Accordingly, this bias occurs not as a result of using probability incorrectly, but as a consequence of disregarding the function of probability entirely. Because this bias is relatively common (Martin, 2006), controlling for it is important.
• Von Restorff effect: this bias occurs due to individuals being prone to remembering events associated with more severe outcomes as compared to less severe outcomes. If this occurs, it potentially misrepresents the perception of probability. For example, should a Delphi method study seek to evaluate a risk perception, then this effect may become widespread due to more extreme events being more likely to be recalled when making a subjective judgment. Krimsky and Golding (1992) identify this bias as especially important because subjects are more likely to overestimate probability values when an exceptionally high magnitude is involved. This overestimation may artificially inflate risk scores for events associated with high severity levels.
• Myside bias: Perkins (1989) suggests that myside bias occurs as a result of subjects generating arguments on only one side of an issue. It may also occur when individuals are unwilling to address objective viewpoints that counter a subjective position. Addressing myside bias is critical to the Delphi method’s success because individuals may be required to adopt a counter position that challenges a previously held viewpoint. Even though myside bias frequently occurs, Baron (2003) suggests that subjects are easily prompted to contrasting arguments on the issue’s other side. He also identifies that the original failure to have considered alternative positions is typically not the result of individuals not knowing the argument on the issue’s other side.
• Recency effect: this bias occurs when participants artificially inflate risk ratings as a result of similar incidents recently occurring outside of the study (i.e., recent events are given inappropriate levels of salience in relation to others). While this bias is fairly common and difficult to manage, eliminating those experts who have recently experienced events related to the study may be the most effective control.
• Primacy effect: this effect transpires as a result of unconsciously assigning importance to initial questions or observations to the detriment of following stimuli (i.e., the first stimulus is considered more important than the final observation). Accordingly, the research team must be aware of the subjects’ predisposition to conform to the primacy effect at the beginning of the Delphi method.
• Dominance: this bias type usually arises because one group member intimidates others to conform to the member’s viewpoint. However, using anonymous feedback in the Delphi method provides an effective control, with equal response weighting providing a further counter to domineering behavior. Anonymity also allows expert subjects to provide their own opinions without undue influence from other panel members.
To reduce the potential of these biases harming the study, research teams using the Delphi method should consider adopting the counter measures presented in Table 2:
Table 2. Example of Full-Width Format for Tables (Source)
Bias Control / counter measure Collective
unconscious
Require panelists to provide response justification during each round.
Contrast effect For each round / expert, randomize question order. Report final results as a median.
Neglect of probability Independently record probability / severity ratings for each risk identified.
Von Restorff effect Require panelists to justify their responses during each round. Have multiple survey
rounds.
Myside bias Require panelists to justify their responses during each round. Report final results as a
median.
Recency effect Eliminate individuals who have experienced recent similar events, ignore outlying
observations, perform multiple rounds, and report final results as a median.
Primacy effect For each round / expert, randomize question order.
Dominance Ensure expert anonymity.
Having addressed potential bias concerns, the starting position for the Delphi questionnaire becomes contingent on whether the questionnaire design is exploratory or confirmatory. Hasson, Keeney, and McKenna (2000) suggest that exploratory questions are best implemented by seeking informants’ views through initial open-ended questions or via a set of preliminary interviews. This approach is especially appropriate for social science research where situations may be vague, ill defined, or contradictory (Day & Bobeva, 2003). In contrast, the confirmatory approach is typically seen during follow-up studies (Brancheau, Janz, & Wetherbe, 1996; Gottschalk, 2000) where the scenario is generally less ambiguous. This confirmatory variety of Delphi design is customarily undertaken by giving the panel a predefined set of issues to explore (Niederman, Brancheau, & Wetherbe, 1991).
2.5.2 Transmit Questionnaire to Expert Panel
Research teams have different options when deciding how to interact with the expert panel. Cramer (1991) identifies Delphi surveys as originally pen and paper-based and returned to the research team via “snail” mail. While this approach is an option, using email and the Internet affords particular advantages to both researchers and panelist alike. These include the expediency provided by a quick turnaround that helps keep interest alive and participation high. Further, email use provides raw data in a digital format, which eliminates transcription requirements.
2.5.3 Collect and Analyze Round Responses
Analyzing data and reporting results are directly related to question type used in the process. Therefore, appropriate analysis techniques must be used premised on question type and data collected. Schmidt (1997) identifies three distinct phases in data collection, which include issue discovery, issue importance, and issue ranking.
• Issue discovery: the panelists should be initially requested to provide at least six important issues in response to the research question. While researchers may limit the number of items an expert can contribute (Couger, 1988), submitting a minimum number enhances the potential for differing respondents to raise the same issue albeit in different terms. The responses are concatenated into a single list, with those items describing the same issue consolidated in a single term. The panel verifies that these have been mapped correctly and the panelists’ ideas fairly represented. Should the research team discover major differences, however, this step may need to be repeated.
• Issue importance: the issues must now be meaningfully ranked. A randomly ordered, consolidated listing from the initial phase is provided to each expert panelist. Each panelist is required to select at least 10 percent of the issues as most important. The research team subsequently eliminates all issues not selected by a respondent majority. Note that, even though this phase aims to reduce the number of options for consideration in the next phase, it may not result in a “reasonable” reduction in the number of list items. In this eventuality, the research team may be challenged to identify alternative actions for trimming the list to a manageable size.
• Issue ranking: the pared list of the top 10 percent of issues is arranged randomly and the expert panelists requested to rank them. Data collection is terminated if broad consensus is achieved as a result of the ranking process. If consensus is not achieved, however, further rounds may be necessary to obtain higher agreement levels from the panel.
Following collection, the data is analyzed. The most appropriate analysis technique depends on the data form collected; it is therefore important to consider that data analysis may involve both qualitative and quantitative data. Hasson et al. (2000) suggest the major statistical techniques used in Delphi method studies are measures of central tendency and level of dispersion (i.e., standard deviation and inter-quartile range) because these most effectively present information regarding the respondents’ collective judgment. Eckman (1983) suggests median use based on a Likert-type scale. Mullen (2003) proposes Bayesian weighting to combine responses from using the Delphi method. For ranking data, statistical analysis may be appropriate; Schmidt (1997) summarizes nonparametric statistical techniques to be used in detail, in which he includes Kendall’s coefficient of concordance (Kendall’s W) to measure consensus level. He suggests that, if Kendall’s W is small or large, dissensus/consensus is easily identifiable and, therefore, the decision whether to proceed easy to make. However, if a moderate Kendall W is achieved, Nelms and Porter (1985) identify a trade-off between feasibility (i.e., panel indulgence, researcher resources, and time required) and the potential gain achieved as a result of further rounds. Thus, with moderate consensus levels, the research team must carefully consider the most appropriate next steps. Kiel et al. (2002) advise that a moderate level of consensus is achieved with a Kendall’s W of 0.500 and strong consensus at 0.700. Schmidt (1997) interprets differing levels of Kendall’s W (see Table 3).
Table 3. Interpretation of Kendall’s W (Schmidt, 1997)
W Interpretation Confidence ranks
0.1 Very weak agreement None
0.3 Weak agreement Low
0.5 Moderate agreement Fair
z0.7 Strong agreement High
0.9 Unusually strong agreement Very high
In contrast, Dietz (1987) suggests that the traditional assumptions made about statistical method use in Delphi method study analysis may be inappropriate. He highlights specific concerns in relation to positive correlation between a panelists’ uncertainty across rounds, increases in accuracy as a result of multiple rounds, forecasts weighted by self-reported confidence becoming more accurate versus unweighted forecasts, and the use of robust estimates of statistical location as summaries of expert opinion yielding better forecasts than non-robust measures. As a result of these concerns, researchers must carefully consider which data analysis method is most appropriate4.
While striving for consensus is often a study’s intention, researchers must also consider the scenario whereby agreement is achieved too quickly. This situation can suggest that:
• The subject choice is inappropriate (i.e., the topic is mature, well understood, and not subject to divergent views),
• The research question is presented in such a way as to “lead” the expert panel to only one possible response, and
• The panel is unwilling or unable to contemplate alternative scenarios.
As a result, the research team must judge whether unintended bias has entered the study. In this eventuality, re-structuring the research question to address bias-type may become necessary. An alternative method is to select a panel member to become “devil’s advocate”. This approach entails a “friendly” expert being selected to provide divergent opinions in such a way that the other panelists are forced to justify, and potentially challenge, their original judgments in greater detail.
2.6 The Utilization Stage
The final stage of the Delphi method reports the result back to the panelists and prepares the findings for publication.
2.6.1 Report Results
It is imperative to provide feedback to the panelists at the study’s conclusion. Not only have they given up their time and expertise to facilitate the study, but they will also have an interest in the results. When reporting the Delphi method results, the research team must include the following details (adapted from Schmidt, 1997):
• The total number of issues under consideration in all rounds • The strength of support for each issue
• The duration of the study
• Approaches to bias management • Panel selection procedure • Consensus approach
• The level of confidence in rankings obtained
4
• The round-by-round levels of consensus and other measures associated to the research hypothesis
• Sufficient raw data to support any statistical calculations • The response rate for the initial call, and
• The number of panelists for each round.
The panelists and research team should also agree on whether and how to publicly acknowledge individual panel participation in subsequent publications. Providing details of the panel in following papers may provide the field with heightened confidence in the research findings, particularly if analysis is proportionally qualitative in nature.
In Section 3, we examine how IS researchers have applied the Delphi method.
3 A Review of Information Systems Delphi Method Use
In this section, we discuss the nature and quality of IS papers that have used the Delphi method based on this paper’s guidelines for undertaking Delphi research. We present results and lessons learned from IS Delphi research studies: specifically, we analyze three Delphi IS papers in detail to highlight diverse method use premised on significantly different method characteristics.
3.1 Research Themes
To facilitate understanding of the nature and attributes of IS Delphi research, we surveyed IS journals and conference proceedings for the period January 1991 to December 2014. We identified 61 prominent IS research papers that apply the Delphi method (Appendix B). The prevalent theme in these IS Delphi studies is issue identification (i.e., recognizing the key strategic IS issues either at a country level, the organizational executive level, or in specific technological contexts). Insofar as IS issues may occur unexpectedly, the Delphi method is well suited for structuring feedback from experts who are best placed to evaluate these based on previous experience. Selected examples of other IS Delphi study topics include the critical elements of IS infrastructure flexibility (Duncan, 1995), development of a taxonomy of knowledge creation mechanisms (Nambisan, Agarwal, & Tanniru, 1999), scope and requirements of a knowledge management systems (Nevo & Chan, 2007), risk identification (Schmidt et al., 2001; Keil et al., 2002), identification of factors necessary for a successful ITIL implementation (Iden & Langeland, 2010), and top ten remedies for runaway IT projects (Iacovou & Dexter, 2004).
Several papers use the Delphi method in support of a larger study (i.e., Delphi use is only part of investigating greater issue) (e.g., Nambisan et al, 1999; Wynekoop & Walz, 2000; Mulligan, 2002; Lin & Chang, 2008; De Haes & Van Gremberg, 2009). Interestingly, even though Delphi method use is aligned to supporting quantitative and/or qualitative investigations in these mixed-method papers, the majority of the authors do not discuss the mixed-method nature or guidelines of their research when analyzing their data. De Haes and Van Grembergen (2009) are an exception: they adopt a research strategy that “triangulates between multiple different research methods: literature research, pilot case research, Delphi method research, benchmark research, and extreme case research. This triangulation enables us to obtain a richer insight in reality” (De Haes & Van Grembergen, 2009, p. 125). Those authors who fail to actively evaluate their studies’ mixed-method nature are limited in their ability to either acquire this level of insight or make further inferences as a result of their investigations:
data analysis in mixed methods research should be done rigorously following the standards that are generally acceptable in quantitative and qualitative research ... the quality of inferences from qualitative and quantitative studies contributes greatly to the process of developing high quality meta-inferences. (Venkatesh, Brown, & Bala, 2013, p. 18)
3.2 Research Objectives
Generally, the papers we reviewed sufficiently articulate their study objectives. Most of the studies are exploratory, with the majority being follow-up research to previously identified challenges. All of the studies conclude with a list of recommendations as a result of the expert feedback. Note that, in four of the studies, the term “Delphi” is not used. However, given the process followed and experts used to generate feedback, we identified these papers as Delphi method studies in all but name.
3.3 Data Collection
As with any investigation, methodological rigor is a cornerstone of “good” research (Skulmoski et al., 2007). Creswell (1994) identifies rigor as critical to quantitative studies, with Sadleowski (1986) doing likewise for qualitative research. Further, Sadleowski (1986) identifies that rigor levels are enhanced as a result of the researcher leaving an audit trail, defined as a clear decision trail of all key theoretical, methodological, and analytical decisions made in the research from beginning to end (Koch, 1994). As part of the data collection process, all of the IS papers we examined adopt the methodology’s anonymity characteristic, with several papers discussing its role in their studies. Dexter, Janson, Kiudorf, and Laast-Lass (1993) argue the method prevents dominant individuals from unduly influencing the results that may arise in face-to-face group meetings. Similarly, Akkermans et al. (2003) recognize anonymous panelist feedback that averts groupthink bias. Gonzalez, Gasco and Llopis (2010, p. 245) suggest that:
anonymity allows the participants to exchange ideas or preferences with no fear to show a conflicting opinion and without any pressures to reach a consensus ... they do not have to worry about the consequences of their answers and are never under the influence of the most dominant personalities.
However, because judgment bias is a recognized limitation of the Delphi method (Hung et al., 2008 – Table 1), and the minimization of judgment bias an important aspect of any rigorous study (Hallowell & Gambatese, 2010), by exclusively focusing on anonymity to negate bias, the authors are able to neutralize only the dominance bias type. Similarly, while some authors advise researchers to randomly order questions in the study phase to address other bias concerns (Pervan, 1994; Brancheau et al., 1996; Dekleva & Zupancic, 1996; Pollard & Hayne, 1996; Schmidt et al., 2001), this course of action addresses only primacy and contract effect biases (Table 2). Thus, with no other biases identified or addressed, findings may be subject to judgment biases that limit the papers’ validity and reliability.
3.4 Results and Lessons Learned
One of the most prominent examples of the IS field exploiting the Delphi technique has been the Society of Information Management ‘s (SIM) annual use of the Delphi (and survey) method to identify key issues in IT systems management since 1980. In a recent SIM analysis, Kappelman, Mclean, Luftman, and Johnson (2013) leverage the Delphi method to increase response rates versus previous iterations: “in the interests of increasing the response rate, additional effort was invested in questionnaire design…with questions modified or added based on previous results and suggestions from the Delphi group” (p. 239). Notwithstanding the subsequent survey achieving the highest response rate ever seen in this study (21.7% or 1002/4612 responses), using the Delphi method facilitates the study’s objective because it provides experts with opportunity to influence research design and questionnaire content. Survey outcomes subsequently establish that the key issues in IT systems management include the need to align IT with the business, security / privacy, business agility / flexibility, business productivity, and IT time-to-market / speed of IT delivery.
A consequence of the SIM study’s consistently achieving desirable outcomes is other IS studies adopting an equivalent methodological framework to accomplish similar outcomes but in different contexts. For instance, the Delphi method is used to identify and rank key issues and concerns facing IS and IT management in country settings that include Taiwan, Australia, and Hong Kong (Pervan, 1994; Wang, 1994; Moores, 1996): “the issue list of this study was based on those checked in previous ones. Specifically, an initial list of 28 issues was adopted from Niederman et al. (1991)” (Wang, 1994, p. 342). Interestingly, as a result of leveraging the SIM study method framework as their methodological template of choice, each of these three studies independently determines that improving IS strategic planning and leveraging IS data as a corporate resource are important issues to be addressed in their respective countries.
The method has also been adopted in studies seeking to identify and rank critical elements and decision variables in organizational settings. Doke and Swanson (1995) use the method to investigate the most important decision variables for selecting prototyping in IS development. They justify using the Delphi method premised on the need to achieve consensus on the relative importance of variables while concurrently seeking “to solicit and aggregate information from a group of disjointed individuals “ (p. 174). The Delphi method’s framework enables this approach because it allows research questions to be communicated asynchronously to a geographically spread panel via a central research team.
Consequently, feedback from managers in 31 firms suggest that the most important variables for selecting prototyping in IS development include clarity of project goals, developer understanding of user requirements, user task comprehension, user contribution, and user availability. Similarly, Duncan (1995) adopts the Delphi method to rank critical issues: she explores the most important items in how practitioners view IS infrastructure flexibility via:
a simple Delphi procedure designed to ferret out the issues perceived as critical from the practitioner’s point of view… the Delphi study was intended to foster and to focus discussion on those characteristics of IS resources most relevant to infrastructure flexibility (p. 45).
In recognizing that the Delphi method is used to generate a list of issues as the basis for subsequent interviews, Delphi method use is justified premised on the study’s exploratory nature that requires a framework to foster input versus one that provides a rigorous levels of sampling. With the method’s anonymous feedback and iterative consensus building advancing this aim, the authors are able to determine the most important items in how practitioners view IS infrastructure flexibility. These include the existence of compatibility rules for communication / networks, data, applications, business management leadership in long-term planning for applications, connectivity of systems across physical locations, and interface standardization. Keil, Lee, and Deng (2013) also adopt the method and identify the most critical IT project manager skills and their relative importance in a rigorous manner (p. 398). As the Delphi technique enables one to identify and rank skills in a single method structure, their use of the method is warranted due to it reducing methodological overheads. Consequently, the authors are able to determine that the top five (of nineteen) most critical skills include leadership, verbal communication, scope management, listening, and project planning.
The Delphi method has also been used in IS framework development. Nambisan et al. (1999) develop a taxonomy of knowledge creation mechanisms using the Delphi method because it is “deemed appropriate when judgmental information is indispensable” (Rowe, Wright, & McColl, 1991, p. 374). In noting that the Delphi method’s goal is to use expert opinion to classify mechanisms, the authors conclude that three knowledge types (context free IT knowledge, industry-specific IT knowledge, and firm-specific IT knowledge) interact with two forms of knowledge creation activity (knowledge acquisition and knowledge conversion). Similarly, Holsapple and Joshi (2002) develop a conceptual framework to identify and characterize a generic set of knowledge manipulation (KM) activities. They justify their use of the Delphi method in part due to it providing the authors with independent expert assessment of framework development in the research process. Perhaps more significantly, however, their use of the Delphi method is warranted due to it being part of a framework that serves “as a common language for discourse about knowledge manipulation. For researchers, it suggests issues that deserve investigation and concepts that must be considered in explorations of KM episodes” (p. 477). Ultimately, the study determines a generic set of elemental knowledge manipulation activities that include acquiring, selecting, internalizing, and using knowledge in KM episodes. Sharma, Ng, Dharmawirya, and Lee (2008) likewise use the method to derive a conceptual framework for analyzing knowledge societies. Their innovative work investigates knowledge assets developed in digital communities during economic or leisure activities. They incorporate the Delphi method into an active research approach that provides a “a qualitative and anecdotal validation of our model… we claim face, content and construct validity” (p. 153). While the legitimizing aspect of validation is rarely discussed in IS Delphi method studies, aligning the Delphi method with active research provides novel research opportunities because both methods are abductive in nature. The authors determine that, even though creating a knowledge society encompasses dimensions related to infrastructure, governance, talent and culture, the key elements for sustaining the society are intangible assets that include governance and culture.
Notwithstanding these papers providing an opportunity to appreciate the method’s outcome diversity, it is essential that lessons be learned from previous Delphi method use to assist future IS Delphi method studies. Insofar as we have acquired considerable insight into the relative research successes of previous IS Delphi method papers, those studies we deemed most successful “go the extra mile” in providing method operationalization details. While acknowledging that some papers may fail to do so as a result of the editorial review process, authors using the Delphi in a standalone context are obliged nonetheless to operationalize the Delphi technique in as much detail as possible to allow readers to fully appreciate levels of methodological rigor. As an example of a paper succeeding in this approach, Daniel and White (2005) adopt a framework that clearly articulates not only the intention of the study, but also the identification of experts, the nature of the instrument used, the number of rounds, participation and non-response rates,
and analysis of responses. Additionally, authors must help readers compare method operationalization with subsequent data generation / use because it provides an overall “snapshot” of the method’s efficacy in achieving the study’s research aims. De Haes and Van Grembergen’s (2009) exploratory study into IT governance implementations and its impact on business / IT alignment provides an effective example of this relationship as Figure 2 shows.
Figure 2. Study Research Process (De Haes & Van Grembergen, 2009)
Note that the majority of the papers we examined incorporate the Delphi method as part of a multi-method approach. In this context, the Delphi method is typically used to leverage subject matter experts as part of a questionnaire derivation process prior to a survey being sent to a sample population. Despite word count restrictions that can potentially limit the same level of method operationalization detail as seen in the standalone context, the complexity of the multi-method approach nevertheless compels authors to show how the Delphi method “sits” in the research study. This requires the author(s) to keep a clear focus on the research aims while ensuring that the reader is able to follow a logical “route” throughout the study. As an example of this aim being achieved, Wynekoop and Walz (2000, p. 189) provide a simple yet informative research design that shows the data collection structure (Figure 3) supported by an explanation of how the methods interact and on what premise (see also Figure 3):
The data collection for the planned research will involve two phases: a three-round Delphi study... of IT managers, following by a field study using different participants… the results of the planned Delphi study of IT managers will serve as input for a subsequent field study.
Figure 3: Study Research Design (Wynekoop & Walz, 2000)
In summary, while it is unfortunate that the lack of methodological clarity offered by some IS Delphi method papers obscures potential knowledge-generation opportunities, papers that supply detailed methodological information allow rigor levels to be determined and thereby enhance the field’s academic reputation.
3.5 Three Delphi Research Studies
By explicitly contrasting a selection of IS papers, the authors further illuminate Delphi method use and thereby confirm the method’s adaptability and diversity. Accordingly, we highlight three papers that adopt the Delphi method but whose characteristics vary significantly. The papers include a review of IT project post mortems (Kasi et al., 2008), a comparison of IT project risks (Liu, Zhang, Keil, & Chen 2010), and a study of inter-organizational IT system linkages (Daniel & White, 2005). The first paper (Kasi et al., 2008) explores an issue identified previously but not investigated; that is, it evaluates how issue choice defines question structure. While the second study (Liu et al, 2010) also investigates an issue, it uses multiple expert panels and an extra concluding phase that uses only a portion of the original expert panel. In contrast, the final paper (Daniel & White, 2005) does not seek issue resolution; instead, it forecasts the future direction of system technology use and, therefore, illustrates the impact of prediction versus issue resolution on method design. Table 4 summarizes these contrasting approaches that are supported by differing numbers of rounds and panel and samples sizes.
Table 4. Example of Full-Width Format for Tables (Source)
Kasi et al. (2008) Liu et al. (2010) Daniel & White (2005)
Study’s purpose Identify barriers of
conducting post mortem analysis for IT projects
Identify and compare IT project risk perceptions of IT project managers in Asian culture
The future of inter-organizational system linkages
Study’s duration Not advised Six months Not advised
Number of rounds 4 4 3
Number of panels 1 2 1
Sample size per round 23, 23, 23, 23 34/30, 34/30, 34/30, 6/4 17, 15, 11
Delphi design type5 e-Delphi Classical, e-Delphi Modified, e-Delphi
Questionnaire design Exploratory Exploratory Exploratory
Panel selection procedure Not advised Not advised Convenience sample,
snowball.
Consensus approach Kendall’s coefficient of
concordance
Kendall’s coefficient of concordance
Frequency of Agreement
Bias management Not advised Not advised Anonymity
Evaluating divergent method use across these three studies confirms Delphi method design and use diversity. And despite these differences, it is clear that apparently dissimilar Delphi studies can be broadly comparable. For example, all three studies are exploratory in nature even though their research question structures differ significantly. We can see a further likeness in the abductive approach that all three studies take in arriving at a final list of factors relevant to each structure. In contrast to other qualitative methods (such as field research) where researchers mainly rely on theory or a priori reasoning to arrive at
5
outcomes deductively (Benbasat et al., 1987), these researchers use the Delphi method in an abductive approach to create theory, with theoretical development derived from the discussions and results of experts with direct experience of a focal topic (Worrell et al., 2013). For a detailed review of these three papers, please see Appendix C.
4 Concluding Comments
In this paper, we explore the nature and use of the Delphi research method, explore how and why it may be applied in IS investigations, survey its use in IS research since 1991, and offer suggestions for improvement. Premised on a guideline for methodology use that is supported by a review of the method’s characteristics and the decision-making process, we present 61 IS papers as having applied the Delphi method since the early 1990s. Consequently, we establish the Delphi method being used principally to explore issues and risks as a result of previous research and to forecast future technology application and best practices. This confirms the Delphi method’s effectiveness in addressing complex research issues where only partial understanding exists about a phenomenon or where expert participation is required to advance subject knowledge. Furthermore, the contrasting approaches available to using the Delphi method support its position as an adaptable and innovative study procedure. As a result, the technique successfully confronts the field’s ongoing concerns that, “despite the gains of qualitative research in the late 20th century, a methodological conservatism has crept upon social science over the last ten years.” (Tracy, 2010, p. 837-838)
This paper makes several important contributions:
1. A literature review shows that previous IS method guidelines and prior IS Delphi method application have mostly ignored the importance of pilot testing as part of the method’s exploratory stage. We recognize the importance of not ignoring this fundamental task as part of research question generation.
2. The importance of understanding and recognizing differing bias types in using the Delphi method. Identified as a method limitation, appreciating the potential severity of bias on IS Delphi study findings necessitates a greater use of counter-bias techniques to remove opportunity for subsequent validity and reliability concerns.
3. A greater appreciation of the reporting process. Too often are results presented with a focus on outcomes without effective consideration of the interim stages and approaches that have supported their creation. Providing greater detail of study structure and findings can only enhance subsequent reader interpretation.
This paper has two limitations. First, it does not consider the differences in method use that may arise as a result of differing philosophical foundations in the IS field. Insofar as Linstone and Turoff (1975) review the main philosophical perspective underlying the Delphi method’s use, a more current philosophical review is required specifically for the IS field. Therefore, future papers should review different approaches to using the IS Delphi premised on contrasting philosophical foundations. Second, this paper focuses on presenting a best-practice approach to Delphi method use premised on the currently accepted structures inherent to the Delphi method. Accordingly, it does not discuss requiring researchers to change any essential aspect of the existing process and thereby accepts the methodology’s existing central characteristics. However, changes may be needed to fundamental Delphi method characteristics because of IS’s distinct nature that include rapid technological change and innovative data collection methods and analysis. Therefore, future IS Delphi method papers should consider not only the impact of the Delphi method on the IS field, but also how the IS field, in turn, influences the Delphi method.
In summary, when considering using the Delphi method, the IS researcher’s fundamental consideration must always be whether using the Delphi method is appropriate. If the problem under review is not articulated clearly or if the initial research question is not structured to facilitate iterative rounds of analysis, then using the Delphi method may fail to provide the research team with the outcomes they desire. Moreover, both expert panelist and research team alike must be mindful of the considerable commitment necessary for successful Delphi research. Consequently, as Worrell et al. (2013) succinctly identify, the challenge becomes to align phenomenon, research, question and method effectively so that the researcher and study make a contribution to the broader literature.
References
Adler, M., & Ziglio, E. (1996). Gazing into the oracle: The Delphi Method and its application to social policy and public health. London: Jessica Kingsley Publishers.
Akkermans, H. A., Bogerd, P., Yucesan, E., & van Wassenhove, L. N. (2003). The impact of ERP on supply chain management: Exploratory findings from a European Delphi study. European Journal of Operational Research, 146(2), 284-301.
Ausadamongkol, K., & Loveridge, D. (2001). Shaping Thailand’s IT future through technology foresight. Foresight, 3(5), 467-473.
Bacon, D., & Fitzgerald, B. (2001). A systemic framework for the field of information systems. Database for Advances in Information Systems, 32(2), 46-67.
Baldwin, A., & Trinkle, B. (2011). The impact of XBRL: A Delphi investigation. International Journal of Digital Accounting Research, 11, 1-24.
Baldwin-Morgan, A. (1993). The impact of expert system audit tools on auditing firms in the year 2001: A Delphi investigation. Journal of Information Systems, 7, 16-34.
Baron, J. (2003). Myside bias in thinking about abortion. Philadelphia: University of Pennsylvania Press. Becker, J., Beverungen, D., Knackstedt, R., Matzner, M., Muller, O., & Poeppelbuss, J. (2013). Designing
interaction routines in service networks: A modularity and social construction-based approach. Scandinavian Journal of Information Systems, 25(1), 1-32.
Bell, W. (1997). Foundations of future studies: Human science for a new era: Volume 1: History, purposes, and knowledge. New Brunswick, NJ: Transaction Publishers.
Benbasat, I., Goldstein, D. K., & Mead, M. (1987). The case research strategy in studies of information systems. MIS Quarterly, 11(3), 369-386.
Bjarnason, T., & Jonsson, S. H. (2005). Contrast effects in perceived risk of substance use. Substance Use & Misuse, 40(1), 1733-1748.
Boje, D. M., & Murnighan, J. K. (1982). Group confidence pressures decisions. Management Science, 28(1), 1187-1196.
Bradley, L., & Stewart, K. (2003). A Delphi study of Internet banking. Marketing Intelligence and Planning, 21(5), 272-281.
Brancheau, J. C., & Wetherbe, J. C. (1987). Key issues in information systems management: A Delphi study of IS executives and corporate general managers. MIS Quarterly, 11(1), 23-45.
Brancheau, J. C., Janz, B., & Wetherbe, J. C. (1996). Key issues in information systems management: 1994-95 SIM Delphi results. MIS Quarterly, 20(2), 225-242.
Brockhoff, K. (1975). The performance of forecasting groups in computer dialogue and face-to-face discussion. In H. A. Linstone & M. Turoff (Eds.), The Delphi method: Techniques and applications (pp. 292-321). Reading, MA: Addison-Wesley.
Brungs, A., & Jamieson, R. (2005). Identification of legal issues for computer forensics. Information Systems Management, 22(2), 57-66.
Cegielski, C. G., Bourrie, D. M., & Hazen, B. T. (2013). Evaluating adoption of emerging IT for corporate IT strategy: Developing a model using a qualitative method. Information Systems Management, 30(3), 235-249.
Chang, S.-I., Gable, G., Smythe, E., & Timbrell, G. (2000). A Delphi examination of public sector ERP implementation issues. In Proceedings 21st International Conference on Information Systems, 494-500.
Chou, C.-H, Zahedani, M. F., & Zhao, H. (2014). Ontology-based evaluation of natural disaster management websites: A multistakeholder perspective. MIS Quarterly, 38(4), 997-1016.
Conboy, K., Fitzgerald, G., & Mathiassen, L. (2012). Qualitative methods research in information systems: Motivations, themes, and contributions. European Journal of Information Systems, 21(2), 113-118. Couger, J. (1988). Key resource issues in IS in the 1990s: View of IS executive versus human resource
executives. Information & Management, 1(4), 161-174.
Cramer, R. (1991). The education of gifted children in the United States: A Delphi study. Gifted Child Quarterly, 35, 84-91
Creswell, J. (1994). Research design: Qualitative & quantitative approaches. Thousand Oaks, USA: Sage. Cuhls, K., Blind, K., & Grupp, H. (1998). Studie Zur globalen Entwicklung von Wissenschaft und Technik. Methoden-und Datenband. Fraunhofer Institut Systemtechnik und Innovationsforschung, Karlsruhe. Custer, R., Scarcella, J., & Stewart, B. (1999). The modified Delphi technique—a rotational modification.
Journal of Vocational and Technical Education, 15(2), 1-10.
Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi-method to the use of experts. Management Science, 9(3), 458-467.
Daniel, E., & White, A. (2005). The future of inter-organisational system linkages: Findings of an international Delphi study. European Journal of Information Systems, 14, 188-203.
Day, J., & Bobeva, M. (2003). Seeking the truth: The use of Delphi studies for IS research. In K. Grant, D. A. Edgar, & M. Jordan (Eds.), Reflection on the past, making sense of today and predicting the future of information systems. 9th Annual UKAIS Conference Proceedings, Glasgow Caledonian University, Glasgow, UK.
Day, J., & Bobeva, M. (2005). A generic toolkit for the successful management of Delphi Studies. The Electronic Journal of Business Research Methods, 3(2), 103-116.
de Haan, J., & Peters, R. (1993). Technology: Toys or tools? Results of a Dutch Delphi study. Information and Management, 25, 283-289.
De Haes, S., & Van Grembergen, W. (2009). An exploratory study into IT governance implementations and its impact on business/IT alignment. Information System Management, 26, 123-137.
Deip, P., Thesen, A., Motiwalla, J., & Seshardi, N. (1977). Systems tools for project planning. Bloomington, Indiana: International Development Institute.
Dekleva, S., & Zupancic, J. (1996). Key issues in information systems management: A Delphi study in Slovenia. Information and Management, 31, 1-11.
Delbecq, A. Van de Ven, A., & Gustafson, D. (1975). Group techniques for program planning. Glenview, Illinois: Scott Foresman.
Dexter, A. S., Janson, M. A., Kiudorf, E., & Laast-Lass, J. (1993). Key information technology issues in Estonia. Journal of Strategic Information Systems, 12(2), 139-152.
Dietz, T. (1987). Methods for analyzing data from Delphi panels: Some evidence from a forecasting study. Technological Forecasting and Social Change, 31(1), 79-85.
Doke, E. R., & Swanson, N. E. (1995). Decision variables for selecting prototyping in information systems development: A Delphi Study of MIS managers. Information and Management, 29(4), 173-183. Duncan, N. B. (1995). Capturing flexibility of information technology infrastructure: A study of resource
characteristics and their measure. Journal of Management Information Systems, 12(2), 37-58. Durkheim, E. (1982). The rules of the sociological method. New York: Free Press.
Eckman, C. A. (1983). Development of an instrument to evaluate intercollegiate athletic coaches: A modified Delphi study (Unpublished doctoral dissertation). West Virginia University, Morgantown. Eggers, R. M., & Jones, C. M. (1998). Practical considerations for conducting Delphi studies: The oracle
