R E S E A R C H A R T I C L E
Open Access
A randomized controlled trial comparing
in-person and wiki-inspired nominal group
techniques for engaging stakeholders in
chronic kidney disease research
prioritization
Meghan J. Elliott
1,2*, Sharon E. Straus
1,3, Neesh Pannu
4, Sofia B. Ahmed
5, Andreas Laupacis
1,2,3,
George C. Chong
6, David R. Hillier
7, Kate T. Huffman
8, Andrew C. Lei
9, Berlene V. Villanueva
10, Donna M. Young
11,
Helen Tam-Tham
12, Maoliosa Donald
5, Erin Lillie
1, Braden J. Manns
5,12and Brenda R. Hemmelgarn
5,12Abstract
Background:Few studies have evaluated stakeholder engagement in chronic kidney disease (CKD) research prioritization. In this two-arm, parallel group randomized controlled trial, we sought to compare an in-person nominal group technique (NGT) approach with an online wiki-inspired alternative to determining the top 10 CKD research priorities, and to evaluate stakeholder engagement and satisfaction with each process.
Methods: Eligible participants included adults ≥18 years with access to a computer and Internet, high health literacy, and from one of the following stakeholder groups: patients with CKD not on dialysis, their caregivers, health care providers who care for patients with CKD, or CKD-related health policymakers. Fifty-six participants were randomized to a wiki-inspired modified NGT that occurred over 3 weeks vs. a 1-day in-person NGT workshop, informed by James Lind Alliance methodology, to determine the top 10 CKD-related research priorities. The primary outcome was the pairwise agreement between the two groups’final top 10 ranked priorities, evaluated using Spearman’s correlation coefficient. Secondary outcomes included participant engagement and satisfaction and wiki tool usability.
Results:Spearman’s rho for correlation between the two lists was 0.139 (95 % confidence interval−0.543 to 0.703,
p= 0.71), suggesting low correlation between the top 10 lists across the two groups. Both groups ranked the same item as the top research priority, with 5 of the top 10 priorities ranked by the wiki group within the top 10 for the in-person group. In comparison to the in-person group, participants from the wiki group were less likely to report: satisfaction with the format (73.7 vs.100 %,p= 0.011); ability to express their views (57.9 vs 96.0 %,p= 0.0003); and perception that they contributed meaningfully to the process (68.4 vs 84.0 %,p= 0.004).
(Continued on next page)
* Correspondence:[email protected] 1
Li Ka Shing Knowledge Institute, St. Michael’s Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
2Institute of Health Policy, Management and Evaluation, University of
Toronto, 155 College Street, Toronto, Ontario M5T 3M6, Canada Full list of author information is available at the end of the article
(Continued from previous page)
Conclusions:A CKD research prioritization approach using an online wiki-like tool identified low correlation in rankings compared with an in-person approach, with less satisfaction and perceptions of active engagement. Modifications to the wiki-inspired tool are required before it can be considered a potential alternative to an in-person workshop for engaging patients in determining research priorities.
Trial registration:(ISRCTN18248625)
Keywords:Chronic kidney disease, Research priorities, Patient preferences, Wiki
Abbreviations:CI, Confidence interval; CKD, Chronic kidney disease; CONSORT, Consolidated Standards of Reporting Trials; eGFR, Estimated glomerular filtration rate; JLA, James Lind Alliance; NGT, Nominal group technique; PCORI, Patient-Centered Outcomes Research Institute; SPOR, Strategy for patient oriented research
Background
Chronic kidney disease (CKD), defined as estimated glomerular filtration rate (eGFR) <60 mL/min/1.73 m2, affects approximately 5–7 % of the adult population [1–3] and contributes to excess morbidity, mortality and health care costs [4, 5]. Despite the tremendous burden CKD places on health care delivery, care for persons affected by CKD remains suboptimal [6]. These care gaps may result from the lack of research that is relevant to end-users, in-cluding patients with CKD, their caregivers, and the clini-cians involved in their care [7, 8]. To address these gaps, there is increasing emphasis on the importance of en-gaging key stakeholders in determining research priorities and establishing user-relevant research objectives [9, 10]. The Strategy for Patient Oriented Research (SPOR) [11] in Canada and the Patient-Centered Outcomes Research Institute (PCORI) in the United States [12] are two initia-tives that aim to engage patients in research and knowledge translation and to focus on interventions and outcomes that are important to research end users.
Although few established methods for involving key stakeholders in research prioritization exist, the James Lind Alliance (JLA) priority setting partnership model is widely considered an inclusive and transparent process and has been applied to a number of disease conditions [13]. The final step of the four-step JLA partnership in-volves an in-person workshop with a nominal group technique (NGT) to use a list of the top 30 research pri-orities to develop a ranked list of the top 10 research priorities from key stakeholders through small and large group activities [14]. While considered the reference standard, this approach has some clear limitations for participants and coordinators, including time and re-source requirements, travel, and logistical consideration [15]. These challenges may be particularly problematic for patients with chronic illness who have difficulty with or are unable to travel, which could limit participant in-volvement and generalizability of findings. Alternative Internet-based approaches to research priority setting, such as an interactive wiki-like platform, could obviate
some of these limitations. A‘wiki’is a collaborative web-based platform that allows users to add and edit content online in real time, and in the context of research prioritization can be used to summarize the preferences of multiple stakeholders [16]. Wiki-inspired tools have been successfully applied in similar, multi-stakeholder consensus approaches, including the creation of an asthma action plan [16], but have not been studied in research prioritization exercises that require significant patient involvement.
In this randomized controlled trial we sought to com-pare the in-person workshop and NGT approach with an online wiki-inspired alternative as the final step in the JLA research prioritization process for key CKD stake-holders, including patients with CKD not on dialysis, their caregivers, and the health care professionals and policy-makers involved in their care. We aimed to compare the top 10 research priorities resulting from each process and to evaluate the engagement and satisfaction of partici-pating stakeholders with their assigned intervention.
Methods
This trial is reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement [17] (trial registration: www.isrctn.org, ISRCTN18248625).
Steps 1–3 took place from July 2014 to June 2015; this trial involved step 4 only.
Trial participants
The steering group enlisted the support of nephrology networks, partner organizations, and nephrologists from across Canada to identify potential participants. Eligible participants were≥18 years of age, English speaking, had access to and were comfortable using high-speed Internet, and had high health literacy (determined by asking the fol-lowing question: “How confident are you in filling out medical forms by yourself?” [18]). Participants were also required to be a member of a stakeholder group: patients with CKD not on dialysis or with a prior transplant; informal caregivers of persons with CKD (relatives, family members, or friends who help patients manage their illness); health care professionals (primary care physicians, nephrologists, nurses, pharmacists, social workers, or dietitians) who care for patients with CKD; or health policymakers (non-clinicians with the ability to influence or determine policies and practices related to health care delivery for CKD at an international, national, regional, or local level). The following indi-viduals were not eligible to participate: patients receiving dialysis and/or who had received a kidney transplant; per-sons with an underlying diagnosis of dementia or cogni-tive impairment; and patients deemed unfit to travel. A research assistant confirmed the eligibility of potential participants, described the trial to them in detail, and obtained their written informed consent.
Interventions
Participants were randomized 1:1 to an in-person NGT-based workshop or an online wiki-inspired platform. Randomization occurred by random number sequence generation using a central electronic system to ensure allocation concealment. Patients and their caregivers were randomized as a dyad to avoid contamination. Additionally, the 6 patient and caregiver steering group members were allocated to the in-person workshop group only, and not randomized, as they had originally committed to participate in the NGT process of the study and it was therefore deemed important to include them in the NGT arm. Participants in both groups were provided with the top 30 uncertainties shortlist (derived from a national survey and prioritized by the steering group) to review and consider prior to the intervention; participants in the in-person workshop group ranked the 30 uncertainties prior to the workshop, while par-ticipants in the wiki-inspired group ranked the top 10 (due to logistical issues of ranking 30 online).
Individuals in the 1-day in-person workshop group participated in a NGT approach [14, 19] in Toronto, Canada, which included a combination of small and
large group exercises facilitated by four individuals with experience in the JLA process. The final ranking of the top 10 research priorities was determined by large group consensus, with voting when needed.
Wiki group participants were provided with a 1-h training webinar on use of the wiki and were given unique usernames and passwords to access the secure website before they began the following tasks to deter-mine the top 10 research priorities. The wiki-based intervention took place online over 3 phases: 1) an individual ranking phase (7 days), where participants submitted their preferences for the top 10 research priorities individually, 2) a group ranking phase (12 days), where participants determined the final top 10 research priorities collaboratively, and 3) a sign-off phase (2 days), where participants could review to confirm their agree-ment, but not re-order the final list. Following the individual phase, individual rankings were aggregated across participants using reverse scoring, such that items ranked first received a score of 10, and those ranked last received a score of 1. These scores were aggregated and presented as a summary rank list to which participants could refer throughout the collabo-rative ranking phase. During the collabocollabo-rative ranking phase, participants could re-order the top 10 priorities in real time and discuss their selections using a chat feature. The chat feature consisted of a text box and display of all participant comments in chronological order, organized by unique user identifier (ie identifying information and stakeholder role were not apparent during chat to avoid differential perception of input by provider role). A facilitator monitored the site daily and stimulated discussion through comments using the chat feature. Technical support was available throughout the process. Due to the nature of the interventions, study participants and intervention administrators were not blinded to group assignment; however, statistical ana-lysis was blinded. The in-person workshop took place during the wiki process; participants in both groups were blinded to the ranking results of the other group.
Outcomes
platform (FluidSurveys platform, Fluidware Inc., Ottawa, Ontario)., Key elements of stakeholder engagement were evaluated in the participant engagement section of the questionnaire, including trust, respect, accountability, legitimacy, fairness, and competence [20, 21].
Sample size
The primary outcome was the pair wise agreement between the two groups’top 10 priorities, therefore the correlation analysis was driven by the number of research priorities considered, which was fixed at 10 for the purposes of this study. As the JLA recommends 20 to 30 participants in the in-person priority setting exercise, with approximately equal representation of patients, clinicians, and caregivers, we estimated that a total of 40–60 participants would be required (we aimed for 50). Inflating this by 10 % to ac-count for potential dropouts, an overall; sample size of 56 was felt to be a conservative estimate for the primary outcome.
Statistical analyses
Baseline demographic information was compared between the in-person workshop and wiki groups using univariate analyses. The unit of analysis for the primary outcome was at the group level, and the pairwise agreement between the two groups’ priorities (provided as ranks) was examined using Spearman’s correlation coefficient and its associated 95 % confidence interval (CI). We also compared the rank order of the top 10 research priorities from each group descriptively, noting the pro-portion that were within the top 10 across both groups. Our secondary outcomes were obtained by survey for both the in-person and wiki groups, with responses mea-sured using a 5-point Likert scale (where 1 = strongly disagree, 5 = strongly agree), and the proportions of participants who agreed with each statement (4 or 5 on the scale) were compared between groups using chi-square tests. Written comments also obtained from the survey by both groups were examined qualitatively. Two researchers (MJE and BRH) reviewed all written com-ments and independently coded them for emerging categories and themes. Resulting themes and discrepan-cies were discussed, and a final coding scheme describing participants’ experiences with each process was estab-lished. Participant observation data collected during the in-person workshop was summarized quantitatively (ie frequencies) and qualitatively (ie brief researcher notes). The time requirements to undertake each process were compared descriptively.
Results Participants
Fifty-six individuals were randomized to either the in-person workshop (n= 28) or online wiki-inspired group
(n= 28). In addition the 6 patient and caregiver steering group members were allocated to the in-person work-shop group only, as established a priori, for a total of 34 participants for the in-person workshop. Overall 25 indi-viduals took part in the 1-day workshop (12 patients, 6 caregivers, 6 health care professionals, and 1 policy-maker), and 26 individuals participated in the 3-week online wiki-like process (12 patients, 8 caregivers, 5 health care professionals, and 1 policymaker). Participant enrolment and reasons for withdrawal are outlined in Fig. 1. During the 3-week period of wiki activity, 21 of 26 participants (80.8 %) logged onto the ranking page at least once during either the individual or group phase (3 patients and 2 caregivers did not access the site during the intervention). All participants contributed to the pri-mary analysis of agreement between the two top 10 re-search priority lists, as this was assessed at the group level. Only participants who completed the post-intervention survey were included in the analysis of participant engage-ment and satisfaction.
The majority of study participants were 40–64 years of age, and 47 % were male (Table 1). 12 patients and 6 caregivers participated in the in-person workshop, and 11 patients and 8 caregivers participated in the online wiki-inspired process. Compared to the in-person group, a smaller proportion of patients and caregivers in the wiki-inspired group were < 40 years of age and female (5.6 vs. 21.1 % and 38.9 vs 52.6 % respectively), and they were less likely to work full time (31.6 vs. 44.4 %) or hold a university degree (26.3 vs. 61.1 %).
Comparison of the top 10 CKD research priorities
measures to slow CKD progression, CKD symptoms, and the impact of lifestyle factors and medications on kidney disease. Two wiki group members (both clinicians) had participated previously in an in-person priority setting workshop for patients on or nearing dialysis [22], which may have influenced their perceptions and experiences of the intervention.
Participant satisfaction and engagement
All 25 participants in the NGT-based workshop group and 19 of 26 participants from the online wiki-inspired group completed the post-intervention questionnaire. The non-responders from the wiki group included 4 pa-tients, 2 caregivers and 1 health care professional. All workshop participants felt the in-person NGT format was well-suited (ie noted agreement or strong agree-ment) to determine CKD research uncertainties, whereas 73.7 % of respondents in the wiki group were satisfied (agree/strongly agree) with the format (p= 0.011) (Fig. 2a). Ninty-six percent of participants in the workshop group felt they could adequately express their views in this for-mat, as compared to 57.9 % in the wiki group (p= 0.003). With respect to participant engagement, workshop participants were significantly more likely than wiki
participants to report that the format used by the group respected the ideas of all participants (100 vs. 68.4 %, p= 0.004) and perceived that their views con-tributed to the final list of research priorities (84.0 vs. 33.3 %, p< 0.001) (Fig. 2b). Respondents from both groups felt they had adequate knowledge (80.0 % in workshop group vs. 83.3 % in wiki group, p= 1.0) and skills (92.0 % in workshop group vs. 89.5 % in wiki group, p= 1.0) to rank the top research priorities, and around one quarter of participants felt that influential participants impacted the group dynamics (28.0 % in workshop vs. 21.1 % in wiki groupp= 0.73).
Written comments were provided in the survey by 11 participants in the in-person NGT group and 10 partici-pants in the wiki-inspired group. The main themes that emerged were related to opportunities for personal inter-actions and meaningful contributions, participation rates, format effectiveness and the importance of com-munication to justify individual rankings (Table 3), with the workshop group generally providing more positive feedback of their experience across these three themes. Written comments from wiki participants highlighted their perceptions of a low level of active participation and inadequacy of the chat feature (which aimed to
allow comprehensive discussion of participants’ views, preferences and justification of changes to the their rankings). Wiki participants felt that the top ranked un-certainties were ordered and re-ordered without a clear understanding of the reasons for the changes and with limited group discussion. In contrast, in-person work-shop participants felt they were able to contribute equally and effectively among all stakeholders in this format.
Observation of participants during the in-person work-shop revealed a moderate to high degree of verbal and physical engagement for the majority of group mem-bers. Three participants (2 patients and 1 caregiver)
demonstrated limited verbal and physical interaction with others. Most individuals appeared to readily voice their views, questions and comments spontaneously or with prompting by the facilitator. While the initial small group discussions were reserved and initiated mainly by the facilitator, more spontaneous and animated discus-sions among the participants arose throughout the day. Some participants contributed more confidently and per-suasively than others, but these individuals were not limited to one stakeholder group and did not appear to negatively dominate any of the group discussions. In the wiki-inspired group, chat discussions were similarly reserved and required facilitator prompting initially.
Table 1Baseline demographics of participants in the in-person workshop and online wiki groups, n (%)
In-person workshop group (n= 25)
Patients and caregivers in in-person workshop (n= 18)
Online wiki-inspired group (n= 26)
Patients and caregivers in online wiki-inspired group (n= 19)
Age
18–39 years 3 (12) 1 (5.6) 4 (15.4) 4 (21.1)
40–64 years 14 (56) 9 (50) 16 (61.5) 9 (47.4)
≥65 years 8 (32) 8 (44.4) 6 (23.1) 6 (31.6)
Sex
Male 13 (52) 11 (61.1) 11 (42.3) 9 (47.4)
Female 12 (48) 7 (38.9) 15 (57.7) 10 (52.6)
Role
Patient 12 (48) 12 (66.7) 11 (42.3) 11 (57.9)
Caregiver 6 (24) 6 (33.3) 8 (30.8) 8 (42.1)
Health Care Provider 6 (24) - 6 (23.1)
-Policymaker 1 (4) - 1 (3.8)
-Marital Status
Married 22 (88) 16 (88.9) 19 (73.1) 13 (68.4)
Common Law 1 (4) 0 (0) 2 (7.7) 1 (5.3)
Divorced/Separated 2 (8) 2 (11.1) 0 (0) 0 (0)
Single 0 (0) 0 (0) 3 (11.5) 3 (15.8)
Not Indicated 0 (0) 0 (0) 2 (7.7) 2 (10.5)
Employment Status
Full-time 14 (56) 8 (44.4) 12 (46.2) 6 (31.6)
Part-time or casual 2 (8) 1 (5.6) 2 (7.7) 1 (5.3)
Retired 9 (36) 9 (50) 10 (38.5) 10 (52.6)
Unemployed 0 (0) 0 (0) 1 (3.8) 1 (5.3)
Not Indicated 0 (0) 0 (0) 1 (3.8) 1 (5.3)
Highest Level of Education
Some High School 0 (0) 0 (0) 1 (3.8) 1 (5.3)
High School Graduate 4 (16) 4 (22.2) 3 (11.5) 3 (15.8)
College Diploma 4 (16) 3 (16.7) 8 (30.8) 8 (42.1)
University Degree 17 (68) 11 (61.1) 12 (46.2) 5 (26.3)
However, throughout the process, participants provided spontaneous input and sought each other’s views in a respectful manner.
Wiki usability
During the 3-week period of wiki activity, 21 of 26 par-ticipants (80.8 %) logged onto the ranking page at least
Table 2Top 10 CKD-related research priorities resulting from the in-person workshop vs. online wiki-based groups
In-person workshop group Wiki rank Online wiki-based group Workshop rank
1. What are the most effective new interventions and treatments to prevent the development and progression of kidney disease?
1 1. What are the most effective new
interventions and treatments to prevent the development and progression of kidney disease?
1
2. What is the best diet to slow progression of kidney disease and what are the benefits and risks of specific diets (ie phosphate restriction, protein restriction, low salt etc.) in terms of kidney disease progression and quality of life?
6 2. What are the harmful effects of
medications used in patients with CKD, and in particular, the combinations of medications used to treat other diseases (such as diabetes and high blood pressure)?
7
3. What are the causes of symptoms in patients with chronic kidney disease, including fatigue, low energy, sleeping problems, depression, anxiety and sexual dysfunction, and how can these best be treated to improve quality of life?a
8 3. What are the best signs or markers
(ie blood tests, urine tests or other tests) to identify and diagnose kidney disease early?
14
4. What are the optimal strategies, such as having access to health information (eg lab test results), sharing of information, and/or improving communication, to help patients manage their health condition(s) themselves and to improve patient experience and outcomes?
Not ranked in the top 10 4. What are the benefits and risks associated with use of vitamins, supplements and alternative/complementary therapies (ie herbal, naturopathic, marijuana etc.) in terms of kidney disease progression and quality of life?
11
5. What is the impact of lifestyle factors (ie exercise, stress) on risk of developing kidney disease, kidney disease progression, and quality of life?
9 5. How can we predict how fast kidney
function will get worse, and when kidneys will fail?
13
6. What are the optimal strategies for the management of CKD (ie those undertaken by the primary care physician, nephrologist, other health care professionals) to delay progression and improve outcomes?
Not ranked in the top 10 6. What is the best diet to slow progression of kidney disease and what are the benefits and risks of specific diets (ie phosphate restriction, protein restriction, low salt etc.) in terms of kidney disease progression and quality of life?
2
7. What are the harmful effects of medications used in patients with CKD, and in particular the combinations of medications used to treat other diseases (such as diabetes and high blood pressure)?
2 7. What are the optimal medications
(eg ACE inhibitors, ARBs, phosphate binders, sodium bicarbonate, etc.) to slow
progression of kidney disease?
19
8. What are the optimal approaches for the prevention and treatment of cardiovascular disease in patients with CKD?
Not ranked in the top 10 8. What are the causes of symptoms in patients with chronic kidney disease, including fatigue, low energy, sleeping problems, depression, anxiety and sexual dysfunction, and how can these best be treated to improve quality of life?a
3
9. What is the best strategy (eg screening, programs targeting high risk groups, programs to increase public awareness) to identify kidney disease early?
Not ranked in the top 10 9. What is the impact of lifestyle factors (ie exercise, stress) on risk of developing kidney disease, kidney disease progression, and quality of life?
5
10. How do we ensure that patients with CKD have equitable access to care (eg nephrologists, allied health clinics) irrespective of location of residence or socio-economic status?
Not ranked in the top 10 10. How can communication regarding patient care be improved and/or streamlined across all disciplines (primary care, nephrology, allied health) to improve outcomes and the patient experience?
12
CKDchronic kidney disease,ACEangiotensin-converting enzyme,ARBangiotensin II receptor blockers
a
once during either the individual or group phase. Those who did not log in during the intervention included 3 pa-tients and 2 caregivers. During the group ranking phase, 13 users (50.0 %) were considered active participants (ie they made direct changes to the top 10 uncertainties list [n= 11, 42.3 %] and/or participated in the chat feature [n= 13, 50.0 %]). These active participants included 6 pa-tients, 3 caregivers, and 4 health professionals. One patient who was admitted to hospital during the individual ranking phase accessed the ranking tool, submitted her individual ranking, and participated in the group ranking and discus-sion from a laptop computer while in hospital.
Over the 12-day group ranking phase, a total of 192 changes were made to the rankings (mean 16 changes/ day, range 1–37) and 104 comments were provided in the group chat (mean 8.7 comments/day, range 2–21). The most changes were made on the first day of the group round (37 changes). Twenty-six changes were made to the list on the final day, and the most group discussion (21 comments) occurred on the final day of the exer-cise. The most active participants included 2 patients and 1 health care provider, each of whom provided between 46 and 58 contributions to the rankings and/or group chat. Half of the wiki-inspired group participants viewed and confirmed they had reviewed the final top 10 list following conclusion of the group ranking phase.
Results from the wiki usability portion of the post-intervention questionnaire are outlined in Additional file 2.
Of the 18 respondents, all felt that 12 days were sufficient to undertake the group ranking process, and 61 % used the tool as often as they would have liked. Eighty-three percent of participants experienced no difficulties in changing the rankings, and 61 % found the chat feature useful for dis-cussing the uncertainties as a group. Six participants con-tacted study coordinators regarding technical issues, all of which related to difficulty logging onto the site. All issues but one were resolved, with this user being able to login after repeated attempts but having ongoing difficulties par-ticipating in the ranking process.
Time requirements
The in-person workshop intervention took place over 1 weekday and required travel time for participants from across Canada to attend. The online wiki-based group could access the ranking site at any time of day, from any location, and as often as participants preferred during a 3-week period.
Discussion
In this randomized trial evaluating two methods of stakeholder engagement to determine CKD research priorities, we found that an online wiki-inspired ap-proach had a low level of correlation compared to the research priorities determined from an in-person NGT approach informed by JLA methodology, with only 5 of the top 10 priorities identified by the wiki-inspired group
within the top 10 priorities of the in-person group. There were also important differences between the groups, with fewer health-services research questions ranked by the wiki-inspired group, and differences in perceptions of en-gagement. In comparison to participants in the in-person workshop, those in the wiki-inspired group were less satis-fied with the format of the exercise and felt they were less likely to contribute meaningfully to the process. Whereas all individuals who attended the in-person workshop con-tributed to the group discussions (either spontaneously or with prompting), 80 % of individuals in the wiki group accessed the tool at least once, and only half were con-sidered ‘active’ participants. We are unable to deter-mine if the differences in research priorities identified were due to the different processes, or differences in
the groups themselves (including the patient and caregiver steering group members participating in the in-person workshop), as priorities may vary by group irrespective of the process used to establish them. This will be important to address in future work.
Our study is one of the first to evaluate an online wiki-based tool to engage patients and other stakeholders in health care research prioritization and to undertake a ran-domized comparison of two distinct research prioritization methods. Although both techniques are based on a NGT approach, the online wiki-inspired option had attractive features including flexibility to contribute at a convenient time during a process that occurred over a longer period of time compared to the 1-day in-person workshop, which the majority of participants felt was adequate. However the
Table 3Main themes identified from written feedback on post-intervention questionnaire
Group Role Comment
Limited Personal Interactions
Wiki Caregiver “It was easier not to put your whole heart into this as it would have been if it were done face to face with the other participants…I felt something was missing, some sort of connection with others facing the same difficulties.”
Wiki Health Care Professional “The wiki seemed to reward the last person to change the rankings. In person, a doc such as me could have had a more subtle way to influence the process without being aggressive or trying to take it over.”
Importance of Communication to Justify Selections
Wiki Patient “When people explained their rationale for their preferences, I understood. When the ranking was changed without using the chat feature to augment our understanding, I could not discern the rationale for the preferences.”
Wiki Patient “Although I enjoyed the discussions that did occur and the mix of group members (caregiver, patient, clinician), I'm not sure, as a whole, we really made the most of the chat feature.” Wiki Health Care Professional “I think any changes in the top 10 should have had comments to support these changes.” Wiki Health Care Professional “In a discussion forum online, I also didn't feel comfortable asking specific people to clarify
or explain their choices.” Format Effectiveness
Workshop Caregiver “The format enabled patients and caregivers to engage with a wide group of professionals, other than the traditional doctor-patient relationship.”
Wiki Patient “Basically, I felt this process was less effective and that we didn't actually reach a group consensus with our outcomes.”
Workshop Patient “The only part that was not entirely satisfactory was the final phase in which the entire group worked to re-order priorities. I felt that it lacked the fluidity and finesse to allow for minor changes.”
Perspectives Dependent on Participation
Wiki Patient “I was disappointed by the participation rate (those who did not participate). I think this affected the quality of the discussions and possibly weakened the final outcome.”
Workshop Health Care Professional “Some opinions/voices might not be represented depending on who attends (is able to attend) and who is approached to attend the workshop.”
Ability to Contribute Meaningfully
Workshop Caregiver “As a layperson I felt that even though I did not have the hard earned scientific expertise of the medical community, all our experiences were considered on their merits.”
Workshop Patient “The workshop was an amazing experience where a collaborative effort of all affected people by CKD joined together to come up with the top 10 important and highly timely research uncertainties that will give better life with the patients.”
longer duration required for participation may be per-ceived as a disadvantage by some. The standard JLA methodology for stakeholder involvement in health care research prioritization, including a final in-person work-shop, has been implemented to date in more than 20 chronic diseases, including prior work by our group to identify research priorities for patients on or nearing dialy-sis [22]. A recent systematic review on research priority setting in kidney disease identified 16 studies, only 4 of which involved patients in the prioritization process [23]. The studies cited in this review used a variety of methods to elicit research priorities, none of which were described in detail nor used an electronic or online collaborative platform to engage stakeholders, and none of which were evaluated in a randomized trial such as this one.
Wiki-like tools have been applied in other collabora-tive health care-related activities involving patients and other stakeholders. Gupta et al. included patients and clinicians in a wiki-inspired collaborative design of an asthma action plan (WikiBuild), where groups of 14 par-ticipants contributed to the action plan’s content and de-sign over a 1-week period [16]. Other studies examining the application of wiki-based tools in patient-centered research, including the development of clinical practice guidelines [24] and patient information leaflets [25], have found these tools to be promising alternatives to more traditional approaches. In contrast to our study, these other studies generally involved input from a large num-bers of patients (>300) and lack the personalized and multi-stakeholder interactions endorsed by the JLA format. Potential advantages of an online wiki-inspired approach for collaborative research prioritization include flexibility in participation, lack of travel requirements limiting par-ticipant eligibility, and lower costs [16]. A particular ad-vantage is the ability to include those whose illnesses may preclude easy travel; for example, one patient in the wiki-inspired group in our study was admitted to hospital during the study period and participated in the process while in hospital. The complex group dynamics often seen with in-person techniques can lead to the development of status hierarchies that favour clinicians over patients and caregivers [19, 26, 27], which could limit the openness of individual contributions to the discussion and influence the final product [28], although our qualitative results would suggest that was not the case in our study. Although neither the wiki group nor the in-person work-shop group indicated the presence of overly influential individuals, wiki-inspired participants indicated the ability to contribute honestly and explicitly in this format.
Although 61 % of wiki-inspired participants responding to the post-intervention questionnaire felt that the chat feature was useful for explaining their views and prefe-rences, many expressed concerns that this format did not allow for as comprehensive a discussion and understanding
as is possible with face-to-face interactions. Gupta et al. de-scribed similar participant concerns for the wiki online chat, as well as the lack of individual accountability for changes made [16]. Other barriers to the adoption of wiki-inspired tools for patient engagement activities de-scribed include concerns regarding suitability of the for-mat to its objectives, poor participation rates, and technical challenges, among others [16, 24, 25]. It is unclear whether patients with poor health literacy or cognitive impairment would be able to participate in a wiki-inspired priority set-ting process, and this is an area for future work. The par-ticipant feedback provided during our trial highlights the need to refine the online collaborative tool, and the chat feature in particular, to ensure participants feel engaged and able to contribute meaningfully to the process.
Our study is one of the first to describe the application of a wiki-inspired tool to a research prioritization exer-cise, with a randomized comparison of methods for eliciting input from a diverse group of stakeholders to determine research priorities. The results, however, should be interpreted in light of the study limitations. The re-quirement for participants to have regular access to a computer and the Internet may have limited eligible participants and thus the generalizability of findings. However, Internet access is increasingly common among patients with chronic disease (up to 70 % of patients with chronic illness are Internet users) [29], and approximately half of patients with advanced CKD access the Internet to obtain health information [30, 31]. Further, little evidence exists to guide the evaluation of stakeholder engagement with such consensus-driven processes [32]. While quanti-tative and written responses on the post-intervention sur-vey provide valuable information, additional insights could be gained through participant interviews evaluating indi-vidual experiences with the interventions. Finally, 6 of the patient and caregiver steering group members participated in the NGT in-person workshop, and were not part of the randomization process. Although these steering group members were involved in the earlier stages of the CKD priority setting partnership (ie identification of research priorities through national survey and refinement of these priorities into a top 30 shortlist), the final ranking of the top 10 priorities was a collaborative process involving all stakeholders participating in the workshop. All partici-pants were encouraged to contribute to the final ranking process, and the top 10 priorities reflect group consensus rather than the views of any one individual.
Conclusions
wiki-inspired process identified different top ten CKD-related research priorities compared with an in-person workshop, and participants in the wiki group were less likely to feel engaged and satisfied. We are unable to determine if the differences in research priorities iden-tified were due to the different processes or differences in the groups themselves, as priorities may vary by group irrespective of the process used to establish them. This is important to address in future work. Modifications to the wiki tool, including optimization of a communication fea-ture to improve the ability of participants to communicate their views and preferences, need to be designed and evaluated before a wiki process can be considered a feasible alternative to an in-person workshop for en-gaging patients in determining research priorities.
Additional files
Additional file 1:Table summarizing methods used for assessing primary and secondary outcomes. (DOCX 104 kb)
Additional file 2:Figure demonstrating usability responses from the wiki-based group on the post-intervention questionnaire. (DOCX 120 kb)
Acknowledgements
We would like to thank Mr. David Newton for his assistance in the development and administration of the online collaborative ranking tool. We also thank Ms. Sarah Gil for her regular communication with and technical assistance to all participants in the wiki group, and Ms. Sarah Gillis for assisting with the coordination of the in-person workshop.
Funding
MJE is supported by an Alberta Innovates-Health Solutions Clinician Fellowship and Kidney Research Scientist Core Education and National Training Program Post-Doctoral Fellowship. BRH, NP, SBA and BJM are supported by an alternative funding plan from the Government of Alberta and the Universities of Alberta and Calgary. SBA is supported by an Alberta Innovates–Health Solutions Clinical Investigator award. HT is supported by an Alberta Innovates–Health Solutions Graduate Studentship. AL is supported by a Canada Research Chair in Health Policy and Citizen Engagement, SES is supported by a Tier 1 Canada Research Chair in Knowledge Translation, and BRH is supported by the Roy and Vi Baay Chair in Kidney Research. This research was supported by the Canadian Institutes of Health Research and an interdisciplinary team grant from Alberta Innovates–Health Solutions, the Interdisciplinary Chronic Disease Collaboration. The sponsors of the study had no role in study design, data collection, data analysis, data interpretation or writing of the report.
Availability of data and material
The data supporting the main conclusions of this article are included within Table 2 (top 10 CKD-related research priorities resulting from the in-person workshop vs. online wiki-based groups). All other data is stored in a secure data holding at the University of Calgary and may be made available from the corresponding author on reasonable request.
Authors’contributions
MJE, BRH, and SES were responsible for the conception and design of this project. MJE, NP, SBA, AL, GCC, DRH, KTH, ACL, BVV, DMY and BRH were members of the JLA steering committee who contributed substantially to the design and implementation of the workshop and wiki interventions. HT, MD, EL, and BJM were additional non-steering committee individuals who provided methodological expertise and significant contributions in the planning and administration of the in-person workshop. MJE conducted and is responsible for the data analysis. All authors drafted or revised the article critically for important intellectual content, and all authors gave final approval of the version to be published.
Competing interests
The authors declare that they have no competing interests.
Consent for publication
Not applicable.
Ethics approval and consent to participate
Ethics approval for this study was obtained from the Institutional Review Board of the University of Calgary. Written informed consent was obtained from all participants.
Author details
1Li Ka Shing Knowledge Institute, St. Michael’s Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada.2Institute of Health Policy, Management and Evaluation, University of Toronto, 155 College Street, Toronto, Ontario M5T 3M6, Canada.3Department of Medicine, University of Toronto, Suite RFE 3-805, 200 Elizabeth Street, Toronto, Ontario M5G 2C4, Canada.4Department of Medicine, University of Alberta, 8440 112 Street, Edmonton, Alberta T6G 2G3, Canada.5Department of Medicine, University of Calgary, 1403 29 Street NW, Calgary, Alberta T2N 2T9, Canada.6Suite 200, 5970 Centre Street SE, Calgary, Alberta T2H 0C1, Canada.7343 Webb Court, Newmarket, Ontario L3Y 5E8, Canada.823-1140 Falcon Drive, Coquitlam, British Columbia V3E 2J6, Canada.94K Spadina Avenue, Suite 1820, Toronto, Ontario M5V 3Y9, Canada.10A4123-409 Tache Avenue, Winnipeg, Manitoba R2H 2A6, Canada.1131 Seligs Road, Prospect Village, Nova Scotia B3T 2A6, Canada.12Department of Community Health Sciences, University of Calgary, 3280 Hospital Drive NW, Calgary, Alberta T2N 4Z6, Canada.
Received: 26 February 2016 Accepted: 18 August 2016
References
1. Coresh J, Selvin E, Stevens LA, Manzi J, Kusek JW, Eggers P, et al. Prevalence of chronic kidney disease in the United States. JAMA. 2007;298(17):2038–47. 2. Arora P, Vasa P, Brenner D, Iglar K, McFarlane P, Morrison H, et al. Prevalence
estimates of chronic kidney disease in Canada: results of a nationally representative survey. CMAJ. 2013;185(9):E417–23.
3. Aitken GR, Roderick PJ, Fraser S, Mindell JS, O'Donoghue D, Day J, et al. Change in prevalence of chronic kidney disease in England over time: comparison of nationally representative cross-sectional surveys from 2003 to 2010. BMJ Open. 2014;4(9), e005480.
4. Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med. 2004;351(13):1296–305.
5. Jha V, Garcia-Garcia G, Iseki K, Li Z, Naicker S, Plattner B, et al. Chronic kidney disease: global dimension and perspectives. Lancet. 2013;382(9888):260–72. 6. Ronksley PE, Hemmelgarn BR. Optimizing care for patients with CKD.
Am J Kidney Dis. 2012;60(1):133–8.
7. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50.
8. Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet. 2011;378(9805):1777–8.
9. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17(5):637–50. 10. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM,
et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.
11. Canadian Institutes of Health Research. Strategy for Patient-Oriented Research (SPOR) http://www.cihr-irsc.gc.ca/e/41204.html. Accessed 15 Aug 2015. 12. Patient-Centered Outcomes Research Institute (PCORI) http://www.pcori.org/.
Accessed 15 Aug 2015.
13. Cowan K. The James Lind alliance: tackling treatment uncertainties together. J Ambul Care Manage. 2010;33(3):241–8.
14. Cowan K. The James Lind Alliance Guidebook. Oxford: James Lind Alliance; 2013. 15. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in
systematic reviews: knowledge transfer for policy and practice. J Contin Educ Health Prof. 2008;28(2):67–72.
17. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Ann Intern Med. 2010;152(11):726–32.
18. Powers BJ, Trinh JV, Bosworth HB. Can this patient read and understand written health information? JAMA. 2010;304(1):76–84.
19. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):i–iv. 1–88.
20. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181–94. 21. Lavallee DC, Williams CJ, Tambor ES, Deverka PA. Stakeholder engagement
in comparative effectiveness research: how will we measure success? J Comp Eff Res. 2012;1(5):397–407.
22. Manns B, Hemmelgarn B, Lillie E, Dip SC, Cyr A, Gladish M, et al. Setting research priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol. 2014;9(10):1813–21.
23. Tong A, Chando S, Crowe S, Manns B, Winkelmayer WC, Hemmelgarn B, et al. Research priority setting in kidney disease: a systematic review. Am J Kidney Dis. 2015;65(5):674–83.
24. den Breejen EM, Nelen WL, Knijnenburg JM, Burgers JS, Hermens RP, Kremer JA. Feasibility of a wiki as a participatory tool for patients in clinical guideline development. J Med Internet Res. 2012;14(5), e138.
25. van de Belt TH, Faber MJ, Knijnenburg JM, van Duijnhoven NT, Nelen WL, Kremer JA. Wikis to facilitate patient participation in developing information leaflets: first experiences. Inform Health Soc Care. 2014;39(2):124–39. 26. Pagliari C, Grimshaw J. Impact of group structure and process on
multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002;8(2):145–53.
27. Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4:67–74.
28. Mansell I, Bennett G, Northway R, Mead D, Moseley L. The learning curve: the advantages and disadvantages in the use of focus groups as a method of data collection. Nurse Res. 2004;11(4):79–88.
29. Kruse RL, Koopman RJ, Wakefield BJ, Wakefield DS, Keplinger LE, Canfield SM, et al. Internet use by primary care patients: where is the digital divide? Fam Med. 2012;44(5):342–7.
30. Schatell D, Wise M, Klicko K, Becker BN. In-center hemodialysis patients’use of the internet in the United States: a national survey. Am J Kidney Dis. 2006;48(2):285–91.
31. Ong SW, Jassal SV, Porter E, Logan AG, Miller JA. Using an electronic self-management tool to support patients with chronic kidney disease (CKD): a CKD clinic self-care model. Semin Dial. 2013;26(2):195–202.
32. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal • We provide round the clock customer support
• Convenient online submission • Thorough peer review
• Inclusion in PubMed and all major indexing services • Maximum visibility for your research
Submit your manuscript at www.biomedcentral.com/submit
