Participatory design of a collaborative clinical trial protocol writing system

j o u r n a l h o m e p a g e : w w w . i n t l . e l s e v i e r h e a l t h . c o m / j o u r n a l s / i j m i

Participatory design of a collaborative clinical trial

protocol writing system

Chunhua Weng

_{, David W. McDonald}

_{, Dana Sparks}

_,

Jason McCoy

_{, John H. Gennari}

a_{Center of Pathology and Oncology Informatics, University of Pittsburgh, PA, United States} b_{Information School, University of Washington, Seattle, United States}

c_{The Southwest Oncology Group, United States}

d_{Cancer Research and Biostatistics, Seattle, WA, United States}

e_{Department of Medical Education and Biomedical Informatics, University of Washington, Seattle, WA, United States}

a r t i c l e

i n f o

a b s t r a c t

Objective:To explore concrete approaches to socio-technical design of collaborative health-care information systems and to design a groupware technology for collaborative clinical trial protocol writing.

Method:We conducted “quick and dirty ethnography” through semi-structured interviews, observational studies, and work artifacts analysis to understand the group work for protocol development. We used participatory design through evolutionary prototyping to explore the feature space of a collaborative writing system. Our design strategies include role-based user advocacy, formative evaluation, and change management.

Results:Quick and dirty ethnography helped us efficiently understand relevant work prac-tice, and participatory design helped us engage users into design and bring out their tacit work knowledge. Our approach that intertwined both techniques helped achieve a “work-informed and user-oriented” design. This research leads to a collaborative writing system that supports in situ communication, group awareness, and effective work progress tracking. The usability evaluation results have been satisfactory. The system design is being trans-ferred to an organizational tool for daily use.

© 2006 Elsevier Ireland Ltd. All rights reserved.

1.

Introduction

Many researchers have realized the importance of social factors and organizational issues for system design [1–3]. However, concrete methods that account for social factors in system design are largely unavailable or have insuffi-cient guidelines. Participatory design (PD) is one approach to socio-technical designs[4]. It is a proactive design method that explicitly advocates active user participation throughout

∗_{Corresponding author. Tel.: +1 412 513 5340; fax: +1 412 647 5380.}

E-mail addresses: cweng@cbmi.pitt.edu (C. Weng), dwmc@u.washington.edu (D.W. McDonald), dsparks@swog.org (D. Sparks), jasonm@crab.org(J. McCoy),gennari@u.washington.edu(J.H. Gennari).

the design process. Unfortunately, PD has not been widely adopted in healthcare information system designs[5]. The major challenges involved in PD include long prototyping design cycles, difficulties to carry out representative partic-ipant selection, and changing or conflicting user inputs. PD without sufficient considerations of work practice may abuse user opinions and generate a perfect technical solution for a wrong workflow. Therefore, our research has two goals: one is to explore the design space of a collaborative clinical trial

1386-5056/$ – see front matter © 2006 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2006.05.035

protocol writing system, and the other is to enrich the partic-ipatory design theory by trying to answer these four research questions:

1. How can we make ethnographic studies complement par-ticipatory design methods?

2. How can we best apply participatory design to the design of collaborative information systems in this healthcare setting?

3. Who should be chosen to best represent users in a partici-patory design of a groupware technology?

4. What are effective strategies or incentives for active user participation?

Our research setting is The Southwest Oncology Group (SWOG), one of a number of cooperative programs that develop cancer clinical trial protocols under the direction of National Cancer Institute (NCI). SWOG members include approximately 4000 leading clinical experts, statisticians, protocol editors, and other clinical trial workers across the country. Each SWOG clinical trial protocol takes 4–9 months to author and the resulting protocol ranges from 60 to more than 100 pages. A combined use of Microsoft Word and email systems is inef-ficient and error-prone for the collaborative protocol writing processes. Prior to our efforts, SWOG had tried different sys-tems to support their protocol authoring process. However, none of those systems were adopted. One possible reason is that those systems failed to address the group work chal-lenges. Our research aims to fill this gap. This paper focuses on our participatory design experiences at SWOG. In the rest of this paper, we first elaborate on our design methodology, and then we present how it led to a usable system design. We conclude with a discussion about the effectiveness and gen-eralizability of our research methodology.

2.

Design methodology

To design the right system in the right way in healthcare settings, it is important to understand both the work context and user preferences. In this research, to achieve a design grounded in work and users, we intertwined “quick and dirty ethnography” [6,7] and participatory design as Fig. 1 shows below. Based on this model, an iterative design process consists of three components: (1) quick and dirty ethnography assembles instances of working roles and their work activi-ties, and sensitizes designers with selected focus of the work practices; (2) participatory design engages role-based user

Fig. 1 – An iterative grounded design.

advocates, who understand users and are able to articulate users needs to designers, in carrying out system design through evolutionary prototyping; (3) formative evaluations are carried out through evaluative ethnography to understand the interaction between user advocates and the prototypes, and also to provide feedback for ethnography and participa-tory design. Evaluation results drive further field ethnography or help adjust subsequent participatory designs, and then improve the prototypes.

In the conventional social–technical design model, field studies and system designs are widely apart, and field stud-ies are mostly design oriented. With this method, there has been a better synergy between field studies and participatory designs. AsFig. 1shows, the prototype serves two roles: one as an evolving representation of the design ideas and the other as a probing tool to deepen the field knowledge.

Change management has been carried out in PD sessions through discussions with participants about the design fea-sibility for improving work experience, the possible changes to the current work practice, and the tradeoff between ben-efits and costs of embracing new changes. The research has received a human subject approval from University of Wash-ington at Seattle. The details for quick and dirty ethnography, participatory design, and evaluative ethnography in the above model are provided below.

2.1. Quick and dirty ethnography

To understand the current work practice, we carried out ethnography using the practical “quick and dirty ethnography” strategy developed by Hughes et al.[6]. Quick and dirty ethnog-raphy is different from conventional ethnogethnog-raphy in that it does not provide an exhaustive description of the whole work context; instead it strategically focuses on selected aspects of the work and provides relevant work details to designers in a short time period[7]. In this research, we are particu-larly interested in group communication, information sharing and integration, and the workflow of protocol writing. The following open-ended research questions are the foci of our ethnography:

1. What are the major steps in the protocol writing process? 2. Who is involved in each step and what are their job

respon-sibilities?

3. How is group communication carried out? 4. What is the most challenging part of the writing? 5. How do protocol writers coordinate the group work? 6. What is the protocol review and revision process like?

Our ethnographic methods include semi-structured inter-views with SWOG protocol writers, observation of protocol review committee (PRC) meetings, and protocol development artifact collection and analysis. These approaches leveraged information gained from one other.

2.1.1. Observational study

Over a period of about 6 months, we observed PRC meetings. At the time of our study, protocols were typically reviewed twice in PRC meetings. Each meeting reviews an average of two pro-tocols, and meetings occurred about once a week. Meetings

were attended primarily by statisticians, although one proto-col coordinator manager from San Antonio also attended via conference call. The first and the last author of this paper par-ticipated and collected notes from the meetings, as well as the official PRC notes that documented the suggested revi-sions and discussion points around a particular protocol. We observed where in protocols the major questions or discussion occur, how reviewers respond to other reviewers’ comments, how reviewers have common ideas or different opinions, and what review comments are typical. We collected notes from about 30 different protocols for later content analysis.

2.1.2. Artifact content analysis

The artifacts that we have collected and analyzed include (1) about 80 selective protocol development emails, which were forwarded to us by protocol writers; (2) organizational pro-tocol development guidelines or standards; and (3) around 1140 protocol review comments for 32 protocols. From these comments, we used grounded theory[8]to derive comment categories, major communication patterns, and group work problems. The major categories of comments are ordered as “revision suggestion,” “question,” “inconsistency,” “answer,” “ incom-pleteness,” and “incompliance with standards provided by NCI.” Therefore, most comments carry a substitution text for revi-sion. This observation led to our design for supporting editors to access review annotations when they revise protocols[4].

2.1.3. Semi-structured interview

Initially, we identified three major roles in protocol develop-ment: protocol coordinator, principle investigator, and statisti-cian. We recruited interviewees for each role using a snowball sampling method. There were seven protocol coordinators in total in SWOG at the time of study. We were able to interview three of them and their manager from San Antonio to under-stand the work of a protocol coordinator. We also interviewed four statisticians and three principle investigators who were recently involved in protocol development and were located in Seattle. During the second part of this research, based on new knowledge gained from our evaluative ethnography, we interviewed two more supportive roles for protocol develop-ment: data coordinator and PRC members, who perform pro-tocol review. We interviewed one data coordinator and two PRC members. Each semi-structured interview lasted about an hour[9]; some interviews took 2 h because those interviewees were willing to talk and provide more details. During the inter-views, we used a “process query” technique by asking each interviewee to draw a diagram of the writing process based on their perceptions. Moreover, we asked our interviewees to recall some extreme scenarios using the “Critical Incident Techniques” (CIT)[10]. Our users told us stories about a few extremely complex protocol development cases. CIT helped us get real personal experiences while minimizing interfer-ence from stereotypical reactions.

This ethnographic study provided us with an in-depth understanding of design-sensitive work practice details, such as communication problems, group interactions, and work-flow. We found that comments were essential to protocol writ-ing processes because reviewers convey their feedback and carry out discussions or negotiations with other reviewers through comments. On the other hand, editors and authors

iterate protocol document revisions based on suggestions in comments from experts. Therefore, effective support for group awareness[11], in situ communication, and comments incor-poration is greatly needed by protocol writers. Interestingly, we found that the process diagrams created by different pro-tocol writers were not exactly the same. Propro-tocol writers have disparate views of the work process and tool support needs. This finding alludes to the importance of involving represen-tative participants for each writing role in the design so that we could pattern their shared needs while coordinating their individual preference in a single system. These results effec-tively informed our participatory design as described below.

2.2. Participatory design

Our participatory design process spanned almost 2 years; it followed our first part of ethnography study and happened in parallel to our evaluative ethnography and the second part of work process ethnography. Our major techniques were role-based user advocacy, iterative prototyping, and change man-agement.

2.2.1. Role-based user advocacy

User-centered design has been proven to help ensure the usability of a system. However, an open research question is “Who could be the participants that represent users?” Muller defined six approaches to selecting representative users based on interpretations from the perspectives of statistics, poli-tics, design practice, and grounded theory[12]. Among these methods, the grounded theory approach ensures diversity among users, and the design practice approach finds the polit-ically representative users. Since our work involved multi-disciplinary users working on different roles, we wanted to combine the features of both approaches. We selected politi-cally representative users for each role in the protocol writing process identified by our ethnographic fieldwork and made sure that they were advocates of a new technology[13]. User advocates evaluated the system prototype in their real work practice and provides timely feedback to designers. There-fore, we called our approach “role-based user advocacy”. We selected participants across three roles to form a design team with us: one protocol editor, one study coordinator, and two statisticians. Although there has been some turnover in indi-viduals, we have kept representatives from all three roles closely involved throughout our iterative prototyping design process. In addition, we also kept a couple of policy makers in the loop all the time. Periodically, we reported the status of the design and demonstrated the ongoing design to these pol-icy makers. The polpol-icy makers understand the organization’s culture, provide leadership, and provide incentives to other participatory users to try out new technology. We found that timely communication with these leading roles is useful and important for a smooth system adoption later.

2.2.2. Iterative prototyping

As Scacchi pointed out, the classic socio-technical design or participatory design approach does not provide critical insights, tools, or guidelines beyond “user participation”[2]. One open research question that we tried to address dur-ing this process is “What counts as an active participation and

how to encourage it?” We utilized two effective props: system prototype mockup and evolutionary prototyping. Both ethno-graphic study results and input from participants inform the design.

First, to assess the logical feasibility of our system design, we began with a paper-based mockup interface and scenar-ios of a new workflow. These mockups were developed using Microsoft Visio, a drawing package that helps create fake screenshots that look as if from real windows systems. Sce-narios helped reconstruct work like contexts in which users are encouraged to formulate design ideas as domain experts. We performed cognitive walkthrough on the scenarios for our participants and elicited their feedback to refine the design [14]. This process was cost-effective and gave a vivid represen-tation of the potential system outlook. The scenarios provided sufficient details for the logical flow of functionalities to partic-ipants. All participants considered the artifacts intuitive and stimulating.

After we reached consent with participants on the interface and workflow design using mockups, we settled on the logical feasibility of our system design, and moved onto web-based prototypes to assess the technical feasibility of the potential design by considering issues such as data backend and concur-rency control strategies. We went through two major rounds of system infrastructure changes. Our initial prototype used a file-based backend, which supported the old SWOG work practice of “only one editor is authorized to make changes on documents” perfectly. However, participants felt encouraged by the first prototype and were willing to try a more syn-chronous collaborative work mode by enabling all the authors to write multiple sections of a protocol concurrently. This new request was not supported by the file system; therefore, we extend the system to a database-driven web application. Participants provided very positive feedback for the second prototype, which remains as the current design. Web-based prototyping enabled concrete experience and modification by prospective users.

Throughout our prototyping design, we held participatory design meetings with our participants as regular as once every 2 weeks. We consulted them about the usability of the inter-face design and system functionalities. Our participants were enthusiastic and felt encouraged because we incorporated many creative design ideas from them. We also felt that the iterative prototyping helped us build a mutual understanding and trust between system designers and user participants. We understood their work practice better, and they understood the technical feasibility better.

2.2.3. Change management through negotiations

A big barrier in innovation deployment is resistance to changes from users [15,16]. Markus identified three cate-gories of user resistance: user-centered, system-centered, and interactional[17]. Since participatory design emphasized user involvement, here we managed user-centered resistance: a resistance to innovation caused by a lack of knowledge or a reluctance to change on the user side. At SWOG we tried to minimize this resistance through frequent negotiation of changes with users. For example, we prepared “What are new changes” documents prior to each participatory eval-uation session and gave participants a clear comparison of

their current work practice and the potential new work prac-tice supported by our system. We elicited their feedback and discussed the tradeoffs with them. This approach served as educational sessions that helped participants understand design rationale, which turned out to be an important fac-tor for them when they made system adoption decisions later.

2.3. Evaluative ethnography

Formative evaluation is an integral part of our participatory design process. Since our design was exploratory, there was no prior peer system; a self-reflective evaluation is more appro-priate than comparison-based evaluations. Formative evalua-tion throughout the design process helped us collect contin-uous feedback from participants as the design evolved. Our evaluation methods largely follow Kaplan’s suggestions [3] from the following two aspects: (1) set up evaluation objectives from multiple perspectives and from the beginning of system design; and (2) use multiple evaluation methods. As Kaplan pointed out, well recognized areas of system evaluation in the medical informatics literature include: (1) technical and sys-tem features that affect syssys-tem use, (2) cost–benefit analysis, (3) user acceptance, and (4) patient outcomes. Here we are par-ticularly interested in evaluation areas (1) and (3); (2) and (4) would be the long-term evaluation objectives that could be carried out later. Therefore, we created a set of usability eval-uation questions for these two selected areas. Below are some example evaluation questions:

1. Can the new design fulfill the required functionalities that are supported by current work mode?

2. Can the new design address the range of problems that we observed via our ethnographic fieldwork?

3. Can the new design improve the group work experiences of these users?

4. Is the new system easy to use and are the system features appropriate and useful?

Following Kaplan’s suggestion that evaluation should use multiple methods [3], our approach integrated interviews, focus-groups, and system use observations. We designed a staged evaluation plan, including the following steps: (1) sys-tem testing from a software engineering perspective; (2) task-oriented, role-centered, and scenario-based user testing; and (3) short-term field trials in a natural work setting. We kept participants closely involved in each stage of design and evalu-ation while incrementally increasing the degree of complexity of group interactions in different evaluation stages so that we could keep some control and better refine the design accord-ing to feedback from participants. So far, we have carried out two field trials for the system focused on interactions between protocol reviewers and protocol coordinators.

3.

Research results

Quick and dirty ethnography, participatory design, and paral-lel evaluative ethnography form a integrated design process in this research and effectively lead to satisfactory research

Fig. 2 – With our PCAT infrastructure, users have a single web portal to the evolving document as well as comments and a library of standards. Communication is still open, but all users work with the same document.

results, including a dynamic comment model that facilitates collaborative writing at the data management level[18], a col-laborative clinical trial writing system prototype[4], and a description of collaborative protocol writing process at SWOG with some important design implications [19,20]. Below we briefly described the major group work problems, features of our collaborative writing system, and formative evaluation results.

3.1. Group work problems

At SWOG, the process of iteratively reviewing and revising protocol documents was error-prone and led to a number of work-coordination and communication problems. We iden-tified problems such as ineffective version control of evolv-ing documents, inefficient group communication via emails, divergent views of the process by different roles, and ineffec-tive group coordination[20]. The major unmet needs of current protocol writers are: (1) awareness of the shared workspace and individual contributions to the evolving protocol, (2) a shared repository of protocol and comments with version con-trol for both, and (3) an integrated reviewing and revising tool that has wide accessibility.

3.2. Current system designs

Our designs so far consist of two parts. The first part is a generic comment model that facilitates in situ docu-ment reviewing and communications[18]. The model defines dynamic text anchor in changing documents, discussion con-text, version information for a comment, as well as activity transition status to support group awareness. Based on this model, we also arrive at a web-based protocol collaborative writing tool (PCAT) that strengthens iterative reviewing and revisions[4]. In our PCAT system, we support a new work-flow model, whose comparison with the old workwork-flow model in SWOG is illustrated inFig. 2.Fig. 3shows a screenshot of the PCAT interface for protocol reviewers. The interface is divided into three panels. The left panel displays threaded comments. The top right panel displays the text of a protocol document, where a reviewer can select arbitrary text and insert com-ments; the commented text is then highlighted. Each reviewer is represented with a different color. The bottom right panel displays the details of a comment, including incorporation sta-tus, the time when the comment is made, who makes the comment, who is supposed to receive the comment, and other such metadata that we describe in our comment model[18].

A comment author can edit or delete his or her own com-ment, and any participant can attach new comments to any comment to facilitate online negotiations or discussions in the context of the shared document. Moreover, a floating window is displayed to show the recent activity of group members and to provide group awareness to collaborative protocol writers.

3.3. System formative evaluation results

Our formative evaluations have been focused on eliciting feed-back about the usability of our design of the new workflow, user interface, and other system features. Our usability eval-uation studies showed that users liked our system design, and particularly the following four aspects of our collabora-tive writing system. First, users liked “comment incorporat-ing” and “notification service” very much. They considered that these features could speed up the reviewing process and help group members stay abreast of the ongoing progress eas-ily. Second, interoperability between the system and the old tools such as email and MS Word was highly preferred. Users needed smooth transition of work between two systems for the best flexibility. We also realize that this is important for any innovation diffusion. We should not force users to give up their familiar tools easily; it is better to give them more than one option and let themselves decide what to keep and what to use. Third, users preferred an “all-in-one” design. In another word, users liked integration. The system currently supports email, editing, commenting, progress report, document man-agement and sharing, and many other features, which are far beyond our original intentions to strengthen the reviewing and revising activities. Such an integrated tool enables users to carry out all work in a single environment. Last, “less is more” is an important principle for work-oriented system design. Fil-ters for information about group awareness or comments are very well received and heavily used. Most users commented that they liked to see information specific to their working doc-ument or their group, and they liked direct links to their inter-ested information without navigating through multiple page. In addition, we found that “change management” remains a challenging task. Users tended to swing between the old practice and the new workflow model. Our users often fought with the question of “Is this change necessary and valuable?” We need further research to find out the answer; maybe a long-term field trial of the system can help convince users, where cost–benefit tradeoffs can be more explicitly measured. More-over, we realize that change management requires efforts from at least two levels: individual level (micro level) and orga-nizational level (macro level). Our “change explanation” docu-ments helped individual participants understand the system design and facilitated change management at a micro level. However, a large organization like SWOG has hundreds of potential users of this system who are used to the old stan-dard of e-mail and MS Word. Negotiating changes among the statisticians, protocol coordinators, and data coordinators is certainly a challenging task but getting the study coordina-tors, a constantly changing list of investigators who may or may not be computer savvy and are distributed widely as well as involved sparsely in this type of work, to use the system is a much more daunting task. It is an unknown quantity to manage changes at the organizational level.

3.4. Design adoption status

Overall, we are encouraged by our evaluation study results. Our design participants so far have given enthusiastic sup-port and positive feedback for the design. Many healthcare information system designs face adoption challenges. In this pilot research, we not only engaged users throughout the sys-tem design process, but also won the adoption of a pilot group of SWOG. Currently the statistical center’s leader is making efforts to transfer the prototype system into a fully-functional tool that will eventually be used daily in the statistical cen-ter of SWOG, which plays the major role in SWOG protocol development. The advantage of starting with a small branch of SWOG is that we can control the complicating factors from distributed clinical trial investigators for now. We also believe that this is a great opportunity to enable further collaborative research with other parts of SWOG or other collaborative clin-ical trials research organizations.

4.

Discussion

At the beginning of this paper, we raised four research ques-tions about participatory design. At this point, we can answer these questions based on our experiences from this research. First, ethnography could be an integral part of a participatory design process and could be used to increase the sensibil-ity of system design to the working context. Our quick and dirty ethnography proved to be practical and efficient in this work. We system designers did not strive for an exhaustive work process description; instead, we just focused on poten-tial work problems that are relevant to the design. We think an important step is to define research questions and pro-vide a focus for a “quick and dirty” ethnography. Interviews, observations, and working artifact analysis working together can sensitize designers to potential work problems in an effi-cient way. Second, collaborative healthcare information sys-tem should consider multiple roles and their needs together. Role-based user advocacy is an effective strategy toward col-laborative system design. Third, to engage users actively in a participatory design, evolutionary system prototypes are a significant improvement from other artifacts because they are concrete artifacts that users can interact with and they are a source of further design ideas; otherwise, it is hard for users to articulate what they want, what might work for them, etc. Moreover, evaluative ethnography should be conducted along the way because it provides timely feedback to designers and helps designers adjust the focus of ethnography studies. Oth-erwise, some design mistakes might be too costly to fix later. Therefore, we conclude that our design model as illustrated in Fig. 1is effective. We hope it could be useful for other health-care information system design.

One of our broad research interests is to improve the pro-cess of specifying software requirements. Currently many healthcare system designers still follow conventional require-ment specification docurequire-ments. The problem is that a large portion of system requirements is tacit and hard to articulate at the beginning of a system design process. As a result, the generated requirement specification documents often convey vague or incomplete system requirements, and are subject

to changes over time. Throughout our participatory design, we found that user requirements are emergent and change often; therefore, we did not use a static requirement speci-fication document. Instead, we had many design proposals, versioned system descriptions, notes on user feedback, a list of design modification tasks and priority features, and notes from meetings with different stakeholders of the project. We view these artifacts as a “dynamic requirements document” that evolves over time as the project proceeds. Such dynamic requirements documents need version control and these ver-sioned documents could provide design rationale to facilitate future system maintenance. It would be interesting to con-duct further study to find out whether this kind of “dynamic requirement specification document” is generalizable to other software projects.

Much of health care work is collaborative; thus, infor-mation systems design should consider group work require-ments. So far few groupware systems that directly address multiple user roles have been studied in medical informatics. Thus, we found it important to consider computer supported cooperative work (CSCW)[21]. This field offers insights into the design of groupware systems. For example, we leveraged ideas about group awareness and work-coordination from CSCW. We also applied the idea of “role-based user advocacy” into our participatory design approach, where we included partic-ipants from all significant roles in the authoring process. This ensures that the system will include incentives for all users, rather than just for certain stakeholders.

Our research extended our understanding of “change man-agement” during system design in the healthcare setting. Healthcare setting is a loosely-coupled collaborative environ-ment. Any system’s installation may affect a very big pool of users with diverse training backgrounds, different levels of expertise, and varied system requirements. How to man-age changes at the macro level to address many users is a new research problem that we discovered from our research process. We think this problem is especially important for collaborative healthcare information system design. Further research needs address this problem.

In summary, we believe that role-based user advocacy, iter-ative evolutionary prototypes, and ethnography are all integral parts to a work-oriented and user-centered design solution. These techniques together with CSCW concepts and tech-nology helped us reduce user resistance and usability errors during early prototyping and arrive at a feasible and user-preferred system design. We have designed our system and applied our methodology specifically in the work setting of SWOG collaborative protocol authoring; however, we hope that our approach can be more broadly applied to healthcare groupware design in general.

Acknowledgements

We thank the director of SWOG Statistical Center, Dr. John Crowley, for his constant support in this research and other SWOG participants for their active involvement, including

Jacqueline Benedetti, Jim Moon, Diana Lowry, and others. We also thank all reviewers for the helpful comments.

r e f e r e n c e s

[1] J. Ash, Organizational factors that influence information technology diffusion in academic health sciences centers, J. Am. Med. Inform. Assoc. 4 (2) (1997) 102–111.

[2] W. Scacchi, Socio-technical design, in: W.S. Bainbridge (Ed.), The Encyclopedia of Human-Computer Interaction, Berkshire Publishing Group, Great Barrington (MA), 2004.

[3] B. Kaplan, Addressing organizational issues into the evaluation of medical systems, J. Am. Med. Inform. Assoc. 4 (2) (1997) 94–101.

[4] C. Weng, J. Gennari, D. McDonald, A collaborative clinical trial protocol writing system, Medinfo 11 (2004) 1481– 1486.

[5] C. Sj ¨oberg, T. Timpka, Participatory design of information systems in health care, J. Am. Med. Inform. Assoc. 5 (2) (1998) 177–183.

[6] J. Hughes, et al., Moving out from the control room: ethnography in system design, in: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, 1994, pp. 429–439.

[7] A. Crabtree, Designing Collaborative Systems: A Practical Guide to Ethnography, Springer, 2003.

[8] B.G. Glaser, A.L. Strauss, The Discovery of Grounded Theory, Aldine Publishing Company, New York, 1967.

[9] L.E. Wood, Semi-structured interviewing for user-centered design, Interactions 4 (2) (1997) 48–61.

[10] J. Flanagan, The critical incident technique, Psychol. Bull. 51 (4) (1954) 327–359.

[11] P. Dourish, V. Bellotti, Awareness and coordination in shared workspaces, in: Proceedings of CSCW’92, ACM Press, New York, 1992, pp. 107–114.

[12] M. Muller, D.R. Millen, What makes a representative user representative? A participatory poster, in: Proceedings of CHI’01, Seattle, 2001, pp. 101–102.

[13] P. Mambrey, G. Mark, U. Pankoke-Babatz, User advocacy in participatory design: designer’s experiences with a new communication channel, Comput. Support. Coop. Work 7 (1998) 291–313.

[14] D. Rowley, D. Rhoades, The cognitive jog-through: a fast-paced user interface evaluation procedure, in: Proceedings of CHI’92, 1992, pp. 389–395.

[15] D.M. Berwick, Disseminating innovations in health care, JAMA 289 (2003) 1969–1975.

[16] N.M. Lorenzi, R.T. Riley, Managing change: an overview, J. Am. Med. Inform. Assoc. 7 (2) (2000) 116–124.

[17] M.L. Markus, Power, politics, and MIS implementation, Comm. ACM 26 (1983) 430–444.

[18] C. Weng, J. Gennari, Asynchronous collaborative writing through annotations, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 578–581.

[19] D.W. McDonald, C. Weng, J.H. Gennari, The multiple views of inter-organizational authoring, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 564–573.

[20] J. Gennari, et al., An ethnographic study of collaborative clinical trial protocol writing, Medinfo 11 (2004) 1461–1465. [21] W. Pratt, et al., Incorporating ideas from

computer-supported cooperative work, J. Biomed. Inform. 37 (2) (2004) 128–137.