desirable effects depends on who you ask. Grover et al. (1996: p 182 cited in Seddon et al. 1999) list four different classes of evaluation perspectives: 1) users, 2) top management, 3) information system personnel, and 4) external entities (e.g., politi-cians).
Some of the evaluation literature points to the issues about the perspective of the evaluation (e.g. Kumar 1990; Seddon et al. 1999; Klecun and Cornford 2005). How-ever, few are concerned with who is to specify the effects. For example, Kumar (1990) points out that evaluation criteria are, in most cases, defined by those who have designed the system or managed the implementation. Not even in rather con-crete tools aimed at supporting evaluation like Effects Measures of Public IT Projects (Telestyrelsen 2007) or Guidelines for Good Evaluation Practices in Health Infor-matics (GEP-HI) (Nykänen and Brender 2008) is there guidance on who is to specify the evaluation criteria.
To summarize, I have argued that formative evaluation is the most useful and con-structive type of evaluation when concerned with the design and implementation of healthcare information systems. Formative evaluation is especially relevant when the purpose of doing evaluation is improvement and learning, as the iterative process in formative evaluation creates opportunities for personal and organisational learning.
The implications in this section do, in principle, account for both summative and formative evaluation. However, formative evaluation, as presented in the preceding sections, has the potential to address the negative implications because the criteria are iteratively reflected upon and open to being changed, as in the approach to forma-tive evaluation presented in Paper VI. However, in order for formaforma-tive evaluation to support learning and improvement, for example, by including emergent and opportu-nity-based changes, it is important to be aware of and address the negative implica-tions of pre-specified evaluation criteria.
5.3 Related work on evaluation in relation to design
Various researchers have proposed frameworks where evaluation plays a central role either in relation to information system development and/or to implementation and organisational change. In the following section I will briefly describe three frame-works in which evaluation plays a significant role.
5.3.1 Benefits realisation management
Benefits realisation can, in some respects, be compared with the formative evaluation approach presented in Section 5.1 and in Papers V and VI. Benefit realisation focuses on how to manage and organise IT-enabled business change “such that the potential benefits arising from the use of IT are actually realised” (Ward and Elvin 1999: p 197). To do so, Ward et al. (1996) suggest a cyclic process model for benefit man-agement that includes five elements: indentify and structure benefits, plan benefits realisation, execute benefits realisation plan, and evaluate and review results. In addi-tion, “effective benefits realisation requires an ongoing commitment to, and focus upon, the benefits, rather than the technology, throughout a system’s development, implementation and operation” (Ashurst et al. 2008: p 352). Ashurst et al. (2008) have investigated a number of information system projects, and they have found that most projects focus on the delivery of the IT solution but they only have a very
lim-54
ited focus on work design, process engineering, organisational and benefits re-alisation (p 365). Consequently, they suggest looking more into the competencies needed in the organisation to mange benefits realisation: benefits planning, benefits delivery, benefits exploitation, and benefits review (Ashurst et al. 2008). More con-cretely, Ward and Elvin suggest building on stakeholder involvement, including sen-ior management commitment, active management of benefits along with approaches to system development and project management, and the need for focusing on under-standing the origins of the need for change.
Benefits realisation is mainly concerned with theoretical models, whereas my re-search aims at contributing to the practice field; hence, I have been more concerned with how to identify and structure benefits and how to plan and execute benefits rea-lisation in terms of effects.
5.3.2 Results-driven incrementalism
Results-driven incrementalism requires a project to be divided into a number of short (approximately 3 months), intensive cycles of implementation, each with a focus on delivering measurable business benefits. The target business results are used to drive decisions, divide the implementation into non-overlapping increments, ensure that each increment implements both software functionality and complementary changes, and to use results as the basis for the next increment (Fichman and Kemerer 1997).
Results-driven incrementalism is based on the proposition that effects occur within three months. For some effects that is a rather short amount of time. Results-driven incrementalism, like benefits realisation, does not take opportunity-based and emer-gent effects into account.
5.3.3 Business engineering
Giaglis (1999) presents a business engineering framework that advocates integrating process-based organisational design and the evaluation of business processes and in-formation systems. In his review of process-based organisation design, inin-formation systems evaluation, and information systems development literature, Giaglis indenti-fies a number points all directed at the need for improved evaluation of information systems in the context of business engineering. For example, he states that informa-tion system development methods do no pay enough atteninforma-tion to the importance of and difficulties with evaluation ex ante. Giaglis also points out that information sys-tem evaluation ought to be integrated into the design of business processes, but that existing information system evaluation methods focus solely on the information sys-tem project without paying explicit attention to the business process. Through this review and a case study, he proposes “to substitute the IS project with the business process as the fundamental unit of analysis in IS evaluation” (Giaglis 1999: p 24).
However, this recommendation seems to be too general compared with the processes and activities of focus in a work system perspective. Furthermore, business engineer-ing is seen from a management perspective and is not concerned with involvengineer-ing the business process participants (Giaglis 1999).
Part Two has been concerned with presenting the four elements that constitute the theoretical background for this thesis. The four elements provide a theoretical basis for thinking and talking about participation, evaluation, design, and implementation
55 of HIS. However, before we are able to use these theoretical elements, either for ana-lysing and understanding participation in the empirical work or for applying the knowledge derived, we need to establish an analytical framework and an approach based on the intersection of the elements. This approach will be established in Part Three.
56
PART THREE: UNDERSTANDING AND APPROACHING PARTICIPATION IN
EVALUATION
Part Three comprises a synthesis of the theoretical and empirical knowledge derived from the work related to this PhD thesis. The first chapter in this part (Chapter Six) accounts for an analytical development of an approach to organisational implementa-tion of HIS based on a participatory design perspective. The approach includes see-ing implementation as ongosee-ing design of work systems and provides a basis for an-swering the research question. The approach is a synthesis of the elements that con-stitute the theoretical background. In Chapter Seven, the approach is used as an ana-lytical framework to analyse and discuss the empirical material from the three stud-ies. Incidents from the papers and studies will be analysed and discussed through the analytical framework in order to increase our understanding of participation in evaluation, as well to provide examples of how participation in evaluation has been approached as part of an application of the framework. Chapter Eight presents and discusses five central contributions of this thesis, apart form the contributions pre-sented in the papers. Chapter Nine comprises some concluding remarks that include implications for research and practice and suggestions for further research.
57
Chapter Six: A participatory design approach to organisational implementation
In the following chapter a synthesis of the theoretical elements from Part Two is out-lined. This synthesis results in an approach to HIS implementation based on forma-tive evaluation as part of ongoing design of healthcare work systems. The approach also serves as an analytical framework for understanding participation in evaluation as part of the design of healthcare work systems.
The approach can be viewed as a result of the research process that was illustrated Chapter Two, Figure 2. The approach has emerged through an interactive process of theoretical inspiration and empirical experiences and findings, a process following the principles of analytical induction. This means, first, that participatory design of healthcare information systems has been defined as the phenomenon of interest. Sec-ond, based on the theoretical background, a framework was formulated as a hypothe-sis to explain the phenomenon. Third, the framework was analytically applied to ex-plain the first two studies, during which the framework was adjusted. Last, the framework was operationalised and applied during the third study, which can serve as the next step of refinement.
In order to present and discus the framework, various intersections between the four different theoretical elements and their mutual influence will be outlined.