Evaluation and Evolution

Once a system has been implemented it is common practice to evaluate how well it is working in practice (summative evaluation). There are many forms that evaluation can take, ranging from technical evaluations (are there ‘bugs’ in the system?), through user problems with usability and training to cost-benefit evaluations. In the cases we examined we found very few examples of these kinds of formal evaluations although it was common practice to set up a group of user champions who would report to the clinical change facilitators any problems that users were having. One consequence of a lack of formal evaluations is that informatics staff were often unaware of the extent of use or non-use of the facilities in the technical system.

One indirect way in which technical systems were evaluated was that the systems were used to generate reports of healthcare pathway performance. These reports shed light on the performance of the technical system in two ways. First, if the data is inadequate to produce complete reports, one reason may be that users are not using the system to enter all of the details of their operational work. Second, failure to meet targets may in part be because the technical system is not performing well or the users are unable to exploit its potential to help them achieve pathway targets. In all the pathway developments e-health systems were perceived as vital to the success of the pathway and hence, one way of evaluating the technical system is by evaluating its role in overall pathway success. However, there is no evidence in these cases that this is a deliberate aim of measuring performance against key performance indicators.

One reason why formal evaluations are not undertaken may be that there is very little evidence of the systems that are implemented achieving a steady state which might support a ‘before’ and ‘after’ comparison. There is an assumption in system development that after implementation there is a period when the system is in use and a new level or form of activity can be detected. There is nothing in the data from these cases that suggests any kind of steady state develops. Following implementation of system changes, it looks as though a process of emergent behaviour begins as users come to terms with new ways of working and that this results in requests for system changes that trigger bottom up forms of development and, as a consequence, the technical system continues to evolve. It is notable that the meetings established in the action research cycle (Appendix 3), which were often conducted between 6 to 12 months after data was collected for the cases, gave many examples of the continuous development of the systems that had been studied. As was remarked in one of the meetings, ‘e- health systems development is like painting the Forth Bridge; it never seems to end’.

It is possible to discern three drivers for the continuous evolution of these systems. The first, as described above, is user learning; as users experience the system and see its potential and its disadvantages so they exert pressure for changes in what has been provided and, in both LHCs, this is expected and there is a mature process for collecting and processing such requests. The second force is the delivery of subsequent parts of the technical system; in most cases the first implementation is just part of the planned system and there are second and third waves, covering, for example, in FEP the links between FUSION and Paris, the implementation of telehealth equipment etc. The third force is the continuing changes that occur in the organisational environment of the pathway as organisational re-configurations take place, as new national initiatives are launched etc which can change what is expected of the pathway and, as a result, what the technical system needs to support. Together these forces create a need for the technical system to keep evolving. For the most part this seems to become a series of smaller bottom up projects building on the system infrastructure already in place but with major convolutions at intervals when new top down initiatives require more radical change.

9.6 Conclusions

This review demonstrates that there are pockets of mature practice in the way that systems are developed in the two LHCs; for example, the process of creating and approving business plans, the project management of technical aspects of the development and the use of clinical facilitators or hybrids to support the interaction between informatics specialists and the user community. However, as one of the informatics directors remarked in a feedback session (Appendix 3), there

is no overarching process in place that can deliver a large project such as may be required for integrated care across a whole healthcare project. Such a process has not only to manage the stakeholder interests of many agencies and disciplines but has to integrate three interrelated but quite different strands of development in process, organisational design and technical system development. If we add that such a programme cannot be delivered in a short time, it is also necessary to acknowledge that the organisational turbulence in the NHS is going to mean the programme will be subjected to changing requirements during its lifetime. It is perhaps not surprising that these programmes become fractured in their delivery. On the basis of this limited sample, the overall effect appears to be that as a result, the system that is delivered is focused more on serving commissioning and management requirements than on operational needs to share information. What appears to happen is that, once a system is in place, compensatory developments occur in the evolution of the system to meet requirements that were not addressed in the first delivery. Whether they can actually be met depends on resources available for internal developments, on technical system flexibility and on governance issues. The e-health strategies in the two LHCS, although different, both emphasize the creation of an IT infrastructure that can facilitate the sharing of information for integrated care. The aim is to achieve this by (a) using a primary information system that is flexible and can be configured to meet different user needs and (b) using ‘middle ware’ systems of various kinds to make links between systems. As a result an evolution appears to be occurring in both LHCs in which there are pockets of integration amidst an array of systems they are not currently inter-operable. Whether the technical bases created in the two LHCs are sufficiently robust and resilient to cope with the on-going requirement for evolution is a question for future research.

10 The State of the Art on ‘Seamless