Application management

There are five Cerner software domains implemented under the eMR project, and two additional installations of FirstNet (SSWAHS, CHW) which have been independently implemented. A State Base Build (SBB) domain exists for development and demonstration of the SBB. These FirstNet domains are described below.

SSWAHS*

CHW*

NSCCH AU

SEAHS_AU

SWAHS_AU

Observations

North Sydney LHD Central Coast LHD South East Sydney LHD Illawarra LHD

Western Sydney LHD Nepean Blue Mountains LHD Sydney LHD

South West Sydney LHD

Children's Hospital at Westmead Have own Cerner implementation - not part of eMR project

Far West LHD Western LHD Murrumbidgee LHD Southern LHD Northern NSW LHD North Coast LHD

Hunter New England LHD GSGW AU

*Not part of eMR project - does not run Cerner software.

NCAH_AU

HNELHD*

There is considerable variance in each of these implementations, including variance in operating systems, Cerner versions, SBB versions, local configuration, and local implementation of capability outside of the eMR scope. Current implementations are summarised in the following table:

Observations

+ Not part of eMR project - run and manage own software and technology

* SBB instance is used for SBB development and demonstration only.

This complex and varied environment presents a difficult and expensive application management challenge.

Application governance model and processes exist, both locally and centralised

The processes to request a change to the State Base Build are well defined and documented. These processes include both local recording and triage processes which then feed into the central triage, recording, assessment and management processes. While these processes are well defined, the local leadership's focus and representation on the governance committees impacts the overall effectiveness of the process. It was reported that ED doctors often find it difficult to attend governance meetings, and this has at times skewed prioritisation and progress toward the interests of those who attend.

Multiple versions of infrastructure, Cerner and build across the State adds to management complexity

There is a plan and strategy to standardise core FirstNet configurations across the State, however, system changes during the rolling implementation process, and the capacity of sites to localise their systems has driven variances in the implementations at each site. This diversity drives both complexity and operational cost. At some sites new capability cannot be implemented as the base build at that site does not support the required changes.

This variance in both the central FirstNet implementation and local integration architecture makes it difficult to get an end-to-end view of any process across the State. This complexity leads to user frustration about the responsiveness of support services, such as the time taken for issue remediation or the effort required to diagnose and respond to a reported issue.

Plan in place to gradually migrate regions to a consistent build

HSS have developed a plan to migrate the required infrastructure and Cerner versions toward the prerequisites for the delivery of a more standardised State Base Build. This plan includes the upgrading of hardware, operating systems and Cerner versions. Successful execution of this plan should lead to a more uniform set of environments that can accept and support new standardised configurations and capability.

Observations

An ED system requires the integration of a number of systems, including FirstNet, the hospital's Patient Administration System, and various order receiving and laboratory management systems. Ownership of these integrating systems is also shared by HSS and the Local Health Districts. The method by which this integration has been implemented varies, and is dependent on what systems and integration technology exists at each site.

We observed a number of integration models at different sites:

Centralised integration

Cerner PAS

Local integration engine

Local systems integrate directly to the eMR SBB integration engine. This has the advantage that there is only one integration infrastructure to maintain, however, it makes the centralised integration engine more complex to manage.

Also, there are many more interfaces to manage at the intersection of responsibility between HSS and the Local Health District. The central integration environment must know about all of the implementation details of the local systems.

In sites where the Cerner Patient Administration System (PAS) is deployed, the Cerner software manages all interactions between the clinical modules and the PAS. This considerably simplifies the integration architecture. If additional Cerner modules are deployed, the integration architecture is further simplified.

Where a local integration engine is deployed the integration between the centralised and locally managed environments is much simpler and more easily managed. Local Health Districts are more clearly responsible for integration of local systems to the local integration engine.

This variety of implementation models increases the overall management complexity of the end-to-end solution and contributes to a general lack of clarity about the management responsibility of the overall architecture.

Observations

4.4.'

Variety of integration models

As described above, we found a variety of integration models deployed across the State, as well as variances and combinations of these models. This variety and complexity has added significantly to the overall management effort required to operate and support the end-to-end solution. This variance in models makes it harder to maintain and diagnose issues, since the responsibility for interface management is different for each model and is often not well defined.

Varying integration reliability and performance

We observed a range of reports on the performance and reliability of interfaces with the eMR solution.

Some sites reported reliable interfaces and little awareness or concern about failed or lost messages.

Elsewhere it was reported that while the integration was reliable the performance was often too slow.

And in some sites FirstNet went live without some core interfaces being implemented, and as a result, staff resorted to printing out orders for sending to Pathology for manual re-entry into the pathology system.

There were also some reports of interfaces that did not reliably translate and deliver the complete message set. However, in most cases, this was associated with PAS integration and was being addressed and rectified. We did not receive any reports of there being specific adverse clinical outcomes as a result of integration issues.

Cerner supports `publish-subscribe' model

The FirstNet solution supports a `publish' model for message integration in which messages are

`published' to an integration engine for translation and transmission. In this architecture, FirstNet effectively passes all responsibility for the delivery of the message to the integration engine. As a consequence, FirstNet does not necessarily find out if a message fails or contained errors. Equally, there is no feedback to the user if a message has errors or fails. While this is not un-common in systems integration architectures, it places considerably more responsibility on the integration environment and manual processes needed to review and action any interface problems.

4.4.2 _._ -As and compliance

Code and message standards have been generally well applied

The use of HL7 for defining messages and SNOMED for the definition of message content (codes) has been generally well applied, and has resulted in a defined and standardised integration architecture.

NSW Health has only implemented a restricted set of messages which has aided with standardisation and management of integration, but has in some cases resulted in restrictions on the use of the system.

For example, the system only allows `new order' or `cancel order' messages which makes it difficult for users if they wish to make a change to an existing order. A number of clinicians expressed the need to change an order. The maintenance of message and code standards across systems is manually undertaken, which contributes to the effort required to manage overall system integration.

Limited visibility of failed messages

It was reported that there is little to no user visibility of any messages that contain errors or have failed.

In most cases this was not a significant problem for users, who generally found integration to be reliable.

In the interface design documentation we observed scant consideration for error processing. In most cases specifications defined error handling functionality with the statement `Use standard error processing'. We expected to find greater detail on what technology, reporting and processes are required to identify and remediate messages that report errors.

Observations

4.4.4-1

Cerner provide support for outbound messages

Cerner is responsible for defining and delivering interfaces into and out of the eMR SBB interface engine.

The interfaces and message content were designed, documented, built and deployed by Cerner. While Cerner was engaged to develop the message set, the Local Health Districts maintain overall responsibility for reliable message delivery.

Local Health Districts provide support for local system integration

Any required message translation and integration with local systems was the responsibility of the Local Health Districts, and any relationship they had with their local system suppliers. These interfaces were developed at each site with little coordination across sites. It was observed that most interfacing systems support the required interfacing capability and standards, or have interfaces to local integration engines that enable appropriate integration to the eMR SBB integration engine.

It was reported that in one example the system used by the Pathology laboratory operator could not match the laboratory results with the corresponding order and encounter within FirstNet (the system could track at a patient identity level only). FirstNet was enhanced to match the result to an encounter;

however, the result of this matching is not completely reliable. To compensate, staff check results at the user level, rather than just at the encounter level.

4.5.1

The primary vendor for the supply and implementation of FirstNet is the Cerner Corporation Pty Limited.

The scope of the vendor's responsibility is comprehensively established via three related contracting document types:

• The original deed of agreement IT-135 between Cerner Corporation Pty Limited "the Contractor" and Health Administration Corporation (i.e. the New South Wales Department of Health), dated 27 September 2002. The original deed of agreement provides for hardware acquisition and installation, hardware maintenance, software licensing, IT consultancy, software development and modification, software support, systems integration, data conversion and migration and transition out services. In addition, the agreement specifies a range of mechanisms and measures to govern the relationship with Cerner.

• GITC Official Orders lodged under the period purchasing agreement IT-135, detailing the scope of services to be provided relating to a specific site implementation.

• Change Requests covering the provision of software development services or support related to system enhancements or fixes.

The vendor's responsibilities appear to be adequately specified via the contracting mechanisms While the requirements for system development and support are formally documented (as described above) the approach taken by NSW Health for managing the relationship with Cerner has generally been less formal. While Cerner provides many of the performance reports as specified in the deed of agreement (IT-35), Cerner has not been formally held to account by HSS in line with the performance governance model specified in the agreements.

That said, by most measures it would appear that HSS established an effective working relationship with the vendor. Evidence of this may be found in the measures taken by Cerner to invest in the success of the program, such as providing capability and services beyond that specified in the agreements, and conducting annual reviews by international specialists.

Observations

4.5.2 'eness

A layered governance model operates to identify, assess and escalate to the vendor support and change requests received from SBB sites. User support and the handling of change requests are facilitated via local site representatives. The local user group escalates to the State AAG and ultimately Cerner if necessary. Under the support model, Cerner's responsibility includes third level FirstNet support and the development and implementation of software changes based on direction from both AAG and at times, local areas or sites.

Evidence of varying support effectiveness, change management and inadequate code configuration

Feedback on overall support effectiveness varied by area and site, and some reports of inadequate system change management and code configuration management were received. For example, users expressed frustration that software changes previously implemented would be lost after subsequent version updates.

The implementation approach adopted was that of a rolling model whereby a team would implement the system at a site, including the delivery of initial training, then move on to the next site. The framework for the Change Management and Communications approach was developed centrally by HSS, including the development of tools and templates. Each of the LHDs then localised the approach and materials to suit their specific requirements and took responsibility for delivery.

Implementation approach did not allow for follow up reinforcement

There is evidence that the time spent by the implementation teams implementing the system and supporting users to learn and become familiar with the system varied by site, and varied according to the size of the site and the number of users. The ED Director of a small rural hospital reported that the implementation team was on site for a total of 5 days. At some of the larger urban hospitals the implementation team was reported as being onsite for weeks prior to and following implementation.

There was general agreement that there was insufficient support during the implementations and insufficient follow up support after the implementation team moved to the next site. Some clinicians reported that their hospitals had invested in dedicated support technicians to trouble-shoot and provide support.

Consensus on the value and quality of the implementation team

Feedback received throughout the review on the value delivered by the implementation teams was consistently high. Clinicians consistently reported that the implementers possessed the requisite knowledge of the system, clinical processes and implementation practices to effectively implement the system.

Varying levels of local leadership and investment to support change management and communications

Evidence was provided to our review of inconsistent support by senior management at some sites for the implementation process, specifically in ensuring training attendance, implementation of changes to local work practices and investment on local support capabilities. This variation generally correlates with the level of user acceptance of the new system at a site.

Transparency of change management lacking

When functional changes are made to the system, (either as a result of change requests or version upgrades that have impacted the way clinicians use the system), the process has not always been

Observations

sufficiently transparent. Clinician's `discovery' of unexpected, unannounced or unsupported changes has resulted in varying degrees of frustration. .

Change impact of the system on local clinical processes was not adequately addressed

There is some evidence that the system was not designed to optimise business processes effectively.

Some clinicians reported that the system imposed changes to clinical processes which they regard as suboptimal (for instance, requiring patients to be `clerked' prior to triaging). In contrast to this, some clinicians felt that the system has improved discipline and enforced best practice, albeit with an accompanying increase in the number of steps in some tasks or activities.

Limited communication or formal process to support sharing of [earnings

Across various sites, as users have become more familiar with the system, local configurations have been made to address many of these usability and process-related problems and enable more effective work practices. Typically, these improved techniques have only benefited the site where they were developed, as there is no formal process or assigned resources to enable improvements that could be shared to be communicated and leveraged by other sites.