CENTRAL CARDIAC AUDIT DATABASE (CCAD) DATA VALIDATION DRAFT REPORT

(1)

SCTS CCAD QAP Visit – Morriston October 2006

1

CENTRAL CARDIAC AUDIT DATABASE (CCAD) DATA

VALIDATION DRAFT REPORT

Data validation visit to Cardiac Surgical Unit, Morriston Hospital, Swansea

Visitors:

Lin Denne (CCAD)

Mr Brian Fabri (QAP assessor – Consultant Surgeon, Cardiothoracic Centre, Liverpool)

Mr Mark Jones (QAP assessor and Lead – Consultant Surgeon, Wythenshawe Hospital, Manchester)

(2)

2

1.Introduction

National data collection in adult cardiac surgery is well established and has evolved to include risk models and more recently public reporting of

outcome data. Implicit in this initiative is the need for accurate data and a proposal for data validation has been made in the Society of Cardiothoracic Surgeons (SCTS) Fifth National Adult Cardiac Surgical Database Report 2003. There is a need for ensuring that data submitted for the CCAD project is robust because of a number of perceived shortcomings

-Lack of accurate recording of the number of operations at some centres -A high level of missing data for the items which are required for adequate risk adjustment in some centres

-Lack of independent validation of submitted mortality data

In an ideal world it may be desirable to impose an independent system where all data collected on all patients undergoing cardiac surgery is validated and corrected by independent personnel. This is not achievable within current available resource. The proposal for SCTS data validation is that each organisation should be subjected to a data validation visit. This would involve an independent review of the data that the hospital had submitted to CCAD, and a review of the processes that should be in place to ensure that the data is robust. The planned visits are to be organised by personnel from CCAD and undertaken by a combined team from CCAD and the SCTS.

The CCAD software has been rewritten over recent months and included in the development is functionality to allow the hospital that is submitting data and the validation team to view aspects of missing data, discrepancies of mortality between submitted and ONS traced data, and potential ‘gaming’ of risk factors. The access rights to this part of the soft ware is only available to the submitting hospital and visiting team, and not to general CCAD users.

The CCAD software development is now in a live format and we have used this as the basis for this validation report.

(3)

3

The visiting team were met at 09.30 by Dr Mark Ramsey, Clinical Director and Consultant Cardiologist. We were shown around the Cardiac Unit with an opportunity to meet and speak to staff. Following this, in the

Seminar Room, a presentation was given by Dr Ramsey which followed the template of the CCAD Data Validation visits. There was ample opportunity during this presentation for question and discussion. At the end of the presentation over lunch we were able to meet with Mr Malcolm Thomas, Divisional General Manager, and other consultant colleagues.

Swansea NHS Trust

The Swansea National Health Service Trust provides a comprehensive range of acute community and mental health services for the 250,000 residents of Swansea, as well as many specialist services for people across South and Mid Wales. The Trust was established on 1st April 1999 as a result of the merger of the former Swansea NHS Trust, Morriston Hospital NHS Trust and part of Glan-y-Môr NHS Trust. Employing in excess of 8,000 people, it had an income of over £300 million in 2004/5, mostly generated from NHS commissioning bodies such as Local Health Boards and Health Commission Wales (Specialist Services).

From 1st April 2003 the Trust implemented revised management arrangements based on Trust wide Divisions. The overriding purpose of this change was to create an organisational culture that concentrates on patients and care processes, not hospital sites.

Cardiothoracic Surgery lies alongside tertiary cardiology within the Directorate of Cardiac Services, which is part of the Division of Cancer and Tertiary Services. Each Directorate within the Division has its services commissioned directly by Health Commission Wales (Specialised Services), an executive agency of the Welsh Assembly Government.

(4)

4

The Divisional General Manager is Mr Malcolm Thomas, The Divisional Clinical Chair is Mr Hamish Laing and the Divisional Nurse is Mrs Linda Bevan. The Clinical Director for Cardiac Services is Dr Mark Ramsey and the Cardiac Services Manager is Mrs Jill Rowe.

The Trust Clinical Governance manager is Ms Nicola Williams and the Divisional Lead is Mrs Linda Bevan.

Physically, the Cardiac Surgery Service is part of a purpose built facility which is integrated with Cardiology, and Thoracic Surgery. It has evolved to its present form in 3 phases since 1997 and the Consultant Surgical staff consists of 5 consultant surgeons and 1 locum thoracic surgeon.

2. Structure of Data Collection Systems

Personnel

Dr Mark Ramsey. Clinical Director and Consultant Cardiologist

Dr Mark Anderson. Clinical Lead for Audit and Consultant Cardiologist, succeeded by Mr Pankaj Kumar, consultant Cardiothoracic surgeon on 1st September 2006.

Mr Aprim Youhana. Audit lead for submission of yearly returns to the SCTS database and quarterly returns to CEPOD.

Mrs Andrea Beniamous. Audit Facilitator (30 hours per week shared between cardiology and cardiac surgery) since June 2004. Responsibility for collection, collation and submission of data to BCIS, MINAP and CCAD.

Mr Stephen Morris. Cardiac Information Systems manager since April 2006. Appointed to support cardiac image archiving systems and maintain computer hardware and software for PATS (Patient Administration and Tracking System)/Dendrite and GE Centricity.

(5)

5

There are no audit clerks to support data collection and verification, and no standalone audit office.

Software System and Network

Data are collected onto the PATS/Dendrite software with a dedicated “UKC2” registry for cardiac surgical activity. There are 23 user licences within the cardiac centre that allow 6 concurrent users. The cardiac network on “Alpha” allows data to be entered on specified PCs throughout the cardiac centre. The IT department have a disaster recovery plan and data are automatically backed-up daily.

The department has recently invested in a new IT solution (GE Centricity) that enables the DICOM images collected in the cardiac catheterisation laboratory, echo laboratories etc to be embedded with the same database as descriptors of patient factors, operative details and outcomes. The advantage of the latter is that the images and database can be web-based allowing them to be more accessible both within the Trust and to external partners. In addition, unlike the PATS/Dendrite system, the Centricity system is a relational database allowing Structured Query Language (SQL) queries. Both the PATS database and the Centricity database have been adapted to allow them to export to national databases (e.g. MINAP, BCIS, CCAD programme etc.).

Overview of Process

All patients are registered onto the hospital’s PAS system with a unique identifier when they are referred to the hospital for the first time. Secretaries or ward receptionists who are trained in the use of PAS perform this task. The minimum dataset for entry on to the PAS system are first name, surname, and date of birth. (NHS number is not mandated). Patient demographics are transferred from the hospital’s PAS system to Dendrite using an automated HL7 link.

Following clinic attendance patients are assessed pre-operatively and the cardiac surgical SpR or consultant completes an A4 questionnaire containing the 17 fields of

(6)

6

the Logistic Euroscore. Each field of the Euroscore has the definition of the relevant field with it (as stated on www.euroscore.org as appropriate). The Audit Facilitator enters these data onto an Excel spreadsheet and the calculated Logistic Euroscores are entered into PATS.

Following cardiac surgery, the cardiac surgical SpR and cardiac perfusionists enter details pertaining to the patient journey through cardiac surgery. The secretaries use these data to generate a discharge summary.

Deaths are registered on the hospital PAS and PATS systems. There are also paper records of deaths held in the cardiac theatres, cardiac ITU, cardiac surgical ward and Patient Affairs Office that are routinely used for crosschecking.

3. Data Collection processes and Crosschecks Weekly

1. Check of all patients on the database versus the theatre list (including cross-checks for emergencies).

2. Check of CITU logbook for deaths and update PATS database.

3. Check cardiac surgery ward records for deaths and update PATS database. 4. Check completion of Euroscore data and records in Excel and PATS.

Monthly

1. Review of all deaths at monthly multidisciplinary audit meeting.

2. Re- run of monthly lists and check all missing entries have been entered and any errors dealt with.

3. Production of monthly reports of activity for the department, both as a stand alone report and check against contracts.

Annually

1. Check of all activity against PAS activity data for each procedure and each consultant.

2. Check of deaths for each procedure against each consultant with: -PAS electronic records

-PATS electronic records

(7)

7

operative details and relevant correspondence -Written records held by Patient Affairs Office.

3. Presentation of activity and outcomes at Commissioner’s audit – this is a joint meeting held between Morriston Hospital, Swansea, University Hospital, Cardiff and Health Commission, Wales.

4. Processes in place to ensure mortality data collection is complete?

This is partly covered by the database validation exercise described in section 3.

The primary source for monitoring data is from the separate but triangulated records of PAS, PATS and the Patient Affairs Office. The latter is thought to be an absolutely robust source of mortality data.

1. The PATS database links directly to the hospital’s PAS system and updates the surgical database held in PATS. Dates of death are automatically linked to the patient’s hospital number.

2. Deaths are sometimes reported to cardiac secretaries who are trained to update the PAS system.

3. Theatre, CITU, Cardiac surgical logbooks are scrutinised weekly.

4. Weekly reports of incomplete data are distributed to each consultant cardiothoracic surgeon for individual attention.

5. Review of surgical activity and outcomes as a routine part of the Cardiac Centre’s monthly multidisciplinary audit programme.

6. Each year, before the Commissioner’s audit, all operations are reviewed to determine survival or death. Results are crosschecked between, PATS, PAS, theatre, CITU & ward records and the Patient Affairs Office. Where necessary, individual patient records are recalled for closer scrutiny.

7. The department sends quarterly and annual returns to NCEPOD for first-time isolated CABG. Any queries are fed back from NCEPOD and records are clarified accordingly.

(8)

8

Activities and outcomes recorded on PAS are sent to the PEDW ( Patient Episode Database for Wales). The PEDW database contains records of the inpatient/daycase care received by all patients in NHS Wales hospitals and for some Welsh residents treated in the other home countries. The

database is used by the Welsh Assembly Government to provide aggregate figures to be used in their publications.

Currently, the department does not receive information from ONS to further validate the data.

Mortality data for 2004/5 have been checked internally on 4 separate occasions by 4 different groups of staff over the past 2 years.

5. What feedback mechanisms are in place to enable surgeons to validate their own data?

Each week the audit facilitator reviews the theatre and ward records to ensure that all operations have been recorded in the department’s PATS system. When this has not occurred, the consultant cardiac surgeon and their SpR are informed. This ensures that no operations are omitted from the database.

Each month, deaths and complications arising from surgery are reviewed in the multidisciplinary audit meeting. Respiratory, neuropsychological, renal and deep and superficial wound infections are reported to the multidisciplinary team (including the surgeons). In addition, re-opening for bleeding and blood usage (stored on the blood bank database) are reviewed routinely. In 1998, the monthly audit detected that there was a sudden increase in the number of deep wound infections that triggered an assessment of the air conditioning in the cardiac theatres. This was found be defective resulting in the closure of the theatre to permit new laminar flow facilities to be installed. Similarly, unexpected increases in blood usage during surgery highlighted at audit have led to changes in practice and routine use of cell saver devices.

(9)

9

Mortality data are supplied annually to each surgeon for review and validation. In the PATS database if a patient died following a second cardiac surgical operation within the same year, the output from the system is that there were 2 deaths (one from each operation). This requires manual correction.

Euroscore predicted outcomes for 2005/6 have been reviewed by the Department in one of its monthly meetings and compared with the latest national data available on the Healthcare Commission website. These were made available to the visiting team. For April 2005 – March 2006 actual survival for first time CABG is 99.2% and for all surgery 98.4%.

Audit of Euroscores

Since April 2005, a separate record of Euroscore data has been compiled which is stored in the PATS system. This has enabled surgical outcomes to be reviewed with risk-stratification available for all patients. A validation exercise of these data has recently been completed by Cardiology Nurse Practitioners.

40 medical records (5%) were chosen at random from the operations performed in 2005/6. A Euroscore was re-calculated by the Nurse Practitioners based upon the information contained within the medical records (and without knowledge of the original Euroscore calculation). These were matched against the original score. Where there was total agreement, notes were returned to medical records. Mr Pankaj Kumar reviewed all other notes.

Out of 680 data fields reviewed, there were 20 variations (3%) affecting 11 of the 17 categories. This resulted in 10 cases scoring lower than the original score and 2 scoring higher.

(10)

10

1. “Unstable angina” meaning “Rest angina requiring i.v. nitrates until arrival in anaesthetic room”

2. LV function. This was difficult to validate because the original echocardiograms were often held at referring hospitals.

3. Recent MI within the previous 90 days. The definition of MI is now less clear because of the interpretation of small rises in troponin.

4. Emergency – meaning operation before the beginning of next working day. 5. Pulmonary Hypertension meaning pressure >60mmHg (this was sometimes

measured intra-operatively rather than pre-operatively).

6. Review of Data

This review of data is based upon the 681patients operated upon at the Morriston Hospital, Swansea for the fiscal year 2004. The national comparison is data submitted on 34,301 patients operated upon in the same time frame.

Review of data shows that there are 0 patients reported alive on the database and dead on the ONS. There are 0 patients reported dead on the database and alive on ONS.

Table 2 shows data completeness for core variables compared with pooled national data.

Table 3 shows a data quality index of 62%.

Table 4 shows the incidence of risk factors compared to pooled national data and the Morriston mean EURO-score of 5.4% versus national EURO-score of 4.6%.

At the time of review, of the 681 patients for the year 2004 through CCAD Lotus notes logic checks were as shown.

(11)

11

Analysis of Data Submitted to CCAD- 2004

Table 1. Discrepancies between submitted and ONS tracked data

Number of patients Reported alive on database: dead on ONS Reported dead on database: alive on ONS 681 0 0

Overall deaths 23 (all deaths at last ONS X check) No. hospital recorded deaths 23

No. ONS recorded deaths 0

Table 2.% Data completeness for core variables: Hospital compared to pooled ‘national’ data

Variable Morriston ‘National’

Age 100 100 Sex 91.3 100 NHS number 99.4 89.3 Post Code 100 99.7 Procedure 100 99.8 Surgeon Identifier 100 90.3

Post operative morbidity 0 76

(12)

12

Table 3: % completeness of EuroSCORE fields compared to ‘national’ data

Risk factor Completeness Morriston

2004 Completeness national 2004 Age 100 100 Sex 91 99 PVD 89 95 Previous surgery 75 87 Renal failure 82 97 Active endocarditis 100 100 Iv Nitrates 0 91 LV dysfunction 75 96

Most recent infarct 13 93

Shock pre-op 0 86 Ventilated pre-op 0 87 IABP 96 86 IV inotropes 0 87 PA systolic 100 86 Urgency 86 99

Non coronary surgery 100 100

Surgery on aorta 100 100

Acute VSD 100 100

(13)

13

Table 4: incidence of risk factors compared to pooled national data

Risk factor Morriston incidence National incidence

Mean age 67.1 65.7

Male 69.6 72.6

Mean EuroSCORE 5.4 4.6

Fair LV 27.6 25.3

Poor LV 7 6.3

Logic checks for the Morriston data are as follows

1. Fatal Errors = 0 ( none expected - records rejected at import) 2. Serious Errors = 1266/2117 (these values need checking but the record is imported)

3. Minor Errors = 2117/2117

Current definitions are as below:

Fatal errors will prevent that record from being uploaded. The only errors which will prevent the record from being uploaded is the absence of a patient identifier or an operation type.

Serious errors will be flagged up a will require attention from the unit. The following problems will flag up a serious error

1. Lack of NHS number

2. Dates should be available for admission, operation and discharge 3. Lack of date order logic – i.e. the following should be in

(14)

14

4. There should be a surgeon identifier which should fit with a recognised list of GMC codes for the submitting unit

5. Discrepancies between submitted and ONS derived mortality (if the ONS derived mortality falls within the hospital stay)

6. Operation type should pass logic checks –

a. if the operation is a CABG, there should be some data that vessel or vessels have been grafted

b. If the operation type is a valve there should be data about which valve has undergone surgery

c. If the operation type is a valve and grafts there should be data on both vessel(s) grafted and valve undergoing surgery

Minor errors will flag up flaws in data, which may prompt further action

from the unit. Absence of data in any field which is required to produce a EuroSCORE for a particular record will flag up a minor error.

Issues relating to the completeness of the CCAD returns for 2004/5

a) The PATS database was established in 1997 when the Cardiac Centre opened at Morriston Hospital.

b) All the registries (echo, pacemaker, angioplasty, CCU etc) were supported by one whole-time officer (Mr Mark Arnold).

c) In 2000, Mark Arnold left the Trust and was not replaced. At this stage the PATS system was 3 years old, had not evolved and was now not supported. Hitherto data completeness had been in the region of 95%. Maintenance of the PATS system was passed on to Louise Williams, (Hospital Information Department) who was only able to support it part-time. Her primary concern was data for contracts and some analysis of PATS data. During this 2 year vacuum it fell upon Mr Youhana to collate the annual returns to the SCTS. No hospital support was available despite an awareness of the problem. This was clearly a frustrating time for the cardiac unit.

(15)

15

d) In December 2002, Adrian Morgan was appointed as audit facilitator (with Louise Williams continuing in her supporting part-time role from the Information Department). Adrian Morgan held this post until February 2004.

e) In June 2004, Andrea Beniamous was appointed to this role (30 hours per week)

f) In April 2006, Stephen Morris was appointed as Cardiac Information System Manager. He supports the IT infrastructure within the cardiac centre and is the designated “owner” of the PATS system within the Trust. Louise Williams no longer supports this role and Stephen Morris has all the administrative rights for the system.

g) The Trust has been seeking additional support for audit clerks from Health Commission Wales (and its predecessor the Specialised Health Service Commission for Wales [SHSCW]) since 2001 but without success.

h) Ownership of the clinical data resides with the relevant consultant for the patient.

i) The department has always been focussed on the objective measure of death following cardiac surgery. Data was supplied to the SCTS by its paper returns from 1998 until 2004. Outcomes have been scrutinised in detail within the Trust and shared with external partners. The Trust recognises that it has not been possible to comply with the wishes of CCAD and the Healthcare Commission in a timely manner owing to the lack of available resources to adapt its current systems.

j) The cardiac surgical database requires some modification to enable all relevant fields to be sent to CCAD. “Re-operation” causes the most problems. It is not possible because more than one question maps to this. It is not possible to report "Number of-Reoperation" to the CCAD dataset .

k) At present, some data fields (e.g. NHS number) are not mandated on the PAS system. This means that missing fields have to be entered manually by the audit facilitator. Potential “Serious Errors” relating to the data for 2004/5 have been corrected whenever reasonably possible. Surgeon but not GMC code are recorded on PATS but these data are easily correctable and available to map across to the CCAD.

(16)

16

Future developments

Considerable discussion has taken place within the Department regarding

the mechanisms for future data collection within Cardiac Surgery and Cardiology including the adoption of corresponding software and hardware. The current plan is that the UKC2 database will be switched off within the next few months. Data pertaining to Cardiac Surgery will be entered directly into the PATS/CCAD database field (rather than in to the UKC2 Registry and mapped across). The surgical discharge summaries use mail-merge documents using the UKC2 Registry and work is currently in progress to modify these for the new fields so that the Department can continue to issues computer generated discharge summaries. In the medium term it is planned to merge all database activities onto one system, namely the GE centricity database. This will reduce maintenance costs and reduce duplication of data entry because it is a relational database. Also because it is web-based accessibility is better and the restrictions on total and concurrent user licensers will have less impact.

With the appointment of a Cardiac Information Systems Manager there is a current update and development of the website both on the Swansea NHS Trust intranet and the internet. Activities and outcomes from the Cardiac Centre have been sent routinely to national audit bodies (who are publishing them more widely) and in future more information relating to Cardiology and Cardiac surgical outcomes will be presented on the Cardiac centre website. It is expected that interventional cardiology audit returns will be sent electronically this year to CCAD (rather than by paper returns to BCIS).

There is acceptance that the infrastructure required to deliver timely, accurate and validated data on cardiac surgical outcomes to external bodies requires significant additional resources. The Trust is confident that the mortality data for the unit over successive years is correct and has been thoroughly validated internally. It recognises that all the fields required to populate fully the CCAD database are not available but it believes that it has provided as much support as reasonably possible within its resources. The focus of the unit has been to record activity and mortality outcomes

(17)

17

accurately. It is asserted that it is not possible to complete and validate all the fields in the current CCAD database without compromising other cardiac databases with the existing resources.

Summary and Recommendations

The visiting team felt that Dr Ramsey and his colleagues had spent considerable time and preparation in providing the information required to inform this review. We felt that sufficient time was available for us to gain a full and complete picture of the mechanisms surrounding data collection, collation, validation, feedback and submission to CCAD.

It is clear that the commissioning of the Cardiac Centre at Morriston Hospital in 1997 was associated with the thoughtful establishment of a data base with support from one whole-time officer. Not surprisingly, however, with the burgeoning workload this one whole-time officer rapidly became overwhelmed and in 2000 when he left the Trust was, sadly, not replaced. There then followed a period of several years when there was a serious lack of personnel support for data management within the Cardiac Centre. During this time considerable efforts were made by clinicians alone (for which they should be congratulated) to at least ensure the accuracy of data pertaining to activity, case mix and mortality.

Since June 2004 with the appointment of Mrs Andrea Beniamous as an audit facilitator and subsequently April 2006 Stephen Morris as Cardiac Information System Manager the situation has been eased.

The visiting team was very impressed with the extent, detail and rigor with which a dynamic, cohesive, and forward looking team is addressing data management. There are, however, increasing demands for outcome reports required by various professional and statutory bodies. Also and quite reasonably the unit has aspirations to provide its population base with better accessibility to outcome data including a website which will require investment of time and money. Further evolution of the database within the Cardiac Centre as described above will also require necessary

(18)

18

support staff including ownership and validation and quality assurance systems in place. All of these issues are implicit to the delivery of modern day health care. Realisation of these ambitions will require sufficient investment .

In comparison to other units which have a similar workload and that have been visited during this data validation exercise the Cardiac Centre at Morriston Hospital is significantly under–resourced. To achieve a base level of audit infrastructure will , we believe , require a financial investment and in addition it would seem reasonable to recommend that two further data co-ordinators are appointed, one in Cardiology and one in Cardiac Surgery in order to facilitate cross-cover for holidays.

Appendix

Background and History of data collection and validation in Cardiac Surgery

National data collection in Adult Cardiac Surgery began in 1977 with the voluntary reporting of basic activity and outcome data on adult cardiac operations. Data were received from 100% of UK NHS and all the Republic of Ireland units and the aggregated national data was fed back to each unit to allow comparison of local results with national average. Since 1997 this included individual surgeons’ results for coronary artery surgery.

The National Adult Cardiac Surgical Database was established in 1994 and the current data set includes demographic, procedural and outcome data for each patient. The reasons for collecting more comprehensive data were firstly a growing public and political interest in cardiac surgical outcomes, secondly ignorance of changing patterns of patient populations with a professional and public misconception about that coronary artery surgery carried little or no risk. Thirdly in North America the release of crude mortality data on Medicare patients in the late 1980s with no risk adjustment for patients’ specific risk factors or co-morbidity caused considerable concern within the cardio-thoracic surgical community.

(19)

19

In the early 1990s the development of the internal market focussed attention on the purchaser/provider split in healthcare provision. It became clear that the success of the new healthcare market depended on an accurate understanding of the nature of the patient population and the

availability of comprehensive data collection for understanding severity of the illness, resource allocation and outcome analysis.

Further important developments in this “data collection journey” have been firstly the introduction of an agreed data set for the national database, secondly the public disclosure of surgeon’s specific outcome data in New York, and thirdly the report of the public enquiry into children’s heart surgery at Bristol Royal Infirmary. All directed attention towards clinical governance, and, in December 1997 there was an extraordinary general meeting held at the Royal College of Surgeons.

This concluded that there was “ a need for quality assurance driven by the change in public perception of doctors and their accountability and the public’s wish for more detailed information about doctors’ activity”

The collection and collation of data from the National Adult Surgical Database has recently resulted in a 5th report (2003) which documents the nature of contemporary cardiac surgery practice in the UK and Ireland. This is a considerable task which has been largely undertaken by one individual, Professor Sir Bruce Keogh, and the success and future of this project is now seen to rest with direct submission of data from individual cardiac surgical units to the central cardiac audit database (CCAD).

As important as the burgeoning momentum for outcomes of cardiac surgical procedures, there has been a growing concern regarding the nature and quality of data, which is used for outcome analysis. It is this, which in 2001 led to the introduction of the Society of Cardiothoracic Surgeons Quality Accreditation Programme whose mission statement was to “recognise and reward good quality monitoring schemes in adult cardiac surgical units”. This meant that an adult cardiac surgical unit and its individual consultants had systems in place for knowing its activity, case mix and outcomes, and had mechanisms in place for validating and verifying the data.

(20)

20

(21)

21