reports with 3 reviewers
Step 1 (2013)
Literature review and Investigation Quality Evaluation Tool 1 (IQET 1)
Step 3 (2014) IQET 1 Applied to 2013 reports to develop IQET 2 (n=62) Step 5 (2015) Developed IQET 3 Step 4 (2015) Qualitative inter-rater reliability testing of IQET 2 Random sample of 4 investigations with 3 reviewers Step 6 (2015/16) Applied IQET 3 to 2014 reports (n=45) Step 7 (2016/17) Quantitative inter- rater reliability testing of IQET 3 Learning from this
to inform IQET 4
Figure 4.1: Showing the application of the IQETas per steps 3 and 6 4.3.1. Sample
The National Director for Quality and Patient Safety (QPS) requested that reports of investigations of incidents that resulted in death and serious harm and which were completed between the 1st of January and 31st of December 2013 would be forwarded to the National Incident Management Team (NIMT) for review.
65 reports of investigations of serious incidents which were completed in 2013 were returned to the NIMT. Four were excluded (See footnote31).
→ 31One was a review of a service at a specific site and was not an investigation of a specific individual incident that resulted in death or serious harm.
→ One was a report of a Look Back Review in relation to a number of incidents; and not an individual investigation of a specific incident that resulted in death or serious harm.
→ One was a draft report of a healthcare record review related to an incident that resulted in serious harm and not a final report of a systems analysis investigation of that incident. This case was followed up to determine whether there was a final report of a full systems analysis investigation conducted in relation this incident and it was confirmed that a full investigation did not take place as safety recommendations were implemented based on the draft report of the healthcare record review.
133
Subsequently, the National Director for Quality Assurance and Verification (QAVD) wrote to the National Directors of the six Divisions of the HSE and requested that reports of investigations of incidents that resulted in death and serious harm and which were completed between the 1st of January and 31st of December 2014 would be attached to the Quality and Patient Safety Incident Information Management Systems (QPS IIMS) for review.
52 reports completed in 2014 were attached to the QPS IIMS. Seven were excluded (See footnote32). See appendix 9 for further details of investigation reports that fell within this study.
4.3.2. Application of IQET to evaluate the 2013 & 2014 reports
IQET 1 was used to evaluate the quality of the 62 investigation reports completed in 2013. It was then enhanced to create IQET 2 based on learning from the process of evaluating the 2013 investigation reports in Study 1 as described in Chapter 3. Qualitative inter-rater reliability testing of IQET 2 was conducted following evaluation of the 2013 investigation reports culminating in the development of IQET 3. This was used to evaluate the 45 investigation reports completed in 2014. The main differences of IQET 1, 2, and 3 are outlined in Chapter 3 and the tools can be found in appendices 3, 4, and 5. The data was inputted on an SPSS data base.
→ One was is a letter from a Consultant involved in delivering care in relation to the incident - to the Risk Office at the hospital; and not an individual investigation of a specific incident that resulted in death or serious harm.
32
Of the 52 investigation reports that were attached to the QPS IIMS, one was an investigation of an incident that was completed in 2013. This investigation was not one that was returned with the 2013 investigation reports already received by the NIMLT. This investigation was reviewed using IQET 3. Of the remaining 51 investigation reports that were returned and which were completed in 2014, six were excluded from this study for reasons outlined as follows:
→ Two were investigations of incidents that were completed in 2015. These investigation reports were not included within this study.
→ One submission was a report of a Look Back Review in relation to a number of incidents; and not an individual investigation of a specific incident that resulted in death or serious harm and was excluded. → One was a draft report of the investigation of a serious incident. Follow up occurred with the relevant division a number of times to determine whether there was a final report of the investigation of this incident. A final report was not furnished by the time the review and analysis took place. Consequently, the draft report was not included within this study.
→ One report was not a systems analysis of an incident, but was rather an investigation of an allegation of abuse of a vulnerable adult under the HSE’s Vulnerable Adults Policy (2014), and so this was not included within this study.
→ One report attached to the QPS IIMS was password protected. Follow up was made with the relevant division seeking the password. A password was not furnished by the time the review and analysis took place so this report could not be included in this study.
134
The structure and content of the 2013 investigation reports varied
significantly. This posed challenges for evaluation. The average amount of time to evaluate and enter data from the 2013 investigation reports was 74.45 minutes; with a minimum duration of eight minutes, and a maximum duration of 204 minutes.
2014 reports were less variable but took more time to analyses because more data was gathered from them.
The average amount of time for 2014 reports was 102.32 minutes; with a minimum duration of 47 minutes and a maximum duration of 209 minutes.
4.3.3. Method for Investigation Quality Scoring (IQS) 2014 reports
Study 1 (Chapter 3) details the Investigation Quality Scoring (IQS) process which was developed to score aspects of the 2014 investigation reports which were considered important for (i) “leaving an analytic trace for investigation findings” (Dekker, 2006), and for (ii) enabling analysis for generalizable learning from groups of investigation reports. Tables 3.3 and 3.4 in Study 1 in Chapter 3 show the criteria that underpinned decisions about which sections of the evaluation tool were used for quality scoring purposes and which sections were not.
4.3.4. Method for determining whether causal factors were adequately supported by evidence
After reading and evaluating the quality of the chronology and before reading and evaluating the Key Causal Factors (KCFs), KCFs were deduced from the evidence presented within the chronology. The KCFs deduced in this manner where then compared with the KCFs that were identified within the actual investigation report by the original investigators. Different and/or additional KCFs were deduced for some investigation reports from this process and these are illustrated via examples of cases in the results section of this chapter. After reading and evaluating the quality of the analysis of the KCFs to identify contributory factors (CFs), the CFs were considered to determine whether they were adequately supported by the evidence within the investigation report. This included considering whether different and/or other CFs were
135
identifiable from the evidence presented within the investigation report. These are illustrated with examples of cases in a later section of this chapter.
4.3.5. Analysis of the quality of investigation reports
Data related to the quality of investigation reports for both the 2013 and the 2014 investigation reports were analysed together where possible. The data were analysed separately where joint analysis was not possible due to differences in the version of the IQET used. The results of this analysis are outlined in section 4.4 below.
4.4. Results
This results sections starts by summarising the quantitative analysis (i.e. analysis of questions with yes/no answers), followed by outlining the results of the Investigation Quality Scoring (IQS) process. Finally, the detailed findings of the evaluation of investigation quality are outlined including using cases of specific investigations to illustrate these findings.
4.4.1. Summary of quantitative results
Table 4.1 below shows the results of the analysis of the questions in the evaluation that elicited yes/no answers.
Note re table 4.1 below:
* and the green colour indicates that this question was used for IQS purposes ¥ Indicates that information from both 2013 and 2014 investigation was included in this analysis Question Frequency Valid Percent33 N/A Not possi ble to tell Total Yes No Yes No
2.1. Does the report state the date the incident occurred? 102 4 96.2 3.8 1 - 106¥
2.3. Does the report state the date the investigation team was established? 4 41 8.9 91.1 - - 45 2.6. Does the report state the date the investigation commenced? 54 52 50.5 49.5 - - 107¥ 2.9. Does the report state the date the investigation was completed? 93 14 86.9 13.1 - - 107¥
2.14. Did the investigation achieve the target timelines? 7 29 19.4 80.6 6 3 36
2.15. If no, is a reason for the delay given in the investigation report? 7 22 24.1 75.9 - - 29
4.1. Does the report state the names of the investigators? 40 5 88.9 11.1 - - 45
4.2. Does the investigation report state the number of investigators? 101 6 94.4 5.6 - - 107¥
4.3. If no, is it possible to deduce the number of investigators? 0 6 0 100 - - 6
4.5. Does the investigation report state the job title of the investigators? 40 5 88.9 11.1 - - 45 4.8. Does the investigation report state whether the investigators were
internal and/or external34 11 34 24.4 75.6 - - 45
33
Valid percent" is the percent when not applicable cases/missing data are excluded from the calculations
136
4.9. If not, is it possible to deduce whether the investigators were internal
and/or external? 32 2 94.1 5.9 - - 34
4.11. Does the investigation report state that the investigators WERE NOT
responsible for the service within which the incident occurred? 1 44 2.3 97.7 - - 45 4.12. If not, is it possible to deduce from the report that the investigators
WERE NOT responsible for the service within which the incident occurred?
8 36 18.2 81.8 - - 44
4.14. Does the investigation report state that the investigators WERE NOT
directly involved in the incident being investigated? 1 44 2.3 97.7 - - 45
5.1. Does the investigation report include the terms of reference (ToR) for
the investigation? 30 15 66.7 33.3 - - 45
5.2. If yes, does the ToR that is included adhere fully to the template ToR
from the HSE Guidelines? 11 19 36.7 63.3 - - 30
5.3. Does the report include a summary of the method used to conduct the
investigation? 34 11 75.6 24.4 - - 45
5.4. Does the summary of the investigation report state that a systems
methodology was used for this investigation? 34 11 75.6 24.4 - - 45
6.1. Does the report include a summary of data collected and reviewed?* 34 11 75.6 24.4 - - 45 6.2. Does the report reflect that the relevant records/healthcare records
were collected and reviewed?* 37 8 82.2 17.8 - - 45
6.3. Does the report reflect that the relevant local/national/international
PPPGs were collected and reviewed?* 15 29 34.1 65.1 1 - 44
6.4. Does the report reflect a search for and review of all relevant
literature?* 10 35 22.2 77.8 - - 45
6.5. Does the report state the number of individual interviewees that were
interviewed?* 47 60 43.9 56.1 - - 107¥
6.7. Does the report reflect that individual interviews were conducted with individuals that observed the incident and/or had information pertinent to the investigation?*
47 60 43.9 56.1 - - 107¥
6.8. Does the report reflect that those harmed/affected were given an
opportunity to participate in an interview?* 16 28 36.4 63.6 1 - 44
7.2. Was the scope in time reasonable?35* 79 25 76.0 24.0 3 - 104¥
7.3. Are events place in chronological order?* 40 5 88.9 11.1 - - 45
7.4. Is the exact date and time of each event specified where this is known?* 37 7 84.1 15.9 1 - 44 7.5. Where exact date and time is not known, is there evidence that
investigators tried to determine this as well as possible from information available and from interviews in order to be able to place the event in chronological order including noting that the time is an approximation where relevant?*
10 8 55.6 44.4 27 - 18
7.6. Is the chronology sufficiently detailed including details about reasons for, results of, and significance of tests/examinations and
interventions?*
6 39 13.3 86.7 - - 45
7.7. Does the investigation report specify the individual that observed each
chronological event?* 22 19 53.7 46.3 4 - 41
7.8. Does the investigation report specify the source of the information
referred to in the chronology?* 20 21 48.8 51.2 4 - 41
8.1. Does the report refer to the definition of key causal factors in line with
HSE investigation guidelines?* 21 24 46.7 53.3 - - 45
8.2. Does the report identify KCFs (or state that none were identified)?* 39 6 86.7 13.3 - - 45 8.4. Are the KCFs adequately supported by evidence presented within the
investigation report?* 29 78 27.1 72.9 - - 107
9.1. Does the report refer to the definition of IFs in line with HSE
investigation guidelines?* 14 31 31.1 68.9 - - 45
9.2. Does the report identify IFs or state that no IFs were identified?* 24 21 53.3 46.7 - - 45 9.3. Are IFs adequately supported by evidence within the investigation
report?* 12 32 27.3 72.7 1 - 44
10.1. Does the report reflect that each individual KCF was analysed
separately to identify CFs?* 11 16 40.7 59.3 18 - 27
10.x. Does the report identify CFs or state that no CFs were identified?* 25 4 86.2 13.8 16 - 29 10.2. Does the report clearly describe the factor type and the CF in line with 6 22 21.4 78.6 17 - 48
34The IQET tool clarified that external investigators were investigators nominated to a
HSE investigation by an external agency such as the Forum of Irish Post-graduate Medical Training Bodies.
35 HSE Investigation Guidelines (2012) stated that the scope in time of an investigation
should be the shortest sufficient period of time necessary to be included to ensure the investigation proposes will be achieved.
137
the framework of CFs?*
10.3. In addition to the information referred to in 10.2. above, does the
report include further details of each CF?* 16 9 64.0 36.0 20 - 24
10.4. Are the CFs adequately supported by the evidence presented within
the investigation report?* 2 24 7.7 92.3 19 - 45
11.1. Does the report make recommendations?* 40 5 88.9 11.1 - - 45
11.2. Are the recommendations inked to CFs?* 32 8 80.0 20.0 5 - 40
11.3. Are recommendations SMART36?* 1 39 2.5 97.5 5 - 40
11.4. Is there reference to the hierarchy of preferred control measures for
developing recommendations?* 9 31 22.5 77.5 5 - 40
11.5. Is there evidence that the hierarchy of preferred control measures was
used in the development of recommendations?* 0 40 0.00
100.
00 5 - 40
12.1. Does the report identify CFs that seem as though they may occur
elsewhere? 34 5 82.1 17.9 6 - 39
12.2. If yes, does the report state that the CFs may occur elsewhere?* 2 32 5.9 94.1 - - 32 12.3. Does the report identify recommendations that seem as though they
may be applicable elsewhere? 38 7 84.4 15.6 - - 45
12.4. If yes, does the report state that the recommendations are applicable
elsewhere?* 2 36 5.3 94.7 - - 38
12.5. Also, if yes, does the report state that nationally applicable
recommendations should be communicated to the relevant National Director(s) for implementation nationally within the body of the report?*
0 38 0.0 100.
0 - - 38
13.1. Does the report state that legal review of the report was undertaken? - - - - 37 8 45 14.1. Is the investigation report anonymised in terms of the person harmed? 42 2 94.5 5.5 1 - 44 14.2. Is the investigation report anonymised in terms of the staff involved? 41 4 91.1 8.9 - - 45
14.3. Is the investigation report anonymised in terms of age? 14 30 31.8 68.2 1 - 44
14.4. Is the investigation report anonymised in terms of gender? 2 42 4.5 95.5 1 - 44
14.5. Is the investigation report anonymised in terms of the site where the
incident occurred? 26 19 57.8 42.2 - - 45
15.1. Does the report consistently use everyday words and avoid jargon?* 19 26 42.2 57.8 - - 45 15.2. Does the report use terms consistently (i.e. heart attack v coronary
event)?* 42 3 93.3 6.7 - - 45
15.3. For abbreviations, are the terms always presented in full text with the
abbreviation beside them the first time they appear in the report?* 21 24 46.7 53.3 - - 45 15.4. For technical terms, are they always explained; and/or is there a
glossary of terms including the source of definitions references?* 15 35 33.3 66.7 - - 45
16.1. Is an apology appropriate?37 25 17 59.5 40.5 3 - 42
16.2. Where an apology is appropriate is an apology given/referred to in the
report? 6 19 24.0 76.0 - - 25
17.1. Does the report reflect that those harmed/affected were given an
opportunity to contribute to the investigation? 16 28 36.4 63.6 1 - 44
17.2. If yes, did they contribute? 12 3 80.0 20.0 - 1 15
17.3. If yes, was their contribution to the investigation acknowledged? 8 4 66.7 33.3 - - 12 17.4. Was the contribution of staff to the investigation acknowledged? 18 17 51.4 48.6 10 - 35 18.1. Does the investigation report refer to the application of fair
procedures?* 17 27 38.6 61.4 1 - 44
18.2. Does the body of the investigation report reflect that interviewees received a draft report (or relevant excerpts) for factual accuracy checking purposes?*
30 46 39.5 60.5 31 - 76¥
18.3. Does the report reflect that those affected received a draft chronology and any other relevant excerpts for factual accuracy checking purposes?*
7 36 16.3 83.7 2 - 43
18.4. Does the report reflect that the draft was modified based on feedback to enhance factual accuracy and to enable the report to achieve its objectives?*
15 30 33.3 66.7 - - 45
19.1. Is the report written impartially (i.e. there is no evidence of bias such as hindsight bias, outcome bias or any other bias in the investigation report)?
18 27 40.0 60.0 - - 45
Table 4.1. Results of analysis of investigation reports related to yes/no questions on IQET
36
HSE Investigation Guidelines (2016) define SMART recommendations as Specific, Measurable, Achievable, Realistic and state a Timeframe for completion (SMART)
37
HSE Investigation Guidelines (2012) stated that it was appropriate for an apology to be given to the individual(s) affected if key causal factors are identified in the report.
138
4.4.2. Results from IQS process
Using the IQS process as outlined in Study 1 in Chapter 3, IQSs for the investigation reports completed in 2014 ranged from 13.79 - 78.13 with the average score being 45.17%.
Figure 4.2 below shows the percentage of the 2014 investigation reports that demonstrated satisfactory evidence in relation to each of the 41 elements of the IQS process under the nine main investigation quality themes.
140
141
As shown in figure 4.2, 70% or more of the investigation reports reflected satisfactory evidence of:
→ Summary of data collected and reviewed
→ Relevant records/healthcare records were collected and reviewed
→ Investigation reports reflected that a reasonable scope in time was covered by the investigation, with events placed in chronological order within the chronology, and with exact date and time of events given where this was known
→ KCFs and CFs were identifiedor a statement that none were identified
→ Recommendations were made when appropriate, and these were linked to CFs
→ Terms were used consistently (i.e. heart attach v coronary event) Conversely, 70% or more did not reflect satisfactory evidence of:
→ Search and review of the relevant literature
→ Sufficient details within the chronology including reasons for, results of, and significance of tests, examinations and interventions.
→ Adequate evidence to support identified KCFs, CF, and IFs or to support a finding that no KCFs, CFs, and IFs were identifiable
→ Clear description of the factor type and contributory factor in line with the framework of contributory factors in HSE guidelines.
→ Demonstrating that recommendations were SMART; and that the “Hierarchy of preferred controls” was referenced and used in developing recommendations.
→ That those affected received a draft report for factual accuracy checking.
→ Generalizability of investigation reports was a problem with only 5.9% of applicable investigation reports stating that CFs may occur elsewhere, only 5.3% of applicable investigation reports stating that recommendations may be applicable elsewhere, and none of the applicable investigation reports stated that nationally applicable recommendations should be
142
communicated to the relevant National Director(s) for implementation nationally.
4.4.3. Detailed results
4.4.3.1. Investigation timeliness
As shown in table 4.2 below, information about the timeframe from
commencing to completing investigation reports was available for 35% (n=38) of investigation reports 38.
Months from date investigation commenced to date completed
2013 Investigations 2014 Investigations
Data available 18 Data available 20
Data missing 44 Data missing 25
Mean 10.15 5.75 Median 6.00 5.00 Mode 6.00 1.00a Std Deviation 8.57 3.52 Variance 73.74 12.38 Rage 33.50 12.50 Minimum 1.75 1.00 Maximum 35.25 13.50
a. Multiple modes exist. The smallest value is shown
Table 4.2: Table comparing the timeframes for completion of investigation reports in 2013 and 2014.
The most common reason for delay cited in investigation reports related to issues of availability/competing priorities of investigation team members. Appendix 10 includes details of the reason for delays in completing investigations cited within investigation reports.
As shown in table 4.1 (i.e. 2.14), 19.4% (n=7) of applicable39 investigations achieved the target timeline of being completed within four months40.
38
Specifically, this information was available for 16.82% (n=18) of 2013 reports; and for 44.44% (n=20) of 2014 reports).
39
In relation to investigations where timelines were categorised as “Not applicable”, all six were preliminary reports only and not a full systems analysis investigation (Case refs 84, 85, 86, 90, 91 and 101). The target timeline related to systems analysis
143
4.4.3.2. Existence of the “Summary of the Methodology”
HSE Investigation Guidelines (Health Service Executive, 2012) required that investigation reports included a methodology section. This, in-turn needed to include a summary of the systems analysis methodology used.