Research Article
Performance of simplified acute physiology score 3 admission score
as a predictor of ICU mortality in a tertiary care hospital
of rural Telangana, India
B. Balaji*, A. Bikshapathi Rao, V. Suresh Kumar, Sammaiah
INTRODUCTION
A critical care program or unit can be assessed using a variety of severity scoring systems or models that allow us to predict outcome and prognosis, guide the clinical decision making process, monitor and assess new therapies, compare care between different centers, standardize medical research and perform cost-benefit analysis with regard to resource utilization. While not specifically designed for individual patient care, scoring systems may guide (but will not replace) clinical decision
making regarding withdrawal of treatment and/or futility of continued aggressive care.1,2
However, reliability of a severity score reportedly deteriorates when applied to different populations, probably due to case mix, the level and quality of care, and the development of new treatment options changing overall patient outcomes.2 Application of a severity scoring system in the intensive care unit with different case mixes raises issues of the system's reliability and validity.1 ICU scores divided into outcome prediction
ABSTRACT
Background: This study was aimed to assess the performance of Simplified Acute Physiology Score 3 (SAPS3) as a predictor of Intensive Care Unit (ICU) mortality in critically ill patients of different case mixes admitted to an intensive care unit.
Methods: This study was performed from 1st August 2014 to 31st July 2015, in ICU of Govt. tertiary hospital in Rural Telangana. Predicted ICU mortality was calculated using SAPS3 global model. Observed versus predicted mortality rates were compared. The discrimination and calibration characteristics of the SAPS3 system to predict ICU mortality were assessed.
Results: A total of 491 patients were included. The majority (370, 75.3%) of the cases included in study were medical cases, with Cerebro-vascular accidents (150, 33.4%) and Shock, all types (96, 19.5%) as the most frequent primary diagnoses. Mean age of patients was 57.2 years with Males (296, 60.3%) predominance. Observed ICU mortality is 140 (28.5%) and SAPS 3 predicted mortality [(%), mean±SD - 41.4±14.80]. The discriminative power of the SAPS 3 model was good for the whole population (AUROC = 0.81, 0.77-0.83. Calibration was seen with Hosmer-Lemeshow goodness of fit.
Conclusions: The global SAPS 3 prediction model showed Good discrimination and Fair or satisfactory calibration in predicting mortality in our intensive care unit. Different levels of discrimination and calibration across the different subgroups analyzed suggest that overall ICU performance is affected by case mix variations. It is recommended that this model be tested in other centers and that a consolidated database be formed.
Keywords: SAPS3, Intensive care unit, Mortality, Predicted mortality, Discrimination, Calibration Department of General Medicine, Kakatiya Medical College, Warangal, Telangana, India
Received: 06 June 2016
Accepted: 02 July 2016
*Correspondence:
Dr. B. Balaji,
E-mail: resdoc555@gmail.com
Copyright: © the author(s), publisher and licensee Medip Academy. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License, which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
scores - based on disease severity on admission (e.g. acute physiology and chronic health evaluation (APACHE), simplified acute physiology score (SAPS), mortality probability model (MPM)), organ dysfunction scores – assess the presence and severity of organ dysfunction (e.g. multiple organ dysfunction score (MODS), sequential organ failure assessment (SOFA)), Scores that assess nursing workload use (e.g. therapeutic intervention scoring system (TISS), nine equivalents of nursing manpower use score (NEMS)).
The simplified acute physiology (SAPS) 3 admission score is one of these models used to predict hospital mortality from admission data taken within the first hour of the patients' admissions (It includes 20 variables divided into 3 sub-scores related to patient characteristics prior to ICU admission, circumstances related to ICU admission and the degree of physiologic derangement within 1 hour before or after of ICU Admission.2 This is in contrast to all prior models that utilized a 24-hour time window.12
The total score can range from 0 to 217) from this score, global and region-specific equations for hospital mortality have been derived.3 The performance of this model has shown mixed results among different case mixes in different studies.3 Discrimination and calibration are two characteristics used to judge a scoring system. Severity scoring system reliability can be quantified in terms of calibration, which represents the level of accordance between observed and predicted probabilities of the outcome (describes how an instrument performs over a wide range of predicted mortalities.
An instrument would be highly calibrated if it were accurate at mortalities of 90%, 50% and 20%).4 This is derived from tests such as the Hosmer-Lemeshow goodness- of- fit test or the calibration belt.4,5,9 Discrimination, another essential quality, is quantified with measures such as sensitivity, Specificity, and more completely, the area under the receiver operating characteristic curve (AUROC) (refers to the accuracy of a given prediction e.g. if a scoring system predicts mortality of 90%, discrimination is perfect if the observed mortality is 90%).5,8
An AUROC of 0.5 indicates that the model does not predict better than chance. The discrimination of a prognostic model is considered perfect if AUROC= 1, good if AUROC >0.8, moderate if AUROC is between 0.6 and 0.8, and poor if AUROC <0.6.6 This study aims to assess the performance of the SAPS 3 in its ability to predict ICU mortality among critically ill patients of different case mixes admitted to our hospital.
METHODS
After ethical committee approval, This retrospective study was conducted in intensive care unit of Mahatma Gandhi Memorial hospital, an 20-beded medical and
surgical units serving adults (>14years) critically ill patients from all departments of the Institution. Data was collected from all ICU admissions from 1st August 2014 to 31st July 2015.
Inclusion criteria
Age >14years,
Both medical and surgical patients ,
At admission data available,
Only the 1st ICU admission of patients with multiple
ICU admissions during single hospital stay
Exclusion criteria
Incomplete data availability,
patients expired with in 1st hour of admission,
Post resuscitation patients.
Data is collected which included all components of the SAPS 3 score described by the original SAPS 3.7 All data was collected by medical residents to formulate the predicted mortality rates based on the SAPS 3 severity score, using SAPS3 Integrated score database tool (Microsoft office Access database, MBD)
Statistical analysis was done MedCalc Software 12.3.0 (MedCalc Software, Belgium) was used to perform the statistical analysis. A p value of less than 0.05 was set as statistically significant.
Discrimination was determined by analysis of the area under a receiver operating characteristic curve (AUROC) using the method described by Hanley and associates.8
Calibration was assessed using the Hosmer- Lemeshow goodness-of-fit statistics in the analysis, lower Hosmer-Lemeshow Ĉ values and a p value of more than 0.05 would indicate a good fit of the model.
RESULTS
Baseline characteristics of patients
There were 577 distinct admissions during the study period. A total 491 (85.09%) of patients were included in this study. 86 patients were excluded, 2 (0.02%) for age <14 years, 39 (45.3%) for readmission to the ICU during the same hospital admission, and 45 (52.3) for incomplete SAPS 3 data.
Calibration of SAPS 3 scores
The SAPS 3 model exhibited satisfactory calibration for the entire population (Ĉ=12.2, p=0.24, Figure 1). Subgroup analysis showed that the SAPS 3 model showed good calibration for age >50 years (Ĉ=5.7, p=0.68), all medical conditions treated as a group (Ĉ= 8.4, p=0.31), and all surgical conditions treated as a whole (Ĉ=4.8, p=0.48), as shown in Table 1.
Poor calibration was noted with patients aged” 50 years (Ĉ= 17.1, p = 0.04) and patients admitted with Pneumonia, of all types (Ĉ= 19.1, p = 0.01).
Figure 1: Calibration curve for SAPS 3 model comparing actual and predicted ICU mortality. Hosmer and Lemeshow "Goodness of fit" test Showed
satisfactory calibration (Ň=12.2, p=0.24).
Table 1: Calibration of SAPS 3 for all patients and various sub-groups.
Group Number Actual mortality number and % Predicted mortality % Goodness of fit
All 491 140 (28.5%) 41.4±10.8 Ĉ=12.2, p=0.24
Medical 370 111 (30%) 42.2±19.1 Ĉ=8.4, p=0.31
Cerebrovascular accidents 150 40 (26.6%) 53.6±20.4 Ĉ=8.4, p=0.31
Shock, all types 96 28 (29.1%) 60.1±17.1 Ĉ=9.3, p=0.52
Acute respiratory failure 36 11 (30.5%) 58.8±21.8 Ĉ=13.8, p=0.09
Septic shock 58 14 (24.1%) 41.6±14.1 Ĉ=6.4, p=0.61
Pneumonia. All types 57 11 (19.3%) 31.1±11.1 Ĉ=19.1, p=0.01
Encephalopathy, all types 82 14 (17.07%) 56.1±25.1 Ĉ=17.2, p=0.04
Heart failure 43 6 (13.9%)
Surgical 121 29 (23.9%) 33.7±12.6 Ĉ=4.8, p=0.48
Intestinal obstruction 41 9 (21.9%)
Peritonitis, all types 50 20 (40%) 22±14.8 Ĉ=4.8, p=0.48
Comparison of discrimination
The discriminative power of the SAPS 3 model was good for the whole population (AUROC=0.81, 0.77-0.83, Figure 2) and exhibited moderate discrimination for
medical cases (AUROC = 0.79, 0.77-0.80) with different discriminatory patterns noted, from poor for Pneumonia, of all causes (AUROC = 0.58, 0.39-M.74) to good for groups like Septic Shock (AUROC = 0.93, 0.74-1.0) and patients with Acute respiratory failure (AUROC=0.88, 0.74-0.91).
Table 2: Discrimination of SAPS3 for all patients and various sub-groups.
GROUP AUROC (95% C.I.)
All 0.81 (0.77-0.83)
Age<50Y 0.77 (0.74-0.79)
Age>50Y 0.82 (0.80-0.84)
Medical 0.79 (0.77-0.80)
Cerobrovascular accidents 0.72 (0.69-0.76)
Shock, all types 0.79 (0.77-0.80)
Septic shock 0.93 (0.74-1.0)
Acute respiratory failure 0.88 (0.74-0.91)
Pneumonia, all types 0.58 (0.39-0.74)
Encephalopathy, all types 0.61 (0.33-0.84)
The model showed better discrimination for surgical cases, with good discriminatory power (AUROC=0.84, 0.81-0.88). Table 2 shows discrimination power of SAPS3 in different sub-groups.
Figure 2: Receiver operating curve for population as defined by SAPS 3.
DISCUSSION
The study evaluated the Validity of the SAPS 3 mortality prediction model when used in a local Government tertiary centre's intensive care unit. Itis important to validate the performance of SAPS3, prior to application to other Similar centres, for its use to make quality of care assessments.2,12
The model showed Satisfactory calibration in our study population, true for both medical and Surgical patients, this was Particularly true inpatients >50 years, Shock in particular for Septic shock, But model's calibration
suffered heterogenisity in different subgroups,
Particularly in patients with Pneumonia and
Encephalopathy where it showed Poor Calibration.
The study showed that the SAPS 3 global model had good discriminative power, which was comparable with other studies for whole population and also hold the same for both medical and surgical patient groups, model's discrimination remained relatively homogenous in
different subgroups except in pneumonias and
encephalopathy, where model's discrimination
performance is poor.2,3,13,14
Evidence of different levels of calibration and discrimination on subgroup analysis supports that the global SAPS 3 model was indeed affected by differences in case mix.
There are several limitations to our study. First, ,this is a retrospective study; another issue is that the study derived its data from a single centre ICU, limiting the sample size as well as the case mix included in the study compared to the original SAPS 3 cohort and affecting generalizability. Our subgroup analyses had smaller samples that make the statistical analysis less robust. The last limitation is one
inherent to the Hosmer-Lemeshow goodness-of-fit test, which depends on the sample size.
CONCLUSION
The global SAPS 3 prediction model showed Good discrimination and Fair or satisfactory calibration in predicting mortality in our intensive care unit. Different levels of discrimination and calibration across the different subgroups analyzed suggest that overall ICU performance is affected by case mix variations.
It is recommended that this model be tested in other centres and that a consolidated database be formed.
Funding: No funding sources Conflict of interest: None declared
Ethical approval: The study was approved by the institutional ethics committee
REFERENCES
1. Sakr Y, Krauss C, Amaral ACKB, Réa-Neto A,
Specht M, Reinhart K, et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth. 2008,101:793- 803.
2. Khwannimit B, Bhurayanontachai R. The
performance and customization of SAPS 3 admission score in a Thai medical intensive care unit. Intensive Care Med 2010,36:342-6.
3. Ledoux D, Canivet JL, Preiser JC, Lefranq J, Damas
P. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med. 2008,34:1873-7.
4. Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G. Calibration belt for quality-of-care assessment based ondichotomous outcomes. PLoS One. 2011;6(2). doi.org/10.1371/journal.pone.0016110.
5. Steyerberg EW, Vickers AJ, Cook NR, Gerds T,
Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128-38.
6. Afessa B, Tefferi A, Dunn WF, Litzow MR, Peters
SG. Intensive care unit support and acute physiology and chronic health evaluation III performance in haematopoietic stem cell transplant recipients. Crit Care Med. 2003;31:1715-21.
7. Moreno RP, Metnitz PG, Almeida E, Jordan B,
Bauer P, Campos RA, et al. SAPS 3 from evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission.Intensive Care Med. 2005;31:1345 -55.
8. Hanley JA, McNeil BJ: The meaning and use of the
9. Lemeshow S, Hosmer DW Jr. A review of goodness of fit statistics for use in the development of logistic
regression models. Am J Epidemiol.
1982;115(1):92-106.
