Summary Statistics: Table
Section 5: Discussion Results Sections 4.1-4.8 (subsections: 1)
All cases in the revised data set were accepted for analysis. The dependent variable was encoded such that death had a value of zero and survival had a value of one. The probabilities generated were for survival rather than death.
Results Sections 4.1-4.8 (subsections: .2 - .5)
The backward likelihood ratio method was used as the means for selecting the regressors. The likelihood is the probability of the observed results given the parameter estimates. Since this is a small number less than 1 the likelihood is usually expressed as -2 times the log of the likelihood. A good model therefore has a high likelihood and therefore a small -2LL. The -2LL for the constant in all the cases was 5363.149 (subsections: .2 ) The -2LL for the whole model is given in subsections .4 and the model chi-square which is the difference between the -2LL for the constant and the whole model is reported in subsections: .5. The degrees of freedom for the model chi-square is the difference between the number of parameters in the two models. The chi-squared value for the entry labelled Block (subsections: .4) is the change in the -2LL between successive entry blocks during model building. Because the variables were entered as a single block for each of the nine models developed, the block chi-square is the same as the model chi- square. The entry labelled Step in subsections: .4 is the change in the -2LL between successive steps in model building. In this study only two models were considered at each stage (constant and a single block) therefore the Step chi-square is the same as the block chi-square. The SPSS program has two other methods of model
selection; the Wald statistic and the Conditional statistic. The conditional statistic option also uses the likelihood ratio test but is computationally less intensive than the likelihood ratio option in SPSS. The Wald statistic in SPSS is calculated by dividing the coefficient by its standard error and has an approximate chi- squared distribution with one degree of freedom. Unfortunately when the coefficient becomes large the estimated standard error becomes too large. This produces a Wald statistic which is too small resulting in failure to reject a predictor (i.e. failure to reject the null hypothesis that the value of the predictor coefficient is zero). Harrell (2002) considers the likelihood ratio test to be the preferred method for selecting predictor variables.
Results Sections 4.1-4.9 (subsections .4)
The worth of a model in linear regression is determined using (see footnote*). This statistic is a function of the Y residuals i.e. it measures the difference between the predicted value (Ÿ) and the actual value. is therefore a measure of the explained variation of Y. In logistic regression R^ should not be used when there are only two possible values for Y. The statistic may under predict the worth of the model even if the model fits nearly perfectly (Ryan, 1997). The Cox and Snell test (1989) is a measure of explained variation in the dependent variable which utilises the ratio of the likelihood for the null model by the likelihood for the model with the regressors. Unfortunately the maximum value that R^ can achieve is .75 using the Cox and Snell method. Nagelkerke (1991) proposed a modification so that the value of 1 could be achieved.
*R^ = y i Ÿ - Ÿ )
I ( Ÿ - Y )
Ÿ = mean value o f the dependent variable.
Ÿ = predicted value for the dependent variable using the model coefficient. Y = actual value for the dependent variable for a given value of the independent variable.
From table 3 the model with the greatest proportion of explained variation is model 5 (HCISS + coded GCS + coded SBP + coded RR + Age). The model with the smallest proportion of explained variation is model 1 (HCISS only). All models with a physiological variable were found to have a greater proportion of explained variation than model 1 (HCISS only).
Results Sections 4.1-4.9 (subsections .6 and .7)
Model validation was performed using internal validation of the fitted model rather than by external validation on another data set or cross-validation. Although this is a less stringent test than external validation it still provides some useful information. Subsequent chapters of this thesis will address in detail the different methods of validating a prognostic model.
The Hosmer Lemeshow statistic in SPSS is calculated using an algorithm approach with variable cell numbers rather than using a fixed cell number method. The SPSS algorithm method divides the cases into roughly 10 approximately equal groups based upon their predicted probabilities. Cases with the same combination of values for the predictor variables are kept in the same group. The different methods of calculating the Hosmer Lemeshow statistic and the drawbacks of each method will be discussed in chapters 5 and 6.
Results Sections 4.1.9
The plot of log odds against HCISS (model 1) and RTS controlling for HCISS (model 6) both generated straight lines demonstrating that for this particular data set the logit model is correct for both models. This is in contrast to a recent study by Osier et al (2002) who found that the log odds for death was not linear when plotted against ISS but was better approximated when a squared term was added to the ISS variable. The study was performed on three data sets; a paediatric data set of 53,113 cases, a New Mexico data set of 3,142 and the Portland data set of 2,916 cases. The reason why the ISS model was not linear for the two adult data sets appears to be due to the fact that the odds (and therefore log odds) was calculated from the actual number of deaths and survivors for a given ISS value. The correct way of calculating the log odds is to use the probabilities generated by the fitted model (Harrell, 2002; Chapter 10). The logistic model should be linear when the log odds are plotted against X p ( X is the predictor{s} variable). For a
model with only one predictor variable: p = p o + p i X i
Therefore Log Odds = Po + PiXi
p o is the intercept and p i defines the slope of the line.
If this line is not linear then mathematical adjustments such as quadratic terms may need to be considered to transform the model into its linear form.
The results for the Hosmer Lemeshow goodness of fit statistic and the ROC analysis (tables 1 and 2) demonstrates several points. Firstly, it confirms the well established observation (Champion, 1983) that the addition of the Revised Trauma Score variable (weighted or unweighted) to the Injury Severity Score variable
results in a substantial improvement in the calibration and discrimination of both models (model 6 and 8) when compared to the model with only the ISS variable (model 1). In this study separation of the Revised Trauma Score variable into its three components resulted in better model calibration but not discrimination when compared to the composite Revised Trauma Score variable without weights (model 4 c.f. to model 6). The addition of MTOS weights to the RTS variable substantially improved model calibration but not discrimination (model 8 c.f. model 6). The addition of age as a continuous variable to the three previous models (models 4, 6 and 8) resulted in an improvement in model calibration for all three models. Discrimination was not however significantly improved with the addition of age to the previous three models (4, 6 and 8). The former results are in broad agreement with the findings by Stephenson et al (2002) i.e. that the addition of age as a continuous variable improves model calibration. Interestingly enough model 9 (HCISS + RTS {MTOS weights} + Age) had the best calibration of all nine models. Model 5 had the best discrimination although this was not statistically significant when compared to models 3-9.
The results of this study are in part agreement with the results of Hannan et al (1999) who also found that the RTS variable was superior when utilised in its component form rather than in its weighted composite form. The magnitude of this difference was not mentioned in the monograph although the inference was that the difference was significant. This is in contrast to this study where the difference in calibration between the component RTS and the weighted RTS was minimal. There are several important
differences between the study by Hannan et al (1999) and the author’s study. Firstly, in this study model validation was performed using internal validation in contrast to the study by Hannan et al who used external validation on a separate data set. Both external validation and cross-validation provide a more stringent test of model fit (Harrell, 1996). Secondly, in the study by Hannah et al the test data set contained only patients with blunt injuries in comparison to the data set used by this author which contained both blunt and penetrating cases. Thirdly, Hannan et al used the ‘deciles of risk’ method (i.e. ten equal sized groups of predicted probabilities) to calculate the HL value. In contrast the SPSS algorithm method (see chapter 5 for a detailed description of this method) was used in this study. The various methods of calculating the Hosmer Lemeshow statistic can result in different values for the test statistic as was pointed out by Hannan et al (1995).
This study also showed that the addition of coded SBP and coded RR to model 2 (HCISS + coded GCS) resulted in a marginal reduction in the HL value. These findings support the work of Becalick et al (2001) who found that SBP and RR were relatively unimportant when compared to the motor and verbal components of the GCS. Their results were based upon a 16 predictor variable model developed using a Neural Network method.
In summary the results of this study are in broad agreement with other authors (Hannan, 1999; Becalick, 2001; Stephenson, 2002). Of interest was that the model with the best calibration was in fact a TRISS type model using age as a continuous variable.