Bootstrap Validation Comparing Non-Linear Functions

Internal validation, to directly compare the fit of the restricted cubic spline and fractional

polynomial strategies, was carried out by calculating the sampling distribution of the difference in the AIC statistics between the models in multiple bootstrap resamples of the data using an in- house developed SAS program (Appendix) based on nonparametric bootstrap analyses (Efron and Tibshirani, 1993). The bootstrap analysis fitted the final models, one based on the restricted cubic spline transformations and one based on the fractional polynomial transformations, to a series of 1000 bootstrapped resamples taken from and with the same size as the original data. Smaller values of the AIC indicate a better model but it is unclear whether AIC statistics from non- nested models will approximate the chi-square distribution for hypothesis testing. Using a fixed level of significance for the comparison of AIC is not appropriate in non-nested models with the same parameters. The difference in AIC statistics for the different models in multiple bootstrap resamples provides a distribution which can then be summarised appropriately with bootstrap percentile confidence intervals derived from the bootstrap distribution of the difference between the restricted cubic spline and fractional polynomial models.

96

The %BOOT macro (Sarle, 2000) in SAS (SAS Institute Inc., 1999) is very computationally intensive carrying out non-parametric bootstrap analyses for simple random samples to produce inferences such as approximate standard errors, bias-corrected estimates and confidence intervals without knowing the type of distribution from which a sample has been taken assuming a normal sampling distribution. The %BOOT macro was used to calculate 1000 resamples of the original dataset using identical univariate transformations, if any, as in the original dataset.

Modelling was carried out in 1000 bootstrap resamples forcing in trial, stage, sex, trt and CA19-9 components (restricted cubic spline or fractional polynomial) and considering all other variables for inclusion (including the borderline variables of AST and BUN) based on backward elimination variable selection method. Two models were created within each bootstrap sample, one based on restricted cubic spline transformations and one based on fractional polynomial transformations. The AIC for each model was output from the screen to a temporary SAS dataset and the difference (AIC[RCS]-AIC[FP]) between them calculated, a positive difference indicating a better fit by the fractional polynomial transformation model. This was repeated in each 1000 bootstrap resamples and the mean difference calculated with 95% confidence interval (Table 7.3). The AIC statistics had to be output from the screen rather than delivered to an output dataset within the modelling process, as the two AIC values differed. This was reported to SAS who replied “you have an excellent point” and confirmed they would investigate further and recommended outputting the AIC from screen using an „ods output‟ statement instead.

97

Table 7.3: Bootstrap Comparison of AIC based on 1000 Resamples

Bootstrap Mean AIC Bootstrap Percentile 95% CI

Restricted Cubic Spline 5498.1 5441.7 to 5667.1

Fractional Polynomial 5487.1 5419.9 to 5648.5

Difference (RCS-FP) 11.0 9.0 to 31.4

The bootstrap mean AIC (Table 7.3) was smallest for the model containing fractional polynomial transformations and on average was a better fit than the model based on restricted cubic spline transformations by a bootstrap mean reduction of 11.0 (bootstrap percentile 95% CI: 9.0 to 31.4).

7.6. Conclusions

Fractional polynomial and restricted cubic spline functions are both polynomial functions

particularly suitable for modelling smooth curved relationships between response and a predictor and both are easily implementable in SAS. Both models containing non-linear transformations in the advanced pancreatic cancer dataset gave a substantially better fit compared to the models which dichotomised or assumed linearity of continuous covariates. The fitted functions generated by restricted cubic splines and fractional polynomials were similar but the model AIC and

bootstrap mean difference in AIC was smallest for the fractional polynomial model. The methods were generally different in the extremities, in the left-hand tails for CA19-9 (values less than 30,000 KU/l), where there is often a paucity of data. Due to the availability of an extremely large dataset in a different disease site, the reproducibility of these results could be investigated in data with an alternative event rate and survival distribution to investigate the stability of the

conclusions. Analyses of data from 42802 cardiac surgery patients are reported in Chapter 8.

98

CHAPTER 8: CARDIAC SURGERY EXAMPLE

 SUMMARY

 Prospective data were collected on 44902 patients undergoing cardiac surgery  5563 (12.4%) patients had died and median follow-up was 5.2 years

 Further analyses possible due to the size of dataset with alternative event rate and survival distribution

 Aim to investigate reproducibility of results seen when comparing non-linear methods in advanced pancreatic cancer dataset

 In addition, a univariate unadjusted fractional polynomial transformation was recalculated within each of 200 bootstrap resamples and compared directly against a 5-knot restricted cubic spline

 The influence of the size of the bootstrap samples was investigated

8.1. Introduction

Cardiovascular disease is the most common cause of premature death in the Western world and is closely related to socio-economic deprivation. Cardiac surgery includes a number of operations known to carry significant prognostic benefit. Prospective data were collected on 44902 patients undergoing cardiac surgery, followed for an median of 5.2 years, to assess whether social deprivation based on post-codes using the Carstairs score influences survival following surgery (Pagano et al., 2009). The event rate (5563 (12.4%) patients had died) and survival distribution (approximate 90% survival at 5-years) in this disease site is very different to advanced pancreatic cancer.

The aim of the analysis was to investigate the reproducibility of the results and conclusions drawn from the comparison of non-linear methods in the advanced pancreatic cancer dataset, where the

99

fractional polynomial transformation gave a better fit compared to the restricted cubic spline transformation. Analysis was carried out based on 42802 patients (5486 deaths) with complete data on the prognostic factors of interest. Seven baseline clinical and demographic variables were considered potential prognostic factors of survival (randomising centre, smoking status, diabetes, surgical procedure, body mass index (BMI), Carstairs score (CS) and EuroScore (ES)), three collected as continuous measurements (BMI, CS and ES). Due to the extreme size of this cardiac surgery dataset, further comparison of the two polynomial based strategies was able to be carried out. Principal analysis investigated each continuous covariate in turn but allowing the univariate fractional polynomial transformation to be recalculated within each of 200 bootstrap resamples taken from the original dataset and compared against a 5-knot restricted cubic spline. Supportive analysis investigated the influence of sample size by decreasing the bootstrap resample sizes compared to the size of the original dataset.