Item 4: How were competing events managed in each prediction model

Information about the management of competing events in each of the 15 prediction model studies is reported in Table 3.7. Statistical methods (such as competing risks regression) were applied to appropriately manage competing events in six (24.0%) individual prediction models from three (20.0%) prediction model studies (Kovalchik et al., 2012, Romond et al., 2012, Travis et al., 2005). Competing risks bias should not be present as the competing events have been appropriately managed. Competing events were censored during the development of nine (36.0%) individual prediction models, from four (26.7%) prognostic model studies (Bevilacqua et al., 2012, Hippisley-Cox and Coupland, 2012, Nyberg and Gustafson, 1997, Sekhri et al., 2008), and prior to the development of one individual prediction model (Sekhri et al., 2008). Competing risks bias is likely to affect these individual prediction models as censoring or excluding competing events causes inflated absolute risk estimates. The eight (53.3%) remaining prediction model studies, which developed ten (40.0%) individual prediction models, did not report on the management of competing events. While two of these (Carey et al., 2008, Schonberg et al., 2009) were classified as no potential for competing risks bias, the remaining prediction model studies were classified with moderate or high potential for competing risks bias, thus it is unlikely that no competing events occurred in these studies. For example, one cohort study followed 12,648 stroke patients, with a mean age of 64, for 5 years (Ariesen et al., 2006); The study does not report any deaths (competing events) or how these were managed in the study, yet it is unlikely that none occurred.

Table 3.7: Management of competing events in each prediction model study Prediction model study reference Potential for competing risks bias

Prediction model outcomes

Competing risks statistical methods used Competing events censored Competing events excluded Validation method (Sekhri et al., 2008) High

Death due to coronary heart disease or non-fatal acute coronary syndrome

No Yes No Apparent

(Ariesen et al.,

2006) High Intracerebral haemorrhage No No No Internal

(Cuschieri et al.,

2014) High

Acute gastrointestinal (GI)

bleeding No No No Apparent

(Carey et al., 2008) No All-cause mortality No No No Internal

(Schonberg et al.,

2009) No 5-year mortality No No No Internal

(Bevilacqua et al.,

2012) High Lymphedema No Yes No Internal

(Briganti et al.,

2010) High Erectile function recovery No No No Internal

(Ezaz et al., 2014) High Heart failure and

cardiomyopathy No No No Internal

(Kovalchik et al.,

2012) High Second primary thyroid cancer Yes No No External

(Mathieu et al.,

2014) High

Urinary toxicity, urinary

frequency, dysuria No No No Apparent

(Romond et al.,

2012) High

Severe congestive heart failure

or cardiac death Yes No No Internal

(Travis et al.,

2005) High Breast cancer Yes No No None

(Nakagawa et al.,

Prediction model study reference

Potential for competing risks

bias

Prediction model outcomes

Competing risks statistical methods used Competing events censored Competing events excluded Validation method (Hippisley-Cox and Coupland, 2012)

The predictive performance of each developed model was evaluated in some way for the majority of models (23, 92.0% models from 14, 93.3% prediction model studies). Apparent performance measures were reported for nine (36.0%) individual prediction models from five (33.3%) prediction model studies; internal validation methods were reported for 11 (44.0%) prediction models from eight (53.3%) studies; and external validation was reported for three (12.0%) models from one (6.7%) study. None of the prediction model studies reported on how competing events were managed during model validation.

3.4 Discussion

In this chapter, a review of the presence, reporting, and management of competing events in prediction model development studies was conducted. In total, 25 individual prediction models were evaluated, as identified from 15 prediction model studies likely to be affected by competing events. The key findings and conclusions of this review are summarised in Box 3.3 and discussed below.

Box 3.3: Key findings

1. Mortality is a common competing event for prediction model outcomes.

2. Prediction model development articles rarely report competing risks, even when studies are conducted in clinical settings where competing events are a likely issue.

3. The management of competing events in prediction model development studies is inconsistent and often not appropriate.

3.4.1 Key findings

The key findings of this review are presented and discussed in more detail:

3.4.1.1 Mortality is a common competing event in prediction models.

Of the 25 individual prediction models included in this review, only 8.0% predicted all-cause mortality, and a further 16.0% predicted cause-specific mortality. Thus mortality, either all-cause or other-cause, was considered a competing event for 92.0% of the prediction models identified in this review. Further, all of the individual prediction models were developed in either frail or elderly populations, and these populations are particularly susceptible to the competing risk of death (Berry et al., 2010). More needs to be done to highlight the importance of appropriately accounting for mortality as a competing event in prediction model studies in elderly or frail populations.

3.4.1.2 Prediction model development articles rarely report competing risks, even when studies are conducted in clinical settings where competing events are a likely issue.

The included prediction model studies were identified for having high potential for competing risks bias and population characteristics that were susceptible to competing events. Despite this, competing events were poorly reported in the published prediction model articles. Key terms related to competing events were not mentioned in 80% of the prediction model study articles (which developed 84.0% of individual prediction models). Further, the number of competing events was not reported in 73.3% of the prediction model study articles (60.0% of individual prediction models). It is unlikely that competing events were not present in the majority of the studies, given the selection criteria and potential for competing risks bias. Thus, the lack of reporting of competing events may indicate the lack of awareness of competing risks bias as an issue in prediction model research.

3.4.1.3 The management of competing events in prediction model development studies is inconsistent and often not appropriate.

The management of competing events during the development of the prediction models was inconsistent; for example, statistical methods to appropriately handle competing events were applied in only 20.0% of the prediction model studies (24.0% of individual prediction models), whilst competing events were censored in 26.7% of the prediction model studies (36.0% of individual prediction models). The management of competing events in the validation of prediction models was not discussed in any of the articles. Not appropriately managing competing events when they are present can lead to bias, inflated absolute risk predictions. Moreover, not accounting for the competing events when validating prediction models can lead to bias in predictive

appropriately in new prediction model development studies where competing events are an issue.

3.4.2 Limitations and further research

This work adds to existing reviews and empirical evaluations of whether competing risks have been accounted for in medical research. Previous systematic reviews investigating competing risk biases have focused on articles containing Kaplan-Meier curves (Schumacher et al., 2016, Walraven and McAlister, 2016), studies with population susceptible to competing risks (Koller et al., 2012), or published randomised controlled trials (Austin and Fine, 2017). This systematic review focuses on competing risks bias in prediction model development studies and thus provides a novel insight into the presence, reporting, and management of competing events during the prediction model development process. The review highlights the high presence of death as a competing event in prediction model studies in elderly and frail populations. Nevertheless, the review does not encompass all prediction model development research. The high presence of competing events is unlikely to be generalizable to all prediction model development studies, given the selection of articles from of high potential for competing risks bias systematic reviews. However, focusing on studies likely to be affected by competing events allowed a concentrated investigation into the reporting and management of competing events in model development articles where competing events were likely to be present and cause bias results if inappropriately accounted for.

There has been relatively little research about the impact of competing events in relation to prediction model research. The way in which competing events may affect prediction model outcomes (if not appropriately accounted for) include: erroneous estimates of prognostic factor associations, inflated absolute risk predictions, mis-

proportion of prediction model development studies which do not appropriately account for competing events. However, further research is needed to evaluate the impact of this bias on the prediction model outcomes (such as absolute risk predictions, and measures of calibration and discrimination). In the following chapters the statistical methods available to appropriately account for competing events will be discussed and applied to develop a prediction model for the risk of antenatal adverse events using data from the Prediction of Risks in Early onset Pre-eclampsia (PREP) study (Thangaratinam et al., 2017).

4 A COMPARISON OF TIME-TO-EVENT PROGNOSTIC