Posttraumatic stress symptoms at two weeks were examined with the Impact of Event Scale (IES);29 a 15-item questionnaire in which patients have to rate whether they experience intrusive
and avoidant posttraumatic stress symptoms on a 5-point Likert scale, resulting in a minimum score of 0 and a maximum score of 75. Scores above 26 were classified as severe, in line with recommendations.30
Selection of prognostic models
Existing prognostic models were identified by screening reference lists of systematic reviews on prognosis following mTBI10,12 and by updating the search strategy of Silverberg et al.10 until
March 2017. Models were considered for our external validation study if they were developed in prospectively collected data on adult (age ≥ 16 years) patients with mTBI (Table 1). To be included, studies had to fulfill at least one out of three quality criteria:21 1) A large sample size
(N > 500 patients); 2) > 10 cases (patients with PCS) for each candidate predictor consider; or 3) the use of shrinkage or internal validation.
Two models met our eligibility and quality criteria (Online Supplement A).19,31 Stulemeijer et al.31
had developed a prediction model based on data from a level I trauma center in the Netherlands (2004-2006) to predict six-month PPCS using the RPQ. They investigated 201 patients among whom 49 developed PPCS. Among the 19 candidate predictors, three (pre-injury physical comorbidities, early PCS, early posttraumatic stress) were included in their final model. The final model had an Area Under the Curve (AUC) of 0.82, which decreased to 0.73 after bootstrap validation.
Cnossen et al.19 had developed a prediction model based on 277 patients from three level
I trauma centers in the United States (2010-2012) applying the RPQ. The RPQ was used as a linear scale and 14 candidate predictors available at the ED were considered. After least absolute shrinkage and selection operator (lasso) procedure and bootstrap validation, the final model with 8 predictors (age, sex, years of education, pre-injury migraine or headache, pre-injury psychiatric disorders, prior TBI, PTA and LOC) explained 14% of the variation in outcome. The final set of predictors was examined in a logistic model with the RPQ dichotomized according to the ICD-10 criteria, resulting in an AUC of 0.74. A list of included and excluded studies as well as detailed characteristics of the included prediction models are presented in the Online Supplements B and C.
Table 1. Eligibility criteria for the external validation of existing prediction models in the curent study o Data:
o Prospectively collected o Patients:
o Patients with mild TBI (GCS 13-15) o Adult patients (age ≥ 16 years) o Outcome:
o Examined at ≥ 6 months post-injury
o Outcome measurement: HISC or another self-reported measurement in which the prevalence of post-concussion symptoms was broadly dichotomized into ‘PCS according to the ICD-10 criteria’ and ‘no PCS according to the ICD-10 criteria’
o Predictors:
o ≥ 80% of the predictors in the model should be measured in the current study o Quality Requirements model:
o Multivariable model of at least two predictors
o At least one out of three quality criteria reported by Mushkuadini et al.21 1. Large sample size (N > 500)
2. > 10 cases (patients with PCS) for each candidate predictor considered 3. The use of shrinkage and/or internal validation
Abbreviations: GCS = Glasgow Coma Scale; HISC = head injury severity scale; ICD = international classification of diseases; PCS = post- concussion symptoms; TBI = traumatic brain injury
Statistical analyses
Patient characteristics were reported by medians and interquartile ranges (IQR) for continuous variables and frequencies and percentages for categorical variables. To assess the possible influence of loss to follow-up, we compared characteristics of patients who completed the six-month outcome assessment and patients who were lost to follow-up using Chi-Square and Mann-Whitney U tests, since all continuous variables had a skewed distribution. Missing data on candidate predictors were subsequently imputed using multiple imputation.
The external validity of two existing models was assessed in terms of calibration and discrimination. Calibration refers to the agreement between observed and predicted outcomes. Calibration was graphically assessed and expressed as calibration-in-the-large, indicating whether predictions are systematically too low (calibration-in-the-large > 0) or too high (calibration-in-the-large < 0) and calibration slope (indicating the average strength of predictor effects and ideally equal to 1). Discrimination refers to the ability of a model to distinguish between patients who will develop PPCS and patients who will not develop PPCS and was expressed as the AUC. An AUC of 1 implies perfect discrimination and an AUC of 0.5 indicates that the model is no better than random chance. All variables from the Stulemeijer et al.31 and Cnossen et al.19 models were available
in the UPFRONT data. However, education was measured as a continuous variable in study by Cnossen et al.19 but as a categorical variable in the UPFRONT dataset. Therefore, for the external
5
7
validation we used the mean years of education for each category (low education: 8 years, middle education: 12 years, higher education: 15 years).
To examine the role of complaints at the ED (headache, nausea or vomiting, neck pain) we developed a new model with all predictors from the Stulemeijer et al.31 and Cnossen et al.19
models in a backwards selection procedure (p < .157).32 We subsequently assessed whether the
addition of complaints at the ED significantly improved this model by comparing goodness-of-fit of a model with and without complaints at the ED. To correct for optimism of the new model, we used bootstrap validation with 100 samples, where all modeling steps were repeated. Analyses were performed using SPSS statistics version 21.0 and R (version 3.2.2) using the rms, foreign,
pROC and mice packages.
Results
Patient population
A total of 1,151 patients were included in the UPFRONT study, of whom 591 (51%) completed the six-month outcome assessment. Included patients had a median age of 51 years (Interquartile range 32 to 64) and 41% (n = 241) were female. Sixteen per cent (n = 94) had intracranial traumatic abnormalities on the initial head computed tomography (CT) scan. Patients included in the study were significantly older (p < 0.01), more often female (p = 0.03) and showed more CT abnormalities (p = 0.01) than patients lost to follow-up. In addition, patients included in the study less often reported pre-injury psychiatric disorders (p = 0.04) and more often reported pre-injury physical disorders (p = 0 .04, Table 2).
Persisting post-concussion symptoms
At six months post-injury, 370 patients (63%) reported at least one out of eight symptoms. A total of 241 patients (41%) reported three or more symptoms, indicating PPCS according to our criteria. Fatigue (38%), concentration problems (36%) and memory problems (35%) were most frequently reported (Figure 1).
External validation of existing models
Both existing models performed poorly; with an AUC of 0.64 (95% CI 0.60-0.68) for the Stulemeijer et al. model31 and an AUC of 0.57 (95% CI 0.52-0.62) for the Cnossen et al. model19 (Figure 2).
Both models systematically underestimated the proportion of patients with PPCS (calibration-in- the-large > 0) and average effects of the set of predictors were too low (calibration slopes < 1).
Table 2. Characteristics of 591 subjects included in the study and 560 subjects lost to follow-up
Included subjects (n = 591)
Subjects lost to follow-up (n = 560)
Variable Missing N (%) Missing N (%) p-value
Demographic and preinjury characteristics
Age (median, IQR range) - 51 (32-64) - 34 (22-53) < .01 Sex (Female) - 241 (41%) 194 (35%) .03
Education† 31 249 .07
– Low 105 (19%) 66 (21%)
– Middle 210 (37%) 134 (43%)
– High 245 (44%) 111 (36%)
Preinjury psychiatric disordersⱢ 30 58 (10%) 238 58 (18%) < .01 Preinjury physical disorders‡ - 185 (31%) - 145 (26%) .04 Preinjury headache or migraine 57 156 (29%) 279 87 (31%) .60 Prior TBI 131 15 (3%) 113 27 (6%) .05 ED characteristics CT abnormalitiesⱡ 13 94 (16%) 8 61 (11%) .01 LOC 3 494 (84%) 3 483 (87%) .20 PTA 37 490 (88%) 41 440 (85%) .08 Headache 87 263 (52%) 63 259 (52%) .98 Nausea or vomiting 92 173 (35%) 78 174 (36%) .64 Neck pain 120 71 (15%) 97 91 (20%) .07
Early postinjury symptoms
2-week PCS¥ 160 240 (56%) 348 108 (51%) .26 2-week posttraumatic stressⱠ 117 114 (22%) 287 69 (25%) .20
†Low education = less than 11 year of formal education; middle education = 11-14 years of formal education; high education = 15 or more years of formal education
ⱢIncludes any psychiatric disorder necessitating treatment by a psychologist or psychiatrists or use of psychotropic medication, or both ‡Includes cerebrovascular accident, heart diseases, hypertension, diabetes, asthma or other respiratory diseases, epilepsy or any malignant disorder
ⱡAny lesions, compressed cisterns or midline shifts
¥Meeting the ICD-10 criteria for post-concussion symptoms, 2 weeks postinjury. ⱠScore on the Impact of Events Scale above 26, 2 weeks postinjury
Abbreviations: ED = emergency department; CT = computed tomography; IQR = interquartile range; LOC = loss of consciousness; PCS = post-concussion symptoms; PTA = posttraumatic amnesia; TBI = traumatic brain injury
Development of the UPFRONT-PPCS model
Backward selection (p < 0.157) with all variables from the Stulemeijer et al. and Cnossen et al. models19,31 resulted in the inclusion of three variables: female sex (OR 1.48, 95% CI 1.01-
2.18), two-week PCS (OR 4.89, 95%CI 3.19-7.49) and two-week posttraumatic stress (OR 2.98, 95% CI 1.88-4.73; Table 3). PCS after two weeks was the strongest predictor: among the 241 patients with six-month PPCS, 192 (80%) already reported three or more symptoms after two weeks and almost all patients (n = 233, 97%) reported at least one symptom after two weeks.
5
7
However, among the patients reporting three or more symptoms after two weeks (n = 333), only half (n = 192) still reported three or more symptoms after six months. In the other half, early PCS resolved over time.
Figure 1. Frequency of post-concussion symptoms six months post-injury
0% 10% 20% 30% 40% 50%
Intolerance to stress and alcohol Irritability Headache Sleep problems Dizziness Memory problems Concentra!on problems Fa!gue Percentage of pa!ents repor!ng that the symptom was worse than before the injury
The addition of complaints at the ED (headache, nausea or vomiting and neck pain) significantly improved the model (p < 0.01). Of these complaints, only neck pain was statistically significantly associated with six-month PPCS (OR 2.58, 95% CI 1.39-4.78). The AUC of the final prediction model was 0.77 and decreased to 0.75 after bootstrap validation (Table 3). An overview of all uni- and multivariable associations of predictors is shown in Table 4.
Table 3. Prediction model for six-month PPCS based on existing models and the role of ED symptoms
Variable OR (95% CI) Female sex 1.48 (1.01-2.18) Nausea or vomiting 0.88 (0.54-1.43) Headache 0.94 (0.61-1.47) Neck Pain 2.58 (1.39-4.78) 2-week PCS 4.89 (3.19-7.49) 2-week posttraumatic stress 2.98 (1.88-4.73)
AUC 0.77
AUC after bootstrap validation 0.75
Intercept: B = -2.241
Figure 2. Calibration plots for external validation Stulemeijer et al. and Cnossen et al. models
X-axis shows predicted probabilities by the model in quantiles of patients and Y-axis shows observed proportion. The dotted diagonal lines represent perfect predictions. The triangles indicate the observed outcome frequency in quantiles of predicted probabilities, with 95% confidence interval. Calibration -i.t.l= Calibration-in-the-large.
Table 4. Univariable and Multivariable Associations of all predictors in this study and 6-month PPCS
Variable Univariable association
OR (95% CI) Multivariable association OR (95% CI) Age† 0.99 (0.99-1.00) 0.99 (0.98-1.01) Female gender† 2.40 (1.60-3.14) 1.56 (1.05-2.32) Education†
– Low vs medium / high 1.10 (0.69-1.73) 0.87 (0.54-1.72) – Medium vs low/high 1.26 (0.87-1.83) 1.39 (0.91-2.13) Preinjury physical comorbidities†‡ 1.05 (0.74-1.50) 1.20 (0.73-1.95) Preinjury migraine or headache† 1.06 (0.74-1.51) 0.96 (0.63-1.47) Preinjury psychiatric disorders† 1.63 (0.92-2.86) 1.18 (0.64-2.18) Prior TBI† 1.22 (0.46-3.27) 1.25 (0.34-4.62) PTA† 1.41 (0.82-2.40) 1.06 (0.55-2.03) LOC† 0.92 (0.59-1.44) 0.89 (0.53-1.50) 2-week PCS‡ 5.82 (3.94-8.60) 5.18 (3.39-7.91) 2-week posttraumatic stress‡ 3.22 (2.12-4.50) 3.15 (1.93-5.16)
ED symptoms
Headache 1.46 (1.03-2.07) 1.15 (0.73-1.75) Nausea or vomiting 1.31 (0.85-2.01) 1.13 (0.73-1.76) Neck pain 3.66 (2.23-6.01) 3.44 (2.05-5.78)
In the multivariable model, all variables are included. Therefore, the effect estimates might diverge slightly from the effect estimates in Table 3
‡Variable derived from Stulemeijer et al. model †Variable derived from Cnossen et al. model