The purpose of a prognostic model is usually to predict the risk of an event. Although the outcome is, therefore, binary (i.e. yes or no), the predictions are almost always intermediate probabilities, rather than 0% (will definitely not happen) or 100% (definitely will happen). A model should, therefore, successfully distinguish between high and low risk groups, but the ability to predict an individual's outcome is almost always limited (Henderson and Keiding2005). Nevertheless, even relatively modest risk stratification can be clinically useful. For example, given a 5% operative risk of stroke and death for endarterectomy for patients with asymptomatic carotid stenosis and an average five-year risk of stroke on medical treatment of about 10%, simply separating patients into a group with a mere 5% five-year unoperated risk and a group with a more worrying 20% five-year risk would substantially improve the targeting of treatment (avoid for the former, recommend for the latter perhaps).
The quality of the data is also important. Measurements should ideally have been made with reliable and reasonably standard methods, preferably without categorization (i.e. they should be recorded as continuous variables as opposed to ad hoc categories, which might limit their predictive value). In multicenter trials or cohorts, it is particularly important to Table 14.3. Table of quality criteria for study method for reliable measurement of prognosis
Study feature Qualities sought
Sample of patients Methods of patient selection from study population described
Inclusion and exclusion criteria defined Diagnostic criteria defined
Clinical and demographic characteristics fully described Representative
Assembled at common (usually early) point in course of disease Complete (all eligible patients included)
Follow-up of patients Sufficiently long, thorough and sensitive to outcomes of interest
Loss to follow-up provided
Outcome Objective, unbiased and fully defined (e.g. assessment blinded to
prognostic information) Appropriate
Known for all or high proportion of patients
Prognostic variable Fully defined
Details of measurement available (methods if relevant) Available for all or high proportion of patients
Analysis Continuous predictor variable analyzed appropriately
Statistical adjustment for all important prognostic factors Treatment subsequent to inclusion
in cohort
Fully described
Treatment standardized or randomized
have consistent methods of measurements and similar definitions of variables across centers (e.g. are the centers all using the same definitions of hypertension?). As complete a set of data as possible is also essential for the development of reliable prognostic models. Even when each variable is reasonably complete, many patients will have missing data for at least one variable, often the majority of patients (Clark and Altman2003). Excluding such cases, as standard statistical packages would automatically do, reduces statistical power and may also introduce bias. The alternative to exclusion of patients with missing data is to impute the missing values (Vach1997; Schafer and Graham2002). Although imputation requires certain assumptions about why data are missing, it can often be preferable to risking selection bias by using only cases with complete data (Schafer and Graham 2002; Burton and Altman2004).
Prognostic models are usually derived using logistic regression (for predicting binary outcomes) or Cox regression (for time-to event data). The sample size required depends on the number of outcomes and not the number of patients (Feinstein 1996; Schmoor et al. 2000). Cohorts with few events per prognostic variable studied are likely to produce unreliable results. It is generally recommended that there should be at least 10–20 outcome events per prognostic variable studied (i.e. not 10–20 per variable eventually included in the model), although reliable models have been derived on smaller numbers (Harrell et al. 1984; Feinstein 1996). However, with a small derivation cohort there will always be a risk of selecting unimportant variables and missing important ones by chance.
Before performing multivariable analysis, many researchers try to reduce the number of candidate variables by means of univariate analyses, eliminating those variables that are not significant univariate predictors (often with a cut-off ofp < 0.1). However, this step is not strictly necessary and it may introduce bias (Sunet al.1996; Babyak2004). It makes more sense to reduce the number of candidate variables using clinical criteria, such as by eliminating variables that are difficult to measure in routine practice (e.g. emboli on transcranial Doppler) or a variable that is likely to be closely correlated with another (e.g. systolic and diastolic blood pressure).
The initial selection of variables can be further reduced automatically using a selection algorithm (often backward elimination or forward selection). Such an automated proced- ure sounds as though it should produce the optimal choice of predictive variables, but it is often necessary in practice to use clinical knowledge to over-ride the statistical process, either to ensure inclusion of a variable that is known from previous studies to be highly predictive or to eliminate variables that might lead to overfitting (i.e. overestimation of the predictive value of the model by inclusion of variables that appear to be predictive in the derivation cohort, probably by chance, but are unlikely to be predictive in other cohorts).
It is also important to look for potential interactions between the predictive value of particular variables (i.e. the predictive value of one variable may depend on the presence or absence of another), especially if there is some a priori clinical or biological reason to suspect an interaction. For example, the predictive value of cholesterol level is likely to fall with age in a model predicting the risk of vascular events (total cholesterol is highly predictive of myocardial infarction in patients in their 40s and 50s, of less value in 60s and early 70s, and of little value in the 80s). Such interactions can be taken into account in the model by including an interaction term, which will increase the predictive power of the model, assuming that the interaction is generalizable to future patients.
Chapter 14: Methods of determining prognosis
Studies developing models should be reported in adequate detail. Several reviews of papers presenting prognostic models have found common deficiencies in methodology and reporting, including a lack of information on the method for selecting the variables in the model and on the coding of variables, and a tendency to have too few events per variable in the derivation cohort (Concatoet al.1993; Costeet al.1995; Laupaciset al.1997; Counsell and Dennis2001; Hackett and Anderson2005; Jacobet al.2005). Authors must report the model in enough detail so that someone else can use it in the clinic and can validate it with their own data. The main issues in assessing studies reporting prognostic models are internal validity, external validity, statistical validity, evaluation of the model and practical- ity of the model (Counsell and Dennis2001; Jacobet al.2005).