Model uncertainty and sensitivity analysis

Greenland (1996) describes sensitivity analysis as a “quantitative extension of the qualitative specu- lations”, and the importance of assessing all major sources of uncertainty is emphasised by Greenland (2005) and discussants. Draper (1995) points out that models generally have parametric and structural uncertainty, and that the latter is less routinely acknowledged. When data are incomplete, taking ac- count of structural uncertainty is crucial as the results from the types of models required display great sensitivity to the assumptions made. The uncertainty extends to an inability to distinguish between a MNAR and MAR missingness from the data (Molenberghs and Kenward, 2007). Indeed, the fit of a MNAR model to the observed data can always be reproduced exactly by a MAR counterpoint, but the two models will produce different predictions for the unobserved data and different inferences (Molenberghs et al., 2008).

A model’s fit to the observed data can be assessed, but its fit to the unobserved data given the observed data cannot be assessed. Consequently, there is a need for sensitivity analysis, in which, for example, several statistical models are considered. The degree of stability of inferences across these models provides an indication of how much confidence can be placed in them. Molenberghs and Kenward (2007), Chapter 20, suggest that a sensitivity analysis for a selection model might involve varying the distributional assumptions, or supplementing the analysis with several pattern mixture models which allow the missing data part to be explicitly modified.

DH advocate what they call the extrapolation factorisation, where the full data model is factored into identified and non-identified components, i.e.

f (y_obs, ymis, m|ω) = f (ymis|yobs, m, ωE)f (yobs, m|ωO).

Here ωE and ωOdenote the parameters of the extrapolation model, f (ymis|yobs, m, ωE), and observed

data model, f (y_obs, m|ωO), respectively. They argue that sensitivity analysis is facilitated if models

for non-ignorable drop-out have one or more parameters with transparent interpretations which are completely non-identified by the observed data. To formalise this idea, DH define sensitivity parame-

model to the observed data. Sensitivity analysis regarding the missing data mechanism can then be carried out by examining inferences across a range of values for these sensitivity parameters. However, fully specified parametric selection models cannot be factored in this way, as all the parameters can be identified using the parametric assumptions in both parts of the model. Using a simple example, DH illustrate how the model of missingness parameters are identified by the observed data and distribu- tional assumptions about the model of interest, and therefore are not sensitivity parameters according to their criteria. By contrast, pattern mixture models are amenable to the extrapolation factorisation. There are also a range of proposals for sensitivity analysis in which a MAR model is taken as a starting point, and the impact of potential deviations towards a MNAR mechanism are explored. One tool that uses a perturbation scheme around a MAR mechanism is local influence (Verbeke et al., 2001; Jansen et al., 2006; Molenberghs and Kenward, 2007). Although the original idea of local influence was to detect observations with a large impact on conclusions due to their aberrant missingness mechanism, it actually detects observations that have a large impact on the model of interest or model of missingness parameters for a wide variety reasons not necessarily related to the missingness mechanism, for example an unusual mean profile or autocorrelation structure. Local influence, which changes the missingness process for one observation from MAR to MNAR, is different to case deletion, where one observation is removed entirely, and can yield different results (Molenberghs et al., 2008). Troxel et al. (2004) propose a screening tool that measures the potential impact of non-ignorability on an analysis, called an index of sensitivity to non-ignorability (ISNI), which studies the sensitivity in the neighbourhood of a MAR model and avoids estimating a full non-ignorable model. Ma et al. (2005) extend the ISNI to longitudinal studies.

White et al. (2008a) quantify the degree of departure from the MAR assumption for binary outcomes by the informative missingness odds ratio (IMOR), which they define as the ratio of the odds of success among individuals with missing outcomes to the odds of success among individuals with observed outcomes. They use the IMOR to set up sensitivity analyses using pattern-mixture models in a meta-analysis context. In a companion paper, White et al. (2008b) show that the IMOR can also be used in a selection model parameterisation, as they develop a hierarchical model for meta-analysis which is implemented using WinBUGS software.

Summary: literature review

‘Statistically principled’ methods are generally recommended for modelling missing data, and Bayesian full probability modelling falls in this category. If the missingness mechanism is thought to be non-ignorable, one approach entails building a joint model based on the selection model factorisation, which consists of a model of interest and a model of missingness. Such Bayesian models can be implemented using MCMC methods, and as with any statistical model their fit should be checked. However, only their fit to the observed data and not to the missing data can be assessed, so sensitivity analysis is regarded as crucial for assessing model uncertainty.

DIC is widely used in Bayesian statistics for comparing models. For non-ignorable missing data models, its construction is complicated by the unobserved data and the need to take account of the missing data mechanism. In this case, constructions based on the observed data likelihood and the full data likelihood have been proposed.

Chapter 3

Data

In this chapter we introduce the main datasets that are used throughout this research. These include three real data examples, two from the British birth cohort studies and one from a clinical trial, and some synthetic datasets that we use in our initial methodological exploration.

3.1

British birth cohort studies

Our main sources of data for developing and testing strategies for Bayesian full probability modelling are the three most recent British birth cohort studies, each of which has their own ‘attrition’ char- acteristics. The earliest, the National Child Development Study (NCDS), originated as the Perinatal Mortality Survey, which was designed to examine the social and obstetric factors associated with still- birth and early infant mortality. It subsequently evolved into a longitudinal study, following all those living in Great Britain who were born in one week in March 1958, and currently has eight sweeps of data available including the original birth survey. The next, the British Cohort Study (BCS70), was designed along similar lines to the NCDS to survey all those living in Great Britain who were born in a particular week in April 1970, and currently has available seven sweeps of data. The BCS70 was originally named the British Births Survey and had a similar initial focus to the NCDS. Both these studies surveyed over 17,000 babies, and their scope has broadened over time to encompass their sub- jects’ health, education, social development, and various aspects of adult life. The response pattern for BCS70 is rather different from that of NCDS, with generally lower response rates for BCS70 (Plewis

et al., 2004). For both these studies, cohort members move in and out of the study over time, allowing

the possibility of using data from subsequent as well as previous sweeps in predicting missing data, in contrast to approaches based on the assumption that attrition is an absorbing state.

The Millennium Cohort Study (MCS) was set up to provide information about children living and growing up in each of the four countries of the UK, including sufficient data for various sub-groups, such as those living in disadvantaged circumstances and of ethnic minorities, to be investigated. Data

about the children’s families is provided in addition to data about the children themselves, through interviews and self-completion forms undertaken by a main respondent (usually the cohort member’s mother) and a partner respondent (usually the father). In order to meet these aims, the design of MCS is very different from the other two studies, with over 18,000 cohort members being selected at random following stratification and clustering from children born in the UK between specified dates at the start of the Millennium (Plewis, 2007a). This has led to substantial non-response in the initial sweep (Plewis (2007a) categorise 15% of the issued sample as non-respondents), in contrast to NCDS and BCS70. Two further sweeps of this cohort are now also available. Non-response in the initial sweep and first follow-up (sweep 2) is discussed by Plewis (2007b), Ketende (2008) concentrates on response in sweeps 2 and 3, and Calderwood et al. (2008) discuss participation in the first three sweeps. All three studies are multi-disciplinary, with domains of interest including health, family background and education. The response patterns for the different domains vary (Plewis et al., 2004), with unusually low response for education in sweep 3 for NCDS, when the cohort members were age 16, which was affected by a change of school leaving age. For sweep 3, 87% of the target sample responded, of which 82% answered questions on education. In view of this sizeable missingness, it seems appropriate to use a subset of the educational data from NCDS to motivate our initial research, using models proposed by Goldstein (1979) for investigating the effects of social class on educational attainment as a starting point. Our other major example from the British birth cohort studies uses data taken from MCS to predict income, for which there is evidence of non-ignorable non-response (Plewis et al., 2008). We will now discuss each of these datasets in turn.