Comparing the Proposed Method with Uncertainty Estimation Methods

Fig 4.9 – Common features of Bootstrap, Jackknife, MI and the proposed method

The proposed method and the uncertainty estimation methods described in the two preceding sections are similar in that they all execute the imputation method (the method being evaluated) repeatedly against the same missing value dataset. This process produces a set of unique imputed datasets, as shown in Fig. 4.9 (although different techniques are used to create these datasets, as explained above). All four of the methods then go on to use the set of parameters that describe the unique datasets to evaluate the results of the imputation process.

However, the four methods also differ in several important respects, as described below.

In its purest form, the Jackknife method requires n repetitions of the imputation method, where n is number of rows in the dataset (see section 4.3.2). This can be impractical for large datasets (e.g. datasets containing millions of rows), particularly since the imputation method itself will take longer to execute as the size of the dataset increases. In addition, the Jackknife differs from the other methods in that the overall process is deterministic - i.e. a repeated execution of the overall Jackknife process (all iterations taken together) against the same dataset will always produce the same uncertainty estimate.

The Bootstrap method introduces a stochastic element into the overall evaluation process by taking random samples from the dataset. In effect, this sampling process is equivalent to a repetitive randomised form of the listwise deletion process described in chapter one. However, there is no way of knowing whether any particular sample will fully reflect the missing data mechanism within the dataset as a whole. This could bias the uncertainty estimate produced by the Bootstrap when the data is MAR. It could be argued that this hardly matters, since the repetitive process should remove this bias - but it is unclear how many Bootstrap iterations would be needed to achieve this for any particular MAR configuration.

Missing value dataset Unique imputed dataset n Unique imputed dataset 2 Unique imputed dataset 1 Imputation evaluation Imputation process 2 Imputation process 1 Imputation process n Dataset 1 parameters Dataset 2 parameters Dataset n parameters

Multiple imputation builds on the ideas underpinning the resampling methods, but differs from them in that it integrates the repetitive stochastic part of the imputation evaluation process with the imputation procedure - so that repeated executions of that procedure will “reflect variation within an imputation model and sensitivity to different imputation models” (Rubin, 1978). However, unlike the resampling methods, MI achieves this while retaining all of the rows in the dataset. The proposed method also retains all of the rows in the dataset, so it has more in common with MI than either of the resampling methods in this respect.

The proposed method differs from the other methods by estimating the accuracy of the imputed values - i.e. the other methods do not record deleted values and then measure

how accurately they have been “put back” by the imputation process. The Bootstrap, MI

and the proposed method are similar in that they all employ stochastic procedures, but the Jackknife method does not share this characteristic. However, the proposed method and the Bootstrap differ from MI in that the stochastic part of these methods is performed before the imputation process starts, rather than being integrated with that process.

The problem of defining a “proper” multiple imputation method in practice

The Bootstrap/Jackknife methods and the proposed method can all be used to evaluate any imputation technique, whereas the MI approach can only be confidently used for evaluation purposes when we are sure that the MI method employed is “proper”, in the sense defined by Rubin (1987, pp. 118-119) and further summarised by Rubin (1996a). One of the clearest defintions of a proper MI method is given by Durrant (2005), who explains that for a proper MI method, equation (4.10) - given in section 4.3.3 - “is indeed a valid formula, providing an approximately unbiased estimator of the variance”. But it is very hard to verify the truth of this statement in practice, as Schafer (1997) points out;

“Except in trivial cases (e.g. univariate data missing completely at random), it can be extremely difficult to determine whether a multiple-imputation method is proper”

Binder and Sun (1996) shed some light on this problem by discussing several complex examples, but these only cover a small proportion of the imputation problems that can occur. In practice, proper MI methods generally employ Bayesian imputation algorithms, even though theoretically this is not deemed to be essential. For example, Schafer’s (1997) implementation of MI employs the Markov chain Monte Carlo method (Tanner, 2005; Gilks et al, 1996) via the Bayesian data augmentation algorithm (Tanner and Wong, 1987). To summarise, we can say that the evaluation of imputation methods via MI uncertainty estimation can only be confidently applied when we are sure that the MI method used is proper - e.g. when that method employs a Bayesian imputation algorithm. However, the other

4.4 Summary

This chapter has described the imputation evaluation method devised by the author and has shown how this method can be used to estimate the predictive accuracy of the imputed values generated by any imputation technique.

A functional overview of the proposed method has been given and the equations and procedures which form the basis of that method have been described in detail. An explanation of how the method can be used to compare the accuracy of the imputed values in different data segments has been given. These descriptions and explanations form the principal contribution to knowledge made by this thesis.

A description of how the general idea of estimating imputation accuracy has been applied by other researchers has been given and it has been shown that the proposed method differs from these approaches in several important respects. The functionality of the most similar methods found within the literature (uncertainty estimation methods) has been described and the limitations of these methods have been discussed. The similarities and the differences between the proposed method and uncertainty estimation methods have been discussed, and it has been shown that the proposed method builds on the ideas underpinning uncertainty estimation methods, but differs from them in several important respects.

The proposed method has been implemented alongside the imputation techniques described in chapters two and three in the form of an integrated software application. The following chapter describes how this application was used to experimentally evaluate the reliability and the validity of the proposed method. Chapter six goes on to explain how the integrated application was used to assess the feasibility of imputing the missing values in the collaborating company’s dataset, thus fulfilling the project objectives.

Chapter Five