Chapter 6 Estimating the number of missing studies in neuroimaging
6.4 E↵ect of missing studies
The zero-truncated regression analysis of the previous sections suggests that if the assumptions of our file drawer model hold and the BrainMap database is repre- sentative of the population of CBMA studies, then there are roughly 9 unpublished experiments per 100 that we observe. Therefore, it is plausible to ask what would be the e↵ect of these unobserved, zero count studies on the outcome of a meta-analysis. In this section we repeat the meta-analysis of emotion studies shown in Chapter 4, adding some studies with no foci and demonstrate the e↵ects on the CBMA results that the inclusion of these studies has.
Using our estimates from Section 6.3 we add 9 zero count studies for every 100 of the original dataset which consists of 855 studies. Hence, there are 941 studies in the new dataset of which 86 report no activations. Analysis is carried out with the LGCP model presented in Chapter 4 using no covariate information. HMC is run for 15,000 iterations, with the first 5,000 being the burn-in period. The last 10,000 are thinned every 10 iterations to obtain a posterior sample of 1,000 intensities. These are used to calculate several quantities of interest which we now present.
The median posterior intensities for several axial slices of the brain are shown in Figure C.4. Qualitatively, the results are almost identical to the ones obtained when analysing the original dataset (see Figure 4.6). Nevertheless, there are still di↵erences between the two analyses. For example, the posterior expected number of foci in the original dataset is 7.15 (95% CI [6.99,7.37]); this is significantly higher compared to the estimate obtained when zero count studies are included which is 6.50 (95% CI [6.33,6.67]). In Figure 6.5 we show posterior distributions for the probability of observing at least one focus in the right and left amygdala, the regions which are mainly activated by emotion processing. Even though there is considerable overlap between the two, we see that the posterior median is roughly 10% lower when we add the zero count studies for both left and right amygdala.
Overall, we see that inclusion of zero count studies in the meta-analysis has lead to lower estimates of the e↵ect of emotion processing in brain activation, as we expected. Obviously, the extent to which the file drawer problem inflates the estimates obtained from a meta-analysis depends on the unknown total number of missing studies. It is therefore essential to account for the possibility of missing studies in any CBMA and examine estimates obtained under di↵erent scenarios.
Left amygdala 0 10 20 30 40 0.12 0.14 0.16 0.18 Right amygdala 0 10 20 30 40 0.09 0.11 0.13 0.15
Figure 6.5: Posterior probabilities of activation for the amygdala, as obtained from the original (green) and the new dataset with added zero foci studies (red). The left panel shows results for left amygdala and right panel for the right amygdala. All distributions are calculated based on a sample of 1,000 posterior draws.
6.5
Discussion
We have proposed a simple method for estimating the number of missing studies from a meta-analysis. Our method uses intrinsic statistical characteristics of the non-zero count data to infer zero counts. We find that the zero-truncated negative binomial distribution provides a good fit for the total number of foci per contrast in the BrainMap database. The analysis suggests that the magnitude of the file drawer, zero-foci experiments slightly varies depending on study characteristics but is generally around 9 per 100 published experiments. The number is significantly greater than zero which indicates the existence of publication bias in coordinate- based meta-analysis. Some of the missing experiments can be attributed to negative contrasts reported in the original publications but not registered in the database but surely some are never published.
Our approach relies on assumptions I and II described in Section 6.3. As- sumption I implies that there is independence between each contrast. However, as one publication can have several contrasts, this assumption is tenuous despite it be- ing a standard assumption for most CBMA methods. To ensure the independence assumption is valid, we subsample the data so that only one randomly selected con- trast per publication is used. Assumption II defines our censoring mechanism, such
that experiments with at least one significant activation are always published. The assumption that no significant research findings are suppressed from the literature has been adopted by authors in classical meta-analysis [Eberly and Casella, 1999] and we believe that is reasonable in the context of CBMA as well. For the unob- served studies, we accept that all experiments reporting null results never appear on the database. Note, that since studies typically examine several contrasts of in- terest, authors have fewer incentives to report non significant experiments because of the other, significant results that can be reported in the publication. At this point, one possible source of bias introduced due to the assumptions of the model is the existence of studies with several non significant experiments; however, in such a case our estimates are only underestimating the file drawer quantity and hence we choose to make no corrections.
Our file drawer model has some limitations. Firstly, the analysis in based on data retrieved from a single database. As a consequence, results are not robust to possible biases in the way publications are included in this particular database. A more thorough analysis would require consideration of other databases and pa- pers that haven’t been registered in BrainMap. Secondly, one may argue that our censoring mechanism is rather simplistic. For example we have not allowed for the possibility of experiments initially having negative results but then changing the analysis pipeline (e.g. random vs fixed e↵ects) to finally obtain some significant ac- tivations. This would be an instance of initially-censored (zero-count) data being ‘promoted’ to a non-zero count through some means. Such models can be fit under the Bayesian paradigm and will be examined in our future work.