Limitations and Challenges of an IMPF

Whilst reading the 20 IMPF articles, I recorded any pitfalls or challenges reported by the authors about the process of conducting IMPF projects (Figure 2.8); the three most common problems reported were: (i) unavailability of IPD for some studies (this may lead to the problem of publication and availability bias (see section 2.4.6 and chapter 7), (ii) different methods of measurement of the PF across studies and (iii) missing data.

[82]

Table 2. 7: Details about considering non-IPD studies and selection bias

Study

ID

First Author Was there a formal assessment of whether the included/selected IPD studies were a potential biased set of all available studies (akin to publication bias)?

If yes, what methods were used and what was concluded? How many studies were asked for their IPD? How many studies provided their IPD? Was IPD obtained from all studies desired? If no (or unclear): was the number of patients in the non-IPD studies given (or % missing)? Were the number of events in the non-IPD studies given (or % missing)? Were details given as to the robustness of meta-analysis results to the inclusion/exclusi on of non-IPD studies? If yes, briefly state what was said?

Was a method used to combine IPD and non-IPD studies? If so, what was it?

1 Butcher No Not relevant Not

relevant

9 Not relevant Not relevant Not relevant

Not relevant Not relevant

2 McHugh No Not relevant Not

relevant

11 Not relevant Not relevant Not relevant

Not relevant Not relevant

3 Murray No Not relevant Not

relevant

11 Not relevant Not relevant Not relevant

Not relevant Not relevant

4 Maas No Not relevant Not

relevant

11 Not relevant Not relevant Not relevant

Not relevant Not relevant 5

Mushkudiani

No Not relevant Not

relevant

11 Not relevant Not relevant Not relevant

Not relevant Not relevant

6 Van Beek No Not relevant Not

relevant

7 Not relevant Not relevant Not relevant

Not relevant Not relevant

7 Koopman No Not relevant 6 5 No No No No No

8 Thakkinstian No Not relevant 13 5 No No No Yes, but despite

there were still no statistically significant effects

Yes, IPD was partially reconstructed using summary data in 3 non- IPD studies, and pooled with IPD studies using logistic regression

9 MeRGE2 Yes Inspecting

funnel plot, and calculating the 2 I statistic and a test for heterogeneity 17 12 No Yes No No No

10 Yap No Not relevant 4 4 Yes Not relevant Not

relevant

Not relevant Not relevant

[83]

Continue from previous page Study

ID

First Author

Was there a formal assessment of whether the included/selected IPD studies were a potential biased set of all available studies (akin to publication bias)

If yes, what methods were used and what was concluded? How many studies were asked for their IPD? How many studies provided their IPD? Was IPD obtained from all studies desired? If no (or unclear): was the number of patients in the non-IPD studies given (or % missing)? Were the number of events in the non- IPD studies given (or % missing)? Were details given as to the robustness of meta-analysis results to the inclusion/exclusi on of non-IPD studies? If yes, briefly state what was said?

Was a method used to combine IPD and non-IPD studies? If so, what was it

11 Rovers No Not relevant 19 6 No No No Yes, 6 out of 10

studies included in the meta analysis, and the 4 excluded trials would have not changed the results of the meta-analysis.

No

12 Trivella No Not relevant 38 18 No No No No No

13 MeRGE1 No Not relevant 32 18 No Yes No No No

14 Noordzij No Not relevant 15 9 No No No No No

15 Lanterna Yes Funnel plot for

asymmetry and regression asymmetry test

8 7 No Yes Yes No No

16 Downing No Not relevant 6 4 No Yes No Yes, they

compare the meta-analysis result from the 4 IPD studies with the published results from the 6 studies

No

17 Sylaja No Not relevant Unclear 4 Unclear Not relevant Not

relevant

Not relevant Not relevant

18 Schaich No Not relevant 8 8 Yes No No No No

19 Warkentin No Not relevant 8 7 No No No No No

20 Goetz No Not relevant Unclear 11 Unclear Non-IPD

studies not mentioned Non-IPD studies not mentioned Non-IPD studies not mentioned Non-IPD studies not mentioned End of Table 2.7

[84]

Figure 2.8: Summary of the challenges facing researchers conducting an IMPF

Identifying all relevant studies

• Unavailability of IPD in some studies

• Time-consuming and costly nature of obtaining, cleaning and analysing the IPD.

Issues within individual studies

• Dealing with skewed continuous variables and possible outliers.

• Inability of IPD to overcome deficiencies of original studies, such as being retrospective rather than prospective, being too small for a multivariable analysis, missing important confounders, missing participant data or being of low methodological quality, etc.

• How to assess the quality of studies identified

• Dealing with the continuous variable if the linearity assumption is not achieved.

Heterogeneity between studies

• Different definitions of disease or outcome.

• Different participant inclusion and exclusion criteria.

• Different methods of measuring for the same prognostic factor.

• For survival data different lengths of follow-up

• Factors measured at different points in time or at different stages of disease across studies.

• Different (or out-dated) treatments strategies, especially when a mixture of older and newer studies is combined.

• Insufficient information about treatment for some of the studies.

Statistical issues for meta-analysis

• Missing data, including: missing factor values and outcome data for some participants within a study, and unavailable factors in some studies.

• Inability to adjust prognostic effects for a consistent set of adjustment factors in each study

• Imposed choice of cut-off levels when individual studies categorise their continuous variables and/or categorise their continuous outcomes in their provided IPD

• Difficulty in using a continuous scale for continuous factors in meta-analysis when some studies give IPD give values on a continuous scale and others do not (e.g. see Rovers et al.97)

• Considering whether it is sensible and/or possible to investigate differential prognostic effects in subgroups

• Potential for study-level confounding when assessing whether study covariates (e.g. year of publication) modify the prognostic effect.

• Difficulty of interpreting summary meta-analysis results in the presence of heterogeneity across studies.

• When and how to account for clustering (‘one-step’, ‘two-step’ or just treat as all one study)

• How to combine IPD and non-IPD studies.

Assessment of potential biases

• Potential for publication bias and availability bias

• How to assess the robustness of IPD meta-analysis results to the inclusion/exclusion of studies only providing summary data; and how to combine IPD studies with summary data studies

[85]

Different Methods of Measurement

If the PF has different units of measurement across studies, there is difficulty in combining the IPD studies for the PF. Of the 20 IMPF articles, seven articles reported the problem of different methods of measurement72 77 78 81 85 86 93; three of these seven articles explained the causes of different methods of measurement of the PF across their IPD studies and explained how they limit and cope with this problem72 78 86. For example, Trivella et al.72 state that ‘there are two main methods for measurement’ - so they do separate analyses for each one.

Even when the same method was used to count microvessel density (i.e., Chalkley vs. all vessels), individual laboratories used very different procedures for measurement of microvessel density. Four of the seven articles that reported the problem of different methods of measurement did not attempt to limit or cope with this problem81 85 93. For instance, Thakkinstian et al.81 stated that “Since the IPD meta-analysis is a retrospective

collaboration, it is difficult to get clinical variables that have been assessed and measured using similar methods across all studies”. So it seems they are aware of it, but haven't been

able to address it.

Missing Data

Missing data occurs when no data are available in PF or outcomes; in particular there are three types of missing data in IMPF articles; first, the missing variables occur when there is no data provided for a certain PF or confounding factors in some studies of the IPD studies provided. For example, in Look et al14. for the 15 IPD dataset, the PAi variable was not available in some of these IPD studies. Second, missing values occurs when some values are missing in a certain PF or confounding factors. For example, in Look et al14. across the 15 IPD dataset, there are 68 missing patients values for uPA across studies. Third, missing outcomes occure when there is some missing values of the outcome for some patients in the

[86]

study. All of the twenty articles reported at least one of these missing data problems. Nine of these twenty articles reported statistical methods to limit the problem of missing data; three of these nine articles directly mentioned to an ‘Imputation Analysis’, and gave a brief information about the methods of doing it74 77 83. Three of nine articles report how they overcome missing data. One of them reported ‘missing value analysis function’ whereby they imputed the missing data per trial using a linear regression method (e.g. Koopman et al.82) ; and in the remaining two articles the authors did not directly report to the statistical methods, they just reported that they imputed the data (e.g. missing outcomes, missing patient level) without any further information75 78 79.