6.3.2 ‘Strength of causal inference’
N Strong causal
6.3.5 Results if ‘Strong’ not used for ‘no effect found’
To see what the results would look like if a causal inference was defined as Li et al.715 had
defined it — only for conclusions after evidence of ‘an effect’ was found, with the weakest rating given to conclusions of ‘no effect’ — all ‘Strong’ causal strength ratings were changed to ‘Not strong’ if a difference in the average outcome between the intervention groups was not observed.
The shift of 41 studies that did not find evidence of an effect, from a rating of ‘Strong’ to ‘Not strong’ causal inference, makes an obvious difference to the relative numbers in Table 6.16. This shift also had an impact on some of the comparisons made in this chapter.
Table 6.16 Alternate definitions of a causal inference and group outcomes comparisons Values referred to in the text are highlighted in magenta
Causal inference definition can encompass
Group outcomes comparisons N ‘Effect’ and ‘No effect’ (Ch.6) ‘Effect’ only (Li et al.) ‘Not strong’ : ‘Strong’ ratio
Similar (null result) 92 51 : 41
92 : 0Different (evidence of causal effect) 196 122 : 74
122 : 74For example, a substantial difference occurred with the relatively low number of articles that did not use a multivariable regression method (Table 6.17). By seeming chance, the ratio has been reversed. Combined with the p-value moving to the other side of 0.05, the inference would change to either one of ‘no effect’, or one where the inference is not clear; as opposed to the weak evidence we found, using our definition of a causal inference, of a much greater proportion of those who didn’t use multivariable regression also favouring strong causal language in the conclusion.
Table 6.17 Alternate definitions of a causal inference and multivariable regression Values referred to in the text are highlighted in magenta
Definition of a causal inference can encompass Multivariable
regression N ‘Effect’ and ‘No effect’ (Ch.6) ‘Effect’ only (Li et al.) Change in inference ‘Not strong’ : ‘Strong’ ratio
Not used 31 12 : 19
19 : 12 reversedUsed 257 161 : 96
195 : 62 no change6.3 Results
However, with most of the comparisons in this chapter exhibiting larger numbers in each category, the shift of some articles from the ‘Strong’ to the ‘Not strong’ column was more evenly balanced, with a similar proportion of articles shifting in each category. Nevertheless, three other comparisons that did change with the different definition of a causal inference are worth noting. The first involved whether the methodology was considered adequately reported (Table 6.18), with the inference changing from an association with the ‘strength of causal inference’ to no clear association, when the definition changes.
Table 6.18 Alternate definitions of a causal inference and reporting of methodology Values referred to in the text are highlighted in magenta
Definition of a causal inference can encompass Description of
methodology N ‘Effect’ and ‘No effect’ (Ch.6) ‘Effect’ only (Li et al.) Change in inference ‘Not strong’ : ‘Strong’ ratio
Inadequate 56 27 : 29
39 : 17 equal to unequalAdequate 232 146 : 86
175 : 57 no changeP-value for difference* 0.04 0.37
* Chi-squared test
Table 6.19 Comparison group type with alternate definitions of a causal inference Values referred to in the text are highlighted in magenta
Definition of a causal inference can encompass Comparison group
intervention type N ‘Effect’ and ‘No effect’ (Ch.6) ‘Effect’ only (Li et al.) Change in inference ‘Not strong’ : ‘Strong’ ratio
Active control 118 61 : 57
84 : 34 equal to unequalInactive control 170 112 : 58
130 : 40 no changeP-value for difference* 0.02 0.31
In Table 6.19, changing the definition of a causal inference likewise changed the inference for having an inactive control group defined in a study, from a possible effect to not clear. Lastly, SPSS appears to have had the greatest proportion of articles change from ‘Strong’ to ‘Not strong’, and with it the inference that SPSS users used stronger causal language in conclusions, on average, than the users of SAS, Stata or R.
Table 6.20 Software type with alternate definitions of a causal inference Values referred to in the text are highlighted in magenta
Definition of a causal inference can encompass
N ‘Effect’ and ‘No effect’ (Ch.6) ‘Effect’ only (Li et al.) Change in inference ‘Not strong’ : ‘Strong’ ratio
Not Specified 42 22 : 20
29 : 13 equal to unequalOther 20 10 : 10
13 : 7 equal to unequalR 35 22 : 13
26 : 9 no changeSAS 110 71 : 39
82 : 28 no changeSPSS 70 33 : 37
47 : 23 equal to unequalStata 56 41 : 15
47 : 9 no changeP-value for difference* 0.04 0.33
* Chi-squared test
6.4 Discussion
The idea motivating this review is that scientific progress depends not only on researchers avoiding bias, but that they also convey the uncertainty that remains when a study is reported. In summary, after a brief review of causal language in the literature, our first objective was to rate the ‘strength of causal inference’ implied in the final study conclusions. The second objective involved assessing whether the ‘strength of causal inference’ might be affected by the use of more advanced statistical techniques, as well as with other study
6.4 Discussion
features associated with the design, interpretation and reporting. Using a broader definition of a causal inference than many researchers might tend to use, this review suggests that 40% of 288 health intervention cohort studies implied relatively ‘Strong’ causal inference in study conclusions, as opposed to ‘Not strong’. We found that articles using either multivariable regression, propensity score methods (compared to other multivariable regression methods), or sensitivity analysis, were more likely to express ‘Not strong’ causal inference in study conclusions. Some associations were also noted with other study features, such as whether an inactive control intervention was used, and whether the outcome was a benefit to health or a harm. Given the evidence of bias summarised in Chapter 4, some of these cohort study conclusions are probably wrong, but confidence that exceeds the uncertainty will only compound any effect of evidence that is false.