Principles for developing the method

The key aim for this chapter was to develop a method for building DAGs from scientific literature. Evidence synthesis is fundamentally concerned with very similar issues and thus is worth brief discussion. In short, evidence synthesis may be defined as the collation and systematic integration of evidence from multiple sources with the goal of comprehensively characterising the scientific understanding of a particular topic (Hanley et al., 2016, Murad et al., 2016). Of the numerous evidence synthesis methodologies that exist, (Sutton et al., 2009) systematic reviews and meta- analysis are perhaps the most common (Higgins and Green, 2011). Clearly then, evidence synthesis protocols share powerful commonalities with the aim to systematise scientific knowledge for building DAGs. However, after appraising the literature, it became clear that evidence synthesis was only able to inform ESC-DAGs very indirectly. There were two mains reasons for this. First, systematic reviews and meta-analyses tend to only be interested in the relationship between very few

independent variables (often only one) and a single dependent variable. In other words, they collate evidence on directed edges between only a handful of concepts, while building a DAG is as concerned with how these concepts relate to the wider causal structure. Secondly, evidence synthesis methods are overwhelmingly concerned with characterising the evidence narratively or numerically, rather than graphically. Thus, existing evidence synthesis protocols are not well suited to the task of building DAGs. However, their more general emphasis on systematic, transparent and reproducible protocols was carried forward when designing ESC-DAGs.

As there was no explicit guidance to refer to, and as the related guidance in the evidence synthesis literature was not well suited to the task of appraising numerous directed edges, ESC-DAGs was mainly developed in reference to several underlying principles of graph theory repeated throughout the literature (Morgan and Winship, 2007, Pearl et al., 2016, Greenland et al., 1999, Textor et al., 2011, Pearl, 2009, VanderWeele, 2015). They are discussed below. However, these principles can prove restrictive in an applied setting, especially when research projects are interested in complex social phenomena typical of observational research (such as parental influences on adolescent alcohol harm). Arguably, the restrictiveness of these principles is a key reason why no methods for building DAGs currently exist. Nonetheless, each fundamentally helped shape ESC-DAGs.

1. DAGs should be built independently of data because causal phenomena exist independently of the ability of social scientists to measure them. There are at least two characteristics of data that should be ignored when building a DAG - the variables themselves and the timing between their measurements. It must first be assumed that all nodes in the DAG can be replaced with a variable that adequately captures the concept in question. Thus, when a DAG is built and then brought to bear on data, unmeasured confounding may be systematically conceptualised by

52 comparing the DAG to the data. Second, the timing of measurement between the variables should also be ignored. Thus, it is the relative timing between nodes on a conceptual level that is important – it must be feasible for the purported cause to precede the effect. Given these assumptions, building a DAG independently of data means that it could be used to inform analysis across multiple different data sources, across different research teams, or even to inform data collection itself.

However, focused analysis of a single longitudinal dataset is common in academic,

governmental and other research, and as such building a bespoke DAG without consideration of that dataset introduces a high degree of redundancy to the process. Specifically, many directed edges may have only one plausible direction, but be assessed for both (e.g. if variable V2 is only measured 36 months after variable V1 then the directed edge V2 → V1, even if plausible on a conceptual level, is technically implausible in terms of cause and effect).

2. DAGs should be ‘saturated’ such that every possible relationship between the nodes in the DAG has been assessed, and relatedly, it is a much stronger assertion to omit or delete a directed edge than it is to include the same edge.

This has at least two implications. Firstly, assessing every possible relationship results in a high volume of assessments. To see why, consider how each time a new node is introduced to a DAG the volume of new assessments required would equal the total number of nodes already present in the DAG. Thus, if there were already 49 nodes, then 49 relationships would need to be assessed if a 50th _{node was added. Similarly, if a 51}st _{node was added, then another 50}

assessments would be required. Thus, 99 relationships would have to be assessed simply to introduce two new nodes. If each relationship is assessed in bi-directionally instead of just uni- directionally, the number of assessments doubles. Figure 4-1 below demonstrates graphically how adding a node results in a non-linear increase in the number of required assessments.

Secondly, if the only directed edges that are not included are those that the researchers are sure do not exist (itself an uncertain concept), then the number of the directed edges, like the number of assessments required to select them, could be very high. This could be problematic in numerous ways, for example when conducting statistical analysis using methods that are susceptible to overfitting. A DAG with 50 nodes might suggest that 48 variables should be adjusted for to analyse the relationship between the remaining pair. It also threatens one of the key strengths of DAGs – their ability to use visualisation to reduce the cognitive load required to conceptualise models during complex data analysis. Having over a thousand directed edges in a DAG may be counterproductive in this regard.

53 Figure 4-1: Increasing number of directed edges per new node added

3. DAGs should be built using methods which emphasise the transparency of decision- making taken in designing the DAG, as DAGs are intended to explicate the causal assumptions of the researcher(s). If a DAG has 50 nodes, then there would be a maximum of 1,225 directed edges (1+2+3… 20+21+22… 48+49+50… etc.). However, a database of 1,225 decisions, while a useful resource, may be cumbersome compared to other options such as relying on the DAGs themselves to communicate the decision-making taken during the DAG-building process. This is discussed below.

4. DAGs should be informed by the empirical literature where possible. Again, this seems eminently sensible. However, it could equate to some form of evidence review for every directed edge in a complex DAG. This is not a practical expectation in the context of most research. The approach taken in ESC-DAGs is to focus on the evidence of the focal

relationships of the research question, both in terms of the how an exposure of interest effects an outcome, and of the wider confounding structure employed in the literature when investigating this effect.

Rather than a procedural description of how these principles were used to develop ESC-DAGs, the method is demonstrated first before revisiting these topics at the end of this chapter. However, there are two secondary aims that were not related to DAGs or evidence synthesis per se that should be noted. First, explicit consideration was given to how to make the method appealing to the DAG community. If not for application, then at least for discussion. A strategy with two parts was used. On one hand, the targeted journal for the publication had an established record of high-profile

0 500 1000 1500 2000 2500 0 5 10 15 20 25 30 35 40 45 N u m b er o f a ss es sm en ts

Number of nodes in DAG

Uni-directional Bi-directional

54 publications in causal inference methods. This was to try and ensure that the method would reach the right audience. On the other hand, ESC-DAGs was designed to be ‘modular’. In other words, the protocol operates across several discrete processes, each of which pertains to distinct tasks. As such, the process pertaining to each task may be ‘swapped out’ for an alternative if users have a preference. This flexibility could also see adaptation of ESC-DAGs or innovation of competitor methods.

The second aim was that ESC-DAGs should be accessible to quantitative researchers who are not familiar with DAGs. This was informed by how this thesis was envisaged as a ‘translational’ project which might contribute to bringing DAGs and counterfactual causal inference closer to mainstream health and social science. For example, both the publication and this thesis take a didactic stance in several instances. Further, both also make extensive use and recommendation of software such as DAGitty. One reason for doing so is that software automates the arduous task of manually applying the backdoor criterion. In other words, it allows users to build informative DAGs in an applied setting without needing to understand the ‘nuts and bolts’ of the process. In this way it is analogous to how any regression command in statistical software can be conducted quickly by an applied researcher who can interpret the results correctly without needing an in-depth knowledge of regression minutia. Thus ESC-DAGs may also be applied without the need to have a full understanding of DAGs or previous experience of applying them.