Four process tracing tests

While conducting process tracing, the probability that the hypothesized causal chain leads to the effect will be assessed by four process tracing tests (Beach & Pederson, 2013; Collier, 2011; consulted in CDI, 2015, p. 3); which are based on the principles of certainty and uniqueness (see figure 4). Thus, the tests define whether the evidence is necessary or sufficient for the correlation between cause and outcome. In other words, the test reviews the evidence under assumption that the hypothesized causal sentence holds (X leads to Y in the theorized way) or by stating, that the hypothesized causal mechanism does not hold (an alternative causal sequence explains the effect). For this, the tested hypothesis is either backed by a clue (evidence supporting the hypothesis) or an inference (incident decreasing confidence in or even rejecting hypothesis); whereas – according to the applied (Straw-in- the-wind, Hoops, Smoking gun or Double decisive) test – the clue or inference will be sufficient or necessary in order to approve or disapprove the hypothesis. For the better understanding, the different tests are explained below:

Figure 4: Matrix for the assessment of certainty and uniqueness of an evidence (Collier, 2011).

The first test is called Straw-in-the-wind test. It is the weakest test of the four, because it consists of a neither necessary nor sufficient evidence in order to confirm a hypothesis. Thus, this test represents low uniqueness and low certainty. Accordingly, a Straw-in-the-wind test is not enough to make either an approving or disapproving conclusion on the hypothesis (Collier, 2011). However, if reliable Straw-in-the-wind evidence exists, as for example in this study’s case: a clue that citizen engagement increases due to the program Tejiendo Desarrollo; consequently, the researcher might have a hint for the hypothesis to be approvable. On the other hand, there might also be an inference to this clue, stating that there does not exist any considerable citizen engagement in Costa Rica; thus, the plausibility of the hypothesis would in this case be decreased but not rejected.

Secondly, the Hoops test provides high certainty. As such, passing this test is necessary to confirm a hypothesis and is required in order to declare a theory to be true. Yet, evidence passing the Hoops test is not enough to approve the hypothesis, because it is not sufficient. Nevertheless, weak evidence can be used to somewhat exclude alternative events, as failing the Hoops test can eliminate a rival hypothesis (Collier, 2011). Thus, compared to the Straw-in-the-wind test, passing the hoop test has stronger implications for rival hypotheses: it somewhat weakens their plausibility, without precluding the possibility that alternative hypotheses may be relevant (Collier, 2011, p. 826). As an example, taking the hypothesis that Guillermo Solís, the president of Costa Rica (2014-2018), created public trust: a potential clue is his personal asset as an influencer. However, even if being an influencer is consistent with the hypothesis, it does not demonstrate itself that it is enough to create public trust (inference). Nevertheless, given Guillermo Solís’ ability and strong charisma to gain people’s vote, the hypothesis passes the test. As such, passing the test does significantly increase the confidence that the hypothesis is true; however, is not enough to prove the hypothesis. Yet, in case of the opposite, the hypothesis can be disconfirmed.

Thirdly, the Smoking gun test is sufficient to confirm a hypothesis due to its high uniqueness of the event. Evidence in favour of the hypothesis, passing the Smoking gun test are enough to approve the hypothesis. Hence, evidence failing the test substantially weakens the confidence in the hypothesis (Collier, 2011). Again, taking the hypothesis that Guillermo Solís created public trust: a potential clue is when Guillermo Solís is found with some statistics in his hands proofing that he personally created public trust. Thus, the investigator

can be confident that the hypothesis is true. Even if an inference interrupts this belief, it does not significantly decrease the confidence in the hypothesis due to the high uniqueness of the evidence in hand.

Lastly, the Double decisive test represents the most demanding one. Passing this test is both necessary and sufficient to confirm a hypothesis, as it provides both a high certainty and uniqueness. As such, evidence passing or failing this test can either approve or disapprove a hypothesis; and thus, this test constitutes at the same time the strongest of all and the hardest to pass (Beach & Pedersen, 2013; consulted in CDI, 2015, p. 3; Collier, 2011). Again, taking the same hypothesis as with the Hoops and the Smoking gun test:

assuming the clue that Guillermo Solís has been the only president in the past 10 years and an official OECD and UN report state, that he had been the reason for the rise in public trust in Costa Rica, then the investigator can be confident, that the hypothesis is true as well as all other alternative hypotheses are false. In the case that the report states of the year of the previous president, this inference would be strong enough to disconfirm the hypothesis.

Yet, besides the many positive aspects of this qualitative methods (George & Bennett, 2005), such as the ability to assess social phenomena and to test them respectively with a set of instruments; the sources to be analysed form a crucial role for the outcome of the study. Thus, empirical observations need to be evaluated for their collection, content, accuracy and probability before being qualified as evidence (Beach & Pedersen, 2013, p. 7). In line with this, a brief introduction on the selected sources and their evaluation is elaborated in the following section.