Calculation of the post confidence

In combination with the prior confidence, the assessment of sensitivity and type I error enables the calculation of the post confidence. To ensure the comparability of values given for sensitivity and type I error across mechanism parts, the evaluation team developed a coherent scale for assessing sensitivity and type I error for each test. One should note that the assessment is discretionary and constitutes a quantifi- cation of qualitative results. The post value is calculated according to the following formula:

Posterior = Prior*Sensitivity/(Prior*Sensitivity + Type I Error*(1 – Prior))

Sensitivity

Definition: The probability of getting a confirmation, from the respective entities, when the hypothesis is true. When the sensitivity is high, the probability is high, too.

Example: The probability of the government admitting that social expenditure increased, when it actually increased, is very high because the government has very little incentive to hide the fact. However, the prob- ability of the government admitting to corruption is low because they have an incentive to hide these facts.

Determinants: Two indicators determine the sensitivity, the likelihood that the actors providing the evi- dence know the answer, and the incentive to conceal it. In other words, the evaluation team assigns values depending on if the interview partner is expected to know the answer or not (see Table 21). The second indicator measures the incentive to conceal the truth by the interview partners. Depending on the topic and the source, the incentives range from high to medium to low, with the equivalent values. To calculate the sensitivity, both indicators are added up.

Table 21 Scale for assessing sensitivity

Expected to know the an- swer

Somewhat likely to know the answer

Somewhat unlikely to know the answer

Knowledge 0.90 0.70 0.45

High incentive to conceal Medium incentive to conceal

Low incentive to conceal Incentive to conceal -0.50 -0.30 -0.05

Source: own Type I error

Definition: The type I error, on the other hand, is the probability of getting a confirmation from the respec- tive entities when the hypothesis is not true.

Example: Using the example of public expenditure, the government has an incentive to state that public expenditure increased, although it did not actually increase.

Determinants: Three indicators (can) influence the calculation of the type I error: triangulation, contradic- tory evidence and the likelihood of biased statements. The overall confidence in the evidence for each test depends on the mix of sources. Confidence is generally higher if the evidence comes from interviews with different stakeholder groups, such as donor and partner representatives and civil society representatives. The mix of sources influences the likelihood of biased statements. For example, if asked how the policy dialogue has developed since the exit, government representatives and donors are assumed to have a higher likelihood for a positive bias than CSOs, because the former are responsible for the quality of the dialogue and might feel induced to euphemize the reality. As shown in Table 22, this likelihood for a bias can be either high, medium or low. If the evidence is coming from only one source types, the type I error is increased by

0.1 to account for the lack of triangulation. In case some pieces of evidence contradict each other, the type I error is increased by 0.15 to account for contradictory evidence.

Table 22 Scale for assessing type I error

High incentive for a bias Medium incentive for a bias Low incentive for a bias

Likelihood of biased statements

0.20 0.15 0.05

Some contradictory evi- dence

No contradictory evidence

Contradictory evidence + 0.15 + 0.00

<2 source types >2 source types Triangulation + 0.10 + 0.00

Source: own

Example for the calculation of process tracing

The hypothesis for the mechanism part is that donors exert control through conditionality linked to budget support payments (see Table 23). The first test assesses if conditionality is used. Since existing budget sup- port documents strongly suggest that conditionality was used, the prior was set to 0.9 for the first hypoth- esis. The sensitivity value is 0.85 because the interview partners (both donor and partner representatives) were expected to know the answer and have a low incentive to conceal the information since this infor- mation is not controversial (0.9 + (-0.05) = 0.85). In comparison, the prior for the test ‘conditionality trig- gered the intended action by the government’ is rated as 0.6 because conditionality has in other country cases not always triggered the intended action and it is therefore only somewhat likely (based on the theo- retical intervention logic). The sensitivity value is composed as follows: the interview partners are somewhat likely to know the answer (mainly statements by donor representatives), but have a low incentive to conceal, which results in a value of 0.7 + (-0.05) = 0.65. The type I error is rated 0.05 for all three tests because there is only a low incentive for bias and neither low triangulation nor contradictory evidence are present. Table 23 Preparation of data to calculate the post value

Mechanism Test Test type Prior –

value Sensitivity – value Type I er- ror – value Post value Donors exert control through con- ditionality linked to BS payments

Conditionality was used Hoop 0.9 0.85 0.05 0.994

Not fulfilling the conditions had

consequences Hoop 0.6 0.85 0.2 0.864

Conditionality triggered in- tended action by the GRZ

Smoking

gun 0.6 0.65 0.05 0.951

Source: own

Using the above formula, the three values – prior, sensitivity and type I error – give the post confidence of each test. The confidence in the mechanism part, which can consist of multiple tests, is calculated using a weighted average. This calculation is explained in the following section of the Annex (7.10).