Chapter 5: How should we prioritise among the world’s problems?
4. Distinguishing problem evaluation from solution evaluation
We haven’t yet got to the bottom of the first two questions (what’s the distinction between cause prioritisation and intervention evaluation? What reason is there for engaging in the cause prioritisation in the first place?). In this section, I argue the distinction between stage B of cause prioritisation and intervention evaluation, if there is one, is very thin.
Suppose that we want to determine which of two causes, X and Y, is the priority.
What we need to do at stage B of cause prioritisation is to plug in some numbers for each of scale, solvability and neglectedness and see what comes out the other end.
But it’s unclear there is any way to do this without considering, implicitly or explicitly, particular solutions to the problems at hand. Where else could the numbers come from? More specifically, as we’re trying to do the most good, the relevant comparison is between the best solutions we are aware of for each problem, as opposed to (say) the median solution to the problem. The quality of our analysis will depend on our inputs to the formula being realistic, hence we want to have actual, particular solutions in mind, even if we are just intuitively weighing these up in our heads. Hence, what we’re doing is evaluating particular solutions to given
Value
Resources Value
Resources Less neglectedness
Calculate impact from here
Calculate impact from here
Cause A Cause
B
183
problems. Yet, intervention evaluation is the process of evaluating particular solutions to given problems, so stage B of cause prioritisation isn’t a distinct process.
We might think that although two processes are answering the same question—how cost-effective is a particular solution to a problem?—these are nevertheless somehow distinct. This is perhaps because we don’t have to consider the whole problem (scale) when we evaluate solutions, or the resources going to the problem (neglectedness). But notice that when we evaluate interventions, if you know the cost to solve a given fraction of the problem and the value of solving that fraction, it’s trivial to extrapolate those and work out the problem’s scale; further, to determine the counter-factual impact of the solution, you do have to consider the resources that are going to the problem anyway.
Even though we may think we are referring to two different processes—stage B of cause prioritisation and intervention evaluation—both require the same inputs.
What’s more, not only do both produce an output in the same terms—good done per additional unit of resources—we’ll presumably get the same answer whether we think we’re doing one or the other: it would be very strange if our cost-effectiveness estimate of contributing extra resources to a problem is not identical to our estimate of how cost-effective the best solution(s) to that problem is. As a further indication these things are not distinct, notice that we could easily relabel any of the figures I introduced as representing the effectiveness of causes, as representing the cost-effectiveness of interventions.
I imagine an advocate of the EA method might try to insist there is a meaningful distinction between stage B of cause prioritisation and intervention evaluation.
There are two distinctions we might draw. First, the former is done intuitively—the three factors are combined in the head to make comparisons—whereas the latter
184
necessarily involves making explicit, quantitative cost-effectiveness assessments.
Second, the former is an assessment of the best solution to the problem, whereas the latter is any assessment of some solution to a problem.
We can grant these distinctions, but it is still the case that stage B of cause prioritisation and intervention evaluation both consist in the same task: evaluating particular solutions to problems. Hence, if we assumed there was some deep difference between them, we are on thin ice. I note the second distinction is somewhat awkward: what follows from it is that if we use the three-factor framework to determine the cost-effectiveness of a problem and do this in our heads, it’s ‘stage B of cause prioritisation’, but if we get out our calculators we’re suddenly doing ‘intervention evaluation’. Note that even if we do stage B in our heads, it is still a quantitative comparison of marginal cost-effectiveness that we are making.
What this analysis suggests is that we should conceptualise the EA method for setting priorities as having three steps:
1. ‘Screen out’ problems where it’s clear all their solutions are cost-ineffective.
This is done by appealing to one or more of scale, neglectedness, and solvability individually.
2. Make an intuitive cost-effectiveness evaluation of the most promising solution(s) to each problem. This is done by combining scale, neglectedness and solvability.
3. Make explicit, quantitative cost-effectiveness evaluations of particular solutions to problems.
These steps map onto (what I’ve called) stage A of cause prioritisation, stage B of cause prioritisation, and intervention evaluation, respectively.
185
We might have thought that evaluating causes consists only in assessing the problem as a whole; that is, all possible solutions to it. What follows from the preceding analysis is that once we’ve done the initial screening step and are trying to determine the relative priority of the causes that remain—i.e. how cost-effective resources will be in each—that analysis rests on an assessment of the cost-effectiveness of a particular solution(s) to the problem. Hence, while it may be more natural to make claims such as “poverty is a higher priority than climate change”, it would be more accurate, though less elegant, to say instead “the best solutions to poverty we are aware of are more cost-effective ways of doing good than the best solutions to climate change we are of”.8 When phrased this latter way, it’s clear that the cost-effectiveness of the solutions is doing the work in determining which of the problems is deemed the priority. Two concerns follow from this.
The first worry is that although the scale-neglectedness-solvability framework seems very sophisticated, it is just assessing the cost-effectiveness of particular solutions; our analysis here is only as good as the information we put in. Hence, if we have overlooked excellent solutions or wrongly estimated the cost-effectiveness of those solutions we (intuitively) considered, we will be mistaken about which problems are higher priority. If we thought that cause prioritisation via the three-factor framework resulted in a more holistic evaluation of the problem, we would not have this worry.
8 Peter Singer asks whether a claim such as “poverty is a higher priority than opera houses” requires us to compare the best solutions in each case. I do not think we need to think carefully about the available solutions in each to defensibly make such a claim, but it still seems that we are (implicitly) appealing to the relative cost-effectiveness of the top solutions. To illustrate this, note how odd it would be to claim “poverty is a lower priority than opera houses”, even if one thought the top poverty solution was more cost-effective than the top opera house solution, simply because there are some ways of doing good via opera houses than are more cost-effectiveness than some solutions to poverty.
186
The second concern compounds the first. The EA method invites us to divide the world up into cause ‘buckets’, look for the best item in each bucket, compare what we’ve found, throw away the buckets that seem to have nothing valuable in them, and then look further in the buckets that remain. The issue with this is that is if we have thrown away a bucket because we overlooked something valuable, the method discourages us from looking in that bucket again. The risk is that causes are determined to be low-priority prematurely: if we’d have looked for potential solutions longer, a good one would have been found. In the previous chapter, I argued mental health had been overlooked; this is perhaps part of the explanation.
The practical upshot of the analysis is as follows: if we want to find the most cost-effective ways to do good, and be confident we haven’t overlooked the best options, we need to get stuck into the particular things we can do to make progress on each problem. Incorrectly believing that we can just take a quick look at a problem, consider its scale, neglectedness, and solvability, and thereby gain an accurate picture of a problem’s cost-effectiveness, is liable to lead us to overlook things. It’s these concerns which motivate, in the next chapter, proposing and testing a new approach to cause prioritisation that aims to overcome this issue.
To be clear, I’m not claiming these concerns are ones that (sophisticated) effective altruists are unaware of, or they demonstrate that effective altruist’s prior prioritisation efforts are systematically mistaken and they must go back to the drawing board.9 I am merely noting the concern and highlighting the limitations of the methodology.
9 I say this despite the fact that, in the next chapter, I go back to the drawing board to (re)consider what the priorities are if we want to make people happier during their lives. This is, however, a combination of both (a) wanting to create and test a different prioritisation method and (b) because,
187
It’s unclear if this proposed reconceptualisation is surprising or not. On the one hand, what I suggest seems to be in tension with comments others have made on this subject. For instance, Michael Dickens writes:
The [three-factor] framework doesn’t apply to interventions as well as it does to causes. In short, cause areas correspond to problems, and interventions correspond to solutions; [the three-factor framework] assesses problems, not solutions.” (emphasis added)10
Hence, Dickens implies that it is possible to assess problems without assessing solutions at all, from which it would follow that cause prioritisation (stages A and B) is really distinct from intervention evaluation. As argued earlier, it’s unclear how this could be possible.
Robert Wiblin says of the three-factor framework that:
This qualitative framework is an alternative to straight cost-effectiveness calculations when they are impractical […] In practice it leads to faster research progress than trying to come up with a persuasive cost-effectiveness estimate when raw information is scarce […] These criteria are 'heuristics' that are designed to point in the direction of something being cost-effective.11 If the framework which we use for cause prioritisation is qualitative, then it must be distinct from the quantitative cost-effectiveness estimates we make of particular interventions. What seems to be going on here is that Wiblin is referring to the older, qualitative version of the three-factor framework I mentioned in the previous
as argued in chapter 4, I think effective altruists have been using a non-ideal measure of happiness and this alone prompts a reevaluation of the priorities.
10 Dickens (2016)
11 Wiblin (2016)
188
section. The newer version—the one we’re using—is a quantitative framework where the factors do combine to produce ‘straight cost-effectiveness calculations’ and so the distinction between using the framework and making cost-effectiveness analyses of interventions disintegrates.
More generally, it is somewhat surprising that no one seems to have made it explicit that the priority-setting process should be broken into (at least) the three steps I have suggested. We might have expected someone to offer a clarificatory description of the EA method along the following lines: “first, we ignore the problems with no good solutions; then, we make intuitive judgments of how cost-effective the best solution to each of the remaining problems is; finally, we make some explicit, numerical estimates of those solutions to check our guesses”.
On the other hand, I don’t think anyone has explicitly denied priority-setting can work in the way I have just stated. Indeed, it seems clear, on reflection, that this is how we can and should approach the task. When we decide where to go on holiday, I presume most people make rough judgements about how much they want to visit entire countries, then consider particular in-country destinations in more detail and, at the final stage, compare prices etc. for the different options.
As a further comment against this being surprising, my analysis seems at least compatible with MacAskill’s. In the opening quote, MacAskill states the three-factor framework allows us to make comparisons between causes even if we lack numerical cost-effectiveness assessments. What MacAskill is perhaps claiming is that the scale-neglectedness-solvability framework allows us to do step 2—make an intuitive comparison between problems—without having to go as far as step 3—making explicit quantitative cost-effectiveness assessments; his use of the word ‘heuristic’
seems to support this interpretation. As it happens, it seems plausible that we can
189
and do use three-factor framework in our heads to construct cost-effectiveness lines for different problems: we think about the size and shape of the curve and adjust where we are on the curve to account for what others will do.
As such, it’s not obvious whether I am suggesting something different from what others thought of making explicit something that was previously implicit.
5. Conclusion
I began this chapter by asking: ‘what’s the distinction between cause prioritisation and intervention evaluation?’ This was motivated by the apparently odd suggestion that we can do the former before the latter. I’ve argued there is a sense in which we can evaluate causes(/problems) as a whole: if and when we can evaluate all their solutions – this was stage A of cause prioritisation. If intervention evaluation requires the assessing of particular solutions in some depth, then stage A of cause prioritisation can happen prior to intervention evaluation. Both stage B of cause prioritisation and intervention evaluation require the cost-effectiveness of solutions to a given problem to be determined. If we stipulate that stage B of cause prioritisation can be done intuitively, whereas intervention evaluation requires the writing down of some numbers and then crunching those, the former can be also prior to the latter; I noted this distinction is pretty flimsy. If we don’t stipulate this
‘in-the-head’ vs ‘on-paper’ distinction, then the two are the same process and hence stage B of cause prioritisation is not prior to intervention evaluation.
The second question related to why we should engage in cause prioritisation rather than leap straight to intervention evaluation. Following what we just said in the last paragraph, if we can evaluate causes as a whole—i.e. what happens at stage A—that
190
can usefully save time. We can now see there’s not much difference between starting at stage B of cause prioritisation or with intervention evaluation in any case.
In this chapter, I’ve set out to address some of the outstanding questions about how the EA priority-setting method functions. What I’ve suggested here is a modest reconceptualisation of the EA method. I have not tried to argue the EA method is mistaken in some deep way, because it is not. Rather, I have tried to clarify something that seemed plausible but the details of which were murky. A practical conclusion emerged from the reconceptualisation: while we might have thought the three-factor cause prioritisation method assessed problems ‘as a whole’, this is only partially true: once we’ve sifted out unpromising problems (ones with no good solutions), our analysis of how problem A compares to problem B is nothing more and nothing less than a comparison of the best solutions to problems A and B that we’ve considered. Hence, if we want to find the most cost-effective ways to do good, we need to look carefully at these solutions.
191