Bias in search and inference cannot be studied in isolation

Though bias in search and inference have generally been studied as separate phenomena, perhaps they should not be - it is arguably only with certain combinations of search and

inference that the real problems arise. Being biased in how one searches for informa- tion isn’t so problematic if one interprets and updates on that information rationally.

Similarly, a bias in how one interprets information is at least less of a problem if one starts with balanced and unbiased information. It is therefore difficult to draw any con-

clusions about confirmation bias as a broad phenomenon without understanding both bias in search and inference, and how they interact. As Klayman (1995) points out, the

tendency to study bias at different stages of reasoning independently, and then to claim each demonstrates a confirmation bias, is perhaps one of the biggest problems with this

literature.

This problem is particularly apparent in the selective exposure literature. I argued that

mixed findings may fundamentally be because ‘selective exposure’, as typically defined, is a poor measure of confirmation bias - whether someone seeks out more confirmatory

evidence or not tells us little about whether they are reasoning in ways biased towards the current hypothesis, since this in turn depends on their motivations and how they

draw inferences from that information. A person might seek out balanced evidence and yet still be guilty of confirmation bias if they evaluate confirmatory and disconfirmatory

evidence by unreasonably different standards. Conversely, a person might ‘selectively expose’ themselves to a great deal more supportive evidence, but if they are aware of

this tendency and accordingly hesitant to draw any strong conclusions from it, then they do not necessarily exhibit a confirmation bias.

Mckenzie (2004) makes this point more formally - arguing in line Klayman (1995) that neither bias in testing hypotheses nor in the evaluation of information, in themselves,

necessarily lead to confirmation bias - but certain combinations do. McKenzie discusses one such combination - a positivity bias in how one tests hypotheses, plus insensitivity to

differences in the diagnosticity of different answers to questions - explaining how neither tendency on its own creates a combination bias, but together they do.8

8

McKenzie also goes on to argue that even this combination leads to bias less often than might be supposed, since the bias seems to decrease when the materials used are familiar (rather than abstract.)

A positivity bias in testing hypotheses is essentially the same as the positive test strategy discussed previously - preferring to ask questions for which a ‘yes’ answer under the

current hypothesis is more likely. Consider 2.5 below - where one’s task is to determine whether a patient has disease A or B, and one can choose to do tests from 1 to 4. The

probabilities in the table indicate the probability that the test will yield a positive result if either disease A or B is present. Positivity bias suggests that, if a doctor currently

thinks disease A is more likely, he will choose to do test 4 - since this is most likely to yield a ‘yes’ answer under hypothesis A - and that he will prefer test 1 if he favors

disease B.

Test Disease A Disease B

1 50% 90%

2 50% 60%

3 50% 10%

4 90% 50%

Table 2.5: Probabilities of observing positive test results given diseases A and B

Mckenzie (2004) explains how this positivity bias - though it looks like a form of con- firmation bias - will not necessarily lead one to irrationally strengthen confidence in the

focal hypothesis, so long as one updates rationally (i.e. in accordance with Bayes’ rule) based on the evidence obtained. Assume that the doctor currently believes the patient

has disease B, and chooses to do test 1. Although a positive result to test 1 is more likely than a negative result, a positive test result is also less diagnostic on account of being more likely - and so should cause the doctor to update his belief less. On average, then, choosing test 1 should not cause one to update in favour of diagnosis B.

In general, even if people display some ‘bias’ towards information expected to support the focal hypothesis, if they are sensitive to the diagnosticity of information, they should

not end up overconfident in that focal hypothesis. This is because likely outcomes are less diagnostic than likely ones - so even if supportive evidence is more likely, unsupportive

evidence should cause one to update more, which on average balances out.

Being insensitive to how diagnostic information is does not, in itself, lead to confirma-

the questions asked. Insensitivity to differential diagnosticity should then affect both hypothesis-confirming and disconfirming evidence equally. What does result in confir- mation bias is the combination of ‘positive tests’ and insensitivity to the diagnosticity of information. Asking questions more likely to ‘support’ the focal hypothesis means that a

result supporting the focal hypothesis should be less diagnostic, since that result is more common. Insensitivity to diagnosticity means that people will, in addition, overestimate

how diagnostic this supportive information is - and so people are both more likely to encounter supportive information, and more likely to overweight it. More intuitively:

asking questions in ways that make you more likely to encounter supportive information does not lead to bias if you account for the fact that a supportive answer was more likely

in how you weigh that evidence - and failing to discriminate between the diagnosticity of evidence does not lead to bias if this failure to discriminate affects supportive and

conflicting evidence equally. It is only when these two ‘biases’ are combined, that gen- uine confirmation bias, and overconfidence in the focal hypothesis, results. Almost all

research on confirmation bias discussed in the literature (including Nickerson, 1998, , which is often cited as conclusive evidence for confirmation bias) fails to appreciate this

important point.