Confirmation bias is defined as:
The tendency to notice and to look for what confirms one's beliefs, and to ignore, not look for, or undervalue the relevance of what contradicts one's beliefs.
When it is pointed out to PSI researchers who claim to have successfully demonstrated ESP, that hundreds of non-PSI researchers have tried to replicate their results and failed, they sometimes attribute this to the ostensible influence that the attitude of both experimenter and subject can have over the results. An experimenter who is hostile towards the concept of ESP, they claim, can exert a negative influence over the results, thereby
counteracting any positive ESP effects that may be present. This is one of the many "outs" PSI researchers have developed that enable them to attribute negative results to extraneous causes, and preserve only the data that is favorable to their preferred hypotheses.
We see exactly the same thing happening in the XP community's evaluation of experience reports from the field.
When presented with a claim of success using XP, the community accepts it without challenge, for it is a welcome confirmation of pre- existing beliefs. However, a claim that XP has failed is an unwelcome affront to their personal convictions. So these claims are scrutinized until an "out" is found - some extraneous factor to which the blame for failure can be assigned. If all else fails, one can claim, as PSI researchers are wont to do, that the attitude of the participants is to blame for the failure.
To illustrate, consider the tabulation below of the four types of experience reports that the XP community can be presented with. The columns represent the two basic modes of XP usage – full and partial. Either you're doing all the XP practices or you're only doing some of them. The rows represent the claimants assessment of the project outcome – success or failure. The table shows the interpretation an XP proponent can confer upon each type of experience report so as to confirm their pre- existing belief in XP.
Full XP Subset of XP
Success XP has succeeded! See how powerful XP is? Even a
subset of the practices can yield success!
Failure You weren't doing xxx as well as you could have, or You weren't committed enough, or
There's something wrong with you etc.
You weren't doing all the practices, so you weren't really doing XP.
The XPers have all their bases covered. No matter what the experience report, there is no need to ever cast doubt upon XP itself – there are always rival causes to be blamed.3 In this way, XP becomes non-falsifiable.
Conclusion
There is an "essential tension"4 between being so skeptical of new technologies and methods that we miss the opportunity to exploit genuine innovations, and being so credulous that we are ourselves exploited by those willing to subjugate integrity to self-interest. Given the software industries' history of fads, trends and passing enthusiasms, we would be wise to approach claims of innovation with caution – where those claims are accompanied by fanaticism and zeal, doubly so. As Thomas Henry Huxley warned:
Trust a witness in all matters in which neither his self-interest, his passions, his prejudices, nor the love of the marvelous is strongly concerned. When they are involved, require corroborative evidence in exact proportion to the contravention of probability by the thing testified.
There is no logical basis for dismissing out of hand every "next big thing" that comes along. But an awareness of confirmation bias, positive outcome bias and their contribution to the development of false beliefs should encourage us to seek evidence beyond that provided by popular media and effusive testimonial.
* First published 5 May 2004 at http://www.hacknot.info/hacknot/action/showEntry?eid=53 1 Cited in Voodoo Science, Robert Park, Oxford, 2000
2 The Skeptic’s Dictionary, Robert Carroll, Wiley, 2003 3 http://c2.com/cgi/wiki?IfXpIsntWorkingYoureNotDoingXp 4 Why People Believe Weird Things, M. Shermer, Owl Books, 2002
Thought Leaders and Thought Followers
* Fowler On "Appeals To Authority"For a brief, shining moment there was hope. Through the exaggeration and braggadocio that so permeates the conversation of the Agile community, there came a fleeting glimpse of self-awareness – a flash of social perspective that could have precipitated a greater moderation and rationality in the methodological discourse. And then it was gone – swept aside by the force of yet another ill-considered generalization.
I’m referring to a recent blog entry by Martin Fowler entitled
AppealToAuthority.1 In this entry, Fowler relates how he occasionally receives the comment "When a guru like you says something, lots of people will blindly do exactly what you say." Fowler denies the existence of such an effect, and counters that what appear to be appeals to authority may really be just an artifact of lazy argument or sloppy self-expression.
The argument from authority is everywhere in the Agile and XP communities, and is a far more potent force than Fowler seems to appreciate. Here are just a few ways that the various so-called "thought leaders" and "spokesmen" employ direct and indirect appeals to authority.
• Statements prefixed with "In my experience", combined with the suggestion that this experience is extensive, are attempts to cast the speaker as a seasoned veteran whose word should be taken seriously. Having many years of experience only establishes that one is old, not that one is correct.
• Sweeping statements and broad generalizations can make for powerful-sounding oratory, and suggest that the speaker possesses some kind of absolute knowledge i.e. that they are simply declaring information that they know to be factual. By abandoning the uncertainty and qualification, the speaker sacrifices accuracy for the sake of impact and elevates opinion to fact.
• By inventing and promulgating cute slogans, folksy homilies and other media-friendly sound bites, speakers encourage others to quote them verbatim and dogmatically. Such quotation invests the statement, and thereby the speaker, with a faux authority.
• With rare exception, the aforementioned comment from Fowler’s being one such case, the "thought-leaders" and "spokesmen" rarely acknowledge, let alone reject, their decoration with such grand titles. There is no attempt to discourage the use of such titles, beyond the occasional token self-deprecation.
• Speakers claiming to represent the opinions and experiences of a group are naturally encouraging a view of themselves as leaders. Such speakers will not hesitate to claim "The Agile community believes X" or "The XP community does X", even though the communities in question have not been consulted or surveyed, and in fact may have wildly varying and inconsistent views on the matter.
Fowler's claim that appeals to authority are not a significant influence strikes me as disingenuous. Not only are such appeals frequent, they are at the very heart of the rhetoric. It should be kept firmly in mind that those most outspoken in this space are almost always consultants specializing in AM/XP.2 Consultants make their money by promoting themselves as authorities on some subject, so that others will hire them for their perceived expertise.
Ruin Your Career With Agility
An interesting blog entry, author unknown, came to my attention recently. Entitled How Agile Development Ruined My Career (Sort Of)3 it is the story of a Senior Director’s attempts to introduce Agile work practices into a company, and the consequences for himself. I have commented on the blog itself, and the XP fraternity has just begun to dissect it on comp.software.extreme-programming4 (posted 23 May 2004) which should make for entertaining reading.
* First published 24 May 2004 at http://www.hacknot.info/hacknot/action/showEntry?eid=55 1 http://martinfowler.com/bliki/AppealToAuthority.html
2 Agile Methods / Extreme Programming
3 http://www.undefined.com/ia/archive/000158.html