Living with Machine Intelligence: 1990s - present

Not all writing on algorithmic culture originates in analytic philosophy and computer science. With the rise of the personal computer and growing profile of the tech sector in the global economy, more and more disciplines took an interest in them. Perhaps most prominent among these new critics of technology in culture is Donna Haraway, whose wide ranging work

56. See Eleanor Rosch et al., “Human categorization,” Studies in cross-cultural psychology 1 (1977): 1–49 and George Lakoff, Women, Fire and Dangerous Things: What Categories Reveal about the Mind (University of Chicago Press, 1987)

manages to tie the futility of the Star Wars missile defense program to a feminist critique of positivistic knowledge and various flavors of post-humanism. While her work remains enormously influential in all manner of studies of technology in culture, it is Haraway’s readers who tend to critique the relationship of computers to culture more specifically – Haraway’s deployment of the cyborg concept, if anything, almost seems to assume a certain frictionless compatibility between machines and people, a compatibility that is mobilized in service of a broader feminist project.⁵⁷ A telling example of Haraway’s legacy, but where people and computers are kept at odds rather than brought together, is Gilles Deleuze’s 1992 “Postscript on the Societies of Control,”

which applies a postmodernist lens to a problem that, so far, we have seen only analytic approaches to.⁵⁸Here, Deleuze argues that Foucault’s “society of disciplinarity” has, in the age of the personal computer, given way to a society of “control.” If disciplinarity was bad, control is even worse:

Deleuze calls it the “new monster” and exhorts readers to “look for new weapons.” This monster is connected specifically to computers, which have replaced the traditional tools of the 19th century (“levers, pulleys, clocks” – presumably the tools of Foucault’s “disciplinary” paradigm) and which usher in a new pernicious form of capitalism. In a post-disciplinary, computerized society, all production is relegated to the third world, while the wealthy engage in financial trade wholly abstracted from material goods. Deleuze’s engagement with computer science is of a general nature – its claims in that regard are vague enough to have been made as reasonably about 1970s and 1990s technology, and the relationship between personal computing and economic trends that he mentions are simply general features of economic globalization – but the article does mark a new kind of pessimism in humanistic writing about technology that is not found in early critiques. Where Weizenbaum, for example, is merely confronting the alarming prospect of man’s transformation into a “clock-work” (and hoping to forestall it), Deleuze regards the transformation as a fait accompli. He does not call for any specific action, which in his voice

57. “People are nowhere near so fluid,” she writes. Whereas “cyborgs are ether, quintessence.”Donna Haraway, “A Cyborg Manifesto: Science, Technology, and Socialist-Feminism in the Late Twentieth Century,” in The Postmodern Turn: New Perspectives on Social Theory, ed. S. Seidman (Cambridge University Press, 1994), 89

58. Gilles Deleuze, “Postscript on the Societies of Control,” October 59 (1992): 3–7.

would probably sound rhetorically naive, but he gives a charismatic expression to a grim diagnosis of the emerging digital culture, one that prefigures the fatalism of many contemporary critics of

“techno-optimism.”⁵⁹

With the rise of personal computing and the first cycle of dot com boom and bust, this kind of pessimism has only increased. Daniel Crevier, in his 1993 AI: The Tumultuous History of the Search for Artificial Intelligence, surveys some many of the emerging concerns about the future of artificial intelligence. In a concluding chapter called “The Silicon Challengers in Our Future,” he discusses the grim prospect of a military takeover by the machines, their potential to facilitate mass surveillance, and the unemployment crisis that could follow the advent of artificially intelligent robots.⁶⁰Yet he also leaves room for what he calls “the blissful scenario,”

where “AI could act as an instrument to further democracy,” and “the advent of [the] personal librarian and intelligent database browsing programs [will] help maintain an informed citizenry.”⁶¹ In this striking passage, what Crevier holds out as the best hope for computer technology – that the personal librarian will lead to a better functioning democracy – turns out to be the very thing that is most frequently indicated as technology’s most deleterious effect; many commentators argue that it is precisely this kind of technology – usually termed “personalization” today – that has led to the media “filter bubble”⁶² and the decay of the democracy in the United States.

Although the book predicts, in general, “mostly beneficial effects,” it also sees AI as “immensely threatening,” and even predicts that the “main battles of the 21st century” will be about who – us or the machines – will control the future of the earth.⁶³

In the first two decades of the 21st century, as critical algorithms studies has matured into an interdisciplinary research area, the focus has shifted from the philosophical dimensions of AI to its practical consequences. In the 1970s, 80s and 90s, writing about the intersection of

59. Samuel Alexander, “A critique of techno-optimism: Efficiency without sufficiency is lost,” Melbourne Sustain-able Society Institute, Working Paper (WP)1 (2014): 14.

60. Crevier, AI: The Tumultuous History of the Search for Artificial Intelligence, 312-334.

61. Ibid., 337.

62. Eli Pariser, The Filter Bubble: What the Internet is Hiding from You (The Penguin Press, 2011).

63. Crevier, AI: The Tumultuous History of the Search for Artificial Intelligence, 341.

technology and society had, in one way or another, flowed from the philosophical issues raised by the prospect of artificial intelligence. Even Joseph Weizenbaum’s impassioned admonitions not to let technology supplant human interaction were basically rooted in abstract intellectual commitments about the limitations of computation; normative though they undoubtedly were, they were still essentially philosophical. Critical algorithm studies today, by contrast, is more interested in ethical and social implications per se; the question is no longer what technology is capable of, nor is it the proper definition of “intelligence.” Even less is the discipline interested in what insights AI might yield to us about the actual function of the mind (the essence of the kind of AI that, in different ways, Searle and Schank had actually endorsed). For a diverse range of scholars confronting the lived reality of an algorithmically mediated world, the primary task is simply to understand that world and critique it where it produces unjust outcomes – which it turns out to do frequently.

A number of the major titles in this area have targeted a general audience. Eli Pariser’s widely read The Filter Bubble (2011), for example, highlights the ways in which our present media culture, monopolized by a handful of corporations and optimized for user retention, poses a threat to democracy. In the era of “personalization” (which Pariser dates to 2009), instead of the free flow of opposing ideas envisioned by the Internet’s early adopters, we are instead “more and more enclosed in our own bubbles.”⁶⁴ Pariser argues, moreover, that the same technology that makes the vast reach of the 21st century Internet manageable – namely, personalization – also prevents it from realizing that emancipatory potential; OK Cupid’s recommendation algorithms (or, for that matter, Spotify’s) and Cambridge Analytica are, in a sense, two sides of the same coin. Christopher Steiner’s 2012 Automate This: How Algorithms Took Over Our Markets, Our Jobs, and the World is a catalogue of the various professions threatened by the algorithmic revolution.⁶⁵Jacob Silverman’s 2015 Terms of Service is a consummate tirade against the social

64. Pariser, The Filter Bubble: What the Internet is Hiding from You, 5.

65. C. Steiner, Automate This: How Algorithms Took Over Our Markets, Our Jobs, and the World (Penguin Publishing Group, 2012).

ills of networked society – “I shared therefore I am,” he quips, satirizing the corrosive way our digital personas have supplanted real human interaction.⁶⁶ Frank Pasquale’s 2016 The Black Box Societyshows how financial institutions frequently embed self-serving and reckless behavior in their decision making algorithms, hiding the same old strategies of ruthless capitalism beneath the apparently neutral veneer of algorithmic technology.⁶⁷Many of these same themes are taken up in Cathy O’Neil’s 2019 Weapons of Math Destruction, which focuses on how big data, and the apparently neutral idea of a mathematical model, can be wielded against the consumer in concerning ways.⁶⁸Safiya Umoja Noble’s 2018 Algorithmic Oppression focuses specifically on the negative impact algorithmic mediation can have for racial justice, a development for which Noble coins the widely adopted phrase “technological redlining.”⁶⁹Shoshana Zuboff’s 2019 The Age of Surveillance Capitalismis an extended critique of the covert sale of surveilled user data, a practice that accounts for a large portion of all Internet commerce. The era of “surveillance capitalism,” Zuboff argues, is one in which the surveillance apparatuses of the world’s major corporations already exceed those of the worst totalitarian states in history. They are, moreover, already leveraged to control our offline behaviors, often in ways we do not perceive: “surveillance capitalism preys on dependent populations who are neither its consumers nor its employees and are largely ignorant of its procedures.” Similar themes are sounded in Kate Crawford’s frequently cited article for the Harvard Business Review, “The Hidden Bias in Big Data.”⁷⁰

There are also many critical works for academic and technical audiences. The possibility of machine bias, defined as “unfair discrimination against certain individuals or groups of individuals,” was spotted in 1996, years before it became a popular subject, by Batya Friedman

66. Jacob Silverman, Terms of Service: Social Media and the Price of Constant Connection (Harper Collins, 2015), ix.

67. Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information (Harvard University Press, 2015).

68. Cathy O’Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Crown Publishing Group, 2016).

69. Noble, Algorithms of Oppression: How Search Engines Reinforce Racism.

70. Kate Crawford, “The Hidden Biases in Big Data,” 2013, accessed August 10, 2019.

and Helena Nissenbaum.⁷¹Elizabeth Van Couvering’s 2007 anthropological study of high ranking employees of various search companies reveals that the “schemas” according to which search engines are designed are thoroughly market-oriented and stand distinctly at odds with the values of

“objectivity, fairness, diversity, and representation.”⁷²Kate Crawford highlights the incongruence between the logic of the algorithm, which, she says, will tend to select a clear winner, and the logic of “agonistic pluralism” (a term borrowed from Chantal Mouffe⁷³), which she proposes as a new design ideal for engineers.⁷⁴ Jenna Burrell in 2016 proposed a subtle taxonomy of algorithmic opacity: opacity can refer to corporate secrecy, consumer illiteracy, or, more subtly, as the mismatch between mathematical and human logic.⁷⁵ Burrell also makes the important point that many academic studies of algorithmic culture, even where they ask the right questions, fail to address the actual facts of the systems in question. Nick Seaver (2017) sounds this same theme, while resuming the scuffles from the discipline of math and logic over what should constitute an algorithm; too much debate takes place over algorithms, he claims, without any real consensus about the subject under consideration. Seaver raises this issue in a way that connects it to his training as an anthropologist, drawing on interviews and participant-observer sessions with the actual designers of various recommendation engines. He concludes that more important than the ontological dimension of algorithms is the cultural one; algorithms should be studied not only in culture, but as culture. They should be thought of “as heterogeneous and diffuse sociotechnical systems, with entanglements beyond the boundaries of proprietary software.”⁷⁶

David Berry (2019) takes Seaver’s idea seriously, calling for a general “critical theory

71. Helena Nissenbaum and Batya Friedman, “Bias in Computer Systems,” ACM Transactions on Information Systems14, no. 3 (1996).

72. Elizabeth Van Couvering, “Is Relevance Relevant? Market, Science, and War: Discourses of Search Engine Quality,” Journal of Computer-Mediated Communication 12, no. 3 (April 2007): 866–887.

73. Chantal Mouffe, “Deliberative Democracy or Agonistic Pluralism?,” Social Research 66, no. 3 (1999): 745–

758.

74. Kate Crawford, “Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics,” Science, Technology, and Human Values41, no. 1 (2016).

75. Jenna Burrell, “How the machine ‘thinks’: Understanding opacity in machine learning algorithms,” Big Data and Society3, no. 1 (2016).

76. Seaver, “Algorithms as culture: Some tactics for the ethnography of algorithmic systems,” 10.

of algorithms,” to combat the “cult of data-ism,” which has in certain cases militated against theorizing itself, including the type of theorizing that is the scientific method.⁷⁷ Berry argues that the best way to counter the “data-ism” trend is to look at particular cases where algorithms operate unjustly and examine them in detail; his examples relate primarily to Amazon’s “Mechanical Turk” service, which enables companies to hire very low wage workers to handle menial tasks that are nevertheless still impossible (or not cost-effective) to automate. As Berry notes, Amazon conceives of the Mechanical Turk program as a modular part of a larger computational system.

This renders the human employees as essentially a subset of the computer program, reversing the traditional way in which we conceive of automation in the workforce. Where for most of the history of computation, the problem has been to get computers to behave like people, here the business model is explicitly predicated on getting it the other way around, and paying the humans almost nothing.⁷⁸ Berry sees this as a strategy for keeping the system’s inherent injustices invisible. It is only by exposing this kind of structure, by “contesting the invisibility of algorithmic infrastructure,” that the badly needed critical theory of algorithms is possible.⁷⁹