Breaking Smart
Introduction (by Marc Andreessen)
In 2007, right before the first iPhone launched, I asked Steve Jobs the obvious qu estion: The design of the iPhone was based on discarding every physical interface element except for a touchscreen. Would users be willing to give up the then-domin ant physical keypads for a soft keyboard?
His answer was brusque: “They’ll learn.”
Steve turned out to be right. Today, touchscreens are ubiquitous and seem normal, and other interfaces are emerging. An entire generation is now coming of age with a completely different tactile relationship to information, validating all over ag ain Marshall McLuhan’s observation that “the medium is the message”.
A great deal of product development is based on the assumption that products must adapt to unchanging human needs or risk being rejected. Yet, time and again, peopl e adapt in unpredictable ways to get the most out of new tech. Creative people tin ker to figure out the most interesting applications, others build on those, and en tire industries are reshaped.
People change, then forget that they changed, and act as though they always behave d a certain way and could never change again. Because of this, unexpected changes in human behavior are often dismissed as regressive rather than as potentially int elligent adaptations.
But change happens anyway. “Software is eating the world” is the most recent historic tr ansformation of this sort.
In 2014, a few of us invited Venkatesh Rao to spend the year at Andreessen Horowit z as a consultant to explore the nature of such historic tech transformations. In particular, we set out to answer the question: Between both the breathless and des pairing extremes of viewing the future, could an intellectually rigorous case be m ade for pragmatic optimism?
As this set of essays argues — many of them inspired by a series of intensive conver sations Venkat and I had — there is indeed such a case, and it follows naturally fro
m the basic premise that people can and do change. To “break smart” is to adapt intelligen tly to new technological possibilities.
With his technological background, satirical eye, and gift for deep and different takes (as anyone who follows his Ribbonfarmblog knows!), there is perhaps nobody bet ter suited than Venkat for telling a story of the future as it breaks smart from the past.
Whether you’re a high school kid figuring out a career or a CEO attempting to naviga te the new economy, Breaking Smart should be on your go-to list of resources for thinkin g about the future, even as you are busy trying to shape it.
— Marc Andreessen ---A New Soft Technology
Something momentous happened around the year 2000: a major new soft technology c ame of age. After written language and money, software is only the third major so ft technology to appear in human civilization. Fifteen years into the age of sof tware, we are still struggling to understand exactly what has happened. Marc And
reessen’s now-familiar line, software is eating the world, hints at the significance, but we are only just beginning to figure out how to think about the world in whi
ch we find ourselves.
Only a handful of general-purpose technologies1 – electricity, steam power, precisio n clocks, written language, token currencies, iron metallurgy and agriculture am ong them – have impacted our world in the sort of deeply transformative way that d eserves the description eating. And only two of these, written language and money, w ere soft technologies: seemingly ephemeral, but capable of being embodied in a var iety of specific physical forms. Software has the same relationship to any specif ic sort of computing hardware as money does to coins or credit cards or writing to clay tablets and paper books.
But only since about 2000 has software acquired the sort of unbridled power, indep endent of hardware specifics, that it possesses today. For the first half centur y of modern computing after World War II, hardware was the driving force. The in dustrial world mostly consumed software to meet existing needs, such as tracking
inventory and payroll, rather than being consumed by it. Serious technologists largely focused on solving the clear and present problems of the industrial age rather than exploring the possibilities of computing, proper.
Sometime around the dot com crash of 2000, though, the nature of software, and i ts relationship with hardware, underwent a shift. It was a shift marked by accel erating growth in the software economy and a peaking in the relative prominence of hardware.2 The shift happened within the information technology industry first, and then began to spread across the rest of the economy.
But the economic numbers only hint at3 the profundity of the resulting societal im pact. As a simple example, a 14-year-old teenager today (too young to show up in labor statistics) can learn programming, contribute significantly to open-sourc e projects, and become a talented professional-grade programmer before age 18. T
his is breaking smart: an economic actor using early mastery of emerging technological leverage — in this case a young individual using software leverage — to wield dispr oportionate influence on the emerging future.
Only a tiny fraction of this enormously valuable activity — the cost of a laptop a nd an Internet connection — would show up in standard economic metrics. Based on v isible economic impact alone, the effects of such activity might even show up as a negative, in the form of technology-driven deflation. But the hidden economic significance of such an invisible story is at least comparable to that of an 18 -year-old paying $100,000 over four years to acquire a traditional college degre e. In the most dramatic cases, it can be as high as the value of an entire indus try. The music industry is an example: a product created by a teenager, Shawn Fa nning’s Napster, triggered a cascade of innovation whose primary visible impact ha s been the vertiginous decline of big record labels, but whose hidden impact inc ludes an explosion in independent music production and rapid growth in the live-music sector.4
Software eating the world is a story of the seen and the unseen: small, measurab le effects that seem underwhelming or even negative, and large invisible and pos itive effects that are easy to miss, unless you know where to look.5
Today, the significance of the unseen story is beginning to be widely appreciate d. But as recently as fifteen years ago, when the main act was getting underway, even veteran technologists were being blindsided by the subtlety of the transit ion to software-first computing.
Perhaps the subtlest element had to do with Moore’s Law, the famous 1965 observation by Intel co-founder Gordon Moore that the density with which transistors can be packed into a silicon chip doubles every 18 months. By 2000, even as semiconduc tor manufacturing firms began running into the fundamental limits of Moore’s Law, ch ip designers and device manufacturers began to figure out how to use Moore’s Law t o drive down the cost and power consumption of processors rather than driving up raw performance. The results were dramatic: low-cost, low-power mobile devices, such as smartphones, began to proliferate, vastly expanding the range of what w e think of as computers. Coupled with reliable and cheap cloud computing infrast ructure and mobile broadband, the result was a radical increase in technological potential. Computing could, and did, become vastly more accessible, to many mor e people in every country on the planet, at radically lower cost and expertise l evels.
One result of this increased potential was that technologists began to grope tow ards a collective vision commonly called the Internet of Things. It is a vision based on the prospect of processors becoming so cheap, miniaturized and low-powe red that they can be embedded, along with power sources, sensors and actuators, in just about anything, from cars and light bulbs to clothing and pills. Estimat es of the economic potential of the Internet of Things – of putting a chip and sof tware into every physical item on Earth – vary from $2.7 trillion to over $14 tril lion: comparable to the entire GDP of the United States today.6
By 2010, it had become clear that given connectivity to nearly limitless cloud c omputing power and advances in battery technologies, programming was no longer s omething only a trained engineer could do to a box connected to a screen and a k eyboard. It was something even a teenager could do, to almost anything.
Only a few years ago services like Uber and Lyft seemed like minor enhancements to the process of procuring and paying for cab rides. Slowly, it became obvious that ridesharing was eliminating the role of human dispatchers and lowering the level of expertise required of drivers. As data accumulated through GPS tracking and ratings mechanisms, it further became clear that trust and safety could inc reasingly be underwritten by data instead of brand promises and regulation. This made it possible to dramatically expand driver supply, and lower ride costs by using underutilized vehicles already on the roads.
As the ridesharing sector took root and grew in city after city, second-order ef fects began to kick in. The increased convenience enables many more urban dwelle rs to adopt carless lifestyles. Increasing supply lowers costs, and increases ac cessibility for people previously limited to inconvenient public transportation. And as the idea of the carless lifestyle began to spread, urban planners began to r ealize that century-old trends like suburbanization, driven in part by car owner ship, could no longer be taken for granted.
The ridesharing future we are seeing emerge now is even more dramatic: the highe r utilization of cars leads to lower demand for cars, and frees up resources for other kinds of consumption. Individual lifestyle costs are being lowered and in surance models are being reimagined. The future of road networks must now be rec onsidered in light of greener and more efficient use of both vehicles and roads. Meanwhile, the emerging software infrastructure created by ridesharing is starti ng to have a cascading impact on businesses, such as delivery services, that rel y on urban transportation and logistics systems. And finally, by proving many ke y component technologies, the rideshare industry is paving the way for the next major development: driverless cars.
These developments herald a major change in our relationship to cars.
To traditionalists, particularly in the United States, the car is a motif for an entire way of life, and the smartphone just an accessory. To early adopters who have integrated ridesharing deeply into their lives, the smartphone is the life style motif, and the car is the accessory. To generations of Americans, owning a car represented freedom. To the next generation, not owning a car will represent fr eedom.
And this dramatic reversal in our relationships to two important technologies – ca rs and smartphones – is being catalyzed by what was initially dismissed as “yet anot her trivial app.”
Similar impact patterns are unfolding in sector after sector. Prominent early ex amples include the publishing, education, cable television, aviation, postal mai l and hotel sectors. The impact is more than economic. Every aspect of the global in dustrial social order is being transformed by the impact of software.
This has happened before of course: money and written language both transformed the world in similarly profound ways. Software, however, is more flexible and powe rful than either.
Writing is very flexible: we can write with a finger on sand or with an electron beam on a pinhead. Money is even more flexible: anything from cigarettes in a p rison to pepper and salt in the ancient world to modern fiat currencies can work . But software can increasingly go wherever writing and money can go, and beyond . Software can also eat both, and take them to places they cannot go on their ow n.
Partly as a consequence of how rarely soft, world-eating technologies erupt into human life, we have been systematically underestimating the magnitude of the fo rces being unleashed by software. While it might seem like software is constantl y in the news, what we have already seen is dwarfed by what still remains unseen .
The effects of this widespread underestimation are dramatic. The opportunities p resented by software are expanding, and the risks of being caught on the wrong s ide of the transformation are dramatically increasing. Those who have correctly calibrated the impact of software are winning. Those who have miscalibrated it a re losing.
And the winners are not winning by small margins or temporarily either. Software -fueled victories in the past decade have tended to be overwhelming and irrevers
ible faits accompli. And this appears to be true at all levels from individuals to businesses to nations. Even totalitarian dictatorships seem unable to resist th e transformation indefinitely.
So to understand how software is eating the world, we have to ask why we have be en systematically underestimating its impact, and how we can recalibrate our exp ectations for the future.
[1] Economists use the term general-purpose technologies to talk about those with broa d impact across sectors. There is, however, no clear consensus on which technolo
gies should make the list
[2] By a rough estimate, between 1977 and 2012 the direct contribution of computin g hardware to the United States GDP increased 14% (from 1.4% in 1977 to 1.6% in 2012), while the direct contribution of software increased 150% (from 0.2% in 19 77 to 0.5% in 2012). Computing hardware peaked in 2000 (at 2.2% of GDP) and has steadily declined since (source: a16z research staff).
[3] See for instance Silicon Valley Doesn’t Believe Productivity is Down (Wall Street Jo urnal, July 16, 2015) and GDP: A Brief but Affectionate History, by Diane Coyle, revie wed in Arnold Kling, GDP and Measuring the Intangible, American Enterprise Institu te, February 2014.
[4] Why We Shouldn’t Worry About The (Alleged) Decline Of The Music Industry, Forbes, Ja nuary 2012.
[5] The idea of seen/unseen effects as an overarching distinction in economic evol ution can be traced to an influential 1850 essay by Frederic Bastiat, That Which i s Seen and That Which is Unseen.
[6] Three independent estimates, all for the year 2020, help us calibrate the potent ial. Gartner estimates $1.9 trillion in value-add by 2020. Cisco estimates a value som ewhere between $14 trillion and $19 trillion. IDC estimates a value around $8.9 trillion ( source: a16z research staff).
---Getting Reoriented
There are four major reasons we underestimate the increasing power of software. Three of these reasons drove similar patterns of miscalibration in previous tech nological revolutions, but one is unique to software.
First, as futurist Roy Amara noted, “We tend to overestimate the effect of a techn ology in the short run and underestimate the effect in the long run.” Technologica l change unfolds exponentially, like compound interest, and we humans seem wired to think about exponential phenomena in flawed ways.1 In the case of software, we expected too much too soon from 1995 to 2000, leading to the crash. Now in 2015 , many apparently silly ideas from 2000, such as home-delivery of groceries orde red on the Internet, have become a mundane part of everyday life in many cities. But the element of surprise has dissipated, so we tend to expect too little, to o far out, and are blindsided by revolutionary change in sector after sector. Ch ange that often looks trivial or banal on the surface, but turns out to have bee n profound once the dust settles.
Second, we have shifted gears from what economic historian Carlota Perez calls t he installation phase of the software revolution, focused on basic infrastructure su ch as operating systems and networking protocols, to a deployment phase focused on c onsumer applications such as social networks, ridesharing and ebooks. In her lan dmark study of the history of technology,2 Perez demonstrates that the shift from installation to deployment phase for every major technology is marked by a chaotic t ransitional phase of wars, financial scandals and deep anxieties about civilizat ional collapse. One consequence of the chaos is that attention is absorbed by tr ansient crises in economic, political and military affairs, and the apocalyptic fears and utopian dreams they provoke. As a result, momentous but quiet change p asses unnoticed.
Third, a great deal of the impact of software today appears in a disguised form. The genomics and nanotechnology sectors appear to be rooted in biology and mate rials science. The “maker” movement around 3d printing and drones appears to be abou t manufacturing and hardware. Dig a little deeper though, and you invariably fin d that the action is being driven by possibilities opened up by software more than f undamental new discoveries in those physical fields. The crashing cost of genome
-sequencing is primarily due to computing, with innovations in wet chemistry playi ng a secondary role. Financial innovations leading to cheaper insurance and cred it are software innovations in disguise. The Nest thermostat achieves energy sav ings not by exploiting new discoveries in thermodynamics, but by using machine l earning algorithms in a creative way. The potential of this software-driven mode l is what prompted Google, a software company, to pay $3B to acquire Nest: a com pany that on the surface appeared to have merely invented a slightly better mouset rap.
These three reasons for under-estimating the power of software had counterparts in previous technology revolutions. The railroad revolution of the nineteenth ce ntury also saw a transitional period marked by systematically flawed expectation s, a bloody civil war in the United States, and extensive patterns of disguised change — such as the rise of urban living, grocery store chains, and meat consumpt ion — whose root cause was cheap rail transport.
The fourth reason we underestimate software, however, is a unique one: it is a r evolution that is being led, in large measure, by brash young kids rather than s ober adults.3
This is perhaps the single most important thing to understand about the revoluti on that we have labeled software eating the world: it is being led by young people , and proceeding largely without adult supervision (though with many adults part icipating). This has unexpected consequences.
As in most periods in history, older generations today run or control all key in stitutions worldwide. They are better organized and politically more powerful. I n the United States for example, the AARP is perhaps the single most influential organization in politics. Within the current structure of the global economy, o lder generations can, and do, borrow unconditionally from the future at the expe nse of the young and the yet-to-be-born.
But unlike most periods in history, young people today do not have to either “wait their turn” or directly confront a social order that is systematically stacked ag ainst them. Operating in the margins by a hacker ethos — a problem solving sensibi lity based on rapid trial-and-error and creative improvisation — they are able to u se software leverage and loose digital forms of organization to create new econo mic, social and political wealth. In the process, young people are indirectly di srupting politics and economics and creating a new parallel social order. Instea d of vying for control of venerable institutions that have already weathered sev eral generational wars, young people are creating new institutions based on the new software and new wealth. These improvised but highly effective institutions repe atedly emerge out of nowhere, and begin accumulating political and economic powe r. Most importantly, they are relatively invisible. Compared to the visible powe r of youth counterculture in the 1960s for instance, today’s youth culture, built around messaging apps and photo-sharing, does not seem like a political force to reckon with. This culture also has a decidedly commercial rather than ideologic al character, as a New York Times writer (rather wistfully) noted in a 2011 piece ap propriately titled Generation Sell.4 Yet, today’s youth culture is arguably morepowerful as a result, representing as it does what Jane Jacobs called the “commerce syndrome” of values, rooted in pluralistic economic pragmatism, rather than the opposed “guard
ian syndrome” of values, rooted in exclusionary and authoritarian political ideolo gies.
Chris Dixon captured this guerrilla pattern of the ongoing shift in political po wer with a succinct observation: what the smartest people do on the weekend is what everyone else will do during the week in ten years.
The result is strange: what in past eras would have been a classic situation of generational conflict based on political confrontation, is instead playing out a s an economy-wide technological disruption involving surprisingly little direct political confrontation. Movements such as #Occupy pale in comparison to their 1 960s counterparts, and more importantly, in comparison to contemporary youth-dri ven economic activity.
This does not mean, of course, that there are no political consequences. Softwar e-driven transformations directly disrupt the middle-class life script, upon whi ch the entire industrial social order is based. In its typical aspirational form
, the traditional script is based on 12 years of regimented industrial schooling , an additional 4 years devoted to economic specialization, lifetime employment with predictable seniority-based promotions, and middle-class lifestyles. Though this script began to unravel as early as the 1970s, even for the minority (whit e, male, straight, abled, native-born) who actually enjoyed it, the social order of our world is still based on it. Instead of software, the traditional script runs on what we might call paperware: bureaucratic processes constructed from the older soft technologies of writing and money. Instead of the hacker ethos of fle xible and creative improvisation, it is based on the credentialist ethos of degr ees, certifications, licenses and regulations. Instead of being based on achievi ng financial autonomy early, it is based on taking on significant debt (for coll ege and home ownership) early.
It is important to note though, that this social order based on credentialism an d paperware worked reasonably well for almost a century between approximately 18 70 and 1970, and created a great deal of new wealth and prosperity. Despite its stifling effects on individualism, creativity and risk-taking, it offered its me mbers a broader range of opportunities and more security than the narrow agraria n provincialism it supplanted. For all its shortcomings, lifetime employment in a large corporation like General Motors, with significantly higher standards of living, was a great improvement over pre-industrial rural life.
But by the 1970s, industrialization had succeeded so wildly, it had undermined i ts own fundamental premises of interchangeability in products, parts and humans. As economists Jeffrey Greenwood and Mehmet Yorkuglu5 argue in a provocative paper titled 1974, that year arguably marked the end of the industrial age and the beginn ing of the information age. Computer-aided industrial automation was making ever -greater scale possible at ever-lower costs. At the same time, variety and uniqu eness in products and services were becoming increasingly valuable to consumers in the developed world. Global competition, especially from Japan and Germany, b egan to directly threaten American industrial leadership. This began to drive pr oduct differentiation, a challenge that demanded originality rather than conform ity from workers. Industry structures that had taken shape in the era of mass-pr oduced products, such as Ford’s famous black Model T, were redefined to serve the demand for increasing variety. The result was arguably a peaking in all aspects of the industrial social order based on mass production and interchangeable work ers roughly around 1974, a phenomenon Balaji Srinivasan has dubbed peak centraliza tion.6
One way to understand the shift from credentialist to hacker modes of social org anization, via young people acquiring technological leverage, is through the myt hological tale of Prometheus stealing fire from the heavens for human use.
The legend of Prometheus has been used as a metaphor for technological progress
at least since Mary Shelley’s Frankenstein: A Modern Prometheus. Technologies capable of eating the world typically have a Promethean character: they emerge within a
mature social order (a metaphoric “heaven” that is the preserve of older elites), bu t their true potential is unleashed by an emerging one (a metaphoric “earth” compris ing creative marginal cultures, in particular youth cultures), which gains relat ive power as a result. Software as a Promethean technology emerged in the heart of the industrial social order, at companies such as AT&T, IBM and Xerox, univer sities such as MIT and Stanford, and government agencies such as DARPA and CERN. But its Promethean character was unleashed, starting with the early hacker move ment, on the open Internet and through Silicon-Valley style startups.
As a result of a Promethean technology being unleashed, younger and older face a similar dilemma: should I abandon some of my investments in the industrial social order and join the dynamic new social order, or hold on to the status quo as lo ng as possible?
The decision is obviously easier if you are younger, with much less to lose. But many who are young still choose the apparent safety of the credentialist script s of their parents. These are what David Brooks called Organization Kids (after Will
iam Whyte’s 1956 classic, The Organization Man7): those who bet (or allow their “Tiger” pa rents8 to bet on their behalf) on the industrial social order. If you are an adult
the decision is harder.
Those with a Promethean mindset and an aggressive approach to pursuing a new pat h can break out of the credentialist life script at any age. Those who are unwil ling or unable to do so are holding on to it more tenaciously than ever.
Young or old, those who are unable to adopt the Promethean mindset end up defaul ting to what we call a pastoral mindset: one marked by yearning for lost or unat tained utopias. Today many still yearn for an updated version of romanticized9 195 0s American middle-class life for instance, featuring flying cars and jetpacks. How and why you should choose the Promethean option, despite its disorienting un certainties and challenges, is the overarching theme of Season 1. It is a choice we call breaking smart, and it is available to almost everybody in the developed wo rld, and a rapidly growing number of people in the newly-connected developing wo rld.
These individual choices matter.
As historians such as Daron Acemoglu and James Robinson10 and Joseph Tainter11 have argued, it is the nature of human problem-solving institutions, rather than the nature of the problems themselves, that determines whether societies fail or suc ceed. Breaking smart at the level of individuals is what leads to organizations and nations breaking smart, which in turn leads to societies succeeding or faili ng.
Today, the future depends on increasing numbers of people choosing the Promethea n option. Fortunately, that is precisely what is happening.
[1] See for example, the phenomenon of hyperbolic discounting, one of the major bias es that affect human temporal reasoning.
[2] Carlota Perez, Technological Revolutions and Financial Capital, 2003.
[3] Though the widespread perception that startup founders are relatively young is de batable, it is clear that software allows talented individuals to begin their entrep reneurial journeys much earlier than other technologies in history, simply due t o the availability and accessibility of the technology at low cost. By contrast, major business leaders in the Robber Baron age, such as Cornelius Vanderbilt an d John D. Rockefeller embarked on their empire building journeys in middle age. [4] William Deresiewicz, Generation Sell, New York Times, 2011.
[5] Jeremy Greenwood and Mehmet Yorukoglu, 1974, Carnegie-Rochester Conference Series on Public Policy, 1997.
[6] Personal communication.
[7] See William Whyte, The Organization Man, first published in 1956 and David Brooks, The Organization Kid, The Atlantic Monthly, 2001.
[8] Amy Chua, Battle Hymn of the Tiger Mother, 2011.
[9] Stephanie Coontz, The Way we Never Were: American Families and the Nostalgia Tra p, 1993.
[10] Daron Acemoglu and James Robinson, Why Nations Fail, 2013. [11] Joseph Tainter, The Collapse of Complex Societies, 1990.
---Towards a Mass Flourishing
In this season of Breaking Smart, I will not attempt to predict the what and when of the fut ure. In fact, a core element of the hacker ethos is the belief that being open t
o possibilities and embracing uncertainty is necessary for the actual future to unfold in positive ways. Or as computing pioneer Alan Kay put it, inventing the future is easier than predicting it.
And this is precisely what tens of thousands of small teams — small enough to be fed by no more than two pizzas, by a rule of thumb made famous by Amazon founder Jef
f Bezos — are doing across the world today.
Prediction as a foundation for facing the future involves risks that go beyond s
imply getting it wrong. The bigger risk is getting attached to a particular what and w hen, a specific vision of a paradise to be sought, preserved or reclaimed. This is often a serious philosophical error — to which pastoralist mindsets are particula rly prone — that seeks to limit the future.
But while I will avoid dwelling too much on the what and when, I will unabashedly advocate for a particular answer to how.Thanks to virtuous cycles already gaining in power
ming decades will emerge out of the hacker ethos, despite its apparent periphera l role today. The credentialist ethos of extensive planning and scripting toward s deterministic futures will play a minor supporting role at best. Those who ado pt a Promethean mindset and break smart will play an expanding role in shaping t he future. Those who adopt a pastoral mindset and retreat towards tradition will play a diminishing role, in the shrinking number of economic sectors where cred entialism is still the more appropriate model.
The nature of problem-solving in the hacker mode, based on trial-and-error, iter ative improvement, testing and adaptation (both automated and human-driven) allo ws us to identify four characteristics of how the future will emerge.
First, despite current pessimism about the continued global leadership of the Un ited States, the US remains the single largest culture that embodies the pragmat ic hacker ethos, nowhere more so than in Silicon Valley. The United States in ge neral, and Silicon Valley in particular, will therefore continue to serve as the global exemplar of Promethean technology-driven change. And as virtual collabora tion technologies improve, the Silicon Valley economic culture will increasingly become the global economic culture.
Second, the future will unfold through very small groups having very large impac ts. One piece of wisdom in Silicon Valley today is that the core of the best sof tware is nearly always written by teams of fewer than a dozen people, not by hug e committee-driven development teams. This means increasing well-being for all w ill be achieved through small two-pizza teams beating large ones. Scale will inc reasingly be achieved via loosely governed ecosystems of additional participants creating wealth in ways that are hard to track using traditional economic measu res. Instead of armies of Organization Men and Women employed within large corpo rations, and Organization Kids marching in at one end and retirees marching out at the other, the world of work will be far more diverse.
Third, the future will unfold through a gradual and continuous improvement of well-b eing and quality of life across the world, not through sudden emergence of a uto pian software-enabled world (or sudden collapse into a dystopian world). The pro cess will be one of fits and starts, toys and experiments, bugginess and brokenn ess. But the overall trend will be upwards, towards increasing prosperity for al l.
Fourth, the future will unfold through rapid declines in the costs of solutions to problems, including in heavily regulated sectors historically resistant to cost-s aving innovations, such as healthcare and higher education. In improvements wrough t by software, poor and expensive solutions have generally been replaced by supe rior and cheaper (often free) solutions, and these substitution effects will acc elerate.
Putting these four characteristics together, we get a picture of messy, emergent
progress that economist Bradford Delong calls “slouching towards utopia“: a condition of gradual, increasing quality of life available, at gradually declining cost,
to a gradually expanding portion of the global population.
A big implication is immediately clear: the asymptotic condition represents a cons umer utopia. As consumers, we will enjoy far more for far less. This means that th e biggest unknown today is our future as producers, which brings us to what many vie w as the central question today: the future of work.
The gist of a robust answer, which we will explore in Understanding Elite Discontent , was anticipated by John Maynard Keynes as far back as 1930,1 though he did not l ike the implications: the majority of the population will be engaged in creating a nd satisfying each other’s new needs in ways that even the most prescient of today’s visionaries will fail to anticipate.
While we cannot predict precisely what workers of the future will be doing — what futu re wants and needs workers will be satisfying — we can predict some things about how t hey will be doing it. Work will take on an experimental, trial-and-error charact
er, and will take place in an environment of rich feedback, self-correction, ada ptation, ongoing improvement, and continuous learning. The social order surround ing work will be a much more fluid descendant of today’s secure but stifling paych eck world on the one hand, and liberating but precarious world of free agency an d contingent labor on the other.
In other words, the hacker ethos will go global and the workforce at large will break smart. As the hacker ethos spreads, we will witness what economist Edmund
Phelps calls a mass flourishing2 — a state of the economy where work will be challengi ng and therefore fulfilling. Unchallenging, predictable work will become the preserv e of machines.
Previous historical periods of mass flourishing, such as the early industrial re volution, were short-lived, and gave way, after a few decades, to societies base d on a new middle class majority built around predictable patterns of work and l ife. This time around, the state of mass flourishing will be a sustained one: a slouching towards a consumer and producer utopia.
If this vision seems overly dramatic, consider once again the comparison to othe r soft technologies: software is perhaps the most imagination-expanding technolo gy humans have invented since writing and money, and possibly more powerful than either. To operate on the assumption that it will transform the world at least as d ramatically, far from being wild-eyed optimism, is sober realism.
[1] The classic 1930 article by John Maynard Keynes, Economic possibilities for our grandchildren, remains the dominant framing for understanding technological unemploy ment. Keynes understood that technological unemployment is a temporary phenomenon and that new wants and needs soon appear to create new employment. He viewed this as a spiritual problem of sorts: that of endlessly expanding materialism and spi ritual degeneracy. We will discuss his proposed solution, the concept of a leisu re society, in a later essay.
[2] Edmund Phelps’ Mass Flourishing (2014) is a magisterial survey of the rise of corporat ism and its stifling effects on the economic dynamism that marked the early deca
des of the industrial revolution. By critically examining a wide variety of econo mic indicators (ranging from job satisfaction and values surveys to employment a nd growth data), Phelps constructs a powerful case for abandoning corporatist ec onomic organization models. Compared to the much more heavily publicized economi
c blockbuster of 2014, Thomas Piketty’s Capital in the Twenty-First Century, which foc used much more narrowly on income inequality, Phelps’ work takes a much broader mu lti-model approach. For readers interested in a broad understanding of the econo mic context of software eating the world, Phelps’ book is probably the single best resource.
---Purists versus Pragmatists
At the heart of the historical development of computing is the age-old philosophic al impasse between purist and pragmatistapproaches to technology, which is particularl y pronounced in software due to its seeming near-Platonic ineffability. One way
to understand the distinction is through a dinnerware analogy.
Purist approaches, which rely on alluring visions, are like precious “good” china: m ostly for display, and reserved exclusively for narrow uses like formal dinners. Damage through careless use can drastically lower the value of a piece. Broken or missing pieces must be replaced for the collection to retain its full display value. To purists, mixing and matching, either with humbler everyday tableware, or with different china patterns, is a kind of sacrilege.
The pragmatic approach on the other hand, is like unrestricted and frequent use of hardier everyday dinnerware. Damage through careless play does not affect value as much. Broken pieces may still be useful, and missing pieces need not be repl aced if they are not actually needed. For pragmatists, mixing and matching avail able resources, far from being sacrilege, is something to be encouraged, especia lly for collaborations such as neighborhood potlucks.
In software, the difference between the two approaches is clearly illustrated by t he history of the web browser.
On January 23, 1993, Marc Andreessen sent out a short email, announcing the releas e of Mosaic, the first graphical web browser:
07:21:17-0800 by [email protected]:
By the power vested in me by nobody in particular, alpha/beta version 0.5 of NCS A’s Motif-based networked information systems and World Wide Web browser, X Mosaic , is hereby released:
Along with Eric Bina he had quickly hacked the prototype together after becoming enthralled by his first view of the World Wide Web, which Tim Berners-Lee had un leashed from CERN in Geneva in 1991. Over the next year, several other colleague s joined the project, equally excited by the possibilities of the web. All were eager to share the excitement they had experienced, and to open up the future of t he web to more people, especially non-technologists.
Over the course of the next few years, the graphical browser escaped the confine s of the government-funded lab (the National Center for Supercomputing Applicati ons at the University of Illinois) where it was born. As it matured at Netscape and later at Microsoft, Mozilla and Google, it steered the web in unexpected (an d to some, undesirable) directions. The rapid evolution triggered both the legen dary browser wars and passionate debates about the future of the Internet. Those late-nineties conflicts shaped the Internet of today.
To some visionary pioneers, such as Ted Nelson, who had been developing a purist hypertext paradigm called Xanadu for decades, the browser represented an undesi rably messy direction for the evolution of the Internet. To pragmatists, the bro wser represented important software evolving as it should: in a pluralistic way, embodying many contending ideas, through what the Internet Engineering Task For ce (IETF) calls “rough consensus and running code.”
While every major software project has drawn inspiration from both purists and pra gmatists, the browser, like other pieces of software that became a mission criti cal part of the Internet, was primarily derived from the work and ideas of pragmat ists: pioneers like Jon Postel, David Clark, Bob Kahn and Vint Cerf, who were in strumental in shaping the early development of the Internet through highly inclu sive institutions like the IETF.
Today, the then-minority tradition of pragmatic hacking has matured into agile d evelopment, the dominant modern approach for making software. But the significan ce of this bit of history goes beyond the Internet. Increasingly, the pragmatic, agile approach to building things has spread to other kinds of engineering and beyond, to business and politics.
The nature of software has come to matter far beyond software. Agile philosophie s are eating all kinds of building philosophies. To understand the nature of the world today, whether or not you are a technologist, it is crucial to understand agility and its roots in the conflict between pragmatic and purist approaches t o computing.
The story of the browser was not exceptional. Until the early 1990s, almost all important software began life as purist architectural visions rather than pragma tic hands-on tinkering.
This was because early programming with punch-card mainframes was a highly const rained process. Iterative refinement was slow and expensive. Agility was a dista nt dream: programmers often had to wait weeks between runs. If your program didn’t work the first time, you might not have gotten another chance. Purist architect ures, worked out on paper, helped minimize risk and maximize results under these conditions.
As a result, early programming was led by creative architects (often mathematici ans and, with rare exceptions like Klari Von Neumann and Grace Hopper, usually m ale) who worked out the structure of complex programs upfront, as completely as possible. The actual coding onto punch cards was done by large teams of hands-on programmers (mostly women1) with much lower autonomy, responsible for working o ut implementation details.
In short, purist architecture led the way and pragmatic hands-on hacking was eff ectively impossible. Trial-and-error was simply too risky and slow, which meant significant hands-on creativity had to be given up in favor of productivity. With the development of smaller computers capable of interactive input hands-on hacking became possible. At early hacker hubs, like MIT through the sixties, a h igh-autonomy culture of hands-on programming began to take root. Though the shif t would not be widely recognized until after 2000, the creative part of programm ing was migrating from visioning to hands-on coding. Already by 1970, important and high-quality software, such as the Unix operating system, had emerged from t he hacker culture growing at the minicomputer margins of industrial programming.
Through the seventies, a tenuous balance of power prevailed between purist archi tects and pragmatic hackers. With the introduction of networked personal computi ng in the eighties, however, hands-on hacking became the defining activity in pr ogramming. The culture of early hacker hubs like MIT and Bell Labs began to diff use broadly through the programming world. The archetypal programmer had evolved : from interchangeable member of a large team, to the uniquely creative hacker, tinkering away at a personal computer, interacting with peers over networks. Ins tead of dutifully obeying an architect, the best programmers were devoting incre asing amounts of creative energy to scratching personal itches.
The balance shifted decisively in favor of pragmatists with the founding of the IETF in 1986. In January of that year, a group of 21 researchers met in San Dieg o and planted the seeds of what would become the modern “government” of the Internet .
Despite its deceptively bureaucratic-sounding name, the IETF is like no standard s organization in history, starting with the fact that it has no membership requ irements and is open to all who want to participate. Its core philosophy can be found in an obscure document, The Tao of the IETF, little known outside the world of technology. It is a document that combines the informality and self-awareness of good blogs, the gravitas of a declaration of independence, and the aphoristic wi sdom of Zen koans. This oft-quoted section illustrates its basic spirit:
In many ways, the IETF runs on the beliefs of its members. One of the “founding bel iefs” is embodied in an early quote about the IETF from David Clark: “We reject king s, presidents and voting. We believe in rough consensus and running code”. Another e arly quote that has become a commonly-held belief in the IETF comes from Jon Pos
tel: “Be conservative in what you send and liberal in what you accept”.
Though the IETF began as a gathering of government-funded researchers, it also r epresented a shift in the center of programming gravity from government labs to the commercial and open-source worlds. Over the nearly three decades since, it h as evolved into the primary steward2 of the inclusive, pluralistic and egalitarian spirit of the Internet. In invisible ways, the IETF has shaped the broader econ omic and political dimensions of software eating the world.
The difference between purist and pragmatic approaches becomes clear when we com pare the evolution of programming in the United States and Japan since the early eighties. Around 1982, Japan chose the purist path over the pragmatic path, by embarking on the ambitious “fifth-generation computing” effort. The highly corporati st government-led program, which caused much anxiety in America at the time, pro ved to be largely a dead-end. The American tradition on the other hand, outgrew its government-funded roots and gave rise to the modern Internet. Japan’s contempo rary contributions to software, such as the hugely popular Ruby language designe d by Yukihiro Matsumoto, belong squarely within the pragmatic hacker tradition. I will argue that this pattern of development is not limited to computer science. Ev ery field eaten by software experiences a migration of the creative part from visi oning activities to hands-on activities, disrupting the social structure of all professions. Classical engineering fields like mechanical, civil and electrical engineering had already largely succumbed to hands-on pragmatic hacking by the n ineties. Non-engineering fields like marketing are beginning to convert.
So the significance of pragmatic approaches prevailing over purist ones cannot b e overstated: in the world of technology, it was the equivalent of the fall of t he Berlin Wall.
[1] This distinction between high-autonomy architects and low-level programmers in the early days of computing is often glossed over in feel-good popular accounts of the role of women in early computing. This popular narrative conflates the high-level work of women like Klari von Neumann and Grace Hopper (the latter is in so me ways the zombie Marie Curie of computing, whose totemic prominence somewhat obscu res the contributions of other pioneering women) with the routine work of rank-a nd-file women programmers. By doing so, the popular narrative manages to oversta te the creative contribution of women in the early days, and thereby, rather iro nically, understates the actual misogyny of the 1940s-50s. This leads to a misle ading narrative of decline from an imagined golden age of women in computing. A cl earer indicator of the history of women in programming would be the rate of their
participation in interactive computing, starting with the early 1960s hacker cultu re of the sort that developed at MIT around the earliest interactive minicompute rs such as the PDP-1. Measured against this baseline, I suspect the participatio n of women in creative hands-on programming has been steadily increasing from an ear ly misogynistic low, and is likely significantly better than in other engineerin g fields. I do not, however, have the data to justify this claim.
[2] This rise, in terms of both institutional power and philosophical influence ha s, of course, attracted criticism. The most common criticism is the expected one from purists: that the IETF philosophy encourages incrementalism and short-term thinking. This sort of criticism is briefly addressed in these essays, but repr esents a fundamental philosophical divide comparable to the Left/Right divide in politics, rather than dissent within the pragmatic philosophy. There has also b een actionable criticism within the pragmatic camp. For instance, Postel’s robustnes s principle cited above (“be conservative in what you send and liberal in what you accept”), has been criticized by Joel Spolsky for creating chaos in standards efforts by allowing too much “soft failure.” This particular criticism is being accommodated by the IETF in the form of the alternate “fail hard and fast” design principle.
---Agility and Illegibility
While pragmatic hacking was on the rise, purist approaches entered a period of s low, painful and costly decline. Even as they grew in ambition, software project s based on purist architecture and teams of interchangeable programmers grew inc reasingly unmanageable. They began to exhibit the predictable failure patterns o f industrial age models: massive cost-overruns, extended delays, failed launches , damaging unintended consequences, and broken, unusable systems.
These failure patterns are characteristic of what political scientist James Scot t1 called authoritarian high modernism: a purist architectural aesthetic driven by t he authoritarian priorities. To authoritarian high-modernists, elements of the e nvironment that do not conform to their purist design visions appear “illegible” and anxiety-provoking. As a result, they attempt to make the environment legible by forcibly removing illegible elements. Failures follow because important element s, critical to the functioning of the environment, get removed in the process. Geometrically laid-out suburbs, for example, are legible and conform to platonic architectural visions, even if they are unlivable and economically stagnant. Sl ums on the other hand, appear illegible and are anxiety-provoking to planners, e ven when they are full of thriving economic life. As a result, authoritarian pla nners level slums and relocate residents into low-cost planned housing. In the p rocess they often destroy economic and cultural vitality.
In software, what authoritarian architects find illegible and anxiety-provoking is the messy, unplanned tinkering hackers use to figure out creative solutions. When the pragmatic model first emerged in the sixties, authoritarian architects reacted like urban planners: by attempting to clear “code slums.” These attempts too k the form of increasingly rigid documentation and control processes inherited f rom manufacturing. In the process, they often lost the hacker knowledge keeping the project alive.
In short, authoritarian high modernism is a kind of tunnel vision. Architects ar e prone to it in environments that are richer than one mind can comprehend. The urge to dictate and organize is destructive, because it leads architects to dest roy the apparent chaos that is vital for success.
The flaws of authoritarian high modernism first became problematic in fields lik e forestry, urban planning and civil engineering. Failures of authoritarian deve lopment in these fields resulted in forests ravaged by disease, unlivable “planned” cities, crony capitalism and endemic corruption. By the 1960s, in the West, pionee ring critics of authoritarian models, such as the urbanist Jane Jacobs and the e nvironmentalist Rachel Carson, had begun to correctly diagnose the problem. By the seventies, liberal democracies had begun to adopt the familiar, more demo cratic consultation processes of today. These processes were adopted in computin g as well, just as the early mainframe era was giving way to the minicomputer er a.
was often lowered development speed, increased cost and more invisible corruptio n. New stakeholders brought competing utopian visions and authoritarian tendenci es to the party. The problem now became reconciling conflicting authoritarian vi sions. Worse, any remaining illegible realities, which were anxiety-provoking to a ll stakeholders, were now even more vulnerable to prejudice and elimination. As a result complex technology projects often slowed to a costly, gridlocked crawl. T yranny of the majority — expressed through autocratic representatives of particula r powerful constituencies — drove whatever progress did occur. The biggest casualt y was innovation, which by definition is driven by ideas that are illegible to a
ll but a few: what Peter Thiel calls secrets — things entrepreneurs believe that nobod y else does, which leads them to unpredictable breakthroughs.
The process was most clearly evident in fields like defense. In major liberal de mocracies, different branches of the military competed to influence the design o f new weaponry, and politicians competed to create jobs in their constituencies. As a result, major projects spiraled out of control and failed in predictable w ays: delayed, too expensive and technologically compromised. In the non liberal-democratic world, the consequences were even worse. Authoritarian high modernism continued (and continues today in countries like Russia and North Korea), unche cked, wreaking avoidable havoc.
Software is no exception to this pathology. As high-profile failures like the la
unch of healthcare.gov2 show, “democratic” processes meant to mitigate risks tend to cre ate stalled or gridlocked processes, compounding the problem.
Both in traditional engineering fields and in software, authoritarian high moder nism leads to a Catch-22 situation: you either get a runaway train wreck due to too much unchecked authoritarianism, or a train that barely moves due to a gridl ock of checks and balances.
Fortunately, agile software development manages to combine both decisive authority a nd pluralistic visions, and mitigate risks without slowing things to a crawl. Th e basic principles of agile development, articulated by a group of 17 programmer s in 2001, in a document known as the Agile Manifesto, represented an evolution of t he pragmatic philosophy first explicitly adopted by the IETF.
The cost of this agility is a seemingly anarchic pattern of progress. Agile deve lopment models catalyze illegible, collective patterns of creativity, weaken ill usions of control, and resist being yoked to driving utopian visions. Adopting agi le models leads individuals and organizations to gradually increase their tolera nce for anxiety in the face of apparent chaos. As a result, agile models can get m ore agile over time.
Not only are agile models driving reform in software, they are also spreading to traditional domains where authoritarian high-modernism first emerged. Software is beginning to eat domains like forestry, urban planning and environment protec tion. Open Geographic Information Systems (GIS) in forestry, open data initiative s in urban governance, and monitoring technologies in agriculture, all increase i nformation availability while eliminating cumbersome paperware processes. As we will see in upcoming essays, enhanced information availability and lowered frict ion can make any field hacker-friendly. Once a field becomes hacker-friendly, so ftware begins to eat it. Development gathers momentum: the train can begin movin g faster, without leading to train wrecks, resolving the Catch-22.
Today the shift from purist to pragmatist has progressed far enough that it is a lso reflected at the level of the economics of software development. In past dec ades, economic purists argued variously for idealized open-source, market-driven or government-led development of important projects. But all found themselves f aced with an emerging reality that was too complex for any one economic ideology to govern. As a result, rough consensus and running economic mechanisms have pr evailed over specific economic ideologies and gridlocked debates. Today, every a vailable economic mechanism — market-based, governmental, nonprofit and even crimi nal — has been deployed at the software frontier. And the same economic pragmatism is spreading to software-eaten fields.
This is a natural consequence of the dramatic increase in both participation lev els and ambitions in the software world. In 1943, only a small handful of people working on classified military projects had access to the earliest computers. E
ven in 1974, the year of Peak Centralization, only a small and privileged group had access to the early hacker-friendly minicomputers like the DEC PDP series. B ut by 1993, the PC revolution had nearly delivered on Bill Gates’ vision of a comp uter at every desk, at least in the developed world. And by 2000, laptops and Bl ackberries were already foreshadowing the world of today, with near-universal ac cess to smartphones, and an exploding number of computers per person.
The IETF slogan of rough consensus and running code (RCRC) has emerged as the only w orkable doctrine for both technological development and associated economic mode ls under these conditions.
As a result of pragmatism prevailing, a nearly ungovernable Promethean fire has been unleashed. Hundreds of thousands of software entrepreneurs are unleashing i nnovations on an unsuspecting world by the power vested in them by “nobody in part icular,” and by any economic means necessary.
It is in the context of the anxiety-inducing chaos and complexity of a mass flou rishing that we then ask: what exactly is software?
---Rough Consensus and Maximal Interestingness
Software possesses an extremely strange property: it is possible to create high-value software products with effectively zero capital outlay. As Mozilla enginee r Sam Penrose put it, software programming is labor that creates capital.
This characteristic make software radically different from engineering materials like steel, and much closer to artistic media such as paint.1 As a consequence, eng ineer and engineering are somewhat inappropriate terms. It is something of a stretch t o even think of software as a kind of engineering “material.” Though all computing r equires a physical substrate, the trillions of tiny electrical charges within co mputer circuits, the physical embodiment of a running program, barely seem like matter.
The closest relative to this strange new medium is paper. But even paper is not as cheap or evanescent. Though we can appreciate the spirit of creative abundanc e with which industrial age poets tossed crumpled-up false starts into trash can s, a part of us registers the wastefulness. Paper almost qualifies as a medium for t rue creative abundance, but falls just short.
Software though, is a medium that not only can, but must be approached with an abund ance mindset. Without a level of extensive trial-and-error and apparent waste th at would bankrupt both traditional engineering and art, good software does not t ake shape. From the earliest days of interactive computing, when programmers cho se to build games while more “serious” problems waited for computer time, to modern complaints about “trivial” apps (which often turn out to be revolutionary), scarcity -oriented thinkers have remained unable to grasp the essential nature of softwar e for fifty years.
The difference has a simple cause: unsullied purist visions have value beyond an xiety-alleviation and planning. They are also a critical authoritarian marketing and signaling tool — like formal dinners featuring expensive china — for attracting and concentrating scarce resources in fields such as architecture. In an enviro nment of abundance, there is much less need for visions to serve such a marketin g purpose. They only need to provide a roughly correct sense of direction to tho se laboring at software development to create capital using whatever tools and i deas they bring to the party — like potluck participants improvising whatever reso urces are necessary to make dinner happen.
Translated to technical terms, the dinnerware analogy is at the heart of softwar e engineering. Purist visions tend to arise when authoritarian architects attemp t to concentrate and use scarce resources optimally, in ways they often sincerel y believe is best for all. By contrast, tinkering is focused on steady progress rather than optimal end-states that realize a totalizing vision. It is usually d riven by individual interests and not obsessively concerned with grand and pater nalistic “best for all” objectives. The result is that purist visions seem more comf orting and aesthetically appealing on the outside, while pragmatic hacking looks messy and unfocused. At the same time purist visions are much less open to new possibilities and bricolage, while pragmatic hacking is highly open to both. Within the world of computing, the importance of abundance-oriented approaches w
as already recognized by the 1960s. With Moore’s Law kicking in, pioneering comput er scientist Alan Kay codified the idea of abundance orientation with the observ ation that programmers ought to “waste transistors” in order to truly unleash the po wer of computing.
But even for young engineers starting out today, used to routinely renting cloud y container-loads2 of computers by the minute, the principle remains difficult to follow. Devoting skills and resources to playful tinkering still seems “wrong,” when there are obvious and serious problems desperately waiting for skilled attentio
n. Like the protagonist in the movie Brewster’s Millions, who struggles to spend $30 m illion within thirty days in order to inherit $300 million, software engineers m
ust unlearn habits born of scarcity before they can be productive in their mediu m.
The principle of rough consensus and running code is perhaps the essence of the abun dance mindset in software.
If you are used to the collaboration processes of authoritarian organizations, t he idea of rough consensus might conjure up the image of a somewhat informal committ ee meeting, but the similarity is superficial. Consensus in traditional organiza tions tends to be brokered by harmony-seeking individuals attuned to the needs o f others, sensitive to constraints, and good at creating “alignment” among competing autocrats. This is a natural mode of operation when consensus is sought in orde r to deal with scarcity. Allocating limited resources is the typical purpose of such industrial-age consensus seeking. Under such conditions, compromise represe nts a spirit of sharing and civility. Unfortunately, it is also a recipe for gri dlock when compromise is hard and creative breakthroughs become necessary. By contrast, software development favors individuals with an autocratic streak, driven by an uncompromising sense of how things ought to be designed and built, which at first blush appears to contradict the idea of consensus.
Paradoxically, the IETF philosophy of eschewing “kings, presidents and voting” means that rough consensus evolves through strong-minded individuals either truly com ing to an agreement, or splitting off to pursue their own dissenting ideas. Conf licts are not sorted out through compromises that leave everybody unhappy. Inste ad they are sorted out through the principle futurist Bob Sutton identified as c ritical for navigating uncertainty: strong views, weakly held.
Pragmatists, unlike the authoritarian high-modernist architects studied by James Scott, hold strong views on the basis of having contributed running code rather than abstract visions. But they also recognize others as autonomous peers, rath er than as unquestioning subordinates or rivals. Faced with conflict, they are w illing to work hard to persuade others, be persuaded themselves, or leave.
Rough consensus favors people who, in traditional organizations, would be consid ered disruptive and stubborn: these are exactly the people prone to “breaking smar t.” In its most powerful form, rough consensus is about finding the most fertile d irections in which to proceed rather than uncovering constraints. Constraints in software tend to be relatively few and obvious. Possibilities, however, tend to be intimidatingly vast. Resisting limiting visions, finding the most fertile di rection, and allying with the right people become the primary challenges.
In a process reminiscent of the “rule of agreement” in improv theater, ideas that un leash the strongest flood of follow-on builds tend to be recognized as the most fertile and adopted as the consensus. Collaborators who spark the most intense c reative chemistry tend to be recognized as the right ones. The consensus is rough be cause it is left as a sense of direction, instead of being worked out into a det ailed, purist vision.
This general principle of fertility-seeking has been repeatedly rediscovered and articulated in a bewildering variety of specific forms. The statements have nam
es such as the principle of least commitment (planning software), the end-to-end princ iple (network design), the procrastination principle (architecture), optionality (investin g), paving the cowpaths (interface design), lazy evaluation(language design) and late bi nding (code execution). While the details, assumptions and scope of applicability
of these different statements vary, they all amount to leaving the future as fre e and unconstrained as possible, by making as few commitments as possible in any given local context.
The principle is in fact an expression of laissez-faire engineering ethics. Donald K nuth, another software pioneer, captured the ethical dimension with his version: p remature optimization is the root of all evil. The principle is the deeper reason autonomy and creativity can migrate downstream to hands-on decision-making. Leav ing more decisions for the future also leads to devolving authority to those who come later.
Such principles might seem dangerously playful and short-sighted, but under cond itions of increasing abundance, with falling costs of failure, they turn out to be wise. It is generally smarter to assume that problems that seem difficult and important today might become trivial or be rendered moot in the future. Behavio rs that would be short-sighted in the context of scarcity become far-sighted in the context of abundance.
The original design of the Mosaic browser, for instance, reflected the optimisti c assumption that everybody would have high-bandwidth access to the Internet in th e future, a statement that was not true at the time, but is now largely true in the developed world. Today, many financial technology entrepreneurs are building products based on the assumption that cryptocurrencies will be widely adopted a nd accepted. Underlying all such optimism about technology is an optimism about humans: a belief that those who come after us will be better informed and have m ore capabilities, and therefore able to make more creative decisions.
The consequences of this optimistic approach are radical. Traditional processes of consensus-seeking drive towards clarity in long-term visions but are usually fuzzy on immediate next steps. By contrast, rough consensus in software delibera tely seeksambiguity in long-term outcomes and extreme clarity in immediate next st eps. It is a heuristic that helps correct the cognitive bias behind Amara’s Law. C larity in next steps counteracts the tendency to overestimate what is possible i n the short term, while comfort with ambiguity in visions counteracts the tenden cy to underestimate what is possible in the long term. At an ethical level, roug h consensus is deeply anti-authoritarian, since it avoids constraining the freedom s of future stakeholders simply to allay present anxieties. The rejection of “voti ng” in the IETF model is a rejection of a false sense of egalitarianism, rather th an a rejection of democratic principles.
In other words, true north in software is often the direction that combines ambig uity and evidence of fertility in the most alluring way: the direction of maximal interestingness.3
The decade after the dot com crash of 2000 demonstrated the value of this princi
ple clearly. Startups derided for prioritizing “growth in eyeballs” (an “interestingne ss” direction) rather than clear models of steady-state profitability (a self-limi ting purist vision of an idealized business) were eventually proven right. Iconi
c “eyeball” based businesses, such as Google and Facebook, turned out to be highly pro fitable. Businesses which prematurely optimized their business model in response
to revenue anxieties limited their own potential and choked off their own growt h.
The great practical advantage of this heuristic is that the direction of maximal interestingness can be very rapidly updated to reflect new information, by evol ving the rough consensus. The term pivot, introduced by Eric Ries as part of the Lea n Startup framework, has recently gained popularity for such reorientation. A pi vot allows the direction of development to change rapidly, without a detailed lo ng-term plan. It is enough to figure out experimental next steps. This ability t o reorient and adopt new mental models quickly (what military strategists call a f ast transient4) is at the heart of agility.
The response to new information is exactly the reverse in authoritarian developm ent models. Because such models are based on detailed purist visions that grow m ore complex over time, it becomes increasingly harder to incorporate new data. A s a result, the typical response to new information is to label it as an irrelev ant distraction, reaffirm commitment to the original vision, and keep going. Thi s is the runaway-train-wreck scenario. On the other hand, if the new information helps ideological opposition cohere within a democratic process, a competing pu rist vision can emerge. This leads to the stalled-train scenario.
gree roughly on the most interesting direction than to either update a complex, detailed vision or bring two or more conflicting complex visions into harmony. For this to work, an equally pragmatic implementation philosophy is necessary. O ne that is very different from the authoritarian high-modernist way, or as it is known in software engineering, the waterfall model (named for the way high-level pu rist plans flow unidirectionally towards low-level implementation work).
Not only does such a pragmatic implementation philosophy exist, it works so well that running code actually tends to outrun even the most uninhibited rough consensu s process without turning into a train wreck. One illustration of this dynamic i s that successful software tends to get both used and extended in ways that the original creators never anticipated – and are often pleasantly surprised by, and s ometimes alarmed by. This is of course the well-known agile model. We will not g et into the engineering details,5 but what matters are the consequences of using i t.
The biggest consequence is this: in the waterfall model, execution usually lags vision, leading to a deficit-driven process. By contrast, in working agile proce sses, running code races ahead, leaving vision to catch up, creating a surplus-d riven process.
Both kinds of gaps contain lurking unknowns, but of very different sorts. The su rplus in the case of working agile processes is the source of many pleasant surp rises: serendipity. The deficit in the case of waterfall models is the source of what William Boyd called zemblanity: “unpleasant unsurprises.”
In software, waterfall processes fail in predictable ways, like classic Greek tr agedies. Agile processes on the other hand, can lead to snowballing serendipity, getting luckier and luckier, and succeeding in unexpected ways. The reason is s imple: waterfall plans constrain the freedom of future participants, leading the m to resent and rebel against the grand plan in predictable ways. By contrast, a gile models empower future participants in a project, catalyzing creativity and unpredictable new value.
The engineering term for the serendipitous, empowering gap between running code and governing vision has now made it into popular culture in the form of a much-misunderstood idea: perpetual beta.
[1] See the essay by Paul Graham, Hackers and Painters.
[2] Modern cloud-computing datacenters often use modular architectures with racks of servers mounted within shipping containers. This allows them to be easily mov ed, swapped out or added.
[3] The importance of the “interestingness” of work extends far beyond software proces ses. As Edmund Phelps (see footnote 2of Towards a Mass Flourishing) notes, based on da ta from the World Values Survey, that “How the survey respondents in a country value d the ‘interestingness of a job’ (c020 in the WVS classification) was significantly related to how well the country scored in several dimensions economic performanc e.”
[4] Fast transient is a term of art in a military doctrine known as maneuver warfare. Maneuver warfare is descended from a long tradition dating back to Sun Tzu’s Art of War and the German Blitzkrieg model in World War II. In its contemporary form, it was developed by Col. John Boyd of the US Air Force. The Lean Startup movement i s in many ways a simplified version of a core concept in Boydian maneuver warfar
e: the OODA loop (Observe, Orient, Decide and Act). The Lean Startup notion ofpivot corres ponds roughly to the idea of a rapid reorientation via a fast transient. A good
discussion of the application of maneuver warfare concepts for business environm ents can be found in Chet Richards’ excellent book, Certain to Win.
[5] For technologists interested in learning agile methodologies, there are many e xcellent books, such as The Principles of Product Development Flow: Second Generat ion Lean Product Development by Donald G. Reinertsen and active communities of pra ctitioners constantly evolving the state of the art.
---Running Code and Perpetual Beta
When Google’s Gmail service finally exited beta status in July 2009, five years aft er it was launched, it already had over 30 million users. By then, it was the th ird largest free email provider after Yahoo and Hotmail, and was growing much fa
