Open source software development as an innovation system

Modeling open source software development as a complex system exhibiting preferential

“It is fairly clear that one cannot code from the ground up in bazaar style. One can test, debug and improve in bazaar style, but it would be very hard to originate a project in bazaar

mode. ... When you start community-building, what you need to be able to present is a

plausible promise” (p. 37).

Plausibility turns out to have its own research literature. Scacchi (2006) surveyed research

related to innovation which might have relevance to open source. Just as considerable thought

was devoted to the origins and organization of living things before open source software existed,

much thought had also been devoted to the origins and organization of ideas. This body of work

is beyond the scope of this research, but the following section dips into it to extract two useful

concepts: recombinant search (Fleming, 1998) and the conditions for beneficial diversity (Page,

2008). Open source provides a laboratory for the study of the development of ideas, as code

repositories record the history of both successful and unsuccessful projects, capturing details of

the changes made to the code in each project, including who worked on it and when. Some of

this data may be understood in light of research into the process of invention.

2.1.2.1 Recombinant search.

When writing a new open source program, a software developer may start by looking for

work done by others to use or modify, as discussed in (Haefliger, Von Krogh & Spaeth, 2008).

The public exposure of source code for search, study and use by others is considered one of the

benefits of the open source approach. The search and development process naturally involves a

creative synthesis where libraries and bits of code are brought together in a new project. Fleming

(1998) explains how such recombinant search produces new inventions. Combining “local”

of the search space, with higher likelihoods for both failure and breakthrough. Fleming uses the

example of ink jet printing to illustrate a combination of familiar factors; the factors (glass slides,

ink, thin-film resistor material and a power source) had all been used previously (Fleming, 1998,

pp. 56-57). Alternatively, a computer using an optical disk drive rather than a magnetic drive

illustrates a discontinuous combination (Fleming, 1998, p. 59). Figure 29 illustrates how factors

of both types may combine over time. “The nodes are inventions, and arcs indicate that a

succeeding invention has drawn from a preceding invention or from outside the extant

technosphere” (Fleming, 1998, p. 27).

Figure 29: Illustration from (Fleming, 1998, Figure 1, p. 27) showing “The holistic web of technological evolution.”

A new open source project may have a better chance of growing when it builds upon

familiar, frequently used ‘standards.’ On the other hand, a new project is likely to require some

untried techniques to generate a breakthrough. Rosenberg (2008) summarized this neatly in

Rosenberg’s Law:

“[Rosenberg's Law]

Software is easy to make, except when you want it to do something new.

[Corollary]

The number of possible factor combinations grows until the search space becomes so large

that the time required for exploration becomes a constraint. Consequently, Weitzman (1998, p.

357) argued that path dependence becomes greater over time as the number of paths not taken

expands combinatorially. Paths not taken include paths taken only so far and then abandoned;

these appear in open source code repositories as projects with no growth. David and Rullani

(2007) noted the resulting “dissipative” effort in open source software development, stating,

“[The Free/Libre Open Source Software] mode of innovation is able to generate a substantial amount of both exploration and exploitation” (p. 21).

The ability of search engines to automate the scanning and indexing of open source

software enhances the visibility of potential factors for recombination. An open source developer

may often be presented with a choice of ‘hits’ in response to a search query. Crowdsourcing the

curation of search results, or finding answers to technical questions as demonstrated at

StackOverflow (http://stackoverflow.com), potentially accelerates the pace of innovation by

shortening the factor search time, as discussed in (Helbing & Balietti, 2011).

2.1.2.2 Project uniqueness.

Open source software development produces unique software projects. Each one is

different. The structures are different. The development teams are different. Projects use

different languages and different development tools in different combinations. Movies made

from open source code repositories using Gource41, a software version control visualization tool,

show the development of projects over time; these animations highlight significant differences in

morphology. Projects do not look alike or grow in identical ways. In particular, multiple projects

41

do not develop the way a field of corn grows from seed or the way a housing development grows

from a single set of blueprints. Each project has its quirks. Thus, understanding the

characteristics of a population of open source software development projects may require taking

an approach suitable for drawing conclusions for individual projects (1 of ), than (non-existent) average or ‘typical’ projects ( ).

One fact can be inferred from this pervasive uniqueness or ‘newness’: most projects will

not grow into successful collaborations. ‘New’ does not mean necessary, worthwhile, suitable or

better. In fact, the most frequent type of project has one and only one developer—possibly

because it suits the needs of only that one developer. Simply as a function of combinatorial

complexity, where errors of omission and commission can multiply as code is written and

shared, projects are fragile to the vagaries of selection by other developers and execution on their

different systems. When projects fail to match the various criteria developers apply before

downloading code, or when the downloaded code fails to run or function as expected, those

projects are cut off from any process of collaborative improvement. Thus, conditions for

beneficial diversity become significant.

2.1.2.3 Conditions for beneficial diversity.

Page (2008) cited four conditions for beneficial diversity:

1. No genius can produce the best solution alone. 2. Contributors are smart relative to the problem.

3. Incremental improvements can be made to proposed solutions.

4. The contributor group is large enough for genuine diversity. (from pp. 159-162)

Some open source projects manifest all these characteristics, allowing the beneficial

collaboration of diverse contributors. Top open source projects involve more code than any solo

contributors (point 4). Developers express and enhance their programming skills (point 2) by

making incremental improvements to open source projects (point 3). Smaller projects can have

the best solution developed by some genius, but then it is the overall system of competition

between such solutions which defines the ‘best solution’ in the context of a large and diverse set

of proffered solutions. Amongst the large numbers of no-growth projects are many second-best

solutions and third-best solutions and so on, any of which could have been ‘best’ if the diversity

of available solutions were not so great. So point 1 applies again, but at a system level rather than

a project level. This is another example of an antifragile strategy at work in open source;

diversity is good.

These conditions are necessary but not sufficient for an open source project to exhibit

beneficial collaboration. There must also be some compelling and challenging problem which

the collaborators agree upon. Whether “scratching a developer’s personal itch” (Raymond, 1999,

p. 25) grows into a group of developers collaborating on a project thus depends significantly on

the nature of the “itch,” particularly the way in which the project compares with other solutions and the level of difficulty involved in working on the project. Thus, the ‘niche’ may be more

important than the itch.