Team of Rivals: Aligning
Technology & Market Drivers in an
Open-Source Standards Testing
Program
Rick Bauer, Open Networking Foundation (ONF)
Ron Milford, Indiana Center for Network Translational Research and Education (InCNTRE), Indiana University, US
Li Zhen, Beijing Internet Institute (BII)
ABSTRACT
An effective testing program (conformance, benchmarking, and interoperability) that supports a rapidly-‐ evolving technology specification is as much art as it is science. Economic, technological, and market drivers must be harmonized, allowing for the simultaneous development of consumer confidence, industry competition, and trustworthy product validation. When the technology specification is globally available in a zero-‐cost license, and the supporting software is in open-‐source format, as in the Open Networking Foundation’s OpenFlow™ networking specification, there are additional variables to consider. This paper examines the manner in which this program was designed and built over the past three years. In a world that increasingly features both open source and proprietary economic models, this testing program also proposes to leverage both collaboration and competition among participants. ONF proposes to create, as it were, of a “team of rivals.”
Keywords/Index Terms
Conformance, testing, technology testing, open source testing, performance benchmarking, interoperability, management of testing labs, OpenFlow
INTRODUCTION
Nascent technologies require customer trust in order to thrive. End-‐user customer acceptance and deployment of new standards can be greatly assisted by the creation of independent, 3rd-‐party
supplies verification of conformance to a specification, benchmarking of performance characteristics, and entity interoperability can provide great benefits to customers. For clarification of these three testing services, see Figure 1.
Figure 1: The Aspects of an Effective Technology Testing & Validation Program
For the greater market acceptance of software defined networking (SDN) and its major networking protocol, OpenFlow, the decision was made to create a testing program. The first phase of the ONF OpenFlow Conformance Testing Program resulted in a confederation of independent testing labs supplying OpenFlow conformance certifications. Over the next two years, they will also produce both performance benchmarking and interoperability validation.
About the Open Networking Foundation
The Open Networking Foundation, (www.opennetworking.org), formed in 2011, included 23 founding member companies. ONF was chartered as “a user-‐driven organization dedicated to the promotion and adoption of Software-‐Defined Networking (SDN) through open standards development.”2 By mid-‐2014,
ONF had over 150 member companies. One of the key technical contributions was the release of the OpenFlow protocol specification from earlier versions at Stanford University into a commercially-‐ adaptable specification managed by ONF. A decision was made at the outset that OpenFlow should remain an open, freely-‐available standard that could be consumed royalty-‐free.3
In 2011, the ONF Testing & Interoperability Working Group (TIWG) was created, with a charter to develop an equally open and comprehensive testing and validation program. A small number of
companies and a few volunteer members designed a process whereby OpenFlow conformance could be independently validated, where benchmarks for system and component performance could be
formulated, and where resources to create an effective interoperability matrix could be created. ONF began sponsored plugfests and other technical gatherings so that companies could discover how their OpenFlow devices (both hardware and software) behaved in more complex environments—6 such week-‐long events have already taken place. The TIWG created the following roadmap for its first 5 years (see Figure 2).
ONF wanted a program that was open to all qualified labs, not a single-‐source, standards development organization (SDO) controlled entity that left manufacturers little bargaining power. There was to be zero financial interest on the part of ONF; several in the leadership cadre had experienced the unhealthy ways that a single testing provider working for the revenue of the SDO could create a monopolistic
entity, and wanted to assiduously guard ONF’s antitrust status. Having evaluated a variety of models, ONF identified possible participants for its testing program. It was not always an easy process (the transitions from one incumbent test lab to multiple labs, and then from one lab per continent to multiple labs per continent, were particularly challenging)4, but at the heart of the ONF ethos was a
desire to include all stakeholders’ interests in an open and transparently competitive testing ecosystem.
I. The Stakeholders of a Testing Ecosystem
The stakeholders in this technology testing ecosystem are illustrated in Figure 3. While much of the nature of the relationships between stakeholders is self-‐evident, striking a balance between competition and collaboration among these entities is vital. A deeper examination is in order at this juncture.
A. Customers/SDN Adopters
In addition to the influence of market forces amid the competition between SDN vendors, end users will make purchasing decisions for SDN products by securing the maximum amount of reliable information prior to purchase. ONF’s Market Education Committee was assigned the dual tasks of creating both the “pull” of informed customers wanting product certifications, as well as the “push” of information to SDN vendors that customers will be increasingly “asking for the OpenFlow label.” Additionally, ONF has the task, through its liaison relationships with other SDOs and government technology agencies throughout the world, to highlight the additional purchasing confidence secured the certified OpenFlow products. If successful, this influence can result in governments standardizing their SDN purchases to include only those products that have independent 3rd party conformance validations.
B. SDN Manufacturers
The complete analysis of the market forces that SDN manufacturers face is far beyond the scope of this endeavor5, but it is important when creating a testing program to understand them clearly. In a
networking equipment market that featured a few hardware incumbents enjoying great market share, SDN technologies have disrupted markets, entrenched suppliers with technology lock-‐in, and accepted ways of deploying and managing networks. With networking product differentiation achieved in a variety of ways (including perceived or actual quality, performance, cost, etc.), conformance to the
OpenFlow standard should become another sought-‐after attribute, worth any additional cost in the development and marketing of new SDN products.
One early difficulty for SDN equipment manufacturers was the release scheduling of the OpenFlow conformance test. With the time needed to develop testing protocols, results handling, agreement among rivals for results handling, and the hundreds of use cases and test code for OpenFlow 1.0, the appearance of its conformance test lagged almost a year after the introduction of the protocol. Many vendors decided to wait for the OpenFlow 1.3 conformance test rather than test to what was perceived to be an “older” version of OpenFlow. When a newer version of the OpenFlow specification is released, the conformance test for that version needed to be much more closely aligned. A more coherent roadmap had to be communicated to SDN product manufacturers.
C. ONF-Approved SDN Product Testing Labs
ONF opted for a testing program very much like its specification development. The lab program must be open to any qualified applicant, with multiple labs creating competition for testing services. While ONF-‐ approved testing labs would collaborate to assure a consistent set of testing experiences and results reporting throughout the world, they would neither collude on price nor attempt to serve only one geographic area of the global market. Finally, while ONF would spend member resources to help develop the OpenFlow specification and such test-‐related software as OpenFlow-‐Test, and also support activities such as plugfests around the world, it would not seek a revenue stream or levy a surcharge on any testing lab or company seeking 3rd party testing from an ONF-‐approved lab. After three years of
development and recruitment, ONF has six such approved SDN testing labs:
• Indiana Center for Network Translational Research and Education (InCNTRE), Indiana University, US (Approved 2012)
• Beijing Internet Institute (BII), Beijing, China (approved 2013)
• University of New Hampshire Interoperability Lab (UNH-‐IOL), US (approved 2013) • Criterion Network Labs (CNLabs), Bangalore, India (approved 2014)
• Network Benchmarking Lab (NBL), co-‐hosted by the Industrial Technology Research Institute (ITRI) and National Chiao Tung University (NCTU), Taiwan (approved 2014)
• China Academy of Telecommunication Research of MIIT (CATR), Beijing, China
These companies have an interest in providing testing services to manufacturers, using ONF’s freely-‐ available specifications for conformance testing (published in 2013), and in the future, both
performance benchmarking and interoperability assessment. The criteria by which a lab is approved by ONF to be part of its approved testing process will be detailed in Part II.
D. SDN Testing and Tools Manufacturing Companies
The OpenFlow protocol details a wide variety of networking behaviors that can manage networks. As such, there was an early effort in the development in the specification (beginning with OpenFlow 1.3) to identify what parts would be considered mandatory (known as “Core OpenFlow”) and what parts were optional and not required in a conformance assessment. In this manner, the specification could grow in its core functionality while still allowing for innovation on the “edges” of the specification. Commercial testing companies provide testing products and services that offer insights into the behaviors of SDN equipment, often in the optional requirements of the OpenFlow specifications. The contribution of these companies, working in harmony with ONF-‐approved testing labs, was vital in creating an ecosystem for testing software that would have ONF managing the OpenFlow core specification conformance text (published as a free and open specification available to the world), but also have testing companies be able to innovate on the more optional areas of the specification. In such an arrangement, all parties benefit from a robust, open-‐source community contributing core test cases, test code, and a shared testing framework that can validate core OpenFlow functionality. Testing labs are able to share efforts in this model, both collaborating and competing with other testing labs; commercial testing companies can leverage a shared core created by the community, and thus are provided with incentives to innovate, and incentives to donate older test code and test cases for optional features in the specification as they become normative.
In the early stages of the development of the OpenFlow conformance testing program, there was little activity on the part of commercial software tool manufacturers, due to market uncertainty about OpenFlow and SDN. Why make software tools for a protocol that may not survive? The business reasons for waiting to engage were many, but several forward-‐looking organizations contributed anyway, and a richer environment of both commercial tool testing frameworks and OF-‐Test (open source) emerged. ONF had to create a process to accurately validate these commercial tools, which became part of its services beginning in 2014.
E. OpenFlow Specification Development & Conformance Testing Program
Development
The OpenFlow protocol evolved quickly. Development from versions 1.0 through 1.3, and from 1.4 and beyond (current discussions are laying out the foundations for the next generation of OpenFlow, to be
detailed more specifically in 2015)6 are running at a brisk pace. What is critical in the specification
development process is a shared understanding that every feature and functionality of the core protocol should have not only the behavior adequately described, but a use case, a test case, and (if at all
possible), test code as a part of any new feature submission. Early in the development of the OpenFlow specification this ecosystem was not properly aligned, and features/functionalities in OpenFlow
multiplied (as one would expect for a growing protocol) without concurrent growth in test cases or test code. Other teams were struggling to keep up with a conformance test that was an accurate reflection of the current version of the specification. To make matters worse, there was no early consensus definition of which new features in the specification were considered “Core OpenFlow” and which features were was optional. Additionally, there was uncertainty on the part of manufacturers about which particular version of OpenFlow to target for validating conformance (most organizations do not produce a conformance test for every iteration of the specification, only its major ones). The
specification and its conformance needed better alignment and timing. These ambiguities resulted in an even greater “whipsaw” effect for the development of a conformance test. Stakeholders, activities, manufacturers, testing labs and companies—all were looking for ONF to provide guidance, a clear roadmap, and a light touch that would allow market forces to drive activities in ONF, and not the opposite.
F. Open Networking Foundation, Managing Sponsor
ONF needs to fairly represent its members’ diverse interests, and to drive all member activity to the mission of the organization. ONF must balance the interests of product manufacturers, conformance testing labs, proprietary testing software developers, specification and code development, all with the priority of commercial adoption in mind. It must do so comprehensively, fairly, legally, and effectively. Since ONF is led by board members that are end user/operators of SDN and not vendors or product manufacturers, there is an overarching goal to make certain that its program exists for the benefit of those individuals and companies that are implementing SDN solutions around the world. ONF not only helps coordinate specification and associated open-‐source software development that includes a test framework, test code, and software tools, but it also supports a community that encourages software contributions for the common good. The testing program is thus both competitive and collaborative. One of the early descriptions used to describe this design was borrowed from U.S. history.
In 2005, historian Doris Kearns Goodwin's book on the Lincoln presidency, Team of Rivals: The Political
Abraham Lincoln showed exceptional political aplomb through the selection and management of the leading vote-‐getters in the 1860 presidential elections, creating, as it were, a “team of rivals.” The author summarizes, “Lincoln basically pulled in all the people who had been running against him into his cabinet,” in a manner that attracted the attention of current U.S. President Barack Obama, who
summarized Goodwin's thesis, adding, “Whatever personal feelings there were, the issue was ‘How can we get this country through this time of crisis?’” While the issues confronting ONF pale in comparison to either present or past aspects of American history, in ONF there was a sense that the only way that the thorny issues of access, freedom of markets, economic and programmatic sustainability, and
fundamental openness of a program could be solved would be to have all voices represented in an open and fair model. The resultant mission statement of the ONF testing programs was reflected in the ONF Conformance Testing Program preamble statement, reproduced as follows:
1. Continue to support the functionality and adoption of OpenFlow through an aggressive and effective specification development program;
2. Create a conformance testing program that is supported with test-‐related software
contributions (test cases, test code, other validation tools) from a wide variety of ecosystem participants, available from an ONF-‐supported online environment, as free and open source software;
3. Support a rigorous testing and interoperability activity in ONF Plugfests, supported by ONF member companies with the highest participation possible;
4. Create and support an outward-‐facing series of activities that convincingly demonstrate the efficacy of OpenFlow-‐supported solutions (technology demonstrations, hackfests,
collaborations at events, etc.);
5. Allow the creation of independent testing labs throughout the world that are accredited by ONF, informed and supported by open communication with ONF and with each other, which will offer testing services that validate conformance with the ONF OpenFlow specification.8
II. Creating a Team of Rivals: Competition and Collaboration
In seeking to create such a program, ONF had a head start. Many of its members had participated in other industry testing labs and programs, and provided guidance in creating and supporting an open environment. Vendors weighed in about wanting freedom to choose from a plurality of qualified labs, not being locked into a single provider whose schedule, offerings, or pricing structure might not be attractive or competitive. The program needed to move from a single initial lab to multiple labs, from one testing tool software provider to multiple competitive companies, all guided by policies, procedures, and clear rules of engagement. It needed a core of software that was available and open source, and a
community that would have incentives to encourage contributors. Since there were few market drivers for creating commercial testing tools, there were even fewer tools available in the open source
community. Troubleshooting tools such as an OpenFlow dissector/plugin for Wireshark9 or other
popular tools to stay current with OpenFlow was a challenge. The initial open source test platform for OpenFlow, named OF-‐Test, did not completely align with the OpenFlow test specification. There were great challenges in creating and using test code, and the problems didn’t stop there. The criteria by which an ONF-‐approved lab could be certified as approved had to be clearer, universally respected, and beyond the control of ONF’s lab members, so it would never be a case of incumbents carving out territories and erecting artificial barriers of entry for new labs. Work soon began on these issues.
One of the first tasks undertaken by the TIWG was to identify the key components of a testing program. They began by examining other testing programs such as the Metro Ethernet Forum, IPv6 Ready™, and the Wi-‐Fi Alliance to identify key components of each program. The selected components included the following:
• Advisory or administrative group to set policies, oversee labs, provide a forum to discuss issues and resolve disputes;
• Conformance test specification outlining required tests, optional tests, profiles, reporting and update policies;
• Reference test code to provide early tool for validation by developers and test labs;
• Troubleshooting tools for developers and testers;
• Pilot testing program to validate test tools, test spec and early devices; • Certified commercial testing tools;
• Certified test labs;
• Logo/trademark Program.
As there were independent specifications to validate the quality and integrity of testing labs, ONF decided that the ISO 17025 specification “General requirements for the competence of testing and calibration laboratories”10, was to serve as a reliable guide. This ISO specification laid out an inclusive set
of management deliverables (in many ways, the ISO 17025 specification is much like the ISO 9001 management specification for general business activities, but set up for lab/testing environments). The specification outlined the responsibilities for relationships that labs were to have with vendors seeking testing services, procedures for handling testing results and the regular auditing of testing labs, as well as general policies and procedures for accountability and transparency of testing processes. Armed with this specification, ONF began to discuss broadening the number of labs for those facilities that were ISO
17025 conformant. It did not take long to find reputable labs that were willing to agree to ONF policies and procedures (antitrust obligations, logo usage and program requirements, etc.) in addition to ISO 17025 conformance. In a few months, the Beijing Internet Institute (BII) and the University of New Hampshire Interoperability Lab (UNH-‐IOL) were added as ONF-‐approved labs, opening the ecosystem from one to three labs. A few months later, a new lab in Bangalore, India, approached ONF with interest; Criterion Network Labs (CNLabs) was then added in early 2014. ONF was soon approached by two additional labs in the late spring of 2014, the Network Benchmarking Lab (NBL), co-‐hosted by the Industrial Technology Research Institute (ITRI) and National Chiao Tung University (NCTU), Taiwan; and the China Academy of Telecommunication Research of MIIT (CATR), Beijing. Among the commercial testing software and tools companies, Spirent and Luxoft quickly joined Ixia in the ONF Testing & Interoperability working group, creating a plurality of competing companies that developed products in the custom testing tools market. Slowly but steadily, a team of competitors was being formed. As one would expect, issues about the extent and nature of competition quickly arose. Why multiple labs in a country (in the case of both the US and China)? Why 6 labs and not 3? It was natural to expect providers to want to limit the number of labs to their benefit; in this matter ONF had to be assiduously neutral. If a company was otherwise qualified and wanted to participate in this ecosystem, so long as they
maintained their qualification status, they were welcome to participate. ONF had to make sure that the rules of engagement were clear, and that incentives for collaboration were as clear in areas where open and spirited competition was also expected to take place.
In some aspects of the program, the “team of rivals” aspect worked to ONF’s benefit. Testing results are now double-‐checked by other labs (all working under an NDA for results) to make certain that accuracy exists throughout the program. Labs collaborate with each other, but they also compete by making certain that the level of quality in the program is consistently high. With the addition to a regular audit by ONF for maintenance of each lab’s ISO 17025 certification, there is another layer of integrity and trust for the program.
III. Leadership & Building Community in an Open-Source
Environment
Progress continued as the ecosystem pieces fell into closer interactions, but there were still significant hurdles to overcome. Where was the voice of the customer? How could ONF stimulate the creation of
additional open-‐source software to encourage a community where shared software needs could be built using an open-‐source approach, and where competitors could continue to innovate, differentiate themselves from competitors, and continue to build what was still an immature market? There needed to be additional interactions in order to build out the initial promise of a team of rivals.
In early 2014 ONF inaugurated the Testing Leadership Council (TLC), a forum to bring the ONF-‐approved testing labs and the leading commercial testing manufacturers together in an effort to identify ways of deeper collaboration. Its leadership structure was intentionally balanced—one chair from the labs, and a vice chair from the testing companies, which rotated every year. Membership on the TLC was composed of one representative from every software company, and one from every ONF-‐approved testing lab. Its goals were ambitious—to encourage the growth of open-‐source software to help the testing program, and to help bring greater coherence between the specification development efforts of ONF and the testing infrastructure represented in commercial testing companies and ONF-‐approved testing labs. The TLC is off to a good start, with a goal of developing peer accountability structures for test results (labs working to validate, under NDA, the results of testing that takes place in other labs). Accountability, competition, alignment, and cooperation—ideas and behaviors that once seemed incompatible, seem to be slowly taking root and growing.
One of the final areas in which community will further develop is in the launch of a new online community uniting all the open-‐source software development efforts in open SDN. While isolated projects had been posted to a shared repository (Open Networking Foundation GitHub software repository11), there was little coherence, publicity, or interaction between projects. As such, some code
that had been launched in the early days of SDN had been neglected, and other projects had forked off and were no longer producing for the benefit of a shared community. In the late spring of 2014, ONF began to launch an open-‐source SDN software community that would house OpenFlow-‐Test, Sample Tap (an OpenFlow network tapping application), and the winner of the 2013-‐2014 OpenFlow Driver competition (entries from around the world vying to win a US $50,000 prize for writing the best
OpenFlow reference driver). This software, and more, all written under Apache 2 license, will help drive greater contributions among a community that will go even beyond the bounds of ONF members. The ONF would participate and assist in sponsorship, but a larger community—a meritocracy determined by code, not conditions like membership, would determine the value and direction of software
SDN operations. A representation of the relationships for this open source development is represented in Figure 4.
CONCLUSION
The ONF OpenFlow conformance testing program, a confederation of independent testing labs that provide validation for specification conformance, benchmarking, and interoperability for SDN products, is very much a work in progress. ONF has intentionally (and with some difficulty) created an ecosystem that is at once both competitive and collaborative. Rivals “knock heads” from time to time, and there is spirited discussion over policies and procedures. Transparency, fairness, and clarity are keys that build trust between participants. With time, both the “team” and the “rival” aspects have grown to be understood and appreciated by all participants. The lessons learned in three short years are many, but best summed up by our colleagues from China, who cite the Chinese philosopher Lao Tze:
Deal with the difficult while it is still easy. Solve large problems when they are still small.
Preventing large problems by taking small steps is easier than solving them. By small actions great things are accomplished.
—Lao Tze, Tao Te Ching
ACKNOWLEDGEMENTS
Erica Johnson and Paul To, chair and vice chair of ONF Testing Leadership Council.
BIOGRAPHIES
Rick Bauer ([email protected]) received combined Master of Science degrees in Technology Management from the Wharton School of Business and in Computer Science from the University of Pennsylvania (EMTM Program) in 2001. His career in technology includes stints as director of technology and a CIO; he has also served in several SDOs and trade associations, including The Storage Networking Industry Association (SNIA), and the Computing Technology Industry Association (CompTIA). He presently serves as Technical Program Manager for the Open Networking Foundation.
Ron Milford ([email protected]) received his BA in Electrical Engineering from the University of Illinois at Chicago in 1992. Ron spent more than 15 years as a network engineer, designing and
operating commercial service provider networks. In 2011, he joined InCNTRE at Indiana University and has led interoperability testing for OpenFlow both in the InCNTRE Lab and at industry events. Ron is also co-‐chair of the ONF Testing & Interoperability WG where he helps develop and drive OpenFlow
conformance & interoperability testing.
Li Zhen ([email protected]) received his BA in Image transmission and progressing from XI DIAN University in 1997, and his MA in Project Management from the Graduate School of the Chinese Academy of Sciences in 2002. He is responsible for research, validation and certification of Next
Generation Internet standards, research and development of IPv6 transition and SDN technologies, and research on testing methods. He presently serves as Technical Director for the Beijing Internet Institute.
Author Contact Information
Rick Bauer ([email protected]) 4915 Hidden Rock Road
Colorado Springs, CO 80908 USA (719) 302-‐2273
Ron Milford ([email protected]
Indiana University/Informatics and Communications Technology Complex (ICTC) Attn: Ron Milford, Director of InCNTRE
535 West Michigan Street Indianapolis, IN 46202 317-‐274-‐0801
Li Zhen ([email protected]) Beijing Internet Institute Attn: Li Zhen
Chaoyang District, Beijing Sanyuan West Bridge Time International Building A 2508 Postal Code: 100028
People’s Republic of China +86 10 58678188 123
ENDNOTES
1 see “Standard Making: A Critical Research Frontier for Information Systems (MISQ Special Issue Workshop
258), The Adoption and Diffusion of Interorganizational System Standards and process Innovations, by Matthew Nelson and Michael J. Shaw. Available at http://www.comp.nus.edu.sg/~teohh/tenure/shaw.pdf
2 https://www.opennetworking.org/about/onf-‐overview
3 An early history of the development of SDN and OpenFlow (“An Intellectual History of Programmable Networks”)
is now available at http://queue.acm.org/detail.cfm?id=2560327 .
4 All ONF meetings and activities are driven in conformance with a strict anti-‐trust policy that would preclude any
organizations could be rigorous about these matters with regard to specification development, yet create a single-‐ source provider of testing services with prices controlled by the organization. For more information on ONF’s antitrust policies in this regard, see https://www.opennetworking.org/about/onf-‐operating-‐documents.
5 Understanding the SDN manufacturing ecosystem was a project that ONF’s Rick Bauer undertook early in his
employment at ONF. Using Michael Porter’s “Five-‐Force Analysis” (from his Competitive Advantage: Creating and
Sustaining Superior Performance (Free Press, 1985) to understand all the drivers in the market, ONF was able to
understand better how to create an open competitive ecosystem that maximized competition.
6 For current information on the OpenFlow specification, please consult the ONF website at
https://www.opennetworking.org/sdn-‐resources/onf-‐specifications/openflow
7 Team of Rivals: the Political Genius of Abraham Lincoln, by Doris Kearns Goodwin. More information available at
http://www.doriskearnsgoodwin.com/books.html
8 For the qualifying criteria for ONF-‐approved labs, please see https://www.opennetworking.org/sdn-‐
resources/onf-‐specifications/openflow-‐conformance#labs
9 http://wiki.wireshark.org/OpenFlow
10 See http://www.standardsbookshop.com/iso-‐17025.htm for the link to this ISO specification, which must be
purchased.
