Thomas Ken-Hsing Lin

(1)

S

SAVI

T

by

Thomas Ken-Hsing Lin

A thesis submitted in conformity with the requirements

for the degree of Master of Applied Science

Edward S. Rogers Sr. Department of Electrical and Computer

Engineering

University of Toronto

(2)

Implementation and Evaluation of an SDN Management System on the SAVI Testbed Thomas Ken-Hsing Lin

Master of Applied Science

Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto

2014

Distributed computing applications often require a variety of infrastructure components, such as storage, computing power, and network access, in order to be deployed. In recent years, the IaaS model of cloud computing has significantly reduced the cost of storage and computing power, thus lowering barriers for developers who wish to deploy such applications. However, allowing administrators and users to exercise fine-grained control over the network fabric itself has been an elusive feature. The Smart Applications on Virtualized Infrastructure (SAVI) testbed is an IaaS cloud aimed at empowering researchers to deploy novel applications and experiments. One of the key traits of the testbed is its attempt to virtualize the entire infrastructure, including non-conventional resources such as FPGAs and wireless access points, while enabling researchers the ability to carry out fine-grained traffic control. This thesis documents the implementation and evaluation of an SDN management system for the SAVI testbed, using a novel infrastructure management concept known as software-defined infrastructure.

(3)

Acknowledgements

It’s been a long 3.5 years completing this Masters, and I’ve grown so much both academically and as a person. Too many people to thank for helping me get through these past years, and too little time. Here’s the short list, and much apologies for anyone I forgot to mention.

To my parents, for their unwavering support and belief in me. To my supervisor, Prof. Leon-Garcia, for believing and taking a chance in me, and for his continual advice, guidance, and support. To my project manager, Hadi Bannazadeh, for his continual technical support and pushing me to find my new limits. To the original SAVI testbed team, Eliot, Hesam, Mo, and Eric, for the great experience working with such smart individuals. To Prof. Liebeherr, for teaching me all the fundamentals of networking, which gave me a solid background for my Masters work, and for serving in my thesis committee. To the staff in the ECE graduate office, and to Vladi, for their administrative support. To my colleagues within the ALG group, for some fascinating conversations and memories, an interesting bunch indeed. To the SAVI team in McGill, for their technical help with the wireless virtualization work. To all the friends I made during my time at UofT, for helping keep me sane and preventing me from becoming a lonely hermit. To the workers in the food industry (specifically, Canton Chili, Tim Hortons, Ideal Catering truck, Wokking on Wheels truck, the no-name white Chinese food truck, 7-11, Subway, and Cora’s Pizza), for helping nourish me with sustenance and caffeine which fuelled my Master’s degree work. To the barkeeps… for you-know-what. Thank you all so much.

On to the next one.

(4)

Acronyms and Abbreviations

AMD Advanced Micro Devices

AMQP Advanced Message Queuing Protocol

ANSI American National Standards Institute

API Application Programming Interface

ARPA Advanced Research Projects Agency

BM Baremetal

BSSID Basic Service Set Identification

C&M Control and Management

CANARIE Canadian Network for the Advancement of Research, Industry, and Education

CPU Central Processing Unit

DHCP Dynamic Host Configuration Protocol

E2E End to End

GENI Global Environment for Network Innovations

GIL Global Interpreter Lock

GSM Global System for Mobile Communications

HDD Hard Drive

HTB Hierarchical Token Bucket

HTTP Hypertext Transfer Protocol

(12)

I/O Input and Output

IaaS Infrastructure as a Service

IAM Identity and Access Management

ICMP Internet Control Message Protocol

IP Internet Protocol

IPC Inter-Process Communication

ISO International Organization for Standardization

ITU International Telecommunication Union

JSON JavaScript Object Notation

MAC Media Access Control

NGN Next Generation Network

NTT Nippon Telegraph and Telephone

OFI OpenFlow Interface

ORION Ontario Research and Innovation Optical Network

OS Operating System

OVS Open vSwitch

PCI Peripheral Component Interconnect

QoS Quality of Service

RAM Random Access Memory

REST Representation State Transfer

RMS Resource Management System

(13)

RPC Remote Procedure Call

SAVI Smart Applications on Virtual Infrastructure

SDI Software-Defined Infrastructure

SDN Software-Defined Network

SQL Structured Query Language

SSD Solid State Drive

SSID Service Set Identification

TCP Transmission Control Protocol

UDP User Datagram Protocol

URL Uniform Resource Locator

VM Virtual Machine

VPN Virtual Private Network

WAP Wireless Access Point

(14)

Chapter 1 Introduction & Background

What we know today as the Internet has revolutionized our world, much like past communication milestones such as the telegraph, telephone, radio, and the telephone. Within the last two decades, the Internet has played a key role in the development of new applications and services, especially those offered via methods of “cloud computing” [1]. Cloud computing is a general term which describes the delivery of a service over a networked connection (i.e. the service comes from “the cloud”), whereby the user need not be concerned with how the service is implemented or operated, just that it is transparently available. While the terms “cloud computing” and “the cloud” may not have been in the public vernacular until sometime within the last decade, it is important to note that the concept itself has been around for much longer. One of the key drivers of cloud computing is the concept of virtualization, wherein a single resource may be partitioned into several virtual resources to be shared amongst different users in a way that is transparent to them. From the user’s point of view, the resource is fully theirs to utilize, and they are blind to the existence of the other virtual resources. Cloud computing is often broken down into three types of service models, each reflecting a different level of virtualization: Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (Iaas) [1]. The SaaS model virtualizes and delivers software and software resources over the Internet to end-users. PaaS, on the other hand, is the virtualization and delivery of platform level resources (e.g. application development and deployment tools) over the Internet. Finally, IaaS aims to virtualize and deliver infrastructure-level services such as storage, computing, and some aspects of networking for users who are not able to purchase their own physical infrastructure. The PaaS and IaaS models support application enablement by offering the tools and resources necessary for users to develop and deploy their own distributed computing applications.

(15)

The proper operation of distributed computing applications often relies on several types of resources. At a high level, these resources can be grouped into three categories: data storage, networking, and computing. A truly flexible application platform should ideally present the ability to elastically provision and decommission each of these resource types on-the-fly, and offer them as a virtualized service to users who wish to use them in the development and delivery of new contents, applications, and services. Such a virtualized application platform would also be ideal as a testbed for researchers engaging in experimentation regarding Future Internet protocols, architectures, and applications. The Smart Applications on Virtualized Infrastructure (SAVI) [2] project includes an attempt to realize such a testbed. The SAVI testbed aims to provide researchers in both academia and industry the ability to conduct their exploratory research on a virtualized infrastructure at scale. While there currently exists other research testbeds that also aim to facilitate at-scale experimentation by researchers [3] [4] [5] [6], the SAVI testbed aims to fully virtualize the infrastructure to enable the flexible and elastic use of the available resources.

The SAVI project postulates that a virtualized application platform requires an effective converged management system that is constantly aware of the state and allocation of all resources of all types, either physical or virtual, in order to leverage the full flexibility and advantages of a virtualized infrastructure. Consider an application, such as a streaming video service, whose efficient and effective performance is influenced by both the state of the underlying compute resources, as well as the network conditions. The application must be able to satisfy certain quality of service metrics, and meet the service level agreements negotiated with the customers of the video service. Thus, there exists a need for a system which enables the users of the application platform to manage not only their computing resources, but also the interconnecting network elements as well.

The purpose of this thesis is to document the design, implementation, and deployment of an infrastructure-aware network management system on cloud infrastructures. The rest of this chapter will be dedicated to covering a brief history of network control and management, the SAVI project which this work directly contributes towards, the description of a novel infrastructure management concept, and finally concluding with an overall outline for this thesis.

(16)

1.1 From Traditional Networking to Software-Defined

Networking

The global interconnection of networks that comprise the Internet is nothing short of a technological sensation. The mere idea that with just a pocket-sized access device, such as a smartphone, a person today can access a large bulk of the repository of human knowledge would have seemed ridiculous to the average man just a quarter of a century ago. The history of the Internet can be traced back to the research efforts of the Advanced Research Projects Agency (ARPA), an agency within the United States Department of Defense, to build a packet-switched network to interconnect various universities and research laboratories throughout the continental United States [7]. At the time, many of these sites had their own existing networks, and ARPA’s aim was to interconnect these individual network “islands” to enable end-to-end communication for researchers and applications [8]. The idea was that since each network island should be open to evolve independently, there would be no global centralized control over the overall end-to-end pathways. Additionally, since an intermediary island and links could go offline or suffer from intermittent failure, the network protocols that were eventually developed utilized distributed computation as a strategy to mitigate this risk. This meant that each individual packet forwarding element can compute and re-compute the packet paths through itself by communicating with its immediate neighbours, achieving a sense of self-management. This decision to use distributed computations ultimately resulted in the first version of the Internet Protocol Suite (TCP/IP) [9], which improved the robustness, reliability, and survivability of the entire network.

As the Internet continued to evolve over the next several decades, the TCP/IP suite has remained ingrained in its design. In the recent decade, under the general term of Next Generation Network (NGN), there has been efforts to consolidate all telecommunication services under a single packet-based network transport architecture. In fact, the ITU-T’s official definition declares that one of the fundamental aspects of an NGN is to be able to resolve different identification schemes to traditional Internet Protocol (IP) addresses such that they may be routed in IP networks [10]. While this may mean that the existing distributed routing protocols that control the traffic over much of the Internet may be utilized in these NGNs, they may not be able to use the link capacities in an efficient manner due to the fact that they make routing decisions on a local level. With the explosive growth of traffic on the Internet in the past decade due to the introduction of new types of services, and the continued growth forecasted in

(17)

the next few years [11] [12], the efficient use of the network’s capacity is growing ever more important. It is clear that continuing to use the current set of protocols and allowing the IP networks to manage themselves is not ideal in the long run.

Researchers in the field of networking have been trying to tackle these issues for a long time through the development of new traffic engineering techniques and/or new network protocols. Both methods face some sense of difficulty in being adopted and tested in wide area networks. In the case of traffic engineering, the network managers must be able to take appropriate action when the network conditions indicate that the existing paths for packet flows in the network are inefficient, which involves dynamically configuring and re-configuring static flows within the switches and routers of the network infrastructure. This requires having direct access to the packet forwarding elements within the network, something network infrastructure administrators may be reluctant to grant to researchers. With the approach of implementing new protocols, the ubiquity of the TCP/IP stack in existing systems makes the adoption of new standards both difficult and expensive. Researchers have thus turned to smaller scale testbeds and datacentres to properly test and hone their novel network management methodologies. However, the task of network management remains difficult.

Many modern day datacentre communication networks are still based on the TCP/IP protocol suite. The forwarding logic of the network are still often based on the traditional distributed protocols that come supported by the forwarding element vendor’s operating systems, which are pre-installed within each forwarding element. This makes it increasingly difficult to manage large scale datacentre networks as a datacentre can easily be comprised of switches and routers belonging to different vendors. Creating an end-to-end path for a new service may involve interacting with different interface systems with different configuration settings. In the case of a network testbed, being locked into the protocols that only the forwarding element vendor’s proprietary operating system supports makes it nearly impossible for researchers involved in Future Internet [13] protocols to experiment directly on the physical network, which would offer the most realistic environment for their experimentations. To address these issues, the emerging concept of Software-Defined Networking (SDN) [14], which decouples the network control plane from the data forwarding plane, claims to be able to simplify and improve network management. The separation of the packet forwarding elements from the management plane means that the packet forwarding logic (implemented in the management plane) can evolve separately, without the need to update/replace each

(18)

forwarding element. In addition, the decoupling enables the centralization of the management plane. A centralized manager has the advantage of having a global view of all the forwarding elements in the network, enabling it to see what the best end-to-end paths are, as well as quickly detect and work around any network failures. SDN has often been said to be causing a paradigm shift in the field of networking, changing how networks are planned, deployed, and operated.

1.1.1 OpenFlow

One method of realizing SDN is OpenFlow [15] [16], which defines a communication protocol between controllers (i.e. the management plane) and the various packet forwarding elements of the network infrastructure (i.e. the data plane). Nearly all modern switches and routers contain some sort of hardware flow table that enables packets to be processed at line-rate. Similarly, OpenFlow was designed around the concept of flow tables, with each table entry being defined by a match with an associated action. The design regarding what the match fields are comprised of, and what the possible actions are, was based on the list of common packet headers and packet actions that the majority of switches are able to support. In this way, the OpenFlow designers hoped to lower the barrier for switch vendors to adopt the protocol. The supported match fields of the OpenFlow 1.0 protocol can be seen in Figure 1.1.

When a packet enters an OpenFlow-enabled switch, the switch attempts to match the packet’s headers with the flow table entries it contains. If it does not match any existing entry, it forwards the packet’s information up to a controller to decide what to do with it. When the controller determines an appropriate action for the packet, it sends the action as a directive back to the switch. Additionally, the controller may also update the flow table within the switch with a new entry matching the packet’s headers such that the next packet in the flow will not need to be forwarded up to the controller, enabling hardware line-rate packet processing by the switch. The overall network traffic

(19)

management strategy is thus implemented as software algorithms within the controller. With SDN and OpenFlow, the management of multiple switches can be centralized, and the management interface becomes uniform across all the switches in the network that support the protocol, irrespective of the switch vendor.

1.1.2 FlowVisor

To help facilitate the goal of simultaneous experimentation on the same network infrastructure, FlowVisor [17] [18] was developed as a way to help virtualize OpenFlow-enabled networks into “slices”. Similar to an operating system hypervisor, which supports the execution of multiple guest operating systems over the physical host, FlowVisor allows multiple OpenFlow controllers to co-exist as controllers over the same set of physical network resources, with each controller controlling a slice of the network (see Figure 1.2). FlowVisor is able to virtualize the network in a manner that is transparent to both the data plane as well as the management plane, by using the same OpenFlow protocol to communicate with both planes. In essence, FlowVisor serves as a transparent OpenFlow controller proxy. Each controller may then implement their own unique network management strategy within their slice of the network.

As FlowVisor enables concurrent experimentation over the shared network infrastructure, it is also responsible for ensuring strong isolation between the various slices. It achieves this by partitioning the 12-dimensional space of all possible packet

Figure 1.2: FlowVisor virtualizes network hardware for multiple network controllers [18]

(20)

header combinations into subspaces, called FlowSpaces. The FlowSpaces are defined by a set of FlowSpace rules, where each rule resembles an OpenFlow flow table entry, in that they match packets based on header information. However, rather than having the match be associated with an action, it is instead associated with a controller and a policy. The policy dictates what the controller is able to do within its slice of the network infrastructure. For example, it is possible to just have a controller that does passive monitoring of network statistics, but not be able to modify the flow table entries within the network. Similar to OpenFlow flow table rules, FlowSpace rules have a priority ID in the event that points within the 12-dimensional space are overlapped by two or more subspaces. This slicing mechanism determines which controller OpenFlow messages are destined for in the northbound path (from the data plane to the management plane). In the southbound path, FlowVisor is similarly responsible for ensuring the flow attempting to be installed, or the action that the controller is attempting to execute, doesn’t violate the FlowSpace partitions. If necessary, FlowVisor will re-write southbound flow installation messages in order to enforce slice isolation. This method of network virtualization contributes to the transparency of FlowVisor, and eases the development for researchers and experimenters as they will not need to be concerned with explicitly programming their controller to avoid interference with other people’s slices.

1.2 SAVI Project

The Smart Applications on Virtual Infrastructure (SAVI) [2] project is a partnership between Canadian academic, industry, and research institutes that aim to address key challenges in designing a future applications platform and Future Internet protocols. This work is separated into five research themes:

1. Smart Applications

2. Extended Cloud Computing 3. Smart Converged Edge

4. Integrated Wireless/Optical Access 5. SAVI Application Platform Testbed

(21)

Key to this work is the building of the testbed (theme 5), upon which the work from the other themes, and more, may be implemented and tested. As an enabler of innovation, the SAVI application platform testbed will be designed and built as a flexible IaaS cloud, supporting the ability to virtualize all aspects of the infrastructure in order to rapidly deploy, maintain, and retire large-scale distributed applications [19]. In addition, the SAVI testbed is to support computing resources beyond that of traditional virtual machines (VMs), which are available in many IaaS clouds.

The SAVI testbed is designed to be a multi-tier cloud, comprising Core nodes, Edge nodes, and Access nodes (see Figure 1.3), each of which may be virtualized to support experiments, applications, and services belonging to different tenants [19] [20]. We define a tenant as an isolated slice of the testbed, dedicated to a single project. We note that for the purposes of this thesis, the terms tenant and project may be used interchangeably. A project may involve one or more active users, and a user may belong to several projects simultaneously.

The Core nodes are envisioned to be massive-scale datacentres, capable of supporting many resources. These massive datacentres may be strategically located at sites with renewable energy resources. By comparison, the Edge nodes are small scale datacentres. While similar in design to the Core nodes, the Edge nodes offer a wider range of available

(22)

resources. These Edge nodes are to be strategically located closer to the end-users conducting the experiments, thus facilitating applications that have low-latency requirements with more specialized resources. At the time of writing this thesis, the specialized heterogeneous computing resources available in the Edge nodes include traditional VMs, baremetal servers, general purpose graphics processing units (GPGPUs), BEE2 FPGAs [21], NetFPGAs [22] (both 1GE and 10GE), BEEcube miniBEEs [23], Terasic DE5-Net FPGAs [24], and programmable Software-Defined Radio chipsets. Finally, the Access nodes are sites connected to the Edge nodes that enable the inclusion of end-user clients into the SAVI testbed. The connection between an Access node and an Edge node is over a dedicated and secure link. Access nodes may contain a variety of connectivity options for application end-users (e.g. RJ45 Ethernet jack, Wi-Fi, GSM, and etc.) to connect and associate their client devices with a testbed project.

Figure 1.4 shows the current deployment of the SAVI testbed at the time of writing this thesis. As observed in the figure, the deployment of the SAVI testbed spans 7 operational nodes with an upcoming 8th_{node at Calgary (denoted by a dotted-line}

border). Within the province of Ontario, the nodes are interconnected via the ORION [25] network, which provides a dedicated Layer 2 optical link between the various sites. Between Ontario and other sites, the connection currently utilizes a virtual private network (VPN) over the Internet, though there are plans in the works to connect to them via the CANARIE [26] network in the near future.

The control and management of the Core and Edge datacentre nodes is a major topic of ongoing research in SAVI. As mentioned earlier in this chapter, a virtualized application platform such as the SAVI testbed would benefit greatly from having a

(23)

converged management system aware of all the resources, regardless of their type and vendor. Such a system would be simultaneously aware of the entire infrastructure, which would enable efficient resource utilization while exposing APIs to application developers to leverage the information gleamed from this global view.

1.2.1 OpenStack

To facilitate the building of an IaaS cloud infrastructure, the SAVI testbed team has leveraged the continuously evolving OpenStack [27] set of projects. OpenStack is a collection of open source projects, each aiming to deliver a different aspect of cloud service. Working together as a whole, the various OpenStack projects can be deployed to deliver an IaaS-type cloud. The various project components are interrelated and users can interact with each of them either directly through RESTful APIs [28], or through a front-end project called Horizon [29], which enables users to use a web interface as an alternative. Hundreds of companies, large and small, have joined the OpenStack project to help in its development, and many more use one or more of its projects in their IT infrastructure. One of the benefits of having a continuously evolving project as part of the open source community means that it is very likely to utilize the latest ideas and incorporate the latest technologies. In addition, bugs can be quickly root caused and fixed in a joint effort by the user community at large.

The current deployment of the SAVI testbed utilizes the following OpenStack components:

• Nova: Controller for virtualized computing resources. Considered to be the core project to a functioning IaaS system [30];

• Swift: Object storage manager, offering cloud storage of data. It was designed to store unstructured data that can grow arbitrarily large [31];

• Cinder: Provides a service for block storage, enabling the provisioning of storage devices that can be connected to a VM to act as an external hard drive [32];

• Neutron (formerly named Quantum): A project that aims to provide Networking as a Service. It was designed to be technology-agnostic by functioning

(24)

as a mere database, relying on vendor-specific plug-ins acting as the actuators for managing the infrastructure’s network fabric [33];

• Glance: A service providing the storage of VM images, enabling users to upload newly created images or take snapshots of existing VMs to serve as a new images [34];

• Horizon: OpenStack’s dashboard, providing a web browser-based graphical user interface (GUI) to the various OpenStack projects [29];

• Keystone: A component which provides Identity, Token, Policy, and Catalogue services for the APIs of the various OpenStack projects. It is essentially responsible for implementing Authentication and Authorization [35];

In the months prior to the completion of this thesis, the following three components were also added to expand the SAVI testbed’s capabilities:

• Ceilometer: Converged monitoring and metering component for collecting statistics on different types of resources in the infrastructure. It is designed to be extendable to enable the collection of other statistics, or even on new infrastructure resource types [36];

• Heat: An orchestration project designed to enable management over the lifecycle of applications. It relies on human-readable templates to describe the infrastructure resources necessary for an application, and allows users to boot up, modify, or retire applications in one shot [37];

• Sahara: Enables the quick provisioning and configuration of entire Apache Hadoop clusters in OpenStack, to the benefit of those engaging in big data analytics [38].

While OpenStack enables an easy deployment of an IaaS-type cloud, having central management over the entire system still presents a challenge due to the fact that some projects were designed to be able to function as a stand-alone component. In recent releases of OpenStack, there has been a shift to centralize some of the APIs for various components through the Nova component. In this way, Nova acts as a proxy for various other projects such as the Neutron networking project and the Cinder block storage project. This strategy, however, assumes that the cloud or IT administrators have

(25)

deployed the Nova component. Thus, there is a need for a central management system whose role is dedicated to simply communicating with the various cloud components, while offering APIs that enable users to orchestrate the services provided by said components.

1.2.2 Testbed Management Requirements

Many IaaS systems primarily offer APIs related to the computing resources of the infrastructure, such as virtual machines (VMs). This focus is understandable as hypervisor technology has made VMs ubiquitous in IaaS-type clouds as an easy and cheap way to provide computing capabilities. However, the ability for users to directly control the network is often limited, and is often offered as high level functions and services. In the case of Amazon, for example, users may allocate themselves a pool of IP addresses, configure firewall rules, or obtain load balancers [39]. Allowing users to do fine-grained network management is rare in such systems. Consider the streaming video service application mentioned at the beginning of this chapter. Having a user management system focused on just the compute with little control over the network may increase the difficulty of achieving optimal performance for such an application, as it is clear the application is inherently dependent on both resources working well in tandem. For example, if link congestion occurs in the network, the user should be able to immediately migrate any affected resources to an unaffected part of the network. Likewise, if for example a new compute resource (e.g. video server) is allocated, the user should be able to immediately install end-to-end paths through the network to deliver them to the closest customers without waiting for the network’s distributed protocols to converge. The user thus should be able to achieve some type of fine-grained control via SDN principles.

As explained in [19], the SAVI testbed aims to be a future application platform upon which distributed applications can be deployed, maintained, and retired. Since a future application platform might very well be run on novel Future Internet protocols, another stated goal of the SAVI testbed is to be flexible enough such that it can be shared amongst network researchers while granting the level of control over the network infrastructure that researchers require. This dual requirement requires a control and management system that can provide combined management of compute, network, and other types of resources. Due to the heterogeneity of the resources in the testbed, as well

(26)

as the desire to support the continuous evolution of the testbed infrastructure, the management system envisioned will likely need to be flexible enough to support future extensions. Finally, as an application enablement platform that aims to support innovation, the management system will have to provide open interfaces for external entities and users to interact with it. These interfaces should ideally be flexible enough to abstract the testbed infrastructure’s details (e.g. operational metrics, physical hardware specifications, performance characteristics, etc.) from those wishing to deploy simple applications and experiments. For developers of smart applications, there should also be a set of interfaces available that enable them to query the detailed infrastructure information. For the various testbed components to communicate with each other, there should also be internal interfaces which are restricted from outside users. We summarize these high level requirements of the SAVI testbed management system as follows:

• Converged management over different types of virtualizable resources, including, but not limited to, computing resources and networking resources;

• Extendable management system that allows the infrastructure to evolve;

• Open interfaces for communicating with users and external entities, as well as internal interfaces for inter-component communication.

1.3 Software-Defined Infrastructure

While OpenStack offers many of the desirable features that SAVI envisions to be part of its applications platform testbed, it does not have the capabilities to provide the converged control and management over heterogeneous resources described in subsection 1.2.2 and [19]. In response to this niche, Software-Defined Infrastructure (SDI) was conceived by the SAVI testbed team as a way to meet these converged management needs [40]. SDI is seen as a way to bridge the management of various resource types, defining a top-level manager that communicates with the individual resource controllers (e.g. compute, networking, etc.) in order to orchestrate and coordinate the management of the entire infrastructure. Alongside this top-level SDI manager would be a topology manager responsible for storing the up-to-date information regarding all the resources in throughout the infrastructure, both physical and virtual. The combination of both

(27)

the SDI manager and the topology manager is termed the SDI Resource Management System (RMS) in [41], and a depiction can be seen in Figure 1.5.

The SDI manager would also have open interfaces to enable interaction for users and other external entities. A user who wishes to deploy an application on the infrastructure may use the SDI manager to query an overall view of the infrastructure’s resources and their associated attributes. The information returned should give the user not only the resource information, but also a topological map of where the resources are located in relation to one another. The user can thus use this information to make appropriate deployment decisions and, again via the SDI manager, allocate the resources for their application. While SDI was conceived for the needs of the SAVI testbed, the SAVI project envisions SDI as having the potential to manage a variety of different infrastructures, including but not limited to: enterprise networks, sensor networks, power grids, transportations networks, and etc. This new concept of SDI has motivated us to design and implement a novel SDI management system that can meet the needs of the virtualized application platform testbed that SAVI is building.

1.3.1 Infrastructure-Aware Network Manager

As previously mentioned in section 1.2.1, the SAVI project will leverage some of the existing OpenStack components to aid in building the testbed. In the initial stages of

(28)

developing the SAVI testbed using OpenStack components, we observed that the default networking behaviour of our cloud was a purely flat network, wherein all the VMs were connected by unmanaged switches. Starting in the Folsom release of OpenStack, a new component was introduced called Neutron (called Quantum at the time of release, but renamed due to legal issues) [42]. Neutron functions as a database of all the network-related data for the cloud, and relies on back-end plugins, created by different vendors and companies, in order to actually affect the running configuration of the network fabric. One such plugin was the Ryu [43] plugin, developed by the Nippon Telegraph & Telephone Corporation’s (NTT) Software Innovation Centre [44]. Ryu is an OpenFlow controller also developed by NTT, and the Ryu plugin enabled Neutron to use Ryu to manage the cloud network via SDN principles. The pre-SDI SAVI testbed architecture can be seen in Figure 1.6.

As already mentioned, this strategy of having an SDN controller responsible for the network while a separate manager is responsible for the computing resources may lead to inefficiencies. Thus, our aim is to port the network control logic from the SDN controller up onto the SDI management system. Similar to the benefits gained from

(29)

having a global network view when the network control is centralized in SDN, we expect to gain further benefits when the infrastructure control is centralized, giving the network control logic access to a global infrastructure view. To do this, we will need to design and implement a network control functionality for the SDI management system. In addition, we will also need to design and implement a new plugin for Neutron that can be compatible with our new SDI-based network management system. In essence, we are aiming to create an infrastructure-aware network manager.

1.4 Thesis Organization

In this section we will outline the composition of the rest of this thesis. Chapter 2 will discuss the high level design of three items: the proposed SDI management system, the SDI-based network manager, and the network slicing strategy. Chapter 3 will then focus on the software implementation details of the same three items, and finish off with a preliminary evaluation of the system. Chapter 4 will analyze the results of the preliminary evaluation, investigate methods to improve the system, and finish with a re-evaluation. In addition, chapter 4 will also include points on potential future work to be done to further improve the system. We then go into a real use case example in chapter 5, where we demonstrate how the SDI-based network management system can be used to influence the end-to-end network traffic of an application deployed on the SAVI testbed. The thesis is then concluded in chapter 6 with a summary of the work and the results.

(30)

Chapter 2 Design of the SAVI SDI Manager

In our effort to design a solution to fulfill the desirable features presented in subsection 1.2.2 of the previous chapter, we strive to keep the high-level architectural design as technologically neutral as possible. Details of our implementation will be presented in the next chapter. The design and implementation of the topology manager mentioned in section 1.3 is out of the scope of this thesis, as our focus here is purely on realizing the converged infrastructure management capabilities mentioned in [19] [40]. In this initial design iteration, we simplify our task by focusing on the design of an SDI manager for a single Core or Edge node, rather than tackling the problem of designing a testbed-wide converged infrastructure manager head-on. In keeping with the vision of a generalized manager that can interoperate with diverse infrastructures, we set some high level design principles for the proposed SDI manager:

• Modularity: The SDI manager should be modular by design, with its control and management (C&M) functions defined by pluggable modules. This modularity means that the SDI manager can be quickly re-purposed by simply adding or removing certain modules. This design choice enables the continuous evolution of the infrastructure as well as opens the door for the management of other distributed infrastructures. In addition, this de-coupling between the manager and its functions enables the independent evolution of a module’s implementation (as management requirements changes) as well as that of the SDI manager’s core system.

• Centralized Interfaces: The SDI manager should act as a central point of contact for infrastructure administrators and users alike. Having a centralized set of open interfaces not only makes it easier for the users of the SDI manager, but it also simplifies the upgrading or downgrading of a person’s level of access (i.e.

(31)

if a user is to be “promoted” to have more access, the interface points will remain the same). It is expected that unifying the interface will result in an acceleration of the time for developing applications and experiments, as users will not need to individually interface with several different resource controllers.

• Proxy Controllers: The SDI manager should liaise with resource managers in order to affect the infrastructure state or its resources (seeFigure 1.5). In other words, the SDI manager should never engage in direct control over the infrastructure elements themselves. This de-couples the infrastructure management logic from the implementation details of the physical infrastructure, thus allowing each to evolve independent from one another.

• Global Infrastructure View: The SDI manager should have a persistent and up-to-date topological view of the entire testbed’s resources. This point is key to the success of the SDI manager, as its C&M modules will need to be able to access a global view of the infrastructure in order to effectively control and manage the entire system. This last task is aided by the topology manager, which is being developed separately.

This chapter will present the high-level architectural design of the SDI manager as implemented in SAVI. We have designated the SAVI SDI project, as well as the component itself, as Janus, named from the ancient Roman god of beginnings, translations, gates, doors, passages, endings, and time (the month of January was aptly named after Janus). In addition, this chapter will also describe the design of the SAVI network control module, which runs on top of Janus. In order to frame the design discussion, a brief requirement analysis will be presented for both designs. While the requirement analysis will be conducted with the goals of the SAVI project kept in mind, we aim for the design to be general enough such that it may also be applicable for other datacentre, testbeds, and enterprise networks.

2.1 Design of the SDI Modular Framework

The SDI manager is seen as a way to realize integrated resource C&M functions for datacentres with heterogeneous resources. We envision the SDI manager as a flexible platform upon which different cloud C&M functionalities can be “plugged in” as different

(32)

modules. For this reason, at its very core, the SDI manager is a module manager whose job is to instantiate and enable the various C&M modules. Without any modules enabled, the SDI manager itself effectively does nothing.

2.1.1 Requirement Analysis

We first identify the types or groups of people who will be actively interacting with the SDI manager. The following three groups will comprise the primary stakeholders of the design:

• The Physical Infrastructure Administrators: The owners and administrators of the physical infrastructure will be concerned about how their equipment is being controlled and used by their tenants. The administrators may use the SDI manager in their administrative tasks, and it is also up to the administrators to define and place limitations onto the degree of flexibility granted to the users of the testbed to control and manage their slice of the infrastructure;

• The C&M Module Developers: It is likely that the initial developers of the modules will be from the SAVI testbed team. However it is also possible that modules developed by other users will eventually be granted permission to be hosted on the SDI manager itself, perhaps for the purposes of testing novel infrastructure C&M schemes;

• The Testbed Users: Testbed users are the owners of the applications and experiments. They may need to occasionally interact with the SDI manager in order to achieve customized resource deployment, control, monitoring, measurement, and management over their slice of the infrastructure.

Any module related to control and management will require some way of communicating with external components, such as the proxy resource controllers. The modules will likely also require access to some type of persistent memory for state-aware control and management. Thus, in addition to enabling modules, the SDI manager will offer the following services for each module:

(33)

1. Server(s) that enable APIs for receiving calls from external clients and components;

2. A central database accessible by all modules;

3. A shared events channel that facilities inter-module communication; 4. Set of drivers for communicating with external components.

Figure 2.1 depicts a high level architecture representing the various aspects we require the SDI manager to have. In the subsequent subsections, we will further elaborate on the four items listed above and explain the need for each of them.

2.1.2 API Servers

The first item listed, servers for enabling APIs on the SDI manager, serve three purposes. First, they provide a way for external clients and components to send event notifications (more on this later) to the SDI manager and its modules. Second, they allow for administrators to remotely manage and configure the SDI manager and its various modules. Since these administrative APIs, which will the capability to affect the

(34)

state of the SDI manager, are a potential vector for attackers, they should be protected and wrapped by an authentication and authorization mechanism. However, the design of such protection mechanisms is out of the scope of this discussion. Finally, the API servers will enable the converged point of contact for both administrators and users to manage their applications and experiments running on the testbed. It is up to each module to define its own set of APIs and implement the back-end for handling them; the APIs can be registered with the SDI manager's main servers which will route the calls to their respective back-ends.

2.1.3 Central Database

The second item listed above, the central database accessible by all modules, is a key element for realizing the integrated nature of the SDI manager's C&M capabilities. While different modules are responsible for writing to and updating various sections of the database, they may all jointly read from it and thus glean information from other modules. As an example, consider a Fault Tolerance module whose job it is to keep the various resources running when a fault is detected within the testbed. In a scenario where a network fault is detected, the fault tolerance module would need to identify the virtual resources affected by such a failure and, if possible, migrate those virtual resources (e.g. VMs) to another area of the testbed unaffected by the network fault. Such a module would require access to information within the shared database regarding the current network configuration and state, as well as the current placement of existing virtual resources. Additionally, the database serves as a back-up of vital infrastructure state information in the event that the SDI manager goes offline or crashes, enabling it to recover any lost state or configuration information.

2.1.4 Shared Events Channel

Another key element towards realizing integrated C&M capabilities is the shared events channel listed as the third item above. It is the responsibility of each module to define its own set of unique events. Those developing new modules will have the ability to register callback functions (i.e. code to be run upon the reception of an event, essentially a back-end) for any event, even those which were not defined by the modules

(35)

themselves. Upon the receipt of a shared event, the SDI manager performs a lookup to identify all the modules that have registered to receive the event, and a call to each callback function is made. The shared events channel can thus be interpreted as a bus amongst the various modules, used to send inter-module notifications to each other. External clients and entities may also initiate events by having the SDI manager define an API whose back-end action simply inserts an event into the shared channel.

2.1.5 External Component Drivers

The final and fourth item listed above describes a set of drivers for communicating with external components. This can be thought of as the converse to the API servers mentioned earlier. One of the high level design principles presented at the beginning of this chapter states that the SDI manager should never engage in direct control over the infrastructure elements, and should instead liaise with specific resource controllers instead. These resource controllers are examples of external components. For each external component that the SDI manager is able to take action upon, it requires a driver in order to interact with said component. In essence, these drivers are meant for implementing remote procedure calls (RPCs). Many base driver types may exist for each type of RPC (e.g. Raw TCP/UDP, HTTP, AMQP, etc.), and can be inherited and extended depending on the specifics of the external component's own APIs. Module developers creating new C&M functions for the SDI manager are expected to use these drivers in order to exact control over the infrastructure elements.

2.2 Design of SDI Network Control Module

The primary purpose of the network control module is to enable multiple network control applications to run concurrently, where each application realizes a set of functions regarding SDN control and management. Hence, the network control module is designed foremost as an applications manager. Network control applications, a.k.a. SDN applications, define the behaviour of the network itself. Some examples of SDN applications include a Hub application, a Learning Switch application, a Topology Discovery application, and etc. Without SDN applications, the module is unable to make choices on how to control the network, and thus serves no purpose. The SDN

(36)

applications are able to interact with the various networking devices throughout the infrastructure using a programmatic interface (i.e. a set of APIs) defined within an external component driver designed for SDN controllers. In this way, the SDN applications are agnostic to the SDN controllers being utilized below the SDI manager. From the point of view of the network control module, the SDN controller simply becomes an interface layer used to communicate with the networking devices within the testbed.

2.2.1 Requirements Analysis

In order to design a solution for network control, we must first fully understand what the requirements are. It is vital throughout the design process that we keep the goals of the SAVI testbed in mind. We will begin by identifying the functions that the network control module must support, and then identify any desirable objectives for the module.

Functions

The SAVI network manager shall provide the following functionalities:

• The network manager shall be able to control the networking devices within the testbed;

• The network manager shall be able to define the switching and routing logic over the testbed network;

• The network manager shall support the ability for users to simultaneously exert control over a slice of the network.

Objectives

The design of the SAVI network manager will aim to achieve the following objectives:

• The network manager should be able to load and unload SDN applications on-the-fly;

• The network manager should be able to interact with different types of SDN controllers;

(37)

• The network manager's state should be resilient against software crashes;

• The network manager's state should be open for query to ease debugging efforts.

2.2.2 SDN Controller Drivers

As the SDN applications are written without regard to the SDN controller used to interface with the network below, a common set of APIs must be available to them. Thus, a set of SDN controller drivers, each designed specifically for a certain type of SDN controller, will serve to translate the API calls made by the SDN applications to calls that the SDN controllers can recognize. These SDN controller drivers are an example of the external component drivers described in subsection 2.1.5. Since most SDN controllers available today are based on the OpenFlow [16] protocol, the initial set of APIs available to the SDN applications will mimic the commands that can be taken on OpenFlow-enabled switches. For example, the APIs available to the SDN applications will include commands to install flow rules based on a match-action scheme, and queries for statistics ranging from switch-level, port-level, or flow-level granularities. It is expected that the introduction of any non-OpenFlow SDN protocols that may come in the future will force a review of the available APIs for the applications. The use of these drivers will fulfill the first and second functions mentioned above, while also meeting the second objective.

2.2.3 Event Notifications

The network control module must be constantly aware of any changes to the network configuration in order to make the appropriate control and management decisions. Thus, the network control module implements a set of APIs for external components to communicate and inform the module of network-related events. Network configuration changes may not be limited to just the physical network, but may include changes to virtual networks as well. A few examples of network configuration changes include, but are not limited to:

(38)

• Creation or deletion of virtual networks;

• Creation or deletion of ports;

• Changes to port configurations (e.g. Line speed, port up/down status, etc.). Other network-related events that are unrelated to the network configuration, but which may trigger a configuration change, are events relating to the network traffic. An example taken from OpenFlow is the “PacketIn” event, which informs the network control module of a packet which the network switch does not know how to handle. Another OpenFlow example is the “Flow Removed” event which informs the module of expiring flows within the network switch’s flow table. Depending on the implementation of the SDN application running atop the network control module, some action(s) may be taken to address these events. While it is likely that the majority of these event notifications will originate from the external components, such as the SDN controller, we also envision the possibility that another network-related C&M module (e.g. green networking, fault tolerance, etc.) initiates a change in the network and thus will need to send out notifications to other modules (i.e. through the shared events channel described in subsection 2.1.4) regarding this change.

2.2.4 Network State Context

When the network control module receives network-related event notifications, the information from each notification should be stored within the central database described in subsection 2.1.3. Over time, the aggregation of network event notifications are used to build an up-to-date state view of the network. This state information serves as the primary context upon which the SDN applications may rely on for making accurate control and management decisions. Similarly, the network state context can also be queried by external clients or read by other modules needing information regarding the current network configuration.

(39)

2.2.5 Network Control Logic

The network control is provided by the SDN applications running on top of the network control module. The SDN application is a collection of back-end callback functions that can be registered directly with either the APIs or shared events. These functions will be called upon if and when certain event notifications or APIs are received, and are used to define how the application wishes to handle different events. An OpenFlow-related example is that of a “PacketIn” event, which will trigger some network control logic to decide what to do with the packet.

2.3 Network Slicing

As one of the stated goals of the SAVI testbed is to enable experimentation in Future Internet protocols, users and experimenters will need a way to control the network themselves. Our desire is to design a testbed network that can be sliced at a level low enough such that users can implement any new protocols they wish on top. However the network is sliced, when shared and used by multiple users and experiments, their traffic should remain isolated from one another. The isolation of traffic is needed so as to prevent packets originating from hosts within one experiment from ending up at a host or resource dedicated to another experiment, thus potentially confusing or corrupting the system.

The slicing and isolation mechanisms must be, at all times, directed by the network control module itself. Our policy is to not permit users and external components the ability to directly define network traffic flows. Users wishing to control the traffic within their own slice of the network must make requests via the network control module’s APIs. The reason for this is two-folds:

1. Slicing Accuracy: As the network control module is constantly aware of the network state and configuration, it is well suited to make the decisions on how to slice the network. Any changes to the network configuration which may affect the existing slices or the isolation between the slices (e.g. connecting a new VM to a virtual network) will immediately be known and the appropriate slicing alterations made.

(40)

2. Security: While the network control module provides APIs for users and external components to query its information, and thus can be argued that these external components can make the adequate slicing decisions with this information, there is no guarantee that these external components can be trusted. There is also the possibility that the external components may have been improperly implemented, corrupted, or hijacked by malicious users. Thus, we require any and all attempts by users and external components to define network traffic flows to be vetted by the network control module.

(41)

Chapter 3 Implementation of Janus

This chapter will focus on the implementation details of Janus, the SAVI SDI manager. In addition, implementation details regarding the network control module will also be discussed. Both implementations will be based off of the high level architectural design as presented in the previous chapter. Where appropriate, a brief discussion regarding the design alternatives and a justification of the implementation choices made will also be presented.

We begin this chapter with a brief discussion on the different programming languages which can be used for implementation. Careful consideration must be given as the minutiae of the chosen language will likely affect many factors including ease of development, ease of maintenance, time to deployment, and most importantly, the implementation details themselves. A preliminary evaluation of the initial implementation will also be conducted towards the end of the chapter in order to understand the limits of the prototype system. Tests and measurements will also be done within the SAVI testbed’s production network so as to see whether the initial implementation of Janus and the network controller can adequately handle the network traffic load.

3.1 Programming Language Alternatives

In regards to the selection of a programming language to use for the task of implementing the SAVI SDI manager, there were three primary candidates: C/C++, Java, and Python. Each of these has their advantages and disadvantages, which will be discussed in the subsections below.

(42)

3.1.1 C/C++

C/C++ is a low-level procedural programming language, designed for general purpose application and systems programming. It offers low-level access to memory and requires explicit memory management, thus providing very fine grain control over how memory resources are accessed and utilized. As C/C++ code is compiled down directly into binary machine instructions, which is executed directly on the CPU, it is more efficient (in terms of CPU-cycle utilization) than languages that rely on intermediary interpreters or virtual machines to translate the user’s code into machine instructions at run-time. However, the explicit memory management also makes development a more strenuous task on the part of the developer, as they must be careful not to access areas of the memory not belonging to the program and remember to release memory that are no longer used. In the event of a program crash, it is often difficult to trace as the system provides limited information regarding the root cause of the error, which results in much time spent debugging for inexperienced debuggers or anyone unfamiliar with the structure of the source code. A major downside for developers is the fact that any code written and developed on one system is not assured to work on a system with a different OS or a different CPU architecture, as the code itself (as well as the compiler) may be tailored for each environment. Despite these caveats, the explicit memory management and environment-sensitive nature of the code, as well as the compiler, results in C/C++ being the best choice as far as performance is concerned.

3.1.2 Java

Java is one of the most popular programming languages used in enterprise systems due to the cross-platform nature of the language. Compiled Java code, called byte-code, can easily be used and run directly on machines with different computer architectures. Its cross-platform nature is due to the fact that the byte-code is executed on an intermediary layer, the Java virtual machine, whose job is to translate the compiled Java byte-code to the machine instructions understood by the host system’s CPU architecture. In essence, the Java virtual machine can be considered an interpreter for the Java byte-code, and this type of run-time translation from byte-code to machine instructions is called “just-in-time” compilation. In the event of a program crash, the error will be caught by the Java virtual machine and prevented from potentially corrupting the host system. The Java virtual machine also explicitly controls the amount

(43)

of memory the program is allowed to consume on the host system, and in addition, implements automatic garbage collection, thus not burdening the programmer with explicit memory management. Java’s downsides are primarily related to its performance. As the compiled Java instructions require the Java virtual machine to run, its memory utilization is much more significant compared to C/C++. Not only do Java objects and data structures include an overhead (used by the Java virtual machine), the Java virtual machine itself is a memory-intensive application. Many classes which the programmer may utilize are not compiled into the Java executable but are instead loaded at program start-up time, thus taking a hit on program speed. Finally, the just-in-time compilation of the program byte-code may also hinder the program execution speed.

3.1.3 Python

The last programming language candidate is Python. Python is high-level programming language considered to be easier to read and easier to code with. The language syntax enforces strict indentation, and was designed to use English words and sentence-like structure where possible, thus contributing to its ease of reading. Unlike C or Java, which require some type of compilation prior to the execution of the program, Python relies on an interpreter to read the code at run-time. One of the reasons for this is to enable Python as an on-the-fly scripting language, useable in a shell environment much like Matlab, enabling quick development of algorithms for solving mathematical and scientific problems, as well as quickly prototyping snippets of code. Similar to Java, the interpreter has an automatic garbage collection mechanism, which frees the developer from having to focus on memory management. Variables in Python are not constrained to a single type, and can change dynamically. The intention for this is to enable developers to reuse variables quickly and easily. In the event of a program crash, the interpreter catches the error and automatically prints out a stack trace, indicating the line of failure, thus facilitating debugging efforts. The program execution speed of Python is often slower than C/C++ or Java, due to the fact that the program is interpreted on-the-fly. Another drawback of Python is that since the variables don’t contain static types, and there is no compilation stage, any type mismatch errors or typos will not be detected until the program is executed and it crashes. The use of an interpreter and reliance on garbage collection also means that it will use up more system memory than C/C++, but depending on the application, may use less than Java due to the Java virtual machine. An interesting thing to note regarding Python is that it is

Thomas Ken-Hsing Lin

S

SAVI

T

by

Thomas Ken-Hsing Lin

A thesis submitted in conformity with the requirements

for the degree of Master of Applied Science

Edward S. Rogers Sr. Department of Electrical and Computer

Engineering

University of Toronto

Acknowledgements

Contents

Acronyms and Abbreviations

Chapter 1

Introduction & Background

1.1

From Traditional Networking to Software-Defined

Networking

1.1.1

OpenFlow

1.1.2

FlowVisor

1.2

SAVI Project

1.2.1

OpenStack

1.2.2

Testbed Management Requirements

1.3

Software-Defined Infrastructure

1.3.1

Infrastructure-Aware Network Manager

1.4

Thesis Organization

Chapter 2

Design of the SAVI SDI Manager

2.1

Design of the SDI Modular Framework

2.1.1

Requirement Analysis

2.1.2

API Servers

2.1.3

Central Database

2.1.4

Shared Events Channel

2.1.5

External Component Drivers

2.2

Design of SDI Network Control Module

2.2.1

Requirements Analysis

2.2.2

SDN Controller Drivers

2.2.3

Event Notifications

2.2.4

Network State Context

2.2.5

Network Control Logic

2.3

Network Slicing

Chapter 3

Implementation of Janus

3.1

Programming Language Alternatives

3.1.1

C/C++

3.1.2

Java

3.1.3

Python