Software Defined Networking
and the design of OpenFlow switches
Paolo Giaccone
Notes for the class on “Packet Switch Architectures” Politecnico di Torino
Outline
1
Introduction to SDN
2
OpenFlow
3
Design of OpenFlow switches
Section 1
Introduction to SDN
Network Architecture
Control plane
how to handle traffic (e.g. routing)
Data plane
forward the traffic based on the control plane decision
Traditional networks
Internet architecture
vertically integrated
control and data planes within each device
control plane distributed across the switches/routers
complex and hard to manage
due to distributed control plane
difficult to understand the state of the network and its history
Software Defined Networking (SDN)
Emerging networking paradigm
Separation between control and data plane
routers and switches just acting as forwarding elements
Flow-based forwarding decision
instead of destination-based forwarding
flow
definition: a set of packet field values acting as a
match rule
and
a set of
actions
to operate on all packets belonging to the same flow
unify the behavior of routers, switches, firewalls, load-balancers,
traffic shapers, etc.
Software Defined Networking (SDN)
Logical centralization of the control plane
unique abstract view of the network state (e.g. topology)
control logic moved to an external entity
SDN controller or Network Operating System (NOS)
Network programmability
network applications run on the SDN controller
similar to computer applications running on computer operating system
SDN controller
software platform running on commodity servers
logically centralized
for scalability and reliability reasons, can be distributed in different servers
open-source examples: NOX, POX, Ryu, ONOS, OpenDaylight, etc.
VERSION 2.01 2Network Infrastructure Data forwa rding elem ents (e.g., Op enFlo w switc hes) Open southbound API
Controller(Pla+orm( Network(Applica4on(s)(
Open northbound API
Fig. 1. Simplified view of an SDN architecture.
network operating system1), simplifying policy enforcement and network (re)configuration and evolution [6]. A simplified view of this architecture is shown in Figure1. It is important to emphasize that a logically centralized programmatic model does not postulate a physically centralized system [7]. In fact, the need to guarantee adequate levels of performance, scala-bility, and reliability would preclude such a solution. Instead, production-level SDN network designs resort to physically distributed control planes [7], [8].
The separation of the control plane and the data plane can be realized by means of a well-defined programming interface between the switches and the SDN controller. The controller exercises direct control over the state in the data-plane elements via this well-defined application programming interface (API), as depicted in Figure1. The most notable example of such an API is OpenFlow [9], [10]. An OpenFlow switch has one or more tables of packet-handling rules (flow table). Each rule matches a subset of the traffic and performs certain actions (dropping, forwarding, modifying, etc.) on the traffic. Depending on the rules installed by a controller application, an OpenFlow switch can – instructed by the controller – behave like a router, switch, firewall, or perform other roles (e.g., load balancer, traffic shaper, and in general those of a middlebox).
An important consequence of the software-defined net-working principles is the separation of concerns introduced between the definition of network policies, their implemen-tation in switching hardware, and the forwarding of traffic. This separation is key to the desired flexibility, breaking the network control problem into tractable pieces, and making it easier to create and introduce new abstractions in networking, simplifying network management and facilitating network evolution and innovation.
Although SDN and OpenFlow started as academic experi-ments [9], they gained significant traction in the industry over the past few years. Most vendors of commercial switches now include support of the OpenFlow API in their equipment. The
1We will use these two terms interchangeably.
SDN momentum was strong enough to make Google, Face-book, Yahoo, Microsoft, Verizon, and Deutsche Telekom fund Open Networking Foundation (ONF) [10] with the main goal of promotion and adoption of SDN through open standards development. As the initial concerns with SDN scalability were addressed [11] – in particular the myth that logical centralization implied a physically centralized controller, an issue we will return to later on – SDN ideas have matured and evolved from an academic exercise to a commercial success. Google, for example, has deployed a software-defined network to interconnect its data centers across the globe. This production network has been in deployment for 3 years, helping the company to improve operational efficiency and sig-nificantly reduce costs [8]. VMware’s network virtualization platform, NSX [12], is another example. NSX is a commercial solution that delivers a fully functional network in software, provisioned independent of the underlying networking devices, entirely based around SDN principles. As a final example, the world’s largest IT companies (from carriers and equipment manufacturers to cloud providers and financial-services com-panies) have recently joined SDN consortia such as the ONF and the OpenDaylight initiative [13], another indication of the importance of SDN from an industrial perspective.
A few recent papers have surveyed specific architectural aspects of SDN [14], [15], [16]. An overview of OpenFlow and a short literature review can be found in [14] and [15]. These OpenFlow-oriented surveys present a relatively simpli-fied three-layer stack composed of high-level network services, controllers, and the controller/switch interface. In [16], the authors go a step further by proposing a taxonomy for SDN. However, similarly to the previous works, the survey is limited in terms of scope and it does not provide an in-depth treatment of fundamental aspects of SDN. In essence, existing surveys lack a thorough discussion of the essential building blocks of an SDN such as the network operating systems, program-ming languages, and interfaces. They also fall short on the analysis of cross-layer issues such as scalability, security, and dependability. A more complete overview of ongoing research efforts, challenges, and related standardization activities is also missing.
In this paper, we present, to the best of our knowledge, the most comprehensive literature survey on SDN to date. We organize this survey as depicted in Figure2. We start, in the next two sections, by explaining the context, introducing the motivation for SDN and explaining the main concepts of this new paradigm and how it differs from traditional networking. Our aim in the early part of the survey is also to explain that SDN is not as novel as a technological advance. Indeed, its existence is rooted at the intersection of a series of “old” ideas, technology drivers, and current and future needs. The concepts underlying SDN – the separation of the control and data planes, the flow abstraction upon which forwarding decisions are made, the (logical) centralization of network control, and the ability to program the network – are not novel by themselves [17]. However, the integration of already tested concepts with recent trends in networking – namely the availability of merchant switch silicon and the huge interest in feasible forms of network virtualization – are Image taken from [Kr15]
SDN controller interfaces
Northbound Interface (NBI) API
APIs available to application developers
CLI (Command Line Interface), GUI (Graphical User Interface), REST APIs
e.g. “curl http://controllerIP:controllerPort/command”
abstracts the low-level instructions to program forwarding devices
needed to develop the network applications with any programming
language
Southbound Interface (SBI) API
protocol to access the switches and to send them commands
enables the interaction between control and data plane
Adoption of SDN
Google uses SDN
in WAN to interconnect data centers within the data center
network operators and ISP are also considering to move to SDN
networks
next generation 5G networks are based on SDN in all the layers
Section 2
OpenFlow
OpenFlow
initial idea proposed at Stanford University in 2006
definition by Open Networking Foundation (ONF)
ONS promotes the adoption of SDN through open standards OpenFlow 1.0 released on Dec 2009
OpenFlow 1.5.1 released on Mar 2015
defines the protocol adopted in the southbound interface (SBI)
Image taken from [OF15]
OpenFlow messages
transported on TLS or TCP connections
Packet-in
from the switch to the controller
to transfer the control of a received packet to the controller carries a copy of the packet (possibly, only the header) generated by default in case of forwarding table misses
Packet-out
from the controller to the switch
to send packets out of a specified port on the switch carries the full packet or an ID buffer of the switch
Flow-mod
from the controller to the switch to modify the flow tables
OpenFlow example
Example of flow tables
Ethernet switch (e.g. to reach 68:a8:6d:00:81:42)
Switch MAC MAC Eth IP IP TCP TCP Action
Port src dst type src dst src port dst port
* * 68:a8:6d:00:81:42 * * * * * Forward P1
IP router (e.g. direct delivery on 130.192.9.0/24)
Switch MAC MAC Eth IP IP TCP TCP Action
Port src dst type src dst src port dst port
* * * 0x0800 130.192.9.* * * * Forward P2
Firewall (e.g. block BitTorrent from all Politecnico’s hosts)
Switch MAC MAC Eth IP IP TCP TCP Action
Port src dst type src dst src port dst port
OpenFlow fields
Version
Fields
OpenFlow 1.0
12
OpenFlow 1.3
40
OpenFlow 1.5
44
Examples: switch input port, Ethernet frame type and addresses, VLAN id
and priority, IP DSCP/ECN flags and protocol and addresses, TCP ports
and flags, UDP ports, ICMP type and code, ARP op code and addresses,
MPLS labels and flags, metadata passed between tables.
Section 3
OpenFlow switches
Flow (or Forwarding) Tables
Each rule is a match-action pair
Match
binary exact match (0,1) ternary match (0,1,*)
Action
drop, forward, modify, goto another table
Flexibile
Enables to implement routers,
switches, firewalls, load balancers,
traffic shapers, etc.
Packet processing
Through a sequence of flow
tables
OpenFlow Switch Specification Version 1.5.0
1 Introduction
This document describes the requirements of an OpenFlow Logical Switch. Additional information describing OpenFlow and Software Defined Networking is available on the Open Networking Foundation website (https://www.opennetworking.org/). This specification covers the components and the basic functions of the switch, and the OpenFlow switch protocol to manage an OpenFlow switch from a remote OpenFlow controller.
Port Port Port Port OpenFlow Channel Flow Table Flow Table Flow Table Controller Pipeline OpenFlow Switch OpenFlow Channel Group Table Meter Table Control Channel Controller Datapath Protocol
Figure 1: Main components of an OpenFlow switch. 2 Switch Components
An OpenFlow Logical Switch consists of one or more flow tables and a group table, which perform packet lookups and forwarding, and one or more OpenFlow channels to an external controller (Figure 1). The switch communicates with the controller and the controller manages the switch via the OpenFlow switch protocol.
Using the OpenFlow switch protocol, the controller can add, update, and delete flow entries in flow tables, both reactively (in response to packets) and proactively. Each flow table in the switch contains a set of flow entries; each flow entry consists of match fields, counters, and a set of instructions to apply to matching packets (see 5.2).
Matching starts at the first flow table and may continue to additional flow tables of the pipeline (see 5.1). Flow entries match packets in priority order, with the first matching entry in each table being used (see 5.3). If a matching entry is found, the instructions associated with the specific flow entry are executed (see 5.5). If no match is found in a flow table, the outcome depends on configuration of the
11 © 2014; The Open Networking Foundation
Image taken from [OF15]
Packet processing within each flow table
Matching based only on the header fields
OpenFlow Switch Specification Version 1.5.0Match Find highest priority matching flow entry Apply-actions {list of actions} • modify packet • update match fields • update pipeline fields • if output or group clone packet
Clear-actions
• empty action set
Write-actions {set of actions}
• merge in action set
Goto-table {table-id} Extract header fields Apply Instructions Flow Table Packet Action Set Pipeline Fields Egress Packet clones Execute Action Set Flow Table flow entry flow entry flow entry flow entry flow entry table miss flow entry
Figure 4: Matching and Instruction exectution in a flow table. 5.5 Instructions
Each flow entry contains a set of instructions that are executed when a packet matches the entry. These instructions result in changes to the packet, action set and/or pipeline processing (see Figure 4). A switch is not required to support all instruction types, just those marked “Required Instruction” below. The controller can also query the switch about which of the “Optional Instruction” types it supports.
• Optional Instruction: Apply-Actions action(s): Applies the specific action(s) immediately, without any change to the Action Set. This instruction may be used to modify the packet between two tables or to execute multiple actions of the same type. The actions are specified as a list of actions (see 5.7).
• Optional Instruction: Clear-Actions: Clears all the actions in the action set immediately. • Required Instruction: Write-Actions action(s): Merges the specified set of action(s) into the
current action set (see 5.6). If an action of the given type exists in the current set, overwrite it, otherwise add it. If a set-field action with a given field type exists in the current set, overwrite it, otherwise add it.
• Optional Instruction: Write-Metadata metadata / mask: Writes the masked metadata value into the metadata field. The mask specifies which bits of the metadata register should be modified (i.e. new metadata = old metadata & ˜mask | value & mask).
Image taken from [OF15]
Implementation on NetFPGA
Implementation in FPGA used as reference model for hardware
architecture of any OpenFlow switch
Figure 5: The OpenFlow switch pipeline is similar to the IPv4 Reference Router Pipeline.
8 cycles, the state machine reads the flow headers stored in the two locations indicated by the two hashes for the first packet. While waiting for the results of the lookups for the first packet, the state machine issues read requests for the flow headers using the second packet’s two hashes. In the meantime, the results from the first packet’s read requests are checked for a match. In the case of a hit, the data from the hit address is read. The same is then done for the second packet.
The results of both wildcard and exact-match lookups are sent to an arbiter that decides which result to choose. Once a decision is reached on the actions to take on a packet, the counters for that flow entry are updated and the actions are specified in new headers prepended at the beginning of the packet by the Packet Editor.
The design allows more stages—OpenFlow Action stages— to be added between the Output Port Lookup and the Out-put Queues. These stages can handle optional packet mod-ifications as specified by the actions in the newly appended headers. It is possible to have multiple Action stages in se-ries each doing one of the actions from the flow entry. This allows adding more actions very easily as the specification matures—the current implementation does not support any packet modifications.
5.
RESULTS
In this section we will first give performance results for our OpenFlow switch. We will then compare the complexity of the Output Port Lookup module for the OpenFlow Switch, the IPv4 Router, the NIC, and the learning Ethernet switch.
5.1 Performance Results
Our NetFPGA OpenFlow switch implementation is eval-uated in three dimensions: flow table size, forwarding rate, and new flow insertion rate.
Table Size: While modern enterprise routers and switches can have tables that are hundreds of thousands of entries (our Gates building router—a Cisco Catalyst 6509—can fit 1M prefixes), the total number of active flows at any point in time is much smaller. Data from the 8,000-host network at LBL [20] indicates that the total number of active flows never exceeded 1200 in any one second. Results collected using Argus [21] from the Computer Science and Electrical Engineering network which connects more than 5500 active hosts are shown in Figure 6. We find that the maximum
Table 1: Summary of NetFPGA OpenFlow switch performance.
Pkt Size Forwarding Full Loop
(bytes) (Mbps) (flows/s)
64 1000 61K
512 1000 41K
1024 1000 28K
1518 1000 19K
number of flows active in any one second over a period of 7 days during the end of January, a busy time, only crosses over 9000 once, and stays below 10000. The number of ac-tive flows seen in both the LBL network and the Stanford EE/CS network fit very easily into the NetFPGA OpenFlow switch’s 32,000-entry exact match table.
It is worth noting that at one point, the number of active flows recorded from a single IP address reached more than 50k active flows in one second for a period of two minutes. However, almost all the flows consisted of two packets, one that initiates the flow and one that ends it after 50 seconds. The host that generated all these flows was contacting an AFS server for backup. We suspect the first packet was a request that timed out, while the second packet was the timeout message. The large number of flows is due to the way Argus counts UDP flows. Even when the flow consists of only two packets spread out over 50 seconds, Argus still counts them as active flows. We do not include the results from this single host because this situation could have eas-ily been remedied in an OpenFlow network using a single wildcard table entry. A more optimized version of the im-plementation can handle more than 64,000 entries, so even without the wildcard entry, we would still be able to run the network.
Forwarding: We ran two tests on our NetFPGA Open-Flow switch. First, to test the hardware’s forwarding rate, we inserted entries into the hardware’s flow table and ran streams across all four ports of the NetFPGA. This was done using a NetFPGA packet generator that can transmit pre-determined packets at line-rate. A NetFPGA packet cap-ture device audited the output from the OpenFlow switch to make sure we received the expected packets. The
for-wardingcolumn in table 1 shows that our switch is capable
of running at the full line-rate across 64, 512, 1024, and 1518 packet sizes.
New Flow Insertion: Second, we tested the rate at which new flows can be inserted into the hardware flow ta-ble. This answered the following question: Assuming an infinitely fast OpenFlow controller, at what rate can new flows be coming into the switch before they start getting dropped? The test was run by connecting the NetFPGA switch to an external host that continuously generated new flows. A simple controller implementing a static Ethernet
switch3was run on the same machine as the switch so that
the local OpenFlow switch manager and the OpenFlow con-troller could communicate through memory. We calculated the rate at which new flows were received by the NetFPGA and the rate at which new entries were created in the NetF-PGA flow table. The results are summarized in the full
loopcolumn of table 1.
The main bottleneck in the system is due to the PCI bus—
3i.e. it uses static routes
Image taken from [Na08]
Programmable packet processing
New programmable architecture compatible with any packet
processing
Compatible with OpenFlow
Input Channels Logical Stage 1
...
Switch State (metadata) Se le ct...
... VLIW Action Match Tables Statistics State Prog. Parser Header Pa yl o a d...
Packets 1 K...
Logical Stage N R e co mb in e Output Channels...
1 K Configurable Output Queues Packets New Header(a) RMT model as a sequence of logical Match-Action stages. Physical Stage 1 Physical Stage 2 Logical Stage 1 Logical Stage 2 Physical Stage M
...
: Ingress logical match tables : Egress logical match tables Logical Stage N(b) Flexible match table configuration.
Packet Header Vector Act io n In p u t Se le ct o r (C ro ssb a r) Action Memory OP code
VLIW Instruction Memory
Ctrl Act io n Unit Packet Header Vector
...
Src 1 Src 2 Src 3 Src 1 Src 2 Src 3 OP code (from inst mem) Match Results...
Match Tables Act io n Unit V e ry W id e H e a d e r Bu s(c) VLIW action architecture. Figure 1: RMT model architecture.
2. Flexible Resource Allocation Minimizing Resource Waste: A physical pipeline stage has some resources (e.g., CPU, memory). The resources needed for a logical stage can vary considerably. For example, a firewall may require all ACLs, a core router may require only prefix matches, and an edge router may require some of each. By flexibly allocating phys-ical stages to logphys-ical stages, one can reconfigure the pipeline to metamorphose from a firewall to a core router in the field. The number of physical stages N should be large enough so that a logical stage that uses few resource will waste at most 1/N -th of the resources. Of course, increasing N will increase overhead (wiring, power): in our chip design we chose N = 32 as a compromise between reducing resource wastage and hardware overhead.
3. Layout Optimality: As shown in Figure1b, a logical stage can be assigned more memory by assigning the logical stage to multiple contiguous physical stages. An alternate design is to assign each logical stage to a decoupled set of memories via a crossbar [4]. While this design is more flexi-ble (any memory bank can be allocated to any stage), worst case wire delays between a processing stage and memories will grow at least aspM , which in router chips that require a large amount of memory M can be large. While these
delays can be ameliorated by pipelining, the ultimate chal-lenge in such a design is wiring: unless the current match and action widths (1280 bits) are reduced, running so many wires between every stage and every memory may well be impossible.
In sum, the advantage of Figure1bis that it uses a tiled architecture with short wires whose resources can be recon-figured with minimal waste. We acknowledge two disadvan-tages. First, having a larger number of physical stages seems to inflate power requirements. Second, this implementation architecture conflates processing and memory allocation. A logical stage that wants more processing must be allocated two physical stages, but then it gets twice as much memory even though it may not need it. In practice, neither issue is significant. Our chip design shows that the power used by the stage processors is at most 10% of the overall power usage. Second, in networking most use cases are dominated by memory use, not processing.
2.2 Restrictions for Realizability
The physical pipeline stage architecture needs restrictions to allow terabit-speed realization:
Image taken from [Bo13]
New OpenFlow-related switch designs
OpenState
state machine within each switch
delegates some control from the controller to the switch
refer to [BC14]
P4
protocol-independent packet processors
new programming language
internal state machines within each switch
possible candidate for OpenFlow 2.0
refer to [Bo14]
Bibliography
BC14 Bianchi, Giuseppe, et al. ”OpenState: programming platform-independent stateful openflow applications inside the switch.” ACM SIGCOMM Computer Communication Review 44.2 (2014): 44-51.
Bo13 Bosshart, Pat, et al. ”Forwarding metamorphosis: Fast programmable match-action processing in hardware for SDN.” ACM SIGCOMM Computer Communication Review. Vol. 43. No. 4. ACM, 2013.
Bo14 Bosshart, Pat, et al. ”P4: Programming protocol-independent packet processors.” ACM SIGCOMM Computer Communication Review 44.3 (2014): 87-95.
Kr15 Kreutz, Diego, et al. ”Software-defined networking: A comprehensive survey.” Proceedings of the IEEE, (2015): 14-76.
Na08 Naous, Jad, et al. ”Implementing an OpenFlow switch on the NetFPGA platform.” Proceedings of the 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems. ACM, 2008.