WH I T E PA P E R
Virtual Network Exceleration
OCe14000 Ethernet Network
Adapters
2 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Emulex OCe14000 Ethernet Network Adapters
High Performance Networking for Enterprise Virtualization and the Cloud
AbstractThe OCe14000 family of Ethernet Network Adapters is the latest and most advanced generation of Emulex Ethernet adapters. Powered by the XE100 controller, OCe14000 adapters offer enhanced support for virtualization, cloud and hyperscale cluster deployments. This paper highlights support for Emulex’s Virtual Network eXceleration initiative that delivers support for Virtual Network Fabrics, Single Root I/O Virtualization, Universal Multi-Channel™ and NIC teaming amongst its many capabilities.
Table of Contents
Emulex OneConnect® adapters powered by the XE100 Ethernet controller ... 4
Virtual Network Exceleration™ (VNeX) ... 5
Scaling Virtualization Host I/O Performance ... 6
Windows Server Dynamic VMQ and VMware NetQueue ... 6
Receive Segment Coalescing /Large Receive Offload ... 6
Receive Side Scaling ... 7
Other Offloads ... 7
Scaling a Multi-tenant Cloud Infrastructure ... 7
Virtual Network Fabrics - Overcoming Limitations of 802.1Q VLAN Based Network Isolation ... 7
Virtual Network Fabrics – Delivering Virtual Machine Mobility ... 8
Virtual Network Fabrics – The Path to Software Defined Networking ... 10
Emulex Enhances VNF Performance ... 11
Optimizing Adapter I/O Bandwidth... 11
Single Root I/O Virtualization ... 11
Universal Multi-Channel and Assigned Bandwidth ... 12
Continuous Networking Availability ... 13
3 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Networking – The common foundation
Networking is an essential infrastructure element of today’s data center applications and services. Networking utilization has skyrocketed as the industry has transitioned from the client-server computing paradigm to web-based applications delivery, pervasive virtualization and emergent cloud architectures.
Driven by this paradigm shift, Cisco’s Global Cloud Index1 forecast suggests some fundamental outcomes in networking traffic patterns by 2016:
1. Total network traffic will grow from 1.8 to 6.6 zettabytes/year.
2. 76% of this traffic will stay within the data center.
1Cisco Global Cloud Index
1.8 2.6 3.3 4.1 5.2 6.6 0.00 2.00 4.00 6.00 8.00 2011 2012 2013 2014 2015 2016 Ze tt ab yte s/year
Global Data Center IP Traffic Growth
76%
7% 17%
Traffic Destination Forecast - 2016
4 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
3. 62% of traffic will be in cloud architected data centers, up from 30% in 2011
The IT trends driving these data center network traffic patterns include:
Growth in virtualization-generated, server to server, intra data center traffic such as vMotion, backup, replication and Virtual Desktop Infrastructure (VDI)
Growth of multi-tiered application workloads that increase traffic between database and application servers Adoption of private, public or hybrid cloud infrastructure and software as a service (SaaS)
Increased consumption of internet streaming video and audio services
The technology powering these fundamental transformations in data center computing is 10Gb Ethernet (10GbE). With the launch of its next generation OneConnect® adapters, Emulex is advancing the state of the art in 10GbE connectivity to enable optimized growth of virtualization and cloud computing.
Emulex OneConnect® adapters powered by the XE100 Ethernet controller
Emulex is a leader in high-performance network connectivity, including converged adapter options and enhanced
capabilities for advancing server virtualization that enable a cloud architected infrastructure. The OCe14000 family of 10Gb (10GbE) and 40Gb (40GbE) Ethernet Network Adapters powered by the XE100 controller, is the latest and most advanced server connectivity platform from Emulex. Supporting multiple network and storage protocols with powerful and efficient hardware offloads, OCe14000 adapters have been architected for the dynamic connectivity requirements of today and future-proofed for the increased demands that tomorrow will bring.
New adapters include:
OCe14101 single-port 10Gb Ethernet Network Adapter
OCe14102 dual-port 10Gb Ethernet Network Adapter
OCe14401 single-port 40Gb Ethernet Network Adapter
30% 39%
46% 52% 57% 62%
70% 61% 54% 48% 43% 38%
2011 2012 2013 2014 2015 2016
Workload Shift
5 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Figure 4 – Single-port OCe14101 and Dual-port OCe14102 10GbE AdapterSingle-port OCe14401 40GbE Adapter
The OCe14000 family will deliver the industry’s most comprehensive set of features for enterprise cloud and software defined networking (SDN) deployments. These include:
Greater bandwidth capacity with 40GbE adapters
Hardware offloads for SDN deployments of Virtual Network Fabrics (VNFs) including NVGRE and VXLAN Enterprise hardened and tested iSCSI and Fibre Channel over Ethernet (FCoE) storage protocol offloads with
superior performance, delivering well over one million I/O operations per second (IOPS)
Universal Multi-Channel (UMC) NIC partitioning to optimize bandwidth allocation between workload I/O and virtualization services
Optimization of high-density virtualized servers with Single Root I/O Virtualization (SR-IOV) Driver capabilities for leveraging multi-core processors to maximize VM density
Working with OEM partners to ensure interoperability and broad availability, 10GbE technology developed by Emulex is available as LAN on Motherboard (LOM) controllers and as PCI Express (PCIe) adapters for blade and rack-mount servers. This white paper focuses on the innovations that deliver performance improvements for networks in highly virtualized and emerging cloud architected data centers.
Virtual Network Exceleration™ (VNeX)
The maturation of virtualization deployments has shifted IT management focus to further leverage this technology by: Optimizing server hardware investments by increasing VM density – the number of VMs per host server Virtualizing Tier 1 workloads which have previously been deployed on dedicated server hardware Adopting cloud services architectures in private or hybrid deployments
The paradigm shift from a purely virtualization driven data center to an infrastructure architecture for supporting cloud computing is also shifting the focus from pure applications consolidation to tenant consolidation. A tenant can be a department or a line of business (LoB) in a private cloud or a distinct corporate entity in a hybrid cloud. The networking implications of this shift are discussed further in this paper.
6 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Scaling I/O performance in virtualization hosts to support increasing VM density Scaling networking infrastructure and streamlining deployment for multi-tenant cloud infrastructures Optimizing adapter I/O bandwidth utilization
Ensuring continuous availability of networking resources
Scaling Virtualization Host I/O Performance
Performance enhancements in VM compute capabilities combined with increases in host VM density require advanced NIC capabilities to ensure the efficient and effective scaling of networking performance and prevent networking I/O
bottlenecks. Windows Server® and VMware vSphere® have both introduced several features to enhance networking scalability over several platform generations.
The Emulex OCe14000 adapter family supports these scalability features, which ensures maximum I/O performance for increasingly demanding deployments. These include:
Windows Server Dynamic VMQ and VMware NetQueue
Virtual Machine Queue (VMQ), originally introduced in Windows Server 2008 R2, allowed a VMQ-enabled network adapter to copy received data directly to a VM’s accessible memory. This avoided multiple packet copies and unnecessary virtual switch routing lookups.
Without VMQ, a single processor in a multi-core CPU host runs all network traffic. With VMQ, interrupts are spread across multiple processors and a single physical NIC appears as multiple NICs to the VMs, creating the illusion of a dedicated NIC for each VM. This delivers an overall performance increase for I/O operations and allows a higher host VM density than might otherwise be possible.
In Windows Server 2012, incoming network traffic processing is dynamically assigned to host processors based on processor usage and network I/O load, adding and releasing processors for network traffic. This results in improved network performance, and more importantly, enables the hosting of additional virtual workloads without fear of networking becoming a choke point for increased VM density.
The OCe14000 family fully supports dynamic VMQ and VMQ capabilities across Windows Server platforms.
VMware NetQueue also allows compatible network adapters to deliver network traffic to the system in multiple receive queues that can be processed separately, allowing processing to be scaled to multiple CPUs and improving receive-side networking performance.
When NetQueue is enabled in vSphere, the Emulex OneConnect adapter provides up to eight individual receive queues on each port and offloads Layer 2 (L2) packet sorting and classification functions from the hypervisor. When Ethernet packets are routed to a OneConnect network controller, they are placed into specific queues, allowing packet processing to be scaled to multiple CPUs and significantly improving receive-side networking performance.
Receive Segment Coalescing /Large Receive Offload
7 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Large Receive Offload (LRO) is the equivalent VMware vSphere feature to coalesce TCP packets from the same connection into a large buffer to reduce CPU interrupts. LRO provides up to a 40% improvement2 in both throughput and CPUefficiency.
Receive Side Scaling
Receive Side Scaling (RSS) enables a NIC to distribute and share the processing of I/O interrupts across multiple processor cores instead of burdening a single processor core and potentially limiting the NIC’s receive throughput. In Windows Server 2012, RSS is enhanced to support User Datagram Protocol (UDP) traffic and works on hosts with more than 64 processors3 to achieve highly scalable configurations.
Other Offloads
The OCe14000 family additionally supports other offloads including checksum validation for IPv4/IPv6 and TCP payload in both receive and transmit modes to further unburden the server’s CPUs from network processing tasks.
Scaling a Multi-tenant Cloud Infrastructure
Emulex has participated with VMware and Microsoft to deliver solutions for the problem of scaling a cloud infrastructure. While different in implementation details, these Virtual Networking Fabric (VNF) solutions address two critical
shortcomings of traditional Ethernet networks that limit scalability for cloud architected data centers. Unique Virtual Local Area Network (VLAN) Identifier IDs limited to 4094 hindering tenant expansion
L2 restrictions for VM communications and VM mobility
Virtual Network Fabrics - Overcoming Limitations of 802.1Q VLAN Based Network Isolation
A cloud data center, private or public, supports multiple departments, LoB or individual customers. Each of these entities requires secure isolation of their individual networks, currently accomplished using 802.1Q VLAN tagging.
VMs belonging to a single group (departmental, LoB, corporate etc.) require isolation based on VLAN tagging for their network traffic (since VMs communicate at L2). However, the IEEE 802.1Q standard, with its 12-bit namespace for a VLAN identifier (VID), allows only 4094 VLANs. A top-of-rack switch may connect to dozens of physical servers, each hosting multiple VMs, each of which may belong to at least one VLAN. A data center can easily contain enough switches to quickly
exceed the VLAN ID limit of 4094. For a cloud infrastructure, the importance of network scalability is in its ability to securely provision tens of thousands of
isolated tenant networks. VNFs, also known as overlay networks, solve this problem. VNF solutions, from VMware and Microsoft address this problem through the addition of a new 24-bit identifier to the Ethernet data frame and
encapsulating the frame in an Internet Protocol (IP) packet.
VMware’s VNF solution, known as Virtual Extensible LAN (VXLAN), calls this 24-bit identifier a VXLAN Network Identifier (VNI) tag, while Microsoft’s Network Virtualization using Generic Routing Encapsulation (NVGRE) named it a Tenant Network Identifier (TNI) tag.
In either case, this new 24-bit identification tag enables more than 16 million isolated networks to operate within the same administrative domain, an improvement of many orders of magnitude over the 4,094 VLAN segments discussed above.
2What’s new in VMware vSphere 4
8 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
The figure below shows how the Emulex OCe14000 adds the necessary VNI or TNI tags to support VNF networks. One advantage of Emulex’s approach is enabling VNF networks without requiring upgrades to the switch infrastructure.Figure 5 – Emulex encapsulates VXLAN & NVGRE packets
Virtual Network Fabrics – Delivering Virtual Machine Mobility
Communications between virtual machines (VMs) as well as the mobility of VMs are restricted to the boundaries of a L2 subnet. This limits the scalability of a cloud infrastructure in multiple usage scenarios.
VM migration due to vMotion
Elastic capacity expansion of applications computing resources to service a tenant may require more servers or VMs on a different rack or cluster within a different L2 subnet in a different Layer 3 (L3) network. These additional VMs will need to communicate with the application’s VM on the primary rack or cluster.
Migration of a VM to another server cluster, that is, workload mobility, to optimize usage of the underlying server hardware resources. This may be driven by scenarios such as:
o Migration from, and shut down of underutilized hosts and consolidating on another more fully utilized host to reduce energy costs
o Decommissioning one or more older hosts and bringing up workloads on newly added infrastructure These use cases require “stretching” the tenant’s L2 subnet to connect the servers or VMs since VM-to-VM communications and/or VM migration between hosts currently requires the hosts to be in the same L2 subnet. Both VNF technologies use packet encapsulation strategies discussed above, not only to create a larger number of VLANs for subnets, but to also extend them across L3 networks. Both standards enable multi-tenant networks that can be shared across cloud and on-premise environments.
9 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
The figure below is an example that shows a simplified network topology to better understand the role played by the Emulex OCe14000 in a VXLAN implementation. Readers are encouraged to read the draft submissions to the Internet Engineering Task Force (IETF) to understand the building blocks of a VXLAN4 or NVGRE5 enabled network.Figure 6 – VXLAN Network Topology – The OCe14000 performs VNI encapsulation and de-encapsulation Stretching an L2 VLAN over Layer 3 with VXLAN or NVGRE is conceptually illustrated below.
Figure 7 –VXLAN or NVGRE networks stretch VLANs to span Layer 3
10 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Virtual Network Fabrics – The Path to Software Defined Networking
In the age of pervasive virtualization, an IT manager needs the ability to dynamically scale applications to match changing business needs, streamline the deployment of networking infrastructure and simplify its provisioning, configuration and management. This application-centric approach to networking requires the entire networking stack to operate with increased agility and lower operational hurdles and expenses. SDN is an emerging approach to delivering this agility while ensuring optimal networking performance.
VNF technology is one key enabler for SDN. Today, network connections are managed manually by IT administrators. Untethering VMs from the underlying physical networks and managing rapid and large numbers of VM relocations requires a degree of software automation so that VM network connectivity and mobility can be set up and dynamically managed. Both VXLAN and NVGRE enable this aspect of SDN, and with an intelligent I/O system like the OCe14000 deployed at the edge, enterprises can leverage the distributed edge processing for more effective I/O and network virtualization. The OCe14000 platform provides the required encapsulation services required for the VNF’s ID tagging. Emulex VNeX technology is platform agnostic and provides this service for both VMware vSphere and Windows Server.
Figure 8 – Intelligent I/O Systems Offload Virtual Switching Functions
11 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Emulex Enhances VNF Performance
Earlier in this paper, we discussed the role played by various I/O offload technologies that scale networking performance. It is imperative to maintain these long-standing networking enhancements while onboarding VNFs into the datacenter. Innovation from Emulex underpins the recent announcement6 of OCe14000 support for the coexistence of hardware offloads and VXLAN in vSphere 5.5.
This coexistence ensures continued delivery of high performance networking capability while delivering the benefits of VNF deployments using OCe14000.These benefits include:
Up to 70% better throughput compared to non-offloaded network adapters
Utilization of 50% less CPU resources versus network adapters that use RSS
Optimizing Adapter I/O Bandwidth
The performance enhancements in VM compute capabilities and host virtualization density require corresponding improvements in scaling networking performance. As more workloads are virtualized and consolidated, the network has the potential to become a performance bottleneck. One of the solutions for this bottleneck is to offload the hypervisor based software emulation of I/O devices to specialized hardware and device drivers.
Single Root I/O Virtualization
The purpose of SR-IOV is to enable a “slice” of the NIC called the virtual function (VF) to be assigned directly to a virtual machine, thus allowing VM networking I/O to bypass the host and move on a path directly between the VM and the NIC, eliminating redundant I/O processing in the hypervisor. This, in turn, allows higher I/O performance and lower CPU utilization as compared to the alternative of software-emulated NIC devices that are implemented in the hypervisor. SR-IOV is a part of the PCIe specification that enables multiple VMs to directly access the NIC’s physical I/O resources. The SR-IOV specification includes physical functions (PFs) and VFs and allows a single PCIe compliant network adapter, like the OCe14000, to appear as multiple physical and virtual devices, using the concept of physical and virtual functions.
Physical function — The PF is a full featured PCIe function of a NIC that supports the SR-IOV interface. The PF includes the SR-IOV extended capability.
Virtual function — A VF is a lightweight PCIe function on a NIC that supports the SR-IOV interface. VFs are associated with VMs and are limited to processing I/O streams, and moving data. The number of supported PFs and VFs can vary; Emulex’s OCe14000 supports up to 8 PFs and up to 63 VFs.
SR-IOV is supported by VMware vSphere 5.1 (and higher), Windows Server 2012(and higher), Red Hat Enterprise Linux 6.0 (and higher) and Solaris.
12 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Figure 8 – Network I/O Path with and without SR-IOVBenefits of SR-IOV enabled NICs include:
Increased network throughput
Reduced network latency
Reduced host CPU overhead required for processing network traffic
This reduced CPU overhead for network traffic processing frees up resources for increasing VM density, thus optimizing the usage of host processors.
Universal Multi-Channel and Assigned Bandwidth
Universal Multi-Channel (UMC) delivers similar functionality to SR-IOV in that a single PCIe NIC port is split into multiple PCI functions. The current implementation enables up to four NIC functions per port of an OCe14000 adapter. Each function is presented to the operating system or hypervisor as a distinct physical port with a separate MAC address – thus allowing up to 8 functions per dual-port OCe14000 adapter.
Additionally, Emulex OCe14000 converged network adapters maximize configuration flexibility for storage environments with full protocol offload for FCoE or iSCSI processing. Network administrators can use UMC to partition a single port to provide specialized NIC functions for different uses such as virtual machine migration and console management, and also assign specific bandwidth for high-demand applications. Because each function appears to be a separate physical port, there is no requirement for changes to operating systems or hypervisors.
13 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
Figure 9 –UMC enables partitioning a single NIC port into up to 4 networking portsThe key benefits of deploying Emulex UMC technology include:
Lower Total Cost of Ownership (TCO)
o Consolidates multiple 1GbE adapters, associated cables and switch ports
o Higher VM workload bandwidth allocation to drive higher VM density on host servers o FCoE or iSCSI storage options save capital expenditures for a dedicated storage adapter o Lower per-Gb bandwidth cost compared to deploying multiple 1GbE adapters
Optimized I/O Utilization
o Granular bandwidth provisioning minimizes wasted idle bandwidth and waste of dedicated 1GbE adapters
o Enables Service Level Agreement (SLA) based provisioning and deployment
Simplified Deployment
o UMC is not dependent on specialized OS support o Works with any 10GbE switch
Continuous Networking Availability
A key requirement for a virtualized or a cloud data center network is resiliency through redundancy. With multiple VM workloads aggregated on a single adapter port, networking I/O is a strategic infrastructure resource. The OCe14000 ensures networking availability for multiple, aggregated VM workloads dependent on a single 10GbE connection for both VMware vSphere and Windows Server 2012 platforms.
NIC teaming, also called load balance and failover (LBFO) or NIC bonding, is the combination of two or more NICs acting as one. If one NIC fails the other adapter can still maintain network connectivity, ensuring continuous availability and
14 Emulex White Paper
Virtual Network Exceleration – OCe14000 Ethernet Network Adapters
NIC teaming and failover has been supported by Emulex network adapters since ESX 3.5. Windows Server 2012 integrates and enables vendor agnostic NIC teaming, meaning users are no longer constrained to work with legacy and proprietary solutions7.This also means that the NICs from different vendors can be teamed as long as they have passed the Windows Hardware Qualification and Logo (WHQL) test. The Emulex OCe14000 is qualified for continuous availability solutions in Windows Server 2012. Microsoft previously did not support NIC teaming8 in Hyper-V.
Conclusion
The evolving data center is experiencing architectural shifts from embryonic to pervasive deployment of virtualization concurrently with the introduction of multi-tenant private, public and hybrid cloud infrastructures.
Emulex’s OCe14000 platform delivers networking capabilities that provide multiple benefits to support this new data center architecture, including:
Delivering increased VM density to maximize infrastructure ROI
Enabling a secure and scalable cloud multi-tenant infrastructure
Optimizing I/O bandwidth allocation to match workload and management requirements
Ensuring continuous availability for mission critical computing
The Emulex solution includes critical new capabilities such as full hardware support for VNF and other stateless offloads that ensure optimum utilization of host computing horsepower for maximizing VM density. The OCe14000 also supports SR-IOV to maximize VM networking I/O throughput without burdening host computing resources further minimizing adapter TCO. NIC teaming technology has been standardized in the most widely used operating system platforms – users are no longer constrained to work with legacy NIC solutions from the 1GbE era.
