Available at http://www.joics.com
VON/K: A Fast Virtual Overlay Network Embedded in
KVM Hypervisor for High Performance Computing
Yuan Tang∗,
Jianping Li, Yuanyuan Huang
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Abstract
With the emergence of cloud computing, it is possible that the Virtual Machines (VMs) hosting High Performance Computing (HPC) applications seamlessly migrate between distributed cloud resources and tightly-coupled cluster resources. However, the performance of existing virtual computing environment integrated with VMs and overlay networks cannot support the high-performance distributed computing. In this paper, we describe the design and implementation of a virtual overlay network, VON/K, which is integrated with Kernel-based Virtual Machine (KVM). VON/K has negligible latency and bandwidth overheads on 1Gbps Ethernet networks, providing near-native access to high performance networks.
Keywords: Virtual Machine; KVM; Overlay Network; Cloud Computing
1
Introduction
Virtual Machines (VMs) can greatly simplify cloud and distributed computing by lowering the level of abstraction from traditional model. Moreover, the utility of virtual overlay network has been clearly demonstrated in the context of tightly-coupled high performance computing and loosely-coupled cloud computing using VMs [1]. In this environment, an application is mapped into a collection of VMs that are instantiated as needed and interconnected on overlay network. However, the current limiting factor in employing distributed computing for tightly-coupled system is the performance of the virtual overlay network system.
Current overlay network systems have low overhead to effectively host loosely-coupled scalable applications, but the performance is insufficient for tightly-coupled applications [2]. Indeed, for loosely-coupled applications, this concept has readily moved from research to practice [3]. It is well known that cloud computing supports minimal intra-node overhead, but the network infrastructure imposes significant and frequently unpredictable performance penalties. If the overhead of virtual network were sufficiently low, it would be practical to use it in tightly-coupled cluster.
∗Corresponding author.
Email address: [email protected] (Yuan Tang).
In response to the above limitation, we have designed and implemented VON/K, a virtual overlay network that provides a simple layer 2 abstraction: a collection of user’s VMs appear to be attached to the user’s local area network, regardless of their actual locations. VON/K is an implementation of overlay that is embedded into Linux kernel, and integrated with open-sourced Kernel-based Virtual Machine (KVM) [4]. The test results demonstrate that VON/K can achieve near-native bandwidth and latency in the 1 Gbps Ethernet networks. Through the use of low overhead overlay network in high-bandwidth, low-latency environments such as current clusters/supercomputers and future data centers, we seek to make it practical to use overlay network at all times, even when running tightly-coupled applications on such high-end environments. This paper’s contributions are we implement the virtual network extending for VMs to clusters and supercomputers with high performance networks.
The rest of the paper is organized as follows. In Section 2, we introduce the overlay network and KVM virtualization infrastructure. In Section 3, the detailed description of VON/K’s design and implementation is given. The performance of VON/K is evaluated in Section 4. The last section presents the conclusions and future work.
2
Overlay and KVM
Overlay network with a layer 2 abstraction provides a powerful model for virtualized wide-area distributed computing resources on collections of VMs. The following subsections discuss the overlay network, KVM, and their integration.
2.1
Virtual Overlay Network
An overlay network is a computer network which is built on top of another network. The utilities of overlay network have been recognized in wide area distributed environments, such as VIOLIN [5], IPOP [6], ViNe [7]. In these systems, it is necessary for administrators to set up overlay links. VIOLIN project aims to build a service-on-demand grid environment based on virtual server technology and virtual networking. It allows for the dynamic setup of an arbitrary private link layer and network layer virtual network among virtual servers. IPOP is a system that leverages P2P technology to create virtual IP networks. ViNe on the other hand, builds IP overlays on top of the Internet. It is similar to the traditional VPN, but solves some issues with VPN.
Perhaps closest to our work is the VNET [8], which is a layer 2 overlay network for virtual machines, and provides the abstraction of a virtual LAN. VNET is among the fastest virtual net-works implemented using user-level code, achieving 21.5 MB/s [9] with a 1 ms latency overhead communicating between Linux 2.6 VMs running on host machines with dual 2.0 GHz Xeon pro-cessors. These speeds are sufficient for its purpose in providing virtual networking for wide-area and/or loosely-coupled distributed computing. They are not, however, sufficient for use within a cluster at gigabit or greater speeds. VNET is fundamentally limited by the kernel/user space transitions needed to handle a guest’s packet send or receive.
2.2
KVM Virtual Machine Monitor
Virtual world
Guest 1 Guest 2 Guest n Real world
App App
App App App
Linux kernel Virtio
NIC KVM driver
Qemu I/O
Qemu I/O Qemu I/O
Fig.1: KVM based structure
native virtualization using AMD-V or Intel VT-x. By adding virtualization capabilities to a standard Linux kernel (see Fig. 1), the commonality and scalability are brought into virtualized environment. Moreover, paravirtualization is also available for Linux and Windows guests using the Virtio [10] framework.
By integrating into the kernel, the KVM hypervisor automatically tracks the latest hardware and scalability features without additional effort. A normal Linux process has two modes of exe-cution: kernel and user. KVM adds a third mode: guest mode (which has its own kernel and user modes, but these do not interest the hypervisor at all). The virtual computing environments inte-grated with KVM and virtual overlay networks play an important role in distributed computing and cloud computing.
2.3
Integrating Overlay with KVM
VON/K is an implementation of overlay network that is integrated with KVM virtual machine monitor. This model supports adaptive computing on distributed computing resources, and parallel execution in a collection of VMs. The VON/K is designed to be a configurable overlay network, which presents a simple layer 2 networking abstraction that user’s VMs are located at user’s local area Ethernet network, regardless of the actual locations or the complexity of network topology. The VON/K layer can be effectively used to monitor the VMs’ traffic and the performance of the underlying physical network.
We evaluate VON/K, finding that it provides negligible overheads on 1 Gbps Ethernet networks. Moreover, VON/K could be implemented in other VMMs, and as such provides a proof-of-concept that virtual networking for VMs, with performance overheads low enough to be inconsequential even in a cluster environment is clearly possible.
3
VON/K Design and Implementation
3.1
VON/K Architecture
The overall architecture of VON/K is shown in Fig. 2. KVM can be running with multiple guest VMs, and each VM provides a virtual (Ethernet) NIC to its guest. Linux injects all of the network packages to the physical network and gets packets from the network through the Ethernet card. For high performance applications, the virtual NIC conforms to the virtio interface [10]. KVM virtio network device is created by kernel and servers as the Ethernet network card for the guest mode. Their network packets sent to outside network are all sent to VON/K by virtio network card. Then these packets are routed by VON/K inside kernel, and then either inject to the virtio devices of other guests in the same host physical machine, or go to the outside network through the VON/K Bridge. The guest VMs have modifications to support PCI passthrough (DMA addresses are offset) that bridge the guest VMs’ network device and the physical NIC. Using PCI passthrough mechanism, the VMs have direct access to the Ethernet devices.
Guest 1 Guest n App App Qemu I/O Virtio driver App App Qemu I/O Virtio driver Passthrough NIC KVM virtio device KVM virtio device
Linux kernel VON/K core
Fig. 2: VON/K architecture
VON/K comprises of three major components: the Core component responsible for the packet routing, the Bridge and Control components, both of which are implemented as kernel modules in the Linux kernel.
3.2
VON/K Core
The VON/K Core component, directly embedded into the Linux kernel, basically is a packet processing and forwarding system. It is responsible for routing Ethernet packets between virtual NICs on the machine or between this machine and remote VON/K Cores on other machines. All of the forwarding rules are based on the layer 2 addresses. VON/K forwards the incoming packets from variant sources, which include outside network and virtual network devices. VON/K internal packet processing logic is illustrated in Fig. 3.
NIC Linux kernel Routing table Outside network Packet dispatcher VON/K core KVM virtio device KVM virtio device KVM device manager Linux user VON/K control VON/K virtio device VON/K bridge
Fig. 3: VON/K core logic
Package routing Routng table Links list Link 1 Link 2 Interface 1 Interface 2
Fig. 4: VON/K routing logic
In addition to the front-end virtio drivers (implemented in the guest) and the back-end virtio drivers (implemented in the KVM), VON/K virtio defines two layers to support guest-to-KVM communication. To attach front-end drivers to back-end drivers, VON/K Virtio network driver needs three virtual queue interfaces. One is for receiving, the other is for transmitting and the three is for control component.
In the Linux kernel, a special virtio network device called VON/K virtio device is created and configured. The virtio is an abstraction for a set of common emulated devices in a paravirtualized hypervisor developed in recently Linux kernel. This design allows the hypervisor to export a common set of emulated devices and make them available through a common API.
The VON/K virtio device is used as the communication device between the VON/K Bridge and the VON/K Core components. The VON/K Core sends all of the raw Ethernet packets that are destined to the VMs outside of this host machine. Along with the raw Ethernet packet, the destination link information is also affiliated. These are sent to VON/K Bridge through the VON/K virtio device.
on. The destination physical machine will receive all the packets transmitting on the whole local network, and send all of the packets to the VON/K Core, where the packets are routed.
3.3
VON/K Control
VON/K supports the configuration from user level application. The VON/K control component allows for remote and local configuration of interfaces and routing rules so that an overlay network can be constructed and changed. The VON/K configuration console allows for local control to be provided from a file or command. The VON/K control component is responsible for validity checking before it transfers the new configuration to the VON/K Core using the bridge device.
4
Performance Evaluation
We consider communication between two machines whose NICs are directly connected. In the virtualized configuration, the guests and performance testing tools run on top of Linux with VON/K. In the native configuration, the same guest environments run directly on the hardware.
4.1
Testbed
The purpose of our performance evaluation is to determine how close VON/K comes to native throughput and latency in the most demanding (lowest latency, highest throughput) hardware environments. We measure round-trip latency using ping by sending ICMP packets. The through-put is measured using ttcp tool. Our testbed consists of two physical machines called host. The host has a Dual quadcore AMD Processor 2.0 GHz, 4 GB RAM, and 1 Gbps Ethernet NIC. We considered three configurations:
• Native: In this configuration, the VON/K and KVM are not used. A Linux runs directly on the host machines.
• Passthrough: The KVM runs on each host machine, while the VON/K is not used. A single VM, which is configured with a single CPU core and 1GB of RAM, has directly access to the Ethernet devices via PCI passthrough device.
• VON/K: The VON/K configuration corresponds to the architectural diagram given in Fig. 2. The kernel running in VM has the same passthrough model as above configuration.
4.2
Results Analysis
device. For the latter, these are immediately re-vectored into the VM, but, nonetheless, at least one VM exit/entry cost is born on each packet send or receive by the passthrough device in the passthrough or VON/K configurations.
Although latency is much important metric of the network, considering the primary goal of VON/K, which is high throughput virtual overlay networking for cluster based local area net-working, we do care more throughput than latency in our test case. The end to end throughput of VON/K on 1G networking is shown in Fig. 5 (b). Ttcp is configured to use 1450 byte packets sent as fast as possible over 60 seconds. For the 1 Gbps network, VON/K has no difficulty achieving native throughput.
Also, we gave the throughput comparison for these three with different UDP packet, illustrated in Fig. 6. When the size of UDP packet closes to the multiples of 1500 bytes (a standard MTU), the throughput reaches its highest point. With the increase of UDP packet, the tread lines indicate that throughput becomes more stable, and VON/K can achieve near-native throughput.
800 600 400 200 0 Mbits/sec 788 706
Native Passthrough VON/K Native 0.17 0.2 0.35 0.4 0.3 0.2 0.1 0 Passthrough VON/K 767 689 753 676 Tcp Udp ms
(a) End-to-end latency (b) End-to-end throughput
Fig.5: VON/K performance test
750 700 650 600 550 500 450 400 Mbits/sec 1.0 1.2 1.4 1.5 2.0 2.5 2.8 3.0 3.2
UDP pkt size (kbytes)
3.5 3.8 4.0 4.2 4.5 4.6 5.0 5.5 5.8 6.0 1.45
Native Passthrough VON/K
Fig.6: UDP throughput test
5
Conclusion
Ethernet and other high performance interconnects. To achieve high performance, VON/K relies on KVM virtualization in the Linux kernel and high-performance network I/O. Virtualization enables VON/K to provide a simple and flexible level 2 Ethernet network abstraction in a large range of systems. We are currently working to further enhance performance of the VON/K in which tightly-coupled applications can seamlessly migrate to and from heterogeneous data center networks on cloud and HPC environment. The functionality enhancement will be focused on supercomputers in future.
References
[1] J. Lange, K. Pedretti, P. Dinda, C. Bae, P. Bridges, P. Soltero, A. Merritt, Minimal-overhead virtualization of a large scale supercomputer, Proceedings of the 2011 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), March 2011
[2] S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, D. Epema, An Early Performance Analysis of Cloud Computing Services for Scientific Computing, Tech. Rep. PDS2008-006, Delft University of Technology, Dec. 2008
[3] E. Walker, Benchmarking amazon EC2 for high performance scientific computing, USENIX; login, 3(8), Oct. 2008, 18-23
[4] KVM homepage, www.linux-kvm.org
[5] P. Ruth, X. Jiang, D. Xu, Violin: Virtual Internetworking on Overlay Infrastructure, Tech. Rep. CSD TR 03-027, Purdue University, July 2003
[6] A. Ganguly, A. Agrawal, P. O. Boykin, R. Figueiredo, IP over P2P: Enabling self-configuring virtual IP networks for grid computing, Proceedings of the International Parallel and Distributed Processing Symposium (IPDPS), 2006
[7] M. Tsugawa, Jose A. B. Fortes, A virtual network (ViNe) architecture for grid computing, Pro-ceedings of the International Parallel and Distributed Processing Symposium (IPDPS), 2006 [8] Ananth I. Sundararaj, Peter A. Dinda, Towards virtual networks for virtual machine grid
com-puting, Proceedings of the 3rd USENIX Virtual Machine Research and Technology Symposium, May 2004
[9] J. Lange, P. Dinda, Transparent network services via a virtual traffic layer for virtual machines, Proceedings of the 16th IEEE International Symposium on High Performance Distributed Com-puting, June 2007
