Network Transport for Cloud Users

(1)

This document is published under the Creative Commons license Attribution 3.0 Netherlands. More information about the license can be found at http://creativecommons.org/licenses/by/3.0/nl/

Network Transport for Cloud Users

Sander Boele and Freek Dijkstra, SURFsara December 2013

Project : Research on Networks (RoN)

Projectyear : 2013

Projectmanager : Ronald van der Pol

Authors (s) : Sander Boele and Freek Dijkstra

Releasedate : 31-12-2013

Version : 1.1

Summary

The SURFsara cloud provides a do-it-yourself system where users can create an environment that satisfies their computing needs. Since the data typically originate from a remote source, the users are responsible for setting up the network

infrastructure as well. This article discusses the three main technologies used: Internet, VPN and lightpaths. Each have their own merit, but we found that in particular lightpaths are difficult for users to master, if only because it can be deployed in so many different ways. This article gives an overview of these technologies, along with the pros and cons, and provides insight how these technologies are used in practice.

(2)

2 Colophon

Programme line : Gigaport3

Part : NI hybrid network

Activity : Ethernet OAM

Deliverable : NGE-13-17R

Access rights : public

External party : SURFsara

This project was made possible by the support of SURF, the collaborative organisation for higher education institutes and research institutes aimed at breakthrough innovations in ICT. More

(3)

3 Fact Sheet

Six Matters one should know about Network Transport for Cloud

Users.

Scenario Data transport between a data provider and a cloud provider.

What is it? A document describing the different network setups fort his scenario.

Whom is it for? Users and network operators that want to implement a network connection

to transport data to a high performance cloud.

How does it work? N/A

What can one do with it?

Use as a guideline for the implementation of practical network scenarios, and as a guide for future developments.

More information This document was prepared by Sander Boele and Freek Dijkstra. For more

(4)

4

5

1. Problem Statement

At SURFsara it is possible to bring your own operating systems to the high performance computing cloud (HPC cloud), where users can craft their virtual cluster by defining the amount of CPU cores and nodes and by building their own private networks. Most users start out with their own system where they collect or generate data and process this data. It can be impossible to entirely move this workflow into the cloud when certain hardware or data is only available or

accessible at the home institution or in the scenario where the cloud resources are only used when capacity runs out at the home site. Often it can be a challenge to "scale out" to a different site, in particular to handle the data transport between different sites, and that is the topic of this paper.

1.1. SURFsara cloud network architecture.

The SURFsara HPC cloud provides an Infrastructure as a Service (IaaS). It is run with OpenNebula on KVM and enables users to create their own private networks. There is one common VLAN that all users can place their virtual machines (VMs) in that is routable via the internet, but users can also create their own private LANs. For example in the drawing below Aron has three hosts that are all connected through a private LAN and all connected to the internet VLAN, whereas Karen only has one internet connected head node that she uses to manage her worker nodes. Other setups are also conceivable.

One important element that is missing in this picture is storage. Central storage is used to house the disk images of each virtual machine and to provide a central storage facility that is accessible via various protocols (e.g. iSCSI, NFS, CIFS, CDMI).

(6)

6

This storage is also accessible through Internet connectivity and can be used to upload images and other data. In the picture below this central storage element is added to the mix.

1.2. Network Requirements

The performance of an HPC cloud is often very dependent on the performance the storage can provide and in the case the storage resides at a different site, the network. Another aspect of having data in the central storage is security. Once data is in central storage, it does not have to traverse a potentially hazardous network anymore, it just travels from one rack to the next in a data centre. Another issue that arises when data resides on some institution is that data has to go through a firewall; a firewall that is often a bottleneck in terms of network bandwidth and limits the type of connections (i.e. the protocol used) that can be made.

The network requirements may differ per application, but in general we distinguish between the following requirements:

● Performance, either in terms of bandwidth or latency.

● Security, in particular some data (e.g. medical data) must remain confidential and secure from snooping attempts.

● Firewall policy constraints. A firewall my forbid incoming connections, so a solutions must be used which only uses outgoing connections.

1.3. Mounting a disk versus file transfer

In the scenario discussed in this paper, the compute nodes and the data storage reside at different locations. To access the data, a compute node can remotely access the data by mounting the disk remotely, or the data can be transferred to the compute site prior to the execution of the computation. The advantage of mounting

(7)

7

the data as a disk is that no unnecessary data is transferred. The advantage of a pre-transfer of data is that the latency between the computing node and the data store will be much less, thereby making the computations more effective with less idle time.

1.4. Organisational Entities

We assume a basic scenario:

Data is located at a data provider (e.g. a home organisation of a scientist were the data is generated), and processed by a cloud provider (e.g. SURFsara). The data is transported over the networks of the data provider, the network provider (typically a Research and Education Network like SURFnet), and the network of the cloud provider.

In an IaaS cloud environment the servers are virtual machines operated by the users. So the operator(s) of the data storage and computer servers are one or two

additional entities in addition to the three listed above. Storage and compute servers can be used to create virtual network equipment, like a vSwitch, firewall or VPN endpoint. So in total there may be five entities that operate network equipment, even in this simple scenario.

The next section describes different solutions for network transport. Some of the solutions require interoperable devices or software installed across the networks of the different entities (including the virtual network in the virtual machines).

Regardless if such device or software is installed by one entity in the network of a different entity, or the two entities agree on a certain configuration, these solutions require the entities to collaborate.

In our experience, the solutions that involve the least amount of organisational entities to collaborate are the solutions most likely to be used, even if this is not the best solution. This is not because there is unwillingness to collaborate, but simply an observation that it is easier to implement a solution when less parties are involved. Collaboration requires time and expertise. The users who operate the computer and storage VMs may not be network experts; they usually are scientists with HPC computing needs.

In the opinion of the authors (and many others), users should not be forced to collaborate with other entities in order to configure an optimal data transport solution. Instead, the users should be given the means to configure the best data transport solution themselves so they can do it themselves. This is the do-it-yourself

data provider cloud provider

(8)

8

concept that is advocated by the IaaS concept in the HPC cloud. Unfortunately some solutions may require a network expert to set up (while the operator of the storage and compute VMs are typically scientists rather than network experts), and some solutions won’t work without involvement of other entities.

The solutions are discussed in the next section, while the trade-offs are discussed in the section thereafter.

(9)

9

2. Network Designs

2.1. The Internet

Transferring data via the Internet to a VM in the cloud uses the available general network infrastructure with no extra cost or work involved. This is often the first solution that is tried, and for a lot of applications this suffices.

Characteristics

Performance: Best-effort, as provided by the regular Internet. In practice, we have

seen that large file transfers may saturate the network uplink of institutes with limited bandwidth (like hospitals), which may affect the network performance of other applications.

Secure transport: None at the transport layer; the application needs to provide a

security layer prior to transferring of files. E.g. using SCP instead of FTP for file transfer. This often incurs a performance penalty.

Firewall policy restrictions: Typically subject to restrictive firewall policies.

Data access: Usually only push (file transfer) is possible, due to firewall restrictions

Ease of use: Provided that the cloud provider provides DHCP or NDP, it works out

of the box.

2.2. VPN

A VPN solution is often employed to ensure a secure transfer of sensitive data (e.g. medical data) or to circumvent certain security policies employed by institutions with firewalls. Some may consider this a lightpath-light because security wise it does not matter much if data traverses the internet encrypted or via a lightpath, but with a severe performance penalty both in terms of latency and bandwidth. Typical technologies in use are IPsec tunnels and OpenVPN based systems.

2.2.1. Location of the VPN endpoints

Either the VM operator or the cloud provider may operate the VPN endpoint. In 2013, we planned to create a OpenVPN-based system that would be installed at both the data provider and at the cloud provider. The idea was that the burden of installing a proper VPN would be placed at the organisation that has most

experience with network configurations, and the scientists would simply get a secure channel. In reality, this solution required too much collaboration. In particular, the data provider we briefly spoke to was hesitant to implement it. Setting up such a device in their network was seen a considerable time investment, and our offer to set it up for them was considered with scepticism: network

operators in general are hesitant to place devices in their network, especially what they consider security devices.

(10)

10

We concluded that a better solution would be for the user to run the VPN. Despite that the functionality would be the same, placing the functionality firmly at the edge of the network, in the realm of the end-users makes it more acceptable for most network operators. This is in line with the do-it-yourself philosophy of an IaaS-cloud. What is still required for this to succeed is that it needs to be simplified before users are actually going to use it. In particular, the recommended OpenVPN certificate distribution is considered very complex, and should be simplified.

2.2.2. vxLAN

vxLAN is a technology that aims to bridge two or more data centres by building Ethernet circuit on top of IP. For the purpose of this article, we consider it on par with other VPN tunnel protocols. vxLAN uses a 24-bit identifier to distinguish the different channels and is thus more scalable than VLAN technology which has 12-bit identifiers. While technologies such as MPLS and PBB run on top of Ethernet, vxLAN runs on top of IP. This means that MPLS and PBB are more geared towards use in transport networks, while vxLAN is more geared towards use by data centre networks, since it requires no modification from the transport network provider.

Characteristics

Performance: Generally poor, as it typically runs on top of the Internet. A dedicated

VPN is better than a protocol like SCP, though. The performance hit because of the encryption seems acceptable; modern hardware seems able to handle 10 Gb/s flows. However, more testing is needed, as not all scientists may have access to modern servers.

Secure transport: Excellent security.

Firewall policy restrictions: A VPN connection can be initiated from one side, while the actual data transport inside the VPN tunnel can be initiated from the other side. This means it has the ability to ‘bypass’ a firewall that forbids incoming

connections. A potential issue is that firewalls may block certain protocols required for an IPsec tunnel.

Data access: Either disk mount or file transfer is possible.

Ease of use: If the VM operator sets up the VPN, no other parties need to be

involved. This is preferred, even if the VM operator is not a VPN expert.

2.3. Lightpath

Another way to create a more powerful “scaled-out” virtual HPC cluster is by means of a dedicated network circuit, often called a lightpath. Lightpaths can be

implemented in various ways; they can connect routers, switches and hosts in different configurations.

Because a lightpath does not use the regular Internet, it has a few advantages: 1. It is inherently less prone to interception of confidential traffic;

(11)

11

3. It does not eat up the institutions Internet bandwidth;

4. The performance in terms of latency and bandwidth is both predictable and constant.

2.3.1. Lightpath to a host

By far the most common way to use a lightpath at SURFsara is to connect a host in the cloud through a lightpath. This can for example be a central storage host with a spare network interface or it maybe a data store of an MRI scanner. In the next picture, Karen has added a host to her virtual private network across a lightpath.

At SURFsara this functionality is implemented as follows: the LANs on the virtual switches also exist on the network hardware and a network admin can extend those through a lightpath. Karen would only have to configure the IP address on her VM to be in the same subnet. It is also possible to extend multiple users private LANs through a single lightpath with VLAN tagging to accommodate multiple users at the same institution.

2.3.2. Lightpath to a switch

Suppose Aron is interested in connecting multiple hosts via a lightpath to his private LAN.

(12)

12

He would then hook up a switch to the lightpath. Typically these lightpaths land at a border switch and the VLAN is transported through some layer 2 network.

2.3.3. Lightpath to a router

Next in this epos, Karen decides she wants to aggregate data from her entire cluster that is connected to a router. She unplugs the lightpath from her host and hooks up her cluster router. She configures the address that she had previously used on her host on the router interface and installs a new route to her physical cluster at the institution on her VMs. The new situation is depicted below.

(13)

13

While the scenario in 2.3.2 (switched) and this scenario (router) may seem similar to a non-network expert, the configuration is rather different. This scenario has the advantage that it scales better, in particular if more than one lightpath is involved (as may be the case with projects that are the result of a collaboration with multiple institutes), because the LANs on each side of a lightpath may use IP addresses in different IP ranges. Another advantage of a router is that hosts only need to be placed in one LAN, rather than having two (virtual) network interfaces. A disadvantage is that hosts on either side of the router require an entry in their routing tables. The configuration of a route on a host often needs to be explained to users in detail. Section 3.2 discusses these issues in more detail.

2.3.4. Lightpath between two routers

When there is also the need to access other resources at SURFsara the lightpath connection is usually moved to one of our core routers where the exchange of routing information is done via the Border Gateway Protocol (BGP). The user then loses the ability to access their private LAN, but it can be used to transport a chunk of data to the central storage and the user can always choose to hook up all VMs to the internet. This scenario combines the flexibility of the routed Internet with the performance characteristics of a lightpath.

Aron's VMs

Aron's private LAN Internet

Karen's VMs

Karen's private LAN central storage lightpath Aron's LAN at a different site Aron's hosts at a different site Karen's router at a her institution Karen's cluster at her institution

(14)

14

More advanced scenarios are also possible (though not deployed at SURFsara). A user could also -for example- deploy a VM-router in the cloud to hook up an

institution to a particular cloud slice with a VM running Quagga or pfSense or some other software-based router, but this is often not in the comfort zone of a sysadmin in the business of setting up virtual clusters. This could be implemented as a service in OpenNebula, or a network export could setup the VM.

Characteristics

Performance: Predictable and stable.

Secure transport: Reasonably secure against snooping of traffic, though for even

more protection it is recommended to use additional encryption.

Firewall policy restrictions: Since a lightpath is dedicated in nature, a firewall is

usually not necessary, and full network connectivity is possible between both sides.

Data access: Since full network access is possible, either disk mount or file transfer

(15)

15

Ease of use: Lightpaths are very versatile, as shown by the many solutions in this section. The downside is that it is still necessary to make a specific network design for each lightpath user. Lightpaths also require collaboration between all parties involved: data provider, network provider, cloud provider and sysadmins of the VMs. Since lightpaths typically come at a cost, the organisations also need to agree who pays for it. Given the budgetary constraints of most scientists, this may not be a trivial question.

2.4. Virtual Networking

The use of virtual machines makes it possible and necessary to also virtualise the networking. While this is not the main focus of this article, we briefly mention a few technologies that are relevant for some of the discussions. In particular there is a trade-off if the separation of different network flows (e.g. to distinguish flows from one VM from flows from a different VM) should be done in software or in hardware.

2.4.1. vSwitch

vSwitch (virtual switching) is a technology where a switch is implemented as a VM in software. This allows for flexibility and scalability, since adding a switch to the network is as simple as simply starting another VM. We consider vSwitch as an example of the more general concept of network functions virtualization (NFV) where network functions (switching, firewalling, etc.) partly happen in software. A disadvantage is that these functions take up a lot of CPU, and performance is

typically poor. While there is a clear trend towards this flexibility, the current ASICs (programmable network chipsets) easily outperform the current software solutions.

2.4.2. Network Card Virtualisation

Intel and other vendors have specified the Virtual Machine Device Queue (VMDq) and Single Root I/O Virtualization (SR-IOV) technologies that basically allow a network card to present itself as multiple (virtual) network cards. Each virtual

network card can be assigned to a different VM. The separation of traffic happens in a hardware sorter that resides on the network card, and is based on VLAN or MAC address. The SR-IOV technology even allows direct memory access (DMA) to and from the VM, meaning that network packets can be placed in memory without a software interrupt. This makes this a virtualization technique that easily

outperforms a pure software solution.

2.4.3. OpenFlow

OpenFlow is a network technology that separates the data plane and control plane in networking and also simplifies the network functions of switches and routers, by using generic concepts for the different network technologies. In the context of cloud networking, it allows both easy programmability of the network and

separation of network streams for the different VMs. The technology is relatively new, and the exact usage in practise is still largely unclear.

(16)

16

2.4.4. ScienceDMZ

ScienceDMZ is actually not a network virtualisation technology, but a best-practice to separate small “Internet” data streams (the “mice”) from big scientific data flows (the “elephants”), and also splits a data stream in a local (small latency) part in a data centre and a wide area (long latency) part for the long distance transmission. This greatly improves the performance, on the pretext that there is no packet loss in the wide area network. An older technology that tries to make the same split

between local and wide area network on the TCP layer is Phoebus from Internet2 and University of Delaware. While these technologies are not the focus of this article, the ideas behind these technologies illustrate that even a very small percentage of packet loss have a dramatic negative impact on performance.

2.5. Overview

The following table gives a short overview of advantages and disadvantages of the three solutions mentioned in this article.

Performance Secure Firewall Data access Ease of use

Internet – – – pull ++

VPN – ++ + pull & push +

(17)

17

3. Issues and solutions

In the previous chapter we have begun to touch on this subject, but now we will delve a little deeper.

3.1. Firewall policy restrictions

The most common firewall policy we encounter is that the data provider only allows outbound connections, not incoming connections. This is also the default policy at SURFsara, though it is possible for users to open up the firewall for specific

protocols and ports using the OpenNebula webinterface. Most other cloud providers we are aware of have a permissive firewall that allows all connections by default. A high number of users of the cloud work with medical data. In that scenario the data provider is a medical institute, whose prevalent policy is not to allow

exceptions to the restrictive firewall rules. The technical ideal situation would be that the compute nodes mount a remote disk with (anonymized) medical data, located at the hospital. From a privacy and security point of view, this would be just about the worst solution possible. So this policy is well justified.

A VPN solves this problem, since the VPN can be initiated from the medical centre, and then the disk can still be mounted through the VPN tunnel. A lightpath is

another solution, since lightpaths are typically subject to more liberal policies (since all traffic is guaranteed to come from a known, trusted site).

3.2. Setting routes on Lightpaths

The scenarios above are all about one user with one institution. The story becomes more complicated for everybody to understand and keep track of when there are several lightpaths involved that mix the pictures above. This is especially the case when one or more parties connect a router and routes have to be installed on the VMs.

One issue we had was that one group of people set up the VMs in the cloud and connected a data server to their private LAN in the cloud via a lightpath. They configured a static IP address on the server in the same IP space as their cloud DHCP servers and they were happily connected. A network admin then connected a second lightpath into the private LAN to his router and he could ping the VMs from his router, so he verified his network setup worked. The network admin was not an admin of the VMs, so he could not set a route on the VMs to the machines in his network and it was hard to convince the VM admins how or what they had to do.

3.3. Setting IP addresses on VLANs

The issue with private LANs is that users need to take care of the IP addressing themselves. To make things easier for users, the private LANs at SURFsara by default contain a recursive name server, a network time server and a DHCP server. This DHCP server hands out private (RFC 1918) addresses to known VMs only (as determined by their MAC address), but not to other hosts. So hosts connected via a

(18)

18

lightpath have to configure static addresses on their interfaces facing the lightpath. The usage of RFC 1918 address space is also sometimes problematic when there is an overlap with address space used at one of the institutions connected via the lightpath.

The user can always choose to deactivate the automatic cloud DHCP server, run their own DHCP server, use link-local IP addresses, or use statically configured IP addresses, using whatever IP space they like. The use of IPv6 makes this easier, since it is easier to acquire globally unique address space. Unfortunately, most users still prefer private IPv4 addresses.

3.4. Setting up an MSP from OpenNebula

A new technology that SURFnet is currently rolling out is the so-called multi services port (MSP). Institutions that have such an MSP can dynamically and

programmatically provision circuits to each other across the SURFnet network via VLAN tagging (dot1q tagging). This can be integrated in OpenNebula so that a user can not only build a virtual network between their virtual hosts, but can also extend this network across an MSP to any MSP connected institute without the intervention of a network admin at SURFsara. This is not yet possible at the moment of writing, but we see this as a potential method that may greatly simplify the use of lightpaths for both users and network admins. In our view, users should be able to gain control over the network via an easy to use graphical user interface provided by

(19)

19

4. Conclusion

Users of the high performance cloud infrastructure at SURFsara make use of a do-it-yourself facility. As such, they are responsible for deciding the network solution to transfer data from their data source to the cloud. This article described three main solutions: Internet, VPN and lightpaths, each with its own merits. The regular

Internet is very easy to use, but is otherwise not the best choice, and should be used if a non-secure best effort solution is good enough. VPNs and Lightpath both score well in terms of security and firewall policy flexibility. VPNs are cheaper and easier to deploy, while lightpaths provide the best performance. Lightpaths in particular are hard to use and understand for common users. It is the task of SURFsara as a cloud provider to make the deployment of these technologies as easy as possible for the users. Lightpaths are deployed in different ways. Unfortunately, we have not been able to identify a single deployment scenario that works for all users. If the SURFnet multi service ports can resolve this, or that this can be remedied by more and better documentation remain to be seen. We hope that this document will be a starting point for further documentation.

(20)

20 Acknowledgements

The authors wish to thank the users of the SURFsara high performance cloud infrastructure that we have spoken to about their networking needs. Winners of Enlighten Your Research 3 grants and the network team of the VU medical centre contributed to our understanding of the needs of the users.

Network Transport for Cloud Users