Use of HPC and Xen in Cloud Computing

The Internet Cloud is an attractive model since it allows the resources to be dynamically provisioned on a demand basis, i.e. cloud users can rent resources as it becomes necessary. Cloud computing is changing the computing landscape by shifting the hardware and management costs from the computational center to the third parties, e.g. Amazon, Yahoo! or Google . This model motivated several academic and non-academic institutions to develop open-source cloud solutions.

Virtualization is playing a pivotal role in cloud computing, since it enables the cloud provider to dynamically provision compute resources on demand and present them to the user in a transparent manner [41]. It includes presenting a user with a customized OS or presenting a virtualized compute cluster ready for the HPC applications. Unlike the grid and cluster computing, which traditionally run on a bare metal operating system, the cloud runs on top of a hypervisor [41].

Amazon Elastic Compute Cloud (EC2) [9], Microsoft Azure [20] and Eucalyptus [87] are the cloud platforms based on the virtualization technology. Like the het- erogeneous clusters, the cloud is fast becoming an heterogeneous environment due to frequent upgrades and purchase of new machines specific to client requirements. HPC in the cloud is a relatively a new paradigm and there is not much of the performance evaluation and comparative analysis literature available. The Amazon EC2 has a heterogeneous system for provisioning the cloud HPC services. The system consists of Intel Xeon , Intel Quad-core Nehalem, and AMD Opterons. Evangelinos et. al [52] have presented a detailed comparison

of MPI implementations namely LAM/MPI, MPICH , OpenMPI and GridMPI to test Amazon’s HPC cloud. A custom application for the atmosphere-ocean climate model and the NAS parallel benchmarks were utilized to evaluate the system. It was concluded that the performance of Amazon’s Xen based cloud is below the level seen at dedicated supercomputer centers. However, performance is comparable with low-cost cluster systems. Significant performance deficiency arises from the messaging performance, which for communication intensive application is 50% slower compared to the similar ‘non-cloud’ compute infrastructures.

In a similar study, Napper et. al [86] have also cited the communication network as a major bottleneck case in Amazon’s EC2 cloud for HPC systems.

However, the research presented above is specific to Amazon cloud and its hardware. It cannot be generalized to other cloud deployments in HPC.

SnowFlock [76] utilizes Xen for rapid VM cloning in cloud computing. The VM is cloned to multiple VMM hosts in less than a second. The methodology is similar to the post-copy approach. Initially the CPU state is transfered to VMM hosts via multicasting. Memory pages are transfered upon a page fault only. The source VM remains active for the life of the parallel job to ensure that the page faults are entertained; hence it does not represent migration. Although the VM is cloned in less than a second, the wallclock time stretch of a parallel job is in the order of seconds due to the on-demand transfer of memory pages [88].

3.7

Chapter Summary

From the literature and our own experiments [97], it was concluded that the benefits of Xen outweigh the potential bottlenecks and it can be used in HPC specifically for achieving high throughput and easy management. The main benefits of virtualization for HPC are given as under:

• Partitioning of Cluster, Different OS / Libs/ Licenses etc.

• Partitioning of serial and parallel jobs.

• Fault Tolerance through Live Migration.

• Balancing out the conflicting requirements.

• Implementing virtualized clusters.

3.7 Chapter Summary

Despite all these potential advantages, the network performance of Xen hypervisor in the communication intensive applications is questionable. This serious shortcoming needs to be investigated in detail and addressed. For this we have performed a through analysis of Xen’s network performance and proposed novel network configurations for the VMs to improve the undeylying network bandwidth. The details are presented in Chapter 4.

Part

III

Infrastructure Improvement

In this part, we discuss our contributions to optimize Xen for the high performance computing environments. We first address the issue of the slower network communication in Chapter 4. The improvement of Xen’s live migration facility is discussed in Chapter 5.

Chapter

4

Communication Infrastructure

Improvement

In this chapter we present the intra and inter-domain communication character- istics of Xen hypervisor with special emphasis on the HPC applications. Using micro and application-level benchmarks, we give performance evaluation of various network configurations exposed to the virtual machines (compute nodes) by the hypervisor. A comparison of the two distinct MPI-2 specific implementations, namely OpenMPI and MPICH, is also provided. As described in the previous chapter, we found that researchers are wary of the communication network performance of Xen and describe it as a major limitation for using Xen in HPC environments, despite its potential advantages. In this chapter, we propose network configurations for the virtualized HPC compute farm that can result in up to 50% improvement in the runtime of a parallel application compared to the default configuration. The proposed configurations are able to match native Linux performance in certain cases.

4.1

Related Work

As concluded in Chapter 3, the HPC community is a concerned about the performance disadvantages associated with virtualization especially in the case of network I/O [64, 108, 73, 103] and disk I/O [107].

Strazdins et. al [97] evaluated a number of Gigabit ethernet network configu- rations for performance enhancement of SMP clusters under Xen virtualization. Channel bonding and VMM bypass were considered, with comparisons being made to equivalent native Linux configurations. The best performance came from configurations using Xen virtualized hosts with VMM-bypass for network I/O. It

was also concluded that the intra-node communication between Xen guests on the same node is an order of magnitude slower than the native shared memory transport. This counteracts the advantages of VMM-bypass for configuring an SMP cluster with a Xen guest on each CPU. It was suggested to implement a shared memory transport for network communication for the guests sharing same physical host.

Numerous attempts have been made to enhance the intra-domain communica- tion performance of Xen. Among them, Xensocket [108], Xway [73], IVC [64] and Xenloop [103] are notable.