Virtual Machine Dynamic Provision and Image
Management for UniCloud
Pangfeng Liu
April 10, 2010
1
Related work
1.1
Virtual Machine
1.1.1 Application performance isolation in virtualization
Performance isolation is that performance of one VM should not affect per-formance of another VM in the same computer, which application providers concerned about because there are several kinds of applications in different vir-tual machine on a same physical machine. Therefore, Somani and Chaudhary presented the good isolation with credit scheduler of Xen, and the other sched-uler called SEDF has shown the better performance than credit schedsched-uler in [14].
1.1.2 Live Migration of Virtual Machines
Clark et al. compared original migration (stop-and-copy) with live migration (pre-copy), and observed performance under VM migration with bandwidth and the number of pre-copy iterations. The result in [9]is that pre-copy is always better than stop-and-copy. In addition, more pre-copy iterations and more bandwidth can effective reduce downtime. In these year, xen and vmware has supported the live migration.
1.1.3 Post-Copy Based Live Virtual Machine Migration Using Adap-tive Pre-Paging and Dynamic Self-Ballooning
Post-Copy is a new method of migration compared with Pre-Copy. Post-Copy migration transfer VM’s processor state to target host first, and then send memory contents [10]. As the result, all the memory content will be transferred at most once, which Pre-Copy migration scheme will iterative transfer the dirty
1.2
Cloud Infrastructure and Management
1.2.1 Memory Buddies: Exploiting Page Sharing for Smart Coloca-tion in Virtualized Data Centers
Memory Buddies [15] is a memory sharing-aware placement system for virtual machines. The goal is to maximize the number of VMs in a data center. It uses the VMware ESX [5] technique content-based page sharing to share pages from different VMs in the same host. The algorithm uses fingerprint to predict what kind of combination saving the most memory pages.
1.2.2 Virtual Organization Cluster
Murphy et al. proposed the idea of using one cluster to simulate several virtual clusters to support virtual organizations [11]. The virtual cluster for individual virtual organization is called virtual organization cluster. Every virtual organi-zation provides VM images they needed. Cluster administrators run a virtual cluster supporting each virtual organization.
1.2.3 Dynamic Provisioning of Virtual Organization Clusters Based on virtual organization cluster [11], Murphy et al. further refine the virtue machine framework so that it can dynamically adjust the number of VMs to support virtual organization [12]. They use condor system and maintain a job queue for system to detect the load of each virtual organization. Using this information, they can decide VM numbers for different virtual organization in different situation.
1.2.4 An Efficient Resource Management System For On-line Vir-tual Cluster Provision
Chen et al. studied how to reduce image preparation overhead by VM im-age caching and multicasting VM imim-age to physical machines [8]. Also the Load-Aware Mapping,a resource mapping strategy, can further reduce deploy-ing overhead by awardeploy-ing VM image cache information and make resource used efficiently.
1.2.5 Dynamic Provisioning of Resources in Data Centers
Simmons et al. proposed the idea of how to allocate the resources in the data center [13]. In this paper, it describes some policies that we should concern to make th balance between the datacenter and the customers. Finally, decisions are based on the model which combines all the policies.
1.3
Tools
1.3.1 Xen
Xen [7] is an open source virtual machine monitor. It can execute sereval guest OS on the same machine. Xen systems include Xen hypervisor as the lowest and most privileged layer to schedule different guest OS across the physical CPUs. It needs modified Unix-like OS to be domain 0 and domain U.
1.3.2 Hadoop
Hadoop [2] is a open source software framework which support data-intensive distributed applications. Its capable for thousands of nodes and petabytes of data for applications. Hadoop also include a distribute file system inspired by Googles GFS and the framework is by Googles MapReduce.
1.3.3 MPI
MPI [3], which stand for message passing interface, allows many computers com-municate with one another. MPI specify a communication protocol for parallel computing. It is a message-passing application interface for programmers. MPI is common used in high-performance computing.
1.3.4 GlusterFS
GlusterFS is a application level distributed share file system [1]. it has client and server components. Server side store the data, And client side using TCP/IP to access data on servers. GlusterFS has two modes to store data. Those modes are file-based striping and file-based mirroring ,respectively.
1.3.5 OpenNebula
OpenNebula can provide an interface to manage vitualInfrastructure including storage, network [4]. And it also provide virtualization technologies to enable migration and deploying VM on distributed system.
1.3.6 XCP- Xen Cloud Platfrom
XCP [6]is a platfrom for the cloud enviroment. It collects several tools in the previous products, and combine the xen machines to a resource pool. Therefore, it can monitor VMs, do load balancing and migrate automatically the pool.
there are some machines can not run VM. These machines is grid01,grid02,grid04 and grid05. Now we test xen 4.0 on grid04 and grid05. It works! Although xen 4.0 is unstable, it fixes many problems in xen 3.2. Now, we describe how we install xen 4.0/3.2 and how to create a VM image.
2.1.1 Install Xen 3.2
We use debian packages to install xen 3.2. We just type “apt-get install” to install packages. It needs packages xen-linux-system-2.6.26-2-xen-686,libc6-xen and bridge-utils. Make sure that menu.lst is correct. It must boot by xen hypervisor. And modify /etc/xen/xend-config.sxp file if need. For example, if you want to do live migration, the line “#(xend-relocation-hosts-allow ’ ’)” must deletes the # sign.
2.1.2 Install Xen 4.0/4.1 unstable version
Type “hg clone http://xenbits.xen.org/xen-unstable.hgs” to download the source code. Enter the directory. Type “make xen; make tools; make install-xen; make install-tools”. At this step, you may need to install some packages like bin86,dev86,etc. Go to http://git.kernel.org/ to download modified kernel sup-porting xen. Compile new kernel for booting. Modify menu.lst and xend-config.sxp. Boot in new kernel.
2.1.3 Create VM images
First, we use dd to create two images, one for disk and one for swap. And we use mkfs.ext3 and mkswap commands to format this two images. In order to install a basic Debian system in our image, we mount the image and run debootstrap. For example, we type “mount -o loop disk /mnt; debootstrap –arch i386 lenny /mnt http://ftp2.de.debian.org/debian”. In this step, debootstrap will ask us to set up root password. Next, we set up our VM environment like network, fstab, hostname, etc. Modify VM configure file. There is an example:
kernel = ’/boot/vmlinuz-2.6.26-2-xen-686’ ramdisk = ’/boot/initrd.img-2.6.26-2-xen-686’ memory = ’512’
root = ’/dev/hda1 ro’ disk = [ ’file:/testvm/swap.img,hda2,w’, ’file:/testvm/disk.img,hda1,w’, ] name = ’testvm1’ vif = [ ’ ’ ] on poweroff = ’destroy’ on reboot = ’restart’ on crash = ’restart’
extra = ’console=hvc0 xencons=tty’
2.2
OpenNEbula
2.2.1 description
we use OpenNebula(ONE) [4] v1.4 to build private cloud on ubuntu 8.04.
2.2.2 before-install
We need install ruby, sqlite3, xmlrpc-c, openssl, ssh, scons, g++, flex, blson on ONE server and Install ruby on all cluster nodes. All cluster nodes should have ssh server running and hypervisor working properly configured and ssh-key exchange to ONE server. Build a share file system which can be accessed by all nodes in order to live migration VM. in our case we use NFS or GlusterFS [1] to build share file system.
2.2.3 install
Download ONE and install it on share file system. 2.2.4 configure
put all VM images on share filesystem. configure etc/one.conf as follow(let ONE use share image mode)
TM MAD = [ name = ”tm nfs”, executable = ”one tm”,
NAME = xen hdp CPU = 1
MEMORY = 2048
OS = [kernel=/boot/vmlinuz-2.6.24-27-xen,initrd=/boot/initrd.img-2.6.24-27-xen,root=hda2 ] DISK = [source= /srv/cloud/xen hdp/disk.img,target=hda2,readonly=”no”]
DISK = [source= /srv/cloud/xen hdp/swap.img,type =swap,size =512,target=hda1,readonly=”no”] NIC = [ip=””]
GRAPHICS = [type=”vnc”,listen=”127.0.0.1”,port=”5900”]
we can use this VM configure file to create VM. And then we can depoly VM on spicific cluster node. Finally we can live-migrate VM between cluster nodes by ONE.
2.3
GlusterFS
2.3.1 description
we use GlusterFS [1] to store our VM images.
2.3.2 before-install
install sshfs, build-essential, flex, bison, byacc, fuse on all node. make directory /export/sdb1 on all server nodes.
2.3.3 install
install glusterFS on all nodes. 2.3.4 configure
Execute glusterfs-volgen to generate glusterFS configure file. For example make 192.168.50.70 and 192.168.48.35 2 node as server.
glusterfs-volgen –name test2 192.168.50.70:/export/sdb1 192.168.48.35:/export/sdb1;
copy this configure file to all server node /etc/glusterfs/glusterfsd.vol. start glusterFS server on all server nodes.
/etc/init.d/glusterfsd start;
copy the configure file to client nodes and make directory for mounting. mounting glusterFS on specific directory without direct IO (direct IO is fast but xen can not boot sucessfuly if image is on direct IO mode filesystem).
glusterfs –disable-direct-io-mode -f /usr/src/glusterfs-2.0.9/test2-tcp.vol /mnt/glusterfs;
2.3.5 potential problem
GlusterFS dose not synchronize clients’ cache data. VM images may not sync when VM migration.
2.4
Hadoop Performance
We first consider how virtualization affects on hadoop performance, and then find opportunities of improve performance running hadoop under virtual ma-chines. We setup a hadoop cluster with three nodes each on heterogeneous machine, say native cluster. Then build virtual machine on each computer to run virtual cluster, which using the same settings with native cluster, and we can obtain their performance. The following table describes our hardware.
Processor Architecture
Intel Core 2 CPU E2180 @ 2.00GHz Intel Core 2 CPU E2180 @ 2.00GHz AMD Athlon 64 Processor 3500+ 2.20GHz Memory 4GB 4GB 4GB Disk 58GB 63GB 500GB Network Connection 1G Host Os Ubuntu 8.04 Guest OS Ubuntu 8.04 Xen version Xen 3.2 Memory to Guest 512MB
Disk (Guest) 2G No. of VCPUs to all OSs 2
We first run a RandomWrite generate 30maps, 10MB per maps and measure the job complete time. The virtual cluster has 71.52% performance compare with native cluster in average. Then we run Sort job to sort the random data generated before, which is 66.73% of native. The following are result data.
RandomWrite
sec Native VM
1 47 66
Sort sec Native VM 1 77 133 2 74 95 3 77 109 4 72 120 5 75 105 Avg 75.00 112.40 Performance 66.73%
2.5
MPI
2.5.1 Benchmark Intel MPI Benchmarks 3.2The IMB 3.2 package consists of 3 parts: • IMB-MPI1
• 2 MPI-2 functionality parts
IMB-EXT (One-sided Communications benchmarks), and IMB-IO (I/O benchmarks).
http://software.intel.com/en-us/articles/intel-mpi-benchmarks 2.5.2 Machine
Processor Architecture
Intel Core 2 CPU E2180 @ 2.00GHz Intel Core 2 CPU E6400 @ 2.13GHz AMD Athlon 64 Processor 3500+ 2.20GHz Memory 4GB 4GB 4GB Disk 63GB 20GB 500GB Network Connection 1G Host OS Ubuntu 8.04 Guest OS Ubuntu 8.04 Xen version Xen 3.2 Memory to Guest 512MB
Disk (Guest) 2G No. of VCPUs to all OSs 2 2.5.3 Performance
We set the number of MPI processes to run IMB be three and one virtual ma-chine per real computer and put the image files on the local, nfs, and the gluster file systems [1].
IMB-MPI1
test1 test2 average local 127m 43.4950s 135m 46.8430s 131m 45.1690s
nfs 162m 0.0770s 157m 18.9650s 159m 39.5210s glusterFS 159m 9.9650s 162m 19.1160s 160m 44.5405s
The time taken to complete “IMB-MPI1” for nfs and glusterFS is nearly same and both of them are greater than the time for local.
IMB-IO
test1 test2 average local 21m 22.9640s 30m 45.2870s 26m 4.1255s
nfs 75m 39.0970s 80m 14.8100s 77m 56.9535s glusterFS 28m 9.9880s 27m 8.4790s 27m 48.7335s
Unlike “IMB-MPI1”, the taken time of “IMB-IO” for glusterFS is close to the time for local, but the time for nfs is still longer than others.
[3] Mpi. http://www.mcs.anl.gov/research/projects/mpi. [4] Opennebula. http://www.opennebula.org/.
[5] Vmware esx. http://www.vmware.com.
[6] Xcp. http://www.xen.org/products/cloudxen.html. [7] Xen. http://www.xen.org/.
[8] Y. Chen, T. Wo, and J. Li. An efficient resource management system for on-line virtual cluster provision. Cloud Computing, IEEE International Conference on, 0:72–79, 2009.
[9] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In NSDI’05: Proceed-ings of the 2nd conference on Symposium on Networked Systems Design & Implementation, pages 273–286, Berkeley, CA, USA, 2005. USENIX Asso-ciation.
[10] M.R. Hines and K. Gopalan. Post-copy based live virtual machine mi-gration using adaptive pre-paging and dynamic self-ballooning. In VEE ’09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international con-ference on Virtual execution environments, pages 51–60, New York, NY, USA, 2009. ACM.
[11] M. A. Murphy, M. Fenn, and S. Goasguen. Virtual organization clusters. In PDP ’09: Proceedings of the 2009 17th Euromicro International Confer-ence on Parallel, Distributed and Network-based Processing, pages 401–408, Washington, DC, USA, 2009. IEEE Computer Society.
[12] M. A. Murphy, B. Kagey, M. Fenn, and S. Goasguen. Dynamic provisioning of virtual organization clusters. Cluster Computing and the Grid, IEEE International Symposium on, 0:364–371, 2009.
[13] B. Simmons, A. McCloskey, and H. Lutfiyya. Dynamic provisioning of re-sources in data centers. In ICAS ’07: Proceedings of the Third International Conference on Autonomic and Autonomous Systems, page 40, Washington, DC, USA, 2007. IEEE Computer Society.
[14] G. Somani and S. Chaudhary. Application performance isolation in virtu-alization. Cloud Computing, IEEE International Conference on, 0:41–48, 2009.
[15] T. Wood, G. Tarasuk-Levin, P. Shenoy, P. Desnoyers, and M. D. Corner E. Cecchet. Memory buddies: exploiting page sharing for smart colocation in virtualized data centers. SIGOPS Oper. Syst. Rev., 43(3):27–36, 2009.
