getting an application up and running on the Amazon cloud.
5.1.3
Experience of using Amazon web services
The details of setting up an Amazon account with a credit card, establishing security and authorization with public and private key pairs and so on is not discussed here as it has changed somewhat from the early days of AWS in 2008. The experience gained by this project in using Amazon Web Services and public datasets stored in Amazon Elastic Block Store was good, as long as running instances were closed down when experiments and analyses had finished. Otherwise the cost of using the services would continue to be charged without warning. In a similar way, the use of Elastic Block Storage for the project’s programs, parameter data and results had to be well managed to minimize storage costs.
5.2
A Local Private Cloud
A collaboration with Ibrahim Musa came about when Dr Musa was looking for some real world biological data to use to test his private cloud which was the basis for his PhD study. He needed to find some data and analysis programs which would provide a range of challenges for his proposed self-service infrastructure container. This project was of interest because it was not merely a private cloud on which to test the R scripts which analyse microarrays, but it was also researching a new way of offering cloud services to scientists. It was agreed to share some of the microarray data and some R scripts to analyze the data. Dr Musa ran the analysis jobs, getting the same results, and using the differing amounts of data for each job to test out some job allocation algorithms within the private cloud structure.
5.2. A Local Private Cloud 88
CMS
Utilization Workload Optimization vCell Delivery Layer
manager Report Provisioner vSwitches XEN Kernel Schedular Allocation Module
Virtualization and Control Service Layer (1) (2) (3) (5) Failure Virtual Storage Server Middleware VM VM vSwitches VM VM VM VM (4)
Container1 Container2 Containe3
Statistics
Physical Fabric Layer
High Capacity Optical Switch Fabric Packet Switch network IT resource IT resource v C e ll O w ne r (e .g S ci e n tis t) V IP P IP Service Console
VIP: vCell Virtual Infrastructure Provider PIP: vCell Physical Infrastructure Provider
VM: Virtual Machine B ro k e r Mediation (6)
CMS: Core Management Services
Figure 5.3: Framework and interaction of components in the proposed virtual infrastructure container http://www.journalofcloudcomputing.com/content/3/1/5[Accessed September 29th, 2014]
5.2.1
Description of the private cloud
Cloud Computing encompasses the service model where extremely large network and IT resources are virtualized and offered over the internet to increasing numbers of users. The users themselves do not have to understand the complex layers of machines and software which comprise the cloud, but are able to use the services at the application or machine level, depending on their needs. Dr Musa’s PhD research was to create a virtual organization of resources in a holistic manner and with a single logical view which he termed a virtual service cell (vCell). Users would request one or more organised virtual machines rather than specifying detailed isolated resources. The network to connect the virtual machines, the
5.2. A Local Private Cloud 89 storage to support task execution requirements and the logic to manage all the components with minimum human intervention, would be provided with the machines. In this manner, user resources and management logic would be contained within the logical composition which simplifies the task of management and ensures high user service automation.
Figure 5.3 shows the implementation of Musa’s virtual service cell. One can see the Physical Fabric Layer at the bottom, with the computers, network and switch components and storage servers, and the vCell Delivery Layer at the top which is where the scientist will interact with the cloud. The Core Management Services (CMS) are shown at the right hand side, interacting with the Service Layer and the Virtualization and Control layer in the middle. The actual computers that Musa used were ten four-year-old PCs with switching equipment and three storage servers. The details of the implementation of the private cloud can be found in Appendix D.
5.2.2
Significance of Musa’s private cloud
Dr Musa’s research was able to conceptualize and test an optimal framework for building a virtual container as a service (vCAAS). This model contributed a framework and the optimization techniques for next-generation cloud computing. The isolation of components within a vCell ensured that network and IT components operated with a vCell boundary which isolated external traffic and enhanced performance. This allowed service management to be attached to each vCell and to components within each vCell. The vCells within a vCAAS were each equipped with a privileged VM acting as the vCell manager (CM). The CM controlled communication between vCells and communication with external com- ponents outside the container (vCAAS). The per vCell management in vCAAS reduced complexities and simplified the technical and operational tasks of managing cloud infras- tructure. In the vCAAS, a subset of network management functions was projected into a privileged virtual machine so that management tasks, such as utilization monitoring, usage accounting, failure recovery, adaptive control and scheduling tasks were managed on a per
5.2. A Local Private Cloud 90 vCell basis rather than on the whole datacentre infrastructure.
vCells were able to be replicated to simplify the initiation of new vCells for a new vCAAS container. In future development of this work it is envisaged that an automated vCell that encapsulates various service technologies would be used. This would enable a cloud administrator to provision a new tenant and configure the required services across the network in minutes. The vCell approach enables the efficient and automated scaling of hundreds of tenants and thousands of virtual machines with just a few steps and less complexity. The vCAAS approach ensures efficient delivery of applications, allowing any combination of network and IT resources to support the needs of the users’ applications.
5.2.3
Experiments on the private cloud
The full description of the private cloud, its hardware, the infrastructure and middleware employed, the experimental setup details and full results and discussion are all given in the published paper: “Self-service Infrastructure Container for Data Intensive Application” [61]. A description of the experiments and figures of the results are given in Appendix D. Here the experiments and the key results are summarized.
The microarray data consisted of GSE experiments with differing numbers of CEL files and therefore differing amounts of data. Each experiment comprised one analysis job. The cloud was set up so that the vCAAS container was able to run a varying number of vCells, and in each vCell the analysis for one GSE experiment was run at a time. Three common job queue scheduling algorithms were used:-
• the Shortest Job First algorithm on K Queues (SJF-KQ) • the Largest Job First algorithm on K Queues (LJF-KQ)
• the First Come First Served algorithm on K Queues (FCFS-KQ)
Initially the jobs were allocated randomly to the queues for each type of queueing algorithm. Then the quantity of data to be accessed for each job was assessed and used in
5.3. Windows Azure Cloud 91