Pragmatic Integration of Cloud and Grid Computing Infrastructures
Thomas Rings, Jens Grabowski
Institute of Computer Science, University of G¨ottingen G¨ottingen, Germany
{rings,grabowski}@cs.uni-goettingen.de
Abstract—The integration of cloud and grid infrastructures is still of current interest, because it provides a way for the scientific area to ensure sustainability of well engineered grid applications. The integration of well established grid infras-tructures with cloud systems also fosters their complementary usage, simplified migration of applications, as well as efficient resource utilization. In this paper, we compare the layered conceptual grid model to the service model of clouds. Based on this comparison, we describe pragmatic possibilities to integrate cloud and grid systems. We analyze the connectivity options on the infrastructure level to gain access to both infrastructures using a unified client. In two case studies, we show the successful integration of the Amazon Web Services cloud with UNICORE 6 and the open source cloud Eucalyptus with Globus Toolkit 4. Based on these implementations, we discuss lessons learned.
Keywords-interoperability, integration, grid, cloud
I. INTRODUCTION
Many companies leverage the cloud computing paradigm to offer and sell their idle running computing and storage resources on demand. These cloud resources are provisioned in a rapid way and can be released with minimal manage-ment effort [1]. The main goal of cloud computing is the provision of a user friendly access to an infrastructure that scales in a transparent and resilient way to support business operations.
In the scientific field, grid computing systems pursue goals similar to the ones of cloud computing systems. A grid computing environment provides access to a unified computing and storage resource infrastructure composed of resources that are administrated and owned by different scientific organizations to foster research.
The integration of both infrastructures provides a way for the scientific area to ensure sustainability of well engineered grid computing applications as well as to secure previous investments in grid developments. An integration with re-sources of grid and cloud systems means to increase avail-able computing and storage power on demand so that their resources are used more efficiently. The added resources can be utilized for migration purposes as well as for business critical applications. After such an integration, it would be possible to instantiate well established grid infrastructures in the cloud on demand.
In this paper, we present a conceptual layered model for grid computing systems and compare it to the service model of cloud computing systems. We analyze possibil-ities for integrating both infrastructures to use them in a complementary and transparent way. In addition, we present connectivity options of combining grid computing systems on the cloud infrastructure level. For the integration of grid and cloud systems, we describe two case studies: the integra-tion of the grid middlewareUniform Interface to Computing Resources(UNICORE) [2] based on theHigh Level Appli-cation Programming Interface (HiLA) [3] with theAmazon Web Services (AWS) [4] cloud and the integration of the grid middlewareGlobus Toolkit 4 (GT4) [5] with the open source cloud Eucalyptus [6]. In the first case study, we choose HiLA, because HiLA can be utilized to access other grid infrastructures that support this abstract Application Programming Interface(API). We choose AWS, because we think that AWS is the major player of Infrastructure as a Service(IaaS) clouds and many other cloud systems adapted its interfaces. Therefore, the AWS interface can be seen as a de-facto standard. The second case study was deployed to identify the architectural differences compared to the first case study in order to analyze portability between different providers. The case studies focus on the integration of grid systems in cloud systems.
This paper is structured as follows. In Section II, we describe the conceptual model of grid systems as well as the service model of cloud systems. Based on a comparison of these models, we analyze the integration of grid and cloud infrastructures in Section III. Afterwards, in Section IV, we present the integration of clouds and grid computing systems on the infrastructure level, which we implement in two case studies that are described in Section V. In Section VI, we consider related work and in Section VII, we shortly describe the lessons learned. We conclude this paper with a summary and an outlook in Section VIII.
II. GRID ANDCLOUDCOMPUTING
In this section, we describe the different models of grid and cloud computing. Grid systems allow efficient and dy-namic sharing and management of resources between any in-terested parties of different organizations. It relies heavily on a grid middleware, which provides secure access to diverse 2012 IEEE Fifth International Conference on Cloud Computing
GridPortal GridApplication
GridCoreServices
GridScheduler Resource Management Information Management Data Management Execution Management Security Local
Resources Compute Storage Sensor Service
Figure 1. Layered conceptual model of grid computing
resources that are managed in a decentralized manner. A grid system provides nontrivial qualities of service through standardized, general purpose protocols and interfaces [7].
Referring to the grid architecture defined in [8] and from our practical experiences [9], we developed the conceptual model of grid computing as depicted in Figure 1. On the bottom layer, the model includes local resources. In the sense of grid computing, local resources are entities that fulfill job requests [10] and are usually deployed within a private network. Grids integrate different types of resources including computing, storage, sensor, and services. These resources usually deploy a predefined software stack. For example, compute clusters, which belong to the computing resource type, are tightly interconnected but operationally independent computers, on which user accessible software runs to manage and control concurrent computing tasks that instantiate a common application program [11]. Similarly, the other resource types deploy an already pre-configured infrastructure that is usable within private networks by utilization of their specific protocols and interfaces. The protocols are then utilized by the grid core services that are offered by a grid middleware to access these local resources from a public network and of other organizations. The grid core services include services for information, data, execution, and resource management.
The grid core services are utilized by grid schedulers that are able to schedule jobs over several grid infrastructures. In addition, these services can be directly used via grid portals or grid applications. Grid middleware systems deploy se-curity services that provide authentication and authorization functionalities for the entire grid core services.
Cloud systems are classified in a layered service model containing the following layers from bottom to top as depicted in Figure 2: Infrastructure as a Service (IaaS),
Platform as a Service (PaaS), and Software as a Service
(SaaS) [1]. Within the illustrated clouds on each level, the figure depicts the interfaces to the services that are provisioned by each layer respectively. The IaaS includes virtualized resources, e.g., storage, processors, and networks. Within these virtualized resources, network architects are able to deploy and run arbitrary software via resource
n
SaaS
Applic
at
io
WebͲbasedApplication
GeneralApplications BusinessServices
MobileApps EndUser D l d tfo rm PaaS RuntimeEnvironment ControlInterface Developer Developedon Pla t Resource Management Fault Tolerance Dynamic Provisioning Load Balancing Deployedon ast ru ct u re IaaS SystemMonitoring Interface ResourceManageͲ ment Interface Architect In fr a Virtualization Network
Computing Storage I/O
Figure 2. Service model of cloud computing
management interfaces. They check the status of the IaaS via the system monitoring interface. Within the platform level, services for automated resource management, fault tolerance, dynamic provisioning, and load balancing are deployed. These functionalities are utilized within the run-time environment in a transparent manner via a control interface, e.g., an API that is usually used by an application developer. The developer has only control over the deployed applications, but not over the resources. The top layer, SaaS, provides web interfaces for end users to access applications without requiring local software installations. The users are only able to apply application specific configuration, but cannot control the cloud infrastructure. Each layer deploys security mechanism to protect the resources and services they offer. For further details of each layer, we refer to [1].
III. INTEGRATION OFCLOUD ANDGRID
INFRASTRUCTURES
Grid and cloud computing systems both offer access to distributed and pooled computing resources and services. Efficient resource use, resources on demand, replication, and migration purposes are the main reasons for a combined usage of grid and cloud computing systems. Therefore, we propose the integration of grid and cloud infrastructures on the interface level based on the layered models that we presented in the previous section. Figure 3 shows the direct comparison of these two models. Both are based on physical hardware. Within our grid model, the local resources are di-rectly deployed on physical hardware. In contrast within the cloud model, IaaS offers a resource management interface to deploy operating systems and applications on the underlying hardware within virtual machines and virtual infrastructure. A similar deployment step is already done in our grid model since the local resources, which are on the bottom layer, deploy already a pre-configured software stack. Therefore,
ServiceModelofClouds
G id G id
ConceptualModelofGrids
GridScheduler Grid Portal Grid Application SaaS
GridCoreServices L l R PaaS LocalResources IaaS Physical Hardware Physical Hardware
Figure 3. Comparison of the conceptual models of grid and cloud systems
a layer with similar functionalities as provided by IaaS does not exist in our grid model.
The grid core services as well as low level computing and storage management services of the local resources, reside in the PaaS level, because they provide interfaces to use and manage resources transparently similar to the functionality exposed by the control interfaces of PaaS. A PaaS cloud varies highly in its abstracted functionalities and usually hides the complexity of scheduling inquiries from the user or developer. It scales with the number of users using an application developed and deployed on this platform. Therefore, a grid scheduler is also a functionality of the PaaS. While in grids, the interfaces are exposed directly to the developer, in clouds they are offered transparently via control interfaces. In grid systems, we have delegated control over resources. In the PaaS cloud model, the user might only be able to configure the application-hosting environment [1]. On top of the grid model are grid portals and applications, which are on the same level as SaaS clouds. They provide transparent access to resources via interfaces that are usually implemented using web protocols.
Many grid and cloud providers exist on the market offer-ing their own custom-made solutions and interfaces. How-ever, customers require a high quality of service, which can be reached by simultaneous utilization of systems offered by different providers via a unified access. These different systems can be used, e.g., for data replication and to decrease costs by choosing the best suited solution. Interoperability and interoperation of these systems is needed on a technical level. For further details about technical interoperability, we refer to [12]. Interoperation takes places on the same level of abstraction, i.e., implementations need to adapt to the same communication protocol. In the cloud domain, two ways of achieving interoperability exist: adapters as well as stan-dards. Adapters are implemented within Cloud APIs, e.g., Deltacloud [13] or Libcloud [14], that define connectors for each supported cloud infrastructure. Emerging cloud stan-dards includeOpen Cloud Computing Interface(OCCI) [15] andCloud Data Management Interface(CDMI) [16]. These allow the extension of grid systems with several cloud
d
d GridCoreServices LocalResources GridCoreServices
LocalResources Deployed onIaas (Becomes PaaS)
Figure 4. Grid-cloud integration on the infrastructure level
systems using one cloud client implementation. In the grid domain, the adapter approach is implemented, e.g., in the HiLA for grid applications as well as in the Java Grid Application Toolkit(JavaGAT) [17] that allow to access grid core services of different grid middleware implementations. In addition, the Open Grid Service Architecture (OGSA) standard family is widely adapted. However, a grid-cloud interoperability standard does not exist.
Through analysis of grid and cloud interfaces, we make a step towards the identification of the requirements of a uni-fied grid-cloud standard. We determined three designs for the interoperation of cloud and grid systems. In the first design, we integrate the stack of grid core services and local resource management services on virtual machines of the IaaS cloud as shown in Figure 4. This allows an indirect communication between the grid and the cloud system based on the protocols implemented in the grid core services. An already installed and configured grid system can then be extended with cloud resources without any changes. This is especially interesting in application domains, where computing power is needed spontaneously and in unpredictable time intervals. It is also a step towards the migration of grid applications into the cloud environments. Trough the deployment of the grid core services and the local resources management software stack, the IaaS cloud becomes a PaaS cloud that offers computing services as well as an API provided by the grid middleware. Figure 5 shows the second design, where we access the grid core services from the level of the PaaS cloud. Usually, PaaS clouds scale with the number of users but not with computing intensive tasks. These tasks are submitted into the grid for further processing. Therefore, new instances in the cloud do not need to be started for High-performance computing (HPC), since the virtualization of the resources in the cloud mitigates the computational performance and, therefore, lessens the efficiency of the computational re-source utilization. To this aim, we deploy libraries to access the grid core services in the PaaS cloud so that these libraries can be utilized within the PaaS cloud.
In the third design, the SaaS cloud uses the grid core services via a grid API to provide grid functionalities in a transparent way to the cloud user. Similarly, a grid portal utilizes the services provided by a PaaS cloud. Consequently, the issue of determining the difference between a grid portal and a SaaS arises. In general, there is not a clear
L
PaaS GridCore Services Grid Libr ar y
Figure 5. Grid-cloud integration on the platform level
difference, since both provide user interfaces that access applications located on the lower levels. This access to the lower interfaces of the other system needs to be adapted in both grid portals and SaaS clouds.
IV. INTEGRATINGIAASANDGRIDSYSTEMS
In this section, we describe a methodology for the integra-tion of grid and cloud systems on the infrastructure level. We present evolutionary steps towards a feasible solution of integrating IaaS clouds and grid computing systems. We assume that the different cloud and grid systems implement standards to allow an extension with systems implementing the same standards.
A. Access to Grid and Cloud Resources
IaaS clouds provide virtual infrastructures for the de-ployment of virtual machine images to start and instantiate virtual machines. If several instances of a virtual machine are started, software for their balanced utilization as well as for executing and managing parallel tasks needs to be installed and configured so that the instances can be accessed via abstracted interfaces. In order to reuse the services provided by a grid system, we deploy grid core services into the IaaS cloud. The grid infrastructure can then be utilized to execute already existing grid applications within the cloud infrastructure and offer new services based on grid technology within the cloud.
An IaaS cloud that utilizes grid protocols can also be used to extend already existing grid infrastructures on-demand in peak times. For utilizing the IaaS cloud as well as the grid infrastructure, applications and clients communicate via the same grid core service protocols. The management of the cloud environment includes the instantiation of the virtual machine images, on which the grid core services are readily pre-configured. After these cloud resources are instantiated, the grid core services are automatically started and registered with the information service of the already existing grid. Afterwards, these services can be used by the grid client. In addition, cluster and storage resources can be deployed in the cloud system as local resource management systems and connected to the grid core services of the cloud.
To increase the usability of managing the grid resources deployed in the cloud, we integrate the management of cloud computing resources into the client layer of the grid environment. Figure 6 shows the extension of the grid client with the cloud client using the API of the cloud environment as well as the deployment of the grid core services into
the cloud. The user uses only one client to setup the cloud environment and to send computational tasks to the grid core services deployed in a cloud environment. In the remainder of this paper, we call the grid core services deployed in a cloud environment shortly grid-in-cloud-services.
B. Connectivity Options
A cloud system is deployed in one of the following models: private, community, public, and hybrid. For their definitions, we refer to [1].
For the initial connectivity option, we deploy the grid-in-cloud-services in a public cloud. The public cloud is configured and managed through the cloud extension of the grid client via a custom-made cloud interface. After deployment, the grid-in-cloud-services are available directly within a public network and can be accessed via a grid gate-way, which is an intermediate access point to multiple grid systems and their local resources. The gateway is required, because the grid core services of both grid deployments do not interact directly with each other. A management entity schedules tasks between the two grid deployments from an upper layer, e.g., a grid scheduler. In our case, the grid client offers scheduling functionalities to utilize both grid infrastructures in parallel. The grid-in-cloud-services register with the information service, i.e., with the registry inside the already existing grid via the grid gateway. Then, the grid client polls this registry to utilize both infrastructures through the grid gateway.
If the public domain of the cloud is accessible via the Internet, the grid-in-cloud-services are exposed to threats. For the protection of the grid-in-cloud-services, security methodologies on instance level are required. Each cloud instance needs to be treated with a specific security con-figuration. This includes the application of firewalls and security software for each instance. This design has a high complexity and overhead for security configurations and is, therefore, prone to error.
We overcome these security issues through the deploy-ment of the grid-in-cloud-services within a private cloud that is itself located in a public cloud as shown in Figure 6. The grid-in-cloud-services are only accessible via the cloud gateway, which offers means to access the resources located in a virtualized dedicated network of our private cloud. All communications between the grid-in-cloud-services and the grid core services of our existing grid take place through the cloud gateway. The resources of the private cloud are not visible publicly, because they are logically separated from the public cloud. Each resource is protected in general by the security concepts deployed by a private cloud. Only the cloud gateway needs to be properly configured to fulfill security requirements. Hence, this solution is more viable than the previous one.
Only the connection between the cloud gateway and the grid gateway is left to be unsecure. However, encryption
Grid Client Cloud Extension Grid Client Cloud Extension
Cloud Interface PublicIaaS Cloud
PrivateIaaS Cloud Grid Core
Services Grid
Grid Gateway Cloud Gateway Local Resources Local Resources Core Services Local Resources Local Resources
Figure 6. Grid core services deployment in a private IaaS cloud
mechanisms can be applied on transferred data using an encryption service of a grid middleware. Another possibility is the extension of the local network of the already existing grid with the private network of the cloud via an encrypted
Virtual Private Network (VPN) tunnel. This means that the cloud gateway is not required anymore. The grid-in-cloud-services appear as local resources inside the already existing grid environment. This solution provides a complete separation from the public networks and, therefore, a high level of security.
An emerging topic is the federated cloud [18], where clouds of different providers are integrated based on trust utilizing theAuthentication and Authorization Infrastructure
(AAI). This topic is still in its infancy, but we think, our approaches help to integrate multiple clouds as well as grids. Grid-cloud standards would increase the feasibility of the federated model.
V. CASESTUDIES
In this section, we describe two case studies that show that grid and IaaS cloud resources can be integrated to solve specific tasks in a complementary way. The case studies are based on [19] and [20].
A. Integration of UNICORE and AWS infrastructures
We integrated the grid middleware UNICORE 6 and the IaaS cloud infrastructure AWS. UNICORE 6 is a common grid middleware implementation that provides the grid core services. We decided to use these implementations because they are commonly and widely used. In contrast, OCCI is still in its infancy and covers only basics of cloud interfaces. To our knowledge, only one implementation for this standard exists. AWS interfaces are widely adapted by other cloud systems and can, therefore, be seen as a de-facto standard. UNICORE implements standards, e.g., OGSA-BES and HPC-BP, as well.
Figure 7 shows the integration based on the solution depicted in Figure 6. We use the HiLA shell client as a grid client, which we extended by utilizing the AWS API. Both
HiLA Shell AWSExtension
P bli AWS Cl d AWSInterface UNICOREGateway
PublicAWSCloud
UNICORE UNICORE GatewayAWS UNICORE
Private Central Registry Site Gateway OGSAͲ* Site OGSAͲ* AWS Cloud XUUDB XNJS XUUDB XNJS XUUDB TORQUE XUUDB TORQUE
Figure 7. Deployment of the UNICORE grid middleware in the IaaS AWS cloud
the already existing grid infrastructure and the private cloud deploy UNICORE, which can be accessed via OGSA-* interfaces that access the UNICORE’s internal execution management engine(XNJS). TheUNICORE User Database
(XUUDB) service performs authentication and authorization in the respective grid environment. In addition, the already existing grid provides a central registry, where all UNICORE services are registered. In both environments, we deployed the open source TORQUE Resource Manager [21].
To access UNICORE resources, three different kinds of clients can be utilized [22]: the UNICORE command line client; the UNICORE rich client, a graphical user interface based on the Eclipse Rich Client Platform [23]; and the open source HiLA shell that allows development of grid clients using Java. The HiLA shell can be extended by the cloud extension with little efforts. It provides a uniform way to access grid core services of different grid middlewares. This leverages the goal to support more than one grid infrastructure as well as more than one cloud infrastructure using the same cloud-extended grid client. Therefore, we utilize the HiLA shell to extend our grid client.
The AWS cloud extension needs to fulfill two function-alities: the management of the AWS environment as well as the control of UNICORE resources in the cloud in an automated manner. These functionalities are implemented as AWS extension for the HiLA shell in a package of Java classes. The classes for the automatic AWS cloud management contain methods to configure, start, and stop the AWS environment utilizing the AWS API.
The classes also allow the instantiation of more than one UNICORE server in the AWS environment. Figure 8 shows the configuration of our solution. We establish a private subnet for each UNICORE server. We use a 24 bit subnet mask to be able to configure 20 private subnets, which is the maximum number of allowed AWS subnets. In each subnet, a maximum of 254 instances can be started.
UNICOREGateway Public AWS PrivateAWS Cloud PrivateSubnets Cloud PrivateSubnet 10 0 1 0/24 (RouteTableA) e.g.,10.0.1.0/24 UNICORE Gateway AWS Internet Gateway s e g 10 0 2 0/24PrivateSubnet ) e.g.,10.0.2.0/24 UNICORE Site TORQUE PrivateSubnet e.g.,10.0.3.0/24 UNICORE Site TORQUE Site
Figure 8. Deployment of the UNICORE grid middleware in the IaaS AWS cloud in multiple private subnets
The instances can be used for the deployment of either TORQUE resources or further UNICORE servers. The UNI-CORE servers communicate with the UNIUNI-CORE gateway of the already existing grid infrastructure through the internal UNICORE gateway, which uses the AWS Internet gateway. Public Internet Protocol (IP) addresses are not needed for the UNICORE servers within the private subnets. Only the UNICORE gateway within the private cloud needs a public IP address to register directly with the external UNICORE gateway. To access the UNICORE servers deployed in a private subnet, a Network Address Translation (NAT) in-stance with a public IP address can be started. Afterwards, all instances within a private subnet can be accessed as well as communicate with services outside the private cloud using the NAT service.
By separating UNICORE servers into different subnets, we can serve organizations independent from each other. The separation provides load balancing, isolates behavior of different applications, and gives flexibility in arranging UNICORE and TORQUE resources.
The classes for the automated deployment of UNICORE resources on AWS instances contain methods to configure and to start the UNICORE components within the AWS cloud environment. This includes the configuration of appro-priate users on each instance, as well as the configuration of the UNICORE gateway and the automatic registration of the started UNICORE components in the registry of the already existing grid system. In addition, we configured the UNICORETarget System Interface(TSI) [24] for using the TORQUE resource management system. We performed the deployment of the TORQUE software in a manual manner. We validated the AWS-UNICORE infrastructure using a simple test framework. We checked if the configuration of the cloud environment and the deployment of the UNICORE resources in the AWS cloud were performed as expected by the cloud extension of the grid client. In addition,
ManagementandSchedulingClient(ShellScript) Globus Toolkit 4 Command Line Client euca2ools Globus Toolkit4CommandLineClient euca2ools
Globus Physical
Machine EucalyptusInterface Physical
Machine Globus
Toolkit4 PhysicalMachines
Cloud Controller Physical Machines Node Controllers TORQUE VirtualMachines VirtualMachine TORQUE Globus Toolkit4
PrivateEucalyptusCloud s
Figure 9. Deployment of GT4 grid middleware on Eucalyptus cloud resources
we submitted computational tasks into the UNICORE grid deployed in the cloud successfully. This shows that it is feasible to integrate AWS and UNICORE resources.
B. Integration of GT4 and Eucalyptus Infrastructures
To identify similarities of integrating cloud and grid systems, we applied a second case study, where we deployed GT4 and TORQUE cluster resources in a cloud configured with the open source cloud software Eucalyptus. For this setting, Figure 9 shows the instantiation of the design depicted in Figure 6. We built the cloud infrastructure from scratch. Therefore, Figure 9 shows the physical machines that are not directly visible in the cloud. The Eucalyptus cloud controller is the entry point to the Eucalyptus cloud and controls the node controllers from a physical machine. The physical machines, which are available for the cloud, are configured as node controllers. A node controller manages, i.e., starts, stops, and inspects one or more virtual machines on the physical machine, on which the node controller is installed. The virtual machine image of the grid front end is configured with the GT4 middleware and a TORQUE server. Both are pre-configured with default values.
We developed a command line client based on shell scripts for the management of the cloud infrastructure and task management for the grid layer. This client utilizes the command line client of GT4 for job submission and the command line tool euca2ools [25] to configure and start the Eucalyptus cloud. It also includes a scheduling module. The process to be performed and the deployment of the GT4 resources on Eucalyptus cloud resources are similar to the ones of UNICORE on AWS resources. We validated the GT4-Eucalyptus infrastructure and client by submitting parallel grid tasks to resources of the already existing GT4 grid system and on GT4 resources deployed in the Eucalyptus cloud successfully.
VI. RELATED WORK
We divide the related work of our contributions in parisons as well as integrations of cloud and grid com-puting infrastructures. Within the first area, Foster et al. compare both infrastructures in a detailed analysis from different perspectives including architecture, security, and programming model [26]. Our contribution extends their architecture comparison by describing the direct relations of the architecture layers. Sadashiv and Kumar compare clusters, grids, and clouds by determining if the respective infrastructure has specific functionalities [27]. However, they do not describe the reasons of the determination. In contrast, we describe the comparison of grids and cloud based on their technical architecture. Similarly, the comparison of Zhang et al. [28] does not provide a direct comparison as in our contribution.
Within the second area, Yamini et. al. present a method for scheduling jobs in an IaaS cloud extended grid envi-ronment [29]. Their approach is similar to our contribution. However, we determine connectivity options for IaaS grid integration as well as develop a model for PaaS cloud-grid integration. Jha et al. describe how clouds can be used as semantical abstraction of grids [30]. It is not clear on which level of the cloud service model they operate. We give a technical description and implementation of the integration of IaaS cloud and grids. Ostermann et al. present an approach to extend the ASKALON grid with instances from three different IaaS clouds [31]. In contrast, we devel-oped a generic model for the cloud and grid integration on different service levels, and implement two case studies for the infrastructure level integration for other grid system im-plementations than ASKALON. Nimbus Infrastructure [32] builds a compute grid only based on virtual machines. It mainly utilizes AWS and is deployed within aWeb Services Resource Framework (WSRF) container. This approach is different to ours, because they create a computational cloud as a service inside the grid computing infrastructure. We deploy the grid core services within a cloud.
VII. EVALUATIONS ANDLESSONSLEARNED
On the infrastructure level, the management is only pos-sible from the upper layer, because the services cannot com-municate directly. To allow this, interoperability adapters need to be implemented. A challenge in the development of the cloud extension was that the cloud interfaces nearly changed monthly [33]. This makes it hard to deploy an automated integration solution for the cloud system.
The integration of grid and IaaS requires expert knowl-edge of both technologies. The highest effort was the au-tomation of the IaaS cloud setup as well as the automatic deployment of the grid core services. The configuration of virtual networks and instances was not an easy task in case of applications requiring to fulfill specific security require-ments, e.g., the setup of a VPN. The implementation of
adapters for interfaces of different standards is challenging, because it is needed to normalize them on the same abstract level. The identification of the requirements of interfaces that fulfill the same properties is not a simple task.
Accounts for the different systems are utilized, because the user accounts are not the same in both systems. An application of AAI, e.g., OpenID [34] or Shibboleth [35] technology could provide a better feasibility. However, AAI needs to be implemented on the infrastructure level. In our example, we mapped users to the respective infrastructure.
The integration can be used for transitions from grid to cloud systems. Grids can be fully based on cloud systems that only instantiate resources on demand. The optimal solution would be a standard for grid cloud integration. We provide a step towards securing sustainability of well engineered grid applications within cloud systems as well as previous investments.
VIII. SUMMARY AND OUTLOOK
In this paper, we compared the layered grid model to the service model of cloud computing infrastructures. Based on this comparison, we developed designs for their integration. We presented connectivity options for their integration on the infrastructure level. We implemented one of these options within two case studies, where a grid client is extended with a cloud management module that utilizes the API provided by the cloud infrastructure. We developed an automated setup for cloud and grid environments so that both infras-tructures can be used in a transparent way based on the grid core service protocols. We integrated UNICORE 6 with the AWS cloud and GT4 with the Eucalyptus cloud successfully. As future work, we consider the development of the cloud extension using a cloud API, e.g., Deltacloud [13] or Libcloud [14] as well as for OCCI [15] and CDMI [16]. However, for the management of virtual machine images and virtual instances, common interfaces are required. In addition, we plan to develop a case study for the integration model depicted in Figure 5. This is not an easy task, since the cloud interfaces on the platform level are restricted in case external communication is performed.
ACKNOWLEDGMENT
We acknowledge Amazon Web Services for the provision of a research grant to use their cloud infrastructure.
REFERENCES
[1] P. Mell and T. Grance, “The NIST Definition of Cloud Computing,” Special Publication 800-145, National Institute of Standards and Technology (NIST), 2011, [Online; http: //csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf fetched on 2012-05-06].
[2] J¨ulich Supercomputing Centre, “UNICORE,” [Online; http: //www.unicore.eu/ fetched on 2012-05-06].
[3] UNICORE Team, “UNICORE HiLA 2.1 Documentation,” 2010, [Online; http://www.unicore.eu/community/ development/hila-reference.pdf fetched on 2012-05-06].
[4] Amazon Web Services LLC, “Amazon Web Services,” [On-line; http://aws.amazon.com fetched on 2012-05-06]. [5] I. Foster, “Globus Toolkit Version 4: Software for
Service-Oriented Systems,” inProceedings of the IFIP International Conference on Network and Parallel Computing (NPC05), ser. LNCS, vol. 3779. Springer, 2005.
[6] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, “The Eucalyptus Open-Source Cloud-Computing System,” in Proceedings of 9th IEEE International Symposium on Cluster Computing and the Grid. IEEE Computer Society, 2009, pp. 124–131. [7] I. Foster, “What is the Grid? A Three Point Checklist,”Grid
Today, vol. 1, no. 6, p. 22, 2002.
[8] ——, “The Grid: A New Infrastructure for 21st Century Science,”Physics Today, vol. 55, no. 2, pp. 42–47, 2002. [9] T. Rings, G. Caryer, J. Gallop, J. Grabowski, T. Kovacikova,
S. Schulz, and I. Stokes-Rees, “Grid and Cloud Computing: Opportunities for Integration with the Next Generation Net-work,” Journal of Grid Computing: Special Issue on Grid Interoperability, JOGC, vol. 7, no. 3, pp. 375 – 393, 2009. [10] K. Krauter, R. Buyya, and M. Maheswaran, “A taxonomy and
survey of grid resource management systems for distributed computing.”Softw., Pract. Exper., vol. 32, no. 2, pp. 135–164, 2002.
[11] T. Sterling,Beowulf Cluster Computing with Windows. The MIT Press, 2001.
[12] T. Rings, J. Grabowski, and S. Schulz, “A Testing Framework for Assessing Grid and Cloud Infrastructure Interoperability,” International Journal On Advances in Systems and Measure-ments (SysMea11v4n12), vol. 3, no. 1&2, pp. 95–108, 2011. [13] Apache, “Deltacloud,” [Online; http://deltacloud.apache.org/
fetched on 2012-05-06].
[14] ——, “Libcloud,” [Online; http://libcloud.apache.org/ fetched on 2012-05-06].
[15] Open Grid Forum, “Open Cloud Computing Interface,” [On-line; http://occi-wg.org fetched on 2012-05-06].
[16] Storage Networking Industry Association, “Cloud Data Management Interface,” [Online; http://www.snia.org/cdmi fetched on 2012-05-06].
[17] Ibis Project, “JavaGAT,” [Online; http://www.cs.vu.nl/ibis/ javagat.html fetched on 2012-05-06].
[18] A. Toosi, R. Calheiros, R. Thulasiram, and R. Buyya, “Re-source Provisioning Policies to Increase IaaS Providers Profit in a Federated Cloud Environment,” in Proceedings of the International Conference on High Performance Computing and Communications. IEEE, 2011.
[19] M. Doleys, “Using Cloud Computing Resources in Grid Systems: An Integration of Amazon Web Services into UNI-CORE 6,” Bachelors thesis. Institute of Computer Science, University of G¨ottingen, Germany, ZAI-BSC-2011-17, 2011.
[20] D. Dahman, “Extension of a Globus Toolkit 4 Grid System by a Virtual Runtime Environment based on Eucalyptus,” Masters thesis. Institute of Computer Science, University of G¨ottingen, Germany, ZFI-MSC-2010-XX, 2011.
[21] Adaptive Computing, “TORQUE Resource Manager,” [On-line; http://www.adaptivecomputing.com/products/torque.php fetched on 2012-05-06].
[22] J¨ulich Supercomputing Centre, “UNICORE Client Layer,” [Online; http://www.unicore.eu/unicore/architecture/ client-layer.php fetched on 2012-05-06].
[23] Eclipse Foundation, “Eclipse Rich Client Platform,” [On-line; http://www.eclipse.org/home/categories/rcp.php fetched on 2012-05-06].
[24] UNICORE Team, “UNICORE TSI: MANUAL,” 2011, [On-line; http://unicore.eu/documentation/manuals/unicore6/files/ tsi/tsi-manual.pdf fetched on 2012-05-06].
[25] Eucalyptus, “Euca2ools User Guide,” [Online; http://open. eucalyptus.com/wiki/Euca2oolsGuide fetched on 2012-05-06].
[26] I. Foster, Y. Zhao, I. Raicu, and S. Lu, “Cloud Computing and Grid Computing 360-Degree Compared,” in Proceedings of the Grid Computing Environments Workshop. IEEE, 2008. [27] N. Sadashiv and D. Kumar, “Cluster, Grid and Cloud
Com-puting: A Detailed Comparison,” in Proceedings of the 6th International Conference on Computer Science & Education (ICCSE). IEEE, 2011.
[28] S. Zhang, X. Chen, S. Zhang, and X. Huo, “The Comparison Between Cloud Computing and Grid Computing,” in Proceed-ings of the International Conference on Computer Application and System Modeling (ICCASM). IEEE, 2010.
[29] L. Yamini, G. LathaSelvi, and S. Mukherjee, “Efficient metascheduling in a cloud extended grid environment,” in Proceedings of the International Conference on Recent Trends in Information Technology (ICRTIT). IEEE, 2011. [30] S. Jha, A. Merzky, and G. Fox, “Using clouds to provide
grids with higher levels of abstraction and explicit support for usage modes,”Wiley Concurrency and Computation: Practice & Experience, vol. 21, no. 8, pp. 1087–1108, 2009. [31] S. Ostermann, R. Prodan, and T. Fahringer, “Extending Grids
with Cloud Resource Management for Scientific Computing,” inProceedings of the 10th International Conference on Grid Computing. IEEE/ACM, 2009.
[32] University of Chicago, “Nimbus,” [Online; http://www. nimbusproject.org/ fetched on 2012-05-06].
[33] Amazon Web Services, “Amazon EC2 API Reference - Doc-ument History,” [Online; http://docs.amazonwebservices.com/ AWSEC2/latest/APIReference/ fetched on 2012-05-06]. [34] OpenID Foundation, “OpenID,” [Online; http://openid.net/
fetched on 2012-05-06].
[35] JISC Advance, “Shibboleth,” [Online; http://www.shibboleth. net/ fetched on 2012-05-06].
