CHT Cloud Orchestration: an Integrated Cloud
System of Virtualization Platform
Chien-Ming Tu
1,2, Shih-Han Ku
1, Ju-Chi Tseng
1, Hsiang-Ting Kao
1, Fang-Sun Lu
1, Feipei Lai
21
Billing Information Laboratory Chunghwa Telecom Laboratories
Taipei 106, Taiwan
{james_tu, ksh, sylvie, shanti, fslu}@cht.com.tw
2
Department of Computer Science and Information Engineering
National Taiwan University Taipei 106, Taiwan
Abstract—Cloud computing changes the conventional way of
virtualization resource allocation, and gives users high flexibility and convenience. However, when the cloud service providers offer Infrastructure as a Service (IaaS), there are many practical problems to be resolved. We propose a cloud system's prototype, through a composition of cloud services to automate workflow and exchange data between services based on Message-Oriented Middleware, and then achieve the goal of virtualization platform integration. We develop efficient schemes for users to use functions, which include a single web portal, virtual machine templates and message translation, to quickly deploy virtual machines in a large-scale.
Keywords—Cloud Computing; Virtualization Platform; Virtual Machine; Hypervisor; Orchestration; Message-Oriented Middleware; Service-Oriented Architecture
I. INTRODUCTION
Chunghwa Telecom is the information and communication technology service market leader in Taiwan, as well as the biggest Internet service provider. In 2013, the subscribers are numbered to 4.24 million, and the market share is 68.9%. Its data center mainly provides leasing services such as dedicated hosting and colocation to large enterprises and government agencies. With the upcoming cloud computing era, public cloud deployment model brings significant use efficiency of resources and economies of scale [1], also compelling traditional hosting service providers to transform into cloud service providers to survive in the competitive market. To every Internet service provider, strengthening cloud computing strategy [2], virtualizing data center, and providing SMEs or start-up companies more value-added services and adding value through service model of infrastructure as a service [3], are unavoidable major challenges. Currently, virtualization solutions launched by various vendors are often applied to their own independent platform exclusively, which makes it very difficult to interconnect in cloud environment and integrate with differentiation.
Whether the data center is physical or virtualized, the most fundamental purpose is to deliver computing resources to meet the needs of front-end users. For virtualization environments,
since all the computing resources are simulated by hypervisor with software, in theory, virtual machines should be built quickly in accordance with user's request, and adjusting the resource settings or recycling resources as needed. However, in practice, the creation of virtual machines, configuration adjustments, and deletion are not as flexible and convenient as ideal.
To address the above issues, we propose a cloud system prototype which is called CHT Cloud Orchestration (CHTCO), which is meant for unified management, monitoring and allocation virtualization resource by integrating hypervisors. Our approach allows multiple heterogeneous systems, middleware, and services to collaborate harmoniously in the cloud, and users could deploy complex virtualization environments in a simple way.
In what follows, we first survey related work in Section II. We then present the design goals of CHTCO (Section III) and its architecture (Section IV). We describe our deployment and implementation in Section V. Conclude and discuss future work in Section VI.
II. RELATED WORKS
Cloud BOSS [4] proposed an efficient way to support the management of cloud services. It is mainly developed by adopting the TM Forum NGOSS framework. For example, it implemented the SID as defined in the XML Specification between modules for information exchange. Cloud Boss consists of four main block components: Web UI, BOSS NBI, BOSS Inventory Block, and Process Service. Process Service Block adopts JMS (Java Message Service) to publish and subscribe information and JMX (Java Management Extensions) to support internal modules to interact and to communicate between external BOSS NBI and the Web UI, so that core services can be accessed.
However, previous work did not specially address the need for MOM as the message communication mechanism and assembly cloud services in an orchestration way to reach automated workflow. Upon such a good foundation, we will
Copyright IEICE - Asia-Pacific Network Operation and Management Symposium (APNOMS) 2014
further design a rapid and efficient virtualized resources management approach with extension flexibility in this paper.
III. DESIGN GOALS
To construct a novel virtualization infrastructure service, one of the most important tasks is to combine cloud services into automated workflows with efficient message exchanging ability between services. Therefore, we adopt this principle in the design of the CHTCO architecture, which also represents that there are several requirements must be met during development:
First, CHTCO may be used by multiple role types of users, such as customer self-service and system administrator maintenance operation. Therefore, there must be some distinguish between different functions and information presented in the user interface.
Second, customers often have consistent requirements of virtual machine specification, with only a few differences between specification parameters. The usability of system will be enhanced if the virtual machine can be constructed in a similar manner of blueprint, users can easily select a template, fill in the set of parameter values, and then a virtual machine can be deployed. System administrators can also create several typical default templates for common application model in advance. When user raises demand, these templates can be applied to automatically complete the setting directly.
Third, virtualization platform commands of different vendors are usually confined to their own platform. It would be helpful to achieve higher system interoperability if we can fulfill the user's needs (e.g., create a virtual machine) with a kind of translation mechanism which is compatible with different commands of various virtualization platforms.
Fourth, even if data centers are fully based on virtualization platform, the management model remains being a small number of system administrators serving a majority of customer. Despite the resource utilization on virtualization platform is far more convenient than physical environment. However, due to the maintenance operation burden concentrates entirely on the management, a system administrator may have to serve dozens, even hundreds of clients at the same time, resulting that the bottleneck of service providing speed falls on system administrator. In addition, when deploying and maintaining virtual machines in the past, system administrators often spend a lot of time on the system and software configuration, and even have to help customers re-start when the system fails. Error occurs inevitably with such complicated manual intervention. In order to improve work accuracy, automated process will help improve this situation.
Finally, since the core concept of cloud computing is to publish resources and software in the form of service, it is often necessary to communicate via messages between different services to operate collaboratively. Thus, a high-performance, reliable, and secure communication infrastructure is very important. We expect a large-scale of message communication on CHTCO, so the message delivery and service processing must be very smooth and efficient.
IV. PROTOTYPE ARCHITECTURE
The CHTCO prototype utilizes five main components collaborating with each other to meet all design goals mentioned above. Fig. 1 shows the high-level architecture of our prototype.
Fig. 1. The CHTCO prototype architecture
A. Cloud Management Portal (CMP)
A single-entry web portal provides customers and system administrators with complete lifecycle management of virtual machine and integration of resources monitoring. Once confirmed by the authenticated and authorized security mechanisms, users can access management interface with the duty-related functions.
The functions of CMP are illustrated in Fig. 2. Window A shows optional computing resource specifications of a virtual machine, window B shows functions to upgrade or downgrade the specification of a virtual machine, window C shows the current status of workflow execution, and window D shows a dashboard monitoring the operational status of virtual machines in real-time.
Fig. 2. Display user operation views of different roles in multiple windows
B. Cloud Template Service (CTS)
CTS templates virtual machine specifications and allows users to fill up the desired operating system, software specification parameters of virtual machine. According to different application workload, CTS can provide templates including database server, web application server, and software license inventory management. Refer to Fig. 3, window A shows selectable CTS server specifications, window B shows Oracle database server templates, window C shows WebLogic application server templates, and window D shows the software license inventory management, including information such as statistics and warning lights.
Fig. 3. Display types of various template and software inventory management provided by Cloud Template Service in multiple windows
C. Cloud Conversion Service (CCS)
Through CCS, messages can be translated into applicable commands on corresponding virtualization platform based on different hypervisor, and be packaged into scripts for virtual machine deployment. Take HP Integrity Virtual Machines for example, the command and description are summarized in Table I [5].
TABLEI.
THE HPINTEGRITY VIRTUAL MACHINES COMMAND DESCRIPTION
Command Description
hpvmcreate Create virtual machines hpvmstatus Display status information hpvmstart Start virtual machines hpvmstop Stop virtual machines
hpvmmodify Modify existing virtual machines hpvmconsole Simulate a hardware console
hpvmmigrate Perform on-line or off-line guest migration hpvmsar Show performance information about the
running guests
hpvmsuspend Suspend virtual machines hpvmresume Resume virtual machines
D. Cloud Workflow Service (CWS)
We need a way to describe complex interactions between cloud services. Accordingly, we follow the web service composition approach applied in service-oriented architecture, use workflow to describe the relationship between web services in an orchestration way, and automate a series of cloud service process steps [6][7], please refer to Fig. 4. As can be seen in Fig. 5, when we want to create a BPEL Process, they will take Eclipse BPEL Visual Designer [8] and Apache ODE [9] respectively to design and implement.
CWS provides answers to the questions of:
Can messages be sent and/or received in any order? What rules govern sequencing of messages?
Is there any relation among any incoming and/or outgoing messages?
Is there a "start" and an "end" of a given sequence? Can a given sequence be partially "undone"?
Can a global view of the overall exchange of messages be drawn?
Fig. 4. Orchestration refers to an executable business process that can interact with cloud services
Fig. 5. Sample of BPEL process design
E. Message-Oriented Middleware (MOM)
MOM not only uses message queues to store message sent and received, but also adopts asynchronous message communication mechanism. In addition to be more efficient on message transmission, it also features hiding address, routing, data format complexity, solving various problems of communication between different machines (e.g., different operating systems, networks) through standardization of specifications, and simplifying cloud services with
communication needs. MOM plays the role as message broker in CHTCO. Each cloud service does not deliver messages itself, MOM deals with messaging instead. Cloud services can be more focused on their own business logic also due to they do not have to deal with the details of data transmission themselves [10][11].
If we do not have a message broker, we need to connect n × (n-1) / 2 times to communicate between message queues. Therefore, each endpoint queue must know the location of the other queue to send messages. When the number of queues grows, this model will become too complex to scale up. As Fig. 6 (a) shows, when there are five cloud services, the number of bidirectional connections is 5 × 4 / 2 = 10. If every cloud services need to send messages to other cloud services, the number of connections is 5 × 4 = 20. As Fig. 6 (b) shows, the message broker serves as a central exchange of messages (hub and spoke architecture, broker routes messages to the destination queue). A message broker provides additional decoupling between senders and receivers. The broker may perform additional functions such as filtering, message transformations and load balancing.
Fig. 6. Comparison of communication between cloud services with or without message broker
We adopt Advanced Message Queuing Protocol (AMQP) standard in MOM, the two main reasons are as follow: (1) lack of standardization: there is little standardization in MOM products (mostly proprietary solutions). For example, JMS is dependent on Java and does not specify a wire protocol but only an API. Therefore different JMS providers are not directly interoperable on the wire level. (2) Need for bridges for interoperability: to achieve interoperability between the different queuing systems, third party vendors offer bridges. These bridges complicate the architecture/topology, increase costs and reduce performance (additional delay).
Besides, one of the characteristics of AMQP is that it is an open protocol for business messaging, with support of industry (e.g., Cisco, Microsoft, Red Hat). At the same time, it is also multi-platform/language messaging system. AMQP defines messaging capabilities (called AMQP model) and Wire-level protocol for interoperability. AMQP has three kinds of messaging patterns in use: (1) request-response (messages delivered to a specific queue), (2) PubSub messages delivered to a set of receiver queues), and (3) Round-robin (distribution of messages to a set of receiver based on availability). Fig. 7 shows the main components of AMQP model [12].
Fig. 7. Main components of AMQP model
MOM transmits messages by at-least-once delivery and transaction-based delivery to ensure that users’ needs are met. Besides, MOM also supports HTTPS and SSL protocols. Important customer data, such as ID number and credit card number, are encrypted before transmission to ensure privacy and security. We choose Apache ActiveMQ [13] to implement MOM functions mentioned above with an intention to combine these cloud service components in a loosely coupled manner to construct a robust and flexible system.
Fig. 8 describes the usage scenario of CHTCO in a sequence diagram. Take the provision process of a virtual machine by example, the user can send VM creation request to CWS on CMP. When receiving this message, CWS will determine the flow of execution services, and notify CTS. CTS will then combine the information of virtual machine template and specification parameters user required, and forward to CCS to convert corresponding virtualization platform commands and package them into a script, then deliver the script to hypervisor to create a virtual machine and required software. Finally, the hypervisor will return the user with corresponding messages after finishing the work. Notably, all the message exchange between services is completed through MOM.
Fig. 8. Sequence diagram of how CHTCO creates a virtual machine
V. DEPLOY AND IMPLEMENTATION
We have implemented our prototype of CHTCO and deployed it in cluster server in data center, with firewalls to protect, load balancers to shunt requests, and high availability backup mechanism. The main integration hypervisor is HP Integrity Virtual Machines. With the help of virtual machine templates and automated process, a virtual machine can be created within an average of 3-5 minutes. We observed online virtual machines and found that the average concentration ratio of physical hosts and online virtual hosts. Database server is about 1:1.5, and application server is 1:2. Besides, since MOM transmits message asynchronously, new cloud services can be
easily added on CHTCO, and the existing cloud services are upgraded without interruption.
VI. CONCLUSION AND FUTURE WORKS
In this paper, we proposed a cloud system prototype which has successfully achieved automated deployment of cloud environment through the combination of cloud services and message transmission through MOM. Our prototype is still work in progress, and there is still a lot of room for improvement. For example, although MOM has high flexibility, it has to be responsible for all message transmission at the same time, and thus tends to be the bottleneck of the message flow. As a result, we should optimize the transmission efficiency of MOM to elevate the overall performance of CHTCO. We hope that CTS can support hypervisors from more vendors to achieve cross virtualization platform usage. We can also adopt Message Mover, which is a kind of service component, in CHTCO to integrate MOM or message queues from different cloud service providers in the future. Therefore, the message transmission will not be restricted to different cloud environments, and each cloud service can be accessed to each other seamlessly in a unified method.
REFERENCES
[1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "Above the Clouds: A berkeley view of cloud computing," EECS Dept, University of California, Berkeley, Tech. Rep. UCB/EECS-2009-28, Feb 2009.
[2] T. M. Takai. "Cloud Computing Strategy," Chief Information Officer, Dept. of Defense, USA, Tech. Rep., July 2012.
[3] P. Mell, T. Grance, "The NIST definition of cloud computing," National Institute of Standards and Technology, Tech. Rep., October 2009. [4] Huan-Guo Lin, Chung-Hua Hu, Hey-Chyi Young, Kuan-Hsiung Liang,
Yung-Yi Hsu, Chia-Chen Chu, Chi-Sheng Wu, Chien-Wei Cheng, and Yao-Te Huang, "Cloud BOSS: Cloud-centric BSS/OSS for enterprise cloud service operations," Network Operations and Management Symposium(APNOMS 2011), Taipei, Taiwan, Sept. 2011.
[5] HP Integrity Virtual Machines 4.3: Installation, Configuration, and
Administration. [Online]. Available:
http://h20565.www2.hp.com/portal/site/hpsc/template.BINARYPORTL ET/public/kb/docDisplay/resource.process/?spf_p.tpst=kbDocDisplay_ ws_BI&spf_p.rid_kbDocDisplay=docDisplayResURL&javax.portlet.be gCacheTok=com.vignette.cachetoken&spf_p.rst_kbDocDisplay=wsrp- resourceState%3DdocId%253Demr_na-c02752979-1%257CdocLocale%253D&javax.portlet.endCacheTok=com.vignette.c achetoken
[6] C. Peltz, "Web service orchestration and choreography," IEEE Computer, vol. 36, no. 10, pp. 46-52, Oct. 2003.
[7] Orchestration, Choreography, Collaboration and Java Technology-based Business Integration. [Online]. Available: https://weblogs.java.net/blog/2003/10/30/orchestration-choreography-collaboration-and-java-technology-based-business-integrat
[8] BPEL Designer Project. [Online]. Available: http://www.eclipse.org/bpel
[9] Apache ODE. [Online]. Available: http://ode.apache.org
[10] Q. H. Mahmoud,(Ed) "Middleware for Communications", West Sussex, Wiley & Sons, 2004.
[11] S. Goel, H. Sharda, and D. Taniar, "Message-oriented-middleware in a distributed environment," in Third International Workshop on Innovative Internet Community Systems, June 2003, pp. 93–103.
[12] MOM - Message Oriented Middleware. [Online]. Available:
http://zh.scribd.com/doc/70272844/MOM-Message-Oriented-Middleware
