民國一 O 四年四月第
45
卷第
2
期
1============================================================ Designing and Constructing a Big Data Processing Platform using Hadoop Open Source Software
Hsiu-Ming Lin Cheng-Ju Yu Chih-Cheng Chien
Abstract
The Apache Hadoop had become the most popular open source software in the big data platform area.However its core services HDFS, the distributed file system and
MapReduce, the distributed computing framework, could not satisfy the requirements varies among different big data usages. It also becomes challenges to manage the servers, the network devices, and the system configurations in a distributed computing environment. In this paper we propose an integrated big data platform architecture comprised of the Hadoop and related open source software. This platform provides the file management service, the cluster resource management service, the distributed computing service, the data warehousing service, the data manipulation service, the workflow management service, along with the deployment interface, the end user interface, the system integration interface, and the system administration interface, to help enterprise ease adoption of the big data platform with Hadoop.
Keyword
Apache Hadoop、Apache Hive、Apache HBase、Apache Oozie、Apache Knox、Apache Ambari、Cloudera Hue
The Open-Source Memory Cache Technology Implementation and Application for Wi-Fi Value-added Service
Chin-Ping Chuang Jia-Wei Gong Shaw-Wei Ting Ya-Shian Wang Hong-Jie Chen Ya-Ping Huang Mei-Chun Chen Abstract
With the rapid growth of smart devices and Social Network Service (SNS) applications, the global mobile data traffic takes on a trend of explosive increase. These smart devices are equipped with Wi-Fi module, which at many locations is a more cost-effective, high-bandwidth access method for mobile broadband. Many mobile operators have built Wi-Fi networks proactively and reaped the rewards of reduced cost and lower churn due to a better user experience. The rapid spread of Wi-Fi networks are also derived the Wi-Fi Location-Based Service (LBS). Wi-Fi LBS value-added services can help mobile operator enhance the user experience or use user location information to optimize the experience and create revenues.This paper discusses the development of RINOS Wi-Fi network management systems by Chunghwa Telecom Laboratories which imported open-source memory cache technology to archive all kinds of new Wi-Fi value-added services requirements.
In order to accommodate the quick and immediate user position queries of Wi-Fi LBS,Wi-Fi network management system must poll the information of AP devicesmore frequently and rapidly to collect a large number of connected user data and provide the stable inquiry API. Finally, the experimental result of actual field trial shows that the RINOS Wi-Fi network management system can meet the huge and immediate inquiry demands of Wi-Fi LBS services and the Wi-Fi LBS services can correctly indicate user’scurrent location within a period of reasonable reaction time to meet the user requirement.
Keyword
Leveraging Open Source Software to Develop Cloud Resource Analysis Service Tzu-Chun Huang Yung-Sen Hsin Chuang-Fa Yang Kun-Hua Tsai Chih-Chien Chang Feng-Been Yang Abstract
Virtualization technologies offer enterprises many benefits likereducing cost, improving equipment efficiency, faster server provisioning and energy saving, etc.Using the
function of faster server provisioning, enterprise can efficiently create lots of virtual machines (VM) for the existing business systems or test requirements. However, creating lots of VMs could cause VM sprawl problem if virtualization resources on virtual machines are not allocated properly. With the expansion of VM sprawl problem, it will result in the performance degradation of the virtualization platform.
This paper illustrates the Cloud Resource Analysis Service (CRAS), which leverages open source distributed storage, O/R mapping, and several popular libraries to construct a high scalability, high reliability and high maintainability service architecture. With the patented calculating technologies inside, CRAS can analyze large amount of
virtualization performance data and then provide recommendations to reallocate resources for optimization purpose. CRAS helps enterprises that have adopted virtualization technologies to control VM Sprawl well.
Keyword
NoSQL： Not only SQL
JSON：JavaScript Object Notation ORM： Object/Relation Mapping OGM： Object/Grid Mapping
Verify a Multitenant Application Platform to Meet the Design Principle of Twelve-Factor App Using Container Technology
Wenda-Da Lain
Abstract
Booming cloud services let us rethink how to adjust the architecture in the traditional application, and how really make good benefits of cloud technologies to achieve the aspirations of the cloud services including high availabilities and scalabilities.
Especially, it is more important to deploy fast. Fortunately there have been many open source corresponding relational databases, cache, and network communication methods support decentralized architectures in cloud. For example, MongoDB, Redis, Mariadb, and Zookeeper can help us meet purposes of high availabilities and scalabilities, but the integrations of these basic software technologies have barriers. Moreover, to achieve the goal of rapid cloud services must deliver in the development, testing, maintenance and operation of on-line, and must integrate those operations seamlessly. Therefore, this paper will adopt container-based technology for applications, and follow the
Twelve-Factor App design principles to verify a cloud service development case as a basis for future development of cloud application platform.
Keyword
Build a Software Defined Storage Based on ZFS Min-Yi Tsai Chun-Yu Chang Kuo-Pao Lee Hsu-Fang Lai Chieh-Feng Li Yung-Yi Hsu Abstract
With the rapid development of cloud computing technology, the need of big data and virtualization also grows year by year. How to build an IaaS, Infrastructure as a Service become more and more important for a company IT.Traditional storage solution cannot longer fulfill the fast growing needs of storage. A far more advanced storage solution is needed.A Software Defined Storage (SDS) is more flexible than traditional storage, administrators can easily adjust the current storage resource based on current requirements and dispatch these resource rapidly. Nowadays, a SDS can utilize old storage hardware which can reduce the cost of purchasing new storage hardware. In addition, a SDS usually can provide features such as high-availability, remote-backup, snapshot, clone, compression, deduplication and RAID. As the result, we propose a method to build a SDS based on Illumos ZFS.The features such as snapshot, clone, compression, deduplication and RAID are already built in ZFS which can easily raise storage space utilization and data availability. Besides, we use ZFS built-in command to develop the remote-backup feature which allows the data can be periodically and automatically copied from local to remote. The high-availability feature is developed based on Pacemaker, an open source high-availability resource manager suitable for both small and large clusters. This SDS supports NFS/iSCSI/SAMBA protocols and built-in software RAID function make it become the storage repository of the hypervisor such as XenServer, VMware, Hyper-V in a virtualization environment to store virtual machine images. We also perform an experiment to compare the
performance of the SDS contain SSDs as write and read cache with the SDS without write and read cache.
Keyword
Design and Implementation of Virtual Machine Monitor Agent Tyng-Jiun Kuo Shu-Fan Shih Wenda-Da Lain Chih-Lung Liao Abstract
Cloud computing technology rapid grows recent years. Hypervisor such as VMWare, Microsoft, Citrix, Virtuoso all provide centralized resource management functions for virtual machines. However, there is no efficient way for VM resource monitoring from Service provider to user without API calls by user themselves in programs.
In this paper, we use the open source Sigar API and JNRPE Server to show the way to monitor resource of VMs cross hypervisor by monitor agents to fulfill a centralized multi-platform VM resource monitoring mechanism. Currently, this virtual machine monitor agent is on-line and supports the operations of CHT hicloud CaaS service.
關鍵字
A Virtual Machine Operation Prioritizing Method for Virtual Infrastructure Management Systems Ko-Jo Lee Yi-Xian Lin Shun-Hsing Hsu Yung-Yi Hsu Abstract
Cloud computing is a very popular technology in recent years. Gartner predicts that Infrastructure-as-a-Service(IaaS) will achieve a compound annual growth rate (CAGR) of 41.3% through 2016, the fastest growing area of public cloud computing the research firm tracks[1]. Therefore, virtual infrastructure management system (VIM), which manages infrastructure and provides virtualization services, plays an important role. Based on an open source software XenServer, CHT Virtuoso (hereinafter referred to as Virtuoso) which develops by CHT is used to manage, monitor and allocate virtual resources. In order for all virtual machines running on top of Virtuosoto work efficiently, this paper presents a non-interference prioritizing method for operations of virtual machines. In Virtuoso, all operations have different characteristics, some operations might be interfering and some others might be interfered with other operations. This scheme will prioritize all kind of operations of the virtual machines according to operation’s characteristics. All operations are assigned different processing orders and executed in sequence from high priority to low priority in order to avoid interference. As a result, the total execution time of the Virtuoso will be reduced significantly.
關鍵字
A Heuristic Resource Load-Balancing Algorithm for Cloud Computing Environments Yu-Wen Huang Jen-Hsien Cheng Po-Han Tseng Shun-Hsing Hsu Yung-Yi Hsu Abstract
Cloud computing is an emerging paradigm in computing research and industry in recent years. In cloud computing, computing resources are virtualized as services as demanded by the users. One of the challenging issues in cloud computing is to balance resource utilization across physical hosts to ensure resources available on demand. We design a cloud model and define the mathematical load-balancing problem which is an NP-hard problem. We propose a heuristic resource load-balancing algorithm to select appropriate VMs that are migrated from the overloaded hosts to others. Instead of rearranging all VMs, our algorithm migrate a few VMs can decrease the difference of resource
utilization among physical hosts. Three criteria are designed for the VM selection. The algorithm is implemented in CHT Virtuoso, a virtual infrastructure management system. Virtuoso inspects the resource usage of the cloud environment regularly and relocates VMs automatically when the loadings are imbalanced. Demonstration results show that the proposed algorithm can achieve load-balancing effectively.
關鍵字
雲端運算 虛擬機器 負載平衡
Abstracting Internal API with an Extension Module in SDN Controller Kai-Yi Cheng Cheng-Yang Chen Chih-Hui Hsiao Tung-Yu Tsai Chung-Hsiung Wu Yu-Ching Chen Wen-Chen Lu Hsiao-Chuan Tsou Chien-Hui Lee Chih-Hsiung Huang Abstract
This research aims to abstract internal APIs in a SDN controller. In SDN, controller is a key entity providing a logically centralized control of the network. When building an application, we may encounter multi-vendor issues due to the lack of standard
northbound APIs which can provide a unified interface among controllers. Though we can achieve an objective of proactive control model via adaptors. Few of applications were implemented with a pure proactive model. Applications need to be implement with the reactive control model to fulfill real-time requirements. However, this model is built by writing a module in a controller, when developing the module for every controller in a multi-vendor environment, we will face the issue of the difference of programming languages and internal APIs.
In this study, we proposed an architecture that adds an extension module to provide real-time service without introducing irrelevant components, and uses message broker to perform message delivery. This approach can simplify the functionalities of the controller, and avoid performance bottlenecks caused by heavy load of data processing. Therefore, it makes developing the applications with the reactive control model a feasible approach. Moreover, the extension module relies less on the controller, which can provide additional benefits by increasing the flexibility of the controller and reduce potential maintenance. 關鍵字 軟體定義網路 抽象化 訊息佇列 主動控制模式 即時控制模式
Design and Implementation of Cloud Rendering System with Open Source Software
Wei-Ting Huang Jia-Bin Yuan
Abstract
As the high-resolution 3D images and animations become more and more popular worldwide, they bring the visual effects production companies a lot of challenges of how to utilize the computing resources of servers in an on-demand, cost-effective way to finish the rendering process of these images and animations before the deadline. To meet the requirements mentioned above, some production companies try to build their own render farm, which consists of a number of computing and storage servers. However, this kind of infrastructure construction results in huge space consumption, energy wasting, and high CAPEX and OPEX. On the other hand, cloud computing technologies nowadays are much more mature than before, and they bring some important essential characteristics, such as resource pooling, on-demand self-service, and pay-as-you-go, into reality.In this paper, we first surveyedthe development status and trend of the visual effect industry by adopting cloud rendering technologies. And then the functionalities of theCloud Rendering System, which has been developed on the basis of open source software (OSS), were described to show how to create a high-availability system with the flexible yet stable architecture. Besides, several operation scenarios were illustrated to highlight the benefits of the system.
關鍵字
開源軟體(Open Source Software，OSS) 雲端算圖(Cloud Rendering)
算圖農場(Render Farm) 高可用度(High Availability)
