}w!"#$%&'()+,-./012345<ya

(1)

MASARYKOVA UNIVERZITA FAKULTA INFORMATIKY

}w !"#$%&'()+,-./012345<yA|

Cloud backend deployment at

Faculty of informatics MU

MASTER THESIS

Ondrej Famˇera

(2)

Declaration

I hereby declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete refer-ence to the due source.

(3)

I would like to thank Yenya, for technical support and feedback, Hanna Kim for support and language corrections and my colleagues for feedback.

(4)

Abstract

This work presents approaches in building cloud storage, network and manage-ment subsystems, compares performance of individual implemanage-mentations and discusses best choices in this field. It also discusses implementation of cloud backend in infrastructure of Faculty of Informatics at Masaryk’s university.

(5)

(6)

Chapter 1 Introduction

The boom of virtualization in IT has never seen such a rapid growth as when it was hidden after the word ”cloud”. This concept made a revolution in this area as stuff around virtualization get highly propagated to non-IT people claim-ing and promisclaim-ing consolidation, money savclaim-ing, ease of use and many other over-hyped conclusions.

This work does not aim at introducing concepts of ”cloud” from user perspec-tive but rather shows the other side of whole cloud ecosystem — the machinery behind it. Despite our determination to bring most recent information, this area is silently and rapidly evolving so we’ll write about most recent state with con-sideration of possible future development.

First, we’ll provide insight into cloud storage concepts. Beginning with def-inition of most suitable kind of storage for cloud. Also, we’ll introduce ”Stor-age layer-bloat” problem which impacts performance expectations planning. Further, we’ll describe distributed file systems and their suitability for storing virtual machines data. Along with these concepts, we’ll also debate about re-dundancy, reliability and robustness of distributed storage systems. As example for our requirements on cloud storage we more deeply introduce two projects: GlusterFS and CEPH. In the end we shorty describe other possible file systems with their main shortcomings that made them out of our list of options for ef-fective deployment.

In third chapter we’ll take a closer look at anatomy of network traffic in cloud environment. We will not only focus on high-level architecture of how virtual machines in cloud communicate, but also on management traffic around cloud. Recommendations for traffic separation, security and performance are also de-scribed here with the aim at distributed file systems presented in previous chap-ter.

(9)

Last theoretical chapter will introduce cloud management systems with their main problem similar to one from storage — layer-bloat. After general discus-sion about cloud management, we’ll introduce two candidates for implementa-tion at Faculty of Informatics of Masaryk’s university (FI MU). First of them is a large project called OpenStack, which is aiming at becoming cloud operating system. The second one is OpenNebula project, which is less ambitious than OpenStack but more focuses on the particular area of cloud management.

Part of this work is also performance testing of selected subsystems that are crucial for cloud deployment. First, we’ll present here testing equipment and testing methods description. We will take look at storage subsystem perfor-mance and compare distributed file systems that had interesting features for deployment at FI MU. Comparisons that we made here are supported by graphs with most important results to make right overview. Later, we’ll discuss about suitability of tested distributed file systems for our deployment.

One whole chapter we’ll dedicate to discussion about cloud management sys-tems. We’ll show differences between two selected in earlier chapters. Then we decide on suitability for our deployment and describe most important advan-tages of them.

The last chapter deals with deployment of systems discussed in our work to improve virtualization environment at FI MU. Here we gather our knowledge from beginning and transform it into practical deployment and implementation suggestions.

(10)

1. INTRODUCTION

1.1 Cloud

Before we dive deeper into cloud technologies, we need to introduce what we are actually dealing with.

Cloud technology provides services at all levels of IT infrastructure, from basic vir-tualization services to operating system services to application level services.[1]

The goal of Cloud Computing is to allow users to take benefit from all of these technologies, without the need for deep knowledge about or expertise with each one of them.[17]

(11)

Most recent accepted standardized definition of Cloud Computing is one by the National Institute of Standards and Technology (NIST).[21] Cloud can be classified in many different ways, depending on which aspect we are looking at. In figure 1.1 we show for example division by cloud provided services. This view is most important for end users. It consists of:

Software as a Service (SaaS) Platform as a Service (Paas) Infrastructure as a Service (Iaas)

From deployments point of view, we can divide cloud into following cate-gories:

Public Cloud — run and provisioned by 3rd party cloud providers, enabling organizations to outsource computer infrastructure outside to decrease to-tal running cost of IT

Private Cloud — contained inside control domain of cloud owner providing better control on whats happening inside along with ability to completely customize it to specific requirements

Hybrid Cloud — mix of public and private cloud bringing the best of both worlds: low cost services and highly customizable own environment This work is aiming at giving more insights to cloud owners into building up private cloud providing IaaS and PaaS services. Our main goal is to find optimal solution for FI MU as a cloud provider.

To build up cloud from the bottom, we should take a deeper look at:

Cloud storage — providing storage for whole cloud infrastructure

Cloud networking — enabling communication management not only inside the cloud but also managing connection to rest of the world

Cloud management — giving administrators and users ability to keep every-thing in cloud under control and provide them with high level interface to work with this environment

(12)

Chapter 2 Cloud storage

Cloud storage is made up of many distributed resources, but still acts as one. It’s highly fault tolerant through redundancy and distribution of data and highly durable through the creation of versioned copies.[22]

Important aspects of Cloud storage are

scalability — it should be easy to scale-out without loosing the data availability

flexibility — it should be possible to use commodity hardware as well as high-performance one

redundancy — data should be protected against errors

recovery — data should be easy to recover or storage should deploy self-healing mechanism

performance — we should take into consideration planned usage of storage system ( distributed computations versus service hosting )

As for Cloud storage solutions, they follow two implementation paths:

hardware path — Big powerful storage systems providing all-in-one solution for storage, however they are expensive and they outdates quickly with new trends.

software path — Approach that builds upon deployment of distributed stor-age created by software to lower cost of specialized hardware. Usually, the commodity hardware which has lower price is used than specialized all-in-one solutions.

Our work aim at providing better insight into distributed storage implemented in software as we believe it provides better performance/price ratio.

(13)

2.1 Distributed storage

Distributed storage (DS) is closely analogous to a distributed object storage. Distributed object storage (DOS) stores objects (unified data structures) among data nodes and provides access to them. These data nodes forms groups of nodes called clusters. Objects on data nodes are building blocks for more com-plicated higher-level structures such as distributed file systems or virtual block devices. Before we continue, we’ll provide you some basic terminology from DS cluster environment:

data nodes — nodes that stores the data (objects)

monitor nodes — nodes that monitors operation of other nodes in cluster

distributed file system (DFS) — file system running on the top of DOS

metadata — extra data stored by DFS

metadata nodes — nodes that takes care about metadata operations in DFS

object replication — way of storing the same object on multiple data nodes providing redundancy of object in case of data node failure

self-healing — ability to react on event of data node failure, for example by allocating more objects to keep number of replicas on desired level

replica — redundant data, exact copy of another object used to increase redun-dancy and so provide fault tolerance

In this work we’ll focus on DS suitable to provide storage for virtual machines (VMs) in virtualization environment. As one of typical use cases of storage for VMs is a non-volatile storage that will hold users data from VM. This can be a disk, disk partition, file, loop device etc. The goal is to provide performance close to physical hardware (bare metal).

2.1.1 Storage layer-bloat

In order to provide high performance storage to VM we face problem that we cannot optimize the storage performance according to underlying hardware. Reason is that there are too many levels (layers) with different characteristics (disk block size, file system block size, DFS block size, etc. ). It’s very hard to determine how we should work with such storage to get the optimal perfor-mance.

(14)

2. CLOUD STORAGE Lets take for demonstration a virtual disk (1) stored in files (2) on distributed file system (3) which is created of objects (4) lying on other conventional file system (5) placed on RAID1 _{(6) on different machines (7). Previous sentence}

with numbers in brackets indicate layer number in which preceding component is. We need to remember that this kind of abstraction makes it not feasible to easily figure out optimal values for things like disk block size to use.

2.1.2 Distributed object storage as VM storage

There are two possible approaches how to allow DOS to become storage for VMs. One of them is file system approach where we don’t consider VM storage to be something special. We treat it as regular file stored on DFS. Problem of this approach is unnecessary overhead of DFS when working with the file such as metadata look-up and update. Second approach is to provide block device created on top of DOS. Here we avoid using DFS and we access DOS directly. Virtual block device maps directly to objects resulting in higher performance as we avoid use of DFS metadata as there is no DFS.

2.1.3 Metadata

Similar to regular local file systems such as ext4, DFS also needs to have me-tadata. This is mainly due to the fact that DFS are actually file systems and they generally require metadata to properly function.

We know about three general approaches to metadata realization in DFS:

centralized metadata systems — Metadata of whole DFS are stored on one cen-tralized system. This system soon becomes performance bottleneck and single point of failure (SPOF).

distributed metadata systems — Metadata are distributed among several sys-tems and can be access from any of them. This requires that all syssys-tems maintain consistency in distributed metadata in real time which poorly scales with increasing number of systems.

algorithmically located metadata – Similar to distributed metadata systems, metadata are on different systems. However metadata now share com-mon function according to which it is possible to compute where to look for them without querying metadata systems. So clients can access meta-data directly as if they were regular meta-data stored in DFS about which they know their location.

(15)

2.2 GlusterFS

GlusterFS is an open source DFS capable of scaling to several petabytes and handling thousands of clients. It is based on a stackable purely user space design.[2]

Some of interesting advantages includes:

Scalability and Performance — algorithmically located data and metadata

Elastic Hash Algorithm — function that allows client to compute where (meta)data are without need to query GlusterFS for this information

High Availability — internal replication and self-healing mechanisms

As basic building block GlusterFS uses a brick. Brick is storage file system that is part of Gluster storage volume (Gluster volume types are discussed later in section 2.2.1). Brick is a regular directory on data node, which holds the DOS objects and is build on top of existing local file system. It’s advised for brick to be on separated partition to prevent interference with other systems or user data. Essential requirement for local file system is support for user extended attributes.

For locating data and metadata, GlusterFS uses only Elastic Hash Algorithm. This allows the clients to calculate themselves where data resides based just on knowledge of structure of GlusterFS volume. In case of replicated volume or undergoing re-balancing of data it might happen that data are not in calculated place. To cover situations when data are not in position yet, GlusterFS uses ref-erences which temporary refers to current location of data.

Configuration of GlusterFS is created by interacting with command line in-terface (CLI) utilities. This is due to complexity of configuration files which gets generated through this CLI in background. From user’s point of view, this is somehow better approach as user can interactively configure GlusterFS, which is simpler than manually editing configuration files.

2.2.1 Distributed, replicated and striped volumes

As mentioned earlier, GlusterFS organizes storage networks into bricks which then forms storage volumes. These volumes can have some special properties which improves their parameter in way of performance, fault-tolerance and dis-tribution of data among nodes. Basic volume types are:

(16)

2. CLOUD STORAGE

distributed volume — files are spread across cluster and are not accumulating on single node

replicated volume — files get replicated to another bricks, number of replicas can be adjusted. By default number of replicas is ’one’ which means no replication.

striped volume — split files to stripes and store them on different bricks, this improves performance in high concurrency environment

These basic types can be further combined together to create more specific attributes. Such an example can be the distributed, striped and replicated vol-ume. This combination brings high performance of stripped volume distributed evenly in cluster to spread the workload, and replicated volume that provides fault tolerance in case of failure of some nodes. All basic types and this example are showed in figure 2.1.

(17)

2.3 CEPH

CEPH is a unified, distributed storage system designed for ex-cellent performance, reliability and scalability.[3] It’s built on top of Reliable Autonomic Distributed Object Store (RADOS) which provides foundation for all other services provided by CEPH.

Essential building blocks of CEPH cluster are[4]:

Object Storage Daemons (OSDs) — store data as objects on storage nodes

Monitors — maintain copies of cluster structure, monitors data nodes and pro-vide information for clients about cluster

OSDs are closely analogous to bricks in GlusterFS. Apparent difference is only in their internal structure. They also recommend separated file system and requires user extended attributes. For locating data in cluster CEPH uses Con-trolled, Scalable, Decentralized Placement of Replicated Data (CRUSH) algorithm.

(18)

2. CLOUD STORAGE CEPH cluster builds it’s services using librados library which is also available as general purpose library for accessing RADOS. RADOS organizes internally it’s structure into storage pools (similar to volumes in GlusterFS) which can have different attributes such as number of replicas or different CRUSH map for data placement. Services that build on RADOS are (also shown in figure 2.2):

RBD — RADOS Block Device (RBD) (discussed later in section 2.3.1)

RADOSGW/RGW — RADOS REST2_{full Gateway}

CEPH FS — POSIX-compliant3_{DFS similar to GlusterFS}

Configuration of CEPH cluster is done through configuration files, which are well documented and provide wide variety of configuration options. All config-uration options are stored in single file. Along with configconfig-uration, CEPH have it’s own authentication protocol similar to Kerberos in order to maintain secu-rity in communication between nodes in cluster.

2.3.1 RADOS Block Device (RBD)

RBD service provides access to virtual block devices built on top of RADOS for use mainly in VMs. These devices can be access using Linux kernel module utilizing kernel objects (KO) or by QEMU hypervisor accessing them directly via librbd library. By default RBD stripes a block device image over multiple objects to increase performance.[12]

2. REST - Representational State Transfer

(19)

2.4 DRBD

For performance comparison use, we also mention Distributed Replicated Block Device (DRBD) as distributed file system with very limited distribution capabilities. It’s suitable for 2-node clusters where it works best. Despite it is able to run in 3 node cluster, its configuration is very complicated and brings very limited use in scenario where we want to use tenths of nodes. As the name suggests its main use is to provide network replicated block device. In figure 2.3 is shown how DRBD works in 2-node cluster scenario. Both nodes can see DRBD block device but only one can actively use it. In case of failure of one node, DRBD relies on external application or manual intervention to change access point for DRBD block device.

Figure 2.3: DRBD operation example[16]

2.5 Other distributed file systems

Following DFSs have some potential in cloud storage environment. But we consider them not to be ready for production environment because of many reasons, which we’ll discuss later. However their concepts are interesting for being included into this work.

(20)

2. CLOUD STORAGE

2.5.1 Moose FS

This DFS is similar in architecture to GlusterFS, but it implements some in-teresting unique features. Most inin-teresting one of them is per-file replication, which allows to define number of replicas in granularity of one file (GlusterFS replication level granularity is only on volume level).[5] Another feature such as file system wide ”trash bin” which allows to delete files not instantly but after selected period of time are rather eye-candy.

List of disadvantages for this DFS starts with existence of metadata server which creates place for performance bottleneck and also it introduces SPOF.[14]. As workaround to problem of SPOF it is possible to use external high-availability solutions such as pacemaker4_{, but it’s not as stable and reliable as mechanisms}

that are parts of DFSs themselves.

2.5.2 SheepDog

Another promising project that addresses exactly the problem of distributed storage for the purpose of virtualization. It supports block devices, replica-tion, snapshots, thin provisioning and other useful features. Current it’s lim-ited for user with QEMU/KVM hypervisor and high-availability features are supported by underlying cluster management software (corosync, zookeeper, accord). [6]. The biggest problem of this DFS is immaturity of whole project and lack of activity in it that would make it widely available in distributions in recent versions. Road-map of project promises a lot but reality is somewhere else.

(21)

Cloud networking

Network traffic in cloud can be categorized into two main types:

VM traffic — generated by running VMs in cloud

management/overhead traffic — needed by various components of cloud to work properly together

VM traffic can be considered as pure as the traffic in the environment where we use bare-metal servers. This traffic is what we can see arriving/leaving the cloud or traveling between VMs in cloud. This represents real user data in net-work. On other side there is so called management (or more precisely overhead) traffic. In one single machine environment this would be equivalent to com-munication between components of that machine. In cluster of computers this includes

– DFS communication between data nodes, – monitoring communication,

– cloud management communication,

– network overhead because of encapsulations and – other communication not related to VM traffic.

3.1 VM traffic

On this level, we may consider VM traffic to be regular traffic, which can be seen real world networks. But the difference is that it would be most probably running over virtualized environment that is hidden from our viewing perspec-tive. Virtualization enables us to virtualize whole networks in ways that most of devices in network topology can be virtual. For end users this concept should be not visible or disruptive. The way how this is achieved is discussed in sec-tion 3.2.2 about management traffic. Away from how the data are transferred over network, we deal with numerous ways how to interconnect with other networks. These concepts are taken from virtualization scenarios and they pro-vide following abilities for communication:

(22)

3. CLOUD NETWORKING

Isolated networking — VMs, which are connected to isolated network, can only communicate between each other.

NAT-ed or routed networking — Same as isolated networking but there is a point in network (computer, router, ... ) which has connectivity to ”outer network”. This single point which defines how to reach ”outer network” can be limited (Network address translation (NAT)) or restricting (static routing).

Direct networking — VMs are connected to some existing network without major limitations and acts as part of that network. Typical example of this is network bridge which interconnects real and virtual network on virtu-alization server.

3.2 Management networking

Non-negligible amount of traffic we observer in physical network of cloud is used for management, monitoring and subsystem communication. This cat-egory covers many different types of traffic which have different requirements on network parameters such as throughput, latency or jitter.

3.2.1 DFS traffic

The largest amount of this traffic comes from DFS subsystem as it’s respon-sible for providing DS over network. This kind of traffic is very sensitive to network latency and jitter which greatly influences its performance. According to speeds of today’s hard drives, DFS have no problem to utilize 1 Gbit network interface. Because 10 Gbit network cards are still not widely used nor cheap, we should try to optimize performance on 1 Gbit network.

First possible optimization is dedicating separated network exclusively to DFS communication. This improves processing at switch level where most of traffic are large packets. Second recommended optimization is increase of Mes-sage transmission unit (MTU) to reduce overhead of packet header. For exam-ple, increasing MTU size from 1500 bytes (standard Ethernet) to 9000 bytes (jumbo frame) requires 6-times less headers and CPU interrupts to send and receive such a packet. This can be very beneficial for DFS traffic as most of it is expected to be large in size.

(23)

3.2.2 Cloud management traffic

Depending on selected cloud management solution, it might be needed to protect communication between cloud management and rest of cloud infras-tructure by separating this traffic. Despite this applies mainly to applications unable to secured communication themselves, it’s also a good practice to do it for hardening management environment. Another reason can be the fact that some of traffic might be time critical and in case it is disrupted, it can paralyze management of cloud or even cause Denial of Service.

3.3 Inter VM network separation techniques

In inter VM communication we face the problem of making communication link between two or more processes (VMs) that can be on one or more virtual-ization servers. Problem is how to separate this traffic easily and dynamically. We are focusing on situation when VMs are on different virtualization servers. Both of VMs that wants to communicate have Virtual NICs (VNICs) that are vis-ible as VNIC on virtualization hosts. One of possvis-ible solutions is to setup static routing based on knowledge of network between VM and route it between ser-vers. Such an approach leads to no separation at all, requires manual setup and is not flexible. Because of that, we describe here better solutions.

3.3.1 Linux bridge

In case we have more of VNICs that we need to put into same network, we can use Linux bridge which groups them together under one bridge. This allows us to easily create distinguishable groups on single virtualization host. But very soon we’ll discover that this approach is also not flexible as it limits us to only interfaces that we have available on virtualization host. It doesn’t provide any separation on network out of virtualization host (except the case we have the server that has as many physical NICs as VMs). To overcome this limitation we need something that would help us extend the influence of separation we have on local machine to the rest of the network.

3.3.2 VLANs

Answer to problem of network separation can be Virtual Local Area Net-works (VLANs) defined in RFC 802.1Q[18]. Thanks to VLANs, we can have more than just one layer 2 network on one single physical NIC. VLANs are amend to 802.3 standard that allows to put small tag on network packet with

(24)

3. CLOUD NETWORKING identification number with value of 1-4094. By using this tag, the network de-vices can identify the network to which the packet belongs and treat it accord-ingly. The catch here is that network devices such as switches and routers must understand this tag and route it to right destination where it belongs to. Setting up VLANs can be done either manually or automatically using GARP/GVRP or MRVP protocols. However this can be problematic for security reasons as administrators might not wish to allow network devices to be reconfigured dy-namically as this can cause harm to the rest of network in case this goes out of control. Another problem is that number of VLANs is limited to 4094 which in large deployments can be easily depleted.

3.3.3 Tunneled protocols and tunnels

Another way to create separated layer 2 networks on single physical NIC is the use of tunneling protocols creating tunnels working on top of existing layer 3 network. This can be very useful as it allows not only to solve problem with VLAN limits but also allows the networks to be interconnected beyond the borders of local layer 2 network. This feature comes with price of overhead as these protocols needs to package data inside the regular packet which means they transfer less data compared to ones that are just tagged as VLANs.

Despite there are quite many encapsulation protocols we picked following that can server purpose of interconnecting VMs:

NVGRE — Network Virtualization using Generic Routing Encapsulation

STT — Stateless Transport Tunneling Protocol for Network Virtualization

VXLAN — A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks

3.3.4 Open vSwitch

This very interesting open source project tries to take best from Linux bridge and combines it with ability to dynamically create VLANs and various encap-sulation tunnels. It’s designed to be configured and monitored using standard-ized protocols such as OpenFlow, sFlow or NetFlow. [7]. We may imagine open vSwitch as very intelligent Linux bridge capable of collaboration with Software defined network (SDN).

(25)

Cloud management

Cloud is very abstract definition and so it could be a problem to define what the Cloud management should be responsible for. We focus on narrowing our definition to IaaS and PaaS in private cloud but we’ll still refer to it as to Cloud management. As essential it should provide unified interface to administration of all cloud subsystems such as storage and network management discussed in earlier chapters. We also believe that this unification should bring easier main-tenance of these systems for administrators. Sadly this is not always true and sometimes it can make things more complicated.

4.1 Layer-bloat problem

In Cloud storage chapter we mentioned problem with layer-bloat in storage management. Here we are facing similar problem, which affects the way how the management system is deployed and used. Many cloud management sys-tems were created on top of other existing utilities and software which provided parts of its functionality. This is not bad as it uses already existing utilities to do the job. Problem arises when these utilities starts to be dependent on each other as all of them provides its own scope of configuration options and its own layer of functionality that builds on top of the other.

The more layers of these utilities are bonded together, the more probable is that failure of any of them would bring breakage of whole system. As an exam-ple serves existence of cloud management for other cloud managements which tries to make abstraction over all other abstractions over management of cloud. Existence of such software indicates disastrous state of these kinds of software.

(26)

4. CLOUD MANAGEMENT

4.2 OpenStack

OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface[8]

Figure 4.1: OpenStack architecture[13]

As description from OpenStack homepage says, OpenStack is aiming at pro-viding solution for complete cloud management. Whole system is logically di-vide into three main categories shown in the figure 4.1. These categories include projects which deals with tasks in different areas:

OpenStack Compute (Nova) OpenStack Image service (Glance) OpenStack Networking (Quantum) OpenStack Object Storage (Swift) OpenStack Block Storage (Cinder)

(27)

Furthermore OpenStack have projects, which overcome categorization and work in general scope of whole OpenStack environment:

OpenStack Identity (Keystone) Metering (Ceilometer)

Orchestration (Heat)

OpenStack Dashboard (Horizon) OpenStack Common Library (Oslo)

Figure 4.2: OpenStack subsystem logical architecture (version Grizzly) [13]

Each of this projects takes care for area of its influence in OpenStack envi-ronment and can be possibly stand-alone. This way every project can be devel-oped on its own and it needs just to provide some interface to rest of OpenStack ecosystem. However, some projects might be dependent on each other and need to communicate. To imagine how this communication is architected, we provide look at newest version of OpenStack named Grizzly in figure 4.2.

To ease configuration of whole system, developers provide not only manuals, but also definition files for automated deployment used by various tools such as Puppet or Chef. The biggest problem could be the number of nearly indepen-dent projects that needs to be configured separately to form OpenStack.

(28)

4. CLOUD MANAGEMENT Distributions powered by OpenStack as of beginning of 2013 include Cloud-scaling, Debian, Fedora, Piston Cloud Computing, Red Hat Enterprise Linux, SwiftStack, openSUSE, Ubuntu and Stackops. OpenStack is also being integrated into software from HP, which wants to create software that would manage whole datacenter. [20]

4.3 OpenNebula

OpenNebula.org is an open-source project developing the industry standard solution for building and managing virtualized enterprise data centers and enterprise private clouds. [9]

Architecture of OpenNebula can be simply divided into three main categories by type: Tool, Component, Interface/API. These types then form the whole en-vironment around it as shown in figure 4.3.

(29)

The main functionality is located in one central component — OpenNebula. Communication with other components is done through XML-RPC API or spe-cialized internal APIs. Spespe-cialized APIs are designated to give control to scripts and programs that communicate directly with physical infrastructure of Open-Nebula cloud environment. Objectives that these programs fulfills are:

Transfer Manager (TM) — transfers configuration files and disk images be-tween nodes in infrastructure

Information Manager (IM) — gathers information about node operation and monitors state of VMs running on it.

Virtual machine Manager (VM) – interacts with local hypervisors and man-ages VMs life cycle on nodes.

Authentication API — authenticates users.

These APIs are on side of physical infrastructure provisioned by scripts mostly written in bash or ruby. This allows great flexibility for future customizations as these scripts can be easily edited. Another special thing is that these scripts are main part of software needed for node to be part of cluster. Except of them only few system utilities are needed depending on used features.

Distributions powered by OpenNebula as of beginning of 2013 include De-bian, Red Hat Enterprise Linux, CentOS, openSUSE and Ubuntu. and Stackops.

(30)

Chapter 5 Performance testing

Important part before deployment of any part of new cloud system is per-formance testing of individual components to maximize perper-formance of future system. We conducted the following tests to determine possible physical limits of available hardware of our deployment and to find best possible combination of components that will bring maximum possible performance for our scenario of deployment.

5.1 Physical testing environment

The testing was conducted on 5 nodes running latest 64bit version of Fe-dora 18 OS. From hardware point of view, there were 4 identical nodes ( later referred to as data nodes) and one more powerful node ( later referred to as

master node). Master node was used as client of DFSs and as master node in cloud management testing.

Configuration of master node:

Processor : 8-core AMD Opteron 6134

Memory : 16 GB ECC DDR2 RAM

Disk : 2x 1 TB Seagate Barracuda 7200.12

Network : 2x Gigabit Intel PRO 1000 Configuration of data node:

Processor : 2-core Intel Core 2 Duo E8500

Memory : 8 GB ECC DDR2 RAM

Disk : 2x 1 TB Seagate Barracuda 7200.12

(31)

Nodes were interconnected using HP 2848 ProCurve switch. Each node was connected to network with both NICs using 1 Gigabit Full-duplex link:

NIC1 - MTU 1500, connected to Internet

NIC2 - MTU 9000, connected to local network on switch

Disks were directly attached to motherboard without using hardware RAID controller. Each node used one disk for OS and one disk for our tests. Before testing, disks were checked for errors and completely rewritten using random pattern.

5.2 Virtual testing environment

To simulate performance in real deployment, some tests were run in VM with following configuration:

CPU: 1x kvm64 CPU on KVM enabled host

Memory: 2048 MB

System: Fedora 18 x86 64 freshly booted from LiveCD

Disk: attached using virtio driver with disabled caching of data

The tests were conducted with SELinux in enforcing mode. Tests in VMs were run from non-graphical terminal to avoid performance drop caused by graphics emulation.

(32)

Chapter 6 Storage performance testing

For comparable performance we have chosen hard drives with similar physi-cal characteristics such as size, linear read and write speed. Comparison of these values was done by a simple test of 2000 reading and writing of 1 MB blocks to disk. The test was conducted using dd program with direct flag set for ensuring that page cache and disk cache are avoided for both reading and writing. The difference between disks was less than 3% during whole testing.

Because most of DFSs use some existing underlying file system on disk we have also tested performance of two file systems (FSs). One was ext4, which is FS widely used as default FS on many Linux distributions and represents conservative FS approach. Second is btrfs, which is FS that gathers lots of pop-ularity by re-making way how modern FSs works. Both of FSs were tested using IOzone File system Benchmark1_{. All raw results of following tests can be found}

as attachment of this work. Test were ran in a way that page and disk cache is avoided. For our work we consider following results from IOzone relevant as they represent extremes which are crucial for wide range of deployments of DFS:

– Linear read performance – Random read performance – Linear write performance – Random write performance

On both file systems, ext4 and btrfs, we ran the test two times, once with 200 MB and once with 400 MB testing file. As a record size we have tested power of 2 from 128 bytes to 8096 bytes (8kB). As relevant record sizes we consider of 512 bytes, 1kB, 4kB and 8kB. While 512 bytes and 4kB are common disk block sizes, 1kB and 8kB are their multiplies. Results of IOzone benchmark on these two FSs show the figure 6.1 and figure 6.2. Please notice that these graphs are scaled to show small differences between FSs.

(33)

Figure 6.1: Linear read/write performance of ext4 and btrfs on physical disks

Figure 6.2: Random read/write performance of ext4 and btrfs on physical disks

6.1 DFS capabilities requirements

For our deployment we decided on following requirements for DFS:

hardware independent — It’s essential that DFS would be independent of hard-ware on which it’s run as most of it would be commodity HW with possi-bly different configurations

distribution independent — There must be at least 2 officially supported Linux distributions with this file system support

redundancy of data — each piece of user data must be stored at least at two physical nodes to provide redundancy and fault tolerance

random read/write performance — DFS should be able to provide high perfor-mance in random reads and writes of small blocks (512 bytes, 1 kB, 4kB, 8kB)

on-line configuration — DFS should provide ability to change number of data nodes without need to shut down whole DFS

self-healing — DFS should be able to heal itself after node failure. Data with reduced redundancy must be always available. It’s not feasible to deny access to data because of undergoing replication.

(34)

6. STORAGE PERFORMANCE TESTING maintenance on multiple data nodes at once without great impact on per-formance of DFS = DFS should not re-balance abruptly

6.2 DRBD

Before moving to ”truly distributed” DFS, we have tested performance of DRBD. This was done to compare it to performance of other DFS. DRBD is ar-chitected to be most efficient in scenario of disk mirroring between two nodes. We expected write and read performance to be very close to native performance of physical disks as if they were stand alone.

In test results we were interested in amount of overhead which brings this networked file system as there is need for synchronization between nodes over network. We also wanted to compare this performance on physical (bare metal) and virtual machine (VM). Results of these tests can be seen in figure 6.3 and figure 6.4. As we can see the overhead of virtualization layer and its impact on I/O (input/output) performance of disks is marginal.

Figure 6.3: Linear read/write performance of ext4 and btrfs on DRBD device in virtualized (virt) and physical (bare) environment

Figure 6.4: Random read/write performance of ext4 and btrfs on DRBD device in virtualized (virt) and physical (bare) environment

(35)

6.3 GlusterFS

First of tested DFSs, which meets most of our requirements, was GlusterFS. Performance tests were conducted using version 3.4-alpha which was stable enough for the testing. In general, it was very easy to start using GlusterFS as it has rapid deployment ability, which only requires few commands to have DFS up and running. There was no need to mess around configuration files, everything was configured via command line interface (CLI). Since the version 3.2 the CLI is only supported way of configuring GlusterFS which makes some manuals around Internet useless.

We tested GlusterFS in configuration with distributed, replicated and striped volume to meet requirements of high performance and redundancy. Exact com-mands used for creating volume can be found in attachment of this work. Be-cause GlusterFS doesn’t provide block device support yet, we had to use files to simulate block device instead. So the virtual disk for VM was in fact RAW non-preallocated image file. Graphs bellow shows the performance of GlusterFS at-tached to VM as storage depending on FS used. In figure 6.5 are tests of linear read and write operations. In figure 6.6 are results for random read and write.

Figure 6.5: Linear read/write performance of ext4 and btrfs on GlusterFS in vir-tualized environment

From the graphs can be seen that, there is a measurable difference in read and write performance of GlusterFS depending on underlying FS and FS used in VM. But in most scenarios this difference is too small to bring some benefits.

(36)

6. STORAGE PERFORMANCE TESTING

Figure 6.6: Random read/write performance of ext4 and btrfs on GlusterFS in virtualized environment

Due to the lack of block device support the previous tests might be discrimi-nating compared to other DFSs that have this functionality. Work on implement-ing this functionality is planned for the final version 3.4 of GlusterFS. However it would be restricted to cluster with one node only which is still not accept-able for deployment in distributed environment. We believe that even despite of that, the performance of block device over DFS would be in that case better than compared to image file over DFS as we tested now.

Another downside we encountered was limited ability to control fail domain. This is very essential thing in planning the outage of several data nodes for maintenance. This is somehow possible in case of replicated volumes where ex-ists concept named Replica-Set. Replica-Set holds list of nodes which can be safely taken down without disruption of whole DFS. However this list of nodes is de-fined statically when volume is created or extended. GlusterFS doesn’t allow dynamic change, which might be needed in some cases. In case we want to turn off machines in one rack that are not in same Replica-Set, there is no possibility to do that or to change it without taking down the DFS.

6.4 CEPH

Second tested DFS that we have chosen was CEPH in version 0.56. Deploy-ment was not so straightforward as in previous DFS and required several stages of file copying over network to establish working cluster. Learning curve to fig-ure out the right configuration was a bit steep.

The tests here used RBD block device provided by CEPH, which was stripped and replicated over the cluster to meet requirements. Block device was attached to system via Linux kernel module, which should bring better performance in

(37)

some cases but doesn’t support some features such as copy-on-write clones. Scripts used to configure testing environment can be founded in attachment of this work. Graphs in figure 6.7 and figure 6.8 shows performance of CEPH RBD device used by VM as storage device. Interesting is good read performance of ext4 used in VM.

Figure 6.7: Linear read/write performance of ext4 and btrfs of CEPH RBD in virtualized environment

Figure 6.8: Linear read/write performance of ext4 and btrfs of CEPH RBD in virtualized environment

Despite quite complex configuration CEPH gives us more options that we can use. For example one configuration, which is separated from main configu-ration file, is CRUSH map. This map describes how the cluster looks and allows us to create logical view over it. So we can organize OSDs into racks, storage rooms, datacenters or anything that we come to into hierarchical structure. But CRUSH map is not about only organizing things but it’s about the weight of each part of tree hierarchy in determining where to place data. This can be es-pecially useful when we need to change physical topology of cluster and then we want to rebalance the data distribution in the cluster. Also it can be used for maintenance when we can tag some parts of tree to be offline for a while but we

(38)

6. STORAGE PERFORMANCE TESTING don’t want to rebalance data that are stored there. This is case when we want to just reboot one rack of computers for maintenance.

6.5 MooseFS and SheepDog

Through initial testing we have also tried other DFSs which got major draw-backs described in earlier chapters.

MooseFS was hot candidate as competitor of GlusterFS however it’s stability was a concern to us because it was susceptible to crashes which lead to DFS un-availability during our testing. Lack of block device support without any plans for near future made this DFS not interesting for further testing.

The main problem of SheepDog project is its immaturity. It still contain some serious flaws and it’s not supported by many distributions in recent versions. It’s high-availability stands on top of other software such as corosync, which we don’t think it’s the good solution for DFS used for virtualization. Still con-cepts presented by this project are really interesting as they describe the need of specialized DFS for virtualization. Despite all tries we were not able to make it run on recent version Fedora.

(39)

6.6 CEPH and GlusterFS performance comparison

We consider the comparison of CEPH and GlusterFS as very interesting. Com-parison of linear read/write speed can be seen in figure 6.9. Despite CEPH clearly wins in speed of linear write, GlusterFS is comparable and sometimes better in linear read.

Figure 6.9: Linear read/write performance between CEPH and GlusterFS

The situation in figure 6.10 however shows mostly better or comparable speed of random read/write operations for CEPH. The reason of dramatically higher read performance in case of ext4 as VM file system is still a mystery for us.

(40)

Chapter 7 Cloud management testing

Our knowledge of cloud management starts from assumption that it could be very similar to libvirt library, which is widely used at Linux platform for unified access to virtualization. As we discovered most of cloud management projects build on top of it. However this new layer of abstraction brings new set of problems in way how the whole system works. Some of them that we encountered are described later in next chapter.

Our vision about the cloud management was that it would be able to work with at least one of tested DFSs. Also it would support the use of some more advanced networking setup either using Open vSwitch or similar.

For consistent testing environment, we first used VMs to test cloud manage-ments as they are easily disposable. This approach saved us hours of sitting in server room testing this on real hardware as VMs were breaking down pretty regularly. The goal of this phase was to test feasibility of deployment and cre-ate customized manuals for future deployment of selected projects in our envi-ronment. Following Linux distributions were used to host cloud management software:

– Fedora 18 x86 64 – Fedora 19 x86 64 – Debian 6 x86 64 – CentOS 6 x86 64

Second phase was bare metal deployment in our 4-node data cluster with one master node. We have tested customization of tested systems to our environ-ment. Compared to isolated virtual environment in first test phase, the biggest problems here lied in network customization. As real network was behaving differently than simulated environment and we interacted with living network.

(41)

Last phase was conducted on working cluster with working cloud manage-ment. Here we tested way how to create VMs, work with them, migrate etc. As essential we took set of functionality provided by our current virtualization management system. This phase revealed some differences in concepts used in cloud management compared to our virtualization management, which would be discussed in following sections.

7.1 OpenStack

After initial enthusiastic look at features of this project, which included nearly all aspects of cloud management, we experienced drastic turnover in our opin-ions. Thanks to this ”we can do everything” approach, the complexity of whole system and it’s configuration has greatly increased. As example of complexity we consider simplified manuals for deployment on Debian1 or Fedora2. The amount of work that need to be conducted to get to basic functionality of Open-Stack is comparable to installation of whole operating system with its configu-ration. We tested version Folsom as it was the one that was supported by most of manuals and recommended for deployment in production environment.

We’ve tried both manuals, for Fedora and Debian, with more or less success. In case of Fedora we got to the partially functional system which was suffer-ing from random crashes or errors that were hard to track down. On Debian we failed on ”ugly hack” script, which was not working at that time. We were testing on Debian 6.0, in time of writing Debian 7.0 was released which should address these problems.

Another way of installation that we tried is called ”devstack”, which com-piles current development version of OpenStack on supported platforms. Sur-prisingly, success rate was higher than in case of earlier mentioned manuals. But still some parts of system were not functioning right or at all.

Whole OpenStack is organized into projects that takes care for individual parts such as networking, image storage etc. Most of projects requires some database store data. These databases are not shared among projects so every one that need database to store data create its own database. This should allow easier upgrade of individual projects.

1. https://wiki.debian.org/OpenStackHowto/Folsom

2. https://fedoraproject.org/wiki/Getting_started_with_OpenStack_on_ Fedora_18

(42)

7. CLOUD MANAGEMENT TESTING Another thing that we considered as a problem is the amount of software, which got installed by OpenStack to reach the working state. We have started with minimal installation of Fedora and ended up with system containing nearly 5x more packages. This number is comparable to desktop installation of Fedora. Maintainability of whole system is affected by this in a bad way.

Our testing of OpenStack ended after several weeks with the idea that instal-lation and maintenance should be rather conducted by team of several people to be sustainable. This scenario is however not acceptable in our environment and we don’t recommend OpenStack for small scale projects.

7.2 OpenNebula

Projects goals here are much humbler than becoming cloud OS. This can be seen in much simpler architecture compared to OpenStack. Installation is well documented and simple3_{. It doesn’t require much of an effort and can be}

de-ployed by one person in less than a day from nothing to testing environment that can be further customized. Both, installation from packages in distributions and from source code were straightforward. Most of issues during installation are solved by installing required packages, but most of them are mentioned in manual.

We successfully deployed OpenNebula in all testing environments. We have used versions 3.8, 3.9.80 (4.0-rc) and 4.0 final version as it came out. Compared to OpenStack, the individual subsystems here are part of OpenNebula project and doesn’t work on their own. This brings better integration and development for the cost of potentially worse upgrade process, where is not possible or hard to do upgrade of single part. This, however, was not confirmed when we were upgrading between version 3.9.80 and 4.0. Only requirement there was upgrade of database and it went without problems.

The number of packages required for OpenNebula to install is very low but might be not the final number. Distribution packages don’t include all possible dependencies, so it might be needed to install some packages manually when they are required by scripts. We really appreciate the way how the nodes that participate in OpenNebula cluster. They only require the dedicated user, which executes scripts that are included in installation. Also, it is well documented

(43)

which special permissions this user needs as not everything can be done with-out root privileges. But it is possible to use sudo to grant only needed capabilities to OpenNebula user.

General impression from OpenNebula was good, and despite of several dif-ferences in way how it differs from pure libvirt, it was acceptable for our sce-nario. That’s why we recommend this for use in smaller deployments. It is fea-sible to understand architecture of it well in matter of days.

(44)

Chapter 8 Deployment at FI MU

From previous tests we have chosen final implementation to be:

DFS: CEPH with RBD

Network: Open vSwitch with VLANs

Cloud management: OpenNebula

Decision for CEPH was based on it’s ability to support block devices. Another important thing was ability to distribute data based on CRUSH algorithm. One downside was more complicated configuration than GlusterFS but it’s still fea-sible to use and maintain it. GlusterFS might be interesting alternative when block device support with reasonable abilities would be introduced. We hope that it could coexist side by side with CEPH to provide alternative for users.

As network subsystem, we wanted something flexible to be able of future development. Open vSwitch with it’s support for OpenFlow protocol allows us for future integration into Software defined network (SDN). As of now, the project is also rapidly evolving but its presence in Linux kernel assure us that it’s quality meets requirements of kernel developers. As transport between nodes, we will use combination of traditional VLANs for interconnecting with existing networks and VXLANs for isolated networks and future use.

Most important thing to choose was cloud management that needs to cover all our requirements and provide simple and easy maintainable architecture. Despite the OpenStack architecture looks very modular, it was very hard to keep it in working state or even to deploy it right. For us, it’s important that maintenance would require minimal number of people.

Decision was made to use OpenNebula as it’s smaller in terms of complexity and architecture. Another advantage that we find a big plus is ability to change behavior of it by just editing shell scripts. This flexibility really helps us to debug errors or to create completely new or customized functionality easily. Most of

(45)

people that would maintain this system prefers editing shells scripts instead of C/C++ code.

8.1 Existing infrastructure and front-end

Currently we use system which is part of fadmin1_{, web frontend with it’s own}

database that communicates directly with virtualization hosts. This communi-cation is provided by Perl module Sys::Virt. Everything from users and limit management to monitoring and reporting is handled by this frontend, which makes it ineffective in many new situations and very impractical to implement new functionality. This software is also maintained only by few of adminis-trators, which work at FI MU. Code is very dependent to specifics of FI MU environment. This system was developed as a part of my bachelor thesis and it serves it’s initial purpose well, and users likes it’s simplicity.

Current solution was developed with goal to act in future as OpenNebula works now. But the architecture was different and in some way more flat, which increased complexity of code significantly. The biggest problem was fusion of all parts into one component that handles front-end and backend operations. Oppose to that OpenNebula exposes only needed parts through its APIs and leaves the rest to programmers to build on top of already self-sustainable man-agement system. Hard work on physical infrastructure is conducted not directly by OpenNebula core.

The advantage in using OpenNebula in favor of current system is XML-RPC API that allows us to use more generic Perl modules for XML-RPC commu-nication. They are compared to Sys::Virt more independent of libvirt version used. This is important as we experienced that after upgrade of libvirt daemon on virtualization host older Sys::Virt suddenly stopped working. That made us unable to communicate with newer version of libvirt on virtualization hosts and required immediate upgrade of Sys::Virt with uncertain results. The only certain result was need for new code to be integrated into our solution. This was one of motivations for us to move from libvirt-only based management to something more high-level. Because our users got used to the integrated web frontend so we wanted to preserve it.

(46)

8. DEPLOYMENT ATFI MU

8.2 Current deployment state

In time of writing, we already altered several modules for displaying infor-mation about VMs in fadmin to use only OpenNebula API. This change greatly simplified code because OpenNebula already do lot of checks that our previous application was implementing itself.

Despite most of functionality in fadmin can be rewritten to use OpenNebula API compared to libvirt API, there are some limitations and changes. As one of them is different concept of VM state monitoring. Most of readings taken from libvirt that we presented in front-end were accurate and realtime, but now it changed. OpenNebula have time interval for monitoring individual parts of itself such as VMs, their states, memory consumption on hosts etc. It collects these data periodically. On other side virtualization hosts are not bothered too often by front-ends queries for information.

Biggest contribution of OpenNebula to our system was user and group man-agement. As for now, user management was stand alone and partially synchro-nized with users of fadmin. However, it was not ready for distributed envi-ronment where user can have multiple VMs on different virtualization hosts, which can dynamically migrate. Changing to OpenNebula user management would require new implementation of synchronization. Compared to keeping user management in fadmin, this brings benefit that we can offload this respon-sibility to OpenNebula. Implementation of synchronization is undergoing and it doesn’t look complicated.

(47)

Conclusion

We have described the background of cloud deployment with focus on peo-ple involved in deployment of cloud systems, and also three main parts : Cloud storage, Cloud Networking and Cloud management. In these parts, we have explained technologies currently used in real world and interesting projects in-volved in this area.

As a part of our work describes practical tests that we conducted to support our vision of cloud deployment at FI MU. Along with that, we described meth-ods used for testing and we described exact manual how to repeat tests to verify results.

In the end we explained current situation around virtualization at FI MU. Accordingly we presented solution that will replace current deployment with cloud management solution and its advantages for current and future use.

As ”Cloud” covers many technologies, we don’t consider our work to be a report of a finished deployment, but rather a complete conceptual manual for how it should be done to meet requirements we expected.

As future for this project we would like to integrate this solution with Win-dows virtualization platform to allow unified access to VMs at FI MU.

(48)

Bibliography

[1] Claude Baudoin. Public cloud service agreements: What to expect and what to negotiate. http://www.cloudstandardscustomercouncil. org/PublicCloudServiceAgreements2.pdf, March 2013.

[2] Community. http://www.gluster.org/about/, November 2009. [3] Community. http://ceph.com/, 2013. [4] Community. http://ceph.com/docs/master/rados/, 2013. [5] Community. http://www.moosefs.org/about-mfs.html, 2013. [6] Community. https://github.com/collie/sheepdog/wiki/ Cluster-Management-Backends-and-dual-NIC_{, 2013.} [7] Community. http://openvswitch.org/, 2013. [8] Community. http://www.openstack.org/software/, 2013. [9] Community. http://opennebula.org/about:about, 2013. [10] Community. http://opennebula.org/documentation:rel4.0: introapis_{, 2013.}

[11] community. Architecture - ceph documentation. http://ceph.com/ docs/master/architecture/_{, 2013.}

[12] Community. Architecture -ceph documentation. http://ceph.com/ docs/master/architecture/, September 2013.

[13] Community. Openstack compute administration guide - grizzly. http://docs.openstack.org/trunk/openstack-compute/ admin/bk-compute-adminguide-trunk.pdf_{, 2013.}

[14] Jeff Darcy. http://hekafs.org/index.php/2012/11/ trying-out-moosefs/_{, November 2012.}

(49)

[15] Muntimadugu Divya and Sriram Anjana, Suparna. Red hat storage 2.0. https://access.redhat.com/site/documentation/en-US/ Red_Hat_Storage/2.0/html/Administration_Guide/, 2013. [16] Lars Ellenberg. Drbd 9 - linux storage replication. http://data.guug.

de/slides/lk2008/le_drbd9-lk2008-slides.pdf, 2008.

[17] Mohammad Hamdaqa. Cloud computing uncovered: A research land-scape. Elsevier Press, pp. 41-85., http://www.stargroup.uwaterloo. ca/˜mhamdaqa/publications/Cloud_Computing_Uncovered. pdf, 2012.

[18] IEEE. Ieee std 802.1qtm-2011. http://standards.ieee.org/ getieee802/download/802.1Q-2011.pdf, 2011.

[19] Sam Johnston. http://upload.wikimedia.org/wikipedia/ commons/b/b5/Cloud_computing.svg, March 2009.

[20] Tom´aˇs Kubica. http://h41267.www4.hp.com/filelib/cz/cs/ 635046518284403705_aad79035bef41a76294dd93f2d0744dc. pdf, May 2013.

[21] T. Grance P. Mell. The nist definition of cloud computing, recommenda-tions of the national institute of standards and technology special publica-tion 800-145. Napublica-tional Institute of Standards and Technology, 2009.

[22] Rhea S., Wells C., Eaton P., Geels D., Zhao B., Weatherspoon H., and Ku-biatowicz J. Maintenance-free global data storage. IEEE Internet Comput-ing, Vol 5, No 5, pp 40-49, http://oceanstore.cs.berkeley.edu/ publications/papers/pdf/ieeeic.pdf, September/October 2001.