Cloud-Hosted Databases: Technologies, Challenges and Opportunities

Cloud-Hosted Databases:

Technologies, Challenges and Opportunities

Sherif Sakr

NICTA and University of New South Wales, Australia

The 9th International Conference on Informatics and Systems (INFOS 2014)

Cloud Computing

Cloud computing is associated with a new paradigm for the provision of computing infrastructure that shifts the location of this infrastructure

to thenetworkand reduces thecostsassociated with the management

of hardware and software resources.

Businesses and users become able to access application services from anywhere in the world on demand which represents the long-held dream

of envisioningcomputing as a utilitywhere the economy of scale

prin-ciples help to drive the cost of computing infrastructure effectively down.

The major enabling features of the cloud includeelasticityof resources,

pay-per-use cost model, low time to market, and the perception of

Cloud Data Management

In practice, the advantages of the cloud computing paradigm opens up new avenues for deploying novel applications which were not economi-cally feasible in a traditional enterprise infrastructure setting.

The apparent next goal of the system and tool manufacturers is to facilitate the job of implementing every application as a distributed,

scalable, and widely-accessible service on the web (e.g., Facebook,

Flickr,YouTube,Zoho, and LinkedIn).

Facebook has recently announced that it has more than a billion user. Each user has an average friendship relation count of 130. Moreover, there are about 900 million objects that registered users interact with such as pages, groups, events, and community pages.

Cloud Data Management

We are witnessing a proliferation in the number of applications with a tremendous increase in the scale of the data generated as well as consumed by such applications.

Cloud-hosted database systems powering these applications form a

Cloud Data Management

In general, successful cloud data management services should satisfy as much as possible from the following goals:

Availability: They must be always accessible even on the occasions where there is a network failure or a whole datacenter has gone offline.

Scalability: They must be able to support very large databases with very high request rates at very low latency.

Elasticity: They must be able to satisfy changing application require-ments in both directions (scaling up or scaling down).

Performance: On public cloud computing platforms, pricing is struc-tured in a way such that one pays only for what one uses, so the vendor price increases linearly with the requisite storage, network bandwidth, and compute power. Therefore, the system performance has a direct effect on its costs.

Data Replication

When the application load increases, there are two main options for achieving scalability at the database tier and make the application able to cope with more client requests:

Scaling up: aims at allocating a bigger machine with more horsepower (e.g. more processors, memory, bandwidth) to act as a database server. Scaling out: aims at replicating and partitioning data across more ma-chines.

The scaling upoption has the main drawback that large machines are

often very expensive and eventually aphysical limitis reached where a

more powerful machine cannot be purchased at any cost.

Thescaling outmodel fits well with thepay-as-you-go pricing philoso-phy.

Disclaimer

This Talk is not going to touch Hadoop.

For those who are interested to learn about Hadoop family of systems, we

refer to our ACM Computing Surveys paper2.

2_{Sherif Sakr et al. The family of mapreduce and large-scale data processing systems.}

Agenda

Alternative Options of Deploying Cloud-Hosted Database Sys-tems

NoSQL Database System (Not Only SQL) Database-as-a-Service (Daas)

Virtualized Database Servers

Research Challenges and Opportunities

True elasticity Live migration

Data replication and consistency management SLA management

Transaction support Benchmarking

Part I

NoSQL Database Systems

For decades, relational database management systems (e.g. MySQL,

PostgreSQL, SQL Server, Oracle) have been considered as the

one-size-fits-all solution for providing data persistence and its retrieval for decades.

The ever increasing need for scalability and new application require-ments have created new challenges for traditional RDBMS.

At some stage, there has been some dissatisfaction with this one-size-fits-all approach in deploying the data storage tier for large scale online web services.

NoSQL Database Systems

NoSQL database systems represent a new generation of low-cost, high-performance database software which is increasingly gaining more and more popularity.

These systems promise to simplify administration, be fault-tolerant and able to scale on commodity hardware (Scale out) which makes them so attractive, that more and more developers prefer relying on them even for smaller self-hosted environments over RDBMS.

BigTable (presented by Google) and Dynamo (presented by Amazon)

systems have provided a proof of concept that inspired and triggered

Design Features of NoSQL Database Systems

The ability to horizontally scale out throughput over many servers.

A simple call level interface or protocol (in contrast to a SQL binding).

Supporting weaker consistency models (e.g. eventual consistency3)

than the strong consistency models of most traditional RDBMS.

Efficient use of distributed indexes and RAM for data storage.

The ability to dynamically define new attributes or data schema.

3

Eventual consistency guarantees that if no new updates are made to a replicated object, eventually all accesses will return the last updated value. Vogels: Eventually consistent. ACM Queue 6, 2008

NoSQL Database Systems

Alternative Design Decisions

Data Model: Key/Value, Row Store, Extensible Record stores, Graph-Oriented, Document-Oriented

Access-Path Optimization: Read-intensive VS Write-intensive, Single-Key Vs Multi-Single-Key

Data Partitioning: Row-Oriented, Column-Orieneted, Multi-Column Oriented

Concurrency Management: Strong Consistency, Eventual Consis-tency, Weak Consistency

Replication Management: Single Master, Multi Master, 2PHC

Too Many Choices Often Make Us Confused :(

We have choices, but ...

There is a wide variety of NoSQL systems in terms of their functional and non-functional offerings

Developers need to understand many things.

Trade-offs among technologies/configurations.

How to fulfill applications’ requirements using which storage with what configurations.

General Limitations of NoSQL Database Systems

Programming Model: NoSQL databases offer few facilities for ad-hoc query and analysis. Even a simple query requires significant program-ming expertise. Missing the support of declaratively expressing the important join operation has been always considered one of the main limitations of these systems.

Transaction Support: The current limited support (if any) of the transaction notion from NoSQL database systems is considered as a big obstacle towards their acceptance in implementing mission critical systems.

No Support for Multiple Data Centers: Except Cassandra, all open source NoSQL projects are designed to run on a cluster of machine in a single data center and they are nor designed to run over multiple data centers.

General Limitations of NoSQL Database Systems

Maturity and Support: Most NoSQL alternatives are in pre-production versions with many key features either not stable enough or yet to be implemented. Therefore, enterprises are still approaching this new wave with extreme caution.

Expertise: Almost every NoSQL developer is in a learning mode. This situation will address naturally over time. However, currently, it is far easier to find experienced RDBMS programmers or administrators than a NoSQL expert.

One Size Does Not Fit All

Therefore, there is still a big debate between the proponents of the NoSQL and RDBMS camps which is centered around the right choice for implementing online transaction processing systems.

RDBMS proponents think that the NoSQL camp has not spent suffi-cient time to understand the foundation of the transaction model. The NoSQL camp argues that the domain-specific optimization op-portunities of NoSQL systems gives more flexibility to the developers. However, they admit that making such optimization decision requires a lot of experience and can be very error-prone and dangerous if not done by experts.

Part II

Multi-tenancy

Data centers are often under-utilized due to over-provisioning as well

as time-varying resource demands of typical enterprise applications.

Multi-tenancy (a technique which is pioneered bysalesforce.com) is an

optimization mechanism in which multiple customers are consolidated onto the same operational system and thus the economy of scale princi-ples help to effectively drive down the cost of computing infrastructure.

Multi-tenancy

One of the main advantages of multi-tenancy is that it allows pool-ing of resources which improves utilization by eliminatpool-ing the need to provision each tenant for their maximum load.

Multi-tenancy is an attractive mechanism for the cloud providers who

are able to serve more customers with a smaller set of machines.

Multi-tenancy is also an attractive mechanism to customers of cloud

services who do not need to pay the price of renting the full capacity of a server but only for the capacity they use.

Multi-tenancy

In practice, there are three main approaches for the implementation of multi-tenant database systems:

Shared Server: where each tenant is offered a separate database in the same database server.

Shared Process: where each tenant is offered its own tables while mul-tiple tenants can share the same database.

Shared Table: where the data of all tenants is stored in the same tables and each tuple has an additional column with the tenant identifier.

The shared server multi-tenancy model is the most commonly used by most commercial Database-as-a-Service providers as it is considered to be the most effective approach to secure the isolation of each tenant’s data and allocated computing resources.

Database-as-a-Service (DaaS)

Database-as-a-service (DaaS) is a new paradigm for data management in which a third party service provider hosts a database as a service. The service provides data management for its customers and thus al-leviates the need for the service user to purchase expensive hardware and software, deal with software upgrades and hire professionals for administrative and maintenance tasks.

Commercial offerings include:

Amazon RDS provides access to the capabilities of MySQL, Oracle

database or SQL Server.

Microsoft SQL Azure has been built to provide the capabilities of

Mi-crosoft SQL Server technologies on the cloud.

Google Cloud SQLis a fully managed services that that provides the

ca-pabilities and functionality of MySQL database servers which are hosted in Google’s cloud.

Heroku Postgresis a cloud database service that provides the capabilities

Main Advantages of Database-as-a-Service (DaaS)

Users of these services can leverage the capabilities of traditional rela-tional database systems such as creating, accessing and manipulating tables, views, indexes, roles, stored procedures, triggers and functions. It can also execute complex queries and joins across multiple tables.

The migration of the database tier of any software application to a relational database service is supposed to require minimal effort if the underlying RDBMS of the existing software application is compatible with the offered service.

Limitations of Database-as-a-Service (DaaS)

The user does not have the full control on provisioning the hardware resources that can achieve his performance and scalability goals.

Not every relational database systems supported by each cloud provider.

Deploying (geographically distributed) database replicas over multiple data center is not, yet, supported.

Some limitations or restrictions might be introduced by the service provider for different reasons such as database count, size limit and connection constraints.

Part III

Virtualized Database Servers

Virtualization is a key technology of the cloud computing paradigm. Virtual machine technologies are increasingly being used to improve the manageability of software systems and lower their total cost of ownership.

Virtualization allow resources to be allocated to different applications on demand and hide the complexity of resource sharing from cloud users by providing a powerful abstraction for application and resource provisioning.

Resource virtualization technologies add a flexible and programmable layer of software between applications and the resources used by these applications.

Database servers, like any other software components, are migrated to run in virtual machines (e.g. Amazon EC2 Instances).

Advantages of Virtualized Database Servers

The easiest option for migrating the database tier of existing software application.

The application can have the full control in dynamically allocating and configuring the physical resources of the database tier (database servers) as needed.

Software applications can fully utilize the elasticity feature of the cloud environment to achieve their defined and customized scalability or cost reduction goals

Enables the software applications to build their geographically

dis-tributed database clusters. Without the cloud, building such in-house cluster would require self-owned infrastructure which represent an op-tion that can be only afforded by big enterprises.

Advantages of Virtualized Database Servers

Limitations of Virtualized Database Servers

It is on the onus of the software application to:

Dynamically provision the allocated physical resources for the database tier (elasticity).

When to provision a database? How to provision a database?

The monetary cost management.

Management of application-defined SLA. Data replication management.

Dynamic Provisioning

Application workloads are quite dynamic especially for internet scale applications.

Application workloads are quite different.

Different applications have different requirements. Different modules of the same application may require different goals (SLA) and conse-quently use different design decisions (rationing approach / pay only when it matters).

Dynamic Provisioning

In general, the decision of when to an increase or decrease in the

allo-cated computed resources is made in alazyfashion for the web and the

application tiers of the software application, in response to an actual or anticipated significant workload change.

Such lazy triggers are appropriate for these tiers since that new capacity can be added relatively in a quick manner and whenever required as the only incurred latency is for virtual machine startup.

Provisioning of a new database replica involves copying and restor-ing a new replica which can take minutes or hours dependrestor-ing on the database size.

Monetary Cost

The focus of many enterprises from using cloud environments is to reduce costs rather than improving the performance.

Evaluation figures where the Y-axis represent the money cost rather

then the performance characteristics will be quite common (if not mandatory) shortly.

Variability of performance of public cloud providers and their direct affect on the cost need to considered (performance SLA).

Admission Control

Cloud DB Administrator is a poor guy.

In practice, finding the right configuration of cloud databases (design decisions) in order to guarantee the high performance or reduce the money cost is not a trivial task at all.

Reacting to dynamic characteristic of application workloads (e.g. elas-ticity spikes) might not be affordable in short period considering that it should momentum and avoid any unrequired waste of money.

Automated/Semi-Automated techniques for managing the database tier in cloud environment is quite important and highly required.

Dolly System

Dolly is an admission control system for dynamically provisioning database replicas in cloud platforms.

It is database platform-agnostic and uses virtualization-based replica-tion mechanisms for efficiently spawning database replicas.

Dolly relies on eager provisioning decisions that incorporates a model

to estimate the latency to spawn a replica (based on the VM snapshot

sizeand thedatabase resynchronization latency) and uses this model to

trigger the replica spawning process well in advance of the anticipated workload increase.

Dolly incorporates an intelligent scheduling technique that can deter-mine whether it is cheaper to take a new VM snapshot or use an older snapshot when spawning a new replica.

SLA Management for Cloud Hosted Database

According to a Gartner market report released in November 2010, SaaS is forecast to have a 15.8% growth ratethrough 2014 which makes SaaS and cloud very interesting to services industry.

The viability of the business models depends on the practicality and the success of the terms and conditions (SLAs) being offered by the service provider(s) in addition to their satisfaction to the service consumers Successful SLA management is a critical factor to be considered by both providers and consumers alike.

SLA indicates the level of service agreed upon as well as the associated cost if the service provider fails to deliver the level of service.

SLA Management for Cloud Hosted Database

Infrastructure service providers charge business service providers for renting computing resources to deploy their applications.

Software service providers may charge the users for processing their

workloads (e.g. Software-as-a-Service) or may process the user requests

for free (cloud-hosted business application)

In both cases, the software service provider needs to guarantee their users’ SLA. Penalties are applied in the case of SaaS and reputation loss is incurred in the case of cloud-hosted business applications.

For example, Amazon found every 100ms of latency cost them 1% in sales and Google found an extra 500ms in search page generation time dropped traffic by 20%

Challenges of SLA Management for Cloud Hosted

Database

Performance variability of cloud resources

Several studies have reported that the variation of the performance of cloud computing resources is high.

Cloud providers do not provide adequate SLA for their service offerings. Particularly, most providers guarantee only the availability (but not the performance) of their services.

Uncertain behavior

One complexity that arises with the virtualization technology is that it becomes harder to provide performance guarantees and to reason about a particular application’s performance.

The performance of an application hosted on a virtual machine becomes a function of applications running in other virtual machines hosted on the same physical machine.

Challenges of SLA Management for Cloud Hosted

Database

Highly dynamic workload

An application service can be used by large numbers of end-users and highly variable load spikes in demand can occur depending on the day and the time of year, and the popularity of the application.

The characteristic of workload could vary significantly from one appli-cation type to another and possible fluctuations on the workload char-acteristics which could be of several orders of magnitude on the same business day may occur.

Predicting the workload behavior (e.g. arrival pattern, I/O behavior, ser-vice time distribution) and consequently accurate planning of the com-puting resource requirements are very challenging tasks.

CloudDB AutoAdmin

CloudDB AutoAdmin5is an SLA-based admission control for a declar-ative management of the application-defined SLA for cloud-hosted databases.

Two main consumer-centric SLA metrics: Transaction response time

andReplication delay.

The SLA of the application is declaratively defined in terms of goals which are subjected to a number of constraints that are specific to the running application.

The defined SLA is fed to the SLA checkercomponent which

contin-uously evaluates them against the reported metrics from a monitoring

module and triggers the execution of necessary corrective actions (e.g. scaling out/in the database tier) when required.

5

http://cdbslaautoadmin.sourceforge.net/

SLA Management for Cloud Hosted Database

Cloud Infrastructure SLA (I-SLA): these SLA are offered by cloud in-frastructure providers to cloud consumers to assure the quality levels of their cloud computing resources (e.g., server performance, network speed, resources availability, storage capacity).

Cloud-hosted Application SLA (A-SLA): these guarantees relate to the levels of quality for an application which is deployed on a cloud infras-tructure. In particular, cloud consumers often offer such guarantees to their application’s customers/ users to assure quality of services they offer to them such as the application’s response time and availability. Mapping application-defined SLA to low-level resource utilization is a very complex and challenging task.

The burden on the consumers applications to assure their applications’ SLA (e.g. performance, throughput, response time) to their customers.

CloudDB AutoAdmin

Action

Module

Load balancer actions Database actions ...

Control Module

Monitor Module

CloudDB AutoAdmin

The trigger of these corrective actions is based on a set ofdeclarative

application-defined rules where the expected system goals (performance or cost) is optimized according to the defined SLA.

Examples of declarative rules are:

Scale outthe underlying database tier if the average percentage of SLA

violation for transactions T1 and T2 exceeds 10% for a continuous period of more than 8 minutes.

Scale inthe database tier if the average percentage of SLA violation for

transactions T1 and T2 is less than 2% for a continuous period that is more than 8 minutes and the average number of concurrent users per underlying database replica is less than 25.

The framework is designed in a way that it acts as an admission control to achieve the SLA management goal for the database tier of software applications non-intrusively with zero change in their line codes.

CloudDB AutoAdmin / SLA-Based Elasticity Manager

Master

Database Replica(1) Replica(N)

Transaction

Monitor CheckerSLA ManagerAction

Database Proxy

Load Balancer

Transaction

SLA ActionRules

Software Application Database Requests A da pti ve A cti on s A dmi ss ion C ont rol

. . .

CloudDB AutoAdmin / SLA-Based Elasticity Manager

SLA Checker: Responsible of checking the results of the monitoring module and comparing them against the application-defined SLA of the different transaction types in order to report the percentage of SLA vi-olations for each of them (if any). The SLA checker relies on a dynamic scheduler that uses the transaction rate as a main metric. Thus, it en-sures that when the load is high, the SLA checks run more frequently since its most likely the violations happen during these periods of time.

Action Manager: Continuously evaluates the condition of the

application-definedaction rulesand executes the necessary action when these

CloudDB AutoAdmin / SLA-Based Elasticity Manager

(Vi) that an end user will pay/get if a cloud consumer application brings

the transaction (Ti) to completion according to the defined SLA (Si).

TransPenalty (Ti,Si, Vi): is the amount of monetary/dissatisfaction units (Vi) that the cloud consumer application (Ti) will pay if it violates the defined SLA (Si) of the end user transaction (Ti).

SLA-Satisfaction (Ti, Pi, Ii): represents the percentage (Pi) of the instances of the application transaction (Ti) that satisfied their defined SLA during the last period of time units (Ii).

SLA-DisSatisfaction (Ti, Pi, Ii): represents the percentage (Pi) of the instances of the application transaction (Ti) that did not fulfill their

CloudDB AutoAdmin / SLA-Based Elasticity Manager

of the instances (Ni) of the application transaction (Ti) that did not

fulfill their defined SLA and the average magnitude of SLA violation (Mi) during the last period of time units (Ii).

SLA-Threshold (Ti, Pi,Ii): defines the minimum percentage (Pi) of the instances of the application transaction (Ti) that should fulfill their

defined SLA during the last period of time units (Ii).

RreplicaCost (C): is the amount of monetary units (C) which will be paid by the cloud consumer to the cloud service provider (per hour) for renting the required resources for hosting an addition replica of the database tier.

[Min/Max]Rreplicas: represents the boundary limits for the mini-mum/maximum number of replicas that can be allocated for the database tier of the software application.

Replication Performance on Virtualized Database Servers

us-west eu-west ap-southeast ap-northeast L2 u s -e a s t-1 a u s -e a s t-1 b L3 Cloudstone benchmark Master

Slave1 Slavek Slavek+1 Slaven

Slave1 Slavek Slavek+1 Slaven

M write operations N r e a d o p e ra tio n s ( d is tr ib u te d )

Replication within the same region and the same availability zone

Replication within the same region but across availability zones

Slave1 Slaven

Replication across regions

u s -e a s t

M / N satisfies pre-defined read/write ratio

L1

CloudDB AutoAdmin / Replication Delay Controller

Replication within and across regions

M / N satisfies pre-defined read/write ratio

Slave eu-west-1 Slave eu-west-2 MySQL Proxy The controller Monitor and manage to scale

Virtualization Design Advisor

A configuration management component that uses information about the anticipated workloads to automatically determine an appropriate configuration for the virtual machine in which it runs so that it can avoid allocating resources to DBMS instances of which little benefit will be obtained.

The advisor relies on cost models that can predict the workload perfor-mance under different resource allocations. For example, the advisor can distinguish CPU intensive workloads from I/O intensive workloads and allocate more CPU to the former case.

Part IV

Challenge 1: True Elasticity

A common characteristic of internet scale applications and services is that they can be used by large numbers of end-users and highly variable load spikes in the demand for services can occur depending on the day and the time of year, and the popularity of the application.

The workload characteristic could vary significantly from one applica-tion type to another where possible fluctuaapplica-tions on the workload char-acteristics which could be of several orders of magnitude on the same business day may also occur.

To unleash the power of the cloud computing paradigm, cloud database systems should be able to transparently manage and utilize the elastic computing resources to deals with fluctuating workloads. In particular, they should allow users to add and remove computing resources as necessary.

Challenges of True Elasticity

Both of the commercial NoSQL cloud offerings (e.g. Amazon Sim-pleDB) and commercial DaaS offerings (e.g. Amazon RDS, Microsoft SQL Azure) do not provide their users with automated facility to dy-namically increase or decrease the allocated computing resources of their applications.

While NoSQL offerings claim to provide elastic services of their ten-ants, they do not provide any guarantee that their provider-side elas-ticity management will provide scalable performance with increasing workloads.

Commercial DaaS pricing models require their users to pre-determine the computing capacity that will be allocated to their database instance

as they provide standard packages of computing resources (e.g. Micro,

Challenges of True Elasticity

With this setup, the user of cloud database services might still tend to over-provision the allocated computing resources for the database tier of their application in order to ensure satisfactory performance for their workloads. As a result of this, the software application is unable to fully utilize the elastic feature of the cloud environment.

The approach of virtualized database server provides software applica-tions with more flexibility and control on being able to dynamically allo-cate and configure the physical resources of the database tier (database servers). However, this requires implementing an admission control component which is responsible for executing the application logic of its elasticity mechanism

Provider-Centric Approach for True Elasticity

The main focus of this approach is on intelligently managing the com-puting resources in a shared multi-tenant database system at the virtual machine level.

The approach uses machine learning techniques to learn a model that describes the potential profit margins for each tenant under different resource allocations. The learned model considers many factors of the environment such as SLA cost, client workload, infrastructure cost and action cost.

The resource allocation decision dynamically adjusts the resource allo-cations, based on the information of the learned model, of the different tenants in order to achieve the optimum profits

8_{Xiong et al.: Intelligent management of virtualized resources for database systems in}

A Declarative Approach for Achieving Elastic OLTP

Workloads.

The approach is based on defining of two main components:

The transaction classes required for the application.

The actual workload with references to the transaction classes.

Using this information, a formal model is defined to analyze the elas-ticity of the workload according to the specified transaction classes. The approach of is more consumer-centric as it enables the software ap-plications to declaratively define their scaling in and out rules according to specific application requirements and policies.

E-Store.

E-Store is an elastic partitioning framework for distributed OLTP DBMSs. E-Store automatically scales resources in response to demand spikes, periodic events, and gradual changes in an application’s workload. E-Store addresses localized bottlenecks through a two-tier data place-ment strategy: cold data is distributed in large chunks, while smaller ranges of hot tuples are assigned explicitly to individual nodes.

10_{Rebecca Taft et al.: EStore: Fine-Grained Elastic Partitioning for Distributed}

Challenge 2: Live Migration

Live migration is an important technique that provides extreme versa-tility for management of cloud resources by allowing applications to be transparently moved across physical machines with a consistent state. It can be used to improve compliance with tenant’s SLA by migrating the tenant with excessive workload to a less loaded server. Thus, it is a main tool for achieving elasticity and dynamic provisioning.

It can be also used to ensure availability by migrating tenants to other servers when the host server is planned to go down for maintenance.

Challenges of Live Migration

Live migration is a resource-intensive operation and can come at a price of degraded service performance during migration due the overhead caused by the extra CPU cycles which are consumed on both of the source and the destination servers in addition to the extra amount of network bandwidth which is consumed for the transmission process. The performance of a migration process is often measured by two main metrics:

The down time metric which represents the duration when the

appli-cation is completely stopped and its appliappli-cation’s service is entirely un-available.

Themigration timewhich represents the total time for all the involved

Live Migration Strategies

The system stops serving updates for the database, takes a snapshot of the database to be moved, moving and loading the data onto the new server, and finally restarting the service operations at the destination. This technique incurs a long service interruption and down time where the length of this period is proportional to the database size. However, it is more efficient in terms of minimizing the amount of data transferred and the total migration time.

Iterative State Replication

The source checkpoints the tenant’s database and starts migrating the checkpoint to the destination, while it continues serving requests. The destination loads the checkpoint, the source maintains the differen-tial changes which are iteratively copied until the amount of change to be transferred is small enough or a maximum iteration count is reached. At this point, a final stop and copy is performed.

This technique incurs a small down time. However, it requires higher consumption of the computing resources due to the overhead incurs during the longer total migration time.

Live Migration Techniques: Zephyr

Zephyr is a technique that have been proposed to efficiently migrate a live database in a shared nothing transactional database architecture. It minimizes the down time for the database being migrated by in-troducing a synchronized dual mode that allows both the source and destination to simultaneously execute transactions.

The migration process starts with the transfer of the tenant’s metadata to the destination which can then start serving new transactions while the source completes the transactions that were active when migration started.

During this time, a methodology of on-demand pull and asynchronous push of data is followed where the source node, initially, owns the read/write access to all pages at the start and the destination nodes acquire the read/write page ownership on-demand as transactions at the destination access those pages.

11

Elmore et al.: Zephyr: live migration in shared nothing databases for elastic cloud platforms. In SIGMOD 2011

Live Migration Techniques: Zephyr

A lightweight synchronization between the source and the destination to guarantees the serializability of transaction execution.

Once the source node completes execution of all active transactions, migration completes with the ownership transfer of all database pages owned by the source to the destination node.

Zephyr can guarantee no service unavailability and few or no aborted transactions.

Live Migration Techniques: Albatross

A very similar approach to Zephyr. However, it is more focused on a

shared process multitenant database environment where the persistent database image is stored in a network attached storage.

Since the storage is shared, Albatross is more focused on optimizing the migration process by copying the cache during migration such that the destination starts with a warm cache and thus it can minimize any impact on transaction latency after migration.

12_{Das et al.: Albatross: lightweight elasticity in shared storage databases for the cloud}

Live Migration Techniques: Slacker

In general, the migration process adds an extra overhead on both of the source and target servers.

Slacker is an iterative state replication technique which tries to min-imize the performance impact of the migration process on both the migrating tenant and collocated tenants, on the source and destination

servers, by leveraging migration slack (resources that can be used for

migration without excessively impacting performance latency).

Since the majority of the resource cost in migration is from reading, writing, or sending a large amount of data, Slacker control the effect of the migration process by controlling the upper bound rate of data transfer to the point at which performance latency steadily increases indicating that the database is overloaded and cannot maintain both its workloads and migrations.

Live Migration Techniques: Slacker

Slacker applies aPID(Proportional-Integral-Derivative) controlling mech-anism that allows the ability to automatically detect and exploit the available migration slack of computing resources in real time according to the dynamics of the executed workloads on both of the source and destination servers.

Live Migration: Open Challenges

There is a clear tradeoff between the migration time, the size of the

database and the amount of update transactions in the workload which are executed during the migration process.

Although various techniques have been proposed to tackle the challenge

of how to migrate, very little attention from the proposed database

migration mechanisms has been given to other important aspects such

as when to migrate.

In amultitenancyenvironment, the challenges of decidingwhichtenant

to migrate andwhere(to which server) this tenant should be migrated

to remain open issues for further investigation and careful consideration. Making use of database partitioning for improving the performance of live migration.

Challenge 3: Data Replication and Consistency

Management

Statelessservices are easy to scale in the cloud since any new replicas can operate completely independently of other instances.

In contrast, scaling statefulservices, such as a database system, needs

to guarantee a consistent view of the system for users of the service. The cost of maintaining several database replicas that are always strongly consistent is very high.

A common feature of the NoSQL and DaaS cloud offerings is the cre-ation and management of multiple replicas (usually 3) of the stored data (on the same data center) while a replication architecture is run-ning behind-the-scenes to enable automatic failover management and ensure high availability of the service.

In general, replicating for performance differs significantly from replicat-ing for availability or fault tolerance in terms of the degree of replication and the need for supporting weak consistency when scalability is the motivating factor for replication.

Experimental Evaluation of Data Replication in Cloud

Environments

The results of the study show that:

The performance variation of the dynamically allocated virtual ma-chines is an inevitable issue that needs to be considered when deploying database in the cloud.

Different configurations of geographic locations can noticeably affect the end-to-end throughput.

As the application workload increases, the replication delay increases. As the number of database replicas increases, the replication delay decreases.

The replication delay showed to be more affected by the workload increase than the configurations of the geographic location of the database replicas.

Consistency-Performance-Availability Trade-offs

15_{Doug Terry: Replicated data consistency explained through baseball. In CACM}

Consistency Rationing

A dynamic consistency strategy for reducing the consistency require-ments when possible and raise them when it matters.

The adaptation is driven by a cost model and different strategies that dictate how the system should behave.

the data items into three categories and treat each category differently depending on the consistency level provided.

TheAcategory represents data items for which we need to ensure strong consistency guarantees as any consistency violation would result in large penalty costs.

TheCcategory represents data items that can be treated using session consistency as temporary inconsistency is acceptable.

TheBcategory comprises all the data items where the consistency re-quirements vary over time depending on the actual availability of an item.

Adaptive Replication Controller

A framework that enables keeping several replicas of the database, on virtualized database servers, in different data centers (with potentially different geographic locations).

The framework provides the software applications with flexible mech-anisms for specifying different levels of service level agreements (SLA) of data freshness for the database replicas.

The framework allows specifying an SLA of data freshness for each database replica and continuously monitor the replication delay of each replica so that once a replica violates its defined SLA, the framework automatically injects new replica at the closest geographic location in order to balance the workload and re-satisfy the defined SLA.

Geo-replicated Data Management

In general, data replication across different data centers is expensive. Synchronous wide-area replication mechanisms are considered to be unfeasible to achieve strong consistency requirements.

Many solutions either rely on asynchronous replication mechanism and weaker consistency guarantees.

Geo-replicated Data Management: PNUTS (Yahoo’s

NoSQL data store)

PNUTS is one of the earliest systems to natively support geographic replication using asynchronous replication mechanism and publish/subscribe message exchange protocol.

It uses a per-record selective replication mechanism by designating one copy of a record as the master and directing all updates of the record to its master copy.

In this record-level mastering mechanism, mastership is assigned on a record-by-record basis, and different records in the same table can be mastered in different clusters.

Each record maintains a metadata field that stores the identity of the current master.

Geo-replicated Data Management: PNUTS (Yahoo’s

NoSQL data store)

When a replica receives an update request, it first reads the record to determine if it is the master, and if not, to which replica to forward the request to.

The mastership of a record can migrate between replicas according to its access pattern.

All updates are propagated to non-master replicas by publishing them to the message broker and once the update is published, the system treats the transaction as committed.

A master publishes its updates to a single broker, and thus updates are delivered to replicas in commit order.

Geo-replicated Data Management: Google Megastore

A system that has been presented as a scalable and highly available datastore which is designed to meet the storage requirements of large scale interactive Internet services.

It relies on the Paxos protocol for achieving synchronous wide area replication.

Megastore’s replication mechanism provides a single, consistent view of the data stored in its underlying database replicas.

Megastore replication semantics is done on entity groupbasis, a priori

grouping of data for fast operations, basis by synchronously replicating the group’s transaction log to a quorum of replicas.

It uses a write-ahead log replication mechanism over a group of sym-metric peers where any node can initiate reads and writes.

Each log append blocks on acknowledgments from a majority of repli-cas, and replicas in the minority catch up as they are able.

19

Challenge 4: SLA management

In practice, resource management and SLA guarantee falls into two

layers: the cloud service providers and the cloud consumers (users of

cloud services).

The cloud service provider is responsible for the efficient utilization of the physical resources and guarantee their availability for their cus-tomers (cloud consumers).

The cloud consumers are responsible for the efficient utilization of their allocated resources in order to satisfy the SLA of their customers (ap-plication end users) and achieve their business goals.

The state-of-the-art cloud databases do not allow the specification of SLA metrics at the application nor at the end-user level.

SLA management

In practice, cloud service providers guarantee only the availability (up-time guarantees), but not the performance, of their services.

In addition, sometimes the granularity of the uptime guarantees is also weak. For example, the uptime guarantees of Amazon EC2 is on a per data center basis where a data center is considered to be unavailable if a customer can not access any of its instances or can not launch replacement instances for a contiguous interval of five minutes.

In practice, traditional cloud monitoring technologies (e.g. Amazon

CloudWatch) focus on low-level computing resources (e.g. CPU speed, CPU utilization, I/O disk speed).

In general, translating the SLO of software application to the thresholds of utilization for low-level computing resources is a very challenging task and is usually done in an ad-hoc manner due to the complexity and dynamism inherent in the interaction between the different tiers

SLA management

Cloud service providers do not automatically detect SLA violation and leave the burden of providing the violation proof on the customer. In the multi-tenancy environment of DaaS, it is an important goal for DaaS providers to promise high performance to their tenants. However, this goal normally conflicts with another goal of minimizing the overall running servers and thus operating costs by tenant consolidation. In general, increasing the degree of multi-tenancy (number of tenants per server) is normally expected to decrease per-tenant allocated re-sources and thus performance, but on the other side, it also reduces the overall operating cost for the DaaS provider and vice versa. It is necessary, but challenging for the DaaS providers to balance be-tween the performance that they can deliver to their tenants and the data center’s operating costs.

Provider-Centric SLA Management

iCBS20 is a cost-aware query scheduling algorithm that takes the query

costs derived from the service level agreements (SLA) between the service provider and its customers (in terms of response time) into account to make cost-aware scheduling decisions that aims to minimize the total expected cost.

SLA-tree21 is another approach that efficiently support profit-oriented decision making of query scheduling. It uses the information about the buffered queries which are waiting to be executed in addition to the service level agreement (SLA) for each query that indicates the different profits for the query for varying query response times and

provides support for the answering of certain profit-oriented ”what if”

type of questions.

20

Chi et al.: ICBS: incremental cost-based scheduling under piecewise linear SLAs. PVLDB 4(9), 2011

Multitenant SLA Management

A framework that takes as input:

The tenant’s workloads. The tenant’s performance SLA.

The server hardware that is available to the DaaS provider.

The framework produces server characterizing models that can be used to provide constraints into an optimization module.

By solving this optimization problem, the framework provides a cost-effective hardware provisioning policy and a tenant scheduling policy on each hardware resource.

The main limitation of this approach is that the input information of the tenant workloads is not always easy to specify and model accurately.

22

SLA Management: Open Challenges

On one hand, it may be necessary, in the future, for cloud providers to offer performance based SLAs for their services with a tiered pricing model, and charge a premium for guaranteed performance.

On the other hand, it is a very challenging goal to delegate the man-agement of the fine-granular SLA requirements of the consumer ap-plications to the side of the cloud service provider due to the wide heterogeneity in the workload characteristics, details and granularity of SLA requirements, and cost management objectives of the very large number of consumer applications (tenants) that can be simultaneously running in a cloud environment.

It is an important requirement for cloud service providers to enable the cloud consumers with a set of facilities, tools and framework that ease their job of achieving this goal effectively.

Challenge 5: Transaction Support

In general, the expertise gained from building distributed database sys-tems by researchers and practitioners have shown that supporting dis-tributed transactions does not allow building scalable and available system.

To satisfy the scalability requirements of large scale internet services, many systems have sacrificed the ability to support distributed trans-actions.

Most of the NoSQL systems supports atomic access only at the gran-ularity of single keys. This design choice allows these systems to hori-zontally partition the tables, without worrying about the need for dis-tributed synchronization and transaction support.

Transaction Support: Microsoft SQL Azure Database

SQL Azure supports the relational data model and ACID transactions. However, it requires manual data partitioning and does not support dis-tributed transactions or queries across multiple data partitions located in different servers.

The database size is constrained to fit on a single node. For larger data sets, an application needs to partition the data among different database instances.

Transaction Support: G-Store

A data store which provides transactional multi key access guarantees over non-overlapping groups of keys using a key-value store.

The main idea of GStore is the Key Group abstraction that defines

a relationship between a group of keys and represents the granule for on-demand transactional access.

This abstraction allows the Key Grouping protocol to collocate control for the keys in the group to allow efficient access to the group of keys. The Key Grouping protocol enables the transfer of ownership for all keys in a group to a single node which then efficiently executes the operations on the Key Group.

24_{Das et al.: G-Store: a scalable data store for transactional multi key access in the}

Transaction Support: G-Store

At any instance of time, each key can only belong to a single group and the Key Group abstraction does not define a relationship between two groups. Thus, groups are guaranteed to be independent of each other and the transactions on a group guarantee consistency only within the confines of a group.

The implementation of the transaction manager component does not require any distributed synchronization and is similar to the transaction manager of any single node relational database management systems.

The transaction notion is limited to smaller related logical entities (key groups).

Transaction Support: Google Megastore

Megastore implements a transactional record manager on top of the BigTable data store and provides transaction support across multiple data items where programmers have to manually link data items into hierarchical groups and each transaction can only access a single group.

Megastore partitions the data into a collection ofentity groups, a priori

user-defined grouping of data for fast operations, where each group is independently and synchronously replicated over a wide area.

Megastore tables are either entity group root tables or child tables. Each child table must declare a single distinguished foreign key ref-erencing a root table. Thus, each child entity references a particular entity in its root table (called the root entity).

Entities within an entity group are mutated with single-phase ACID transactions (for which the commit record is replicated via Paxos). Operations across entity groups could rely on expensive two-phase com-mit operations.

Transaction Support: Accordion

Accordion, a dynamic data placement system for partition-based DBM-Ses that support ACID transactions.

Accordion explicitly considers the affinity between partitions, which in-dicates the frequency in which they are accessed together by the same transactions.

Accordion estimates the capacity of a server by explicitly considering the impact of distributed transactions and affinity on the maximum throughput of the server.

Accordion then integrates this estimation in a mixed-integer linear pro-gram to explore the space of possible configurations and decide whether to scale out.

Challenge 6: Benchmarking

In principle, benchmarks need to play an effective role in empowering cloud users to make better decisions regarding choosing the adequate systems and technologies that suit their application’s requirements. It is important that cloud users become able to paint a comprehensive picture of the relationship between the capabilities of the different type of cloud database services, the application characteristics and work-loads, and the geographical distribution of the application clients and the underlying database replicas.

Benchmarking: AppScale

An open source implementation26 of the Google App Engine (GAE)

Datastore which unifies access to a wide range of open source dis-tributed database technologies.

It has been used for conducting an evaluation of the performance

char-acteristics of several NoSQL systems including: HBase, Hypertable,

Cassandra,Voldemort andMongoDB.

26

http://www.appscale.com/

Yahoo! Cloud Serving Benchmark (YCSB)

The benchmark28 consists of a package of workloads with different

characteristics (e.g. read-heavy workloads, write-heavy workloads, scan workloads, etc).

The implementation of the YCSB benchmark has been used for

evalu-ating four systems: Cassandra,HBase,PNUTS, and a simplesharded

MySQLin terms of their performance and elasticity characteristics.

28_{https://github.com/brianfrankcooper/YCSB} 29

Benchmarking: YCSB++

YCSB benchmark extension30 supports complex features such as:

Multi-tester coordination for increased load. Eventual consistency measurement.

Multi-phase workloads to quantify the consequences of work deferment and the benefits of anticipatory configuration optimization such as B-tree pre-splitting or bulk loading.

30

http://www.pdl.cs.cmu.edu/ycsb++/index.shtml

Benchmarking: Consistency Benchmarking

A benchmark for quantifying the consistency guarantees of cloud-hosted storage systems.

The architecture of benchmark framework consists of the four main

components: workload Generator, tenant simulator, failure injector

andconsistency measurement component.

Evaluate the effects of geo-replication and different workloads on two popular NoSQL systems: Cassandra and MongoDB.

32_{Bermbach et al.: Towards Comprehensive Measurement of Consistency Guarantees}

Benchmarking: Open Challenges

Not enough standard benchmarks.

No comprehensive assessments and measurements for the different as-pects of cloud database services including:

The performance (e.g. response time, throughput) Scalability

Elasticity

Consistency guarantees Replication schemes

Challenges Summary

Research Aspect Related Factors Open Research Challenges

Elasticity Man-agement

- Application workload - SLA Satisfaction - Monetary Cost - Side of Control (Provider or Consumer)

- Designing accurate models for characterizing and predicting Internet scale application workloads.

- Desiging flexible dynamic provisioning mechanisms that carefully con-sider the target consumer application SLA and the target monetary costs.

- Enabling the consumer applications with powerful and flexible tools (admission controllers) to declaratively define and control their elas-ticity policies. Data Repli-cation and Consistency Management - CAP Theorem - Levels of Consistency Guarantee - Replica Locations

- Designing adaptable consistency mechanisms that can be flexibly configured on the runtime according to the application context. - Desiging efficient data replication and consistency management pro-tocols across different data centers in the cloud environment. - Further understanding to the practical limits of the CAP theorem.

Live Migration - Down Time - Migration Time - SLA Effect

- Triggering of Migration Need (When to Migerate?)

- Optimizing the down time and migration time metrics of the live migration techniques.

- Minimizing the performance effect and SLA degradation of the co-located tenants during the migration process.

- Desiging partitioning-aware live database migration techniques. - Desiging intelligent schedulers for the activities of the migration pro-cesses.

- Desiging intelligent techniques for deciding the the optimal source and destination tenants and servers with aim of optimizing the over-all system performance and the overover-all utilization of the computing resources.

Challenges Summary

Research Aspect Related Factors Open Research Challenges

SLA Manage-ment

- Side of Control - SLA Granularity - Monetary Cost

- Designing efficient mechanisms for monitoring and timely detecting SLA violations in cloud environments.

- Providing fine-granular SLA guarantess for cloud hosting database services.

- Desiging cost-aware SLA managemnt techniques.

- Enabling the consumer applications with flexible mechanisms to declaratively define, monitor and control their SLA requirements.

Transaction Support

- Granularity of Atomicity - Distributed Transactions - Performance

- Providing efficient multi row atomicity guarantees on NoSQL sys-tems.

- Desiging intelligent workload-aware and trasnaction-aware database partitioning mechanisms for cloud-hosted databases.

- Providing scalable transactional guarantees over multiple partitions for distributed database (across different data centers) in cloud envi-ronments.

Conclusions

NoSQL systems are viable solutions for applications that require scal-able data stores which can easily scale out over multiple servers and support flexible data model and storage scheme.

The access pattern of applications using NoSQL systems should not require much join operations and can work with limited transaction support and weaker consistency guarantees.

In general, NoSQL systems are recommended for newly developed ap-plications but not for migrating existing apap-plications which are written on top of traditional relational database systems.

With the wide options and variety of currently available NoSQL sys-tems, software developers need to well understand the requirements of their application to choose the NoSQL system with adequate design decisions of their applications.

Conclusions

DaaS are recommended for software applications which are built on

top of relational databases. They can be easily migrated to cloud

servers and alleviate the need to purchase expensive hardware, deal with software upgrades and hire professionals for administrative and maintenance tasks.

The applications relying on DaaS technology should have the ability to accurately predict their application workloads and provision the ad-equate resources that can achiever their performance requirements. Virtualized database servers are recommended for applications which require to leverage the full elasticity and scalability promises of cloud services and need to have full control on the performance of their appli-cations. However, these application need to build and configure their admission control for managing the database tier of their applications. There are still many open challenges and research opportunities for im-proving the state-of-the-art of hosting database on cloud environments.

Publications

Liang Zhao, Sherif Sakr and Anna Liu. Consumer-Centric SLA Manager for Cloud-Hosted

Databases. ACM International Conference on Information and Knowledge Management

(CIKM), 2013.

David Bermbach, Liang Zhao and Sherif Sakr. Towards Comprehensive Measurement of

Consistency Guarantees for Cloud-Hosted Data Storage Services. 5th TPC Technology

Conference on Performance Evaluation and Benchmarking (TPCTC), 2013.

Sherif Sakr. Cloud-Hosted Databases: Technologies, Challenges and Opportunities. Cluser Computing, Springer, 2013.

Sherif Sakr and Anna Liu.Is Your Cloud-Hosted Database Truly Elastic?. IEEE 9th World Congress on Services (IEEE SERVICES), 2013.

Liang Zhao, Sherif Sakr and Anna Liu. A Framework for Consumer-Centric SLA

Manage-ment of Cloud-Hosted Databases. IEEE Transactions on Service Computing (IEEE TSC),

2013.

Liang Zhao, Sherif Sakr and Anna Liu. Application-Managed Replication Controller for

Cloud-Hosted Databases. IEEE CLOUD, 2012.

Sherif Sakr and Anna Liu. SLA-Based and Consumer-Centric Dynamic Provisioning for

Publications

Liang Zhao, Sherif Sakr, Alan Fekete, Hiroshi Wada and Anna Liu. Application-Managed

Database Replication on Virtualized Cloud Environments. ICDE Workshops, 2012.

Basem Suleiman, Sherif Sakr, Ross Jeffery and Anna Liu.On Understanding the Economics

and Elasticity Challenges of Deploying Business Applications on Public Cloud Infrastructure.

Journal of Internet Services and Applications (JISA), Springer, 2011.

Sherif Sakr, Liang Zhao, Hiroshi Wada and Anna Liu. CloudDB AutoAdmin: Towards a

Truly Elastic Cloud-Based Data Store. ICWS, 2011.

Liang Zhao, Sherif Sakr and Anna Liu. On the Spectrum of Web Scale Data Management. Book Chapter in Cloud Computing: Methodology, System, and Applications, CRC, Taylor and Francis group, 2011.

Hiroshi Wada, Alan Fekete, Liang Zhao, Kevin Lee and Anna Liu. Data Consistency

Properties and the Trade-offs in Commercial Cloud Storage: the Consumers’ Perspective.

Fifth Biennial Conference on Innovative Data Systems Research (CIDR), 2011.

Sherif Sakr, Anna Liu, Daniel Batista, Mohamed Alomari. A Survey of Large Scale Data

Management Approaches in Cloud Environments. IEEE Communications Surveys and