• No results found

Lessons Learnt from the Development of GIS Application on Azure Cloud Platform


Academic year: 2021

Share "Lessons Learnt from the Development of GIS Application on Azure Cloud Platform"


Loading.... (view fulltext now)

Full text


Lessons Learnt from the Development of GIS

Application on Azure Cloud Platform

Dinesh Agarwal and Sushil K. Prasad


Department of Computer Science

Georgia State University

Atlanta - 30303, USA

Email: dagarwal2, sprasad@gsu.edu

Abstract—Spatial overlay processing is a widely used

compute-intensive GIS application that involves aggregation of two or more layers of maps to facilitate intelligent querying on the collocated output data. When large GIS data sets are represented in polyg-onal (vector) form, spatial analysis runs for extended periods of time, which is undesirable for time-sensitive applications such as emergency response. We have, for the first time, created an

open-architecture-based system namedCrayons for Azure cloud

platform using state-of-the-art techniques. During the course of

development of Crayons system, we faced numerous challenges

and gained invaluable insights into Azure cloud platform, which are presented in detail in this paper. The challenges range from limitations of cloud storage and computational services to the choices of tools and technologies used for high performance computing (HPC) application design. We report our findings to provide concrete guidelines to an eScience developer for 1) choice of persistent data storage mechanism, 2) data structure representation, 3) communication and synchronization among nodes, 4) building robust failsafe applications, and 5) optimal cost-effective utilization of resources. Our insights into each challenge faced, the solution to overcome it, and the discussion on the lessons learnt from each challenge can be of help to eScience developers starting application development on Azure and possibly other cloud platforms.

Index Terms—Engineering HPC applications on Azure cloud,

Scientific applications on cloud, Cloud computing, Parallel over-lay operations, Data-intensive applications, Irregular computa-tions


Researchers in Geographic information systems and science (GIS) mostly work with large volumes of geospatial data. More often than not, this data is irregular in nature, which makes its processing a complex and compute-intensive task. Additionally, the current state-of-the-art is desktop-based pro-cessing systems that do not even utilize threaded versions of fundamental overlay processing routines. This lack of ability to utilize parallelism results in spatial data processing modules running for extended periods of time ranging from a few hours to days and even weeks in some cases.

Cloud computing allows researchers to experiment with large-scale systems without worrying about the management of resources [1]. Researchers working with scientific applications, that require large scale storage and computing resources, have started relying on cloud platforms [2]–[4]. The ease of availing cloud resources as various utility based services —‘‘Platform

1This work is partially supported by NSF CCF 1048200 project.

as a service’’ (PaaS), ‘‘Infrastructure as a service’’ (IaaS), and ‘‘Software as a service’’ (SaaS), etc.— has enabled its proliferation in the scientific research community. The emerg-ing cloud platforms, such as Microsoft’s Azure — with their potential for large scale computing and storage capabilities, easy accessibility by common users and scientists, on demand availability, easy maintenance, sustainability, and portability — have the promise to be the platform of choice for these scientific applications.

In spite of facilitating easy access to large scale compu-tational and storage resources, cloud platform proves to be challenging to develop (or port) scientific applications for it [5]–[7]. Cloud computing is a new paradigm for research and application development and thus requires training the researchers and eScience developers on cloud programming. Although cloud service providers offer developers familiar tools to program for cloud platform, there are still barriers of complexities that hinder the productivity of an eScience developer [8], [9] untrained in programming for cloud plat-form.

Literature reports some work identifying the challenges when eScience developers interact with Azure cloud platform. In [10] we give a generic framework along with compre-hensive micro benchmarks of Azure cloud platform’s storage services to help developers make informed choices in terms of application design. Lu et al., [8] detail the challenges faced during porting of a well known sequence alignment application

BLAST [11]. Ekanayake et al., [9] have identified the lack of Azure platform’s capability to provide efficient communication and synchronization framework among processors. Varia et al., [12] provide a list of best practices for application development on cloud platform.

There is a wide range of scientific applications that deal with large-scale data and compute intensive processing on cloud platform [2]–[7]. There are, however, only a few projects from Geosciences that have been designed specially for cloud platform. Most relevant among these include ModisAzure project for download, re-projection, and reduction of satellite imagery [5], [7], [13], and Smart Sensors and Data Fusion applications project for ocean observation [14].

We have created, for the first time, a parallel system named

Crayons[15], an end-to-end GIS polygonal overlay processing application on Azure cloud platform. During the course of developingCrayons, we faced numerous challenges presented


Step 1 Step 4 Web role Blob Container Put blob Step 2 Worker n Worker .. Worker 1 Step 3 Worker n Worker .. Worker 1

put file names get file names




get blob IDs task completion



mes-sage count

flush blocks togmlfile



put blobs

put blobs Read User Selection

Commit Output File

Input Queue

Blob Blob

Blob Blob

Download and Parse Files

Create Intersection Graph

Partition Graph

Task Pool Queue Termination Indicator Queue

Store Output

Process task

Check For Work

Fig. 1: Crayons’ architecture

by idiosyncrasies of Azure cloud platform. Each challenge provided us with valuable insights into Azure cloud platform. In this paper, we first briefly describe the Crayons system along with our experimental results to set up the stage for pointing bottlenecks and challenges at different stages of meta-algorithm utilized to implement theCrayonssystem. We then detail the individual challenges, provide a solution, and summarize the lessons learnt from the respective challenge. Although our implementation is based on Azure cloud plat-form, many of the challenges are generic and thus can apply to other cloud platforms as well. Specifically, our technical contributions are:

• providing guidelines for selecting persistent data storage

on cloud platform,

• suggesting an efficient method for data structure

repre-sentation on cloud platform,

• identifying and solving the problem of synchronizing and

communicating among distributed processors,

• providing guidelines to failsafe a scientific application,


• using a real world application to identify the best

prac-tices for cost-effective utilization of cloud resources. We have organized the paper into the following sections. Section II discusses the preliminaries related to our work. Section III summarizes the architecture and performance of

Crayons system. We discuss the implementations issues and lessons learnt in Section IV, and Section V concludes this paper.


A. Windows Azure platform

Windows Azure cloud platform provides computing and storage services where resources are hosted in Microsoft data centers. Windows Azure cloud platform is made up of three architectural components 1) Compute service, 2) Storage service, and 3) Azure Fabric. Compute service is responsible for running applications and Storage service manages data in Azure cloud platform. Azure Fabric manages both Compute

and Storage services. A Windows application on Azure cloud platform consists ofweb roleandworker roleinstances.

A web role presents the user interface and thus runs on a virtual machine that supports a subset of ASP.NET and Windows Communication Foundation (WCF) technologies.

Worker roleinstances are the processing entities that perform background processing in a scientific application. There could be multiple instances of each role as well based on the demand of the application. For instance, an application can have one set ofworker roleinstances to pre-process the input data and other set ofworker roleinstances to perform core computation on the pre-processed data.

B. Spatial overlay operations

Crayons system performs overlay operations on spatial vector-data. Geographic features are normally captured by a specific theme and thus can be organized as a series of thematic layers that can be overlaid for querying the aggre-gated data. For instance, after overlaying a map layer that represents transportation network of a city with a map layer that represents the tourist attractions in the city, the resultant map layer can answer queries such as finding out which transportation medium connects two tourist attractions.


Figure 1 shows the architectural diagram of Crayons with distributed dynamic load balancing. The entire workflow for this architecture is divided into four steps.

The first step starts with the user selecting two maps (GML files), base layer and overlay layer, to be overlaid along with the type of overlay operation to be performed. Web role simply forwards this information to worker roles through the communication queue shown as Input Queue in Figure 1.

The worker roles continuously check for work in the Input Queue. As soon as the information is located as a message in the queue, worker roles read the message, extract the encapsulated information, and start downloading the relevant maps from the cloud storage shown as the blob (file) container in Figure 1. Once the files have been downloaded, worker role


instances start parsing those files and store them as collection of spatial polygonal objects.

Polygons are converted to their GML representations (seri-alization) to store as blobs in the Blob storage. One polygon from base layer and all intersecting polygons from overlay layer constitute one independent task.

Each task is identified by a unique ID assigned to it when a worker role stores it in the Blob storage. This ID is communicated as a message to a queue named Task Pool Queue as shown in Figure 1. Task Pool Queue serves as a shared task pool among workers and thus facilitates dynamic load balancing.

When a worker role instance reads a message from the task pool, it downloads the blob with ID stored in this message, converts the content of the downloaded blob to get the original base and overlay polygon objects back (deserialization), and performs the spatial overlay operation by passing a pair of base polygon and one overlay polygon at a time toGPClibrary for sequential processing. Once the worker role instances have finished processing, as the final step, the web role instance commits the output blob producing the output GML file.

A. Performance results

We have used two different sets of input GML files for benchmarking Crayons system. The smaller data set has a skewed load distribution. Some of the base layer polygons interact with a large number of overlay layer polygons, while many others do not interact with any overlay layer polygon. The large data set has a comparatively uniform load distribu-tion.

Figure 2 is a representation of maximum time taken by individual Crayons modules for both data sets. Input file handling subprocess encapsulates file download and parsing. Task creation is the process of creating the neighborhood-relationship-graph and creating independent tasks. Task storing process involves storing the tasks in the Blob storage and putting the blob ID on the Task Pool Queue.

Figure 2(b) shows how Crayons behaves with larger data sets. Since the load is comparatively uniform for this data set,

Crayonsshows much better performance. The relative end-to-end speedup ofCrayonsgoes to more than 40x for larger data set (Figure 2(b)), from 10x for smaller data set (Figure 2(a)).


HPC program development over the still-emerging Azure platform is very difficult and tedious even for experienced parallel programmers and algorithm developers. Our primary contribution is that Crayons system is an “engineering” feat to create an Azure cloud based overlay processing system with an open architecture. During the course of engineering

Crayons system, we ran into multiple issues caused variously by the development tools and platforms used, and learnt some invaluable lessons.

In this section we detail those issues and lessons and suggest our solutions to provide a roadmap for readers interested in development on Azure platform. We believe that this discus-sion will help readers make informed design choices for their respective cloud applications.

(a) Small data set

(b) Large data set

Fig. 2: Crayons’ execution times for subprocess and end-to-end speedup

A. Organized storage vs. unrestrictive storage

Azure cloud platform provides Table storage and Blob storage for persistent data storage. Table storage can store data in an organized way so that it can be queried. This looks promising for GIS data and thus Table storage was the first choice for Crayons system. The idea was to store the GIS features (polygons and holes) in an Azure Table storage to enable easy spatial querying and static task assignment to workers virtually arranged in a 2D grid.

However, a table can only store entities (rows) that are a maximum of 1 MB in size. Since Crayons (and most of the Scientific applications) handles and stores tasks that are usually larger than 1 MB, Table storage was soon realized not to be a good fit in this case.

Table storage does not have a strict schema for tables, i.e., two separate rows in same table can have different schema both in terms of number of columns (properties) per row as well as type of those columns. We could have divided and stored GIS features into co-ordinates of each point by creating multiple columns. However, each entity in an Azure Table can only have a maximum of255properties making it difficult to store

a polygon with more than255 co-ordinates. It can be argued

that a GIS feature can be stored into multiple rows but we chose not to do that as it will increase the complexity of the


code without any necessary performance gains.

Therefore, we chose to proceed with Blob storage that can handle large blobs (files) of size up to 200 GB. A blob can either store up to 64 MB of data as a single entity (file) or for larger files the data can be divided into smaller blocks of size up to 4 MB. A Block blob can store a maximum of 50,000 such blocks, i.e., the maximum size of a Block blob can be as large as 200 GB. In our case, we used Block blobs to store input GML files, independent tasks, and output GML map.

Lesson: If the fundamental unit of an application is larger than 1 MB, Blob storage should be used for persistent storage needs rather than storing it in Table storage as multiple parts.

B. Blob storage: random vs. sequential access

There are two types of blobs in Windows Azure: Block blobs and Page blobs. Block blobs are used to store data that are to be accessed sequentially. The maximum size of a Block blob cannot exceed 200 GB.

The Page blob artifact, made of 512-bytes sized pages, was introduced to facilitate random read/write operations on blobs. Pages in a Page blob can be added at any location by specifying the page offset. The offset boundary should be divisible by 512-bytes page boundaries, and the total data that can be updated in one operation is 4 MB. A Page blob can store up to 1 TB of data.

Fig. 3: Page blob (random access) vs. Block blob (sequential access)

Figure 3 demonstrates the relative performance of Page blobs compared with Block blobs for uploading and down-loading blobs from cloud storage. It can be seen that Page blobs are superior for uploading data to a Page blob at random location, compared with adding blocks to a Block blob. The download operation demonstrates almost similar performance even though pages were accessed randomly while blocks were downloaded sequentially starting at the first block.

When multiple workers are trying to upload to a single blob, Page blobs provide better throughput than Block blobs [10]. This is because Page blobs support random read/write from/to pages in the blob, at the cost of the restriction that the page offset boundary should be divisible by 512-bytes page boundaries. For downloading an entire blob, both Page blobs and Block blobs perform almost equally. For partial data download from a blob, a Page blob is preferred when access can be random, and a Block blob is advised when the access is

going to be sequential. As the number of worker role instances increases, Block blobs perform better than Page blobs for blob download operations as shown in Figure 3.

Furthermore, the maximum size of a Page blob must be specified in advance during initialization, while a Block blob can grow up to 200 GB. In the case of Crayons system, the input data is uploaded by the user and hence the maximum input data size cannot be predicted in advance. Moreover, unlike Page blobs where we need to control the offset of a page to avoid overwriting of existing data, Block blobs allow us to add blocks in the output blob concurrently without managing the placement of a block manually. Therefore, we currently use Block blobs over Page blobs. However, since the allocation of work in Crayons system is based on the IDs of the worker role instance, we are currently experimenting with Page blobs to see if we can reduce the downloading time during Step 2 (see Figure 1) of the meta-algorithm by allowing each worker to only download the required partition of a blob.

Lesson: If an application creates a blob by adding partial blob content at a pre-specified location in a blob, Page blobs should be chosen, provided maximum file size can be pre-dicted. For general purpose file access, however, Block blobs are more convenient.

C. Representation and storage of data structures

Azure Blob storage can be used to store a polygon object from C# only after it has been represented as a byte array or a string of text. Microsoft C#.Net provides a serialization library that can take an object and convert it into a byte array or a string that can later be deserialized and restored back as the original object. Although serialization of objects significantly simplifies the distribution of objects among workers and web role instances, as objects can now be stored as Blobs, the serialization library stores object as a hierarchy of objects and adds a significant amount of metadata to ensure that the object can be created back (deserialized) from the serialized string and thus is a rather time consuming process. Since

Crayonsdeals with an enormously large number of polygons, this process costs a lot of chargeable compute hours as well as storage hours on Azure platform.

To alleviate this problem, we created an in-house serial-ization library that incorporates the information needed to be stored and converts an object to its GML representation. Since we have the liberty to know the structure of a polygon object beforehand, thanks to strict schema of GML data representation, we can easily parse it in a way to relate it to a flat string representation. For instance, we parse a polygon object, extract its co-ordinates and attribute information, and then store it as a long string delimited by a predefined symbol. To convert an entire independent task to a string, all the polygons in the task are converted to their respective GML representations and then concatenated in such a way that the polygon from the base layer is the first polygon. With this information the polygons can be converted to strings and vice versa.

Lesson:If the data structure is known a priori and does not include excessive references, it is best to manually convert it


to string in an application-specific manner than relying on the serialization library.

D. Queues without head or tail and message visibility timeout Crayons uses Queue storage service of Azure platform to communicate between the web role and the worker roles. Azure Queue storage differs from the traditional queue data structure as it lacks the ability to guarantee a FIFO operation. This lack of guarantee for FIFO operation can create issues if a queue is to be used to signal a special event.

For instance, in a common scenario, the producer and the consumers share the task pool. The producer keeps adding the tasks to the task pool and the workers consume the task from the task pool. In order for a consumer to know when to stop looking for work there are two possibilities (i) either the task pool is empty, or (ii) the last read message indicated the end of the work. However, in case of Crayons, the Task Pool Queue can be empty due to starvation, i.e., when there are more consumers than needed to process available work.

Moreover, if a producer puts a special end message in the task, it might be read before the previously generated messages for the tasks. This scenario is possible because Azure Queue storage does not guarantee FIFO behavior. The reason for this lack of guarantee is the fact that a message that was earlier read by a worker might reappear in a queue if it was not deleted permanently. Therefore, it is advisable to not reuse a queue for signaling special events through message based flags. Our solution was to employ a separate Termination Indicator Queue wherein all workers insert special messages for each task completed and web role keeps the count.

Additionally, Azure Queue storage imposes a limit on how long a message can stay alive in the queue. In Azure terminol-ogy, this time is known asvisibilityT imeoutof a message. It

used to be2hours prior to October 2011, but it was increased

to a week during October 2011 API update. It is left to the eScience developer to ensure that the producers never produce more work than can be finished within visibilityT imeout


The reason why visibilityT imeout matters is if the task

pool has so many tasks that it might take consumers longer thanvisibilityT imeoutperiod to process them, the messages

will disappear from the queue and hence the final result will be incorrect. We faced this issue during execution of Crayons

system with fewer processors. The end-to-end execution time for Crayons system with 1 worker role instance is more than six hours and hence some of the messages used to disappear from the Task Pool Queue before they were processed.

A solution to this problem can be what we have imple-mented in our design of Crayons system for Linux clusters [17]. In this case, the producer creates a limited number of tasks and assigns them to the available workers. After the workers finish the allocated task, only then the producer creates next set of tasks.

Lesson:If a message requires a sense of order, it should be communicated through a separate queue.

Fig. 4:Ratio of access times over a separate queue per worker to a single queue shared among workers

E. Shared queue vs. separate queues

As discussed previously in Section IV-D, queues are primar-ily used for communication among multiple role instances. Although Queue storage scales very well for concurrent access, queues perform even better when each worker is communicating with its own queue. In our experiments to evaluate Azure Queue storage, we found that queues can perform almost twice as good when there are large number of workers and each worker accesses its own queue compared to all of the workers accessing a shared queue.

Figure 4 shows the ratio of access times of Queue storage for up to 100 workers. For these experiments, workers inserted a total of 20K messages, divided equally among workers, in the Queue storage and then deleted them after reading (peeking). Figure 4 illustrates the ratio of time taken by workers when there was a separate queue per worker to time taken when each worker was accessing the same shared queue, for various queue operations. The time to access shared queue increases with increasing number of worker role instances, i.e., as the contention on the storage increases, the performance on shared queue gradually decreases. The ratio approaches 2-fold in the case of 96 workers [10]. In case of Crayons system, we currently employ a single shared queue shown as Task Pool Queue in Figure 1. We are planning to split the Task Pool Queue into multiple smaller local queues where each worker accesses its own queue as long as there is work, and in case of starvation it can steal work from other queues as well.

Lesson:When there are multiple clients that require access to a shared queue, it is beneficial to split the queue among multiple smaller queues, if allowable by the application.

F. Debugging and development platform vs. cloud platform

Cloud platforms do not provide an interactive debugging environment that most eScience developers are accustomed to. Therefore, a typical development cycle for cloud based appli-cations start with a local development and testing phase. Once the application behaves as expected in a local environment, it is then deployed to the cloud platform. Azure platform, similarly, provides developers with an Azure cloud simulator that can be installed on the development machine to test the application before deploying it to the cloud platform. However, we found


that the Azure cloud platform simulator does not necessarily replicate the local Windows OS and .Net platform accurately. In our case, the issue was absence of a standard Windows dynamic link library(dll)file named msvcrt100.dllon Azure

cloud platform. It was only after a few weeks of debugging that we were able to locate this issue and eventually solve it. The application started executing as expected once this dll file was manually packed inside the package that was deployed to the Azure cloud. In order to detect such errors, the best method is to login to the remote VM through remote desktop connection and check the local error logs at the respective VM.

Lesson:If an application works locally but not in the cloud, check for both native and non-native dependencies.

G. Significant resource allocation and deallocation times

One of the crucial challenges while working with cloud platforms is the time taken by resource allocation and deallo-cation process. When an applideallo-cation is deployed to the Azure cloud platform, the Windows Azure fabric controller prepares the web role and worker role instances requested by the appli-cation. Similarly, when the application configuration changes — to allocate additional instances or to deallocate some of the allocated instances — it is also handled by Windows Azure fabric controller. This process is a time consuming process compared to deploying an application to a traditional web server. The resource allocation and deallocation time varies with the number of instances requested by the application. Usually, web role instances take longer than worker role instances as web role instances require virtual machines with support for ASP.Net and Windows Communication Foundation (WCF) technologies.

Figure 5 shows the time taken by deployment of an applica-tion to Azure cloud platform. The time is measured from the Visual Studio integrated development environment’s tool to publish a Windows Azure application on cloud platform. The application has one instance of small web role and varying number of worker role instances. It can be clearly seen that the minimum time taken to deploy an application to Azure cloud platform is over 500 seconds. The deployment time does not show any pattern for fewer than 10 instances. However, after that the time keeps increasing with increasing number of instances. We only have a maximum quota of 100 instances for our account and therefore we cannot test the scalability beyond that, but it is clear that the deployment time is a function of number of instances.

In case ofCrayons, the low fidelity of Azure simulator and time consuming deployment process significantly hindered the productivity, as in order to test the application after every minor change it would take a long time to configure it on the cloud platform. The deployment timing has reduced now, after one year, compared to what it was during the beginning of year 2011 but it still is a major barrier to the productivity. Moreover, for an application in the production environment, on-demand configuration changes can add substantial delays to the overall processing time if not managed adequately by predicting the demand well in advance. Currently, there are

Fig. 5:Application deployment time for Azure cloud platform no easily implementable management APIs in Azure cloud platform that can provide a mechanism to allocate and deallo-cate resources in a manner similar to what traditional parallel programming paradigms provide to spawn or destroy multiple threads. As a result, we cannot deallocate idling processors without manual intervention resulting in significant amount of chargeable Azure compute hours.

Lesson: During the deployment, an application should be tested thoroughly on the simulator before deploying it to the cloud. For production scenarios, the application should be able to predict the demand of resources in near-future and request reconfiguration well ahead of time when those resources are required.

H. Parallel constructs and throttling of Azure cloud storage

Microsoft .Net platform supports multiple parallel direc-tives such as P arallel.F or loop among others to introduce

thread-level parallelism to applications. The concurrency con-trol mechanism underlying these constructs, however, is not aware of Azure platform’s limitations to handle concurrent requests. Azure storage can handle up to a maximum of 5000 transactions (entities/messages/blobs) per second for an account. Moreover, each of the storage services can handle up to 500 transactions per second. For instance, a Queue can handle only 500 messages per second [10]. These constructs can easily throttle the Azure storage services by sending more requests than allowed by Azure cloud. As a result the deployed application will throwunhandled exceptionswhich

are difficult to interpret and resolve. Additionally, continued storage exceptions can lead an instance to restart continuously resulting in Azure fabric controller starting a fresh instance to replace the current instance. In case of Crayons system, we initially had the web role create and store independent tasks into the Blob storage usingP arallel.F orEachconstruct

which led to this bug. Although we were able to catch the exception, the returned error message did not hint at throttling of Azure storage and hence it took us a long time to resolve this issue.

Our solution was to manually control the concurrency to not let the application issue more requests than that can be handled by Azure cloud platform. This was achieved by maintaining a thread pool with parameterized set of threads.


Lesson: Manually controlling the parallel constructs is essential to avoid throttling of Azure cloud storage services.

I. Synchronization among distributed processors

Azure cloud platform is primarily targeted for bag-of-task applications where processors can independently execute tasks. However, an application can require explicit synchro-nization while switching between tasks or modules within an application. Azure platform does not provide any API in the Azure software development kit that can facilitate a barrier like functionality.

However, a queue can be used as a shared memory resource to implement explicit synchronization among multiple worker role instances. Each worker can put a message on a designated queue that acts as a barrier. When the number of messages in the queue is equal to the number of workers, it is safe to assume that all workers have touched the barrier and hence all of them can cross it.

This synchronization mechanism is illustrated by us in [10]. What makes it interesting is that if a worker deletes the message(s) after crossing the barrier, those workers that have touched the barrier, but haven’t crossed it yet, will never be able to cross it. On the other hand, if the worker does not delete the message(s), the number of workers will never match the number of messages in the queue after first synchronization cycle. Therefore, it is required to maintain a variable that accounts for the messages left in the queue during previous synchronization phases. Moreover, since a large number of requests to get the message count can throttle the queue, each worker should wait for a brief moment before issuing the next request.

Lesson:If an application demands barrier synchronization, Queue storage can be treated as a global shared memory resource to facilitate barrier like functionality.

J. Role instance failures

Cloud platform abstracts the resource provisioning mech-anism to provide resources as a utility based service. These resources are distributed across multiple data centers scattered throughout the world. Each data center potentially consists of millions of machines and provides a multi-tenant environment. Furthermore, each physical machine can host multiple virtual machines (VM), each of which can run a separate application instance. It is inevitable that a small fraction of these machines fail during the course of execution of an application due to various reasons. It is strongly recommended that an application should have the provision for such failures.

Crayonsscales well with increasing number of worker roles as can be seen from Figure 2. However, it is possible that a worker role dies during either Step 2 or Step 3 of the meta algorithm. We created a version of Crayons that used static load balancing where each worker role would maintain the list of tasks locally, instead of storing them in the task pool as shown inCrayonsarchitecture in Figure 1. Therefore, even if other workers have finished their processing, they will have to wait for the worker that died to come back up again and finish its own work. Although in the best case this version

performs better than the current version based on shared Task Pool Queue, it is not safe to deploy that version to production environment.

Task pool queues (shared or local) are better vehicles for robustness. The visitbilityT imeout of a queue can be

exploited to make an application both failsafe and efficient. Messages in the queue behaving as a task pool should have a

visibilityT imeout set to be a little over the maximum time

that processing a task is expected to consume. For instance, if a worker role does not return the success message within

stipulated amount of time, the message (task) that this worker was working on should reappear in the task pool so that other workers can process it. On the other hand, duplication of output needs to be avoided.

Lesson: An application should be oblivious to state of a role instance. Azure queues can be a fault-tolerant mechanism to ensure eventual completion.

K. Load imbalance and worker idle time

Cloud computing platform offers utility based services where users are charged for the time their resources were active regardless of the actual processing that happened at that core. One major portion of wasted compute time in cloud based applications comes from idling of role instances. For instance, if the roles need to wait for the next job after finishing one job, this gap is an obvious wastage of compute time and hence lead to underutilization of resources. When we first started building Crayons system, we had a web role as a centralized producer of tasks. Even though we used an extra large virtual machine, with 8cores, as the web role instance,

it could not produce task fast enough to keep up with large number of hungry workers. Hence, Crayons system did not scale well beyond 16worker roles. After post-analysis of our

system, we realized that the task pool queue was always empty when there were more than 16 worker roles, i.e., as soon as a task was produced one of the idle workers will grab it and process it while others will idly wait for the next job.

Therefore, we delegated the process of task production to the worker roles themselves. Crayons became scalable for large number of worker role instances.

We are currently working on Crayons system to analyze further aspects of its architecture to gain insights into factors that contribute towards performance as well as running costs. We are currently working on optimizing the number of total storage transactions, reducing the storage requirement of in-dependent tasks, and creating a relation between number of polygonal features (input data sets) and required number of processors to facilitate automatic scaling ofCrayons.

Lesson: It is essential to perform a post-analysis of an application to ensure that there is always enough work for all processors.

L. Clipper library specific lessons

For polygon overlay operations, General Polygon Clipper (GPC) is a well-known open-source library. The developers of GPClibrary have made some design choices that limit its applicability in an application for spatial overlay processing.


First and foremost, the GPC library supports only four opera-tions - intersection, exclusive-or (XOR), union, and difference, while there is an extensive set of relationship such as Equals,

Crosses,Within,Contains,Disjoint,Touches, andOverlap, that a GIS scientist would like to test on the input data sets.

Secondly, although GPC library is capable of handling polygons that contain holes, the resulting polygon does not discriminate between a polygon contour and a hole. It is left to the user to compute that information by some means. Murta et al. [16] suggest that to associate holesH1,H2... Hn with

external contours E1, E2 ... Em, one can use the clipper to

compute the difference of the ith hole Hi and each external

contour Ej, for all j from 1 to m. Any difference which

gives an empty result indicates that hole ilies within external

contourj. This clearly addsO(mn)complexity wheremand n are the number of resultant polygons and holes.

Moreover, it does not maintain all of the attributes of polygons, such as polygon ID and dimension information among others. We had to store this information locally to copy it over to the resultant polygon after the library returns the output. When the output of GPC library was compared with that from state-of-the-art GIS solution ArcGIS, it was found that the output polygon boundaries had a little deviation. A small number of vertices (1-2%) were missing from the output too. However, the overall timing characteristics of theCrayons

system should hold for intersection and related operations.

GPClibrary is packaged withCrayonssystem as adynamic link library (dll)file and hence can be replaced with any other third party library, with same API signature and underlying data structure (polygon), without any change in the Crayons

system’s code. Due to such open-architecture, the applicability of Crayonssystem can be further extended to related applica-tions from GIS domain.

Lesson:Design a large system with an open architecture to enable easy replacement of third-party components.


In this paper, we have documented the details of lessons learnt from development of a fundamental GIS application on cloud platform. This work provides insight into a gamut of services offered by Azure cloud platform ranging from storage services to compute services. Crayons system can be seen as a good representative of a generic scientific application. We believe our experience will guide eScience developers in the right direction with their high performance computing (HPC) application design choices for cloud platform.Crayons

is an open-source software system2 facilitating the eScience developers and researchers to refer to our code for pragmatic solutions. We have also portedCrayonssystem on Linux clus-ter [17] to compare the performance of scientific applications on traditional HPC clusters against cloud platform.

Development of Crayons system has helped us identify a generic framework for HPC applications of cloud platform. As a next step toCrayonssystem, we are working on creating 2More details and code forCrayonssystem can be found at http://www. cs.gsu.edu/dimos/content/gis-vector-data-overlay-processing-azure-platform. html

a generic framework that will provide researchers and develop-ers with a skeleton to start building their scientific applications on cloud platform.


[1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, “Above the Clouds: A Berkeley View of Cloud Computing,” EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2009-28, Feb 2009. [Online]. Available: http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html [2] I. Foster, Y. Zhao, I. Raicu, and S. Lu, “Cloud Computing and Grid

Computing 360-Degree Compared,”ArXiv e-prints, vol. 901, Dec. 2009. [3] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, “Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility,” Future Gener. Comput. Syst., vol. 25, pp. 599–616, June 2009. [Online]. Available: http://dl.acm.org/citation.cfm?id=1528937.1529211

[4] J. J. Rehr, F. D. Vila, J. P. Gardner, L. Svec, and M. Prange, “Scientific computing in the cloud,”Computing in Science and Engineering, vol. 12, pp. 34–43, 2010.

[5] C. A. Lee, “A perspective on scientific cloud computing,” in Proceed-ings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC ’10. New York, NY, USA: ACM, 2010, pp. 451–459.

[6] G. Turcu, I. Foster, and S. Nestorov, “Reshaping text data for efficient processing on Amazon EC2,” inProceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC ’10. New York, NY, USA: ACM, 2010, pp. 435–444. [Online]. Available: http://doi.acm.org/10.1145/1851476.1851540 [7] A. Thakar and A. Szalay, “Migrating a (large) science database to

the cloud,” inProceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ser. HPDC ’10. New York, NY, USA: ACM, 2010, pp. 430–434. [Online]. Available: http://doi.acm.org/10.1145/1851476.1851539

[8] W. Lu, J. Jackson, J. Ekanayake, R. S. Barga, and N. Araujo, “Performing large science experiments on azure: Pitfalls and solutions,” in Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science, ser. CLOUDCOM ’10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 209–217. [Online]. Available: http://dx.doi.org/10.1109/CloudCom.2010.104 [9] J. Ekanayake, J. Jackson, W. Lu, R. Barga, and A. S. Balkir, “A scalable

communication runtime for clouds,” inProceedings of the 2011 IEEE 4th International Conference on Cloud Computing, ser. CLOUD ’11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 211–218. [Online]. Available: http://dx.doi.org/10.1109/CLOUD.2011.21 [10] D. Agarwal and S. K. Prasad, “Azurebench: Benchmarking the storage

services of the Azure Cloud Platform,” inIEEE International Parallel and Distributed Processing Symposium workshops, Shanghai, China, May 2012.

[11] S. F. Altschul, W. Gish, W. Miller, E. W. Myers, and D. J. Lipman, “Basic local alignment search tool.” Journal of molecular biology, vol. 215, no. 3, pp. 403–410, Oct. 1990. [Online]. Available: http://dx.doi.org/10.1006/jmbi.1990.9999

[12] J. Varia, Architecting for the Cloud: Best Practices, Jan. 2010. [Online]. Available: http://jineshvaria.s3.amazonaws.com/public/ cloudbestpractices-jvaria.pdf

[13] J. Li, M. Humphrey, D. Agarwal, K. Jackson, C. van Ingen, and Y. Ryu, “eScience in the cloud: A MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform,” in Parallel Distributed Processing (IPDPS), 2010 IEEE International Symposium on, april 2010, pp. 1 –10.

[14] J. R. Delaney and R. S. Barga,The Fourth Paradigm: Data Intensive Scientific Discovery. Microsoft Research,, 2009, ch. Observing the Oceans - A 2020 Vision for Ocean Science.

[15] D. Agarwal, S. Puri, X. He, and S. K. Prasad, “Cloud computing for fundamental spatial operations on polygonal gis data,” inCloud Futures 2012 - Hot Topics in Research and Education, Berkeley, California, United States, May 2012.

[16] A. Murta, “A general polygon clipping library,” http://www.cs.man.ac.uk/ toby/alan/software/gpc.html, 1997.

[17] D. Agarwal, S. Puri, X. He, and S. K. Prasad, “A system for GIS polygonal overlay computation on linux cluster - an experience and performance report,” in IEEE International Parallel and Distributed Processing Symposium workshops, Shanghai, China, May 2012.


Related documents

The rural areas need focus as they are much behind urban areas in a number of socioeconomic parameters like infrastructure, access to health and education services, gender

Hybrid of comprehensive learning particle swarm optimization and SQP algorithm for large scale economic load dispatch optimization of power system.. Science China

Sogeti suggested that Wellmark host the data on Windows Azure, which serves as the Microsoft cloud services development, hosting, and management environment.. Windows Azure

On the other hand, the existence of efficient (in the sense of breaking the general lower bound) low-congestion shortcuts is known for several major graph classes, as well as

We reviewed long-term efficacy and safety data from a number of pivotal studies that reported significant survival benefits with the addition of 1 year of adjuvant

The solutions that are part of the Noah Mobile cloud service are based on the core Platform-as-a-Service (PaaS) services on the Microsoft Azure Platform.. The Noah Mobile

Accenture and Avanade’s application services for the Microsoft Windows Azure platform provides customers with deep expertise with application development and management services. •

I certify that an Examination Committee has met on 24 th March 2009 to conduct the final examination of Siti Nur Afifi Ahmad on her Master of Science thesis entitled