The software stack to implement the Cloudlet Platform includes: ZeroMQ as the distributed application enabling messaging library, node.js as the preferred programming language, CouchDB as the datastore, Mongrel2 as the webserver, nginx as the load balancer, and OpenStack as the cloud platform.
7.1.1 Languages
Most components will be written in node.js, the platforms preferred programing language. The benefits of node.js are: 1) it is a dynamic language which allows for rapid prototyping, 2) it can
handle thousands of concurrent connections with low overhead, 3) its strong support with JSON (our preferred API and messaging format), 4) its event loops, and 5) a number of application frameworks have been built for the language e.g. express and Geddy.
However, a recurring criticism of node.js is that it’s codebases are difficult to maintain as it grows;
splitting the Cloudlet Platform into many smaller components will help address this inherent drawback with the language.
7.1.2 Messaging Framework
ZeroMQ [9] is an open source socket library that acts as a concurrency framework for scalable, distributed and concurrent application. It includes features that allow messages to be sent via in-process, inter-in-process, TCP and multicast as well as the ability to support many-to-many connections [10]. ZeroMQ libraries are available in a wide range of languages including C, C++, Java, .NET, node.js and Python. Software patterns utilise asynchronous I/O in order to build scalable multicore message passing applications. Its suitability to our platform are categorised as follows:
Patterns – ZeroMQ supports a number of communication patterns including: pipeline, fan-out, fan-in, publish-subscribe, task distribution and request-reply and exclusive pair. These patterns allow us to create multiple topologies within the one distributed application. I.e.
workers/components can communicate with each other as they wish.
Asynchronous I/O – it handles asynchronous I/O in background threads so there are no locks or wait states in concurrent ZeroMQ applications.
Multiple transport methods - Different ways of passing messages including: TCP, multicast, in-process, and inter-process which means that the same application code can be deployed on a large cluster of computers or alternatively a single personal plug server.
Languages Supported – libraries are available in a wide range of languages including C, C++, Java, .NET, node.js, and Python. While node.js is the preferred language the development of some components may be accelerated by utilising code libraries and applications written in another language like Java.
Libraries - there are many libraries that build upon the native APIs.
OSs – supported on the main operating systems including Windows, Mac and Linux.
7.1.3 DataStore
Primarily due their scalability, NoSQL databases appear to the most appropriate solution for the OPENi datastore. NoSQL databases are designed for high performance and throughput when handling vast amounts of data from large numbers of concurrent users. Of the many NoSQL datastores we compared (See Appendix II for the full analysis) we opted for CouchDB.
Couch was selected because of its multitenancy support; it can support millions of databases allowing a database per user. In addition it is lightweight enough to install in a personal plug server.
CouchDB’s support for the map-reduce paradigm is essential for the data aggregation component.
Additionally, CouchDB has built in compression, native JSON and JavaScript support, and can support secondary indexes.
7.1.3.1 CouchDB
CouchDB (cluster of unreliable commodity hardware) is an open source databases system, managed by the Apache Software Foundation. CouchDB is a document based storage system that persist data in a JSON format. According to the CAP theorem partition tolerant distributed databases have to make a design choice between availability and consistency, CouchDB provides high availability and partition tolerance with eventual consistency.
The simplistic data model employed by CouchDB is a flat collection of independent self-contained documents. Structures can be applied dynamically to the data through the creation of views. Views are controlled by implementing JavaScript functions, these describe how to aggregate and filter documents to produce a view. These functions are analogous to the mapper function associated with Map Reduce. View representations of the data do not affect the underlying document; it is an illustration of data within documents based on some criteria.
CouchDB operates in a peer-to-peer distributed architecture. Each instance can have its own independent replica of the data. Each replica can be altered independently and changes will eventually be synchronised in a bi-directional and incremental (only what has changed since the last update) fashion. Instances can also continue to operate when connectivity is lost with other nodes and it will continue to operate as normal when the problem is solved.
Secondary index support has been added to the views to allow for more efficient querying. As the database changes the views and their indexes are incrementally updated to reflect the alterations.
The view engine is optimised to only evaluate documents that are new or have changed since the previous view update. This incremental approach as well as the option to allow data to be manipulated in memory (which is flushed to disk when a threshold is reached) is designed to reduce the time used writing to disk. Since version 1.2.0 compression is enabled by default with Google’s snappy, with the option to use Gzip as an alternative.
The primary method of accessing data is through a set of REST APIs and HTTP. This feature makes CouchDB extremely interoperable with any programming paradigm. Extensible security and validation functionality is delivered. JavaScript functions can be used to create a more complex access control system than the basic one provided.
7.1.4 Web Server
Mongrel2 [11] is a protocol and language agnostic web server. It has supports HTTP, Flash sockets, WebSockets, Long Polling on the user facing side and exposes ZeroMQ handlers on the backend. The Cloudlet Platforms workers will integrate with Mongrel2’s handlers. Conveniently the node open-source community created a node.js handler for Mongrel2 called m2node.
7.1.5 Load Balancer
nginx [12] is a compact software package containing a HTTP proxy, load balancer, edge cache and origin server. nginx has many features which cover protocols and performance, request routing, security, edge cache and origin server, and configuration and management. Some of the key features are IPv6 support, support for up to a million concurrent connections, multi-tenancy, session persistence, connection and request policing, header manipulation, load balancing and many more.
nginx provides reverse proxy and load balancing features that allows the act of load balancing to be done in software rather than hardware. The configuration of the load balancers is done in a configuration files which allow for the fine tuning of the logic behind the load balancing. Weights can be assigned to each interface and this will dictate the amount of traffic routed to each as well as IP hashing to ensure that returning IP addresses are returned to the same interface [13].
7.1.6 Cloud Platform
When deciding on which cloud technology we should employ on the OPENi project we analysed a number of Infrastructure-as-a-Service (Iaas) and (Paas) offerings (see Appendix III for the full analysis). Of the PaaS offerings CloudFoundry was the closest to meeting our stack requirements however it doesn’t provide built in support for ZeroMQ. Therefore CloudFoundry and the other PaaS offerings were excluded.
After reviewing the IaaS solutions it was felt that OpenStack was the most suitable for this project due to several key factors. OpenStack is an open source project and as such there is no cost associated with airing the use of the technology and utilising it for the purpose of this project. Its interoperability with other cloud providers and hypervisor technologies allows for freedom from vendor lock-in. The OpenStack Foundation which protects, empowers and promotes the OpenStack community consists of many internationally renowned organisations such as AT&T, Canonical(Ubuntu), IBM, Rackspace, Red Hat, Cisco, VMWare and many more [14]. The collaboration with these technological leading organisations shows that OpenStack will not easily be pushed from its place among the top cloud technologies and provides a sense of reassurance about the future of OpenStack.
7.1.6.1 OpenStack
OpenStack [15], as described in deliverable 2.2, is a collection of open source components used to deliver public and private clouds. OpenStack’s interoperability with multiple hypervisors, such as Xen, KVM and VMWare ESXi, and other cloud providers, like Amazon EC2 and S3, have made it a popular solution for cloud platforms.
OpenStack provides access to the underlying hardware with the use of the OpenStack APIs. This coupled with the ability to use a command line interface for the management of the physical hosts allows for finer control of the system as a whole.
Managing the virtual machines in the system can be done several ways. OpenStack has a web
“Dashboard” GUI that can be used to manage all the main aspects of the system including but not limited to VM instance and image management, VM provisioning, user management and other security features. The OpenStack API can also be used to manage these aspects of the system as well using the Amazon EC2 API as OpenStack has implemented the system so that it can support the EC2 API.
The key security features are the ability to create security groups, user groups and the user management system. The security groups allow for the finer control over the flow of information in the system and can be used to isolate groups of machines from one another and mitigate any
possible security risks that would be present in a system where all machines had access to one another.
7.1.7 Orchestration
We will use Chef to orchestrate the OPENi platform’s hardware nodes, virtual machines, and ZeroMQ workers. Ganglia and Nagios are system monitoring tools.
7.1.7.1 Chef
Chef is an IT infrastructure automation and configuration management tool used by system administrators and DevOps personnel [16]. Chef uses ruby as its main language and its configuration groups and files called “cookbooks” and “receipts”. The CLI tool known as “knife” provides an interface between the local Chef repo and the Chef server; it is also the main means to manage cookbooks and receipts.
A Chef cookbook is a collection of receipts that can be used to configure a system in a certain manner. It could have a “database” cookbook which could install all the services required on a machine to be a database. Normally separate cookbooks would be created for each application, i.e.
the user could create an Apache cookbook and then download all the receipts for apache into that cookbook. The apache cookbook could then be included in different configurations i.e. a database configuration, which would then be pushed to all machines using that configuration.
The “run list” knife command is used to push or configure what cookbooks need to be running on a certain machine. Using the knife command the Chef server can connect to its hosts and push the configuration cookbooks to the host.
7.1.7.2 Ganglia
Ganglia [17] is an open source distributed monitoring system for systems such as clusters, grids and clouds. It is optimized in such a way that it has very low per node overhead while still providing accurate statistics about each node using a combination of XML, XDR and RRDTool.
Ganglia has a hierarchical structure and runs a script on each node in the system. The information is gathered and then passed up the tree; this collected information is then displayed upon a web interface that shows graphs of many of the details of the system including CPU usage, network usage, memory usage and many more. Ganglia can be extended to application level monitoring. E.g.
The data authorisation component could notify Ganglia each time an app is refused access to a cloudlet. The resulting Ganglia graphs give the platform manager insights into usage patterns over time.
7.1.7.3 Nagios
Nagios [18] is an IT infrastructure monitoring system that allows organisations to identify and resolve issues when they first appear and before they progress further. It can be set up to monitor different
aspects of the IT infrastructure such as system metrics, network infrastructure, applications, services and servers.
Nagios includes many useful features including the ability to set up alerts for critical failures of the system that will be sent via email, SMS or a custom script. These alerts can also be grouped in different levels of alert so that if an alert is not handled promptly it can be escalated to a higher level of priority. It can generate detailed historical reports for review that include outages, events, notifications and alerts