the libraries. Then, in section3.6, we will detail how RPCs are implemented and their usage.
In section 3.7, we put everything together to show some real applications that can be written with the framework. In section3.8 we describe churn management. Finally, in section 3.9, we will talk about sandboxing design choices.
3.2
Controller
The controller plays an essential role in our system. It is implemented as a set of cooperating processes and executes on one or several trusted servers. The only central component is a database that stores all data pertaining to participating hosts and applications. Command line Web interface Web services Trace Script log log ctl ctl ctl splayd splayd splayd splayd splayctl blacklist unseen jobs churn SQL DB
Figure 3.2 – Architecture of the Splay controller (note that all components may be distributed on different machines).
The controller (see Figure3.2) keeps track of all active Splay daemons and applications in the system. Upon startup, a splayd initiates a secure connection (SSL) to a ctl process. Afterwards, the splayd will also open a connection to a log process for each running application.
The actual Ruby implementation permits us to manage several thousands of splayds using a multi processes architecture on one server. This multi-process architecture was
initially the result of the lack of native threads in Ruby 1.8 and the poor performances of its internal threading system. Now, this architecture also permits us to scale easily by increasing the number of servers and distributing the processes.
The deployment of a distributed application is achieved by submitting a job through a command-line or Web-based interface. These interfaces also permits us to monitor the status of the running jobs and access the collected logs. Splay also provides a Web services API that can be used by other projects.
The nodes participating in the deployment can be specified explicitly as a list of hosts, or one can simply indicate the number of nodes on which deployment has to take place, regardless of their identity. One can also specify requirements in terms of resources that must be available at the participating nodes (e.g., bandwidth) or in terms of geographical location (e.g., nodes in a specific country or within a given distance from some position). Incremental deployment, i.e., adding nodes at different times, can be performed using several jobs or with the churn manager. More information about the deployment are available in section 3.4.
Each daemon and job are associated with records in the database that store information about the applications and active hosts running them (or scheduled for later execution). The controller monitors the daemons and uses a session mechanism to tolerate short- term disconnections (i.e., a daemon is considered alive if it shows activity at least once during a given time period). Only after a long-term disconnection (typically one hour) does the controller reset the status of the daemon and clean up the associated entries in the database.
Several controller processes monitor one or more tables and react to specific events (modifications in the tables). More details about how each process works are given in the next section.
The controller actually does not manage directly users access rights, usage quota and external splayds registration. These features can be added through additional interfaces that directly act on specific controller’s tables. 1 One implementation of this external interface that brings some of these features is the web-based interface SplayWeb. SplayWeb manages users, permits them to register their own splayds and to run their jobs using a web interface. It also provides some rudimentary web services APIs to access these features from a third party application.
1It would be better to provide a low level controller API for that in the future, rather than giving
3.2 Controller
3.2.1
Controller’s Processes
As previously explained, the controller architecture is build of several cooperating pro- cesses. This permits us to achieve a very good scalability, separation of concern and easier management (e.g., to restart some processes individually).
Before seeing each processes more in details, it is important to give some insights about the database structure and tables that are used to store the data.
First the splayds table stores all the attributes and configuration of a splayd. This table is primarily used to select splayds using various criteria when a new job is submitted. The splayd_availabilities table contains the history of the availability of each splayd. Using this table we can measure the average availability of each splayd and filter them using that criteria (e.g., to avoid running a long experiment on unstable nodes).
The jobs table contains all jobs submitted. When a job is deployed, the table splayd_jobs will contain the list of the splayds where the job has been deployed.
Finally, the actions table is used as a queue to send commands to the splayds.
ctl processes
When the controller starts, a pool of ctl processes are launched and during the regis- tration part, each splayd will receive the instructions to connect to one of them. After the communication initialization and state synchronization, the ctl process will monitor the actions queue (queue of commands for the connected splayd), send them and finally update the local splayd status in the database after having received the command’s reply.
jobs processes
The jobs process dequeues jobs from the database and searches for a set of splayds matching the constraints specified by the user. In case the job needs to run under churn, an additional churn process is created to monitor the states of the running splayds and to switch them on or off during the time of the experiment.
This process has the complex task to find enough splayds meeting the job’s criteria and to deploy it on them.
The splayd_selections table will be used to store the initial selection of splayds as well as the final list. Another table, job_mandatory_splayds, is used when submitting a job that absolutely must includes some specific splayds (by default, the controller chooses the most responsive, less loaded splayds).
log processes
As for the ctl processes, a pool of log processes are launched.
Each splayd will receive during the registration protocol the instructions to connect to one of them. The accesses to the log are restricted by several security checks, as is the total log size that can be received by a single splayd.
The cumulative output of all splayds running the job is stored in a plain text file named using the job’s reference.
The role of log is to collect information about or from running applications. This information can then be processed to debug algorithms, analyze their performance, gather runtime statistics, etc.
other processes
The unseen process will monitor the state of splayds by sending them a regular ping to verify their availability (the ping will be added as a command in the actions table). The blacklist process is responsible to ensure that all running splayds have the most
up to date blacklist of hosts. Support for blacklisting has been added to provide
protection against malicious Splay usage and if complains about someone receiving unwanted connections from one or more splayds are received. In this case, all successive attempts will be denied (before taking other measures).