Data Archive Cloud

The data archive cloud is comprised of storage nodes which are distributed web servers with mounted file systems containing files to be processed by worker nodes. Each of the storage nodes within the cloud provides a set of services to the pipeline as shown in the Table 5.1. Each service is accessed either via the control system or as HTTP requests. The function of servicing web based image file requests separated from the storage of image data allows for a flexible method of data storage and upload. The storage node architecture is shown in Figure 5.3.

Storage Node Control System Distributed Worker Queue Processing Cloud Science Payload WebServer Mount Storage Advertise Data iSCSI EBS S3

Figure 5.3: Storage Node Architecture

Service Description

Upload There are three primary methods of upload used, the first is for a storage device

containing images to be mounted by the web server, the second is via HTTP POST requests to send data to the web server, the third is to instruct the web server to initiate a download via HTTP from another data source.

Storage Image data can be stored on remote network storage system such as ISCSI

devices, or it can be locally attached storage. For the purposes of the web server the method of mounting or attaching storage is irrelevant, however it may cause some delays in servicing files depending on the read time of the storage. Where possible files should be stored in a compressed state to reduce the file transfer times.

Advertise A storage node will be required to populate an AWS Simple Queue Service

(SQS) message queue with the URL of all of the files that it currently has stored. The storage node will do this via direct commands from the control system, but could also perform this via a HTTP command. Work is advertised and the contents of the message describe the location of the stored files and how they may be accessed.

Download A web server is used to service HTTP requests providing access to files. This

NginX webserver is highly optimised for servicing static pages and can be tuned to allow large number of simultaneous connections. Any web server however is permissible, including the Apache server and S3.

Science Payload

This service provides a static location for the downloading of the worker package to each worker. This is a small package containing all of the instructions a worker node will need to operate, including where to obtain work, where to put results, what work to perform and how to perform the work. The function of the payload and its operations are discussed within the Control system

to the web server of the storage node. No specific method is presented within the NIMBUS pipeline, but data transfer times have been reviewed in the ACN Pipeline in Figure 4.9. Within the NIMBUS pipeline the data archive cloud was populated with images downloaded from the ACN AWS S3 storage buckets using the s3fs FUSE system which mounts S3 storage buckets allowing files to be copied to mounted storage blocks.

• Storage within the pipeline is flexible to the point that any mounted storage device which can be accessed by a webserver can be accommodated. The speed at which a storage node serves file requests will be determined by the network, the number of concurrent request supported and the read speed of an image from the storage node. Three different configura- tions were constructed to demonstrate this flexibility within the implemented pipeline. The HEAnet iSCSI storage was mount on DIT based storage nodes while the AWS based storage systems used the AWS Elastic Block Storage service.

An advantage of these devices is that through the use of the NFS file system, the storage device can be mounted for use by multiple servers. An NFS storage for example could be written to from a telescope location, but function as a read only mounted device running a web server. The performance of the read and write of the storage does depend on the configuration of the raw storage. If the storage uses multiple spindles then the write or read times may be quite reasonable. A disturbed file system such as S3 is likely to provide a more scaleable storage solution if there is a high rate of concurrent reading or writing.

• Advertise is the process of requesting a storage node to review the contents of its datastore and to create a message for each file found. That file is then written to an SQS queue as an advertisement of that file, indicating that it is accessible and ready for processing. The control system can instruct a storage node to review its storage and write the messages. In a production system it would be more realistic for the storage server to monitor for changes in storage and to create new messages when files were added to the data storage devices. The ability to reset and advertise everything rather than operate in an incremental fashion would also be a reasonable requirement. Message formats are explained further under the Distributed Queue Worker cloud section in this chapter.

• Download services the simple requirement of servicing static urls which provide access to files. Port 80 is a ubiquitously open port which allows all workers to access the image files. A worker node will read a message containing the URL of the image file and simply issue a request to download using a Python script. The NginX web server is a fast static web server which was used for most storage nodes. The entire pipeline works on the assumption that work is obtained through URL downloads.

• Science Payload is essential to the creation of workers which can be reconfigured easily.

figured before it starts performing any work. The science payload solution is initiated from the control system, but uses the storage node as a central point of advertising packages. When requested, the central web node responsible for package management removes the existing package from a standard location on the web server and creates and publishes a new version of the package. New package details are obtained from a central GIT repository.