BioSimGrid data file formats

The largest volume of the trajectory comprises of coordinates and velocities of atoms as this information is required for every timestep. Each contains three double precision numbers for velocity and position of each atom; the X, Y and Z compo- nents. As this data is not relational by nature and is often accessed in chunks, it is not very well suited to a RDBMS; non-relational data is discussed in section 2.3.5.

As discussed in section 2.6 file systems are databases for storing and retrieving files and can be better suited to large volumes of data than databases. It is for these reasons that the coordinates and velocities are stored in flatfiles.

Example 5.10:Shows the information stored in the metadata file for each trajectory. This information is used for accessing the data files using the appropriate code methods.

1 { 2 ’NumberOfAtomsPerFrame’: 1855, 3 ’FrameSize’: 22260, 4 ’DataType’: ’f’, 5 ’MaxFileSize’: 524288000, 6 ’ArrayWidth’: 3, 7 ’FileNames’: [’BioSimGrid_GB-STH_8.met’], 8 ’FramesPerFile’: 23552, 9 ’DataTypeSize’: 4 10 }

uses Python to serialise objects to files and the second writes out the data purely as a series of IEEE double precision floating point numbers (Parhami, 1999,IEEE, 1985). The two formats have come about as an optimisation during the development of BioSimGrid and both store the same data and support the same methods.

Both formats have the same structure for storing a trajectory. Each trajectory resides in its own directory where there is a flatfile metadata file and a series of data files. It is important to note that this is not the metadata which BioSimGrid stores about the trajectory, rather it is information about the flatfiles. The flatfile metadata file stores information about the size of frames, the number of atoms per frame, the type of data used and a list of files which contain the data (see example 5.10). This metadata file is used so that trajectory storage formats can change, but the supporting code will still be able to read the original trajectories.

Each trajectory is split into many files with a maximum size of 500 Mebibyte (MiB) for multiple reasons: i) to ensure that Operating System (OS) file size limits are not reached, ii) to speed up checksum calculations, iii) to assist with the general manageability of the data. The maximum filesize is set in the flatfile metadata file and can be changed on a per trajectory basis.

Pickled frames

This method uses the inbuilt Python serialisation method to convert Numeric ar- rays (Oliphant, 2006) to a bytestream. This bytestream is then written to a file. The serialised object can easily be re-instantiated as a Python object by reading the bytestream written from disk as shown in section 2.6.4. Figure 5.9 shows the con- tent of a single frame as well as the content of a data file. The single frame contains all the serialised data as well as some metadata so the serialiser knows the type and size of the object when reading. The data file comprises of a series of frame objects. Frame objects are never split across files and a file size is limited to a maximum size. Which often results in the file being slightly smaller than the maximum size. As all the frames are the same size it is easy to calculate the location of thenthframe which makes access quick for any frame.

Metadata X , Y , Z , /n X , Y , Z , /n X , Y , Z , /n ... Metadata X , Y , Z , /n X , Y , Z , /n X , Y , Z , /n ... Metadata X , Y , Z , /n X , Y , Z , /n X , Y , Z , /n ... Metadata X , Y , Z , /n X , Y , Z , /n X , Y , Z , /n ... . . . Frame size

Actual file size Maximum file size

Data file Object serialisation metadata Coordinates Frame Single frame

Figure 5.9:Shows the contents of a single pickled frame and shows how the frames are written in a file. Each frame comprises of metadata and coordinate tuples, ending in a delimiter.

As each frame object is serialised using the Python inbuilt functions there is no control over how this is implemented. This poses two limitations: i) there is no guarantee how it will perform in the future and ii) the implementations vary between operating systems.

For example each frame has some data stored at the beginning, essentially the object name and its size. If this data were extended in the next implementation of Python then it would not be possible to update this object as it may no longerfit

the allocated space; without having to rewrite the entire file. Although BioSimGrid contains frame object size checks future implementations run the risk of corrupting data if a frame size increases.

In some cases users require one atom from each frame in the whole file resulting in the entire file having to be read, as each object needs to be reconstructed in mem- ory; this is very time consuming. File methods were added to extract the required atom coordinates from within the frame object. As the object serialisation meta- data is a fixed data size calculating the position in a file is relatively easy, but this breaks down between operating systems. As it is difficult to ensure that extracted coordinates are accurate the decision was taken to rewrite the flatfile data format.

Serialised frames

An alternative to using the Python pickle functions is to implement a custom serial- isation method. As each file only stores objects of the same type there is no need to store the metadata for every frame. This saves space and simplifies the calculation of the coordinates for a single atom. Figure 5.10 shows how this format differs from pickled frames.

This format overcomes some of the limitations of the Python pickle method and poses an interesting example of two different data formats which implement the same file methods; this is discussed in section 5.7.2.

X , Y , Z X , Y , Z X , Y , Z ... X , Y , Z X , Y , Z X , Y , Z ... X , Y , Z X , Y , Z X , Y , Z ... X , Y , Z X , Y , Z X , Y , Z ... . . . Frame size

Actual file size Maximum file size

Data file

Coordinates Single frame

Figure 5.10:Shows the custom format used to store data in flatfiles. Each frame only contains coor- dinates, eliminating the need for per frame metadata. This reduces space requirements and means that single atom coordinated can easily be extracted.

Performance results

The decision to store data using flatfiles rather than using an RDBMS to manage all the data was based on a series of performance tests (Murdock et al., 2004,Ng et al., 2004). The tests compared the performance of the BioSimGrid coordinate data in an RDBMS, DB2 (Mullins, 2004) with the performance of the Python pickle approach. To complete the test an existing standard for flatfiles, netCDF (Rew et al., 1997) was also compared; a summary of the results can be seen in table 5.1.

Table 5.1:Shows the performance results between two flatfile approaches and one using a RDBMS (Ng et al., 2004).

DB2 netCDF Python Pickle

Size (GiB) 7.5 3.0 3.0

Random access (Sec) 560.8 16.4 18.6

Sequential access (Sec) 389.0 4.9 5.5

Each test shows the storage requirements and the total time for a random and sequential data access. The random access involved accessing and retrieving 1,000 frames from across a trajectory in a random order. The sequential access involved accessing the first 1,000 frames in a trajectory. The performance of the serialised frames described in section 5.7.1 is comparable to the Python pickled frames.

It is important to note that RDBMS are not slow, rather that the data is not suited to the database methodology. The database was using non-clustered indices and could potentially be optimised further. The large difference in time comes from Structured Query Language (SQL) not guaranteeing the order of data returned, thus forcing the results to be sorted in the result set.

Storage resource broker

BioSimGrid uses SRB to manage all its data files across the many distributed sites. Section 2.6.3 describes the features and capabilities offered by SRB and this section describes how BioSimGrid is implemented using SRB.

Local site SRB Server Flat files Client/Application SRB Client BioSimGrid Application SRBInterface Master site SRB Server Flat files MCAT

Figure 5.11:Shows how the Storage Resource Broker (SRB) manages resources. The client requests a file from the localSRB Server, the file is located using the MCAT and sent directly to the requesting client.

Each site has its own storage resource (≈4 TiB ) which are managed by a single Metadata Catalog (MCAT) database currently located at Southampton. The MCAT manages all the data files in the entire BioSimGrid repository as shown in figure 5.11.

Since it is important to reduce the volume of data transported over the network each site defaults to its own resource. This results in any data deposited at a site being written to the local resource, thus reducing cross-site network traffic.

For availability and backup issues all data is replicated to at least one other site. This is managed by a daily job executed at each site. This results in data residing in many locations which helps balance load and eliminates the need to backup such large volumes of data. The replication integrity is managed using MD5 (Rivest, 1992, Schneier, 1996) hashes for each file to ensure that the data is consistent throughout the repository.

SRB is accessed using a Python interface (Johnston, 2005a) which provides a mechanism for accessing SRB files like local file system files. This results in the same code existing on all sites providing location transparent access to the data.

The MCAT is a single point of failure for BioSimGrid which can be seen as a limitation.