Implementation Overview

The implementation of the VSS required creation of new services (RM, DCE), as well as modifications of the existing ones.

The following sections will discuss in detail the implementation of new functionalities across the existing and new system services.

3.1. Storage Access Time Estimation

The storage access time estimation service provides access to an external estimator. Estimator is a legacy program, which returns estimated storage access time for a given physical file stored on the underlying storage system.

As a part of previous studies, an estimator for the DiskXtender HSM system has been developed and used. In case of using other storage systems, an appropriate estimator should be provided by the system administrator.

The storage access time estimation is implemented as a part of the DCE in a form of new service. However, in order to provide this functionality to the users, an extension of the interface of the Data Broker was necessary, as well as addition of a new method to the MR. The method gets all locations of a file and asks each location’s storage access time estimation service for the estimated storage access time of that file location. The existing methods have not been modified.

In Figure 2 the UML sequence diagram for Storage Access Time Estimation is shown.

Figure 2. Storage Access Time Estimation sequence diagram

3.2. File Ordering

Similarly to the storage access time estimation, the file ordering functionality is part of the DCE. It is represented by a newly defined service. Also this time, apart from the extension of the Data Broker’s interface, a new method in the Metadata Repository had to be implemented, but no modifications to the existing methods were necessary. The new method is responsible for the selection of an optimal location for a file order and forwarding the file order to that location.

The file ordering service for each new file order computes the time, when to start file transfer from tapes to disk cache. The time is computed according to the user requirement given in the file order and the estimated storage access time (with some safety margin). When the operation of file caching is finished, the file is locked in cache, till the user specified moment of requesting. After that moment the lock is

removed no matter if the file has actually been accessed. The lock removal allows the file to be removed, if necessary, by the HSM system according to its purging policy.

In Figure 3 the UML sequence diagram for File Order is shown.

Figure 3. File Ordering sequence diagram

3.3. Access to File Fragments

The access to file fragments is implemented in the file ordering service. The user can specify a file fragment by defining its byte range. The access to file fragments is done in the same way as the file ordering. The only difference is, that the system (more precisely file ordering service) is not copying the whole file to the cache but only the requested fragment. When the fragment is no longer needed it is removed from the cache.

The presented above functionalities – Storage Access Time Estimation, File Ordering and Access to File Fragments – can be easily reused, they do not depend on any other system service and are implemented as web services grouped in DCE. 3.4. Replica Selection Algorithm

The algorithm is an integral part of the Metadata Repository, so the modification of the existing code was necessary. The existing, old algorithm has been entirely replaced by the new one. The new algorithm chooses the replica with the minimal access time, which is estimated based on the current system condition (storage system load and performance, network latency).

3.5. Automatic Replication

The automatic replication is implemented as a new service – RM. The service periodically checks the file access logs and evaluates decisions about replication.

For supporting the automatic replication, new specific methods have been added to the MR. These methods are used for selection of the optimal location for a new file and for the addition of a new replica to the repository.

In order to perform file transfers, the adequate file transfer service had to be im- plemented. The service was implemented as a part of the DCE. It allows to upload and download files from or to the SE. Actually only the FTP protocol is supported.

The Replica Manager works accordingly to the algorithm, which outline is presen- ted bellow:

1. get the file access logs,

2. logs analysis – selection of files, which need to be replicated,

3. for each selected file:

a) get optimal4 storage element for new file,

b) get optimal5 source file location,

c) transfer the file using the DCE file transfer service,

d) add new replica to the repository.

In order to get the file access logs, Replica Manager executes a script provided by the system administrator. The script should place the logs in a proper directory, which is set in the RM configuration file. In the case, when RM is deployed on the same machine as the Metadata Repository, the script can simply copy the files or do nothing (depending on configuration). In the other cases, it should transfer the logs using appropriate file transfer protocol.

Automatic replication UML sequence diagram is shown in Figure 4.

Figure 4. Automatic Replication sequence diagram

4_{The optimal storage element is selected based on its storage class, storage system load and performance,}

network latency

5_{The optimal source file location is the location with the minimum access time for the selected storage}

3.6. Replica Coherency

Maintaining replica coherency is a complex task that requires cooperation between Metadata Repository and Replica Manager. MR is responsible for preserving file consistency, while RM controls replicas propagation process.

MR as a basic DMS service has the ability to preserve file consistency by keeping the replicated files read-only. In order to allow replicated file updates, the replica coherency algorithm has been implemented in MR. In the case of replicated file update MR marks all replicas as stale and allows the user to update one replica only. While updated, MR allows access to the updated replica and blocks access to the remaining replicas. MR sends the list of the stale replicas and the location of the updated replica to RM.

During the replica propagation process, RM sends file transfer requests to appropriate file transfer service. After every successful upload, MR is notified and allows access to the replica just uploaded.

The cooperation of RM and MR in process of replica update is shown in Figure 5.

Figure 5. Replicas Update sequence diagram