Data Management

Data Management Services in the gLite middleware use the file as the primary unit for data management. A file is usually replicated for performance and reliability reasons, but after the creation a replica cannot be modified not to break the consistency among the replicas.

Different names are used to identify a file in the Grid:

• GUID, Grid Unique IDentifier: it is used to identify unambiguously a file into

the Grid. Its format is: guid:<unique string> where the unique string is made of a MAC address plus a timestamp.

• LFN, Logical File Name: this is a human readable string, chosen by users, that

they will normally use to refer to a file irrespective of its location. The format is lfn:<any string>.

• SURL and TURL, Storage and Transport URL: they refer to a physical file,

meaning to a specific replica stored in a specific SE. The SURL (also known as Physical File Name, PFN) includes the name of the host where the file is stored, plus the path to its current location.

The format is either sfn:<SE_hostname>/<path> or srm:<SE_hostname>/<path>

depending on the type of SE, with or without an SRM interface3_{. The TURL}

has also a protocol specification and it is used to retrieve or move the file. The format is <protocol>://<SE_hostname>:port/<path>.

Files are stored in Storage Elements, but the mapping between GUID, LFN and SURL is kept on a replica catalogue. The catalogue supported by gLite is the LFC File Catalogue, that we discuss later.

Different Storage Elements and access protocols are supported in gLite. As re- gards the Storage Elements, three types of storage elements are supported: Castor, dCache and LCG Disk pool manager. Castor is a mass storage system that uses a disk buffer as front end to a tape robot. Files are kept on tapes and transferred to the disk buffer before being accessed. dCache is a system of disk pools managed by a single server which presents a single virtual file system to the users. This system can be used both as front end for mass storage systems or as disk server. LCG Disk pool manager is a lightweight disk server solution for small sites.

All the three types of Storage Element provide a uniform interface to users, the Storage Resource Manager (SRM) interface, which allows users to do operations like

requesting files, moving them from tape to disk buffer in case of mass storage system and reserving space, in the same way on all the SEs.

As we said, the catalogue used in gLite to locate files is the LHC File Catalogue (LFC). It contains the mapping between GUIDs and LFNs and between GUIDs and SURLs (the TURL can be obtained dynamically from the SURL). This catalogue is implemented using Oracle as backend database, but MySQL is also supported. Using Oracle, the catalogue can be replicated to several sites, but only with a single master configuration, that is, only one replica can be updated and the others are read- only synchronised copies. This type of replication is done using Oracle Streams (see Section 3.5), as we saw in the LCG-3D project in Section 4.4. The LFC publishes its endpoint in the Information Service, so that it can be found and queried by other services, like the Workload Management System. The LFC provides a command line interface through which users can query the catalogue to retrieve the list of registered files. In the LFC, LFN entries are organised like a Unix filesystem: users can create directories, list their contents, change their access rights and so on (for the complete list of commands see [54]).

For replica management gLite offers different solutions. The most proper way to manage replicas is through a set of command line client tools called lcg_utils. They provide commands to move data into or out of the Grid, to create or delete replicas, to move replicas from one SE to another one, to list the replicas of a given LFN, to retrieve the GUID from an LFN or SURL and so on.

It is worth emphasising that files are considered read-only: if a user modifies directly a single replica, all the other replicas became stale without warning the users of the Grid. There is no replica consistency service in gLite.

When using the LFC commands a user has also to be aware that the operations he does on the catalogue do not involve the files, but just the catalogue entry. For ex- ample, when removing an LFN from the catalogue, all the replicated files associated to that LFN are not removed from their SEs, but they will not appear as part of the Grid. Thus, these replicas will be unavailable but will still take space in the SE.

Another gLite Data Management Tool is the File Transfer Service (FTS) [12], which can be used to schedule file transfers between a source and a destination SE in the Grid. The FTS service is based on GridFTP, the standard protocol for data transfers in Grid computing. In order to use the FTS command line clients a user

has to have a proxy certificate4_{, after which he can use the command to submit a file}

transfer job either specifying sources and destinations as arguments, or giving a file

4_{In this case the proxy is a long-term proxy, created using a special service called MyProxy [26]}

5.2. What is a middleware and what is it made of 69

where sources and destinations are listed. The submission commands return to the user an identification string that can be used to query the status of the job or to cancel it.

Lower level tools also exist to execute data management operations, but users are encouraged to use the high level services that we just presented, mainly for two reasons: they are easier to use and they protect against possible inconsistencies that can be generated when using the lower level tools.