at the University of Oslo (USIT)) while the indexing service and entry point are located at central machines in Ørestaden near Copenhagen. To avoid having all the traffic passing through a central endpoint the actual file transfer is redirected to go directly between the source point and the selected data pool (part of the GridFTP V2 protocol [50]).
4.2
Distributed Storage
An essential part of distributed data management is of course how to store the data. To create a distributed storage solution involves facilitating data transfer and data access, in a secure manner, i.e., in a manner so that at any point in the lifetime of the data only users with read access can access entries and only users with write access can change the entries. In the same manner as one can distin- guish between high-performance computing and high-throughput computing, one can distinguish between distributed storage for high performance and distributed storage for high throughput. While the first is designed to be deployed in a Local Area Network (LAN), inside the same firewall and with full control of the network connections, the latter is designed to work over the Wide Area Network (WAN) without control of the network connection, high network latency and high probabil- ity of failing connections. While it is quite common for high throughput storage to use high performance storage solutions as local storage backends2, we will here fo- cus on the high throughput variant of distributed storage, and the term distributed storage should be read as distributed storage for high throughput computing in the remainder of this chapter.
Due to the need for connecting over the WAN, distributed storage solutions need to be exposed to the Internet, both internally and externally. This poses several challenges:
• Security: When being confined to a single LAN, a malicious user will have to break into the LAN (or otherwise gain access to the LAN) to pick up data transfers. File transfers over WAN are exposed to any user and sensitive data need to be secured through encryption.
• Access: In a LAN environment there is a limited number of users, all of them accessing the system with a local username and password. In a WAN environment there is a much greater number of potential users and traditional access control methods are not designed to scale to huge numbers of users. • Fault tolerance: While a LAN environment is not void of errors, the po-
tential sources of errors increase in a WAN environment as more connection 2For example, the dCache pool at the University of Oslo uses GPFS to physically store the data.
points need to be passed between the resources. Additionally, the infras- tructure (switches, routers, cables, etc.) is not physically controlled by one system administrator. Hence, one should expect a higher fault rate in a WAN environment, and design the storage system accordingly.
• Performance: Whereas in a LAN environment latency can be down to microseconds and bandwidth can reach levels near internal harddrive per- formance, in a WAN environment it is not unusual with latency up to a second and bandwidth in the order of megabits per second. A message-flow which performs very well in a LAN environment does not necessarily perform optimally in a WAN environment.
• Scalability: While having a storage system distributed over WAN drasti- cally increases the possibility of storing huge amounts of data, it also increases the needed efforts in managing and monitoring the data. Existing solutions for managing and monitoring data in a LAN are most likely not designed to work in a distributed environment.
• Metadata handling: Accessing a storage system, be it for localizing files, downloading files or uploading files, requires accessing metadata. The meta- data are stored in a database which can grow very large when the number of entries in the system increases. To achieve a reasonable performance in the system, this meta-database needs to be very efficient. To have a reliable system the meta-database needs to be consistent. Having the metadata on a single machine can provide consistency, but will limit the performance on database lookups, make a single point of failure and limit the scalability of the system. As the need to distribute a storage system is closely connected to the question of scalability, a major challenge in distributed storage is how to consistently and efficiently replicate the metadata over several machines. All these challenges are quite demanding and their solutions depend, to a cer- tain degree, on the intended use of the storage system at hand. For example, if the storage system is intended to be used for a large scale physics experiment, high performance and scalability may be prioritized while lowering the security demands. On the other hand, a multinational company in need of sharing their customer database between the local offices may set security much above perfor- mance. It is not given that a storage solution that is considered very good in one case will even be considered usable in the other case.
As a result of the diverging needs of potential users, many distributed storage solutions have emerged over the years, usually based on either grid or cloud ideas. While many of these solutions are highly specialized to solve specific tasks, some solutions are intended to be used for more generic data storage. Examples of such solutions are dCache and DPM/LFC (see Section 4.1), both used at ATLAS