The purpose of a file system is to provide a common abstraction for data storage using files and directories. Filesystem data comprises the information stored in files while metadata describes the additional information needed to describe and organize the data. Operating systems provide a common set of data and metadata operations that can be used by appli- cations and typically do not depend on the exact type of the file system used.
Local file systems store data on a block-based storage device such as a disk. Distributed file systems allow access to data on a remote system using the same file system abstraction. Basic distributed file system models include client-server, SAN, and parallel file systems, which many production filesystems combine into hybrids.
Because of the higher likelihood of failure in a distributed system, the semantics for data and metadata access and persistence can vary considerably between different (and even differently configured) file systems. This fact must be considered when comparing perfor- mance.
The role of the file system in data storage is constantly changing. Although the amount of unstructured (e.g., sensor) data is increasing, increases in disk capacities and hierarchical storage management techniques permit the indefinite retention of data. Thus, the amount of metadata has been constantly increasing while the file system APIs have remained basically unchanged for years.
With the upcoming introduction of solid-state storage devices to file server systems, local file systems will become faster and additional opportunities for acceleration will become available. At the same time, the efficiency of metadata operations in distributed file systems will depend less greatly upon the storage device and more greatly upon the semantics and protocol.
Chapter 3
Distributed metadata benchmarking
This chapter discusses previous work in the field of metadata benchmarking and the capa- bilities of existing benchmarks. Based upon this discussion, it identifies additional require- ments that influenced the architecture of the DMetabench framework and presents analogies to application benchmarking in HPC. It concludes by presenting the design and implemen- tation of DMetabench.
3.1 Previous and related work
The field of file system benchmarking has been thoroughly explored in previous publica- tions, many of which described the different benchmarks proposed to obtain performance numbers for metadata operations. Benchmarks can be categorized into micro-benchmarks, which selectively stress particular operations or aspects of file systems (e.g., sequential through- put) and macro-benchmarks, which simulate application behaviour or even complete multi- application workloads.
3.1.1 Andrew Benchmark
The importance of metadata performance for scalability in distributed file systems was rec- ognized quite early when Howard et al. used a benchmarking tool in their work about AFS [HKM+88]. They described their benchmark as a script that simulates different phases of a program compilation by creating a directory, populating it with files, and then reading and compiling these files. The load on the file system generated by a single instance of the script is called a Load Unit. Howard et al. then examined the performance of AFS and NFS servers based up a given number of load units. The benchmark script later named the Andrew bench- mark is an example of an application-level benchmark.
3.1.2 Trace-based benchmarking tools
A trace is a recording of the sequence of requests sent and received in a distributed system. In the context of distributed file systems, traces can be used to simulate the load imposed on the file system during the original recording. Trace-based tools are typically used for testing NFS servers. Because it is mostly free of state, the NFS protocol can be easily recorded and processed.
LADDIS and SPEC SFS
LADDIS is a distributed benchmark tailored to measuring NFS server performance [WK93]. An updated version is part of the SPEC SFS 97 benchmark suite from the Standard Perfor- mance Evaluation Council (SPEC). LADDIS uses multiple processes on several nodes called load generators to submit NFS RPC requests to a given NFS server. As the NFS client and file system layer is bypassed, greater control can be achieved over timing and latency values. A benchmark run generates an increasing load on the NFS server until it is completely sat- urated. Saturation is detected when the RPC response time reaches a threshold. The load itself consists of a pre-defined mix of NFS operations. The original LADDIS paper described the load as half file name and attribute operations (LOOKUP and GETATTR), roughly one-third I/O-operations (READ and WRITE) and the remaining one-sixth spread among six other operations. The number of files and their size scales proportionally with 5 MB of data and 40 files for every operation per second of load applied. The final result is a table and a graph showing the RPC request latency, depending upon the number of NFS operations per second (see Fig. 3.1).
Figure 3.1: Example results of a SPEC SFS measurement of an NFS server
SPEC SFS 97 is a well-accepted parallel NFS benchmark that maintains a large share of metadata operations in the workload. In 2008 SPEC SFS was updated to SPEC SFS 2008, which modernized the workloads used (i.e. increased file sizes) and added a similar but separate benchmark for the CIFS distributed file system [SPE08]. The benchmark suite is still by design tied to the NFS/CIFS protocol and thus not useful for testing other distributed
3.1. Previous and related work 39
file systems. The main focus is on the performance of the server and it does not (again, by design) take into account the behaviour of the client software. Therefore, SPEC SFS is useful and accepted among storage vendors for comparing NFS and CIFS server performance.
TBBT
In [ZCCE03] Zhu et al. discussed the use of traces to benchmark NFS servers and then pro- posed their own toolkit they called Trace Based filesystem benchmarking tool (TBBT). The paper contains an interesting discussion about how to scale up and down a given trace to generate a higher load: a spatial scale-up generates the load on disjoint directories and enlarges the working set while a temporal scale-up speeds up the playback of a trace and maintains the amount of data. TBBT is, however, unable to test any filesystems other than NFS.
3.1.3 Lmbench-suite and Lat.fs
McVoy et. al introduced the microbenchmark suite lmbench, which contains different com- ponents for measuring memory bandwidth, memory latencies, system calls, signal process- ing times, and other system parameters [MS96]. A part of lmbench is lat.fs, a file system benchmark that measures the ”file system latency” - the time necessary to create or delete an empty file. In his paper, McVoy compared the file system latencies between the different local file and operating systems of the 1990ies. Interestingly, the paper briefly described why the persistence semantics of the file system is an influential factor in performance.
3.1.4 Postmark
Postmark, another macro-benchmark developed by J. Katcher, an employee of Network Ap- pliance, simulates the work of a simple mail server that receives messages, puts them into the file system, and then reads and deletes them [Kat97]. A postmark run consists of three phases. In the creation phase, a number of subdirectories and files are created. The trans- action phase simulates the workload of the imaginary mailserver by creating, appending, reading, and deleting files. The third phase removes the files and subdirectories.
The benchmark itself is written in C and is strictly single-threaded. Therefore, it also mainly measures file system latency and is not able to show the maximum scalability of a file system for multi-threaded applications.∗
3.1.5 FileBench
A newer benchmark, FileBench from Sun Microsystems, is a framework for the simulation of application loads in file systems [MM06]. FileBench uses a definition language for operations ∗As Network Appliance sells NFS server appliances using a non-volatile memory cache that reduces latency
and several pre-defined models that simulate the workload of common applications, such as mailservers or database servers. The chosen workload model can be simulated using several threads or processes in parallel. Unfortunately, as of 2008, it is not possible to distribute the simulation among several computers.
3.1.6 IOzone
IOzone is a data throughput benchmark that can also perform intra-node and multi-node parallel measurements [NC06]. Multi-node measurements are coordinated using an RSH/SSH based login. A recent addition is the fileops program which offers metadata performance measurements for all basic metadata operations in a single-process scenario but does not allow parallel measurements.
3.1.7 Fstress
Fstress is an older distributed NFS benchmark quite similar to SPEC SFS 97 but purely syn- thetic [And02]. It includes several example workloads (e.g., a ”web server”). The output of Fstress also shows the per-operation latency, depending upon the number of operations per second.
3.1.8 Clusterpunch
Clusterpunch is a software framework that can be used to execute small, customizable bench- marks on UNIX-based compute clusters [Krz03]. Default benchmarks available in cluster- punch stress the CPU, memory and file I/O. Each benchmark run is very short (a punch) so that the framework can be used to determine the current load and to classify cluster nodes by their benchmark throughput. Possible applications include monitoring and improving load balancing by quickly estimating the load of nodes in a cluster.
3.1.9 Parallel I/O benchmarks
Several benchmarks have been designed for measuring the performance of parallel file I/O, including b eff io and the NASA NAS parallel benchmarks[RKPH01, WdW03]. These bench- marks are able to utilize a large parallel environment using the MPI programming interface. Parallel file I/O is, however, the exact opposite of metadata-intensive operations, as it results in simultaneous access by many processes to a single file.
3.2. Discussion of objectives for a new benchmark framework 41