Of all the challenges presented by the NFS data set, the most serious was the presence of large files. In this section we detail the three changes we made to ourlokoimplementation to allow for these large files.
6.3.1 REDUCING MEMORY USAGE
Given the potentially large sizes of NFS files, we can no longer expect loko to store all objects in main memory. For keyspaces whose total size exceeds a threshold d, loko stores the value of the contents field on disk instead of in main memory. Without this change,loko proxies would run out of memory before even starting an experiment. Note that sinceownerandperms can both be represented within a fixed number of bits (e.g., an integer), keyspaces are large if and only if thecontents of the files they represent are large, and so storing contents on disk is sufficient to mitigate main-memory capacity concerns from large files.
We expect a tradeoff when setting d: Large values will allow more objects to reside in main memory, potentially filling it to capacity and causing thrashing. Small values will cause more objects to be stored on disk, freeing space in main memory but increasing latency for any queries to those objects, due to increased disk seek time (vs. main memory).
6.3.2 REDUCING NETWORK LOAD: MIGRATION
In order to prevent network congestion due to migration of large keyspaces, nfs-loko disables migration of keyspaces exceeding a certain size c—the “migration cutoff.” Small values forcwill disable migration for many objects, reducing network load due to migration but thus increasing—or, more accurately, potentially failing to reduce—request latency for requests to those objects. By not reducing the number of hops per request, the lack of migration would also fail to reduce the number of messages for requests to the affected objects, and so we expect that a value of c that is too small may cause more harm than good from a performance standpoint. On the other hand, values that are too large will allow large keyspaces to migrate among proxies, potentially causing network slowdowns throughout the network, particulary in the case of low migration thresholds m that could cause some migration jitter in extreme cases.
6.3.3 REDUCING NETWORK LOAD: BLOCK REQUESTS
loko is designed to respond to read requests with a copy of the entire keyspace re- quested, in order to facilitate caching and read clustering (see Section 4.1). Very large files, as frequently encountered in NFS filesystems, are impractical to repeatedly send across the network on the critical path of a response. Furthermore, doing so would not faithfully mir- ror the behavior of NFS, which forces clients who wish to fully download a large file to do so over the course of many block-level requests, not all at once. Similarly, we have adapted nfs-loko to use block-level requests in the case of large files. We made several changes toloko to that end.
First, all requests for a keyspace’s contents now carry with them the block they wish to access and the corresponding payload size, obtained from the original NFS logs. That is, each read request carries the offset (within the file) being read and the length of the payload that was returned by the read (in the original data set). Similarly, write requests carry with them an offset and a payload to write to that offset. Requests and responses also carry a boolean flag called is-block.
For small files, loko handles read requests normally, forming clusters when possible and caching objects along the way at its discretion. However, if a file’s size exceeds a thresholdb, then responses to requests for that object will not contain the entire keyspace. Instead, their is-block flag will be set to true, and their only payload will be the payload needed to answer the particular read request that reached the object.
When a response reaches a proxy along the path back to the originating proxy, it may encounter read requests that were paused awaiting its arrival. In the is-block=false
case, the response would answer these reads and proceed along its path. But in the
is-block=true case, the response does not carry enough information to satisfy any read
except the one that reached the object.1 So, the response marks these waiting requests as
1
We ignore the case in which some paused reads asked for exactly the same block and wanted exactly the same response payload, as we believed this case to be relatively rare and not worth the extra bookkeeping
is-block=true and then unpauses them, freeing them all to proceed to the object. Future proxies receiving these unpaused requests will see the is-block=true flag and know not to pause those requests, as no clustering is possible for block requests. The added network overhead incurred by the loss of read clustering for these large objects should be neglible compared to the savings from not transferring the objects themselves over the network.