Distributed storage systems attempt to provide two types of reliability: avail- ability and durability [46]. A data object is available when it can be recon- structed from the data stored on currently available nodes. The durability of a data object is maintained if it has not been lost due to permanent node fail- ures, that is, it may be available at some point in the future. Important coding metrics that concur in determining a reliable system are:
1. Fault tolerance, i.e., the number of node failures that can be tolerated by the system without data loss;
2. Storage overhead, i.e., the ratio of the storage space dedicated to the re- dundancy over the total storage space;
3. Speed of encoding and decoding.
The higher the storage overhead and the fault tolerance the more reliable the system. But while a high fault tolerance is desirable, a high storage overhead is not. This leads to choose coding techniques with length n and dimension k such that:
• n − k is small compared to k, typically n−k k <1;
• the minimum distance d is as close as possible to the Singleton bound, that is, to n − k + 1 for linear codes.
This motivates the traditional choice of Reed-Solomon codes, that can be built for any k and n and reach the Singleton bound with equality.
Regarding the speed of encoding and decoding, in their fastest instance rs(k,n − k) codes can be encoded and decoded in quadratic time with respect to k, see Section 2.1. More precisely, algorithms for encoding and decoding Reed- Solomon codes in time O(n log2n log log n) are known, but for small values of
n, quadratic time algorithms are faster than the theoretically, asymptotically fast algorithms [27].
When coding is used on a distributed storage system, if a node fails, in order to maintain the same level of reliability, the system needs to re-create encoded information at a replacement node. The consideration of the repair network
Section Length Dimension Fault toler an ce Repair lo calit y Help er no des Systematic Vector co de Binary Year MDS [16] 2.1 n k n− k k 3 7 7 1960 LRC [56] 4.3.1 n(r + 1) kr n− k r 3 7 7 2014 LRC [57] 4.3.2 n+ τ − 1 k n− k ka 3 7 7 2013 LRC [40] 4.3.3 16 10 4 5 3 7 7 2013 LRC [41] 4.3.4 10 6 3 3 3 7 7 2012 LRC [58] 4.3.5 n k n− k − ⌈k r⌉ + 1 r 7 b 7 7 2014 LRC [59] 4.3.6 n k n− k − ⌈k r⌉ + 1 r c 7d 7 7 2016 MBR [55] 4.4.2 n k [k, n − 1] 3 3 7 2011 MSR [51] 4.4.3 n k [2k − 2, n − 1] 3 3 7 2015 MSR [60] 4.4.4 k+ 2 k 2 k+ 1 3 3 3 2016
aA more general construction exists, which anyway improves information locality only. For
simplicity, we divide the k data blocks into τ groups with τ∣ k, create 1 redundant block per group and leave unchanged n− k − 1 global parities. Then, the information locality is kτ.
bThe authors offer a systematic version of the LRC code, for which the optimality of the
minimum distance is generally not guaranteed.
cIt must be r+ 1 ∣ n.
dThe generator matrix described in [59] is not in systematic form. Nevertheless, the authors
describe a method to make it systematic, by preserving the locality and distance properties.
Table 4.1: State-of-the-art codes summary.
traffic gives rise to new design challenges [53]. Thus, besides the traditional metrics, three major repair cost metrics have been identified in [56]:
4. Disk I/O, i.e., the number of bits read during the repair;
5. Repair bandwidth, i.e, the number of bits communicated in the network during the repair;
6. Repair locality, i.e., the number of nodes that participate in the repair process.
We now describe these new metrics in more detail.
4.2.1
Disk I/O
The disk read overhead during a repair using a Reed-Solomon code of dimension kis k times that under replication. Indeed, to reconstruct a single block of size b, k blocks of size b are read from the nodes participating to the repair [51]. Consequently, under RS codes, repair requires a large amount of disk I/O, rising the need for lighter techniques.
4.2.2
Repair bandwidth
When a node fails, storage systems must set up a replacement node using in- formation from the functioning nodes. Hence, a key question is how to restore the information in the replacement node while transferring as little data as pos- sible across the network. If a rs(k,n − k) code is used, for every block on the failed node, k blocks must be downloaded from surviving ones and combined by
the replacement node to reconstruct the lost coded block. Downloading a full block from each of k surviving nodes is suboptimal [46]. Indeed, data recon- struction can be performed by downloading functions of the data stored in the surviving nodes.
To redeem Reed-Solomon codes, we cite a very recent work by Guruswami, showing that RS codes can also take advantage of the freedom to download partial symbols for the exact repair problem. The exact repair problem for high- rate rs(k,n − k) codes can be solved with repair bandwidth of O(n) bits [54].
4.2.3
Repair locality
Repair locality deals with the repair of single node failures, i.e., with the recon- struction of a single erased block in a codeword [40]. The repair locality of a code is the maximum over the blocks locality [61]. The block locality of a block bis the smallest integer r such that b is a linear combination of exactly r other blocks. In other words, the block is said to have locality r if it can be recovered by accessing at most r other blocks [61]. The block locality, and thus the repair locality, ranges from 1, if the block is replicated, to k as this is the number of source blocks. For repair efficiency, the lower the better.
A code with repair locality r has at least one block having block locality exactly r and every block is a linear combination of at most r other blocks. When it is clear from the context, we shall write locality for both block locality and repair locality. If we deal with the locality of the code we refer to repair locality, while block locality refers to the locality of a block of a representative codeword. Locality can be weakened to protect only data blocks: the information locality of a code is the maximum over all the data block localities. Moreover, we refer to a code offering the same block locality r for every block as a code with all-blocks locality r. For example, MDS codes of dimension k have all-blocks locality k, which is the highest, thus the worst, possible.
We now give operational definitions of block locality. Let g1, . . . , gn be the
columns of a generator matrix G for a certain code. We say that the block in coordinate 1 ≤ i ≤ n has locality r, if the column gi can be written as linear
combination of r other columns of G (and not less than r). Equivalently, for any coordinate 1 ≤ i ≤ n, there exists a row in the parity-check matrix of the code of Hamming weight at most r + 1, whose support includes i [62]. The definition of code locality can be relaxed as in [56] to include both linear and non-linear codes and to allow the size of the input and output blocks to be different.