&
%
PARALLEL I/O FOR HIGH
PERFORMANCE COMPUTING
Skriptum zur Vorlesung
Peter Brezany
Institut f¨
ur Scientific Computing
Universit¨
at Wien
E-Mail: [email protected]
&
%
The Need for Parallelism
The fastest individual disk drives can read and write data at about 25 MB/sec. This is far too slow for modern parallel computers, which need to access data at hundreds or thousands of megabytes per second. An obvious solution is to use parallel I/O.
The main parallel I/O technique is disk striping (see Figure on the next slide). A computer writing a large quantity of data can split the data into pieces and write them simultaneously to separate disks in a disk array.
The data is generally divided into fixed-size blocks, and the blocks are distributed cyclically to the disks.
The number of disks in this simple striping scheme is called the stripe factor, and the size of the blocks is the stripe depth.
The stripe depth can be as little as one bit or as much as several disk sectors. The appropriate choice depends on how the disk array will be used, as
&
%
In principle, a system could be built in this way that striped data over hundreds of disks and moved data at many gigabytes per second.
The problem with a simple striping scheme is reliability. If a file is striped over multiple disks, the loss of even one disk could make the entire file useless, and the risk of loosing a disk rises approximately in proportion to the number of disks.
&
%
RAID
RAID - Redundant array of inexpensive disks, proposed at the University of California, Berkeley, in 1988.
Now “RAID” is taken to stand for “redundant arrays of independent disks”. The largest RAID systems move data at more than 200 MB/sec and store over 1 TB.
The central idea in RAID is to replicate data over several disks so that no data will be lost if one disk fails.
The original RAID publication describes five (one to five) strategies, called
RAID levels with different performance characteristics and different ways of replicating data.
Disk striping with no redundant storage is sometimes called RAID 0.
RAID technology has spread from large storage systems used in computer centers to compact units designed to work with desktop computers.
&
%
RAID Level 0
Simple disk striping with no redundancy, so the loss of a single disk will corrupt data.
0
1
2
3
4
5
6
7
8
9
10 11
13
14
15
16
17
18
19
12
In this and following figures, each column represents a disk; all examples have the equivalent of 4 data disks. Each row represents one stripe, with one
&
%
Otherwise known as disk mirroring. Disks are paired, and every write is sent to both disks. If a disk fails, its mirror can be used instead.
Usually, the writes can proceed in parallel, so the write transfer rate is about the same as it would be for a single disk. For reads, duplicating the data improves the access time. The system can improve to the performance by retrieving alternate blocks from both disks in parallel.
0
4
8
16
12
1
5
9
13
17
2
6
14
18
3
7
11
15
19
0
4
8
12
16
1
5
9
13
17
2
6
10
15
19
10
3
15
19
11
7
&
%
Hamming code. Data is striped across N data disks. Hamming code is computed for each group of N bits, one taken from each data disk at
corresponding positions, to produce a larger set of bits. Several “check” disks are added, so we can distribute the coded bits one per disk. Since a Hamming code is designed to detect and correct errors, the bit lost due to a disk failure can be recovered using the extra Hamming-code bits stored on the check disks.
1 2 3
4 5 6 7
8 9 10 11
13 14 15 16 17 18 19 12
0 C C C
C C C
C C C C C C
C C C
Shaded striping units represent redundant information
(C for check bits).
For N=10 data disks, 4 check disks are required; for N=25 data disks, 5 check disks are required.
This technique uses disk space less efficiently than parity-based methods discussed below. So it is rarely implemented.
&
%
A simpler and more efficient way to protect data than for instance Hamming codes, is parity checking. This technique relies on the properties of the
exclusive-or (XOR) operator.
c = a XOR b (if a==1 and b==1 or a==0 and b==0 then c = 0 else c = 1) When applied to two collections of bits, such as a pair of bytes, XOR operates individually on corresponding pairs of bits. For example:
01101010 (data byte 1) XOR 11001001 (data byte 2)
---10100011 (parity byte)
XOR is reversible. Suppose that 3 bytes listed above were stored on separate disks, and the first disk, containing data byte 1, failed. Recovery:
11001001 (data byte 2) XOR 10100011 (parity byte)
&
%
Parity Checking (Cont.)
Likewise, an entire block of parity bytes can back up two blocks of data bytes, as long as only one of the blocks needs to be recovered.
In fact, a block of parity bytes can back up any number of data blocks, since the XOR operator works with any number of input bits. When multiple bits are combined with the XOR operator, the result is zero if the number of ones in the input is even, and the result is one if the number of ones is odd.
RAID levels 3, 4, and 5 all use parity schemes. The difference among these levels are in how they store the parity data.
&
%
RAID Level 3
Single-bit parity. Since when the disk fails, it is known to have failed, and the identity of the failed disk is known, a single parity bit for each N-bit data is sufficient to reproduce the lost bit in that word. Thus, RAID level 3 uses only one “parity disk” for any group of size N.
0
1
2
3
4
5
6
7
8
9
10 11
13
14
15
16
17
18
19
12
P
P
P
P
P
Shaded striping units
represent redundant information
(P for parity bits)
&
%
RAID Level 4
Block-sized striping depth. Parity is computed in the same way as in Raid 3: one parity bit is produced from N bits, one from each disk and corresponding positions.
0 1 2 3
4 5 6 7
8 9 10 11
13 14 15 16 17 18 19 12
P P
P
P
P
Shaded striping units
represent redundant information (P for parity bits)
&
%
RAID Level 5
Rotated parity blocks. Notice that in a workload of small reads and writes, RAID 4 requires several one-block I/Os to write a single data block: read the old data and parity blocks, compute the new parity blocks, and write the new data and parity blocks. Although the data reads and writes are spread over N disks, the parity disk is used for every write request, and thus becomes a
bottleneck. RAID level 5 solve this problem by distributing parity blocks, but their positions are different on each stripe,
Shaded striping units represent 0 1 2 3
4 5 6
8 9 11
14 15 17 18 19 12
P
P
P P
P
redundant information (P for parity bits)
&
%
RAID - Conclusions
The most common RAIDs in use are RAID level 0 (when reliability is not an issue), RAID level 1 (primarily in critical database applications), RAID level 3 (for high-bandwidth large-read and -write applications), and RAID level 5 (for applications with small I/O requests).
