The raidtools package, also maintained by Ingo Molnar, provides a set of utilities for creating and managing software arrays. raidtools has been the standard software RAID management package for Linux since the inception of the software RAID driver. Over the years, raidtools has proven cumbersome to use, mostly because it relies on a configuration file (/etc/raidtab) that is difficult to maintain, and partly because its features are limited. In August 2001, Neil Brown released an alternative.
His mdadm package provides a simple, yet robust way to manage software arrays.
mdadm is now at version 1.0.1 and has proven quite stable after its first year of devel-opment. It has received a positive response on the Linux-raid mailing list and will likely become widespread in the future. The rest of this chapter will provide you with examples of building and managing arrays, using the raidtools package, as well as mdadm.
raidtools
raidtools provides a small, simple command set that allows administrators to create, activate, and stop software arrays. Each array managed by the system is defined in a global configuration file (/etc/raidtab) that describes which physical disks are mem-bers of each array. /etc/raidtab also contains metadata about every array, such as its RAID level and failover capabilities. The commands included with the raidtools package use the information in /etc/raidtab to interface with arrays.
There are two notable versions of raidtools. Version 0.90 is the most recent imple-mentation, and the only one being maintained. It should be used only in conjunc-tion with the 0.90 driver. An older version (0.42) of raidtools is also still available.
This version should only be used when working with older systems that still use the 0.36 software RAID driver. Never use raidtools with the wrong driver!
Also included with raidtools are sample configuration files and documentation (man-ual pages and cookbooks). Unfortunately, much of the documentation is seriously outdated, sometimes by a factor of years. Adhering to the open source philosophy that “old documentation is better than no documentation,” a lot of outdated infor-mation is included with the raidtools package. It’s probably best to double-check a Linux Documentation Project mirror for the most recent information and consult additional reference material and mailing lists in lieu of relying on the documenta-tion in the package.
As of this writing, raidtools-1.0 has been released in a limited capacity. It can be found in Red Hat 8.0 and will likely become pervasive during the next few months. It fixes several minor bugs and can be used as a replacement for raidtools-0.90.
Chapter 4 presents more information about raidtools-1.0 and some new utilities included with it.
raidtools-0.90
Most modern Linux distributions ship with the raidtools package. You can install them from your installation media or download them from your vendor. In a Red Hat CD directory, for example, issue the following command:
# rpm -ihv ./raidtools-0.90-20.i386.rpm
Debian users can use apt-get to install the raidtools:
# apt-get install raidtools2
Note that under Debian, raidtools2, NOT raidtools, provides the 0.90 raidtools pack-age. Please be certain you have the proper package installed.
You can also download the source code and compile the package yourself. The most recent version is available at ftp://ftp.kernel.org/pub/daemons/raid/alpha. As of this writing, the most recent version is ftp://ftp.kernel.org/pub/linux/daemons/raid/alpha/
raidtools-19990824-0.90.tar.gz. You might be aware that a seemingly newer version is available from http://people.redhat.com/mingo/raidtools/. There are several known problems with this version (raidtools-20010914), including its lack of a key manage-ment utility (raidsetfaulty). Use this version at your own risk. The raidtools package should compile without problems on any recent system, though it does require the popt libraries. There are currently no notable configuration options. Download and unpack the archive.
# cd /usr/src/raidtools-YYYYMMDD
# ./configure
# make install
Compilation will leave you with several executables. Of particular note are mkraid, used to create new arrays, and raidstart, used to activate arrays that have already been initialized with mkraid. Installation will copy binaries to /sbin and create sym-bolic links that invoke specific functions for each utility. raidstop, for example, is simply a symbolic link that causes raidstart to invoke a case switch based on the command line. Manual pages are also installed. Chapter 4 covers each command included with the raidtools package in detail.
At the time of this writing, version 1.0 of raidtools has just appeared in Red Hat Linux 8.0. Version 1.0 of raidtools corrects the issues associated with raidtools-20010914 and includes some major improvements, such as bug fixes and a couple of new utilities. lsraid will allow you to query arrays and member disks. It will even let you generate an /etc/raidtab file by querying a running array. raidreconf allows users
to add (or remove) disks to a RAID-0 or RAID-5, usually without data loss. Although these utilities are somewhat new, I’ll discuss them throughout this book, since they are particularly useful and will more than likely become widespread.
/etc/raidtab
/etc/raidtab is raidtools’ primary configuration file. /etc/raidtab provides a function similar to /etc/fstab. It provides the kernel with a description of each software RAID, including its RAID level and a list of member disks. All RAID devices must have an entry in /etc/raidtab to be created or activated using raidtools.
Users create /etc/raidtab by hand, using any text editor. Once /etc/raidtab is created and contains configuration information about arrays, the mkraid command can be used to construct an array based on the parameters defined in /etc/raidtab. mkraid will complain if it cannot find a valid file at /etc/raidtab. This process is similar to cre-ating the boot configuration file /etc/lilo.conf and consequently running /sbin/lilo to rebuild the boot block.
Here is a typical /etc/raidtab that describes a RAID-0 with two ATA component disks:
raiddev /dev/md0 raid-level 0 persistent-superblock 1 chunk-size 64
nr-raid-disks 2 nr-spare-disks 0
device /dev/hda1 raid-disk 0
device /dev/hdb1 raid-disk 1
Each array begins with araiddeventry, followed by a list of array properties and an entry for each array member. A valid entry in /etc/raidtab must contain at least a definedraid-level, achunk-size, the total number of raid disks (nr-raid-disks) and two device/raid-disk pairs. chunk-size is a bit counterintuitive because it applies only to arrays that support disk striping (RAID-0, RAID-4, and RAID-5), yet it is required for any array you define in /etc/raidtab. That’s because the utilities in the raidtools package (mkraid, most notably) check for achunk-sizeeven if the defined RAID level doesn’t support one. So when you work with linear mode or RAID-1, for example, you must provide achunk-size, even though it’s technically only a place-holder.
There are no global options in /etc/raidtab. Each section, or stanza, applies to the last parsedraiddeventry. Once an array is defined, it may be included as a member disk
in a subsequent array section. This allows you to combine multiple arrays into a hybrid array (as in RAID-10, for example).
Chapter 4 provides details about each /etc/raidtab parameter.
raidtools-0.42
It’s unlikely that you will be able to locate a package file for this outdated version of raidtools. So, in most cases, you will be stuck compiling the tools yourself. Down-load version 0.42 of the raidtools (ftp://ftp.kernel.org/pub/linux/daemons/raid/
raidtools-pre3-0.42.tar.gz) and unpack it.
One important caveat when working with this version is that it requires a patched kernel to compile. By default, the configure command assumes that your kernel is in /usr/src/linux, but you can change that by using the--with-linux parameter.
# cd /usr/src/raidtools-0.42
# ./configure --with-linux=/usr/src/linux-2.0.37
# make
# make install
Installation will leave you with an older version of mkraid, as well as several depre-cated utilities that are used to start (mdrun), stop (mdstop), and manage (mdcreate, ckraid) arrays. Manual pages are also installed by default.
Debian users can install version 0.42 of raidtools using apt-get:
# apt-get install raidtools
The rest of this chapter covers the current version of raidtools and the md driver.
However, if you are working with legacy arrays, then you will have to use the 0.42 version of raidtools or mdadm to manage them. Istrongly recommend that you upgrade old arrays whenever possible.
mdadm
Over the years, many users have become quite frustrated by the problems involved in using raidtools and /etc/raidtab to manage software RAID. On small systems, the problems presented by raidtools are tolerable. But on large systems with multiple arrays, each with the potential for dozens of member disks, managing /etc/raidtab can become daunting. mdadm provides a slightly different approach from raidtools.
The idea behind mdadm is that the kernel, as well as administrators, should be able to manage arrays without resorting to a complicated, structured configuration file.
mdadm uses a Universally Unique ID (UUID) to identify each array and member disk. A UUID is a 128-bit number that is guaranteed to be reasonably unique on both the local system and across other systems. It is randomly generated, using sys-tem hardware and timestamps as part of its seed. mdadm uses the UUIDs found in
the array superblock to scan member disks, determining which array they belong to and what the array properties are. Many programs use UUIDs to tag devices uniqely.
In fact, kernel RAID autodetection uses these UUIDs, too. See the uuidgen and lib-uuid manual pages for more information.
mdadm provides all the same functions that raidtools provides, in addition to some new features. The only disadvantage of mdadm is that it’s new. Neil Brown began working on mdadm in June of 2001, and it has undergone only a few revisions.
mdadm can also manage legacy arrays created under the 0.36 md driver, which are normally managed with raidtools-0.42. In this book. I’ll describe how to manage arrays using both raidtools and mdadm. The final decision is yours to make, but I urge you to at least try mdadm.
You can download the most recent mdadm tarball from http://www.cse.unsw.edu.au/
~neilb/source/mdadm/ and issue make install to compile and install mdadm and its documentation. In addition to the binary, some manual pages and example files are included.
# tar xvf ./mdadm-1.0.0.tgz
# cd mdadm-1.0.0.tgz
# make install
Or, you can download and install the package file found in the rpm directory under the above URL.
# rpm -ihv mdadm-1.0.0.i386.rpm
mdadm has five major modes of operation. The first three modes—Create, Assem-ble, and Build—are used to configure and activate arrays. Manage mode is used to manipulate devices in an active array. Follow or Monitor mode allows administra-tors to configure event notification and actions for arrays. The remaining options are used for various housekeeping tasks and are not attached to a specific mode of oper-ation, although the mdadm documentation calls these options Misc mode. The fol-lowing list describes the major modes of operation in mdadm.
Create mode
Used to create a new array. With Create mode, you have the option to define a new array at the command line or create an array defined in /etc/mdadm.conf.
Assemble mode
Used to start an array that already exists.
Build mode
Used only for creating or starting legacy arrays (kernel version 0.36). It should never be used with the 0.90 md driver.
Manage mode
Used to add and remove disks to a running array. This mode is useful for remov-ing failed disks and addremov-ing spare or replacement disks. Manage mode can also be used to mark a member disk as failed. Manage mode replicates the functions of raidtools programs such as raidsetfaulty, raidhotremove, and raidhotadd.
Follow (or Monitor) mode
Used to implement some of mdadm’s best and unique features. With Follow/
Monitor mode, you can daemonize mdadm and configure it to send email alerts to system administrators when arrays encounter errors or fail. You can also use Follow mode to arbitrarily execute commands when a disk fails. For example, you might want to try removing and reinserting a failed disk in an attempt to correct a nonfatal failure without user intervention.
Follow/Monitor mode also allows arrays to share spare disks, a feature that has been lacking in Linux software RAID since its inception. That means you need to provide only one spare disk for a group of arrays or for all arrays. It also means that system administrators don’t have to manually intervene to shuffle around spare disks when arrays fail. Previously, this functionality was available only with hardware RAID. When Follow/Monitor mode is invoked, it polls arrays at regular intervals. When a disk failure is detected on an array without a spare disk, mdadm will remove an available spare disk from another array and insert it into the array with the failed disk.
The remaining options, which fall under the Misc mode, are used for tasks that include stopping arrays, marking arrays as read-only or read/write, and clearing the RAID superblock from a disk.
/etc/mdadm.conf
/etc/mdadm.conf is the primary configuration file in mdadm. Unlike /etc/raidtab, mdadm does not rely on /etc/mdadm.conf to create or manage arrays. Rather, mdadm.
conf is simply an extra way of keeping track of software RAIDs. Using a configura-tion file with mdadm is useful, but not required. Having a configuraconfigura-tion file means that you can quickly manage arrays without spending extra time figuring out what the array properties are and where disks belong. Unlike the configuration file for raidtools, mdadm.conf is concise and simply lists disks and arrays. The configuration file usually contains two types of lines, each starting with either the DEVICEorARRAY keyword. Whitespace separates the keyword from the configuration information.
DEVICElines specify a list of devices that are potential member disks.ARRAYlines spec-ify device entries for arrays, as well as identifier information. This information can include lists of one or more UUIDs, md device minor numbers, or member devices.
A simple mdadm.conf file might look like this:
DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1 ARRAY /dev/md1 devices=/dev/sdc1,/dev/sdd1
In general, it’s best to set up an /etc/mdadm.conf file after you have created an array.
You should update the file when new arrays are created. Without an mdadm.conf file, you’d need to specify more detailed information about an array on the com-mand line in order to activate it. That means you’d have to remember which devices belonged to which arrays. That could be a hassle on systems with a lot of disks.
mdadm even provides an easy way to generateARRAYlines. The output is a single long line, but I have broken it here to fit the page:
# mdadm --detail --scan
ARRAY /dev/md0 level=raid0 num-devices=2 \ UUID=410a299e:4cdd535e:169d3df4:48b7144a
If there were multiple arrays running on the system, mdadm would generate an array line for each one. Thus, after you’re done building arrays, you could redirect the out-put of mdadm --detail --scan to /etc/mdadm.conf. Just make sure you manually create aDEVICE entry as well. Chapter 4 contains a complete reference for mdadm.conf.