Linux Client DM-Multipath Best Practice for CDP-NSS

(1)

(2)

Linux DM-Multipath Configuration for CDP/NSS

Best Practice Guide

FalconStor Software, Inc.

2 Huntington Quadrangle, Suite 2S01 Melville, NY 11747

Phone: 631-777-5188 Fax: 631-501-7633

Website: www.falconstor.com

FalconStor Software, FalconStor, and IPStor are registered trademarks of FalconStor Software, Inc. in the United States and other countries.

Windows is a registered trademark of Microsoft Corporation.

All other brand and product names are trademarks or registered trademarks of their respective owners. FalconStor Software reserves the right to make changes in the information contained in this publication without prior notice. The reader should in all cases consult FalconStor to determine whether any such changes have been made.

(3)

Introduction

Linux Device Mapper Multipathing (DM-Multipath) is a native path-failover solution available with Red Hat Enterprise Linux 4.0 U2, SUSE SLES 9.0 PS2, and above. DM-Multipath makes it

possible to configure multiple I/O paths between the storage server and storage in order to provide transparent failover and failback. DM-Multipath supports path redundancy and throughput aggregation.

This document describes best practices for configuring Linux DM-Multipath to manage multipathing for storage provisioned by CDP/ NSS. Two options are included:

 If DM-Multipath is being configured for use with FALCON IPStor Disks, follow instructions in

Configure DM-Multipath for FALCON Disks to run the configuration script. You can add disks other than FALCON disks at a later time.

 If DM-Multipath is being configured/has been configured for disks other than FALCON

IPStor Disks, follow instructions in Add FALCON Disks to the DM-Multipath Configuration to include FALCON IPStor Disks in the devices section of the configuration file.

For more information on the DM-Multipath configuration file, refer to documentation for your operating system.

Check DM-Multipath package

DM-Multipath includes packages as shown below. Make sure you have the latest version of each package and update if necessary.

 Red Hat Enterprise Linux  device-mapper

 device-mapper-multipath  SUSE Linux Enterprise Server

 device-mapper  multipath-tools

 Debian Linux Enterprise Server (Ubuntu)  multipath-tools

O

PERATING

S

YSTEM

D

OCUMENTATION

 Red Hat Enterprise Linux 4 DM Multipath

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4.8/pdf/DM_Multipath.pdf

 Red Hat Enterprise Linux 5.1 DM Multipath

http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/pdf/DM_Multipath.pdf

 How to set up/use multipathing on SUSE Linux Enterprise Server 9

(5)

Contents

 Storage Administration Guide - SUSE Linux Enterprise Server 10

http://www.novell.com/documentation/sles10/pdfdoc/stor_evms/stor_evms.pdf

 Storage Administration Guide - SUSE Linux Enterprise Server 11

http://www.novell.com/documentation/sles11/pdfdoc/stor_admin/stor_admin.pdf

A

DDITIONAL

R

EFERENCES

 Using Device-Mapper Multipath

http://www.redhat.com/docs/manuals/csgfs/browse/4.6/DM_Multipath/index.html

 Online Storage Guide for Red Hat Enterprise Linux 5

(6)

Configure DM-Multipath

for FALCON Disks

Perform the following operations on any Linux client machine that is using DM-Multipath.

Create the DM-Multipath configuration file for FALCON disks

The script updates the configuration file and restarts the daemon to initiate a new configuration. It accepts an input parameter to identify disks for which you do not want multipathing enabled, such as a local SCSI drive such as /dev/sda .

In addition, the script includes default values and a multipath section for storage devices; you can replace these defaults in the Devices _{section of /etc/multipath.conf and also remove}

comments.

Run the script

DM-Multipath also supports group paths based on Asymmetric Logical Unit Access (ALUA) state.

To run the script without ALUA, enter the following:

sh multipath.sh nonalua “^sda[0-9] ^sdb[0-9]”

To run the script with ALUA support, enter the following:

sh multipath.sh alua “^sda[0-9] ^sdb[0-9]”

This section of the command represents the devices to be blacklisted:

^sda[0-9] or ^sdb[0-9]

U

SE THIS SCRIPT

:

#!/bin/sh configure() {

touch multipath.conf

# defaults for user_friendly_names echo "defaults {" > multipath.conf

echo " user_friendly_names yes" >> multipath.conf echo "}" >> multipath.conf

echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf

(7)

Configure DM-Multipath for FALCON Disks

# blacklisting devices

echo "blacklist {" >> multipath.conf for i in $*

do

if [ $i != alua -a $i != nonalua ] then

echo " devnode \"$i\"" >> multipath.conf fi done echo "}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf

# uncomment the defaults section to override default values echo "#defaults {" >> multipath.conf

echo "# udev_dir /dev" >> multipath.conf echo "# polling_interval 10" >> multipath.conf echo "# selector \"round-robin 0\"" >> multipath.conf

echo "# path_grouping_policy multibus" >> multipath.conf echo "# path_checker tur" >> multipath.conf

echo "# rr_min_io 100" >> multipath.conf echo "# max_fds 8192" >> multipath.conf

echo "# rr_weight priorities" >> multipath.conf echo "# failback immediate" >> multipath.conf echo "# no_path_retry queue" >> multipath.conf echo "# user_friendly_names yes" >> multipath.conf echo "# features \"1 queue_if_no_path\"" >> multipath.conf

echo "#}" >> multipath.conf echo >> multipath.conf

echo >> multipath.conf echo >> multipath.conf

# multipaths section to override devices section echo "#multipaths {" >> multipath.conf

echo "# multipath {" >> multipath.conf

echo "# wwid

3600508b4000156d70001200000b0000" >> multipath.conf

echo "# alias yellow" >> multipath.conf echo "# path_grouping_policy multibus" >>

multipath.conf

echo "# path_checker readsector0" >> multipath.conf

echo "# path_selector \"round-robin 0\"" >> multipath.conf

echo "# failback manual" >> multipath.conf echo "# rr_weight priorities" >>

multipath.conf

(8)

echo "# }" >> multipath.conf

echo "# multipath {" >> multipath.conf

echo "# wwid 1DEC_____321816758474" >> multipath.conf

echo "# alias red" >> multipath.conf echo "# rr_weight priorities" >>

multipath.conf echo "# }" >> multipath.conf echo "#}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf

# values per storage device override the defaults section echo "devices {" >> multipath.conf

echo " device {" >> multipath.conf

echo " vendor \"FALCON\"" >> multipath.conf echo " product \"IPSTOR DISK\"" >>

multipath.conf

echo " path_selector \"round-robin 0\"" >> multipath.conf

echo " path_grouping_policy multibus" >> multipath.conf if [ -f /etc/redhat-release ]

then

if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then

if [ `cat /etc/redhat-release | awk '{print $7}' | awk -F"." '{print $1}'` -eq 6 ]

then if [ $1 = alua ] then echo " prio alua" >> multipath.conf else echo " prio const" >> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else

(9)

Configure DM-Multipath for FALCON Disks echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/SuSE-release ] then

if [ `cat /etc/SuSE-release | awk '{print $3}' | grep 11` -eq 11 ]

then

if [ $1 = alua ] then

echo " prio alua"

>> multipath.conf

else

echo " prio const"

>> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/lsb-release ] then

if [ `cat /etc/lsb-release | grep -c Ubuntu` -ge 1 ] then

if [ $1 = nonalua ] then

echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi

echo " path_checker tur" >> multipath.conf echo " rr_min_io 100" >> multipath.conf

echo " rr_weight priorities" >> multipath.conf echo " failback immediate" >> multipath.conf echo " features \"1 queue_if_no_path\"" >> multipath.conf

(10)

Configure DM-Multipath for FALCON Disks echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf device return }

# restart the multipathd service

restart() {

if [ -f /etc/redhat-release ] then

if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then

if [ `cat /etc/redhatrelease | awk '{print $7}' | awk -F"." '{print $1}'` -eq 6 ]

then

mpathconf --enable fi

fi

service multipathd restart chkconfig multipathd on echo

"======================================================================="

echo " Reloading dm-multipath"

echo "=======================================================================" multipath -F multipath -r multipath -ll elif [ -f /etc/SuSE-release ] then

service multipathd restart chkconfig multipathd on echo

"======================================================================="

echo "=======================================================================" multipath -F multipath -r multipath -ll else

(11)

sudo multipathd restart echo

"======================================================================="

echo "=======================================================================" sudo multipath -F sudo multipath sudo multipath -ll fi return } device() { echo "}" >> multipath.conf if [ -f /etc/lsb-release ] then

sudo mv multipath.conf /etc/. else mv multipath.conf /etc/. fi restart } usage() {

echo "multipath.sh alua|nonalua \"blacklist devices\"" echo "example:"

echo "multipath.sh alua \"^sda[0-9]\"" exit 0 } if [ $# -lt 2 ]; then usage fi if [ $# -eq 2 ]; then

if [ $1 != alua -a $1 != nonalua ]; then usage

fi fi

if [ $# -gt 2 ]; then

if [ $1 != alua -a $1 != nonalua ]; then usage fi fi if [ -f "/etc/multipath.conf" ] then if [ -f /etc/lsb-release ] then

sudo mv /etc/multipath.conf /etc/multipath.conf.old else

(12)

fi

configure $1 $2 exit 0

Add parameters for a storage device

After you run the script, you can edit the devices portion of /etc/multipath.conf to add

device parameters for storage devices other than FALCON IPSTOR DISK.

Reload multipath.conf

To ensure the modified configuration file is read and adopted after modifying device parameters, reloadmultipath.conf by entering the following commands:

multipath –F (Flush the impending IO and remove devices from the device-mapper.) multipath –r (“multipath” in Ubuntu) (Reload the configuration file to apply new

changes.)

Note: There may be some outstanding IOs on multipath devices; if “multipath -F” fails, restart the multipath daemon and then reload the configuration file:

service multipathd restart (e.g., RHES and SLES)

To list multipath devices, enter the following command:

multipath –ll (Display the long list of multipath devices.)

Note: When the user_friendly_names option in the multipath configuration file is set to yes, the name of a multipath device is in the form mpathn. For versions of Red Hat

Enterprise Linux earlier than version 6, nis an integer. For Red Hat Enterprise Linux 6, nis

an alphabetic character; therefore, the name of a multipath device might be mpatha or mpathb. For more information, refer to

(13)

Add FALCON Disks to

the DM-Multipath

Configuration

You can override the default configuration values for DM-Multipath by editing

/etc/multipath.conf to suit your environment. Make a backup of the original file before

making any changes.

1. Configure devices

When DM-Multipath is being used to support storage devices other than IPSTOR DISK, you must define parameters for IPSTOR DISK in the “devices” section of the configuration file. Add

the section similar to the one shown below to include CDP/NSS-managed disks when

generating multipath devices. In addition, you can set the appropriate parameters for a failover environment. There may be slight differences for other operating systems, with and without ALUA.

This example shows the “device” section of the /etc/multipath.conf file including with ALUA for Redhat 5.6 and below:

devices {

device {

vendor "FALCON"

product "IPSTOR DISK"

getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout "/sbin/mpath_prio_alua %d" features "1 queue_if_no_path" hardware_handler "0" path_selector "round-robin 0" path_grouping_policy group_by_prio failback immediate rr_weight prioritiesuniform rr_min_io 100 path_checker tur } }

CDP/NSS v6.15 and above supports asymmetric logical unit access (ALUA). To enable ALUA, set the priority callout program or the priority option as shown below. If you are using the iSCSI protocol, set the path_grouping_policy to multibusfor Red Hat Enterprise Linux 4 update 7

(14)

Add FALCON Disks to the DM-Multipath Configuration The following parameter “priority callout (prio_callout)” must be changed to support specific

operating system versions and ALUA: With ALUA:

 SLES 10 SP2 and later – prio “alua”  RED HAT 6 and above – prio “alua”

 Other - prio_callout “/sbin/mpath_prio_alua %d”

Without ALUA:

 SLES 10 SP2 and later – prio const  RED HAT 6 and above – prio const  Other - prio_callout /bin/true

2. Restart the multipath daemon

Whenever you modify the configuration file in any way, you must enter this command to restart the multipath daemon when you are done:

/etc/init.d/multipathd restart

3. Reload the configuration file

multipath –F (Flush all inpending IOs)

multipath –r (“multipath” in Ubuntu) (Reload the new configuration)

4. Check configuration

Enter this command to display current multipathing status:

multipath -ll

mpath1 (36000d7780000e5dc65d6877cf8618d14) dm-0 FALCON,IPSTOR DISK [size=9.8G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=200][active] \_ 3:0:0:0 sdb 8:16 [active][ready] \_ 4:0:0:0 sdc 8:32 [active][ready] \_ 3:0:1:0 sdd 8:48 [active][ready] \_ 4:0:1:0 sde 8:64 [active][ready]

Some useful commands are listed below:

 multipath –F – Flush impending IO and remove devices from the device-mapper.  multipath –r – Reload the configuration file to apply new changes.

(15)

Add FALCON Disks to the DM-Multipath Configuration

 multipath –ll - Display the long list of multipath devices.  dmsetup – Configure DM devices.

 pvcreate – Initialize physical volumes for use by LVM.

5. Start the DM-Multipath daemon at boot time

Use the chkconfig command to configure the run level of the DM-Multipath daemon service.

 Red Hat Enterprise Linux

 chkconfig add multipathd  chkconfig multipathd on  SUSE Linux Enterprise Server

 chkconfig add boot.multipath  chkconfig add multipathd  chkconfig boot.multipath on  chkconfig multipathd on

(16)

Configure HBAs

Perform the following operations on any client machine that is using DM-Multipath.

Tune HBAs for fast path switch

For the multipath environment, path failure must be detected as quickly as possible in order for I/O to be redistributed to a healthy path. It is recommended that you configure the time-out setting to disable failover at the HBA level, which allows the failure of the HBA or cable to be propagated to the multipath layer.

For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or

/etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the

system using the mkinitrd command to reflect the change. options qla2xxx qlport_down_retry=1

Tune HBAs for a single-path host

For the single-path environment, path failure should be delayed as long as possible in order for I/O to be resumed by the underlying storage subsystem. It is recommended that you configure the time-out setting to prolong the HBA retry period.

For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or /etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the

system using the mkinitrd command to reflect the change. options qla2xxx qlport_down_retry=255

For an Emulex HBA, the following is recommended:

Linux Client DM-Multipath Best Practice for CDP-NSS

Linux DM-Multipath Configuration for CDP/NSS

Best Practice Guide

Contents

Introduction ... 1

Configure DM-Multipath for FALCON Disks ... 3

Add FALCON Disks to the DM-Multipath Configuration ... 10

Configure HBAs ... 13

Introduction

Check DM-Multipath package

O

S

D

A

R

Configure DM-Multipath

for FALCON Disks

Create the DM-Multipath configuration file for FALCON disks

Run the script

U

:

Add parameters for a storage device

Reload multipath.conf

Add FALCON Disks to

the DM-Multipath

Configuration

1. Configure devices

2. Restart the multipath daemon

3. Reload the configuration file

4. Check configuration

5. Start the DM-Multipath daemon at boot time

Configure HBAs

Tune HBAs for fast path switch

Tune HBAs for a single-path host