Linux DM-Multipath Configuration for CDP/NSS Best Practice Guide

FalconStor Software, Inc. 2 Huntington Quadrangle, Suite 2S01 Melville, NY 11747 Phone: 631-777-5188 Fax: 631-501-7633 Website: www.falconstor.com

Copyright 2011 FalconStor Software. All Rights Reserved. FalconStor Software, FalconStor, and IPStor are registered trademarks of FalconStor Software, Inc. in the United States and other countries. Windows is a registered trademark of Microsoft Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. FalconStor Software reserves the right to make changes in the information contained in this publication without prior notice. The reader should in all cases consult FalconStor to determine whether any such changes have been made. 6.17.2011

Contents
Introduction ............................................................................................... 1
Check DM-Multipath package ................................................................................................. 1

Configure DM-Multipath for FALCON Disks ........................................... 3

Create the DM-Multipath configuration file for FALCON disks ............................................. 3 Run the script ...................................................................................................................... 3 Add parameters for a storage device ..................................................................................... 9 Reload multipath.conf ......................................................................................................... 9

Add FALCON Disks to the DM-Multipath Configuration ...................... 10

1. 2. 3. 4. 5. Configure devices ............................................................................................................10 Restart the multipath daemon .........................................................................................11 Reload the configuration file ...........................................................................................11 Check configuration ........................................................................................................11 Start the DM-Multipath daemon at boot time .................................................................12

Configure HBAs ...................................................................................... 13

Tune HBAs for fast path switch .............................................................................................13 Tune HBAs for a single-path host..........................................................................................13

Linux DM-Multipath Configuration for CDP/NSS

Introduction
Linux Device Mapper Multipathing (DM-Multipath) is a native path-failover solution available with Red Hat Enterprise Linux 4.0 U2, SUSE SLES 9.0 PS2, and above. DM-Multipath makes it possible to configure multiple I/O paths between the storage server and storage in order to provide transparent failover and failback. DM-Multipath supports path redundancy and throughput aggregation. This document describes best practices for configuring Linux DM-Multipath to manage multipathing for storage provisioned by CDP/ NSS. Two options are included:

If DM-Multipath is being configured for use with FALCON IPStor Disks, follow instructions in Configure DM-Multipath for FALCON Disks to run the configuration script. You can add disks other than FALCON disks at a later time. If DM-Multipath is being configured/has been configured for disks other than FALCON IPStor Disks, follow instructions in Add FALCON Disks to the DM-Multipath Configuration to include FALCON IPStor Disks in the devices section of the configuration file.

For more information on the DM-Multipath configuration file, refer to documentation for your operating system.

Check DM-Multipath package

DM-Multipath includes packages as shown below. Make sure you have the latest version of each package and update if necessary.

Red Hat Enterprise Linux

device-mapper device-mapper-multipath

SUSE Linux Enterprise Server device-mapper multipath-tools

Debian Linux Enterprise Server (Ubuntu) multipath-tools

OPERATING SYSTEM DOCUMENTATION

Red Hat Enterprise Linux 4 DM Multipath http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4.8/pdf/DM_Multipath.pdf Red Hat Enterprise Linux 5.1 DM Multipath http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/enUS/RHEL510/pdf/DM_Multipath.pdf How to set up/use multipathing on SUSE Linux Enterprise Server 9 http://support.novell.com/techcenter/sdb/en/2005/04/sles_multipathing.html Linux DM-Multipath Configuration for CDP/NSS
1

Contents

Storage Administration Guide - SUSE Linux Enterprise Server 10 http://www.novell.com/documentation/sles10/pdfdoc/stor_evms/stor_evms.pdf Storage Administration Guide - SUSE Linux Enterprise Server 11 http://www.novell.com/documentation/sles11/pdfdoc/stor_admin/stor_admin.pdf

ADDITIONAL REFERENCES
Using Device-Mapper Multipath http://www.redhat.com/docs/manuals/csgfs/browse/4.6/DM_Multipath/index.html Online Storage Guide for Red Hat Enterprise Linux 5 http://www.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/html/Online_Storage_Reconfiguration_Guide/index.html

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

Perform the following operations on any Linux client machine that is using DM-Multipath.

Create the DM-Multipath configuration file for FALCON disks

The script updates the configuration file and restarts the daemon to initiate a new configuration. It accepts an input parameter to identify disks for which you do not want multipathing enabled, such as a local SCSI drive such as /dev/sda. In addition, the script includes default values and a multipath section for storage devices; you can replace these defaults in the Devices section of /etc/multipath.conf and also remove comments.

Run the script

DM-Multipath also supports group paths based on Asymmetric Logical Unit Access (ALUA) state. To run the script without ALUA, enter the following:
sh multipath.sh nonalua ^sda[0-9] ^sdb[0-9]

To run the script with ALUA support, enter the following:

sh multipath.sh alua ^sda[0-9] ^sdb[0-9]

This section of the command represents the devices to be blacklisted:

^sda[0-9] or ^sdb[0-9]

USE THIS SCRIPT:

#!/bin/sh configure() { touch multipath.conf # defaults for user_friendly_names echo "defaults {" > multipath.conf echo " user_friendly_names yes" >> multipath.conf echo "}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

# blacklisting devices echo "blacklist {" >> multipath.conf for i in $* do if [ $i != alua -a $i != nonalua ] then echo " devnode \"$i\"" >> multipath.conf fi done echo "}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # uncomment the defaults section to override default values echo "#defaults {" >> multipath.conf echo "# udev_dir echo "# polling_interval echo "# selector multipath.conf echo "# path_grouping_policy echo "# path_checker echo "# rr_min_io echo "# max_fds echo "# rr_weight echo "# failback echo "# no_path_retry echo "# user_friendly_names echo "# features multipath.conf echo "#}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # multipaths section to override devices section echo "#multipaths {" >> multipath.conf echo "# multipath {" >> multipath.conf echo "# wwid 3600508b4000156d70001200000b0000" >> multipath.conf echo "# alias yellow" >> multipath.conf echo "# path_grouping_policy multibus" >> multipath.conf echo "# path_checker readsector0" >> multipath.conf echo "# path_selector \"round-robin 0\"" >> multipath.conf echo "# failback manual" >> multipath.conf echo "# rr_weight priorities" >> multipath.conf echo "# no_path_retry 5" >> multipath.conf /dev" >> multipath.conf 10" >> multipath.conf \"round-robin 0\"" >> multibus" >> multipath.conf tur" >> multipath.conf 100" >> multipath.conf 8192" >> multipath.conf priorities" >> multipath.conf immediate" >> multipath.conf queue" >> multipath.conf yes" >> multipath.conf \"1 queue_if_no_path\"" >>

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

echo "# }" >> multipath.conf echo "# multipath {" >> multipath.conf echo "# wwid 1DEC_____321816758474" >> multipath.conf echo "# alias red" >> multipath.conf echo "# rr_weight priorities" >> multipath.conf echo "# }" >> multipath.conf echo "#}" >> multipath.conf echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf # values per storage device override the defaults section echo "devices {" >> multipath.conf echo " device {" >> multipath.conf echo " vendor echo " product multipath.conf echo " path_selector multipath.conf echo " path_grouping_policy

\"FALCON\"" >> multipath.conf \"IPSTOR DISK\"" >> \"round-robin 0\"" >> multibus" >> multipath.conf

if [ -f /etc/redhat-release ] then if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then if [ `cat /etc/redhat-release | awk '{print $7}' | awk -F"." '{print $1}'` -eq 6 ] then if [ $1 = alua ] then echo " prio alua" >> multipath.conf else echo " prio const" >> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi else if [ $1 = nonalua ] then echo " \"/bin/true\"" >> multipath.conf else prio_callout

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/SuSE-release ] then if [ `cat /etc/SuSE-release | awk '{print $3}' | grep 11` -eq 11 ] then if [ $1 = alua ] then echo " >> multipath.conf else echo " >> multipath.conf fi else if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi if [ -f /etc/lsb-release ] then if [ `cat /etc/lsb-release | grep -c Ubuntu` -ge 1 ] then if [ $1 = nonalua ] then echo " prio_callout \"/bin/true\"" >> multipath.conf else echo " prio_callout \"/sbin/mpath_prio_alua %d\"" >> multipath.conf fi fi fi prio const" prio alua"

echo " echo " echo " echo " echo " multipath.conf echo "

path_checker rr_min_io rr_weight failback features }" >> multipath.conf

tur" >> multipath.conf 100" >> multipath.conf priorities" >> multipath.conf immediate" >> multipath.conf \"1 queue_if_no_path\"" >>

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

echo >> multipath.conf echo >> multipath.conf echo >> multipath.conf device return }

# restart the multipathd service

restart() { if [ -f /etc/redhat-release ] then if [ `cat /etc/redhat-release | awk '{print $1}'` = "Red" ] then if [ `cat /etc/redhat-release | awk '{print $7}' | awk F"." '{print $1}'` -eq 6 ] then mpathconf --enable fi fi service multipathd restart chkconfig multipathd on echo "=======================================================================" echo " Reloading dm-multipath"

echo "=======================================================================" multipath -F multipath -r multipath -ll elif [ -f /etc/SuSE-release ] then service multipathd restart chkconfig multipathd on echo "=======================================================================" echo " Reloading dm-multipath" echo "=======================================================================" multipath -F multipath -r multipath -ll else

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

sudo multipathd restart echo "=======================================================================" echo " Reloading dm-multipath" echo "=======================================================================" sudo multipath -F sudo multipath sudo multipath -ll fi return } device() { echo "}" >> multipath.conf if [ -f /etc/lsb-release ] then sudo mv multipath.conf /etc/. else mv multipath.conf /etc/. fi restart } usage() { echo echo echo exit } if [ $# -lt 2 ]; then usage fi if [ $# -eq 2 ]; then if [ $1 != alua -a $1 != nonalua ]; then usage fi fi if [ $# -gt 2 ]; then if [ $1 != alua -a $1 != nonalua ]; then usage fi fi if [ -f "/etc/multipath.conf" ] then if [ -f /etc/lsb-release ] then sudo mv /etc/multipath.conf /etc/multipath.conf.old else mv /etc/multipath.conf /etc/multipath.conf.old fi "multipath.sh alua|nonalua \"blacklist devices\"" "example:" "multipath.sh alua \"^sda[0-9]\"" 0

Linux DM-Multipath Configuration for CDP/NSS

Configure DM-Multipath for FALCON Disks

fi configure $1 $2 exit 0

Add parameters for a storage device

After you run the script, you can edit the devices portion of /etc/multipath.conf to add device parameters for storage devices other than FALCON IPSTOR DISK.

Reload multipath.conf
To ensure the modified configuration file is read and adopted after modifying device parameters, reload multipath.conf by entering the following commands:
multipath F (Flush the impending IO and remove devices from the device-mapper.) multipath r (multipath in Ubuntu) (Reload the configuration file to apply new

changes.) Note: There may be some outstanding IOs on multipath devices; if multipath-F fails, restart the multipath daemon and then reload the configuration file:
service multipathd restart (e.g., RHES and SLES)

To list multipath devices, enter the following command:

multipath ll (Display the long list of multipath devices.)

Note: When the user_friendly_names option in the multipath configuration file is set to yes, the name of a multipath device is in the form mpathn. For versions of Red Hat Enterprise Linux earlier than version 6, n is an integer. For Red Hat Enterprise Linux 6, n is an alphabetic character; therefore, the name of a multipath device might be mpatha or mpathb. For more information, refer to http://docs.redhat.com/docs/enUS/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/MPIO_Overview.html.

Linux DM-Multipath Configuration for CDP/NSS

Add FALCON Disks to the DM-Multipath Configuration

You can override the default configuration values for DM-Multipath by editing /etc/multipath.conf to suit your environment. Make a backup of the original file before making any changes.

1. Configure devices
When DM-Multipath is being used to support storage devices other than IPSTOR DISK, you must define parameters for IPSTOR DISK in the devices section of the configuration file. Add the section similar to the one shown below to include CDP/NSS-managed disks when generating multipath devices. In addition, you can set the appropriate parameters for a failover environment. There may be slight differences for other operating systems, with and without ALUA. This example shows the device section of the /etc/multipath.conf file including with ALUA for Redhat 5.6 and below:
devices { device { vendor product getuid_callout prio_callout features hardware_handler path_selector path_grouping_policy failback rr_weight rr_min_io path_checker } } "FALCON" "IPSTOR DISK" "/sbin/scsi_id -g -u -s /block/%n" "/sbin/mpath_prio_alua %d" "1 queue_if_no_path" "0" "round-robin 0" group_by_prio immediate prioritiesuniform 100 tur

CDP/NSS v6.15 and above supports asymmetric logical unit access (ALUA). To enable ALUA, set the priority callout program or the priority option as shown below. If you are using the iSCSI protocol, set the path_grouping_policy to multibus for Red Hat Enterprise Linux 4 update 7 and earlier. Linux DM-Multipath Configuration for CDP/NSS
10

Add FALCON Disks to the DM-Multipath Configuration The following parameter priority callout (prio_callout) must be changed to support specific operating system versions and ALUA: With ALUA:

SLES 10 SP2 and later prio alua RED HAT 6 and above prio alua Other - prio_callout /sbin/mpath_prio_alua %d

Without ALUA: SLES 10 SP2 and later prio const RED HAT 6 and above prio const Other - prio_callout /bin/true

2. Restart the multipath daemon

Whenever you modify the configuration file in any way, you must enter this command to restart the multipath daemon when you are done:
/etc/init.d/multipathd restart

3. Reload the configuration file

multipath F (Flush all inpending IOs) multipath r (multipath in Ubuntu) (Reload the new configuration)

4. Check configuration
Enter this command to display current multipathing status:
multipath -ll mpath1 (36000d7780000e5dc65d6877cf8618d14) dm-0 FALCON,IPSTOR DISK [size=9.8G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=200][active] \_ 3:0:0:0 sdb 8:16 \_ 4:0:0:0 sdc 8:32 \_ 3:0:1:0 sdd 8:48 \_ 4:0:1:0 sde 8:64 [active][ready] [active][ready] [active][ready] [active][ready]

Some useful commands are listed below:

multipath F Flush impending IO and remove devices from the device-mapper. multipath r Reload the configuration file to apply new changes. multipath-l - List multipath devices.

Linux DM-Multipath Configuration for CDP/NSS

11

Add FALCON Disks to the DM-Multipath Configuration

multipath ll - Display the long list of multipath devices. dmsetup Configure DM devices. pvcreate Initialize physical volumes for use by LVM.

5. Start the DM-Multipath daemon at boot time

Use the chkconfig command to configure the run level of the DM-Multipath daemon service.

Red Hat Enterprise Linux

chkconfig add multipathd chkconfig multipathd on chkconfig add boot.multipath chkconfig add multipathd chkconfig boot.multipath on chkconfig multipathd on

SUSE Linux Enterprise Server

Linux DM-Multipath Configuration for CDP/NSS

12

Configure HBAs
Perform the following operations on any client machine that is using DM-Multipath.

Tune HBAs for fast path switch

For the multipath environment, path failure must be detected as quickly as possible in order for I/O to be redistributed to a healthy path. It is recommended that you configure the time-out setting to disable failover at the HBA level, which allows the failure of the HBA or cable to be propagated to the multipath layer. For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or /etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the system using the mkinitrd command to reflect the change.
options qla2xxx qlport_down_retry=1

Tune HBAs for a single-path host

For the single-path environment, path failure should be delayed as long as possible in order for I/O to be resumed by the underlying storage subsystem. It is recommended that you configure the time-out setting to prolong the HBA retry period. For example, setting driver parameters of a QLogic qla2xxx HBA in /etc/modprobe.conf or /etc/modprobe.conf.local is recommended. You may need to recreate the INITRD on the system using the mkinitrd command to reflect the change.
options qla2xxx qlport_down_retry=255

For an Emulex HBA, the following is recommended:

options lpfc_nodev_tmo=60

Linux DM-Multipath Configuration for CDP/NSS

13