Escolar Documentos
Profissional Documentos
Cultura Documentos
I/O for Logical Unit A, going directly to Node A owning the Logical Unit
In this acronym, Logical Unit Access part is well understood but Asymmetrical at
the beginning does make it sound little complicated.
So, What exactly is asymmetrical? Well, if you google for the word asymmetrical
you will find many definitions around this word but in general, most definitions are
trying to convey the same message. In a nutshell it means when the two halves are
not equal, from storage multipathing perspective it means Not all paths available
to a LUN, necessarily have same equal access path.
In this document we shall try and cover all aspects around ALUA & multipathing.
Ashwin Pawar
Sep, 2014
www.simplysan.com
Why multipathing & ALUA needed
When you have a device [LUN] presented to the host using multiple ports [multiple
paths] it does add complexity at the OS level. In other words, getting the device to
show up properly as a single [pseudo] device is one thing and then have the OS
understand its port characteristics is virtually impossible without breaking into the
operating system code and writing a module to sit in the storage stack to tap into these
features. This led to the development of Multipathing, which basically provides high
availability, performance and fault tolerance at the front-end/Host side.
In configurations in which there are multiple paths to a logical unit, each path might
have different bandwidth and latency characteristics. Due to these different
characteristics, the target ports might need to indicate which path is most efficient.
Also, if a failure occurs on a target port, the SCSI target device might change its
internal configuration, causing a path to go offline.
The access characteristics of a path are defined by the target port's asymmetric access
state, which is returned by the SCSI command REPORT TARGET PORT GROUPS
(RTPG). Access states are set through the command SET TARGET PORT GROUPS
(STPG) or by the target device. This will be covered in detailed in the later pages.
TPG: TargetPortalGroup
Following is my own illustration of ALUA for learning purpose only.
Figure 1
I/O for Logical Unit 'A' going directly to the Node 'A' owning the Lun.
TPG: TargetPortGroup
ALUA allows you to see any given LUN via both storage processors as active but
only one of these storage processors owns the LUN and because of that there will
be optimized and non-optimized paths. The optimized paths are the ones with a direct
path to the storage processor [Node-A] that owns the LUN. The non-optimized paths
have a connection with the storage processor that does not own the LUN but have an
indirect path to the storage processor that does own it via an interconnect bus.
Which storage array types are candidates for ALUA?
Active/Active Storage Systems are the ideal candidate for ALUA. That means, ALUA
does not apply to Active/Passive Storage Systems. You might ask, why is that?
Therefore, for storage devices with ALUA feature implemented, there must be at least
two target port groups, first one would be direct/Optimized TPG [Controller A] and
second one Indirect/Un-optimized TPG [Controller B]
Multi-pathing software can query ALUA compliant arrays to load balance only
between paths connected to the preferred controller and use the paths to the non-
preferred controller for automatic path failover if all of the paths to the primary
controller fail.
Symmetrical Active/Active (SAA) storage systems:
These types of arrays do not have a primary or preferred controller per LUN. I/O
requests can be issued over all paths mapped to a LUN. Some models of the HP
StorageWorks XP Disk Array family are symmetrical active/active arrays.
Microsoft Windows 2008 Introduced Native MPIO with feature that utilizes ALUA
for path selection. Hence, if you are running Windows 2008, you don't have to worry
about installing vendor specific DSM for Active/Active storage systems that supports
ALUA. Native MPIO can handle this for you. This has been made possible through
the standardization of ALUA in SCSI-3 specification.
Similarly, many OS vendors are now providing ALUA feature in the Native
Multipathing software. Following is the rough estimation of the time line since
various vendors adopted ALUA.
As ALUA support has been widely adopted and delivered in the Host side OS, no
special storage plug-ins are required, which means volume manager and arrays based
plug-ins are becoming less dominant or unneeded for Active/Active Storage arrays.
Implicit ALUA: With the implicit ALUA style, the host multipathing software can
monitor the path states but cannot change them, either automatically or manually. Of
the active paths, a path may be specified as preferred (optimized), and as non-
preferred (non-optimized). If there are active preferred paths, then only these paths
will receive commands, and will be load balanced to evenly distribute the commands.
If there are no active preferred paths, then the active non-preferred paths are used in a
round-robin fashion. If there are no active non-preferred paths, then the LUN cannot
be accessed until the controller activates its standby paths.
Explicit ALUA: Devices allow the host to use the Set Target Port Group task
management command to set the Target Port Group's state. In implicit ALUA, the
target device itself manages a devices Target Port Group states.
Target Port Groups (TPG) allows path grouping and dynamic load balancing. Each
port in the same TPG has the same port state, which can be one of these:
1. Active/Optimized
2. Active/Non-Optimized
3. Standby
4. Unavailable
5. In-transition
Active/Optimized = Local/Fast/Primary
Active/Non-Optimized = Partner/Proxy/Slow/Secondary
Unavailable = Cluster IC is down, path is not functional
Transitioning = Path is transitioning to another state
Asymmetric Logical Unit Access (ALUA) was included in SPC-2 and updated in
the SPC-3 specification. This interface allows an initiator to discover Target Port
Groups - groups of ports expected to provide common failover behaviour for specific
logical units. The Target Port Group Support (TPGS) field in the standard INQUIRY
response describes the logical unit's adherence to the standard, whether the logical
unit provides symmetric or asymmetric access, and whether the logical unit uses
explicit or implicit failover. The standard provides a standard explicit failover
command, and commands to determine which ports are members of a target port
group and other information about the multipath configuration.
In short - ALUA enables support for SCSI-3 target port group commands.
What is DM-Multipath [Redhat]?
Device mapper multipathing (DM-Multipath) allows you to configure multiple I/O
paths between server nodes and storage arrays into a single device. Without DM-
Multipath, each path from a server node to a storage controller is treated by the
system as a separate device, even when the I/O path connects the same server node to
the same storage controller. DM-Multipath provides a way of organizing the I/O paths
logically, by creating a single multipath device on top of the underlying devices
If any element of an I/O path (the cable, switch, or controller) fails, DM-Multipath
switches to an alternate path.
Let's list down all the possible components that could fail in a typical I/O path
between server & storage?
With DM-Multipath configured, a failure at any of these points will cause DM-
Multipath to switch to the alternate I/O path.
After creating multipath devices, you can use the multipath device names just as you
would use a physical device name when creating an LVM physical volume. For
example, if /dev/mapper/mpathn [n=device number] is the name of a multipath
device, the following command will mark /dev/mapper/mpathn as a physical volume.
pvcreate /dev/mapper/mpathn
For example: In our case, we have one multipath device available by name
'mpath53'
[root@redhat /]# cd /dev/mapper/
[root@redhat mapper]# ll
total 0
crw------- 1 root root 10, 60 Sep 19 14:25 control
brw-rw---- 1 root disk 253, 0 Sep 20 13:30 mpath53
Now, using pvcreate we can create a physical volume to be used for LVM purpose:
Next step: Once we have one or more physical volumes created, we can create a
volume group on top of PVs using the vgcreate command. In our case, we have
created just one 'physical volume' using pvcreate command. Following command
creates 'volumegroup' on top of Physical Volume and on top volume group we can
create a Logicalvolume to be used for filesystem purpose.
Logical volume create command: [For example purpose, we will create 1GB disk]
Next, let's lay/install the filesystem on top of the Logival volume we just created.
First we will find out how our logical volume looks using 'lvdisplay' command:
Important: If you are using LVM on top of Multipath, then make sure Multipath is
be loaded before LVM to ensure that multipath maps are built correctly. Loading
multipath after LVM can result in incomplete device maps for a multipath device
because LVM locks the device, and MPIO cannot create the maps properly.
Querying the multipath I/O status outputs the current status of the multipath maps.
This is perhaps the first thing you would do to find out if paths show up?
1. multipath -l
2. multipath -ll
The multipath -l option displays the current path status as of the last time that
the path checker was run. It does not run the path checker.
The multipath -ll option runs the path checker, updates the path information,
then displays the current status information. This option always the displays
the latest information about the path status.
3600601607cf30e00184589a37a31d911
[size=127 GB][features="0"][hwhandler="1 alua"]
\_ round-robin 0 [active][first]
\_ 1:0:1:2 sdav 66:240 [ready ][active]
\_ 0:0:1:2 sdr 65:16 [ready ][active]
\_ round-robin 0 [enabled]
\_ 1:0:0:2 sdag 66:0 [ready ][active]
\_ 0:0:0:2 sdc 8:32 [ready ][active]
You can also use dmsetup command to see the number of paths:
In this example, I am using 'show paths' command to see the multipaths on my host:
multipathd>
multipathd> show multipaths stats
name path_faults switch_grp map_loads total_q_time q_timeouts
mpath53 0 0 4 0 0
multipathd>
Now, let's simulate path failure - We will fail the path 'sdc'
As you can see 'sdc' is now marked 'faulty' but due to constant polling, default interval
is 5 seconds, the path should come back up as Active immediately.
The goal of multipath I/O is to provide connectivity fault tolerance between the
storage system and the server. When you configure multipath I/O for a stand-alone
server, the retry setting protects the server operating system from receiving I/O errors
as long as possible. It queues messages until a multipath failover occurs and provides
a healthy connection.
However, when connectivity errors occur for a cluster node, you want to report the
I/O failure in order to trigger the resource failover instead of waiting for a multipath
failover to be resolved.
In cluster environments, you must modify the retry setting so that the cluster node
receives an I/O error in relation to the cluster verification process. Please read the
OEM document for recommended retry settings.
Where & how to enable ALUA on the Host when using NetApp Active/Active
storage systems:
Where:
On NetApp Storage: ALUA is enabled or disabled on the igroup mapped to a NetApp
LUN on the NetApp controller.
Only FCP igroups support ALUA in ontap 7-mode or simple Ontap HA.
You might ask why only FCP and not iSCSI in 7-mode, because there is no proxy
path in 7-mode, as both controllers have different IP address and the IP addresses are
tied to the Physical NIC/Ports.
However, with Ontap cmode or cluster mode, access characteristics changes, as the
physical adapters are virtualised in cmode, and clients accesses the data via LIF
[Virtual Adapter/Logical Interface] and in this case, the IP addresses are tied to the
LIF and not to the Physical NIC/Ports, which means if port failure is detected, a LIF
can be migrated to another working port on the another node with-in the cluster, and
hence ALUA is now supported with iSCSI in cmode setups.
This table only shows Windows OS, but ALUA is supported with iSCSI & ontap
cmode on non-windows OS. Please check NetApp interoperability matrix table.
How to enable ALUA:
If ALUA is not enabled for your igroup, you can manually enable it by setting the
alua option to yes. If you map multiple igroups to a LUN and you enable one of the
igroups for ALUA, you must enable all the igroups for ALUA.
Steps
1. Run the following command to check if ALUA is enabled:
filer> igroup show -v igroup_name
2.If ALUA is not enabled, then manually enable ALUA on the igroup following this
command:
filer>igroup set igroup alua yes
On the Host:
1.Validate the host OS and the multipathing software as well as the storage controller
software support ALUA. If yes, then proceed.
For example, ALUA is not supported for VMware ESX until vSphere 4.0. Check with
the host OS vendor for supportability.
2.Check the host system for any script that might be managing the paths automatically
and if so, disable it.
3.If using SnapDrive, verify that there are no settings disabling the ALUA set in the
configuration file.
Note: The output of igroup show -v displays the FCP initiator logged in on physical
ports as well as a port called "vtic". VTIC is an abbreviation for "virtual target
interconnect". VTIC provides a connection between the two nodes in an HA pair,
enabling LUNs to be served through target ports on both nodes. It is normal to see
VTIC as one of the ports in the output of igroup show -v.
This parameter shows whether or not the device supports implicit ALUA. You cannot
set this option as it is a property of the LUN.
explicit_support
This parameter shows whether or not the device supports explicit ALUA. You cannot
set this option as it is a property of the LUN.
explicit_allow
This parameter shows whether or not the user allows the SATP to exercise its explicit
ALUA capability if the need arises during path failure. This only matters if the device
actually supports explicit ALUA (that is, explicit_support is on). This option is turned
on using the esxcli command enable_explicit_alua and turned off using the esxcli
command disable_explicit_alua.
alua_followover
This parameter shows whether or not the user allows the SATP to exercise the follow-
over policy, which prevents path thrashing in multi-host setups. This option is turned
on using the esxcli command enable_alua_followover and turned off using the esxcli
command disable_alua_followover.
fc.20000000c987f8c5:10000000c987f8c5-fc.50060160bce0383c:500601663ce0383c-
naa.60060160455025000aa724285e1ddf11
Runtime Name: vmhba2:C0:T0:L0
Device: naa.60060160455025000aa724285e1ddf11
Device Display Name: DGC Fibre Channel Disk
(naa.60060160455025000aa724285e1ddf11)
Group State: active
Array Priority: 1
Storage Array Type Path Config:
{TPG_id=1,TPG_state=AO,RTP_id=7,RTP_health=UP}
Path Selection Policy Path Config: {current: no; preferred: no}
In the output:
TPG_state = ANO means Active/Non-Optimized
TPG_state = AO means Active/Optimized.
Frequently Used Terms
Command Descriptor Block (CDB): The standard format for SCSI commands.
CDBs are commonly 6, 10, or 12 bytes long, though they can be 16 bytes or of
variable length.
Multipath I/O (MPIO): A method by which data can take multiple redundant
paths between a server and storage.
SCSI Target: The receiving end of a SCSI session, typically a device such as a
disk drive, solid state drive, tape drive, or scanner.
Target Portal Group (TPG): A list of IP addresses and TCP port numbers that
determines which interfaces a specific iSCSI target will listen to.
LUN The logical unit number (LUN) is a SCSI convention used to enumerate
LU elements; for example, the host recognizes a particular Vdisk by its
assigned LUN.
Ashwin Pawar
Sep, 2014
www.simplysan.com
ashwinwriter@gmail.com