Universal SAS and ACP Guidelines

Universal SAS and ACP Cabling Guide
NetApp, Inc. 495 East Java Drive Sunnyvale, CA 94089 USA Telephone: +1 (408) 822-6000 Fax: +1 (408) 822-4501 Support telephone: +1 (888) 4-NETAPP Documentation comments: doccomments@netapp.com Information Web: www.netapp.com Part number: 215-05500_B0 November 2011
Copyright and trademark information
Copyright information
Copyright 1994-2011 NetApp, Inc. All rights reserved. Printed in the U.S.A. No part of this document covered by copyright may be reproduced in any form or by any means graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval systemwithout prior written permission of the copyright owner. Software derived from copyrighted NetApp material is subject to the following license and disclaimer: THIS SOFTWARE IS PROVIDED BY NETAPP AS IS AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp. The product described in this manual may be protected by one or more U.S.A. patents, foreign patents, or pending applications. RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).
Trademark information
NetApp, the NetApp logo, Network Appliance, the Network Appliance logo, Akorri, ApplianceWatch, ASUP, AutoSupport, BalancePoint, BalancePoint Predictor, Bycast, Campaign Express, ComplianceClock, Cryptainer, CryptoShred, Data ONTAP, DataFabric, DataFort, Decru, Decru DataFort, DenseStak, Engenio, Engenio logo, E-Stack, FAServer, FastStak, FilerView, FlexCache, FlexClone, FlexPod, FlexScale, FlexShare, FlexSuite, FlexVol, FPolicy, GetSuccessful, gFiler, Go further, faster, Imagine Virtually Anything, Lifetime Key Management, LockVault, Manage ONTAP, MetroCluster, MultiStore, NearStore, NetCache, NOW (NetApp on the Web), Onaro, OnCommand, ONTAPI, OpenKey, PerformanceStak, RAID-DP, ReplicatorX, SANscreen, SANshare, SANtricity, SecureAdmin, SecureShare, Select, Service Builder, Shadow Tape, Simplicity, Simulate ONTAP, SnapCopy, SnapDirector, SnapDrive, SnapFilter, SnapLock, SnapManager, SnapMigrator, SnapMirror, SnapMover, SnapProtect, SnapRestore, Snapshot, SnapSuite, SnapValidator, SnapVault, StorageGRID, StoreVault, the StoreVault logo, SyncMirror, Tech OnTap, The evolution of storage, Topio, vFiler, VFM, Virtual File Manager, VPolicy, WAFL, Web Filer, and XBB are trademarks or registered trademarks of NetApp, Inc. in the United States, other countries, or both.
ii
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml. Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the U.S.A. and/or other countries. Microsoft is a registered trademark and Windows Media is a trademark of Microsoft Corporation in the U.S.A. and/or other countries. RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and RealMedia, RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the U.S.A. and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks. NetApp, Inc. NetCache is certified RealSystem compatible.
iv
Table of Contents
Chapter 1 Chapter 2 Cabling requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 SAS cabling for new storage systems. . . . . . . . . . . . . . . . . . . . . . 3 SAS cabling rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Cabling SAS connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 SAS cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Chapter 3 ACP cabling for new storage systems . . . . . . . . . . . . . . . . . . . . 29 ACP cabling rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Cabling ACP connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 ACP cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Appendix A Appendix B Terms and symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Document update record . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Table of Contents
vi
Table of Contents
Cabling requirements
Before you begin
To cable a new storage system with SAS disk drives, your storage system must meet the following requirements:
N
Your storage system must be running the minimum software release required to support your model of SAS disk shelves. See the Installation and Service Guide for your particular disk shelf model or the System Configuration Guide at support.netapp.com.
N N N
If your storage system was not shipped in a system cabinet, the controllers and disk shelves must already be installed in a rack. If your storage system will be using one or more PCIe SAS HBAs, they must already be installed and available for cabling the disk shelves. If you are cabling for the ACP capability and your storage system does not have a native onboard ACP port, you must be prepared to designate a network interface for each controller at system setup. If you are using a four-port Ethernet card for this purpose, this card must already be installed and available in each controller.
Chapter 1: Cabling requirements
Cabling requirements
SAS cabling for new storage systems

About this chapter
This chapter provides information about cabling the disk shelf SAS connections for a new storage system installation. This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at support.netapp.com for your disk shelf model. Note References to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.
Topics in this chapter
This chapter discusses the following topics:

N N N
SAS cabling rules on page 4 Cabling SAS connections on page 11 SAS cabling examples on page 16
Chapter 2: SAS cabling for new storage systems
SAS cabling rules
Topics in this section
The following topics are covered in this section:

N N N N N
General SAS cabling rules on page 4 Onboard SAS port rule on page 5 Controller-to-controller rule (2240 systems only) on page 6 Shelf-to-shelf connection rule on page 6 Controller-to-shelf connection rules on page 7
General SAS cabling rules
The following SAS cabling rules apply to all supported storage systems using SAS HBAs and onboard SAS ports, except where otherwise noted: Supported configurations (except single-controller 2240 systems): Multipath HA is the supported configuration, with the following exceptions:
O O
2040 storage systems use single-path; all 2040 systems in an HA pair configuration use single-path HA. 2240 systems in an HA pair with no external SAS storage may use single-path HA, which requires no external cabling. However, the recommended configuration is to enable Multipath HA for the internal disks by connecting the SAS ports of one controller module to the SAS ports of the partner.
Supported configurations (single-controller 2240 systems): In 2240 systems in a single-controller configuration, the internal disks are single-path but the system can use either single- or dual-path connectivity to the external shelves. Both are supported. Dual-path provides greater resiliency, but when dual-path is used Data ONTAP 8.1 and 8.1.1 issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SAS storage are a supported option in these configurations. SAS cables used: QSFP to QSFP SAS cables are used to connect disk shelves together and to connect disk shelves to the SAS ports on the controller.
SAS cabling rules
QSFP to mini-SAS cables are used for 2050 configurations only, and connect disk shelves to the dual-port mini-SAS HBAs on the controller.
SAS port designations: All SAS ports on the disk shelf are designated by a square symbol or a circle symbol. The following illustration shows square and circle labeling on the IOM ports. Note SAS connectivity does not use the concept of In ports and Out ports.
IOM A ACP SAS ACP IOM B SAS
Onboard SAS port rule
To ensure that the SAS cabling rules apply to all SAS HBAs on a controller, whether in a physical PCI slot (slot 1-N) or on board the base controller, an on board HBA is defined as occupying PCI slot 0. Each port is named with sequentially increasing letters (0a, 0b, 0c, and so on). For example, the following illustration shows a controller with one onboard SAS HBA in slot 0 and three SAS HBAs in physical PCI slots 1, 2, and 3.
Slot 0 Slot 1 A B A B C D
Slot 2 A B C D
Slot 3 A B C D
Exception: The only exception is the 2040 storage system. Treat the onboard SAS port labeled 0d as if it were port 0a. Treating the port as 0a enables you to apply all the SAS cabling rules. (Data ONTAP continues to recognize this port as 0d.)
2040 controller
0a
0d
Controller-tocontroller rule (2240 systems only)
On 2240 systems in HA configurations with no external storage, external cables should connect the SAS ports on the two controller modules to enable Multipath HA for the internal disks.
Shelf-to-shelf connection rule
All the disk shelves in a stack are daisy-chained when you have more than one disk shelf in a stack.
N N
IOM A circle port is connected to the next IOM A square port. IOM B circle port is connected to the next IOM B square port.
Stack of three shelves
ACP
Top shelf
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
SAS cabling rules
Controller-to-shelf connection rules
Each controller connects to each stack of disk shelves in the system through the controller SAS ports, which can be A, B, C, and D, (and can be on a SAS HBA in a physical PCI slot [slot 1-N] or on board the base controller). For quad-port SAS HBAs, the controller-to-shelf connection rules ensure resiliency for the storage system based on the ASIC chip design. Ports A and B are on one ASIC chip, and ports C and D are on a second ASIC chip. By using ports A and C to connect to the top shelf in each stack, and using ports B and D to connect to the bottom shelf in each stack, the controllers maintains connectivity to the disk shelves if an ASIC chip fails. The following illustration shows a quad-port SAS HBA with the two ASIC chips and their designated ports.
Quad-port SAS HBA
ASIC chip PCI-e chip ASIC chip A port B port C port D port
SAS port A and C connections: SAS ports A and C on a controller are connected to the stacks of disk shelves in this way:
N
SAS ports A and C are connected to the top disk shelf square port in each stack of disk shelves.
O O
The A and C ports on controller 1/A connect to IOM A. The A and C ports on controller 2/B connect to IOM B.
Controller 1/A
Slot 1 Slot 1
Controller 2/B
A B C D
A B C D
Stack 1
Stack 2
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
Top shelf
IOM A
IOM B
IOM A
IOM B
Bottom shelf
SAS ports A and C are connected to the disk shelf stacks in sequential order of the HBA slots (slot 0, slot 1, slot 2, and so on). For example, a controller with slots 1, 2, and 3 occupied with quad-port SAS HBAs is cabled in order of HBAs and ports 1a, 2a, 3a, 1c, 2c, 3c, as shown below.
Controller 1/A
Slot 1 Slot 2 Slot 3
A B C D
A B C D
A B C D
SAS port B and D connections: SAS ports B and D on the controller are connected to the stacks of disk shelves in this way:
N
Ports B and D are connected to the bottom disk shelf circle port in each stack of disk shelves.
O O
The B and D ports on controller 1/A connect to IOM B. The B and D ports on controller 2/B connect to IOM A.
SAS cabling rules
Controller 1/A
Slot 1 Slot 1
Controller 2/B
A B C D
A B C D
Stack 1
Stack 2
ACP
Top shelf
SAS
ACP
SAS
ACP
SAS
ACP
SAS
IOM A
IOM B
IOM A
IOM B
Bottom shelf
SAS ports B and D are connected to the disk shelf stacks by offsetting the order of the HBA slots by one so that the first HBA port is cabled last. The cabling offset ensures system resiliency. For example, a controller with three quad-port SAS HBAs in slots 1, 2, and 3 is cabled in order of HBAs and ports 2b, 3b, 1d, 2d, 3d, 1b, as shown below.
Controller 1/A
Slot 1 Slot 2 Slot 3
A B C D
A B C D
A B C D
A controller with one HBA is cabled in order of port 1d and then 1b, as shown below.
Controller 1/A
Slot 1
A B C D
Stack cabling sequence: Controllers are cabled to the stacks in sequence of the stacks, such as stack 1, stack 2, stack 3, and so on. 20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.
10
SAS cabling rules
Cabling SAS connections

N N N N
Completing the SAS cabling worksheet on page 11 SAS cabling worksheet on page 13 Example worksheet on page 13 Cabling SAS ports on page 14
Completing the SAS cabling worksheet Step 1 Action
By making a list of the SAS ports for each controller that need to be cabled to the disk shelf IOMs, you can simplify the cabling task of your storage system.
Make a list of the top connections from the controller by listing all SAS HBA A ports and then all SAS HBA C ports on each controller. You can use the SAS cabling worksheet on page 13 to write your information. Note If you have an HA pair configuration, repeat this list for the second controller. Examples: A controller with three quad-port SAS HBAs is listed as 1a, 2a, 3a, 1c, 2c, 3c. A controller with one quad-port SAS HBA is listed as 1a, 1c. A controller with one dual-port SAS HBA is listed as 1a. A 2040 controller is listed as 0a. 2040 system configurations only have a top connection to the disk shelves.
11
Step 2
Action Make a list of the bottom connections from each controller. You can use the SAS cabling worksheet on page 13 to write your information. a. List the SAS HBA B ports and then all SAS HBA D ports. Examples: A controller with three quad-port SAS HBAs is listed as 1b, 2b, 3b, 1d, 2d, 3d. A controller with one quad-port SAS HBA is listed as 1b, 1d. A controller with one dual-port SAS HBA is listed as 1b. b. Copy the first port in your list to the end of your list, and then cross out the first port in your list.
For example:
1b, 2b, 3b, 1d, 2d, 3d, 1b
Offsetting the list by one (so that the bottom connection cabling begins on the second HBA) ensures system resiliency after you cable your storage system. Examples: A controller with three quad-port SAS HBAs is listed as 2b, 3b, 1d, 2d, 3d, 1b. A controller with one quad-port SAS HBA is listed as 1d, 1b. A controller with one dual-port SAS HBA is listed as 1b. Note If you have an HA pair configuration, repeat this list for the second controller. 3 Go to Cabling SAS ports on page 14 to cable your storage system.
12
SAS cabling worksheet
By filling in this worksheet as instructed in the Completing the SAS cabling worksheet on page 11, you identify the SAS ports for each controller that need to be cabled to the disk shelf IOMs.
Connection type Top connections Bottom connections
Controller C1/A C2/B C1/A C2/B
SAS port cabling sequence
Cable to disk shelf IOM To top shelf IOM A To top shelf IOM B To bottom shelf IOM B To bottom shelf IOM A
Example worksheet
This is an example of a completed worksheet for an HA pair configuration with three quad-port SAS HBAs and three stacks of disk shelves. Note The grayed out SAS ports are unused because of the limited number of stacks in this configuration. If there were six stacks, all SAS ports would be used.
Connection type Top connections Bottom connections
Controller C1/A C2/B C1/A C2/B
SAS port cabling sequence 1a, 2a, 3a, 1c, 2c, 3c 1a, 2a, 3a, 1c, 2c, 3c 2b, 3b, 1d, 2d, 3d, 1b 2b, 3b, 1d, 2d, 3d, 1b
Cable to disk shelf IOM To top shelf IOM A To top shelf IOM B To bottom shelf IOM B To bottom shelf IOM A
13
Cabling SAS ports Step 1 Action
To cable the SAS ports, complete the following steps.
If you have more than one disk shelf in a stack, cable the shelf-to-shelf SAS connections; otherwise, go to Step 2. Using the QSFP to QSFP SAS cables that came with your system: a. b. Connect IOM A circle port to the next IOM A square port. Connect IOM B circle port to the next IOM B square port.
Note The QSFP connectors are keyed and click into place when oriented correctly into a SAS port. Do not force a connector into a port.
Stack of three shelves
ACP
Top shelf
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
Use your completed worksheet to cable the top connections for each controller. Note If you are cabling a 2050 storage system, use the QSFP to mini-SAS cables for all controller to shelf connections; otherwise, use the QSFP to QSFP SAS cables.
14
Step 3
Action Use your completed worksheet to cable the bottom connections for each controller. Note For quad-port SAS HBAs, because ports A and C never connect to the same stack of disk shelves and ports B and D never connect to the same stack of disk shelves, you might have unused HBA ports depending on the number of stacks in your configuration. For example, a single-controller with one quad-port SAS HBA and one stack of shelves only uses HBA ports A and D. 4 Verify the SAS connections by entering the applicable command:
N
For 7-Mode, enter the following command at the system console:

sasadmin expander_map
For HA pair configurations, you can run this command on either node.
N
For Cluster-Mode, enter the following command at the clustershell prompt:

node run -node <node name> -command sasadmin expander_map
Result: The output shows information about SAS expanders attached to SAS channels on the controllers. 5 The next step depends on the output. If... The output lists all of the IOMs Any of the IOMs are not shown - either the output will not show the IOM that is cabled incorrectly, or will not show all of the IOMs downstream from it. 6 Then... The IOMs have connectivity. Go to Step 6. Repeat Steps 1 through 5 to correct cabling errors.
The next step depends on whether you are using the ACP capability. If... You are using the ACP capability You are not using the ACP capability Then... Go to Cabling ACP connections on page 32. Boot the storage system and begin setup.
15
SAS cabling examples
The following topics are covered in this section: Note The cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.
N N N N N
Examples of HA pair quad-port SAS HBA cabling on page 16 Examples of single-controller quad-port SAS HBA cabling on page 19 Examples of 2240 system SAS cabling on page 21 Examples of 2040 SAS cabling on page 24 Examples of 2050 SAS cabling on page 26
Examples of HA pair quad-port SAS HBA cabling
The following cabling examples show the SAS cabling sequence for HA pair configurations using quad-port SAS HBAs. One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1d.
Controller 1/A
Slot 1 A B C D A
Controller 2/B
Slot 1 B C D
1.
Ethernet port
2. 4.
Ethernet port
Stack 1
3.
ACP SAS ACP SAS
IOM A
IOM B
16
Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a; Bottom: 2b, 1d.
Controller 1/A
Slot 1 Slot 2 A A B C D A
Controller 2/B
Slot 1 Slot 2 A B C D
4. 7.
Ethernet port
1.
B C D
Ethernet port
2. 3.
B C D
8.
Stack 1
ACP SAS ACP SAS ACP
Stack 2
SAS ACP SAS
Top shelf
IOM A
IOM B
IOM A
IOM B
5.
6.
Bottom shelf
17
Three quad-port SAS HBAs and three stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a; Bottom: 2b, 3b, 1d.
Controller 1/A Controller 2/B
Slot 1 Slot 2 A B C D Slot 3 A B C D
1. 9. 7.
Slot 1 A B C D
Slot 2 A B C D
Slot 3 A B C D
4. 8.
A B C D
6. 11. 12.
Stack 1
ACP
Top shelf IOM A IOM B
2.
3.
Stack 2
ACP SAS ACP
5.
10.
Stack 3
ACP SAS ACP SAS
SAS
ACP
SAS
SAS
IOM A
IOM B
IOM A
IOM B
Bottom shelf
Four quad-port SAS HBAs and four stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a; Bottom: 2b, 3b, 4b, 1d. Four quad-port SAS HBAs and eight stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 1c, 2c, 3c, 4c; Bottom: 2b, 3b, 4b, 1d, 2d, 3d, 4d, 1b. Five quad-port SAS HBAs and five stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a; Bottom: 2b, 3b, 4b, 5b, 1d. Five quad-port SAS HBAs and 10 stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a, 1c, 2c, 3c, 4c, 5c; Bottom: 2b, 3b, 4b, 5b, 1d, 2d, 3d, 4d, 5d, 1b.
18
Examples of singlecontroller quad-port SAS HBA cabling
The following cabling examples show the SAS cabling sequence for singlecontroller configurations using quad-port SAS HBAs. One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence is 1a, 1d.
Controller 1/A
Slot 1 A B
1.
C D
Stack 1
2.
SAS
ACP Top shelf
SAS
ACP
IOM A
IOM B
Bottom shelf
19
One quad-port SAS HBA and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 1c; Bottom: 1d, 1b.
Controller 1/A
1. 3.
Slot 1
A B C D
4.
2.
Stack 1 Stack 2
ACP
Top shelf
SAS
ACP
SAS
ACP
SAS
ACP
SAS
IOM A
IOM B
IOM A
IOM B
Bottom shelf
Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 2a; Bottom: 2b, 1d.
Controller 1/A
1. 4.
Slot 1 Slot 2 A B C D A B C D
3.
Ethernet port
2.
Stack 2
Stack 1
ACP
Top shelf IOM A IOM B IOM A
SAS
ACP
SAS
ACP
SAS
ACP
SAS
IOM B
Bottom shelf
20
Examples of 2240 system SAS cabling
The following cabling examples show the SAS cabling sequence for 2240 HA pair and single-controller configurations. Note These examples show a 2240-2, but the cabling also applies to the 2240-4, even though the controllers are in a different position in the 2240-4. 2240 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a, 0b.
SAS SAS
0b
0a
0b
0a
PSU
PSU
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
21
2240 single-controller with one stack of disk shelves (dual-path): The SAS port cabling sequence is Top: 0a.
SAS ACP
0b
0a
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
Note Dual-path provides greater resiliency, but when dual-path is used Data ONTAP 8.1 issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SAS storage are a supported option in these configurations.
22
2240 single-controller with one stack of disk shelves (single-path): The SAS port cabling sequence is Top: 0a.
SAS
10101 e0a e0c
0b
0a
e0b e0d
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
2240 HA pair configuration with no external disk shelves: The SAS port cabling sequence for each controller is Top: 0a, 0b.
SAS SAS
0b
0a
0b
0a
23
Examples of 2040 SAS cabling
The following cabling examples show the SAS cabling sequence for 2040 HA pair and single-controller configurations. Note For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. Note The 2040 storage systems onboard SAS port labeled 0d is treated as port 0a for cabling purposes. When the onboard port is treated as port 0a, all the SAS cabling rules can be applied.
24
2040 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a.
2040 configuration
Controller 1/A
0a
Controller 2/B
0a 0d
0d
1.
Ethernet port e0P
Ethernet port e0P Stack 1
2.
ACP
Top shelf
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
2040 single-controller with one stack of disk shelves: The SAS port cabling sequence is Top: 0a.
2040 configuration
Controller 2/B
0a 0d Ethernet port e0P
1.
Stack 1
ACP
Top shelf
IOM A IOM B
SAS
ACP
SAS
Bottom shelf
25
Examples of 2050 SAS cabling
The following cabling examples show the SAS cabling sequence for 2050 HA pair and single-controller configurations. Note For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. 2050 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1b.
2050 configuration
Controller 1/A
Slot 1 A B
Controller 2/B
Slot 1 A B
1.
Ethernet port
2.
Ethernet port
Stack 1
4.
ACP
Top shelf
3.
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
26
2050 HA pair configuration with one disk shelf: The SAS port cabling sequence for both controllers is Top: 1a; Bottom: 1b.
2050 configuration
Controller 1/A
Slot 1 A B A
Controller 2/B
Slot 1 B
1.
Ethernet port
Ethernet port
4.
Stack 1
2.
ACP
3.
SAS
ACP
SAS
IOM A
IOM B
27
2050 single-controller configuration with one stack of disk shelves: The SAS port cabling sequence is Top: 1a; Bottom:1b.
2050 configuration
Controller 2/B
Slot 1 A B
1.
2.
Stack 1
ACP
Top shelf
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
2050 single-controller configuration with one disk shelf: The SAS port cabling sequence is Top: 1a; Bottom: 1b.
2050 configuration
Controller 2/B
Slot 1 A B
Ethernet port
1.
2.
Stack 1
ACP
SAS
ACP
SAS
IOM A
IOM B
28
ACP cabling for new storage systems

About this chapter
This chapter provides information about cabling the disk shelf ACP connections for a new storage system installation. This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at support.netapp.com for your disk shelf model. Note References to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.
Topics in this chapter
This chapter discusses the following topics:

N N N
ACP cabling rules on page 30 Cabling ACP connections on page 32 ACP cabling examples on page 38
Chapter 3: ACP cabling for new storage systems
29
ACP cabling rules

N N
General ACP cabling rules on page 30 ACP connectivity rules on page 30
General ACP cabling rules
These ACP cabling rules apply to all supported storage systems using SAS storage:
N N
You must use CAT6 Ethernet cables with RJ-45 connectors for ACP connections. If your storage system does not have a dedicated on board network interface for each controller, you must dedicate one for each controller at system setup. You can use a quad-port Ethernet card. All ACP connections to the disk shelf are cabled through the ACP ports, which are designated by a square symbol or a circle symbol.
ACP connectivity rules
All supported storage systems using the ACP capability are cabled in this way: Shelf-to-shelf connections: All the disk shelves in a stack are daisy-chained when you have more than one shelf in a stack.
N N
IOM A circle port is connected to the next IOM A square port. IOM B circle port is connected to the next IOM B square port.
Intrastack connections: The bottom shelf and top shelf in each stack are connected to provide resiliency within each stack of disk shelves. The bottom shelf IOM A circle port is connected to the top shelf IOM B square port. This applies to stacks containing one or more disk shelves. Stack-to-stack connections: When you have two or more stacks of disk shelves, the last shelf in a stack is connected to the first shelf in the next stack until all stacks are connected. The bottom shelf IOM B circle port of a stack is connected to the top shelf IOM A square port of the next stack. Controller-to-stack connections: Each storage system controller is connected to each stack of disk shelves through a dedicated Ethernet port:
30 ACP cabling rules
N N
Controller 1/A always connects to the top shelf IOM A square port in a stack. Controller 2/B always connects to the bottom shelf IOM B circle port in a stack.
Controller-to-controller connections (2240 systems with no external SAS shelves): Each storage system controller is connected to the partner controller through a dedicated Ethernet port marked with the locked wrench icon. This rule applies to 2240 storage systems only. 20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.
31
Cabling ACP connections
Cabling ACP in an HA pair with no external SAS storage Step 1 Action
The following cabling procedure applies to 2240 HA storage systems with no external SAS storage:
Cable the locked wrench ACP port on one controller module to the same port on the partner controller module.
SAS ACP (Locked wrench) SAS ACP (Locked wrench)
0b
0a
0b
0a
PSU
PSU
Note These examples show a 2240-2, but the cabling also applies to the 2240-4, even though the controllers are in a different position in the 2240-4.
32
Cabling ACP on systems with external SAS storage Step 1 Action
Any storage system supporting the ACP capability and having external SAS storage can be cabled using this cabling procedure.
If you have more than one disk shelf in a stack, cable your shelf-to-shelf ACP connections; otherwise, go to Step 2. Using the Ethernet cables that came with your system: a. b. Connect each shelf IOM A circle port to the next shelf IOM A square port until all shelves in each stack are connected. Connect each shelf IOM B circle port to the next shelf IOM B square port until all shelves in each stack are connected.
Daisy-chain connections
ACP SAS ACP SAS
Top shelf
IOM A IOM B
Bottom shelf
33
Step 2
Action Cable the intrastack connection. If... You have more than one disk shelf in a stack of disk shelves Then... Connect the bottom shelf IOM A circle port to the first shelf IOM B square port and repeat for all stacks.
Intrastack connection
ACP
Top shelf
SAS
ACP
SAS
IOM A
IOM B
Bottom shelf
You have only one disk shelf in a stack of disk shelves
Connect the IOMs within the disk shelf by connecting IOM A circle port to IOM B square of the disk shelf.
Intrastack connection
ACP
SAS
ACP
SAS
IOM A
IOM B
34
Step 3
Action If you have more than one stack of disk shelves, cable the stack-to-stack connections; otherwise, go to Step 4. a. b. Beginning with Stack 1, connect the bottom shelf IOM B circle port to the next stack top shelf IOM A square port. Repeat Step a until all stacks are connected.
Stack-to-stack connections
Stack 1
ACP
Top shelf IOM A IOM B IOM A IOM B IOM A IOM B
Stack 2
SAS ACP SAS ACP SAS ACP
Stack 3
SAS ACP SAS
SAS
ACP
Bottom shelf
35
Step 4
Action Cable controller-to-stack connections. If... If your system has a controller 1/A If your system has a controller 2/B Then... Cable controller 1/A to the Stack 1 top shelf IOM A square port. Cable controller 2/B to the last stack bottom shelf IOM B circle port.
For example, an HA pair configuration with two stacks of disk shelves has the controller-to-stack connections cabled as follows:
Ethernet port
Ethernet port
Stack 1
ACP
Stack 2
SAS ACP SAS ACP SAS
SAS
ACP
IOM B
Bottom shelf
Note 20xx storage systems use the e0P port on each controller. 5 Verify that the ACP cabling is correct by entering the applicable command:
N
For 7-Mode, enter the following command from the system console:
storage show acp
If you have an HA pair configuration, run this command on both nodes.

N
For Cluster-Mode, enter the following command from the clustershell:

node run -node local -command storage show acp
36
Step 6
Action The next step depends on what the command output is for each node. If the result in the output shows ACP connectivity status... Full Connectivity Then... ACP cabling is correct. The number of shelf IOMs detected out-of-band (through ACP) and in-band (through SAS) are the same. No Connectivity No ACP ports on the shelf IOMs are connected to the storage controllers. Repeat Step 4. Partial Connectivity Repeat Steps 1 through 6 to see what ACP connections you missed. Fewer shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS). Additional Connectivity Repeat the Cabling SAS connections on page 11 to see what SAS connections you missed. More shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).
Boot the storage system and begin setup. See the Installation and Setup Instructions that came with your storage system. Note If your storage system does not have a dedicated ACP on board Ethernet port on each controller for ACP, you will designate them at setup.
37
ACP cabling examples
The following topics are covered in this section: Note The cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.
N N N N
Examples of HA pair configuration ACP cabling on page 38 Examples of ACP cabling for 2240 systems with no external storage on page 40 Examples of ACP cabling for all systems except 20xx on page 41 Examples of 20xx single-controller ACP cabling on page 42
Examples of HA pair configuration ACP cabling
The following cabling examples show ACP cabling for all supported HA pair configurations. HA pair configuration with one disk shelf:
Ethernet port
Ethernet port
Stack 1
ACP
Top shelf IOM A IOM B
SAS
ACP
SAS
38
HA pair configuration with two stacks of disk shelves:

Ethernet port
Ethernet port
Stack 1
ACP
Stack 2
SAS ACP SAS ACP SAS
SAS
ACP
IOM B
Bottom shelf
HA pair configuration with three stacks of disk shelves:

Ethernet port
Ethernet port
Stack 1
Stack 2
Stack 3
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
ACP
SAS
Top shelf
IOM A
IOM B
IOM A
IOM B
IOM A
IOM B
Bottom shelf
39
Examples of ACP cabling for 2240 systems with no external storage
The following examples show ACP cabling for 2240 systems with no external disk shelves: 2240-2 HA configuration with no external disk shelves:
SAS ACP (Locked wrench) SAS ACP (Locked wrench)
0b
0a
0b
0a
PSU
PSU
2240-4 HA configuration with no external disk shelves:

SAS
Controller A
ACP
Controller B
40
Examples of ACP cabling for all systems except 20xx
The following cabling examples show ACP cabling for all supported storage systems other than 20xx systems. Single-controller configuration with one stack of disk shelves (except the 20xx systems):
Non 20xx configuration
Controller 1/A
Ethernet port
Stack 1
ACP SAS ACP SAS
Top shelf
IOM A IOM B
Bottom shelf
41
Single-controller configuration with two stacks of disk shelves (except the 20xx systems):
Non 20xx configuration
Controller 1/A
Ethernet port
Stack 1
ACP
Stack 2
SAS ACP SAS ACP SAS
SAS
ACP
IOM B
Bottom shelf
Examples of 20xx single-controller ACP cabling
The following cabling examples show ACP cabling for 20xx single-controller configurations. Note For single-controller 20xx configurations, the controller is in Slot B (the bottom slot); therefore, the controller needs to be cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.
42
20xx single-controller configuration with one stack of disk shelves:

20xx configuration
Controller 2/B
Ethernet port
Stack 1
ACP
Top shelf
IOM A IOM B
SAS
ACP
SAS
Bottom shelf
43
20xx single-controller configuration with two stacks of disk shelves:

20xx configuration
Controller 2/B
Ethernet port
Stack 1
ACP
Stack 2
SAS ACP SAS ACP SAS
SAS
ACP
IOM B
Bottom shelf
44
Terms and symbols

About this appendix
This appendix defines terms and port icon symbols particular to SAS disk shelves and also includes common terms about connectivity. The terms are organized by relationship to each other instead of alphabetical order.
List of terms and symbols
You should be familiar with the following terms:

N
Square and circle symbols: The SAS ports and ACP ports on the disk shelf IOMs are designated by square and circle symbols. All cabling is done in reference to connecting to a square port or a circle port. Note SAS connectivity does not use the concept of In ports and Out ports.
IOM: The SAS shelf I/O module is located in the back of the disk shelf. It connects the individual disk drives to the rest of the storage system and controls the disk shelf operator display panel LEDs. Each disk shelf has two IOMs: IOM A and IOM B. These are also referred to as SBB A and SBB B on the slot map label on the back of the disk shelf.
SAS stack: Also referred to as stack. A group of one or more SAS disk shelves connected (daisy-chained) together and connected to the controller through the top disk shelf in the stack and the bottom disk shelf in the stack (as needed). The maximum number of disk shelves in a stack of disk shelves and the number of disk shelf stacks in a configuration are dependent on the type of storage system.
ACP: Alternate Control Path is a protocol that enables Data ONTAP to manage and control the disk shelf storage subsystem. It uses a separate network from the data path, so management communication is not dependent on the data path being intact and available. Use of ACP requires that all disk shelf IOMs and storage system controllers connect through the ACP ports on the IOMs and the designated network interface on each controller.
QSFP: The standard SAS cable connector for all NetAppSAS systems. QSFP to QSFP SAS cables are used to daisy-chain SAS disk shelf ports and connect
Appendix A: Terms and symbols
45
to the quad-port SAS host bus adapter (HBA) or onboard SAS ports. QSFP to mini-SAS cables are used to connect disk shelves to the dual-port miniSAS HBA.
N
Software-based disk ownership: Software-based disk drive ownership is stored on the disk drive rather than determined by the topology of the storage systems physical connections. It gives you increased flexibility and control over disk drive use. SAS and SATA disk drives use software-based disk drive ownership. Therefore, the hardware-based disk ownership concept that the local node owns all direct-attached disk drives connected to it through the A channel (the loop attached to the A module on the disk shelf) and its partner owns the disk drives connected to the local node on the B channel, does not apply. This also means that the hardware-based disk drive ownership terms do not apply: primary path, standby path, redundant path, and redundant standby path. Instead, connectivity between the controller and SAS disk shelf is addressed in terms of: top connections and bottom connections.
Top connection: The cabling connection from the controller to the top disk shelf in a stack of disk shelves. References to the top disk shelf in this document are to the logical top of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.
Bottom connection: The cabling connection from the controller to the bottom disk shelf in a stack of disk shelves. References to the bottom disk shelf in this document are to the logical bottom of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.
Shelf-to-shelf (daisy-chain): The cabling connection between disk shelves in a stack of more than one disk shelf. Each stack of disk shelves is daisy-chained through its SAS ports and, if you are using the ACP capability, each disk shelf is also daisychained through its ACP ports.
Controller: The component of your storage system that runs Data ONTAP software and manages the storage subsystem. The controller located in the top slot in a storage system chassis can be referred to as Controller 1 or Controller A. The controller located in the bottom slot in a storage system chassis can be referred to as Controller 2 or Controller B.
46
Terms and symbols
In this document controllers are referred to as Controller 1/A and Controller 2/B.
N
Single-controller configuration: A storage system having one controller. HA pair configuration: A storage system having two controllers. This can be two controllers in one chassis or it can be two controllers, each in a separate chassis. The two controllers serve data for each other if one of the controllers becomes impaired.
Multipath HA: Each controller in the HA pair configuration has multiple ways to connect to a disk drive. Multipath HA cabling is the most resilient and preferred configuration for HA pair configurations. This is because it takes full advantage of the resiliency capability of SAS, which means that the storage system continues to have access to disk drives in the event of cable, HBA, and IOM failure. A single failure of a cable, HBA, or IOM does not result in a controller failover.
Dual-path: The single-controller storage system has two ways to connect to a disk drive. This is the preferred configuration for single-controller configurations.
Single-path HA: Each controller in the HA pair configuration has one way to connect to a disk drive. This means an IOM or cable failure requires a controller failover.
Single-path: The single-controller storage system has one way to connect to a disk drive. This configuration has multiple single points of failure and is not a preferred configuration.
Configured systems: A new storage system that ships with SAS disk shelves and preinstalled SAS HBAs (if applicable).
Add-on disk shelves: SAS disk shelves that are shipped individuallynot shipped as part of a configured system.
Appendix A: Terms and symbols
47
48
Terms and symbols
Document update record

Document update history The following table lists and describes the history of changes made to this document. Changes are not dictated by Data ONTAP releases.
B
Feature release date November 2011 September 2010
Feature updates
N N
Updated with rules and exceptions for 2240 systems. Initial release of the SAS and ACP cabling information as a stand-alone document and updates.
O
The SAS disk shelf illustrations are now generic so that they apply to any NetApp SAS disk shelf. A generic disk shelf is shown as two IOMs side-by-side (no power supplies are shown). HBAs (1, 2, 3 etc.) are now referred to as slots (1, 2, 3 etc.) Introduces the new rule for referring to onboard SAS ports as belonging to slot 0 and the 2040 system exception - the single onboard port is 0a. Introduces the new rule that for singlecontroller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. The term active/active is updated to HA pair throughout. The term Multipath is updated to Multipath HA. September 2009
O O
O O N
Initial release of the SAS and ACP cabling information was part of the DS4243 Installation and Service Guide.
Appendix B: Document update record
49
50
Document update record
Index
A
ACP cabling procedure 33 ACP capability 30 cabling SAS examples 19 rules 4 verfiying SAS connections 15 worksheet 13 worksheet procedure 11
C
cabling ACP examples 42 procedure 33 verifying ACP connections 36
T
terminology 45
Index
51
52
Index

Universal SAS and ACP Guidelines

Enviado por

Dados do documento

Descrição original:

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Universal SAS and ACP Guidelines

Enviado por

Direitos autorais:

Formatos disponíveis

Universal SAS and ACP Cabling Guide

Copyright and trademark information

Copyright and trademark information

Copyright and trademark information

Chapter 1: Cabling requirements

SAS cabling for new storage systems

Topics in this chapter

This chapter discusses the following topics:

Chapter 2: SAS cabling for new storage systems

SAS cabling rules

Topics in this section

The following topics are covered in this section:

General SAS cabling rules

SAS cabling rules

Onboard SAS port rule

Chapter 2: SAS cabling for new storage systems

Controller-tocontroller rule (2240 systems only)

Shelf-to-shelf connection rule

SAS cabling rules

Controller-to-shelf connection rules

Chapter 2: SAS cabling for new storage systems

SAS cabling rules

Chapter 2: SAS cabling for new storage systems

SAS cabling rules

Cabling SAS connections

Topics in this section

The following topics are covered in this section:

Completing the SAS cabling worksheet Step 1 Action

Chapter 2: SAS cabling for new storage systems

1b, 2b, 3b, 1d, 2d, 3d, 1b

Cabling SAS connections

SAS cabling worksheet

Connection type Top connections Bottom connections

Controller C1/A C2/B C1/A C2/B

SAS port cabling sequence

Connection type Top connections Bottom connections

Controller C1/A C2/B C1/A C2/B

Chapter 2: SAS cabling for new storage systems

Cabling SAS ports Step 1 Action

To cable the SAS ports, complete the following steps.

Cabling SAS connections

For 7-Mode, enter the following command at the system console:

For Cluster-Mode, enter the following command at the clustershell prompt:

Chapter 2: SAS cabling for new storage systems

SAS cabling examples

Topics in this section

Examples of HA pair quad-port SAS HBA cabling

SAS cabling examples

Chapter 2: SAS cabling for new storage systems

SAS cabling examples

Examples of singlecontroller quad-port SAS HBA cabling

ACP Top shelf

Chapter 2: SAS cabling for new storage systems

SAS cabling examples

Examples of 2240 system SAS cabling

Chapter 2: SAS cabling for new storage systems

SAS cabling examples

Chapter 2: SAS cabling for new storage systems

Examples of 2040 SAS cabling

SAS cabling examples

Ethernet port e0P

Ethernet port e0P Stack 1

Chapter 2: SAS cabling for new storage systems