Escolar Documentos
Profissional Documentos
Cultura Documentos
www.mellanox.com
Rev 1.0
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (PRODUCT(S)) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400 Fax: (408) 970-3403
Mellanox Technologies, Ltd. Beit Mellanox PO Box 586 Yokneam 20692 Israel www.mellanox.com Tel: +972 (0)74 723 7200 Fax: +972 (0)4 959 3245
Copyright 2013. Mellanox Technologies. All Rights Reserved. Mellanox, Mellanox logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, MLNX-OS, PhyX, SwitchX, UFM, Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd. Connect-IB, FabricIT, Mellanox Open Ethernet, MetroX, MetroDX, ScalableHPC, Unbreakable-Link are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.
Mellanox Technologies
Rev 1.1
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.1 1.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Serial Number and Product Version Information . . . . . . . . . . . . . . . . . . . . . . . . 11
Chapter 4 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 Installation Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Kits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Package Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mechanical Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grounding the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Power Connections and Initial Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Battery Replacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Switch the First Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managed (Internally Managed) Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rerunning the Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Connecting to the Switch Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 29 29 29 34 34 35 36 36 42 42
Mellanox Technologies
Rev 1.1
Chapter 6 Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.1 6.2 Splitting the Data Stream Using Breakout (Fanout) Cables . . . . . . . . . . . . . . . . 47 Cable Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Chapter 7 Switch Shut Down Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Chapter 8 Disassembly of the Switch from the Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.1 8.2 Disposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Upgrading Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
SFP+ Removal Tool Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 QSFP Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 SFP+ Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 RJ-45 CONSOLE and I2C Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Accessory and Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Avertissements de scurit dinstallation (French) . . . . . . . . . . . . . . . . . . 65 Installation - Sicherheitshinweise (German) . . . . . . . . . . . . . . . . . . . . . . . 69 Advertencias de seguridad para la instalacin (Spanish) . . . . . . . . . . . . 72
Mellanox Technologies
Rev 1.1
List of Figures
Figure 1: Figure 2: Figure 3: Figure 4: Figure 5: Figure 6: Figure 7: Figure 8: Figure 9: Connector Side View of the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Pull Out Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 QSFP Switch System Power and Connector Side Panels . . . . . . . . . . . . . . . . . . . . . . . 14 System Status Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Fan Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Fan Status LED Power side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Power Status LED Connector Side . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Power Side Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 PS Unit Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 10: Bad Port LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Figure 11: Port LED to Port Location Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Figure 12: Port LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Figure 13: Component Airflow Mismatch (Wrong) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Figure 14: Port Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 15: Top and Bottom QSFP Port Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 16: Top and Bottom SFP+ Port Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Figure 17: Reset Button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Figure 18: Rack Rail Kit Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Figure 19: Placement of Switch in Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Figure 20: Mounting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Figure 21: Screwing on the Rail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Figure 22: Inserting the Caged Nuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 23: Slide the Rail into the Rail Slide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 24: Status LEDs 5 Minutes After Power On . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 25: Two Power Inlets - Electric Caution Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Figure 26: Console Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Figure 27: Fan and PS unit Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Figure 28: Power Supply Unit Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Figure 29: PS Unit Pulled Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Figure 30: Air Flow Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Figure 31: Fan Module Latches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Figure 32: Breakout or Fanout Cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Figure 33: Port Splitting Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Figure 34: SFP+ Cable Orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Figure 35: QSFP Connector Male and Female Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Mellanox Technologies
Rev 1.1
Mellanox Technologies
Rev 1.1
List of Tables
Table 1: Table 2: Table 3: Table 4: Table 5: Table 6: Table 7: Table 8: Table 9: Table 10: Table 11: Table 12: Table 13: Table 14: Table 15: Table 16: Table 17: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Reference Documents and Web Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 System Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Fan Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 PS Unit Status LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Bad Port LED Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Port Connector Physical and Logical Link Assignments for Ethernet . . . . . . . . . . . . . . . 20 Air Flow Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Installation Kit According to Rack Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Serial Terminal Program Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Configuration Wizard Session - IP Configuration by DHCP . . . . . . . . . . . . . . . . . . . . . . 37 Configuration Wizard Session - IP Zeroconf Configuration . . . . . . . . . . . . . . . . . . . . . . 39 Configuration Wizard Session - Static IP Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 40 SX1024 Specification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 RJ-45 CONSOLE Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 RJ-45 I2C Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 OPNs for Replacement Parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Mellanox Technologies
Rev 1.1
Revision History
Table 1 - Revision History Table
Date Revision Description
March 2013
1.1
Added SFP+ cable and module removal tool to contents. Added figures to appendix. Added note use only FCC compliant Ethernet cables. Added note concerning USB insertion Replaced wizard tables from MLNX-OS doc added section opening the CLI Initial Release
January 2013
1.0
Mellanox Technologies
Rev 1.1
Intended Audience
This manual is intended for users and system administrators responsible for installing, running, and managing switch platforms. The manual assumes familiarity with the Ethernet specification.
Related Documentation
Additional Documentation available from Mellanox:
Table 2 - Reference Documents and Web Sites
Document Name Description
SwitchX-2 Switch Hardware Installation The Installation Guide contains instructions for Installation in a rack and initialization of the switch platform. This document can be found on the product web Guide page. SwitchX-2 Switch Product Hardware For possible hardware issues see the switch support product page. This document can be found on the support web page for this product. Release Notes This requires a customer support login. Look up the relevant SwitchXbased switch system/series release note file.
Mellanox MLNX-OS Software User This document contains information regarding configuring and managManual ing Mellanox Technologies Switch Platforms. MLNX-OS Software Command Ref- Command Reference Guide for MLNX-OS listing all of the commands available through MLNX-OS with explanations and examples. erence Guide
All of these documents can be found on the Mellanox Website at www.mellanox.com. They are available either through the product pages www.mellanox.com Products InfiniBand/VPI Switch Systems or through the support page with a login and password.
Conventions
Throughout this manual, the name SX1024 and the term switch are used to describe the 60 port SFP+/QSFP switch unless explicitly indicated otherwise. The following icons are used throughout this document to indicate information that is important to the user.
Mellanox Technologies
Rev 1.1
This icon indicates a situation that can potentially cause damage to hardware or software.
BEWARE! This icon indicates a situation that can potentially cause personal injury or damage to hardware or software.
Abbreviations
Used to indicate a 4X port running at 56Gb/s
FF
16 = 640Gb/s 24 = 960Gb/s 35 = 2880 Gb/s Light version of the Management software 36 = 2880 Gb/s
C P M Y R
Data Rate Separator # Power Supplies Depth of the Unit Air Flow direction Chip Generation
0=0, 1=1, 2=2.... S = standard depth, B = short depth R= Connector side to Power side airflow F= Power Side side to Connector side airflow R SwitchX S SwitchX-2
10
Mellanox Technologies
Rev 1.1
Overview
The SX1024 Ethernet switch family provides the highest-performing fabric solution in a 1U form factor by delivering up to 1.152 Gb/s of non-blocking throughput to High-Performance Computing (HPC), high frequency trading and Enterprise Data Centers (EDC), with ultra-low latency. It has 48 10GbE ports that when connected to 10GbE based servers, provides a unique non-blocking solution for Ethernet Data-Centers.
Figure 1: Connector Side View of the Switch
SX1024
MGT
CONSOL
Mellanox
UID RST
1.1
Features
Based on Mellanox SwitchX-2 Supports 48 SFP+ ports of 10 GbE Supports 12 QSFP ports of 40 GbE Upgradable with a license to 56 GbE 2x 10/100/1000BASE-T RJ-45 Ethernet port 1x RS232 RJ-45 Console connector 1x USB connector 1 + 1 redundant 300W AC/DC hot swappable Power supplies 3 + 1 redundant fans ordered in two fan units (Support Air in/out) On board Temperature monitoring Reset Push button on front panel Various status LEDs 19 rack mountable chassis, 1U with redundant power supplies and Fan units
1.2
Mellanox Technologies
11
Rev 1.1
Overview
SX1024
59 60
mgmt0
Mellanox
S/N:MT1244X03494 P/N:MSX1024B1BFS MAC:0002C9020040 Rev:A1 MGTMAC: 00:02:C9:11:A2:01 244X6511240006
12
Mellanox Technologies
Rev 1.1
Ethernet Ports
This Ethernet switch system supports 48 SFP+ ports, 12 QSFP ports and maintains a 4:1 ratio. Each SFP+ port is capable of 10GbE/1GbE port speeds. Up to 24 SFP+ ports are capable of 1/2/ 4/8G FC. Ports 1-12 are grouped as FC ports as well as ports 38-48. All Ethernet ports can be manually configured through the CLI management interface.
40 GbE and 56 GbE are only guaranteed to work with approved cables and modules.
Each of the 12 QSFP ports is capable of up to 40GbE, upgradable to 56 GbE with a license, as well as 10GbE with a QSA QSFP to SFP+ adapter or QSFP to SFP+ Mellanox cables. Each of the 48 SFP+ ports is capable of up to 10GbE.
Mellanox Technologies
13
Rev 1.1
Basic Operation
3
3.1
Basic Operation
Switch Platform Hardware Overview
Figure 3 shows the power side panel view of the switch. The figure shows the hot-swap power supply locations and the hot-swap fan modules, and an I2C banana connector. The figure also shows the connector side panel views of the internally managed switch. The figure shows port configurations for the switch systems. Managed systems have 2 Ethernet RJ-45 connectors for management, 1 RJ-45 connector for connecting to a host PC, 1 USB connector. All switches have various status LEDs for on site switch status information.
Figure 3: QSFP Switch System Power and Connector Side Panels
OK ! OK ! OK ! OK !
Mellanox
50
51
52
53
54
55
56
57
58
59
60
UID
Mellanox
RST
3.2
3.2.1
Status LEDs
System Status Indicators
The System Status Indicators are located to the left of the connectors on the connector side panel, and on the power side at the far right. Both of these LEDs give identical information.
Figure 4: System Status Indicators
Green OK Yellow a fault in the system Red Major Fault or Fatal Error
PS
UID
RST
14
Mellanox Technologies
Rev 1.1
The system status indicators should display as follows: when the switch is plugged in, within five minutes the STATUS LED should light up green the FAN LED should light up green. If the FAN LED shows red, troubleshoot the fan module. See Troubleshooting on page 52.
As long as there is power to the switch (one PS unit is connected), and the switch is booted up and running, the status LED should be green.
If the STATUS LED shows red after five minutes, unplug the switch and call your Mellanox representative for assistance.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes and replug in the switch. For more information See Troubleshooting on page 52.
PS
UID
RST
OK The system is up and running. The system is booting up. This assignment is valid on managed systems only. To be implemented in a future SW release. Major Error Possible damage can result to the switch. Turn off immediately. e.g. bad firmware, cant boot, overheated. If the system is booting up, it can take up to 5 minutes for the status led to change to green. Off The system has no power.
Off
Mellanox Technologies
15
Rev 1.1
Basic Operation
3.2.2
PS
UID
RST
The LED indicator on the Fan Module is on the right side of the module.
Figure 6: Fan Status LED Power side
UID RST
OK All fans are up and running. Error One or more fans is not operating properly. The system should be powered down and troubleshoot the fan module. Off The fan unit is not receiving any power. Check that the fan unit is properly and completely inserted.
16
Mellanox Technologies
Rev 1.1
All fans must be operating while the power supply is plugged in.
If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes and replug in the switch. For more information See Troubleshooting on page 52.
3.2.3
These are two color LEDs Green OK Red Major fault Off No power
UID PS
RST
Mellanox
Fan Units
Each PS unit has a single 2 color LED on the right side of the PS unit, that indicates the internal status of the unit.
Mellanox Technologies
17
Rev 1.1
Basic Operation
OK The Power supply is delivering the correct voltage 12VDC. Error The PS unit is not operational. Off There is no power to the system (neither PS unit is receiving power). If one PS unit is showing green and the second PS unit is unplugged it will show a red indication.
UID
RST
The Bad Port indicator is located on the left side of the connector side panel of the unit. Table 6 shows the fan status LED assignment.
18
Mellanox Technologies
Rev 1.1
OK No symbol errors have been received in last few seconds. Error One or possibly more ports have just received a symbol error Possible causes are:
Bad cable Bad connection Bad connector This LED lights up when one or more ports is receiving a symbol error. The LED immediately goes off until the next symbol error is received. 3.2.3.2 UID LED Switch Identifier The UID LED is a debug feature that lights a blue LED on the switch connector side panel for ease in finding a particular switch. 3.2.3.3 Port Connector LED Indicators The SFP+ ports come three rows to a column. The LEDs refer to the ports according to the color coding in Figure 11.
Figure 11: Port LED to Port Location Assignment
43 44 43 44 45 45
46 47 46 47 48
49
50 49
50 48
Mellanox Technologies
19
Rev 1.1
Basic Operation
49
50
51
52
53
Table 7 - Port Connector Physical and Logical Link Assignments for Ethernet
LED Configuration LED Description
Physical link is down / Default. Physical link is up with no traffic. Physical link is up with traffic. Physical errors.
3.3
Air Flow
These switches can come with two air flow patterns. The two patterns are: Connector side inlet to power side outlet Power side inlet to connector side outlet
The air flow is specified in the product model number. See Mellanox Part Numbering Legend on page 10.On the PS unit and fan modules the air flow direction can be seen on the power side panel. All components need to have the same air flow. A mismatch in the air flow will affect the heat dissipation.
All PS units must have the same air flow as the fan modules.
20
Mellanox Technologies
Rev 1.1
All air flow labels must be the same. Any mismatch of these labels will affect the switch temperature.
OK
OK
OK
OK
Table 8 shows the icons that indicate airflow using arrows and colors.
Mellanox Technologies
21
Rev 1.1
Basic Operation
3.4
3.5
3.5.1
Interfaces
Port Connector Interfaces
There are 48 SFP+ ports and 12 QSFP ports. They are arranged according to the figure below.
22
Mellanox Technologies
Rev 1.1
50 52
54
56
58
60
3.5.2
3.5.3
1 I2C banana connector on the power supply side for FAE use only
Mellanox Technologies
23
Rev 1.1
Basic Operation
3.5.3.1 RJ-45 Connector The port labeled is for a local host connection to the management module. This is used the first time the switch is connected. A harness is included in the package to connect to a DB9 connection on a host PC. Connecting to a local host PC and following the instructions in the Installation Guide, Configuring the Switch for the First Time, must be done before any remote management is available. For the Socket pinout see RJ-45 CONSOLE and I2C Interface on page 63. 3.5.3.2 RJ-45 Ethernet Connector The 2 Ethernet connections labeled provide access for remote management. The switch can be connected to any Ethernet port. The Ethernet port on the connector side of the switch is mgmt0. The Ethernet port on the power side of the switch is mgmt1.
Each Ethernet connector gets connected to Ethernet switches. These switches must be configured to 100M/1G auto-negotiation.
3.5.3.3 USB Connector There is a single USB connector. This connector can be used to install software and or firmware upgrades using a memory device that has a USB connector. This connector is USB 2.0 compliant.
Do NOT use excessive force when inserting or extracting the USB from the connector.
3.5.3.4 I2C Connector There is an I2C connector on the far right of the power side of the switch. This interface is for Debug and Troubleshooting only. This connector can be used to install firmware upgrades, should the firmware image be damaged and cannot be upgraded through a host PC or remotely. This interface is for support personnel and advanced users only.
24
Mellanox Technologies
Rev 1.1
3.5.4
Reset Button
On the connector side panel under the system LEDs is a reset button. This reset button requires a tool to be pressed.
DO NOT use a sharp pointed object such as needle or push pin for pressing the Reset button. Sharp objects can cause damage, use a flat object to push the reset button.
Press the reset button to reset the main and management CPUs and to delete the existing password.
PS
UID
RST
This button resets both the CPU of the switch device and the CPU of the management module. It thereby resets all of the ports by bringing them down and powering them up when the button is pushed. A quick push of this button performs this reset. When the button is held down for 15 seconds the switch is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin.
Mellanox Technologies
25
Rev 1.1
Installation
Installation
Installation and initialization of the switch platform are straightforward processes, requiring attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment.
4.1
2. Over-temperature
This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 45C (113F). Moreover, to guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings.
26
Mellanox Technologies
Rev 1.1
8.
9. Equipment Installation
This equipment should be installed, replaced, or serviced only by trained and qualified personnel.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards: IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:2001
Mellanox Technologies
27
Rev 1.1
Installation
The switch platform can be rack mounted and is designed for installation in a standard 19 rack. The power side of the switch includes a hot-swap power supply module, a blank cover for an optional second PS unit for redundancy, and 2 hot-swap fan trays. There are two possible air flow directions. Be sure that the switch air flow direction is compatible with your system, rack, and PS units. The connector side of the switch has the ports, system LEDs, and management connection ports. The switch platform contains auto-sensing 100 - 240 VAC connections for all possible PS units. The installer should use a rack capable of supporting the mechanical and environmental characteristics of a fully populated platform.
The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Take precautions to guarantee proper ventilation in order to maintain good airflow at ambient temperature. Cable routing in particular should not impede the air exhaust from the chassis.
28
Mellanox Technologies
Rev 1.1
4.2
Installation Kits
There are two Installation kit options. One long and one short. Both the standard and the short switches can be mounted using the long rail kit.
Table 9 - Installation Kit According to Rack Size
Kit OPN Rack Size
MSX60-BKIT MSX60-SKIT
4.3
Package Contents
Before you install your new switch, unpack the system and check to make sure that all the parts have been sent, check this against the parts list below. Check the parts for visible damage that may have occurred during shipping. The switch comes packed with the following items: 1 X switch 1 X installation kit 1X power cable one for each PS unit Type C13-C14 1 X Harness HAR000028 Harness RS232 2M cable DB9 to RJ-45 1 X SFP+ cable and module removal tool 1 X Quick Start Guide 1 X China RoHS statement
If anything is damaged or missing, contact your customer representative immediately. For customer support go to: www.mellanox.com =>Support => Customer Support Portal Login
4.4
Mechanical Installation
Tools and Customer Supplied Parts: Phillips Screwdrivers #1 and #2 ESD strap ESD mat Grounding screw Grounding wire sufficient to reach a valid ground.
For racks from 60cm to 80cm deep either use the standard depth switches with the long rail kit or the short switches with the long rail kit. Parts included in the rail kit: 2 rails 2 rail slides 2 switch slides 16 recessed flat head screws You will have extras! 10 caged nuts 10 pan head screws M6
Mellanox Technologies
29
Rev 1.1
Installation
Switch slide x2
Rail x2
Rail slide x2
1. Place the ESD mat on the floor where you will be working and put on the ESD strap. Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground. 2. Choose which side of the switch you want even with the rack vertical support. Either the side with the power supply units or the side with the QSFP connectors can be even with one of the vertical rack supports.
30
Mellanox Technologies
Rev 1.1
19" (482.6mm)
Switch
This edge of the switch is even with the rack vertical support Insufficient room for recommended bending radius
Consider the following when deciding where and how to mount the rails and rail slides: The distance between the rack and the door can be as little as 1.57 (4 cm) on one side of the rack and as much as 7(18 cm) on the other side of the rack. Keep in mind that there can be as many as 60 cables connected to the switch. Do you want the connector side recessed in the rack to allow for a larger cable bending radius? It is possible to recess the connector side by 2 (5cm) by optional placement of the switch rails. See Figure 20,Mounting Options. Will the connector side be recessed past other equipment in the rack and will this be problematic? The installation kit allows for a 2 (5cm) recess of the switch past the vertical support.
Mellanox Technologies
31
Rev 1.1
Installation
3. Decide which mounting option you want to use. 4. Screw the switch slides onto the switch. Use 5 flat head screws for short switches and 7 screws for standard depth switches, to connect each switch slide.
Figure 21: Screwing on the Rail
5 screws per side are needed for the short switch. 7 screws per side are needed for the standard switch. There are 16 screws in the kit; you will have left over screws.
32
Mellanox Technologies
Rev 1.1
Second side
5. Clip 6 caged nuts into the holes in the rack on the side of the rack you will be sliding the switch into. Check that both sides of the switch, power side and connector side, are at the same level in the rack. 6. Clip 4 more caged nuts into the holes on the opposite side of the rack. Check that both sides of the switch, left and right, are the same level in the rack. 7. Slide the rail into the rail slide. 8. Using two of the bolts for each corner install the rails and rail slides in the rack. Do not tighten the bolts yet.
Figure 23: Slide the Rail into the Rail Slide
This side of the rail kit goes on the side of the rack you will slide the switch into. This is the same side of the switch that will be next to the vertical support.
Mellanox Technologies
33
Rev 1.1
Installation
9. Slide the switch into the rails. 10. Tighten the bolts to 9.2 Nm or 81.5 pound inches. 11. Put the switch into place and screw the bolts into the nuts. Tighten the bolts to 9.2 Nm or 81.5 pound inches. 12. Ground the switch. 13. Plug in the power cables. 14. Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation.
Figure 24: Status LEDs 5 Minutes After Power On
PS
UID
RST
Warning: Any yellow or red status LEDs are cause for concern and must be dealt with immediately. It can take up to 5 minutes to boot up, during which time the status LED may indicate red.
15. You can start connecting all of the cables to the switch.
4.5
4.6
34
Mellanox Technologies
Rev 1.1
tacle. The input voltage is auto-adjusting for 100 - 240VAC, 50-60Hz power connections. The power cords should be standard 3-wire AC power cords including a safety ground and rated for 15A or higher.
Caution: The switch platform will automatically power on when AC power is applied. There is no power switch. Check all boards, power supplies, and fan tray modules for proper insertion before plugging in a power cable.
This unit is intended for installation in a Restricted Access Location. A restricted access area can be accessed only through the use of a special tool, lock and key, or other means of security.
Caution: After inserting a power cable and confirming the green system status LED light is on; make sure that the Fan Status indicator shows green. If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and/or obstacles.
Caution: When turning off the switch, make sure ALL Connector LEDS are off to ensure a powered down status.
Do not hot swap the power supply if your switch has only one power supply. You must power down the system to replace the power supply unit when there is only one PS unit in the switch. Figure 25: Two Power Inlets - Electric Caution Notifications
CAUTION
Risk of electric shock and energy hazard. The two PS units are independent. Disconnect all power supplies to ensure a powered down state inside of the switch platform.
ACHTUNG
Gafahr des elektrischen Schocks. Entferrnen des Netzsteckers elnes Netzteils spannungsfrei. Um alle Einhieten spannungsfrei zu machen sind die Netzstecker aller Netzteile zu entfernen
Risque de choc et de danger electriques. Le debranchment dune seule alimentation stabilisee ne debranch uniquement quun module Alimentation Stabilisee. Pour isoler completement le module en cause, Il faut debrancher toutes les alimentations stabilisees.
ATTENTION
4.7
Battery Replacement
Mellanox Technologies does not support battery replacement. Customer removal of the switch cover will void the warrantee. Only Remove the cover to comply with WEEE directives or to disassemble for environmentally approved disposal.
Mellanox Technologies
35
Rev 1.1
Installation
4.8
4.9
1. Connect the host PC to the Console (RJ-45) port of the switch system using the supplied cable. The Console ports for systems are shown below as examples.
Figure 26: Console Port
SX1024
6 57 58 59 60
Mellanox
Make sure to connect to the Console RJ-45 port of the switch and not to the (Ethernet) MGT port.
36
Mellanox Technologies
Rev 1.1
2. Configure a serial terminal program (for example, HyperTerminal, minicom, or Tera Term) on your host PC with the settings described in Table 10.
Table 10 - Serial Terminal Program Configuration
Parameter Setting
3. Login (from a serial terminal program) as admin and use admin as password. This starts the Mellanox configuration wizard. 4. Go through the Mellanox configuration wizard. Table 11 shows an example of a wizard session.
Table 11 - Configuration Wizard Session - IP Configuration by DHCP (Sheet 1 of 2)
Wizard Session Display (Example) Comments
Mellanox configuration wizard You must perform this configuration the first time you operate the Do you want to use the wizard for initial configu- switch or after resetting the switch. Type y and then press ration? yes <Enter>. Step1: Hostname? [switch-1] Step 2: Use DHCP on mgmt0 interface? [no] If you wish to accept the default hostname, then press <Enter>. Otherwise, type a different hostname and press <Enter>. Perform this step to obtain an IP address for the switch. (eth0 is the management port of the switch.) If you wish the DHCP server to assign the IP address, type yes and press <Enter>. If you type no (no DHCP), then you will be asked whether you wish to use the zeroconf configuration or not. If you enter yes (yes Zeroconf), the session will continue as shown in Table 12. If you enter no (no Zeroconf), then you need to enter a static IP, and the session will continue as shown in Table 13. Step 3: Enable IPv6? [yes] yes Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no The management interface will be able to use IPv6 addresses. This turns on autoconfiguration of the IPv6 addresses. This is unsuitable for DHCPv6.
Mellanox Technologies
37
Rev 1.1
Installation
Step 5: Admin password (Press <Enter> to leave To avoid illegal access to the machine, please type a password and then press <Enter>. Then confirm the password by re-enterunchanged)? <new_password> ing it. Step 6: Confirm admin password? <new_password> Note that password characters are not printed. You have entered the following information:
1. Hostname: <switch name> 2. Use DHCP on mgmt0 interface: yes 3. Enable IPv6: yes 4. Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no 5. Admin password (Enter to leave unchanged): (CHANGED)
The wizard displays a summary of your choices and then asks you to confirm the choices or to re-edit them.
Either press <Enter> to save changes and exit, or enter the configuration step number that you wish to return to.
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: <Enter> Configuration changes saved. To return to the wizard from the CLI, enter the configuration jump-start command from configuration mode. Launching CLI... <switch name>
Note: To run the command configuration jump-start you must be in Config mode.
38
Mellanox Technologies
Rev 1.1
Mellanox configuration wizard Do you want to use the wizard for initial configuration? y Step 1: Hostname? [switch-112126] Step 2: Use DHCP on mgmt0 interface? [no] Step 3: Use zeroconf on mgmt0 interface? [no] yes Step 4: Default gateway? [192.168.10.1] Step 5: Primary DNS server? Step 6: Domain name? Step 7: Enable IPv6? [yes] yes Step 8: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no Step 9: Admin password (Enter to leave unchanged)? You have entered the following information:
1. 2. 3. 4. 5. 6. 7. 8. 9. Hostname: switch-112126 Use DHCP on mgmt0 interface: no Use zeroconf on mgmt0 interface: yes Default gateway: 192.168.10.1 Primary DNS server: Domain name: Enable IPv6: yes Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: yes Admin password (Enter to leave unchanged): (unchanged)
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... <switch name> [standalone: master] >
Mellanox Technologies
39
Rev 1.1
Installation
Mellanox configuration wizard Do you want to use the wizard for initial configuration? y Step 1: Hostname? [switch-112126] Step 2: Use DHCP on mgmt0 interface? [yes] n Step 3: Use zeroconf on mgmt0 interface? [no] Step 4: Primary IP address? 192.168.10.4 Mask length may not be zero if address is not zero (interface mgmt0) Step 5: Netmask? [0.0.0.0] 255.255.255.0 Step 6: Default gateway? 192.168.10.1 Step 7: Primary DNS server? Step 8: Domain name? Step 9: Enable IPv6? [yes] yes Step 10: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no Step 11: Admin password (Enter to leave unchanged)? You have entered the following information:
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. Hostname: switch-112126 Use DHCP on mgmt0 interface: no Use zeroconf on mgmt0 interface: no Primary IP address: 192.168.10.4 Netmask: 255.255.255.0 Default gateway: 192.168.10.1 Primary DNS server: Domain name: Enable IPv6: yes Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no Admin password (Enter to leave unchanged): (unchanged)
To change an answer, enter the step number to return to. Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the "configuration jump-start" command from configure mode. Launching CLI... <switch name>[standalone: master] >
5. Before attempting a remote (for example, SSH) connection to the switch, check the mgmt0 interface configuration. Specifically, verify the existence of an IP address. To check the current mgmt0 configuration, enter the following commands:
40
Mellanox Technologies
Rev 1.1
sx-43 [standalone: master] > enable sx-43 [standalone: master] # configure terminal sx-43 [standalone: master] (config) # show interfaces mgmt0
yes yes 192.168.10.43 255.255.255.0 1000Mb/s (auto) full (auto) ethernet physical 1500 00:02:C9:11:2A:AE
1343502058 17589211 0 0 0 0
RX mcast packets:
TX TX TX TX TX
0 0 0 0 1000
For managed switches, confirm that the latest SW revision is installed on this device. Use the CLI to get the currently installed revision and then go to Mellanox.com to check the revision of the latest version.
7. Compare the results of this command with the latest version for your switch posted on the support page for your switch. This page will need a support login and password. Starting the Command Line (CLI) 1. Set up an Ethernet connection between the switch and a local network machine using a standard RJ-45 connector. 2. Start a remote secured shell (SSH) to the switch using the command ssh -l <username> <switch ip address>.
rem_mach1 > ssh -l <username> <ip address>
Mellanox Technologies
41
Rev 1.1
Installation
Mellanox MLNX-OS Switch Management Last login: Thu Apr 28 11:24:13 2011 from 192.168.10.1 Mellanox Switch switch >
3. Login to the switch (default username is admin, password admin) 4. Once you get the prompt, you are ready to use the system. Refer to MLNX-OS Command Reference Guide for additional information on the CLI commands.
4.10
4.11
4.12
42
Mellanox Technologies
Rev 1.1
5
5.1
Mellanox
Power supply
See Section 3.3 on page 20 for more information on air flow direction.
Power supply units have directional air flows similar to the fan module. The Fan module airflow must coincide with the airflow of all of the PS units. If the PS unit airflow direction is different from the fan module airflow direction the switch internal temperature will be affected. Figure 28: Power Supply Unit Extraction
OK ! OK ! OK !
Push Here
Latch release
If you are not immediately installing a new PSU you must install a blank so that the air flow for cooling the switch is not compromised.
Mellanox Technologies
43
Rev 1.1
To extract a PS unit: 1. Remove the power cord from the power supply unit. 2. Grasping the handle with your right hand, push the latch release with your thumb while pulling the handle outward. As the PS unit unseats, the PS unit status indicators will turn off. 3. Remove the PS unit. To insert a PS unit: 1. Make sure the mating connector of the new unit is free of any dirt and/or obstacles.
Do not attempt to insert a PS unit with a power cord connected to it.
2. Insert the PS unit by sliding it into the opening until a slight resistance is felt. 3. Continue pressing the PS unit until it seats completely. The latch will snap into place confirming the proper installation. 4. Insert the power cord into the supply connector. 5. Insert the other end of the power cord into an outlet of the correct voltage.
The green PS unit indicator should light. If not, repeat the whole procedure to extract the PS unit and re-insert it.
5.2
44
Mellanox Technologies
Rev 1.1
Risk of Electric Shock! With the fan module removed power pins are accessible within the module cavity. DO NOT insert tools or body parts into the fan module cavity.
Operation without a fan unit should not exceed two minutes. During fan hot-swap, if the LED indicator is OFF then the fan unit is disconnected.
There are two possible air flows for the fan unit. The air flow depends on the switch model. See Air Flow on page 20 for an explanation of the model numbers and labels.
Make sure that the fans have the air flow that matches the model number. An air flow opposite to the switch design will cause the switch to operate at a higher (less than optimal) temperature. Figure 30: Air Flow Labels
OK ! OK ! OK ! OK !
These air flow labels must be the same. Blue is power side inlet to connector side outlet
OK
OK
OK
OK
Red is connector side inlet to power side outlet To extract a Fan Unit: 1. Grasping the handle with your right hand, push the latch release with your thumb while pulling the handle outward. As the fan unit unseats, the fan unit status indicators will turn off. 2. Remove the fan unit.
Mellanox Technologies
45
Rev 1.1
Latch release
1. Make sure the mating connector of the new unit is free of any dirt and/or obstacles. 2. Insert the fan unit by sliding it into the opening until slight resistance is felt. Continue pressing the fan unit until it seats completely.
The green fan status indicator should light. If not, extract the fan unit and reinsert it. After two unsuccessful attempts to install the fan unit, power off the switch before attempting any system debug.
46
Mellanox Technologies
Rev 1.1
6
6.1
Cabling
Splitting the Data Stream Using Breakout (Fanout) Cables
Specific ports can be split by using a QSFP 1X4 breakout cable to split one 40 GbE port into 4 lanes (4 SFP+ connectors 10GbE each). These 4 lanes then go, one lane to each of the 4 SFP+ connectors.
Figure 32: Breakout or Fanout Cable
50
52
54
56
58
60
This port can be split into 4 10Gb/s SFP+ This port is unmapped according to the Port Splitting Table The maximum 10 Gb/s Ethernet ports configurable with this switch is 64.
When using a port to split a data stream into 4-10Gb/s data streams (four lanes) one of the other ports on the switch will be disabled (unmapped). When using this feature you MUST go into the MLNX-OS chassis management system and reconfigure the individual ports to split -4. See the MLNX-OS Software Command Reference Guide for instructions on port configuration.
Mellanox Technologies
47
Rev 1.1
Cabling
6.2
Cable Installation
56Gb/s and 40Gb/s are only guaranteed to work with approved Mellanox Cables.
All cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. The LED indicator, corresponding to each data port, will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port). After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When a logical connection is made the LED will change to green. When data is being transferred the light will blink green.
Do not force the cable into the cage with more than 40 newtons / 9.0 pounds force / 4 kg force. Greater insertion force may cause damage to the cable or the cage. Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row.
To remove, disengage the locks and slowly pull the connector away from the port receptacle. The LED indicator for that port will turn off when the cable is unseated. Care should be taken not to impede the air flow through the ventilation holes next to the connector ports. Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack.
48
Mellanox Technologies
Rev 1.1
The switch cannot be restarted remotely! To restart the switch you must physically go to the switch and unplug and plug in the power cord.
Mellanox Technologies
49
Rev 1.1
Support the weight of the switch when you remove the screws so that the switch does not fall.
5. Slide the switch from the rack. 6. Remove the rail slides from the rack. 7. Remove the ten caged nuts.
8.1
Disposal
According to the WEEE Directive 2002/96/EC, all waste electrical and electronic equipment (EEE) should be collected separately and not disposed of with regular household waste. Dispose of this product and all of its parts in a responsible and environmentally friendly way.
Follow the instructions found in the Mellanox Web site at mellanox.com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive. This button resets both the CPU of the switch device and the CPU of the management module. It
Press the reset button to reset the main CPU and the management CPU and to delete the existing password.
PS
UID
RST
thereby resets all of the ports by bringing them down and powering them up when the button is pushed. A quick push of this button performs this reset. When the button is held down for 15 sec-
50
Mellanox Technologies
Rev 1.1
onds the switch is reset and the password is deleted. You will then be able to enter without a password and make a new password for the user admin. You will need the MLNX-OS package or Mellanox to update firmware for this switch. The MLNX-OS package can be downloaded from: http://www.mellanox.com. => Products > Management Software > MLNX-OS. The Mellanox UFM Management software is also an option to more closely manage your fabric, switches and nodes. The switch can be managed either remotely, or Out-of-Band using MLNX-OS. The SX10XX switches come standard with a management software module for chassis management called Mellanox Operating System (MLNX-OS). MLNX-OS is installed on all based managed switch systems. MLNX-OS includes a CLI, chassis management software
The Ethernet ports for remote management connect to Ethernet switches. These switches must be configured to 100M/1G auto-negotiation.
8.2
Upgrading Software
Software and firmware updates are available from the Mellanox Support website. Copy the update to a known location on a Remote server within the users LAN. Use the CLI in order to perform software upgrades. For further information please refer to the the MLNX-OS Software User Manual section Upgrading MLNX-OS Software. Be sure to read and follow all of the instructions regarding the updating of the software on your switch system.
Mellanox Technologies
51
Rev 1.1
Troubleshooting
Troubleshooting
As soon as a switch is plugged in make sure that the green power LEDs on the PS units are on.
Issue 1. Issue 2. Status LED and or Status Health LED:
If either of these two LEDs is red unplug the switch and call your Mellanox representative.
Power supply unit:
If the LED on the PS unit is not lit or is red, check that the power cable is plugged into a working outlet.
1. Check that the power cable has a voltage within the range of 100 - 240 volts AC. 2. Check that the air flow direction of the PS units are consistent with the Fan module
air flow. 3. Remove and reinstall the power cable. 4. Remove and reinstall the PS unit.
Issue 3. The power LED for the switch shuts off: 1. Check that the there is adequate ventilation. Are the fan LEDs showing that the fans
are all up and running? 2. Make sure that there is nothing blocking the front or rear of the chassis and that the fan modules and ventilation holes are not blocked (especially dust over the holes). 3. If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner.
Issue 4. The green power LED for the fans does not come on: 1. Check that the Power LEDs are on. 2. Remove and reinstall the fan unit. Make sure the mating connector of the new unit is
free of any dirt and/or obstacles. See Section 7, Switch Shut Down Procedure, on page 49. 3. Check that the mating connector pins are not bent.
Do not run the switch if the System Status LED for the Fans is Red!
Issue 5.
The link LED for the connector does not come on:
1. Check that both ends of the cable are connected. 2. Check that the locks on the ends are secured. 3. Make sure that the latest firmware version is installed on all of the HCA cards and
the switch. 4. If media adapters are used, check that the all connections are good, tight, and secure.
Issue 6.
Issue 7.
52
Mellanox Technologies
Rev 1.1
5. If the switch comes on, Use the MLNX-OS management CLI to determine the
U-Boot 2009.01-mlnx1.4 (<date>) CPU: AMCC PowerPC 460EX Rev. A at 1000 MHz (PLB=200, OPB=100, EBC=100 MHz) Security/Kasumi support Bootstrap Option H - Boot ROM Location I2C (Addr 0x52) Internal PCI arbiter disabled 32 kB I-Cache 32 kB D-Cache Board: Mellanox PPC460EX Board FDEF: No I2C: ready DRAM: 2 GB (ECC enabled, 400 MHz, CL3) FLASH: 16 MB NAND: 1024 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE1: successfully set as root-complex 01 00 15b3 bd34 0c06 00 Net: ppc_4xx_eth0, ppc_4xx_eth1 Hit Ctrl+B to stop autoboot: 0 Mellanox MLNX-OS Boot Menu: 1. SX_PPC_M460EX 3.2.0000-dev-HA 2013-04-10 12:02:49 ppc
Mellanox Technologies
53
Rev 1.1
Troubleshooting
2. SX_PPC_M460EX 3.2.0000-dev-HA 2013-04-11 14:05:39 ppc 3. U-Boot prompt Choice: 5. Select the image to boot.
54
Mellanox Technologies
Rev 1.1
Appendix A: Specification
Table 14 - SX1024 Specification Data
Physical
Size /Weight
Mounting
Mounting: 19 Rack mount SerDes Speeds: 10 or 1Gb/s per port Connector Types: QSFP Can use QSFP to SFP+ adapter modules 912.24 BTUs/hr
Management CPU:
Power Consumption:
Typ: 40 GbE Passive cables: 152W Active Cables: 218W Optical cables: 221W SFP+ MAX 1.5 W TYP: 1.0 W
Power numbers
Max: 40GbE Passive cables: 175 W Active Cables: 263 W Optical cables: 267 W QSFP MAX 2.0 W TYP: 1.5 W Shock and Vibration: ETSI EN 300 019-2-2: 1999-09
Mellanox Technologies
55
Rev 1.1
Ethernet: Default 10Gb/s, must be manually configured to 1Gb/s through the GUI or the CLI. Data Rate: 10Gb/s
Data Rate
Regulatory Compliance
Safety:
Safety \ EMC (Emissions)
EMC (Emissions):
USA: FCC, Class A Canada: ICES, Class A EU: EN55022, Class A EU: EN55024, Class A EU: EN61000-3-2, Class A EU: EN61000-3-3, Class A Japan: VCCI, Class A ISO 7779 ETS 300 753
Environmental:
Environmental / Acoustic
EU: IEC 60068-2-64: Random Vibration EU: IEC 60068-2-29: Shocks, Type I / II EU: IEC 60068-2-32: Fall Test
Acoustic:
Switching Performance:
2.30Tb/s
Hot-Swappable: Fan Module Hot swapability / Redundancy 1+1 Redundant: Power Supplies
A.1
Approved Cables
For a list of all approved cables see: http://www.mellanox.com/related-docs/user_manuals/Mellanox_approved_cables.pdf
A.2
EMC Certifications
The list of approved certifications per switch in different regions of the world is located on the Mellanox Website at: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.pdf EMC statements are also in the Regulatory and Compliance Guide.
56
Mellanox Technologies
Rev 1.1
A.3
Mellanox Technologies
57
Rev 1.1
4 5
58
Mellanox Technologies
Rev 1.1
1 2 3 4
J3 and J6 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 GND Rx2n Rx2p GND Rx4n Rx4p GND ModPrsL IntL VccTx Vcc1 LPMode GND Tx3p Tx3n GND Tx1p Tx1n GND GND Rx1n Rx1p GND Rx3n Rx3p GND SDA SCL Vcc Rx ResetL ModSelL GND Tx4p Tx4n GND Tx2p Tx2n GND 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
GND Tx2n Tx2p GND Tx4n Tx4p GND ModSelL ResetL Vcc Rx SCL SDA GND Rx3p Rx3n GND Rx1p Rx1n GND GND Rx2n Rx2p GND Rx4n Rx4p GND ModPrsL IntL Vcc Tx Vcc 1 LPMode GND Tx3p Tx3n GND Tx1p Tx1n GND
Ground Transmitter Inverted Data Input Transmitter Non-Inverted Data Input Ground Transmitter Inverted Data Input Transmitter Non-Inverted Data Input Ground Module Select Module Reset +3.3 V Power supply receiver 2-wire serial interface clock 2-wire serial interface data Ground Receiver Non-Inverted Data Output Receiver Inverted Data Output Ground Receiver Non-Inverted Data Output Receiver Inverted Data Output Ground Ground Receiver Inverted Data Output 3 Receiver Non-Inverted Data Output 3 Ground Receiver Inverted Data Output 3 Receiver Non-Inverted Data Output 3 Ground Module Present Interrupt +3.3 V Power supply transmitter +3.3 V Power Supply Low Power Mode Ground Transmitter Non-Inverted Data Input Transmitter Inverted Data Input Ground Transmitter Non-Inverted Data Input Transmitter Inverted Data Input Ground
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Mellanox Technologies
59
8.50
Top
1 38 GND
Rev 1.1
60
GND
GND
37 TX1n
19
GND 20
TX2n
RX1n
36 TX1p
18
RX2n 21
TX2p
RX1p
35 GND
17
RX2p 22
GND
GND
Mellanox Technologies
16
34 TX3n
GND 23
TX4n
RX3n
33 TX3p
15
RX4n 24
6 5
TX4p
14
RX4p 25
7 32 GND
RX3p
GND
GND
31 LPMode
13
GND 26
8.50
ModSelL
SDA
30 Vcc1
12
ModPrsL 27
ResetL
SCL
29 VccTx
IntL 28
VccRx
VccRx
Top 18.35
28 IntL
VccTx 29
11 1 0 9
18.35
SCL
11 1 0 9
Vcc1 30 ResetL
12
SDA
27 ModPrsL
LPMode 3 1 ModSelL
GND
26 GND
GND 3 2 GND
14 13
RX3p
TX4p 25 RX4p
24 RX4n
TX3p 3 3
15
RX3n
TX4n
23 GND
TX3n 3 4
16
GND
GND
22 RX2p
GND 3 5
17
RX1p
TX2p
21 RX2n
TX1p 3 6
18
RX1n
TX2n
20 GND
TX1n 3 7
19
GND
1 GND
GND 3 8
Rev 1.1
14.80
13.60
13.70
11.85
1.55 0.50
8.50
1.80
2.55
0.60 44.95
8.50
VeeR
VeeR
VeeT 20
1 2
VccR
VeeT 17
VccT
RD+
TD19
3
TD+ 18
4
RD-
15
14
13
12
11
16
5 6
10
TX_Fault
TX_Disable
VeeR
MOD_ABS
RX_LOS
SCL
RS0
VeeT
RS1
SDA
Pin 1 2 3
Description Transmitter Ground (Common with Receiver Ground) a Transmitter Fault.b Transmitter Disable. Laser output disabled on high or open. c
Mellanox Technologies
61
Rev 1.1
Pin 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Symbol Name SDA SCL MOD_ABS RS0 RX_LOS RS1 VeeR VeeR RDRD+ VeeR VccR VccT VeeT TD+ TDVeeT
Description 2-wire Serial Interface Data Line d 2-wire Serial Interface Clock Line d Module Absent. Grounded within the module d No connection required Loss of Signal indication. Logic 0 indicates normal operation. e No connection required Receiver Ground (Common with Transmitter Ground) a Receiver Ground (Common with Transmitter Ground)a Receiver Inverted DATA out. AC Coupled Receiver Non-inverted DATA out. AC Coupled Receiver Ground (Common with Transmitter Ground) a Receiver Power Supply Transmitter Power Supply Transmitter Ground (Common with Receiver Ground) a Transmitter Non-Inverted DATA in. AC Coupled. Transmitter Inverted DATA in. AC Coupled. Transmitter Ground (Common with Receiver Ground)a
a. Circuit ground is internally isolated from chassis ground. b. TFAULT is an open collector/drain output, which should be pulled up with a 4.7k 10k Ohms resistor on the host board if intended for use. Pull up voltage should be between 2.0V to Vcc + 0.3V. A high output indicates a transmitter fault caused by either the TX bias current or the TX output power exceeding the preset alarm thresholds. A low output indicates normal operation. In the low state, the output is pulled to <0.8V. c. Laser output disabled on TDIS >2.0V or open, enabled on TDIS <0.8V d. Should be pulled up with 4.7k 10k on host board to a voltage between 2.0V and 3.6V. MOD_ABS pulls line low to indicate module is plugged in. e. LOS is open collector output. Should be pulled up with 4.7k 10k on host board to a voltage between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal.
62
Mellanox Technologies
Rev 1.1
J34
Pin 1
TXD+ TXDRXD+
Pin 8
TXD+ TXDJ34
1 2 3 4 5 6 7 8
I2C
RXD+ GND
RJ45_SDA GND GND GND GND RJ45_SCL GND GND
Pin 1
GND RXDGND
Pin 8
GND
Looking into the Socket
Mellanox Technologies
63
Rev 1.1
I2C DB9 or RJ-45 to USB Adapter Fan module with power supply side to connector side air flow Fan module with connector side to power supply side air flow 300W Power Supply w/ Power Supply Side to Connector side air flow 300W Power Supply w/ Connector side to Power Supply side air flow Power cord Type C13-C14 Installation Kit for short switches in racks 40-60 cm deep Installation Kit for short or standard switches in racks 60-80 cm deep
64
Mellanox Technologies
Rev 1.1
2. Temprature excessive
Ce matriel ne doit pas fonctionner dans une zone avec une temprature ambiante dpassant le maximum recommand de 45C (113F). Un flux dair de 200LFM cette temprature ambiante maximale est ncessaire. En outre, pour garantir un bon coulement de lair, laissez au moins 8 cm (3 pouces) despace libre autour des ouvertures de ventilation.
3. Empilage du chssis
Le chssis ne doit pas tre empil sur un autre matriel. Si le chssis tombe, il peut provoquer des blessures corporelles et des dgradations de biens.
Mellanox Technologies
65
Rev 1.1
13. Ajouter une information de connexion la masse Connect a Valid Ground to this Device
Avant de brancher lappareil la conduite dalimentation, les vis de protection la terre du terminal de lappareil doivent tre appliques linstallation de protection la Terre du btiment.
66
Mellanox Technologies
Rev 1.1
PRODUIT LASER DE CLASSE 1 et rfrences aux normes laser les plus rcentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825-1:1994+A1:1996+ A2:2001
Mellanox Technologies
67
Rev 1.1
22.
40 - 70 lbs 18 - 32 kgs
Conformment la Directive WEEE 2002/96/EC, les dchets d'quipements lectriques et lectroniques (EEE) doivent tre tris et ne peuvent tre limins avec les dchets mnagers. Veuillez disposer de ce produit et de tous ses composants de manire responsable et respectueuse de lenvironnement.
68
Mellanox Technologies
Rev 1.1
2. bertemperatur
Dieses Gert sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45C (113F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Auerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belftungsffnungen sein, um einen einwandfreien Luftstrom zu gewhrleisten.
Mellanox Technologies
69
Rev 1.1
9. Gerteinstallation
Diese Gert sollte nur von geschultem und qualifiziertem Personal installiert, ausgetauscht oder gewartet werden.
10. Gerteentsorgung
Die Entsorgung dieses Gerts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen.
13. Strahlenkontakt
Achtung Nutzung von Steuerungen oder Einstellungen oder Ausfhrung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefhrlichem Strahlenkontakt fhren.. Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards : ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+ A2:2001
70
Mellanox Technologies
Rev 1.1
40 - 70 lbs 18 - 32 kgs
Mellanox Technologies
71
Rev 1.1
Appendix I:
1. Instrucciones de instalacin
Antes de conectar el equipo a la fuente de alimentacin, leer todas las instrucciones de instalacin.
40 - 70 lbs 18 - 32 kgs
4. Sobrecalentamiento
No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la mxima recomendada: 45C. Adems, para garantizar una circulacin de aire adecuada, se debe dejar como mnimo un espacio de 8 cm (3 pulgadas) alrededor de las aberturas de ventilacin.
72
Mellanox Technologies
Rev 1.1
13. Cdigos elctricos locales y nacionales Este equipo se debe instalar conforme a los cdigos elctricos locales y nacionales.
Mellanox Technologies
73
Rev 1.1
Conforme a la Directiva 2002/96/CE sobre RAEE, todos los residuos de equipos elctricos y electrnicos (EEE) se deben recolectar por separado y no se deben eliminar junto con residuos domsticos. Al deshacerse de este producto y de todas sus partes, hgalo de una manera responsable y respetuosa con el medio ambiente.
74
Mellanox Technologies