TECHNICAL SUPPORT BULLETIN

July 23, 2012

TSB 2012-009-O

SEVERITY: Medium Operational

PRODUC T S AFFECTED:
DCX, DCX-4S, DCX 8510-4 and DCX 8510-8
CORRECTED IN RELEASE:
Credit Recovery tools available in FOS 6.3.2d, FOS 6.4.2a and higher, and all versions of FOS 7.X
provide the capabilities necessary to detect and correct loss of credits on back-end links. An
enhanced set of Credit Loss Detection and Recovery tools for back-end links will be available in FOS
6.4.3a and FOS 7.0.2 including default enablement of recovery from a complete loss of credit.

BULLETIN OVERVIEW
Although rare, the occurrence of back-end credit loss on DCX platforms has the potential of
impacting performance, and under extremely rare conditions, causing an outage. To remove the
risk of these conditions, it is recommended to enable Automatic Credit Recovery tools for backend links on all 8Gbps and 16Gbps DCX platforms.
This is a Brocade and OEM internal document,
Do not forward or distribute directly to any end user Customer.

PROBLEM STATEMENT
Due to the inherent design of Fibre Channel networks, it is possible to lose credit on any FC link,
including the internal back-end links of Brocade DCX, DCX-4S and DCX 8510 directors. The
existing Automatic Credit Recovery tools for back-end ports available in current versions of FOS
must be manually enabled for DCX, DCX-4S, and DCX 8510 platforms. Several end users have
been impacted by credit loss that could have been detected and recovered if this feature had
been enabled. Other end user environments are not operating with the minimum firmware levels
that provide these Credit Recovery tools and are also being impacted by the occurrence of
undetected back-end link credit loss. .
While most switches will never encounter any credit loss, it is possible for any switch to lose
credit as credit can be lost due to the normal rate of occurrence of bit error. Credit can also be
lost due to degrading or failing hardware components such as SFPs (Optics).

2012 Brocade Communications Systems, Inc. All Rights Reserved.

TSB 2012-009-O

1 of 5

RISK ASSESSMENT
Credit loss on external front-end links can result in congestion and back-pressure in a fabric that
under certain conditions could lead to dropped frames on ISLs or on back-end links within a DCX, DCX4S or DCX 8510. Complete credit loss on a front-end link will be automatically detected after 2
seconds of no traffic, and the link will automatically be recovered by the issuing of a Link Reset (LR).
Partial credit loss on a back-end link can also result in congestion, but only the most sensitive
environments (those without redundant paths or adequate bandwidth) might actually notice any
performance impact. The system would also need to be over-subscribed for any performance impact
to occur. Due to the Brocade switching ASICs ability to borrow extra credits when needed, most
end users would never be aware that they have lost some credit on a back-end link.
A complete loss of credit on a back-end link will cause frames to be discarded and traffic will be
impacted. Some DCX platform port blades have two physically trunked links, and before frames are
discarded, the complete loss of credit would have to exist on both of the physical links, an extremely
rare occurrence.

SYMPTOMS
Credit loss on a front-end link would be observed as a Link Reset (LR) occurring on that front-end port
and in the supportsave log file a corresponding entry would be listed as follows:
C2-5021 or C3-5021
With much lower probability, errors could occur on a back-end link and affect certain control words
causing frames or R_RDYs to be lost. When this occurs, credit will be lost on the link. If these types of
errors are occurring, the following supportsave log entries will be made
C2-5828 or C2-5832 or C3-5832
The probability of credit loss due to the occurrence of these events (noted by the supportsave log
entries above) is typically very low, as this is an expected and normal scenario in a FC environment.
However, if the bit error count rate should increase significantly, the probability of impacting a critical
control word also increases, causing a higher likelihood of credit loss. If the bit error counts rate
increase to a level where this might be happening, the system will post the following supportsave log
entries:
C2-1006 or C3-1006
C2-1010 or C3-1010
C2-5679 (on 6.X firmware, and 7.0.2 and higher)
C2-5678 (on 7.0.0x and 7.0.1x)
The firmware is always monitoring for lost credit. On systems running FOS v6.3.2b, v6.4.2a or higher
and all v7.X firmware, the following supportsave log entries indicate that credit loss has been detected
by the firmware:
C2-1012 or C3-1011
These log entries indicate that credit loss has occurred and was not the result of congestion causing
credits to be depleted. On 8Gbps DCX platforms (DCX and DCX-4S) when the Automatic Credit
Recovery tools are enabled an LR will be issued on the identified link to automatically recover credit.
On 16Gbps DCX systems (DCX 8510-4 and 8510-8), the error recovery is built into the switching ASIC
and the credit will be recovered without the need to issue a LR in most cases. Should a LR be
2012 Brocade Communications Systems, Inc. All Rights Reserved.

TSB 2012-009-O

2 of 5

required to recover credit for any 8Gbps or 16Gbps DCX system, then one of the following log entries
will be seen:
C2-1014 or C3-1014

WORKAROUND
If a back-end link is believed to have lost credit, it is recommended to bounce the core blade that
contains the suspect link by issuing slotpower offon commands for that core blade. This will reset all
back-end links to that core blade and any lost credit will be recovered.
The upcoming FOS v6.4.3a and FOS v7.0.2 firmware releases will include the following new tools to
assist in the recovery and verification of lost credits.

New command option added to the bottleneckmon CLI command that will allow a Link
Reset to be performed on a single back-end link. This will remove the need for having to
bounce the entire core blade to recover lost credit on a single back-end link. Below is the
full syntax:
bottleneckmon --linkreset <slot>/<bladeport>
The two required parameters with the -linkreset flag are the following:
slot the slot number where the VC is observed in the logs that may have lost
possible credits on a VC
bladeport the blade port number where the VC is observed in the logs that may
have lost possible credits on a VC

Note: The blade port number can be located in the Bpt column output of the
bladeportmap command which is documented in the Troubleshooting guide
A Link Reset will be performed and a RAS log indicating that this LR was sent in response
to a manual request will be indicated as noted in the following example:
2012/04/11-19:47:52, [C2-1014], 2/2, SLOT 5 | CHASSIS, WARNING,
Brocade_DCX, Link Reset on Internal Port S2,P-1(33) vc_no=0 crd(s)lost=0
manual trigger., OID:0x43228021, c2_ops.c, line: 1858, comp:insmod,
ltime:2001/06/28-19:53:00:556838

Manual Credit Check. A new Manual Credit Check tool will be provided. Should a
support organization suspect a back-end link might have lost credit, this tool can be used
to verify the back-end link and specific VC. This tool will confirm if a back-end link has
lost credit or if it is actually a case of congestion and the issue lies elsewhere.
The enhancement adds an additional flag with parameter (-check) that can be utilized
in the bottleneckmon command. Below is the full syntax.
bottleneckmon --cfgcredittools -intport check <slot>/<bladeport>,<VC>

2012 Brocade Communications Systems, Inc. All Rights Reserved.

TSB 2012-009-O

3 of 5

The three required parameters with the -check flag are the following:
slot the slot number where the VC is observed in the logs that may have lost
possible credits on a VC
bladeport the blade port number where the VC is observed in the logs that may
have lost possible credits on a VC
VC the virtual channel to check that may have lost credits

Note: The blade port number can be located in the Bpt column output of the
bladeportmap command which is documented in the Troubleshooting guide
Example usage:
bottleneckmon --cfgcredittools -intport -check 3/159,1
Started Credit loss Detection on slot 3 port 159 VC 1.
Please wait....

If no frames are transmitted and loss of credit is confirmed, and automatic credit
recovery has been enabled, the following will be displayed after the Please wait
message:
Please wait....
Detected credit loss.
Link Reset performed.

A Link Reset will be performed to recover the credits. A C2-1027 followed by a C2-1014
RAS message will be generated to record that the LR was issued due to the manual CLI
check request:
2012/04/11-19:47:52, [C2-1027], 11, SLOT 6 | CHASSIS, INFO,
Brocade_DCX, Detected credit loss on Peer internal Port of Slot
3, Port -1(39) vc_no=1 crd(s)lost=3.

2012/04/11-19:47:52, [C2-1014], 12, SLOT 6 | CHASSIS, WARNING,

Brocade_DCX, Link Reset on Internal Port S3,P-1(39) vc_no=1
crd(s)lost=3 auto trigger., OID:0x43228021, c2_ops.c, line: 1858,
comp:insmod, ltime:2001/06/28-19:53:00:556838.

If the internal credit recovery is disabled, no action will be taken and it will alert that the
recovery mechanism is off.
Please wait....
Detected credit loss.
Recovery not attempted: recovery using Link Reset is turned off.

If frames transmissions are detected on the specific VC during the polling cycle, then the
command will alert that this is not a case of lost credit, and the following will be seen
following the Please wait message:

2012 Brocade Communications Systems, Inc. All Rights Reserved.

TSB 2012-009-O

4 of 5

Please wait....
Detected no credit loss.

If Link Reset recovery is disabled, it may be enabled by the following command:

bottleneckmon --cfgcredittools -intport -recover onLrOnly

To disable internal credit recovery once the link is recovered, the following command may
be utilized.
bottleneckmon --cfgcredittools -intport -recover off

CORRECTIVE ACTION
In all recommended Target pPath versions of FOS, including FOS v6.3.2d, v6.4.2a and above, and all
versions of FOS 7.X:

Activate the back-end credit recovery mechanism via the bottleneckmon CLI command. This
will instruct the firmware to issue a LR whenever a loss of credit condition is detected on a
back-end link. The firmware will continuously scan the links, and during any 2-second window
of inactivity, credit levels will be confirmed.

If Link Reset recovery is disabled, it may be enabled by the following command:

bottleneckmon --cfgcredittools -intport -recover onLrOnly

In upcoming FOS releases v6.4.3a and v7.0.2 (and later):

FOS will automatically check for lost credit whenever a dropped frame occurs on a back-end
link, and should credit loss be found, the firmware will automatically issue a LR on the single
back-end link. This will provide added protection when a hardware failure causes a rapid loss
of credits and provide rapid recovery (within 2-3 seconds) after a complete credit loss
condition is detected.

The LR will only be issued by default upon the detection of a full loss of credit. A C21027 followed by a C2-1014 RAS message will be generated to record that the LR was
issued due to the auto trigger of the credit check.
2012/04/11-19:47:52, [C2-1027], 11, SLOT 6 | CHASSIS, INFO,
Brocade_DCX, Detected credit loss on Peer internal Port of Slot 3, Port
-1(39) vc_no=1 crd(s)lost=3

2012/04/11-19:47:52, [C2-1014], 12, SLOT 6 | CHASSIS, WARNING,

Brocade_DCX, Link Reset on Internal Port S3,P-1(39) vc_no=1 crd(s)lost=3
auto trigger., OID:0x43228021, c2_ops.c, line: 1858, comp:insmod,
ltime:2001/06/28-19:53:00:556838
2012 Brocade Communications Systems, Inc. All Rights Reserved.

TSB 2012-009-O

5 of 5