Fibre Channel

Basics
Module 2
33072-rev0702

2007 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice

Objectives

FC terminology and port types

Fibre Channel topologies

Point to point
Arbitrated loop
Switched fabric

Fibre Channel layers

Classes of service

Frame structure

Fibre Channel addressing

Introduction to
SANS

33072-rev0702

Fibre Channel

High speed, full duplex connections

Multi-topology environments
Long distance links
16m+ address space
10-12 error rate
Reliable delivery mechanism
Support for multiple ULPs (inc. SCSI, IP and
HiPPI)
Capable of sustaining 90%+ of stated bandwidth

Introduction to
SANS

33072-rev0702

Why not SCSI?

FC overcomes the
physical limitations of
SCSI

Eliminates SCSI skew

Improved distances

Superior throughput
and average bandwidth

Enhanced address
space

Network topology
rather than bus-based

Number of
addresses

16

16 million

Bus length

12m
/25m

120km

Throughpu
t MB/s

320

400

SCSI skew
4

Introduction to
SANS

33072-rev0702

Fibre
SCSI Channel

Nodes, Ports and Links

Host
Director switch

N port

Node N port

node

link

Disk
Array

Introduction to
SANS

33072-rev0702

port

SAN Topologies

Three connection topologies outlined by the Fibre Channel

standard:

Point-to-point (FC-P2P)

Arbitrated Loop (FC-AL)

Switched Fabric (FC-SW)

You can distinguish between the different

topologies by two criteria:
1) do we have a loop?
2) is there a switch connected?

(*) - also known as direct fabric attach (DFA)

Introduction to
SANS

33072-rev0702

Private loop

The private loop can accommodate up to 126 NL_ports

A private loop does not contain an FL_port

Contains only devices which are not fabric-aware

Cannot communicate with fabric devices

Server

Tape Library
Tx

Rx

Rx

Tx
HUB

NL-port

Introduction to
SANS

NL-port

33072-rev0702

Arbitrated loop hubs

Provide a physical star topology for a loop

configuration

Complete connections between transmitters and

receivers on a port-by-port basis through
multiplexer circuitry

Finish the loop by connecting the transmitter of

the last hub port to the receiver of the first

Have bypass circuitry at each port

Enables the loop to circumvent a disabled or

disconnected node

Introduction to
SANS

33072-rev0702

Arbitrated loop topology

Node B
Receiver

Transmitter
Receiver

Transmitter

NL_Port 11
NL_Port

NL_Port 00
NL_Port

Node A

Fibre Channel hub

Node C
Receiver
Transmitter

Introduction to
SANS

33072-rev0702

Transmitter
Receiver

NL_Port 22
NL_Port

NL_Port 3
NL_Port

Node D

Switched fabric topology

NL_Ports

Rx

Node B
Tx

NL_Port

NL_Port

Node A

Node B

Rx

Tx

HUB

NL_Port

Tx

Receiver

N_Port 0
N_Port 0

Receiver

Receiver

Transmitter

33072-rev0702

F_Port

Transmitte
r

Introduction to
SANS

Transmitter

Transmitter

Receiver

fabric

Node A

10

Receiver

N_Port 1

Transmitter

Rx

FL_Port

Node D

F_Port

FC_AL Fibre Channel

Public loop

If there is an active FL_port on the loop, it is considered a public

loop. The public loop can accommodate up to 126 NL_ports and
one FL_port

The FL_port extends the number of nodes for communication.

Using this port, a private loop can be connected to the public loop.

11

Introduction to
SANS

33072-rev0702

Topology comparison
Advantages
Point to Point

Arbitrated
Loop

Full Bandwidth for the link

Disadvantages
High cost for hardware
No scalability

Cost effective

All ports share bandwidth

Scalability

Max. 126 ports per loop only

Good topology for disk drive

I/O

Failure of one port forces Loop

Initialization
Performance is strongly dependent
on loop length and number of
NL_Ports

Highly scalable
Multiple devices communicate
simultaneously

Switched
Fabric

12

Introduction to
SANS

Loss of one component does

not interrupt the link
Full bandwidth for each switch
port

Performance only minimally

dependant on length
33072-rev0702

Higher initial cost compared to AL

Topology summary

13

Introduction to
SANS

33072-rev0702

Fibre Channel port types (1 of 3)

Node

Node
N_Port
fc-sw,
24 bit

fc-al,
24 bit

U-port

U-port

U-port

U-port

N_Port

NL_Port

fc-sw,
24 bit

fc-al,
24 bit

Node
14

NL_Port

Introduction to
SANS

Switch ports exist in U

universal state and need to
question the Node port to
ascertain final communication
topology (e.g. loop or fabric
point to point)

Node
33072-rev0702

Fibre Channel port types (2 of 3)

N_Port

NL_Port

fc-sw,
24 bit

fc-al,
24 bit

F_Port
F_Port

15

FL-port
FL-port

N_Port

NL_Port

fc-sw,
24 bit

fc-al,
24 bit

Introduction to
SANS

33072-rev0702

Switch port will query the newly

connected Nx-port to determine
if it supports loop operation
and/or is capable of 24bit
addressing.
In this way the U-port at the
switch can adapt its behaviour
to suit the topology of the newly
connected Nx-port.

Fibre Channel port types (3 of 3)

E-port
ISL
E-port
E-port

B-port

16

Introduction to
SANS

33072-rev0702

In

the event that the

recently connected port is
NOT an Nx-port Link Service
frames are exchanged to
determine the operating
parameters of the device at
the other end of the link.

WAN
connectivity

Brocade Switch Port Types

Definitions
Switch Ports:
All ports on the Brocade Switch are Universal Ports. The
U_Port is a designator for any G/E/F/FL/L port
Automatically configured ports:
E_port - Expansion port, used for inter-switch links
(ISLs)
F_port - Fabric port to which an N_Port (node)
attaches
FL_port - Fabric Loop port to which a Loop attaches
G_Port A Generic port that is in a transitional state
either to become an E_Port or F_Port
Manually configured ports:
QuickLoop (QL) port Emulated Private Loop (EPL)
17

Introduction to
SANS

33072-rev0702

Fabric Ports Types

F_Port

FL_Port

N_Port

F_Port

Connects N_Ports to the fabric

Supports Class 2 and Class 3 Services
Uses 24-bit fabric address
NL_Ports

FL_Port
Connects Arbitrated Loop NL Ports to the
fabric
Public Loop uses 24 bit fabric address
Allows Private Loop target device
connections via translative mode
18

Introduction to
SANS

33072-rev0702

Brocade Switch Port Type E_Port

ISL = 8 Virtual Channel (VC)

E_Port

E_Port
Inter Switch Link

Expansion Port Connects two SilkWorm switches in a Fabric

Support for Class F Service, Switch-to-switch communication
(Brocade)
Connections are called Inter-Switch Links (ISL)
An ISL connects two E_Ports on two different switches in a Fabric
SilkWorm Switches have eight virtual channels (VC) per ISL
VCs are a way of prioritizing congestion and improving performance
Note: The E_Port was actually designated to connect two switches from
different vendors; However this is still under definition in the FC Standards

19

Introduction to
SANS

33072-rev0702

Translative Mode
Fabric direct attach
Nodes

Fabric Assigned
Address
NN NN NN
Public Loop
Address

Private Loop
Address

LL LL PP

00 00 PP

Public Loop

Public Nodes
20

Introduction to
SANS

33072-rev0702

Private
Loop

Private
Node

Initialization Process of SilkWorm

Ports
U-Port:
y/n

y/n

FL-Port:

G-Port

Is something plugged into the port?

(Transition 1)

Do you want to talk Loop?

(Transition 2)

Waiting for someone to talk to me.

(State 3)

Are you a switch or a Fabric pt-to-pt device?

(Transition 3)

F-Port:

E-Port:
Note: Silkworm firmware will automatically attempt to reinitialize a faulty port every 2 seconds
21

Introduction to
SANS

33072-rev0702

Fibre Channel architecture

Function levels
Channels

Networks

Upper Level Protocol mappings

SCSI-3

HIPPI

SBCCS

802.
2

IP

ATM

Common Services

FC-2

Signaling / Framing Protocol / Flow

Control
Encode / Decode

FC-1
FC-0

22

133Mb/s

Introduction to
SANS

266Mb/s

33072-rev0702

531Mb/s

1063Mb/s

2125Mb/s

4250Mb/
s

Port (FC-PI)

FC-3

Node

FC-4

FC-0 Physical level

Defines the physical link in the Fibre Channel

system
Media type
Connection
Transceivers
Electrical and optical characteristics needed to connect
ports

23

Includes data rate

Introduction to
SANS

33072-rev0702

Fibre Channel cables

Copper

Optical (glass fiber)

24

Introduction to
SANS

33072-rev0702

Single-mode fiber

Highest bandwidth and lowest performance loss

One coherent stream of light travels a single
path
Long wave lasers
Single-mode, step-index fiber
9 Micron Diameter

Core

25

Introduction to
SANS

33072-rev0702

Cladding
125
Micron

Single-Mode

Single-mode step-index fiber

The preferred medium for long-distance telecommunications

26

Introduction to
SANS

33072-rev0702

Multi-mode fiber

Multiple streams of light to travel different paths

Most popular for networking

50/62.5 Micron Diameter

Core

27

Introduction to
SANS

33072-rev0702

Cladding
125
Micron

Multimode

Multi-mode fiber

28

Introduction to
SANS

33072-rev0702

Attenuation

Loss of power as a signal travels over a distance

Specified in decibels per kilometer (dB/km)

Lessened with higher-quality, more expensive,

single mode fibers

Greater with lower quality, less expensive,

multimode fibers

Results from:
Light absorption caused by material impurities
Light scattering caused by material impurities or by the
defects at the core/cladding interface, and by the
scattering of the molecules of the medium
Macro bends
Micro bends
Scattering and reflection at cable splices

29

Introduction to
SANS

33072-rev0702

Dispersion

Degree of scattering of the light beam as it travels

Light is lost

Light ray

30

Introduction to
SANS

33072-rev0702

Cable bends and damage

Micro bend
Can be the cause of difficult to
diagnose bit transport errors

Macro bend
Can reduce effective data transport
distance due to signal degradation

31

Introduction to
SANS

33072-rev0702

FC-1 Coding layer

FC-1

8b/10b encode/decode

Encodes 8bit byte in to a 10bit byte for transmission

Embeds clocking signal for serial transport
Provides single byte error detection

Used

also in :

PCI Express
IEEE 1394b
Serial ATA
SSA
Gigabit Ethernet
InfiniBand

32

Introduction to
SANS

33072-rev0702

FC-1 Coding layer

Defines the 8-bit/10-bit encoding and
decoding scheme

33

Introduction to
SANS

33072-rev0702

FC-1 encoding process (8b/10b)

disparity control

+/parallel input data from buffer (8b)

a
1

b
1

0
c

d
0

e
1

5b/6b encoding switch

a
1

1
b

0
c

d
1

e
0

0f

g
1

h
1

3b/4b encoding switch

0i

0f

g
1

1
h

0j

encoded data to serialization engine (10b)

34

Introduction to
SANS

33072-rev0702

FC-2 Signaling Protocol level

35

The transport mechanism of Fibre Channel

Framing rules

Payload

Service classes and control mechanisms

Management of data transfer sequence

Building blocks

Ordered sets

Frames

Sequences

Exchanges

Introduction to
SANS

33072-rev0702

Fibre Channel terminology

Frame
Constructed of FC Words and
Ordered Sets

Frame
Frame

Sequence 1

Used to carry ULP data in the

payload

Frame

Sequence
Single direction only

Frame
Sequence 2

May consist of only one frame

Frame

Equates directly to SCSI

Information Unit

Frame

Exchange

Frame
Frame

Sequence 3

Equates to SCSI Read or SCSI

Write

Frame

Exchange
36

Introduction to
SANS

33072-rev0702

Bi-directional

Contains multiple sequences

37
Introduction to
SANS
33072-rev0702

2112 byte data field

Primitive signal

Ordered Set

Data Payload

Data Word

0-2048

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

0-64

Data Word

Opt.
Headers

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

Data Word

24

Data Word

Frame
Header

Data Word

Data Word

Data Word

Data Word

S
O
F

Data Word

Data Word

Ordered Set

Frame structure
Frame Consists of
SOF delimiter
Frame header

Optional headers
Payload

CRC

EOF delimiter

4 4
4

C E
R O
C F
I
D
L
E

Frame header
SEQ_ID

D_ID

S_ID

Unique ID allocated to any

given sequence within a
specific exchange

24 bit FC_ID of
destination port

24 bit FC_ID of transmitting port

SEQ_CNT
Incremented by x0001 for
each frame sent within a
given sequence. May
continuously increment
when sequences are
interleaved.

OX_ID
RX_ID

Originator ID temporary
and re-useable ID given on
a per Exchange basis.

Responder ID temporary
and re-useable ID given on
a per Exchange basis.

2112 byte data field

38

Introduction to
SANS

Frame
Header

Opt.
Headers

33072-rev0702

Data Payload

fill bytes

S
O
F

C E
R O
C F

Class of Service
Class of
service

39

Fibre Channel description

Class 1

Dedicated connection
In-order delivery, acknowledge first frame only
No flow control after first frame of connection

Class 2

Connectionless
Frame switched
Out-of-order delivery possible
Acknowledge each frame
Buffer-to-buffer and end-to-end flow control for all frames

Class 3

Frame switched
Out-of-order delivery possible
No acknowledgments
Buffer-to-buffer frame control for all frames

Class 4

Connection oriented
Virtual circuit
In-order delivery

Class 5

True Isynchronous no longer used

Class 6

Connection oriented
Multicast service

Class F

Introduction to
SANS

Connectionless and acknowledged (similar to Class 2)

Used by switches for fabric related traffic
Out-of-order delivery possible
Fabric may reject frames if not delivered within ED_TOV

33072-rev0702

Indicates frame
delivery
importance to
transport layer

Most widespread
CoS is Class 3

Most fabric
switches and HBAs
support CoS 2 and
3

Disk arrays tend to

support Class 3
only

FC-3 Common Services

41

The FC-3 level of the FC standard is intended to provide the common

services required for advanced switch features such as:

Services for multiple

ports on one node

Port Striping

RAID

Virtualisation

Compression

Encryption

Hunt Groups

Multicast

Introduction to
SANS

33072-rev0702

FC-3 defines common services

necessary for the higher level
capabilities. While the FC-2 layer
deals with individual N_Ports, the
FC-3 layer covers functions that
can span multiple N_Ports.

FC-4 ULP mappings

Each upper-level protocol supported by the Fibre Channel transport requires a mapping
for its Information Units to be presented to the FC lower levels for transport. The FC-4
layer provides these mappings for:
Small Computer System Interface (SCSI)
Internet Protocol (IP)
High Performance Parallel Interface (HIPPI)
Asynchronous Transfer Mode - Adaption Layer 5 (ATM-AAL5)
Intelligent Peripheral Interface - 3 (IPI-3) (disk and tape)
Single Byte Command Code Sets (SBCCS)
future ULPs...

42

Introduction to
SANS

33072-rev0702

43

Introduction to
SANS

33072-rev0702