Waseem Fayed's blog

For the IT savvy :)

Home HowTo Uncategorized Virtualization

Type text to search here...

Home > HowTo > Replication Using FCIP & Brocade 7800 SAN switches

https://wfayed.wordpress.com/2012/10/27/fcip-on-a-brocade-7800-san-switch/

October 27, 2012wfayedLeave a commentGo to comments

I was involved for a project for one of our customers that involved configuring remote replication
between two Sun 6180 storage boxes, one located in Cairo and the other located in Port Said
(which is 200km away).

We had two brocade 7800 SAN switches. The two sites are connected via a WAN link.

The WAN link (if you are not familiar with it), is usually a long distance link between two routers.
These routers are connected via a medium that supports the required distance; for example an
E1 link, or a bundle of E1s to increase bandwidth. The result is two network subnets connected
via the routers. The diagram below illustrates how network clients on both sides of the routers
can reach each other via regular network routing.

FC over IP = FCIP

SAN replication uses the FC protocol. To allow the FC protocol to travel over the WAN link we will
use the FCIP protocol. FCIP will encapsulate the FC packets into IP packets which can then be
routed over the long distance network between the two sites.

Sending large amounts of data over a WAN link usually includes sizing. The WAN connection
must have enough bandwidth to accommodate the traffic. Proper sizing has to be performed
such that the link can complete the data transfer complete in a timely fashion according to the
environment requirements.

Connections & Configuration

Connecting the 6180s to the SAN switch is pretty straight forward, just connect the replication
port of the 6180 to one of the FC ports on the SAN switch.

Configuring the FCIP tunnel is where all the work is. The entire FCIP configuration has to be
performed on both SAN switches via command line. The configuration should match on both
sides, otherwise it will not work.

First we need to configure the network ports on the SAN switch, this will just define the IP
addresses of the FC network ports. Dont get this mixed up with the management port and IP,
this is a different port and different configuration.

Once the network ports have been configured we then need to define the network routes. This
way the packets know which router to use to reach the remote subnet.

Finally, we configure the tunnel. The tunnel defines the source and target IPs in addition to any
other options required for the connection, such as IPSEC, fastwrite, compression, etc On the
brocade 7800 switch, the FCIP tunnel is designated a virtual port number. This port is not a
physical port, it is a virtual port used to identify and configure the tunnel. The first FCIP port on
the 7800 is port 16, which we will be using.

Connection Diagram

The following diagram illustrates the target configuration;

Command Syntax

portcfg ipif <network_interface> create <IP_Address> <Subnetmask> <MTU>

portcfg iproute <network_interface> create <host_subnet> <Subnetmask> <Gateway>

portcfg fciptunnel 16 create <destination_IP> <source_IP> <speed_in_bytes>

Step-by-Step Configuration

To configure the tunnel as shown in the configuration diagram, log on to the Main Switch and
run the following commands;

This following command configures the physical ge0 port with;

IP: 192.168.1.100

subnet: 255.255.255.0

MTU: 1500

SAN-MAIN:admin> portcfg ipif ge0 create 192.168.1.100 255.255.255.0 1500

Now configure port ge0 such that;

To reach 192.168.2.0/24 (the remote SAN switch FC network) use the router 192.168.1.1

SAN-MAIN:admin> portcfg iproute ge0 create 192.168.2.0 255.255.255.0 192.168.1.1

Finally, create the FCIP tunnel 16 with target 192.168.2.100 and source 192.168.1.100

SAN-MAIN:admin> portcfg fciptunnel 16 create 192.168.2.100 192.168.1.100 10000

By running the above three commands, the configuration on the MAIN SAN switch is done. We
now need to configure the other side of the tunnel on the DR switch;
SAN-DR:admin> portcfg ipif ge0 create 192.168.2.100 255.255.255.0 1500

SAN-DR:admin> portcfg iproute ge0 create 192.168.1.0 255.255.255.0 192.168.2.1

SAN-DR:admin> portcfg fciptunnel 16 create 192.168.1.100 192.168.2.100 10000

The configuration is now complete, the two 6180s should be able to communicate as if they
were connected to the same SAN switch.

Finally, additional configuration such as fast write, compression and IPSEC security can be
configured by modifying the tunnel configuration. Note that this will disrupt the FCIP traffic for a
few seconds while the reconfiguration completes. Again, this has to be done on both SAN
switches otherwise the tunnel will not come up.

Syntax for the options we used is as follows;

portcfg fciptunnel <fcip_port> modify -f 1 -c 1 -i 1 -K <32-byte key>

To apply this to our example, the 32-byte key is in hexadecimal, and can be anything as long as it
is the same on both sides.

portcfg fciptunnel 16 modify -f 1 -c 1 -i 1 -K AABBCCDDEEFF112233445566778899AA

You can also limit the bandwidth used by the tunnel. In this example I am limiting the tunnel to
use 10Mbps.

SAN-MAIN:admin> portCfg fciptunnel 16 modify -b 10000 -B 10000

!!!! WARNING !!!!

Modify operation can disrupt the traffic on the fciptunnel specified for a brief period of time.
This operation will bring the existing tunnel down (if tunnel is up) before applying new
configuration.
Continue with Modification (Y,y,N,n): [ n] y

Circuit 16.0 modify: Operation Succeeded

Viewing Your Configuration

The following output show the commands and example output of a configured tunnel;

Viewing the ge interface configuration

SAN-DR:admin> portshow ipif ge0

Port: ge0

Interface IPv4 Address NetMask Effective MTU Flags

--------------------------------------------------------------

0 10.3.202.241 255.255.255.0 1500 URM

Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running

N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp

SAN-DR:admin> portshow iproute ge0

Port: ge0

IP Address Mask Gateway Metric Flags

-------------------------------------------------------------

10.2.3.0 255.255.255.0 10.3.202.150 0 UGS

10.3.202.0 255.255.255.0 * 0 UC

10.3.202.150 255.255.255.255 * 0 UHL

Flags: U=Usable G=Gateway H=Host C=Created(Interface) S=Static L=LinkLayer(Arp)

Viewing the FCIP tunnel configuration

SAN-MAIN:admin> portshow fciptunnel all

-------------------------------------------------------------------------------

Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met

-------------------------------------------------------------------------------

16 - Up cf--- 11m13s 0.00 0.00 4 - -

-------------------------------------------------------------------------------

Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

circuit: s=sack

SAN-MAIN:admin> portshow fciptunnel all -c

-------------------------------------------------------------------------------

Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met

-------------------------------------------------------------------------------

16 - Up cf--- 11m27s 0.00 0.00 4 - -

16 0 ge0 Up ----s 11m27s 0.00 0.00 4 1000/1000 0

-------------------------------------------------------------------------------

Flags: tunnel: c=compression f=fastwrite t=Tapepipelining F=FICON T=TPerf

circuit: s=sack

SAN-DR:admin> portshow fciptunnel 16

-------------------------------------------

Tunnel ID: 16

Tunnel Description:

Admin Status: Enabled

 Oper Status: Up

Compression: On (Standard)

Fastwrite: On

Tape Acceleration: Off

TPerf Option: Off

IPSec: Enabled

Remote WWN: Not Configured

Local WWN: 10:00:00:05:33:d1:ac:c2

Peer WWN: 10:00:00:05:33:a3:73:fa

Circuit Count: 1

Flags: 0x00000000

FICON: Off

SAN-DR:admin> portshow fciptunnel 16 -c

-------------------------------------------

Tunnel ID: 16

Tunnel Description:

Admin Status: Enabled

Oper Status: Up

Compression: On (Standard)

Fastwrite: On

Tape Acceleration: Off

TPerf Option: Off

IPSec: Enabled

Remote WWN: Not Configured

Local WWN: 10:00:00:05:33:d1:ac:c2

 Peer WWN: 10:00:00:05:33:a3:73:fa

Circuit Count: 1

Flags: 0x00000000

FICON: Off

-------------------------------------------

Circuit ID: 16.0

Circuit Num: 0

Admin Status: Enabled

Oper Status: Up

Remote IP: 10.2.3.241

Local IP: 10.3.202.241

Metric: 0

Min Comm Rt: 10000

Max Comm Rt: 10000

SACK: On

Min Retrans Time: 100

Max Retransmits: 8

Keepalive Timeout: 10000

Path MTU Disc: 0

VLAN ID: (Not Configured)

L2CoS: (VLAN Not Configured)

DSCP: F: 0 H: 0 M: 0 L: 0

Flags: 0x00000000

Deleting FCIP tunnel

If you want to change the IP addresses of the ge interfaces you will need to delete the entire
configuration and start over.
The following sequence of commands will delete the FCPIP tunnel, the route and IP address
assigned to the FC network interface. The tunnel is FC IP is dependent on the route, and the
route is dependent on the tunnel. The commands are arranged by dependency, so if you want to
delete a components, you must also delete all the components above it.

portcfg fciptunnel <fcip_port_number> delete

portcfg iproute <fc_network_interface> delete <host_subnet> <subnet_mask>

portcfg ipif <fc_network_interface> delete <fc_ip> <subnetmas>

Advertisements

Share this:

PrintEmailMore

Categories: HowToTags: FCIP, Replication, SAN

Comments (5) Trackbacks (0) Leave a commentTrackback

Tammy

August 30, 2013 at 9:00 pm Reply

Hello everyone, its my first visit at this site,

and article is really fruitful designed for me, keep up posting these content.

Like

Ahmed

October 29, 2014 at 8:23 am Reply

Thanks for the great post. I have a question: How do I make the two switches act as one switch?
For example, how do I manage the ports on one SAN switch using the other switch after creating
an FCIP tunnel between the two?

Like

wfayed

November 6, 2014 at 3:32 am Reply

I did want to configure zoning between the two switches in this solution but was not able to. I
dont remember if the fabric actually joined or not, Ill update again soon with notes on this
though.

But your question made me curious to look into it again so quick search and I found this blog
post. The author says that the fabric should merge, and that if different zoning exists then zone
conflicts will occur. Check it out and let me know how it works out for you.

Like

Jorge

January 23, 2015 at 8:47 am Reply

Hello,

Could you show us which would be the trace from 192.168.1.100 to 192.168.1.200 ??

portcmd tracerout ge0 -s ipsource -d ipdestination

Like

wfayed

July 18, 2016 at 4:46 pm Reply

I am so sorry, I never saw this comment, I know this is ancient, but I thought I should answer
anyways.

I dont have access to this environment anymore, nor did I still have it back when the comment
was made, so I am really sorry I cannot provide the traceroute output.

Like

Leave a Reply
Enter your comment here...

Mirroring Internal disks with SVM My first post

RSS feed

Twitter

Follow Blog via Email

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 6 other followers

Enter your email address

Follow

Archives

January 2013 (1)

December 2012 (6)

October 2012 (7)

Top Posts & Pages

Replication Using FCIP & Brocade 7800 SAN switches

Top

Create a free website or blog at WordPress.com.

Follow

:)