M 06 Res 01

This module focuses on network connectivity options for designing and implementing SRDF solutions.
Design considerations for SRDF over Fibre Channel and GigE, as well as optical networks are described. Some examples of SRDF implementation are also provided.
Copyright 2011 EMC Corporation. All rights reserved.
Module 6: Remote Replication - Network Connectivity
This lesson covers concepts of flow control, write acceleration, ISL trunking, congestion avoidance, TCP scaling, MTU size, hardware and software compression applicable to SRDF over Fibre Channel and GigE.
A dedicated or a shared network must be in place for remote replication. Storage arrays will have dedicated ESCON, Fibre Channel or Gigabit Ethernet adapters which are used for remote replication. The network between the two arrays could be ESCON or Fibre Channel for the entire distance. Such networks would be typically used for shorter distances. For extended distances, an optical or IP network must be used. Examples of optical networks are DWDM and SONET (discussed later). To connect the ESCON or Fibre Channel adapters from the arrays to these network protocols, converters may have to be used. Gigabit Ethernet adapters can be connected directly to the IP network.
ESCON is IBMs proprietary fiber-optic channel link best known for connecting mainframes and mainframe storage systems. ESCON provides 200 Mb/s serial fiber links and supports switching. ESCON is not supported with Enginuity 5874, hence we will not discuss ESCON any further in this course. Fibre Channel is a set of standards that define the physical, signaling, switching and upperlevel protocol mapping of a 1Gb, 2Gb, 4 Gb or 8 Gb interface. Low cost copper links support 25-meter connections; multimode fibre-optic links, 500 meters for 1Gb, 300 meters for 2 Gb and 150 meters for 4 Gb; single mode fibre, up to 35 km. Longer distances can be achieved with repeaters. SCSI HIPPI, ATM and IP protocols have been mapped to Fibre Channel transport. The Gigabit Ehternet has been in existence for quite sometime and is a part of most backbone connections for IT today. The GigE protocol may vary on physical connection as well as content transfer. It uses the same basic Ethernet protocol used for present network connections.
We will discuss the Fibre Channel protocol considerations as it applies to remote replication. The topics listed will be discussed.
The fibre channel protocol requires two round trips for each write. This is usually not an issue in normal host to storage operations within a SAN, because the distances that the signal has to traverse is so short that the latency due to these two round trips is negligible. The latency due to the two round trips starts to become an issue when remote replication is done over Fibre Channel. Speed of light in glass dictates that every 200 km will add 1 ms of latency. Thus if two sites are 200 km apart, two round trips would mean traversing a distance of 800 km (2 round trips * 2*200). The latency for every writes would be 4 ms. The latency in the remote replication links is an issue only for Synchronous replication, because the write cannot be acknowledged till the write has been received at the remote location. For asynchronous replication, the host response times are not affected by the distance between sites.
Write Acceleration/Fast Write is a performance enhancement feature offered by some FC switch and distance extension products. The goal of Write Acceleration or Fast Write is to minimize the latency caused by the two round trips that the FC protocol requires thus improving performance during synchronous replication. The improved performance is achieved via intelligence built into the distance equipment or switches. The intelligent switch that is local to the initiator emulates a Transfer Ready when it receives a Write Command. This allows the Initiator to start the data transfer as soon as it receives the local transfer ready. In the mean while, the local intelligent switch sends the Write Command to the Receiver and eventually receives the real Transfer Ready. The latency is reduced because the Initiator did not have to wait for the real Transfer Ready from the Receiver Port. Once the data is received by the Receiver, it sends a Status Good back to the Initiator. Data integrity is ensured because the Status Good is not emulated, the Status Good is sent by the Receiver Port only if the Write was successful. The intelligent switches have additional buffers and the switches at either end have to coordinate operations. The latency due Write Acceleration/Fast Write is equivalent to that of one round trip. References: Fibre Channel Write Acceleration Cisco White Paper Optimizing Storage Data Transfers over FC ISLs and Long-Distance FCIP WAN Links with FastWrite Brocade SAN Tech note.
The Single Round Trip (SiRT) feature for Symmetrix Fibre Channel RDF (RF) directors was introduced in November 2007 with the 5772 Service Release. SiRT is applicable only for the Synchronous SRDF mode. SiRT needs is enabled by default for 5773 and higher. For 5772 and 5573 the algorithm will only be used when I/O sizes are less than 32 KB and if the RDF link latency exceeds a particular value. The link latency limit depends on the I/O size, but one can assume that if the distances between sites are above 12 Km, SiRT will probably apply. For 5874 and above SiRT is enabled for all block sizes.
Write Acceleration/Fast Write/SiRT will always help in an SRDF/S environment. It can also help in an SRDF/AR or SRDF adaptive copy environment because the SRDF throughput is improved by about 20%. Typically, the two roundtrips (FC) versus one (GigE or Write Acceleration) does not impact SRDF/A performance since SRDF/A will send continuous stream of writes until the cycle is empty. If the links are kept full, there is no waiting for roundtrip delay and SRDF/A performance is not impacted by two roundtrips. Fibre Channel fast writes will improve performance for SRDF/A when there are "gaps" on the links due to retransmissions after link errors, or small writes that throttle SRDF throughput due to hitting I/Os per second limits on the links. The number of active volumes in a cycle can also affect performance on links where the links may be hitting the 32 writes/volume limit. EMC SiRT is compatible with fast write/write acceleration switches and extenders, as it will measure link latency and disable automatically if connected to these devices. As a best practice, it is recommended that either the EMC SiRT feature or the third-party fast write feature should be used. Both should not be enabled simultaneously.
Fibre Channel uses the BB_Credit mechanism for hardware-based flow control. The mechanism works by providing for frame acknowledgement via the R_RDY primitive that is defined in the FC1 layer. In a switched fabric, BB_Credit management may occur between any two Fibre Channel ports that are connected, thus: Between an N_Port and an F_Port (flow control between a HBA/storage_port and the switch it connects to) Between two E_Ports (flow control on an ISL between a pair of switches) An R_RDY (Receiver Ready) is sent from the receiving port to the transmitting port for every available buffer on the receiving side. The transmitting port maintains a count of free receiver buffers, and will continue to send frames if the count is greater than zero. The flow control algorithm is as follows: At login, the transmitter and the receiver exchange parameters and establish the BB_Credit value. The transmitter's count initializes to the BB_Credit value. If (BB_Credit > 0), transmitter sends a frame, and decrements the count per transmitted frame. The transmitter increments the count by 1 for each R_RDY it receives from the receiver, and returns to step 3. If the number of credits reaches 0, FC transmissions cannot occur. A timeout value will be triggered causing the link to reinitialize. Upon a link reset, the BB_Credit value resets to the value negotiated upon login. The diagram provides a view of the BB_Credit mechanism. As viewed from Port As perspective, when a link is established with Port B, BB_Credit information is exchanged. In this case, Port B provided a BB_Credit count of 5 to Port A. For Port A this means that it can transmit up to five Fibre Channel frames without receiving an R_RDY. In this manner, a port on a switched fabric has the ability to pace the frame flow into its processing buffers. This eliminates the need for switching hardware to discard frames due to high congestion.
10
Fabric extension via long-distance ISLs (up to 35 km with long wavelength optics) has become prevalent predominantly for business continuity applications. Even longer distances are possible with technologies such as Dense Wave Division Multiplexing (DWDM) and Coarse Wave Division Multiplexing (CWDM). Throughput on long links can degrade quickly if not enough frames are on the link. The longer the link, the more frames must be sent contiguously down the link to prevent this degradation. This is because the signal itself can propagate no faster than the speed of light. Light propagation time in glass is 5 microseconds per km. Due to this, a standard 2148-byte Fibre Channel frame is approximately 4 km long on a 1-Gb/s link (At 1 Gbit/sec, it takes roughly 20 microseconds to transmit 2 Kbytes; and the signal can propagate up to 4 km distance within these 20 microseconds in glass medium). Transmitting one frame across a 100 km link is similar to running a 4 km-long train on a 100 km track the track remains vastly under-utilized with just one train on the track. Configuring sufficient Buffer Credits ensures that the pipe is being used efficiently.
11
Configuring sufficient Buffer Credits ensures that the pipe is being used efficiently. The table in the slide gives the formula to compute the required number of BB_Credits assuming 100% utilization of the link. In the table, the factor of 2 in the formulas accounts for the time taken by light to travel the entire roundtrip distance: the frame itself from transmitter to the receiver; and the R_RDY back to transmitter. The 4 in the denominator comes from the discussion on the previous slide A 2K FC frame is approximately 4 km long on a 1 Gb/s link. For 10 Gb/s links, the multiplier at the end of the formula is 12 (instead of 10) because 10 Gb/s Fibre Channel transmission uses 64b/66b encoding - which is more efficient than the traditional 8b/10b encoding that is used for 1, 2 or 4 Gb/s links. As stated earlier, these computations assume that 100% utilization of the link bandwidth is the desired goal. In most cases, this is a reasonable goal since the link is likely to be expensive, and it makes sense to avoid starving it by imposing peripheral limitations. If the ISL may be permitted to go under-utilized, greater distances may be achieved with a given value of BB_Credits. For example, using a pair of 2 Gb/s switch ports capable of 120 BB_Credits, a maximum distance of 120 km can be supported based on the table. If the link utilization were allowed to degrade to 50%, the supportable distance would double to 240 km.
12
Fibre Channel distance extension can be achieved in a number of ways: Depending of the specific vendor with long wave laser optics ISL distances of up to 35 km can be achieved. FC traffic over CWDM/DWDM or SONET equipment can extend the FC SANs even further We will discuss CWDM/DWDM and SONET later on in this module. FCIP can also be used to extend FC SANs. We will discuss FCIP the next two slides. Fabric Isolation: Fabric isolation is a method to bound fabric definition. Boundaries can be defined functionally or geographically. The goal is to define the fabrics in such a way that events (such as network problems, administrative configurations changes, or hardware issues) occurring in one fabric will not impact other fabrics. In distance extension networks, isolating traffic locally while minimizing traffic across the WAN leads to remote replication deployments which are higher performing and less problematic. Dedicated fabrics: Isolation between fabrics can be achieved by using dedicated switches to form the link across the WAN. In a typical scenario each site (local and remote) will have its operational fabric where hosts access storage. A third fabric is created for traffic across the WAN. The WAN fabric will not be connected to the local and remote operational fabrics. Disadvantages of this method include the need for additional hardware and the inability to share resources between the local and remote sites. FC routing connects two or more physically separate fabrics to enable shared access to storage resources from any fabric. Only the shared resources will be presented (routed) on each side of the WAN.
13
FCIP is a tunneling protocol that transports Fibre Channel ISL traffic. FCIP uses IP as the transport protocol. An FCIP link tunnels all SRDF traffic between locations and may have one or more TCP connections between a pair of IP nodes for the tunnel end-points. From the Fibre Channel fabric view, an FCIP link is an ISL transporting all Fibre Channel control and data frames between switches, with the IP network and protocols transparent. One can configure one or more ISLs between Fibre Channel switches using FCIP links. Proper network design is required given the speed and bandwidth differences between Fibre Channel and a typical IP network. This design should take into consideration the management of congestion and overload conditions. Fibre Channel fabrics connected via FCIP link are not isolated; isolation can be achieved by using a dedicated fabric, FC routing or using a Virtual SAN.
14
There is no translation of Fibre Channel IDs. Frames are encapsulated "as is" into the IP payload. FC Frames are typically 2148 bytes in size while the MTU (Maximum Transmission Unit) size on IP networks is typically 1500 bytes The MTU can be higher up to 9000 bytes. Thus depending on the MTU size, it is possible that one FC frame may be a part of one or more TCP segments. FCIP is a tunneling protocol that allows geographically distributed fibre channel SAN islands to be transparently interconnected over existing IP-based local, metropolitan, and wide area networks. Each interconnection is called a FCIP Link. The result of a successful FCIP link (tunnel) between two SAN islands is a fully merged Fibre Channel fabric. The TCP/IP protocol is used as the underlying transport to provide congestion control and inorder delivery of error-free data. Some data storage, replication, and backup applications require large amounts of bandwidth, therefore IP networks should be provisioned for the amount of bandwidth that is required. Data traffic throttling or flow control is not handled by FCIP but would rather be controlled by the communicating fibre channel switches and devices within the fabric.
15
Trunking allows multiple ISLs to function as a single logical ISL. As a result of trunking, all of the ISLs appear as one aggregate logical link to the FSPF (Fabric Shortest Path First) algorithm. This allows each of the high-activity initiators to distribute its traffic over all the ISLs, thereby improving throughput and allowing better utilization of the resources. ISL trunking for remote replication traffic should be considered when the number of remote ports connected to a switch, multiplied by the port speed, plus other subscriber or business continuity traffic, is greater than the number of ISLs multiplied by the ISL speed. Trunking should also be implemented if any ISL is observed to be over-utilized while an adjacent ISL is minimally utilized.
Exchange based trunking: Available on Brocade, Cisco, and QLogic switches. It works by distributing exchanges evenly over a group of ISLs. Exchange-based trunking prevents frames within an exchange from getting out of order by routing all of the frames for the exchange down the same ISL.
Flow based/Open trunking: Developed by Brocade M Series and is currently marketed as Open Trunking. It works by making periodic changes to the hardware routing tables based on the amount of time being spent at zero BB transmit Credit and a percent utilization measurement. Frame based trunking: Available on Brocade B series switches. It involves the aggregation of several physical ISLs between any two adjacent switches into one logical unit so that FC frames may be evenly distributed across all of the ISLs in a trunk. Once the frames are received on the other side, they are reassembled in the proper order and sent on their way to the destination.
16
In addition to the Fibre Channel mechanisms and restrictions, SRDF flow control can be governed by the SRDF Flow Control Algorithm which allocates the number of I/Os that can be sent from the SRDF layer. This feature, introduced for Fibre Channel in Enginuity 5671, defaults to DISABLED and should only be enabled for all extended distance asynchronous links (SRDF/A, SRDF/AR, and Adaptive Copy). It should not be used for SRDF/S. The SRDF algorithm first tracks, analyzes, and monitors for certain I/O conditions. It then determines the optimal number of I/Os (jobs) to send out on the SRDF link, based on certain RTT windows: 0-40ms; 40-120ms; and over 120ms. The algorithm maintains a manageable number of I/Os (job queues) while making efforts to avoid time-out scenarios. The Fibre Channel flow control is managed from the primary (sending) side of the SRDF Fibre Channel (RF) links, based on "activity" of each of the secondary ports, and monitoring for primaryside "resource shortage" reported by the Fibre Channel link hardware.
17
Symmetrix DMX-series Eight-Port/Four-Processor RF Directors support up to four SRDF/Fibre Channel links (one per processor or slice) at 2 or 4 Gb/s. The Fibre Channel ports will autonegotiate to match 1 Gb/s ports on older Symmetrix models or Fibre Channel switches. Each processor provides two host connections if not configured for SRDF. If one port is configured for SRDF, the other port on that processor cannot be used. SRDF must be configured using Port 0 on each processor slice, starting with D0 downwards. The Symmetrix DMX series supports the four-processor/two-port and the fourprocessor/four-port Multiprotocol Channel Director (MPCD) configured for GigE (MPCDGigE), iSCSI, or FICON. The Multiprotocol Director for DMX-3 added support for Fibre Channel (FA) and SRDF over Fibre Channel (RF). The FA ports can be configured 2 ports per processor, while SRDF always requires a dedicated processor. RF links will use port 0, while port 1 of the processor (or slice) will be disabled. When SRDF over Fibre Channel (RF) ports are configured along with host FA ports, the SRDF RF ports will always start with a D0 port and end with an A0 port.
18
The architecture of the Symmetrix VMAX is very different from that of the DMX. Symmetrix VMAX systems provide a maximum of eight VMAX Engines within a system bay. Symmetrix VMAX SE systems provide one VMAX Engine within a system bay. The VMAX Engine is a system bay component that provides physical memory, front-end host connectivity (including SRDF), back-end connectivity, and connection to other VMAX Engines. Each VMAX Engine is fully populated, provides 32, 64, or 128 GB of physical memory, multiple host configuration options, and connection to eight disk array enclosures. Beginning with Enginuity 5874 Q4 2009 SR, software compression is available with the Fibre Channel port. Software compression can be enabled at the SRDF group level and SRDF mode of operation. Both sides of the link must support software compression. Higher host response times can result due to the software compression. If host response time is not a concern, bandwidth savings can be realized. Software compression should be disabled when Write Acceleration is enabled on an external Fibre Channel switch.
19
Gigabit Ethernet and IP Network considerations will be discussed in this section.
20
Flow controls are necessary because senders and receivers are often unmatched in capacity and processing power. A receiver might not be able to process TCP packets at the same speed as the sender, causing the receiver to send a pause message to its link partner to temporarily reduce the amount of data its transmitting. With Gigabit Ethernet, if buffers fill, packets are dropped. The goal of flow-control mechanisms is to prevent dropped packets that then must be retransmitted. TCP has Slow Start and Congestion Avoidance algorithms for flow control (discussed on the next slide). Some array vendors have built in flow control mechanisms on the Gigabit Ethernet card to control the amount of data placed in the replication queue to be transmitted to the secondary side. The flow control mechanism keeps track of how many I/Os are being sent across the link, and dynamically adjusts the number of I/Os waiting in the job queue. This prevents I/Os from timing out while in the queue or when sent across the link.
21
TCP has Slow Start and Congestion Avoidance algorithms to reduce the number of packets being sent. When packet loss is encountered, Slow Start will start by sending a single packet and waiting for an ACK (acknowledgement), then send two packets and wait for an ACK, then four packets, and so on. This exponential increase will continue until it reaches a window size of half the size of the one in which the packet loss had occurred. After this halfway value is reached, Congestion Avoidance takes over and the increase in the window size should be at most one packet each round-trip time. This mechanism was designed for handling small numbers of packets in a network with packet loss. When the network becomes congested, packets are dropped. Traditional TCP has a tendency to overreact to packet drops by halving the TCP window size. The resulting reduction in throughput that can occur in traditional TCP implementations is unacceptable to many storage applications. Starting with Enginuity 5671, DMX GigE supports the option to disable TCP Slow Start and just use flow control mechanisms built on the Gigabit Ethernet card. For dedicated bandwidth and loss-less network configurations, it is recommended to disable TCP Slow Start. Disabling TCP Slow Start, the GigE director will not back off to 5% I/O rate during retransmissions. Instead, it will try to maintain I/O rate just below the Speed Limit setting. It is also recommended to set Speed Limit at 90% of available bandwidth per GigE port.
22
A TCP window is the amount of data a sender can send without waiting for an ACK from the receiver. The TCP window is a flow control mechanism and ensures no congestion occurs in the network. The ordinary TCP header allocates only 16 bits for window advertisement. This limits the maximum window that can be advertised to 64 KB, limiting the throughput. The window scaling option allows windows greater than 64 KB. Both the endpoints must agree to use window scaling during connection establishment. The window scale extension expands the definition of the TCP window to 32 bits and then uses a scale factor to carry this 32-bit value in the 16-bit Window field of the TCP header. The window scale is fixed in each direction when a connection is opened.
A standard Ethernet packet is 1500 bytes. Larger packets (jumbo frames) are any packet larger than 1500 bytes, up to 9000 bytes. Maximum transmission unit (MTU) is the largest TCP packet size that can be sent across the network without the packet being fragmented. Packet fragmentation can have a negative impact. For instance, packets may not be reassembled or packets may be dropped. Fragmentation can cause such a high processor load on the device performing the fragmentation (for example, routers, encryptors, etc.) that it can severally impact (limit) throughput and/or cause the dropping of packets. To prevent packet fragmentation, verify that every device in the network path has the same MTU value set so that fragmentation does not occur.
23
In a well-designed network, the latency should be as low as possible with the number of hops kept to a minimum, given the geographical distance separating the sites. For SRDF/S, the acceptable roundtrip time (RTT) is dependent on application tolerance to latency. A best practice, however, is for latency between SRDF ports to not exceed 5ms RTT. Packet loss should never exceed 0.1%. It is recommended, however, that packet loss should not exceed 0.01% over any one-second period and should be as close to 0% as possible. SRDF/A and SRDF adaptive copy are extensively tested to ensure that SRDF continues to function with latencies to a maximum of 200 ms RTT and a packet loss up to 1%. It is recommended, however, that packet loss not exceed 0.1% for these SRDF modes.
Networks that experience packet loss greater than the recommended figures mentioned above are considered poor-quality links and will severely impact throughput rates. With SRDF/S, host response times will significantly increase resulting in application timeouts. Packets should not be delivered out of order. This will result in SRDF link bounces or will create a major degradation in performance.
The rapid variation in the latency on the network (known as jitter) should also be as low as possible. The figure shows an example of the impact of packet loss over varying degrees of latency. The problems caused by high latency and packet loss can be mitigated with the use of WAN Optimization devices. WAN optimization devices can maximize the amount of data that can be transmitted over a link. One device should be deployed at each end point. The acceleration is achieved by increased window sizes, more efficient block streaming protocols, caching of data, improved error correction etc.
24
In order to get the maximum benefit from scaling, it is important that all Gigabit ethernet ports on the primary and secondary arrays can see each other across the network environment. When GigE director ports are connected to a switched or routed network, a TCP connection is established between all ports. For example, a 2x2 configuration would have 4 TCP connections per side. Performance across multiple TCP sessions scales very well.
25
The Symmetrix DMX-series supports the four-processor/two-port and the fourprocessor/four-port Multiprotocol Channel Director (MPCD) configured for GigE (MPCDGigE), iSCSI, or FICON. On a four-port director, for DMX-1 and DMX-2, the ports can be intermixed with SRDF (GigE ) and host (iSCSI or FICON) emulations. The Multiprotocol Director for DMX-3 and DMX-4 has added support for Fibre Channel (FA) and SRDF over Fibre Channel (RF). The GigE ports on the MPCD and the GigE director interface boards have a hardware compression chip. Depending on the compressibility of the data, this can lower network bandwidth requirements. The SRDF data is compressed prior to encapsulation into IP packets.
IPsec (Internet Protocol Security) and IPv6 are available on the GigE/IPsec Multiprotocol Channel Director beginning with 5773 microcode. The embedded line-grade encryption coprocessor for this director provides strong security without degrading performance. Preshared encryption keys simplify management while maintaining security for data in transit.
26
The architecture of the Symmetrix VMAX is very different from that of the DMX. Symmetrix VMAX systems provide a maximum of eight VMAX Engines within a system bay. Symmetrix VMAX SE systems provide one VMAX Engine within a system bay. The VMAX Engine is a system bay component that provides physical memory, front-end host connectivity (including SRDF), back-end connectivity, and connection to other VMAX Engines. Each VMAX Engine is fully populated, provides 32, 64, or 128 GB of physical memory, multiple host configuration options, and connection to eight disk array enclosures. Beginning with Enginuity 5874 Q4 2009 SR, software compression can be used along with hardware compression. The result is a better compression ratio. Hardware and software compression is available for GigE SRDF only. Fibre Channel SRDF can use software compression only. Software compression must be enabled since it is disabled by default. Software compression can be enabled at the SRDF group level and SRDF mode of operation. Both sides of the link must support software compression. Higher host response times can result due to the software compression. If host response time is not a concern, bandwidth savings can be realized through the additional software compression.
27
Starting with Enginuity 5670, the GigE director has a built-in flow control mechanism to control the amount of data placed in the SRDF queue to be transmitted to the secondary side. The SRDF flow control mechanism keeps track of how many I/Os are being sent across the link, and dynamically adjusts the number of I/Os waiting in the job queue. This prevents I/Os from timing out while in the queue or when sent across the link. The GigE director has the ability to speed limit (throttle) the amount of traffic that it sends out of each port. The speed limit value can be changed dynamically (not persistent) or it can be set in the Symmetrix Configuration file (persistent). Speed limit values can be set in whole number values from 10-1000 Mb. Speed limits are set by EMC Customer Service. It is best practice recommendation to always set the speed limit when full bandwidth (1 Gb) per port is not available. Speed limit can be configured on a per port basis or across a group of ports (recommended) using a feature called "auto-adjust" speed limit. With auto-adjusting speed limits, you will combine a set of SRDF over GigE directors into a single speed limit group with an aggregate bandwidth for the entire speed limit group. Each SRDF port will automatically set its speed limit based on its share of the aggregate bandwidth, which is the total bandwidth of the speed limit group divided by the number of active members in the speed limit group. If auto-adjust speed limit is not used, it is recommended to set the speed limit/throttle rate on each individual GigE ports. To determine the optimal speed limit for an individual GigE port, first look at the total amount of network bandwidth available for SRDF and then set the speed limit on each port, ensuring that the sum of all the ports and their speed limit values do not exceed the available network bandwidth.
28
The GigE director supports jumbo frames, which is set in the Symmetrix Configuration file. However, the entire network, end-to-end, must be configured to support jumbo frames (up to 9000 bytes) or the packets may be fragmented or dropped. The GigE director MTU can be lowered to less than 1500 bytes. In cases where VPN encryption or other tunneling occurs within the network, it may be necessary to compensate for the additional overhead caused by these encapsulation protocols. In cases such as these, additional bytes may be added to the packet. The GigE director will try to negotiate the maximum MTU across the network. Since some devices do not negotiate the MTU with the GigE director, problems can arise where you may need to hard-set the desired MTU in the Symmetrix Configuration file.
In order to get the maximum benefit from scaling, it is important that all GigE ports on the primary and secondary Symmetrix can see each other across the network environment. When Symmetrix GigE director ports are connected to a switched or routed network, a TCP connection is established between all ports. For example, a 4x4 configuration would have 16 TCP connections per side. Performance across multiple TCP sessions scales very well.
29
This lesson covers the concepts of optical networks such as DWDM, CWDM and Sonet/SDH.
30
WDM and SONET/SDH Optical Networks will be discussed in this section.
31
Dense Wavelength Division Multiplexing (DWDM) is a process in which different channels of data are carried at different wavelengths over one pair of fiber-optic links. This is in contrast with a conventional fiber-optic system in which just one channel is carried over a single wavelength traveling through a single fiber. Using DWDM, several separate wavelengths (or channels) of data can be multiplexed into a multicolored light stream transmitted on a single optical fiber (dark fiber). This technique to transmit several independent data streams over a single fiber link is an approach to opening up the conventional optical fiber bandwidth by breaking it up into many channels, each at a different optical wavelength (a different color of light). Each wavelength can carry a signal at any bit rate less than an upper limit defined by the electronics, typically up to several gigabits per second. Different data formats being transmitted at different data rates can be transmitted together. Specifically, IP data, ESCON, Fibre Channel, SONET data, and ATM data can all be traveling at the same time within the optical fiber. A key benefit of DWDM is protocol transparency. Since DWDM is an optical transmission technique, the same interface type can be used to transport any bit rate or protocol. It also allows different bit rates and protocol data streams to be mixed on the same optical fiber. Available DWDM topologies include point-to-point and ring configurations with protected and unprotected schemas. DWDM technology can also be used to tie two or more metro area data centers together as one virtual data center.
32
Coarse Wave Division Multiplexing (CWDM), like DWDM, uses similar processes of multiplexing and de-multiplexing different channels by assigning different wavelengths to each channel. With CWDM technology, since the wavelengths are relatively far apart (compared to DWDM), the lasers and transponders are generally inexpensive compared to DWDM. CWDM is intended to consolidate environments containing a low number of channels at a reduced cost. The choice of using CWDM or DWDM is dependant on the specific situation. Businesses may opt for DWDM, only if their requirements cannot be satisfied by CWDM.
33
SONET a (Synchronous Optical Network) and SDH (Synchronous Digital Hierarchy) are a set of related standards for synchronous data transmission over fiber optic networks. SONET/SDH defines a technology for carrying different capacity signals through a synchronous optical network. The standard defines a byte-interleaved multiplexed transport occupying the physical layer of the OSI model. SONET/SDH is the core backbone of all major ISPs, providing low latency, high availability, zero loss, and high bandwidth, all of which make it ideally suited for replicating large amounts of data over a WAN. SONET/SDH is also found in MANs and can be a suitable transport for lower bandwidth storage applications or for those customers without access to dark fiber. SONET is the United States version of the standard published by the American National Standards Institute (ANSI). SDH is the international version of the standard published by the International Telecommunications Union (ITU).
34
SONET/SDH is not a communications protocol in and of itself, but rather a generic and allpurpose transport container for moving both voice and data. SONET/SDH can be used directly to support either ATM or Packet over SONET (PoS) networking. Generic Framing Procedure allows flexible multiplexing so that data from multiple clients sessions can be sent over the same link. GFP supports the transport of both packet-oriented (e.g., Ethernet, IP, etc.) and character-oriented (e.g., Fiber Channel) data. GFP is a framing procedure for mapping variable-length payloads into SONET/SDH synchronous-payload envelopes. The value of GFP is it provides simple encapsulation for native data protocols (such as Ethernet or Fibre Channel) with extremely low overhead requirements. GFP mapping to SONET also promises interoperability by allowing any vendors GFP-compatible equipment to pass and offload GFP payloads. The actual GFP mapping does not necessarily occur at the SONET/SDH device. If the Fibre Channel or Ethernet signal originates at a WDM device, these devices often contain cards that can wrap the native signal within GFP before converting the signal to a WDM wavelength. In this case, a SONET/SDH payload will already exist at handoff from WDM to the SONET/SDH network.
35
The basic SONET/SDH signal operates at 51.840 Mb/s and is designated STS-1 (Synchronous Transport Signal one). The STS-1 frame is the basic unit of transmission in SONET/SDH. Higher speed circuits are formed by aggregating multiple STS-1 circuits. STS-1 is equivalent to OC1. For SONET/SDH implementations, the majority of system and network performance tests will be addressed by the service provider that provides the SONET/SDH facility. Over long distances, end-to-end latency is the critical factor that determines the achievable throughput. For each Fibre Channel port, it is recommended that the number of associated STS-1s or STS-3s be less than the maximum required to achieve maximum throughput for the specified latency.
36
Virtual Concatenation (VCAT) enables multiple low-capacity SONET/SDH circuits to be grouped together by inverse multiplexing into a higher capacity link. For example, pre-VCAT, if a customer had an OC-3 and needed more bandwidth, an OC-12 had to be purchased. The OC-12 was required even if the customer required just a bit more bandwidth than available with an OC-3. STS-3c-2v (equivalent to OC-6), or STS-3c-3v (equivalent to OC-9) are options available with VCAT. SONET/SDH has the ability to dynamically scale bandwidth by using Link Capacity Adjustment Scheme (LCAS) to increase or decrease the bandwidth of virtual concatenated containers (VCAT).
37
The figure shows an example remote replication distance extension using SONET. In this example, EMC's Symmetrix Remote Data Facility is being used to replicate data between the two Symmetrix at either end. The SRDF traffic leaves the source Symmetrix over FC remote adapters. The FC traffic reaches a WDM network element via the FC switches. The WDM devices does the GFP encapsulation of the FC data and sends this to the SONET network. At the remote location, the GFP envelopes are received by the remote WDM device, which coverts the data back to FC and sends it to the FC remote adapters on the remote Symmetrix via FC switches. GFP-T is a specific implementation of the GFP protocol that "transparently" encapsulates any 8B/10B encoded frame into SONET packets for transport across the WAN. GFP-T can be used for transport of a wide range of protocols including ESCON, FICON, Fibre Channel, Gigabit Ethernet, HDTV and InfiniBand. Using GFP-T, 8B/10B characters are decoded and re-encoded into a 64B/65B word. Eight of these words are then grouped together into an octet along with control and error flags. Eight octets are then assembled into a "superblock," which is scrambled and subject to cyclic redundancy checks. The resulting packet is fully SONET compatible. GFP-F: Framed GFP encapsulates bursty client traffic such as Ethernet. The received client signal frame is mapped in its entirety into one GFP frame. GFP-F works relative to client frame boundaries (e.g.: 1 SONET frame per GFP-F frame).
38
Data compression is the process of encoding information using fewer bits than an unencoded representation would use through the use of specific encoding schemes. This is accomplished through algorithms that look for data having statistical redundancy. Compression programs simply eliminate the redundancy. Instead of repeating data over and over again, a compression program will store the data once and then refer back to it whenever it appears in the original data stream. Software compression is typically a lower cost option suited more for lower bandwidth environments (for example, OC-3). Hardware compression is typically a higher cost option, but more suitable for higher bandwidth environments. Efficiency of the compression is dependent on the algorithms used and the compressibility of the data (random versus sequential). Compression is not recommended where the available bandwidth meets or exceeds the throughput requirements.
39
Encryption devices can be used to ensure the secrecy of data transmitted between two points. Encryption refers to algorithmic schemes that encode plain text into non-readable form or cyphertext. The receiver of the encrypted text uses a key to decrypt the message, returning it to its original plain text form. A key is a randomly or user-generated set of numbers and/or characters used to encrypt and/or decrypt information.
40
This lesson provides examples of SRDF implementations in Campus, MAN and WAN configurations.
41
Campus Provides limited-distance direct attachment alternatives for dark fiber environments. Metropolitan Area Network (MAN) Provides SRDF connectivity for distances typically less than 100 km, but up to 200 km or more, utilizing dark fiber, DWDMs, or SONET. This environment is characterized by extremely low error rates and high bandwidth of fiber configurations, so error recovery and compression are less significant in choosing SRDF connectivity options. Wide Area Network (WAN) Provides SRDF connectivity over long distances using telecommunications networks, such as IP, SONET, or ATM. WANs are differentiated from MANs as being non-lossless and limited bandwidth environments, where error-recovery, data buffering, and compression capabilities of connectivity options are extremely important.
42
Please refer to the EMC Symmetrix Remote Data Facility (SRDF) Connectivity Guide for exact cable specifications and supported distances. GigE ports have higher latency than Fibre Channel ports, which will impact synchronous performance in campus configurations.
43
The use of MAN is determined by the ability of the customer or carrier to install/provision dark fiber cable that allows the customer to use the entire bandwidth on that fiber. A Metro environment is characterized by extremely low error rates and high bandwidth of fiber configurations, so error recovery and compression are less significant in choosing SRDF connectivity options. Dark fiber is very expensive and in some cases, depending on the distance separating the sites, may be impractical for use as a WAN. Dark fiber is better suited for use in a metropolitan area network (MAN). In cases where the distance is not an issue and large amounts of bandwidth are needed, dark fiber may be the best choice.
Synchronous SRDF applications requiring high throughput and low latency are excellent candidates for DWDM/CWDM over dark fiber. For instance, SRDF/S configurations within Metropolitan Area.
Networks or MANs) are commonly deployed using Fibre Channel directors or switches with DWDM over dark fiber.
44
Customers using SRDF in extended distance configurations have many connectivity options, including native GigE, Fibre Channel over IP and Fibre Channel over SONET. The most important consideration is the reliability and robustness of the WAN network. It is certainly the most costly component, but also the key to a successful and well-performing SRDF application. Bandwidth sizing, based on SRDF mode and workload, depends heavily on network quality and also on connectivity features such as compression, load-balancing, and traffic shaping/prioritization.
45
The figure shows an example where CWDM edge devices at the customer sites are using GFP to map multiple transparent Fibre Channel and GigE services into OC-48 (STS-48C) SONET payloads which are in turn carried as OC-48 lambdas (wavelengths) to the metro edge DWDM office. These signals are then transported as an OC-48 lambdas over DWDM to telco carrier point of presence (POP). From there, the OC-48 signals are demultiplexed from DWDM and sent to a SONET multiplexer and transported over extended distance via an OC192 SONET network. At the final metro edge site, the reverse process occurs, taking the GFP mapped OC-48 signal back to the corresponding transparent service (FC, GigE, etc.).
46
The illustration is an example of an extended distance EMC SRDF configuration that utilizes a SONET network for Symmetrix GigE director ports. The same WAN configuration could also support FCIP extension over dedicated SONET network, rather than a routed IP network.
47
Many businesses require the highest levels of disaster recovery. A common architecture used to attain this level of disaster recovery is through the deployment of Concurrent Remote replication via Synchronous replication within a metropolitan area and an Asynchronous replication over the WAN. The figure shows a sample deployment where EMC SRDF/S over DWDM is used between the primary and Metro Secondary data centers. EMC SRDF/A, in turn, is used over SONET to locations over metro and WAN networks. This architecture provides the highest level of recoverability in the event of either local or metropolitan-wide disasters.
48
Much of the material in this module was drawn from the two references cited in the slide. These documents are periodically updated. Please download the latest version from Powerlink.
49
This module covered network connectivity options for designing and implementing SRDF solutions. Design considerations for SRDF over Fibre Channel and GigE, as well as optical networks were described. Some examples of SRDF implementation were also provided.
50
51
52

M 06 Res 01

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

M 06 Res 01

Enviado por

Direitos autorais:

Formatos disponíveis

This module focuses on network connectivity options for designing and implementing SRDF solutions.

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Gigabit Ethernet and IP Network considerations will be discussed in this section.

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

Copyright 2011 EMC Corporation. All rights reserved.

Module 6: Remote Replication - Network Connectivity

WDM and SONET/SDH Optical Networks will be discussed in this section.