SRDF v33

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.
Welcome to Symmetrix Business Continuity: SRDF Solutions. Copyright 2007 EMC Corporation. All rights reserved. These materials may not be copied without EMC's written consent. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC and SRDF are registered trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners.
SRDF Introduction
-1
SRDF Introduction
-2
SRDF Introduction
-3
The objectives for this course are shown here. Please take a moment to review them.
SRDF Introduction
-4
The objectives for this module are shown here. Please take a moment to review them.
SRDF Introduction
-5
The following modules are presented over the next three days. All training materials presented in this program are for training purposes only.
SRDF Introduction
-6
Training Schedule for SRDF Solutions, day one and two.
SRDF Introduction
-7
Training Schedule for SRDF Solutions, day three.
SRDF Introduction
-8
The above software environment supports the BCL (Business Continuance Remote) program. Appendix I is a Command Quick Reference Guide created to support this training program. All students attending this program should have a good understanding of the following: Unix file systems. The Unix Vi editor A general understanding of a Unix Volume Manager EMC Solutions Enabler Symm 5 / Symm 6 overview
SRDF Introduction
-9
Business Continuity remains at the top of every executives priority list. Yet executives find themselves in a financial tug-of-war between business continuity solutions and other projects competing for the limited resources. Fundamental to business continuity is the need to understand an organizations practices relative to the protection, availability, and usability of data.
SRDF Introduction
- 10
Failures happen - hardware, software, natural disasters etc. Downtime has a significant impact, the cost is more than just financial loss. What can we do to avoid downtime or minimize the length of time we are down? EMC offers Business Continuity Solutions that help address common failures or outages.
Host to Storage failures and Performance bottleneck of a Host Bus Adapter: PowerPath Local Storage Protection with local mirroring: TimeFinder Family of Products Remote Storage Protection and Site Protection: SRDF Family of Products
SRDF Introduction
- 11
Recovery Point Objective (RPO): Maximum amount of data loss an application can tolerate as measured in time. In other words, the amount of data loss that can be tolerated (cost of transaction versus risk). Individual customer business needs drive the technology chosen to meet specific recovery point objectives. This is also known at RPO (Recovery Point Objective). Data Characteristics that Influence Data Storage Decisions: Several factors affect the value of data including: Legislation, which can mandate how long the data must be accessible, and by whom; Business processes that are tied to points in time (book closing, quarterly reports, tax deadlines, billing cycles, etc.) Business processes that are tied to customer satisfaction service levels associated with the data as its purpose changes. For example, data can start out as transactional, then migrate to billing, then reporting and customer service, then to scoring data for a marketing system, and finally to archival. The usefulness of data to the business will vary over time, and hence the necessity to have immediate access to the data changes. The decisions about where data is placed in the storage infrastructure and the methods used to protect that data are fundamentally driven by three factors: The time required to access the data relative to the cost of the access (that is, usefulness to the business versus cost). Recovery Time Objective (RTO): This refers to the maximum time a company budgets to bring an application back online in the event of a disaster. In other words, the time it takes to recover the data once a disaster or other recovery event is declared (risk versus cost). Each change in data placement and protection criteria represents a stage in the life cycle of the data and is directly related to the usefulness or importance of the data to keep the business functioning.
SRDF Introduction
- 12
The SRDF family of software is the most powerful suite of remote storage replication solutions available for disaster recovery and business continuity. It leverages high-end Symmetrix storage architecture to offer unmatched deployment flexibility and massive scalability so you can meet mixed service level requirements with minimal operational impact. The SRDF family is the most widely deployed set of high-end remote replication solutions, and is installed in tens of thousands of demanding environments worldwide. The SRDF family is the only product that provides cross volume and storage system consistency, tight integration with industry-leading applications, and automated management for simplified usage.
SRDF Introduction
- 13
EMC has several remote replication offerings for various service level requirements. For zero data exposure, EMC offers the industry leader for synchronous mirroring: SRDF. However, as with any synchronous solution, there are characteristics that must be understood. Distance is limited by application time-outs and bandwidth must be sized for peak workload at all times. SRDF/Asynchronous is a solution for service level requirements that need Recovery Point Objectives (RPO) in the seconds-to-minutes area. SRDF/AR delivers solutions that combine SRDF with TimeFinder to create single-hop and multi-hop environments for specialized needs. These solutions offer different RPOs and have different requirements for bandwidth, supported distances, etc. No matter what your requirements are, EMC can help deliver the right Remote Replication Solution.
SRDF Introduction
- 14
SRDF/Star is a three-site1 disaster recovery solution that uses concurrent RDF technology to replicate data from a primary production site (referred to as the workload site) to a nearby remote site and a distant remote site. Data is transferred in SRDF/Synchronous (SRDF/S) mode to the nearby remote site (referred to as the synchronous target site) and in SRDF/Asynchronous (SRDF/A) mode to the distant remote site (referred to as the asynchronous target site). SRDF/Star provides consistent data protection and incremental data recovery between target sites in the event of a workload site failure
SRDF Introduction
- 15
In addition to SYMCLI (Solutions Enabler), SRDF can be managed by a variety of management tools such as EMC Control Center and Symmetrix Management Console. Through its graphical user interface, EMC ControlCenter and SMC software will organize related devices into device groups. SRDF operations may be performed on all devices in this device group by using a single command. The group information is maintained in the SYMAPI database. Another management tool is EMC Replication Manager. It is an application that automates, simplifies, and manages disk-based replications by using SRDF operations.
SRDF Introduction
- 16
EMC offers a Graphical User Interface (GUI) product called Symmetrix Management Console, or SMC. SMC enables the user to deploy web-based device management within a Symmetrix environment. The user can now choose the right management product or products to meet a set of specific requirements. Almost anything you can do using the Solutions Enabler command line (CLI) can now be done using the SMC GUI. SMC manages all Symmetrix systems running Enginuity version 5568 and up, and supports new hardware and software features and functionality at the time of product release. Addressing customer demands for enhanced platform interoperability, SMC is an independent application which runs using its own lightweight Windows/Linux server. The client runs in a browser window, supporting nearly any client with remote access to the server. The SMC GUI features closely match Solutions Enabler CLI features to include all basic monitoring, configuration, and control of Symmetrix arrays. SMC has no Symmetrix-related database other than the Solutions Enabler database, so all data automatically transfers to the full-feature ControlCenter when discovered by the Symmetrix agent. Due to the combination of being light-weight yet feature rich, early response to SMC shows it to be a product leader in this space.
SRDF Introduction
- 17
Primary SMC features include providing the facility to manage SMC access and permission levels, discover, and configure, Symmetrix arrays. Also SMC enables monitoring of replication operations within Symmetrix arrays. Configuration activities include Create devices, map and mask devices, create device groups, set Symmetrix attributes; create Symmetrix Logical Volumes from un-configured storage, create meta volume devices, both concatenated and striped. Additionally, the ability to modify existing configured storage after un-mapping it from the hosts; map and un-map one or more logical volumes to a port or ports; delete devices to convert configured storage into un-configured space; manage virtual and save devices, and manage dynamic spares. SMC enables you to monitor device status, device attributes, and operations status, perform and monitor TimeFinder/Mirror, TimeFinder/Clone, and TimeFinder/Snap operations, SRDF operations, Open Replicator sessions, Optimizer and Quality of Service features, as well as monitor and report alerts.
SRDF Introduction
- 18
The major components of the SMC Interface are highlighted above. To monitor and control operations, you can select a single object or a single folder of multiple objects in the SMC Navigation Tree above. The View Bar is used to switch between five different views: 1. Properties, 2. Configuration Session, 3. Alerts, 4. Command History and 5. Replication Monitor. The current View selected will display details in the view area. The View button and corresponding view display are color coded to match. Note the Alert counter in the top right, which also selects the Alert View. Tree Selection, View Selection, and Object Selection within the View area determine the current display. The View may be split horizontally into two or three areas depending on the detail associated with the selection.
SRDF Introduction
- 19
Almost all replication technologies available on the Symmetrix Arrays can be monitored and managed via the Symmetrix Management Console (SMC) application. Note: At the present time SRDF/Star cannot be managed via SMC.
SRDF Introduction
- 20
SMC Device Group creation is a multi-step process, similar to creating SYMCLI device groups using symdg, symld, and symbcv. The wizard is launched by right clicking the Device Group Folder and choosing the Device Group Management, and then the Create Device Group option. Once a Device Group has been created selecting Replication / SRDF Controls, Settings, or Config enables SMC / RDF control functionality on the selected Device Group. The above SMC01 Device Group has two R1 devices (02EF, and 02F0) paired with two RDF / R2 devices (02EF, and 02F0). The above device group is currently in synchronized, with no Alerts to report.
SRDF Introduction
- 21
SRDF Mode operations are shown in this slide. Right Click an RDF device group and choose Replication SRDF Settings. The dialog box shows the current mode and pair states. The mode can be changed by using the Set Mode pull down. When the Set Mode is pulled down, the user sees a Synchronous, Asynchronous, Semi Synchronous, to mention just a few. In this example, the mode can be changed from Synchronous to Asynchronous.
SRDF Introduction
- 22
These are the key points covered in this module. Please take a moment to review them.
SRDF Introduction
- 23
The objectives for this module are shown here. Please take a moment to read them.
SRDF/S (Synchronous) - 1
Symmetrix Remote Data Facility (SRDF) is a Symmetrix system based business continuance, disaster recovery, restart, and data mobility solution. In the simplest terms, SRDF is a configuration of multiple Symmetrix units that maintains real time copies of logical volume data in more than one location. The Symmetrix units can be in the same room, in different buildings within the same campus, or hundreds and even thousands of miles apart.
This slide displays the representation of the mirror positions when both the Source and the Target SRDF Logical Volumes have local protection (RAID-1). In this diagram, the Target-R2 volume is also represented with 4 mirror positions and has local protection implemented. Three of the mirror positions are used. The first two mirror positions represent local mirrors and the third mirror is occupied by SRDF. If a BCV is established with the R2 volume, then it will occupy the next available mirror position. Under normal circumstances, the R1 volume presents a Read-Write (RW) status to the host which access it, and the R2 presents Write-Disabled (WD) to its host.
A Remote Link Director is a hardware that provides communication and data paths between local and remote Symmetrix units. The Symmetrix can be configured with the following RLDs: Fibre Channel directors (RF) ESCON directors (RA) Multiprotocol Channel Directors (MPCD) available with these channel connections: FICON iSCSI for host GigE (RE) for SRDF
An SRDF group, also known as RDF group or RA group, logically defines relationships between Symmetrix systems. An SRDF group is a set of SRDF director port connections configured to communicate with a another set of SRDF director ports in another Symmetrix system. Logical volumes (devices) are assigned to SRDF groups. Many SRDF groups can share a physical link between the Remote Link Directors. There are two ways to create an RDF group - static and dynamic. Both share the same features and functionality, the difference between the two types is how they are created. Static RDF groups are created during the Symmetrix configuration, and almost always by EMC personnel. Dynamic RDF groups are created and deleted by users through a set of Symmetrix command line interface (SYMCLI) commands. Prior to SE 6.3, the Symmetrix DMX supported up to 64 total RDF groups. With SE 6.4 and 5772, 1 to 250 RDF groups are supported.
SRDF offers three types of link configurations between source (local) and target (remote) Symmetrix systems: Uni-Directional, Bidirectional and Dual Configuration. SRDF Unidirectional Link Configuration If all primary (source or R1) volumes reside in one Symmetrix system and all secondary (target or R2) volumes reside in another Symmetrix system, write operations move in one direction, from primary to secondary. Data moves in the same direction over every link in the SRDF group. SRDF Bidirectional Link Configuration If an SRDF group contains both primary and secondary volumes, write operations move data in both directions over the SRDF links for that group. SRDF Dual-Directional Link Configuration With a dual-directional configuration, multiple SRDF groups are used; some groups send data in one direction, while other groups send data in the opposite direction.
Listed are the operational modes for SRDF operations: Synchronous mode, Semi-Synchronous mode, Adaptive Copy-Write Pending mode, Adaptive Copy-Disk Copy mode, and Asynchronous mode. These operational modes are selectable based on many requirements such as RPO, bandwidth, and performance. One of the two primary SRDF modes of operations is set at the source (R1) volume during Symmetrix configuration. All source (R1) volumes are configured for either the Synchronous or Semi-Synchronous mode. These two modes are considered to be pre-determined SRDF modes, which may be altered using SymCli. Adaptive copy is the secondary mode that facilitates data sharing and migration. Asynchronous mode continually collects and sends data to the remote Symmetrix. Asynchronous mode must be set for the entire RA group. Users can set SRDF to function in a secondary or Asynchronous mode. SRDF will revert to the pre-determined primary mode if it cannot maintain the criteria to remain in the secondary mode. Domino Mode could be classified as an SRDF attribute. Not necessarily a Mode. This attribute is set or used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if the target volume become unavailable, or if all SRDF links become unavailable.
SRDF Synchronous Mode is used primarily in SRDF campus environments. In this mode of operation, Symmetrix maintains a real-time mirror image of the data of the remotely mirrored volumes. Data on the source (R1) volumes and target (R2) volumes are always fully synchronized at the completion of an I/O sequence. The sequence of operations is: A write is received from the host/server at the source. The write is transmitted to the target. An acknowledgment is provided by the target back to the source. The write is acknowledged to the Host. If step 3 never happens, the source SRDF services the I/O after a pre-determined timeout to keep the production machine running.
SRDF Semi-Synchronous Mode is used primarily in extended distance environments. Semisynchronous mode allows the primary and secondary volumes to be out of synchronization by one write I/O operation. Data must be successfully stored in the Symmetrix system containing the primary volume before an acknowledgement is sent to the local host. Semi-synchronous mode will not allow the next write operation to a primary device until a positive acknowledgement is received from the target Symmetrix system that the first write operation was received in the target Symmetrix global memory. However, any number of read operations can be performed to the primary device while awaiting acknowledgement of the first write operation. Semisynchronous mode writes data to the primary device in the source Symmetrix system, completes the I/O, and then synchronizes The data with the secondary device in the target Symmetrix. The sequence of operations is: An I/O write is received from the host/server at the source. The I/O is serviced to the host/server. The I/O is transmitted to the cache of the target. An acknowledgment is provided by the target back to the source.
SRDF Adaptive Copy Mode is used primarily for data migrations and data center moves. This operational mode is not recommended for use when mirroring for disaster recovery/restart purposes unless used with TimeFinder. This mode is very useful for initial synchronization, especially over long distances. (Used within a SRDF/Star configuration). SRDF Adaptive Copy Mode allows the source (R1) volumes and target (R2) volumes to be a out of synchronization by a number of I/Os that users can define, a skew value. There are two types of adaptive copy: Write Pending Mode and Disk Mode. Adaptive Copy data movement is handled at the track level. The target data is only usable after a full synchronization. The sequence of operations is: An I/O write is received from the host/server at the source. I/O is accumulating. I/O is serviced. The I/O is transmitted to the target. An acknowledgment is provided by the target back to the source. In Write Pending Mode, the unit of transfer across the SRDF link is the updated blocks rather than an entire track, resulting in more efficient use of SRDF link bandwidth. Data is read from global memory than from disk, thus improving overall system performance. However, the global memory is temporarily consumed by the data until it is transferred across the link. In Disk Mode, while less global memory is consumed it is typically slower to read data from disk than from global memory, additionally, more bandwidth is used because the unit of transfer is the entire track. Additionally, because it is slower to read data from disk than global memory, device resynchronization time increases. Adaptive copy disk mode should not be used if the primary volumes are not RAID protected.
SRDF/A provides a long-distance replication solution with minimal impact on performance. This protection level is intended for customers requiring minimal host application impact, who need to maintain a restartable copy of data at the target site at all time. SRDF/A continually process Write I/Os in batches. The interval between batches is referred to as a cycle. The sequence of operations is: An I/O write is received from the host/server into the cache of the source. I/O is accumulating. I/O is serviced. The I/O is transmitted to the target.
Domino Mode is used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if target volume become unavailable, or if all SRDF links become unavailable. User will need to manually re-enable the source volumes. While such a shutdown temporarily halts production processing, domino modes can prevent data integrity exposure that causes the inconsistent image on the target volume.
SRDF offers considerable flexibility for various levels of synchronization. To determine the level of synchronization, one must understand the required Recovery Point Objective. This is the amount of data that can be lost in the event of a site outage. There are other factors like distance, bandwidth, and response time latency that must be considered before determining a synchronization level.
Serialization maintains the order in which writes are received at the remote (target) Symmetrix. SRDF serialization must be maintained in order to have a recoverable/restartable copy of data at a target site. Through serialization, write fidelity is guaranteed. In normal operations, SRDF maintains order writes with Synchronous, Semi-synchronous, and Asynchronous modes. But when the link becomes unavailable for any reason, writes accumulate as invalid tracks which the application continues to function on the host. When the link is restored, the Adaptive Copy mode is used to propagate changes across the link. This introduces risk, since serialization is not maintained with Adaptive Copy.
The slide shows Adaptive Copy Disk Mode during in operation. SRDF does not guarantee serialization of the tracks being transferred in this mode. In this example, track 2 and track 8 may not be present on the target volume at the time of disaster rendering the target volume useless. Therefore, the target volume will not serve as a disaster protection mechanism. The consistency of the target volume is not maintained during the replication process in Adaptive Copy Write Pending or Disk Mode. The target is consistent only after the replication has completed.
Prior to Dynamic SRDF, the R1 and R2 pairings were static and defined in the configuration file (BIN File) on the Symmetrix. Any changes to SRDF device pairing required a new BIN file to be defined and loaded into the Source and Target Symmetrix. Dynamic SRDF available with 5x68 Enginuity code will provide the capability to change device pairings on the fly without requiring a BIN file configuration change to be performed by an EMC Customer Engineers.
An R1/R2 personality swap (or R1/R2 swap) refers to when the RDF personality of the RDF device designations of a specified device group are swapped so that source R1 device(s) become target R2 device(s) and target R2 device(s) become source R1 device(s). Dynamic RDF swaps are available with Enginuity version 5567 or later. To perform an R1/R2 swap, you must have an SRDF license with Symmetrix 5567 microcode or higher and Dynamic RDF must be enabled in your Symmetrix configuration. Sample scenarios for R1/R2 Swap - Symmetrix Load Balancing In todays rapidly changing computing environments, it is often necessary to deploy applications and storage on a different Symmetrix without having to give up disaster protection. R1/R2 swap can enable this redeployment with minimal disruption, while offering the benefit of load balancing across two Symmetrix storage arrays. - Primary Data Center Relocation Sometimes a primary data center needs to be relocated to accommodate business practices. For example, several financial institutions in New York City routinely relocate their primary data center across the Hudson River to New Jersey as part of their disaster drills. R1/R2 swaps allow these customers to run their primary applications in their New Jersey data centers. The Manhattan data centers now act as the disaster protection site. - Post-Failover Temporary Protection Measure If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of production computing to the target site without giving up the security of remote data protection. When all problems have been solved on the local Symmetrix, you have to failover again and swap the personality of the devices to go back to the original configuration.
Concurrent SRDF allows two remote SRDF mirrors of a single R1 device, e.g. use one remote copy for disaster recovery, and another for decision support or backup. Each Remote Link Director is assigned to an RA Group. With ESCON, only one RA group per RLD is allowed, but Fibre Channel SRDF RA Groups can be defined to the same RLD. Any mixture of SRDF modes is allowed, except for Sync and Semi-sync configuration and Async and Async configuration. A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: The SRDF IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode.
A BCV can only be established with one of the Target volumes, not both. In case the source is locally protected, the BCV device cannot be established with its source, because all four(4) mirror positions will be occupied 2 Synchronous remote mirrors : A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: The SRDF IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode. The same general principle applies when both remote mirrors are operating in Semi-Sync mode.
SRDF Operations - 1
SRDF Operations - 2
A device group is a logical grouping of Symmetrix volumes. There are three types of device groups: regular, rdf1, and rdf2. A device group with type regular cannot contain RDF volumes. Therefore, users must create a device group with rdf1 or rdf2 for SRDF operations. The device group definition is stored in the SYMAPI database on the host where the symdg create command was executed.
SRDF Operations - 3
SRDF Operations - 4
SRDF Operations - 5
SRDF Operations - 6
Users can perform a number of Symmetrix SRDF operations using host-based SYMCLI commands. Major SRDF operations include: ping, control, or modify operations on a device group; composite group, device file, or on a device within a device or composite group; performs Dynamic RDF group controls to add, modify, and remove a dynamic RDF group. Please refer to EMC Solutions Enabler Symmetrix CLI Version 6.3 COMMAND REFERENCE P/N 300-000-877 REV A08 for the complete list of capability of the symrdf command.
SRDF Operations - 7
The symrdf ping command checks if RDF link communication is operational. The symcfg command lists configuration information of all remote directors.
SRDF Operations - 8
The symrdf suspend command stops data transfer between specified pairs. New writes to source volume accumulate as invalid tracks.
SRDF Operations - 9
The symrdf resume command resumes data transfer between pair. The Invalid tracks will start to synchronize. However, remember that the serialization is not maintained during the synchronization. Note: The #export SYMCLI_DG=srcdg command at the top of the slide. This command is setting the SYMCLI_DG variable, to equal a specific device group name. Setting this enables the user to perform Symrdf commands on a default device group. Example: Performing a query on the device group srcdg without the SYMCLI_DG variable set.
# symrdf g srcdg query
Performing a query on the device group srcdg with the SYMCLI_DG variable set.
# symrdf query
Because the SYMCLI_DG variable has been set the command symrdf query DEV001 is performing a query on device DEV001 within the device group srcdg. Note For this module, from this point on, the SYMCLI_DG variable has been set.
SRDF Operations - 10
The symrdf set mode command changes SRDF operation mode. The noprompt (-nop) is used to bypass a confirmation question from the command line. This switch is applicable to most operations with the symrdf command.
The name of the device group can be exported as a SYMCLI environment variable so that you have to type it in each time. The query shows the SRDF status of the device group.
The disaster recovery operations for SRDF devices are: Failover from the source side to the target side, switching data processing to the target side. Failback from the target side to the source side by switching data processing to the source side. Update the source side after a failover while the target side may still be operational to its local host.
In a period of scheduled downtime for maintenance, or after a serious system problem which has rendered either the host or Symmetrix unit containing the source (R1) devices unreachable, no read/write operations can occur on the source (R1) device. In this situation, the failover operation should be initiated to make the target (R2) devices read/write enabled to their local host(s).
While the target (R2) device is still operational (Write Enabled to its local host(s)), an incremental data copy from the target (R2) device to the source (R1) device can be initiated in order to update the R1 mirror with changed tracks from the target (R2) device.
A failback, or source (R1) device takeover, is performed when you are ready to resume normal SRDF operations by initiating read/write operations on the source (R1) devices, and stopping read/write operations on the target (R2) devices. The target (R2) devices become read-only to their local host(s) while the source (R1) devices are read/write enabled to their local host(s). Host activity should be stopped prior to execution: Stop Application Unmount file systems and deactivate volume groups May be executed on either source or target Symmetrix. Will abort if data integrity cannot be guaranteed.
The decision support operations for SRDF devices are: Establish an SRDF pair by initiating a data copy from the source side to target side. The operation can be full or incremental. Restore remote mirroring. Initiates a data copy from the target side to the source side. The operation can be full or incremental. Split an SRDF pair which stops mirroring for the SRDF pairs in a device group.
The command suspends the link between source (R1) and Target (R2) volumes. It also enables read and write operations on both source and target volumes. Changes to source are kept track of as R2 invalids. Changes to target are kept track of as R1 invalids.
Resume SRDF operation retaining data from source and overwriting data on target.
Resumes SRDF operation retaining data on target and overwriting data on source.
Status of RDF Volume is a specific device group: [-i] interval [-c] count Provides summary of synchronization rate and estimated completion time.
The symrdf addgrp command creates an empty dynamic RDF group that represents another RDF link between Symmetrix 000187400011 and Symmetrix 000187400093. It adds dynamic RDF group 58 on the local Symmetrix, and RDF group 58 on the remote Symmetrix. You must specify a group label (grp58 in this case) that can be used when modifying or deleting the group. Creation of the dynamic RDF group includes director 4D from the local Symmetrix and 3C from the remote Symmetrix as the director end points of this connection. The symrdf removegrp command deletes the dynamic group. The group must be empty to be deleted. It is important to be aware of your network topology when creating dynamic RDF groups between two Symmetrix arrays. To create a dynamic RDF link (a connection) between RA directors, the director end points must be able to see each other through the Fibre Channel fabric. For example, a dynamic RDF link can be created between local and remote directors only if the Fibre Channel zoning is set up so that the two directors can see each other through the fabric.
Since Enginuity 5568, devices can be configured to be Dynamic RDF-capable devices. Dynamic RDF functionality enables you to create, delete, and swap SRDF pairs while the Symmetrix array is in operation. Using Dynamic RDF technology, you can establish SRDF device pairs from non-configured SRDF devices, then synchronize and manage them in the same way as configured SRDF pairs. Note: Running the symcfg list command to check for concurrent RDF configuration.
Using Dynamic RDF technology, you can establish SRDF device pairs from non-SRDF devices using the symrdf createpair command. Once established, the new SRDF pairs can be synchronized and managed in the same way as configured SRDF pairs. Prior to Enginuity version 5568, SRDF device pairing was limited to the static SRDF pairs set at Symmetrix configuration time. Dynamic RDF enables the creation and deletion of SRDF pairs while the Symmetrix array is in operation. The symrdf deletepair command is used to cancel the Dynamic SRDF pairing.
Consistent split allows you to avoid inconsistencies and restart problems that can occur if you split a database-related BCV without first quiescing the database. TimeFinder provides the capabilities to split off a consistent, restartable copy of a database with negligible impact on the production environment. A Composite Group is a user-defined group of SRDF devices that act in unison to maintain the integrity of a database distributed across multiple Symmetrix units or multiple RDF groups within a single Symmetrix. If a source R1 device in the Composite Group cannot propagate data to the target R2 device, data propagation from all R1 devices in the Composite Group is halted. This suspension is called tripping the Composite Group.
Composite group is another logical grouping of Symmetrix devices. There are three types of composite groups: REGULAR, RDF1, RDF2. Remote consistency protection is applicable only to the RDF type composite group. Consistency groups protect the consistency of one or more database management system (DBMS) that span RDF groups during a disaster. A Consistency Group provides synchronous disaster restart with zero data loss, even when databases span multiple hosts and Symmetrix.
Suspending or splitting the Consistency Group creates an on-demand, DBMS restartable copy of the database on the R2 target side. symrdf cg suspend The R2 devices are in the write-disabled state at the end of the trip and cannot be accessed by targetside hosts. (This maintains the consistency of the R2 database copy with the production copy on the R1 side.) symrdf cg split The R2 devices are enabled for both reads and writes by the target-side hosts.
The diagram shows that the log write cannot reach the remote Symmetrix because of a problem with the SRDF link. Meanwhile, the data write has reached the remote Symmetrix via another SRDF link. This is known as data ahead of log condition. Almost all database management systems restart from this condition, without an error or any correction. The integrity of the data in the DBMS is compromised and the data is inconsistent.
Now that the composite group is defined and consistency protection enabled, our rolling disaster begins with the loss of the SRDF links from bottom source Symmetrix to the bottom target Symmetrix. A sense code is sent back stating that the data from volume A could not be propagated to its target side. The composite group started task on the mainframe or the SYMCLI/SYMAPI detects the sense code and works with ECA or PowerPath to hold the I/O. While the I/O is held, two I/Os are sent per Symmetrix. The first request sets the volumes in the Composite Group in a suspend pending state for all volumes in the composite group, and the second request suspends the relationship between the source and target volumes (R1 and R2s). The I/O is released within milliseconds. I/O continues to occur on the source host until the complete disaster happens. The DBMS or application are not aware that we held the I/O and created a Dependent Write Consistent copy or DBMS restartable copy of the data on the target side. We have simulated a local power failure at the target side at the point of the beginning of the rolling disaster. After complete failure on the source side, the target side can be restarted and the DBMS can be restarted, which provides transactional consistency.
SRDF Control operations are performed by right clicking the RDF Device Group and choosing Replication SRDF Control. The user selected a Split operation in the above slide. The same dialog box can be used for the other actions like Failback, Failover, Establish etc. The SRDF Control window has 2 pages. On page 1 the action (Split for example) and the device pairs are chosen. Page 2 allows the user to choose options that relate to a specific action and to execute the action via the Finish button.
The above slide shows the RDF Device Group SMC01 in a split state. Selecting the SMC01 device group within the SMC Navigation Tree automatically presents the current status in the View Area window. Remember, the SMC GUI closely match Solutions Enabler CLI command, to include all basic monitoring, configuration, and control for any attached Symmetrix arrays.
SRDF/A (Asynchronous) - 1
SRRD/As unique architecture delivers a remote mirroring solution that has no impact on production applications over extended distance because I/O requests from the host are acknowledged locally. Changes made to the same data blocks are periodically sent only once to the remote Symmetrix. This enables significant operational savings through reduced bandwidth requirements. Moreover, SRDF/A provides an alternative Disaster Recovery solution in addition to SRDF/S by maintaining a consistent image of RDBMS on the R2 at all times. SRDF/A is a single solution supporting both Mainframe and Open Systems attaches. It also compliments SRDF solutions to meet mixed service level requirements. In fact, it can also share the same communication links as SRDF.
Traditional approaches to asynchronous mirroring have their architectural shortfalls. Uses a combination of cache and files to perform mirroring. Timestamps or sequence numbers are applied to each and every incoming write. Each write must have a timestamp applied to it before sending it to the remote side. That means every single write MUST be sent to the remote side, and because they do not necessarily arrive in order, they must be re-sequenced before being applied to disk. If you have writes number 100-200 pending at the remote side, all waiting for write number 99 to arrive, the system must resequence and wait for number 99 to arrive before committing writes 100-200. This creates a significant amount of overhead and data management activity in both the source and target systems as they scramble to time-stamp, send, re-order, wait for a dependant time-stamp, and then eventually commit the writes to disk at the target side.
When the SRDF/A cycle (N) is active on the source Symmetrix, it collects any new writes in the R1 Symmetrix cache, overwriting any duplicate tracks intended for data transfer over the link. The cycle is active for a pre-determined amount of time that can be configured on the Symmetrix at the time of the initial configuration of the SRDF/A environment; the default time is 30 seconds. After the set time has been reached, the delta set data inherits the next cycle position (N-1) and begins transferring the delta set over the link to the R2. Then, a new cycle N begins collecting new writes again for the next delta set transfer. In cycle (N-1), the delta set is temporarily collected on the R2 side for destaging. When the (N-1) cycle has finished transferring data into the R2 and the minimum cycle time has elapsed, the delta set inherits the next cycle position (N-2) and begins destaging the data to the R2 storage devices. The delta set data is considered committed to the R2 in cycle (N-2). Thus, it takes two(2) cycles for the changes from R1 to get to the R2 which make the shortest RPO of an SRDF/A environment to be twice the cycle. That is, if the cycle time is 30 seconds, the RPO is at least 60 seconds.
All commonly used database management systems are inherently dependent write consistent. For instance, a DBMS will not perform a log write, indicating that a transaction is complete, until it has received an acknowledgement from the storage subsystem that the log data pertaining to the transaction itself was completely written to disk. Symmetrix honors this logic in SRDF/A by treating any successor I/O , which arrives after a predecessor I/O as a dependent I/O.
As noted earlier, SRDF/A is supported under all topologies. However, if the attributes indicated in bold are also set, then the end user can create Dynamic RDF Groups, create dynamic pairs of SRDF devices, and perform Concurrent RDF operations. The switched RDF Configuration State must be enabled from the service processor. The other attributes can be set using Symmetrix Configuration Manager command symconfigure. The use of Host Throttle, and Maximum Cache Usage attributes are explained later in this module.
The output has been truncated and formatted to show just the relevant information. As displayed, the pair of Symmetrix units have four Remote Adapters each 1D, 2D, 15D, 16D. Currently all four are Online (Status).
The output shows the pairings between the local and remote RA Groups (RDF Groups). These pairings provide the logical connection between the two Symmetrix. For example, SRDF devices in RA Group 1 in sid 35 have their remote mirrors in RA Group 1 in sid 56. RA Groups 1 and 2 are of the Type Static. These were created by the CE from the service processor. RA Groups 3 and 4 are of the Type Dynamic. These were created by the user from the command line (SYMCLI). The Directors have been configured in Fibre Channel Switched mode (F-S). RA Group 4 is in SRDF/A Active state (XAS), and consistency has been enabled for this group. Legend: Group (T)ype Group Flags : : : X = Enabled, . = Disabled S = Static, D = Dynamic
Prevent Auto (L)ink Recovery
Prevent RAs Online Upon (P)ower On: X = Enabled, . = Disabled Link (D)omino Director (C)onfig : : X = Enabled, . = Disabled F-S = Fibre-Switched, F-H = Fibre-Hub G = GIGE, E = ESCON, T = T3, - = N/A RDFA Flags : - = N/A - = N/A
(C)onsistency : X = Enabled, . = Disabled, (S)tatus (R)DFA Mode : A = Active, I = Inactive,
: S = Single-session, M = MSC, - = N/A
The list of devices in RDF Group 3 are displayed here. The two devices 00B and 00C are members of the RDF Group. They are in Synchronous mode of SRDF operations.
SRDF/A can be enabled when the device pairs are operating in any of the listed modes. In the case of Adaptive Copy to SRDF/A transitions, it takes two additional cycle switches after resynchronization of data for the R2 devices to be consistent.
The device pairs are operating in SRDF Synchronous mode (S..) and the pair states are Synchronized, prior to enabling SRDF/A.
Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) A(daptive Copy) : X = Enabled, . = Disabled : D = Disk Mode, W = WP Mode, . = ACp off
Transition to SRDF/A is immediate (A..X) and the pair state is immediately Consistent. Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) A(daptive Copy) : X = Enabled, . = Disabled : D = Disk Mode, W = WP Mode, . = ACp off
C(onsistency State): X = Enabled, . = Disabled, - = N/A
In this example, the device pairs are operating in SRDF Adaptive Copy Disk Mode (C.D). There are a number of invalid tracks owed to the R2 devices. This is governed by the skew value that has been set.
Note that the transition into SRDF/A is immediate (A..X), and the group has been enabled for consistency. However, the pair state is SyncInProg. R2 device does not have consistent data until the invalid tracks owed have been resynchronized, and a further two cycle switches have occurred.
In this example, all invalid tracks were synchronized within the first cycle. Consequently, by the third cycle the RDF Pair STATE has become Consistent. Tracks not Committed to the R2 Side is a measure of data in Capture, Transmit, and Receive cycles at the time of the query.
When devices are in SRDF/A enabled state, the display includes their RDFA information. Session Number represents the RDF group number.
The Consistent Deactivation process works in three phases. In the first phase, SRDF/A operates normally. When the request to transition from SRDF/A to synchronous is received, at the next cycle switch, which guarantees an empty active cycle at the R1 side, a transition to the second phase occurs. In the second phase, new writes at the R1 side are sent directly in synchronous mode to the R2 side, with one key exception. As these write arrive at the R2, they are kept in the inactive (receive) cycle at the R2 side. And, the inactive (transmit) cycle at the R1 side continues to send data to the inactive (receive) cycle at the R2. At the next cycle switch (two switches into the process), a transition to the third phase occurs. From phase 2, when the next cycle switch is received (two cycle switches into the process), the inactive cycle at the R2 side becomes the active cycle, and the SRDF/A restore process begins and ends. At the end of the restore, when all tracks are marked write pending to the R2 devices, the Consistent Deactivation is complete.
The limitations to this function are described above. Please note the additional cache which may be required at the R2 side. Because Enginuity changes are required at both R1 and R2 side Symmetrix systems, this is a 5x71 only feature. This function fails if attempted on a 5x71 to 5670 SRDF connected Symmetrix pair.
Data on the R2 devices is always consistent in SRDF/A, even with the loss of link. However, during resynchronization, data is temporarily inconsistent on the R2 until all the invalid tracks have been sent over to R2. For this reason, it is preferable to have a point-in-time copy on a BCV, for example, prior to starting the resynchronization process. Resynchronization can be initiated by issuing symrdf establish command. The logical connection between R1 and R2 can be lost under several conditions. Some of them are listed below: Network problems leading to loss of physical connection between source and target Symmetrix dropping the links due to link saturation User issued commands such as symrdf suspend/split
SRDF/A Transmit Idle is a Reserve Capacity enhancement to EMCs SRDF/A feature that provides SRDF/A with the capability of dynamically and transparently extending the Capture, Transmit, and Receive phases of the SRDF/A cycle while masking the effects of an all SRDF links lost event. Without the SRDF/A Transmit Idle enhancement, an all SRDF links lost event would normally result in the abnormal termination of SRDF/A with either a CACA.20 or CACA.40 error. The SRDF/A Transmit Idle enhancement has been specifically designed to prevent this event from occurring. The versions of 5x71 which support Transmit Idle are
5671 Minimum release level 59.64 with Epack# 1017. 5771 Minimum release level of 92.99 with Epack# 1016 and
SRDF/A Delta Set Extension (DSE) provides a mechanism for augmenting the cache-based Delta Set buffering mechanism of SRDF/A with a disk-based buffering ability. This extended Delta Set buffering ability may allow SRDF/A to ride through larger and/or longer SRDF/A throughput imbalances than would be possible with cache-based Delta Set buffering alone.
SRDF/A DSE Pools and Save devices are managed in the same way as TimeFinder/Snap pools. A RDF group can have at most one pool of each emulation. A single rdfa_dse pool can be associated with more than one RDF group, similar to snap pools shared by multiple snap sessions. SRDF/A DSE Threshold sets the percentage of cache used for SRDF/A that will start offloading cache to disk. DSE must be enabled on both the source and target arrays. Extension on only one side of a link would lead to failure of the SRDF/A recovery with SRDF/A dropping because the R2 side would fail to have enough cache to hold the large and extended Transmit cycle.
SRDF/A has always buffered Delta Set data in cache. However a buffer full condition causes SRDF/A to drop. With SRDF/A DSE delta set data is offloaded to disk buffers (DSE Pools) by the DSE task. We will look at this in more details in the next few slides.
SRDF/A DSE should be used with the Transmit Idle feature. Thus SRDF/A can ride through a temporary link loss, once the DSE threshold is reached, data is paged out to the DSE Pools.
As noted earlier, during resynchronization R2 does not have consistent data. A BCV copy of the consistent R2 data prior to resynchronization can safeguard against unexpected failures during the resynchronization process. When the link is resumed, if there are a large number of invalid tracks owed by the R1 to its R2, it is recommended that SRDF/A not be enabled right away. Enabling SRDF/A right after link resumption causes a surge of traffic on the link due to (a) shipping of accumulated invalid tracks, and (b) the new data added to the SRDF/A cycles. This would lead to SRDF/A consuming more cache and reaching the System Write Pending limit. If this happens, SRDF/A would drop again. Like with SRDF/A, resynchronization should be performed during periods of relatively low production activity. Resynchronization in Adaptive Copy Write Pending mode minimizes the impact on the production host. New writes are buffered and these, along with the R2 invalids, are sent across the link. The time it takes to resynchronize is elongated. Resynchronization in Synchronous mode impacts the production host. New writes have to be sent preferentially across the link while the R2 invalids are also shipped. Switching to Synchronous is possible only if the distances and other factors permit. For instance, if the norm is to run in SRDF/S and toggle into SRDF/A for batch processing (due to higher bandwidth requirement). In this case, if a loss of links occurs during the batch processing, it might be possible to resynchronize in SRDF/S. In either case, R2 data is inconsistent until all the invalid tracks are sent over. Therefore, it is advisable to enable SRDF/A after the two sides are completely synchronized.
In this example there is a workload on the devices in SRDF/A enabled state. A permanent loss of link place the devices in a Partitioned state. Production work continues on the R1 devices and the new writes arriving for the R1 devices are marked as invalid or owed to the R2.
When the links are active again, note that the pair state has moved to Suspended. Also note there are R2 Invalid Tracks on the R1 side AND R1 Invalid Tracks on the R2 side. In Synchronous SRDF, one would have this condition only if changes are made to both the R1s and the R2s. However, in SRDF/A data in Receive cycle during loss of link are marked as R1 invalids. The data in the Capture and Transmit cycles, and new writes after link loss, are marked as R2 invalid on the R1 side. This is one of the behavioral differences between SRDF/S and SRDF/A.
As per recommendation, we place the devices in SRDF Adaptive Copy Write Pending mode (C.W.) and wait for the pair states to become Synchronized. Prior to changing the mode to ACP_WP, we have to disable consistency protection via symrdf g vsrdfadg3 disable. When consistency is enabled, one cannot switch out of SRDF/A without first disabling it. Once the synchronization is complete we can then enable SRDF/A and enable consistency.
The symrecover command was available through EMC Services prior to SE 6.4. symrecover offers the ability to detect and automate restarting SRDF links in a safe and secure manner.
Again, it is advisable to split off a BCV copy of the R2 prior to executing a failback operation. When workload is resumed on the R1 devices immediately after a failback, accumulated invalid tracks have to be synchronized from the R2 to the R1, and new writes must be shipped from the R1 to R2. If there is an interruption now, data on the R2 is not consistent. Even though SRDF/A can be enabled right after a failback, for reasons stated earlier, it should be enabled after the SRDF pairs entered the Synchronized state.
RA Group 3 uses all four RDF Directors 1D, 2D, 15D, and 16D. RA Group 4 uses two of the RDF Directors 15D, and 16D.
The RA Groups 3 and 4 are in SRDF/A mode (XAS), as shown in the output above.
The devices in RA Group 3 are consistent, independently of the devices in RA Group 4 (shown in the next slide).
Devices in RA Group 4 are in SRDF/A Active state and are consistent. Note that RA Group 4 has a different Cycle Number (62), than RA Group 3 (2972).
Loss of links on directors 15D, and 16D causes SRDF/A to drop for RA Group 4.
However, RA Group 3 still has 2 links available (1D, and 2D), and continues to be in SRDF/A Active state and the device pairs are Consistent.
Since Enginuity 5671, consistency protection for SRDF/Asynchronous devices is provided using Multi Session Consistency (MSC). If one or more source (R1) devices in an SRDF/A MSC enabled RDF consistency group cannot propagate data to their corresponding target (R2) devices. The MSC process suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2 targets SRDF/A with MSC supported by an RDF process daemon that performs cycle-switching and cache recovery operations across all SRDF/A sessions in the group. This ensures that a consistent R2 data copy of the database exists at the point-in-time any interruption occurs. A composite group must be created using the RDF consistency protection option (-rdf_consistency) and must be enabled using the symcg enable command before the RDF daemon begins monitoring and managing the MSC consistency group.
The RDF process daemon maintains consistency for enabled composite groups across multiple Symmetrix arrays for SRDF/A with MSC. For the MSC option (-rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host performing management operations must run an instance of the RDF daemon (storrdfd). Each host running storrdfd must also run an instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel application syscalls. Optionally, if the Group Naming Services (GNS) daemon is also running, it communicates the composite group definitions back to the RDF daemon. If the GNS daemon is not running, the composite group must be defined on each host individually.
If one or more sessions in MSC complete their Transmit and Apply cycles ahead of other sessions, they have to wait for all sessions to complete, prior to a cycle switch.
In this illustration we have two SRDF/A sessions, 1 and 2. Each have their own cycle numbers N and M. When they are placed in MSC, the RDF Daemon assigns a Tag number to the capture cycle. This is so the cycle numbers themselves do not have to be synchronized. The Tag number is incremented at every cycle switch. It is this Tag number that is compared for recovery purposes.
There are three ways the RDF daemon can be started. If the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Note: Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the information stored in it to establish contact with your Symmetrix arrays. Alternatively, the daemon can be started manually via the stordaemon command line utility as follows: - stordaemon start storrdfd [-wait Seconds] Note: The stordaemon command requires a path of /usr/storapi/storbin. By default, the stordaemon command waits 30 seconds to verify that the daemon is running. To override this, use the -wait option. Additionally, the daemon can be set to start automatically every time the local host is booted using the following command line: - stordaemon install storrdfd autostart. Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load. If the daemon is stopped for some reason, it can optionally be restarted automatically by an internal Solutions Enabler watchdog mechanism. A combination of the watchdog mechanism and the auto-start option described above can be used to ensure that the daemon is always running. To stop the RDF daemon, use the following command: - stordaemon shutdown storrdfd|all [-wait Seconds] Applying the all option stops all of the daemons currently running, such as storapid, storgnsd, and storrdfd.
The RDF process daemon maintains consistency for enabled composite groups across multiple Symmetrix arrays for SRDF/A with MSC.(Multi Session Control) For the MSC option (rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host performing management operations must run an instance of the RDF daemon (storrdfd). Each host must also be running an instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel application syscalls. Additional data about the current state of a composite group is communicated to the RDF daemon via files written to the Symmetrix file system. MSC requires that the RDF daemon exist and every attempt is made to start or restart the daemon to perform cycle switching for SRDF/A. Failure to switch SRDF/A cycles may cause all SRDF/A sessions to be dropped due to a full cache slot. If SRDF/A sessions have been dropped, the SYMAPI and RDF daemon logic determines whether to commit or discard the data accumulated in cache memory. For redundant consistency protection of RDF composite groups, multiple instances of the RDF daemon can be running at the same time on separate hosts. Each host must have a common view of the composite group being monitored. All redundant daemons run simultaneously, monitoring and switching independently of each other. If one of the redundant daemons fails, the other existing daemon(s) completes the task.
Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the stored information to establish contact with your Symmetrix arrays. There are three ways the RDF daemon can be started. First, if the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Second, the daemon can be started manually via the stordaemon command line utility as follows: stordaemon start storrdfd [-wait Seconds] By default, the stordaemon command waits 30 seconds to verify the daemon is running. To override this, use the -wait option. Third, the daemon can be set to start automatically every time the local host is booted using the following command line: stordaemon install storrdfd -autostart Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load.
We wish to control RA Groups 3 and 4 as one entity. A composite group has been created and enabled using the following commands: symdg dg2cg vsrdfadg3 vsmscdg34 rdf_consistency symdg dg2cg vsrdfadg4 vsmscdg34 rdf_consistency rename symcg cg vsmscdg34 enable Note the CSR flags, indicating that Multi-session Consistency has been enabled for these two RA Groups.
Legend: RDFA Flags: C(onsistency) : X = Enabled, . = Disabled, - = N/A
(RDFA) S(tatus) : A = Active, I = Inactive, - = N/A R(DFA Mode) : S = Single-session mode, M = MSC mode, - = N/A
Similar to the previous example (independent), we have a total loss of links for RA Group 4. The devices in this group go to a Partitioned state as before. However, note that the devices in RA Group 3 have been placed in Suspended state. Loss of links for one RA Group trips the Multi-session Consistency Group. This prevents the propagation of data for the other RA Groups in the MSC-CG, thus preserving a consistent image of data on ALL R2s at the time of link loss. As stated earlier, the storrdfd daemon is responsible for coordinating cycle switches between the RA groups, to trip the MSC-CG and perform any recovery/cache cleanup that might be necessary when the links are resumed. Recovering from this state can be accomplished as usual: symrdf cg vsmscdg34 establish Once the invalid tracks are marked, merged, and synchronized, MSC protection is automatically reinstated. i.e User does not have to issue symcg cg vsmscdg34 enable again.
The first scenario is easy to understand. In this instance, the Receive cycles contain the most recent and consistent data. The second situation arises if there is a failure when some Receive cycles are complete while the others are in transit. In this case, clearly it is only the Apply cycles of all sessions that contain the consistent data. Therefore, ALL Receive cycles are discarded. To understand the third scenario, keep in mind the following: For a cycle switch to be initiated, ALL Transmits must be empty and all Applys must be empty. This means that the failure has occurred DURING the cycle switch process in this case. Receive can only be promoted to Apply on a cycle switch. In MSC, cycle switch is sent to all sessions at once. So each of them is in the process of executing a cycle switch. For example, failure occurred prior to promoting some Receives to Apply. Therefore, the Receives not yet promoted should be committed along with those that have already been promoted.
Each Symmetrix has an array-wide Max # of System Write Pending Slots limit (generally calculated as 80% of available cache slots). The purpose of this limit is to ensure that cache is not filled with Write Pending (WP) tracks, potentially preventing fast writes from hosts, because there is no place to put the I/O in cache. SRDF/A creates WP tracks as part of each cycle.
Examples are displayed in the next several slides.
Note that the Cache Slots available for all SRDF/A sessions is 94% of the System Write Pending Limit (3.16/3.6). In this example, RA Group 5 is utilizing 55% of the available cache. All SRDF/A sessions have the default Priority value of 33.
This output was captured from the R1 side. As displayed, the Minimum cycle time for each of the SRDF/A session is at the default of 30 seconds. The Active Cycle is the Capture and the Inactive is the Transmit, as this output is from the R1 (source) perspective.
For this example, the maximum available cache slots for SRDF/A has been reduced to 50% (1.68/3.36). Now, cache utilization reaches a 100% of available cache, and RDFG 5 is dropped (Inactive).
As seen earlier, the Capture and Transmit data on the R1 side are marked as R2 invalids. The Receive data on the R2 side is marked as R1 invalid. The pair state is suspended.
The dropping of the SRDF/A session can also be displayed via the symevent command.
In this instance, the Cache available for SRDF/A has been set to 80% (2.69/3.36). When 100% (103.2) of this is used, we see that the session with the least priority is dropped first RA Group 4 with a priority of 33.
There are three factors that affects RPO of an SRDF/A implementation: SRDF bandwidth, Symmetrix Cache, and Workload. During a SRDF/A cycle, new changes are captured in the local Symmetrix cache before being sent via the SRDF link to the remote Symmetrix. The Symmetrix should have enough cache to accommodate these changes occurring before the cycle switch time has elapsed. At the same time, there should also be a sufficient bandwidth for SRDF link to push these changes to the remote site. If there is not enough cache or bandwidth, SRDF/A may not be able to maintain the RPO at twice the cycle time. The process to determine these three factors should involve EMC personnel and is essential to a successful SRDF/A implementation.
Industry Estimates of Bandwidth Pricing.
In this example, there is 1 OC-3 Link (15.5 MB/s) with 2:1 Compression between the local and remote site. The workload produces 6 peak writes above the bandwidth.
The single OC-3 bandwidth is not sufficient to push the data surge during peak writes. As a result, the data is accumulating the cache on the local Symmetrix. This drives the cache requirement up to around 80GB.
If the additional cache was added, SRDF/A RPO would be almost 800 seconds with a single OC-3 link. Even though the SRDF/A cycle may have been set at 30 seconds, SRDF/A would not be able to maintain it.
This example explores the option if the second OC-3 link was added to the configuration. The bandwidth would be 31 MB/sec. The workload would not produce any peak writes that are above the bandwidth.
This shows that the implementation does not need additional cache to accommodate peak writes. We have eliminated the 80 GB peak cache with an extra OC-3 link.
Therefore, SRDF/A would be able to maintain the 30-second cycle resulting in the RPO of 60 seconds.
We have examined the implementation of SRDF/A with one and two OC-3 links. Now, lets look at an alternative in terms of link bandwidth with 5 T3 links.
With five T-3 links, SRDF/A would need 30 GB of cache to accommodate the sample workload.
This implementation yields the RPO of 300 seconds. The balance of bandwidth and cache to achieve a desirable RPO is specific to a workload pattern as shown by the three examples. A methodical and thorough planning is crucial to a successful SRDF/A implementation.
In an SRDF configuration, a single source (R1) device can concurrently be remotely mirrored to two target (R2) devices. This allows you to have two identical remote copies available at any point in time. It is valuable for duplicate restarts or disaster recovery, or for increased flexibility in data mobility and migrating applications. Concurrent RDF technology can use two different RA adapters (RAs, RAFs, or RFs) in the interface link to achieve the connection between the R1 device and its two concurrent R2 mirrors. Each of the two concurrent mirrors must belong to a different RDF (RA) group Enginuity 5671 supports Concurrent SRDF with SRDF/A. Only one of the SRDF mirrors is allowed to be in Asynchronous mode, regardless if SRDF/A is active or not.
The list is not conclusive. Please refer to EMC Solutions Enabler Symmetrix SRDF Family CLI Version 6.0 PRODUCT GUIDE P/N 300-000-877 for more information.
SRDF/AR allows users to automate the sequence of SRDF and TimeFinder mirror operations. The automated sequence, cycle, is performed on a user-defined interval called cycle time. The replication cycles automatically loop indefinitely or to the number of cycles specified by the users. Users perform all SRDF/AR operations, setup, start, stop, restart, and query, through the symreplicate command. Even though the SRDF link can be set to all SRDF operational mode, except Asynchronous, it is usually set to operate in Adaptive Copy mode due to the long distance between local and remote sites. This allows the users to save on network bandwidth thus minimizing the network costs without compromising the integrity of the data.
SRDF/AR (Automated Replication) - 1
SRDF/Automated Replication (SRDF/AR) provides the ability to automate data copies across SRDF links, providing a restartable image of the data at the remote site in the event of a disaster at the production site. It combines both SRDF and TimeFinder to complete its operations. This slide lists the requirements for both SRDF and TimeFinder in an SRDF/AR environment. Please refer to the EMC Support Matrix for the latest information.
The copy path for a single-hop configuration is from the local R1/BCV pair (1) to the SRDF pair (2) to the remote BCV pair (3). The remotely associated BCV holds the DBMS restartable copy. The amount of data loss is a function of the replication cycle time (period of time between the start of one copy cycle and the start of another copy cycle). Copy cycle time is affected by distance, bandwidth, I/O update rate, and locality of reference for the updates. Update rate and locality of reference tend to equate to changed tracks. The maximum data loss would be one copy cycle, thus makes the RPO ~ One Cycle Time. Single Hop benefits include: The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target site, reducing required network bandwidth Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations
There are 6 steps to prepare the states of all mirrors involved in SRDF/AR Single Hop setup prior to running symreplicate command. To begin a replication session in the single-hop configuration, the mirrors must be in the following states: Local BCV pair must be synchronized SRDF pair must be suspended Remote BCV pair must be split Note 1: Users must wait and check for the full synchronization after issuing the command before proceeding to the next step.
The copy path for a multi-hop configuration is from the local SRDF pair (1) to the remote BCV pair (2) to the remote SRDF pair (3) to the Target BCV (4). If your configuration does not include Target BCVs, the path stops at (3). Automated replication with the BCVs at Target is applicable if you want a zero data loss solution but cannot risk the loss of both the Local site and Bunker site at the same time. With this configuration, there are two possible disaster restart possibilities: If only the Local site is lost, the result is zero data loss at the Target restart site. If both the Local and Bunker site are lost, the result is a DBMS restartable copy at the Target restart site with controlled data loss. The amount of data loss is a function of the replicate copy cycle time between the Bunker site and the Target restart site. Multi-Hop benefits include: The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target (Multi-Hop) site, reducing required network bandwidth Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations The ability to use the SRDF Multi-Hop site to provide disaster recovery testing, point-in-time backups, decision support operations, third-party software testing, and application upgrade testing or the testing of new applications
There are 5 steps to prepare the states of all mirrors involved in SRDF/AR Multi Hop setup prior to running symreplicate command. To begin a replicate session in the multi-hop configuration, the mirrors must be in the following states: Local SRDF pair must be synchronized BCV pair on Bunker must be synchronized Remote SRDF link between Bunker and Target must be suspended BCV pair on Target must be split Note 1: Users must wait and check for the full synchronization after issuing the command before proceeding to the next step. Note 2: This automatically suspends the SRDF link between the R1/BCV on Bunker and R2 on the Target sites. Note 3: The BCV must associated with the device group with the rrdf option.
Symreplicate invokes a replicate session that generates automated, recurrent background copies of the standard data across SRDF links and cascading BCVs. You can start, stop, and restart the replicate session.
The option file is a text file that contains parameters such as SRDF/AR hop type and cycle time. This file is required and is used in conjunction with the symreplicate command to start a replication session. This slide lists the variables available to be placed in the SRDF/AR options file. There are a minimum of two options that must exist for the symreplicate to be executed. The two required options are: SYMCLI_REPLICATE_HOP_TYPE and one of SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY. As of Solutions Enable 6.2 additional SYMCLI REPLICATE variables have been added.. Ref the SE product guide for a complete detailed explanation.
SYMCLI_REPLICATE_CONS_SPLIT_RETRY, SYMCLI_REPLICATE_R1_BCV_EST_TYPE, SYMCLI_REPLICATE_R1_BCV_DELAY, SYMCLI_REPLICATE_FINAL_BCV_EST_TYPE, SYMCLI_REPLICATE_FINAL_BCV_DELAY, and SYMCLI_REPLICATE_PERSISTENT_LOCKS
The two options that must exist for the symreplicate to be executed are: SYMCLI_REPLICATE_HOP_TYPE and either SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY. SYMCLI_REPLICATE_HOP_TYPE is self explanatory and is directly dependent on the configuration. SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY requires a time interval for its value.
EMC recommends using default settings initially. Time limit parameter changes should only be made based on recommendations from EMC.
EMC recommends using default settings initially. Sleet time parameter changes should only be made based on recommendations from EMC.
The three parameters SYMCLI_REPLICATE_CYCLE, SYMCLI_REPLICATE_CYCLE_OVERFLOW, and SYMCLI_REPLICATE_CYCLE_DELAY significantly contribute to how the next replication cycle starts after the current cycle. The next two slides show different combinations of cycle time, delay time, and overflow behavior to achieve various results. Reference points on the cycle time lines are marked ACT (actual completion time), NCS (next cycle start), and MDT (minimum delay time). Short cycle and delay times (in minutes) were chosen for illustration purposes only. Copy cycles #1 and #2 have the same cycle time (2) and no delay time. When copy cycle #1 runs longer than two minutes, the overflow setting of Next results in a new copy cycle beginning at the four-minute mark.
Copy cycles #3 and #4 have the same cycle time (2) and the same delay time (3). When copy cycle #3 runs longer than two minutes, the system waits three minutes and then resumes copying at the next scheduled copy cycle (the six-minute mark). When copy cycle #4 runs longer than two minutes, the system waits three minutes and then resumes copying as soon as the wait period completes.
Copy cycle #5 finishes before its scheduled cycle time of three minutes, waits two minutes, and begins copying again at the next scheduled copy cycle (the six-minute mark). Copy cycle #6 performs continuous copy cycles when its cycle time is set to zero. At the end of a cycle, the system waits three minutes and then begins another copy cycle, regardless of the overflow setting.
Protected BCV establish: NONE: normal TimeFinder operations LOCAL: protected establish of local BCVs in a single-hop replication REMOTE: protected establish of remote BCVs in a single-hop replication FIRST_HOP: protected establish of Hop-1 BCV pairs in a multi-hop replication SECOND_HOP: protected establish of Hop-2 BCV pairs in a multi-hop replication BOTH: protected establish of both Hop-1 and Hop-2 BCV pairs in a multi-hop replication, and both local and remote BCV pairs in single-hop replication Clone Emulation: Beginning with Solutions Enabler 6.0, if ALL BCVs are RAID-5 BCVs, TimeFinder/Mirror commands automatically map to corresponding TimeFinder/Clone commands. However, if there is a mixed environment, i.e. some RAID-5 BCVs and others are RAID1/unprotected, then this option must be explicitly set to TRUE. Default is FALSE.
It is recommended to be generous with time parameters for cycles, and adjust once more information is collected for various cycles. The times configured in the options file should be configured for worst case scenarios. Beginning with Solutions Enabler 6.1, you can display statistical information for cycle time and invalid tracks by using the symreplicate stats command. The command can be issued by device group (-g) or composite group (-cg) for a specified Symmetrix (-sid) and information can optionally be written to a specified log file (-log). The -all option is the default and will display both the cycle time and invalid tracks statistics. The following example will display both cycle time and invalid track for device group abcdg on Symmetrix 123,
symreplicate -g abcdg -sid 23 -all stats log abcdg.log
Before using the symreplicate command, perform the initial synchronization tasks. All devices must be in a specific paired state before executing the SRDF/AR cycle.
Beginning in SE 5.4, SRDF/AR supports automatic setup using: symreplicate setup symreplicate start -setup The setup command: Sets-up required pair states Options file indicates single or multi hop type setup Same options file used for setup and subsequent start command Executes one cycle, which may take some time, and then exits Setup command executes just one cycle, regardless of the number of cycles specified in the options file The start command with the -setup option: Sets-up the required pair states If successful, begins the symreplicate session Options file defines the hop type and the copy cycle parameters that you have chosen for the session
The symreplicate command automatically executes the above steps in each cycle during a replication session. These steps are similar to those used to initializing the mirror states. It is important to understand these inner working steps because users can choose to terminate a session after the current step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must be to TRUE for symreplicate to log the information after each step to the SYMAPI log file. The location of the SYMAPI varies from one platform to another. The format of the file name, symapi-YYYYMMDD.log, is identical across platforms.
A simple Options file specifying single-hop replication of 10 minute cycle times, to be repeated 6 times.
The setup command runs one cycle and exits. The foreground option provides details of each of the steps being executed.
Query shows the setup has completed. We are now ready to start the single-hop replication cycles.
At the start of a cycle, the R1/BCVs are established and synchronized with the Standard devices.
The BCVs on the Target site are split from their R2s.
The link between the R1/BCVs and their R2s on the Target site is Suspended.
Determination of the states of the devices and deducing the cycle step using the states can be performed from a host on the target side, using appropriate device files. These operations can be performed from a different host on the source side, if the failure only affected the primary host, and the array and site are still available and accessible.
All BCVs are split: If the host on the primary site has failed, but the array and links are available, the R1/BCVs can either be established with the Standard devices or can be split from the Standard devices and transmitting data to the R2. In the former case, both the R2 and BCV contain identical and consistent data, and restart can be done from either set of devices. In the latter case, the most recent data received by the R2 and BCV has consistent data up to the last cycle. In this case, one can wait for transmission to complete and then restart from the R2. Prior to restart it would be advisable to reestablish the BCVs, synchronize, and split to maintain a gold copy. In the third case, R1/BCVs still have tracks owed to the R2. So the consistent data resides on the BCV. One can restart using these or preferably restore the BCV data to the R2s and restart from the R2. All BCVs are established: It is clear that both R2 and BCVs have the most recent and consistent data at the end of the BCV synchronization. One can then split the BCVs (to maintain a gold copy) and restart from the R2s. Some BCVs are split while others are still established: The R2s have the most recent and consistent data. Re-establish all the BCVs with the R2s, synchronize, and split (to maintain a gold copy), prior to restarting from the R2s.
The symreplicate command automatically executes the above steps in each cycle during a replication session. These steps are similar to those used to initialize the mirror states. It is important to understand these inner working steps because users can choose to terminate a session after the current step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must be set to TRUE for symreplicate to log the information after each step to the SYMAPI log file. The location of the SYMAPI varies from one platform to another. The format of the file name, symapiYYYYMMDD.log, is identical across platforms.
Since Enginuity 5669, Symmetrix arrays support clustered SRDF/AR environments for multiple node (host) capability. Clustered SRDF/AR provides the capability to start, stop, and restart replication sessions from any host connected to any local Symmetrix array participating in the replication session. The clustered SRDF/AR environment allows the replication log file to be written directly to the Symmetrix File System (SFS) instead of the local host directory of the node that began the session. If the primary node should fail, then any locally attached host to the Symmetrix array containing the log file would then be able to restart the SRDF/AR session from where it left off. If you begin a session and specify a user log file name (-log), you must specify the -log option for all other commands in the session sequence. To write the log file to the SFS, you must specify the ID of the Symmetrix array (-sid) where the log file is to be stored at the start of the replication session, along with a group name (-g, -cg) and an optional user log filename (-log). For example: symreplicate start -g session1 -log srdfar1.log -sid 201 Note: Not specifying the Symmetrix ID (-sid) at the start of the session, causes the log file to be written to local disk using the default SYMAPI log directory, which is not restartable from another node.
Note that BCVs at the target site are optional in a multi-hop configuration (SYMCLI_REPLICATE_USE_FINAL_BCV=<TRUE|FALSE>). In case BCVs are used at the target site, we must consider their pair states if we wish to preserve copy of data from previous cycle.
R1/BCVs are established with their R2s: In this case we can wait for synchronization between R1/BCVs and the bunker R2s. Then split the R1/BCVs and resume the link between R1/BCVs and target R2s. The optional BCVs at the target site can be left in a split state to preserve consistent data from the previous cycle. R1/BCVs are split from their R2s: If the target BCVs are split, then one can re-establish the R1/BCVs with the bunker R2s, wait for synchronization, split, and resume the link between R1/BCVs and the target R2s. The BCVs at the target will again have consistent data up to the last cycle. If BCVs at the target site are in an established state, one can wait for or verify synchronization, split the BCVs (to preserve data from the previous cycle), then perform the re-establish, synchronization, split, and resume of the R1/BCVs. Some R1/BCVs are split while others are established with their respective R2s: In this instance, one can re-establish ALL the R1/BCVs with the bunker R2s, synchronize, split, and resume links to get the most recent and consistent data over to the target R2s.
RW enabling of R2s can be performed via the symrdf failover command with the appropriate device file definitions.
These are the key points covered in this training. Please take a moment to review them. This concludes the training.

SRDF v33

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

SRDF v33

Enviado por

Direitos autorais:

Formatos disponíveis

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Training Schedule for SRDF Solutions, day one and two.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Training Schedule for SRDF Solutions, day three.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.