SRDF Solutions

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.
Symmetrix Business Continuity: SRDF Solutions
2007 EMC Corporation. All rights reserved.
Welcome to Symmetrix Business Continuity: SRDF Solutions. Copyright 2007 EMC Corporation. All rights reserved. These materials may not be copied without EMC's written consent. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC and SRDF are registered trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners.
SRDF Introduction
-1
Revision History
Rev Number
1.0a 1.2a 3.0a 3.1 3.2 3.2a 3.3
Course Date
March 2005 July 2005 September 2005 August 2006 November 2006 December 2006 June 2007
Revisions
Complete Complete Complete Complete
Updated SE 6.3 Updated for SE 6.4/72
SRDF Introduction - 2
SRDF Introduction
-2
Program Administration
Attendance roster Restrooms Telephones / Etiquette Attendance Rules Fire / Evacuation Procedures Cafeteria Labs Local Sites of Interest Class Evaluations
2007 EMC Corporation. All rights reserved. SRDF Introduction - 3
SRDF Introduction
-3
Course Objectives
Upon completion of this course, you will be able to: Describe the relevancy of SRDF solutions with different (RPO) Recovery Point Objective needs Identify SRDF concepts, terminology and functionality Discus EMCs Symmetrix Management Console (SMC) Describe SRDF host considerations and configurations within Sun Solaris, HP-UX, IBM AIX, and Windows LVM environments Describe SRDF/A theory of operation, and execute SRDF/A operations Identify the architectural components of SRDF/A Describe and execute SRDF/AR operations Describe EMC Consistency Technology
The objectives for this program are shown here. Please take a moment and review them.
SRDF Introduction
-4
SRDF Introduction
Upon completion of this module, you will be able to: Discuss the Concept of Business Continuity Identify SRDF (Symmetrix Remote Data Facility) solutions Identify SRDF solutions to satisfy different RPO (Recovery Point Objective) needs Describe EMCs Symmetrix Management Console (SMC)
The objectives for this module are shown here. Please take a moment and review them.
SRDF Introduction
-5
Documentation
Student Resource Guide
Module 1: BCR - Business Continuance Remote Introduction Module 2: SRDF/S (Synchronous) Module 3: SRDF Operations Module 4: SRDF/A (Asynchronous) Module 5: SRDF/AR (Automated Replication)
Labs Guide
SRDF - BCR Labs 1-7
Appendix
BCR (Business Continuance Remote) Command Lookup Guide
All Material is for Training Purposes Only

The following modules will be presented over the next three days. All training materials presented in this program are for training purposes only.
SRDF Introduction
-6
Training Schedule: Day 1 & 2
Training Schedule for SRDF Solutions, day one and day two.
SRDF Introduction
-7
Training Schedule: Day 3
Open Lab - Activity
Training Schedule for SRDF Solutions, day three.
SRDF Introduction
-8
Software Environment
The following Software Environment has been established for each student:
Sun Solaris 5.8 OS
OS Recommended Patches J2SE Cluster Patch Veritas Volume Manager (VxVM Ver 4.0)
HPux 11.00 with 2004 patch bundle

HP-LVM
AIX 5.2 / ML03 (Maintenance Level)

AIX-LVM
W2K EMC Solutions Enabler 6.4 Enginuity Code 5672 Custom Shell Scripts
The above software environment supports the BCL (Business Continuance Remote) program. Appendix I is a Command Quick Reference Guide, created to support this training program. All students attending this program should have a good understanding of the following: Unix file systems. The Unix Vi editor A general understanding of a Unix Volume Manager EMC Solutions Enabler Symm 5 / Symm 6 overview
SRDF Introduction
-9
What is Business Continuity?

Business Continuity is the preparation for, response to, and recovery from an application outage that adversely affects business operations Business Continuity Solutions addresses systems unavailability, degraded application performance, or unacceptable recovery strategies
Business Continuity remains at the top of every executives priority list. Yet executives find themselves in a financial tug-of-war between business continuity solutions and other projects competing for the limited resources. Fundamental to business continuity is the need to understand an organizations practices relative to the protection, availability, and usability of data.
SRDF Introduction
- 10
Why Business Continuity?

Loss Revenue Number of employees impacted X hours out X burdened hourly rate Loss Revenue Direct loss Compensatory payments Lost future revenue Billing losses Investment losses Financial Performance Revenue recognition Cash flow Lost discounts (A/P) Payment guarantees Credit rating Stock price
Damaged Reputation Customers Suppliers Financial markets Banks Business partners

Know the downtime costs per hour, day, two days...
Other Expenses Temporary employees, equipment rental, overtime costs, extra shipping costs, travel expenses...
Failures happen - hardware, software, natural disasters etc. Downtime has a significant impact, the cost is more than just financial loss. What can we do to avoid downtime or minimize the length of time we are down? EMC offers Business Continuity Solutions that help address common failures or outages. Host to Storage failures and Performance bottleneck of a Host Bus Adapter: PowerPath Local Storage Protection with local mirroring: TimeFinder Family of Products Remote Storage Protection and Site Protection: SRDF Family of Products
SRDF Introduction
- 11
SRDF Recovery Point Objectives (RPO)

Business Needs Drive the Technology Choice
Wk s Day s Hrs Min s Sec s Sec s Min s Hrs Day s Wk s
Recovery Point
Recovery Time
Tape Backup
Periodic Replication Asynchronous Replication Synchronous Replication
12
Recovery Point Objective (RPO): Maximum amount of data loss an application can tolerate as measured in time. In other words, the amount of data loss that can be tolerated (cost of transaction versus risk). Individual customer business needs drive the technology chosen to meet specific recovery point objectives. This is also known at RPO (Recovery Point Objective). Data Characteristics that Influence Data Storage Decisions: Several factors affect the value of data including: Legislation, which can mandate how long the data must be accessible, and by whom; Business processes that are tied to points in time (book closing, quarterly reports, tax deadlines, billing cycles, etc.) Business processes that are tied to customer satisfaction service levels associated with the data as its purpose changes. For example, data can start out as transactional, then migrate to billing, then reporting and customer service, then to scoring data for a marketing system, and finally to archival. The usefulness of data to the business will vary over time, and hence the necessity to have immediate access to the data changes. The decisions about where data is placed in the storage infrastructure and the methods used to protect that data are fundamentally driven by three factors: The time required to access the data relative to the cost of the access (that is, usefulness to the business versus cost). Recovery Time Objective (RTO): This refers to the maximum time a company budgets to bring an application back online in the event of a disaster. In other words, the time it takes to recover the data once a disaster or other recovery event is declared (risk versus cost). Each change in data placement and protection criteria represents a stage in the life cycle of the data and is directly related to the usefulness or importance of the data to keep the business functioning.
SRDF Introduction
- 12
What are SRDF Solution Sets?

Symmetrix Logical Volumes are mirrored between geographically dispersed locations Maintain real-time physically separate mirrors of data with SRDF Maintains near real-time physically separate mirrors of selected volumes with SRDF/A (Asynchronous) or SRDF/AR (Automated Replication) Continue running through events such as individual drive/link failures Mirror copy can be split and used for disaster recovery or business continuance applications
Report Generation Decision Support
TimeFinder
BCV
SRDF/S
ERP
ERP
SRDF/A
E-mail
SRDF/AR
E-mail
Tape Backup
EMC Snap
The SRDF family of software is the most powerful suite of remote storage replication solutions available for disaster recovery and business continuity. It leverages high-end Symmetrix storage architecture to offer unmatched deployment flexibility and massive scalability so you can meet mixed service level requirements with minimal operational impact. The SRDF family is the most widely deployed set of high-end remote replication solutions, and is installed in tens of thousands of demanding environments worldwide. The SRDF family is the only product that provides cross volume and storage system consistency, tight integration with industry-leading applications, and automated management for simplified usage.
SRDF Introduction
- 13
SRDF Solutions
SRDF/S

Source Limited Distance
Target
No data exposure Some performance impact Limited distance

Source
SRDF/A

Target ExtendedDistance
Seconds of data exposure No performance impact Extended distance

Source Unlimited Distance Target
SRDF/AR

Hours of data exposure No performance impact Unlimited distance Requires BCVs

Prod
EMC has several remote replication offerings for various service level requirements. For zero data exposure, EMC offers the industry leader for synchronous mirroring: SRDF. However, as with any synchronous solution, there are characteristics that must be understood. Distance is limited by application time-outs and bandwidth must be sized for peak workload at all times. SRDF/Asynchronous is a solution for service level requirements that need Recovery Point Objectives (RPO) in the seconds-to-minutes area. SRDF/AR delivers solutions that combine SRDF with TimeFinder to create single-hop and multi-hop environments for specialized needs. These solutions offer different RPOs and have different requirements for bandwidth, supported distances, etc. No matter what your requirements are, EMC can help deliver the right Remote Replication Solution.
SRDF Introduction
- 14
SRDF/Star (5671)
Synchronous Target Site
Target
Bunker Site (Local)
Asynchronous Site
Workload Site
X
Primary Site
Source Target
Long Distance Site
SRDF/STAR (Symmetrix Triangular Automated Replication)

Start with the scenario of a Primary site with multi-site protection
An SRDF/S link to a nearby bunker site (zero time lag RPO) An SRDF/A link to a long distance site (Seconds - Minutes time lag RPO)
"Disaster" strikes and the primary site is lost

Bunker site and long distance site are incrementally synchronized to most recent data Third SRDF/A link between the 2 remaining sites maintains continuous protection
SRDF/Star is a three-site1 disaster recovery solution that uses concurrent RDF technology to replicate data from a primary production site (referred to as the workload site) to a nearby remote site and a distant remote site. Data is transferred in SRDF/Synchronous (SRDF/S) mode to the nearby remote site (referred to as the synchronous target site) and in SRDF/Asynchronous (SRDF/A) mode to the distant remote site (referred to as the asynchronous target site). SRDF/Star provides consistent data protection and incremental data recovery between target sites in the event of a workload site failure
SRDF Introduction
- 15
SRDF Management Tools

Solutions Enabler EMC Control Center
Easy, point-and-click access Excellent for ad hoc TimeFinder operations
SMC: Symmetrix Management Console

Web-based application for managing the Symmetrix Works the same as the CLI/API the same rules apply Makes it easier to perform complex CLI tasks
EMC Replication Manager

Discovers replication environments Automates replication process Integrates replication technologies at the application level
In addition to SYMCLI (Solutions Enabler), SRDF can be managed by a variety of management tools such as EMC Control Center and Symmetrix Management Console. Through its graphical user interface, EMC ControlCenter and SMC software will organize related devices into device groups. SRDF operations may be performed on all devices in this device group by using a single command. The group information is maintained in the SYMAPI database. Another management tool is EMC Replication Manager. It is an application that automates, simplifies, and manages disk-based replications by using SRDF operations.
SRDF Introduction
- 16
Symmetrix Management Console (SMC)

Independent, light-weight, web-based application
Simple and easy to use browser interface Hosted on small Windows/Linux server Enables remote access and management from nearly any client
Enables access, configuration, and basic operation of Symmetrix arrays

Supports all configuration capabilities of Solutions Enabler/CLI
SRM Planning and Provisioning SRM Monitoring and Reporting
Storage Management
Services
Supports multiple generations of Symmetrix

Enginuity version 5x68 and newer
Device Management
Provides day-one support of new Symmetrix features when released Adding full-feature ControlCenter does not require management data to be migrated
EMC offers a Graphical User Interface (GUI) product called Symmetrix Management Console, or SMC. SMC enables the user to deploy web-based device management within a Symmetrix environment. The user can now choose the right management product or products to meet a set of specific requirements. Almost anything you can do using the Solutions Enabler command line (CLI) can now be done using the SMC GUI. SMC manages all Symmetrix systems running Enginuity version 5568 and up, and supports new hardware and software features and functionality at the time of product release. Addressing customer demands for enhanced platform interoperability, SMC is an independent application which runs using its own lightweight Windows/Linux server. The client runs in a browser window, supporting nearly any client with remote access to the server. The SMC GUI features closely match Solutions Enabler CLI features to include all basic monitoring, configuration, and control of Symmetrix arrays. SMC has no Symmetrix-related database other than the Solutions Enabler database, so all data automatically transfers to the full-feature ControlCenter when discovered by the Symmetrix agent. Due to the combination of being light-weight yet feature rich, early response to SMC shows it to be a product leader in this space.
SRDF Introduction
- 17
SMC Functionality
Access Management
Manage users, permissions/roles Symmetrix Access Controls
Configuration Management
Create devices, map and mask devices, create device groups, set Symmetrix attributes
Replication Management
TF/Clone, TF/Mirror, TF/Snap, SRDF/S, SRDF/A, SRDF/DM, Open Replicator, Optimizer
Alerts and Monitoring

Monitor Device status, device attributes, operations status Monitor array alerts
Primary SMC features include providing the facility to manage SMC access and permission levels, discover, and configure, Symmetrix arrays. Also SMC enables monitoring of replication operations within Symmetrix arrays. Configuration activities include Create devices, map and mask devices, create device groups, set Symmetrix attributes; create Symmetrix Logical Volumes from un-configured storage, create meta volume devices, both concatenated and striped. Additionally, the ability to modify existing configured storage after un-mapping it from the hosts; map and un-map one or more logical volumes to a port or ports; delete devices to convert configured storage into un-configured space; manage virtual and save devices, and manage dynamic spares. SMC enables you to monitor device status, device attributes, and operations status, perform and monitor TimeFinder/Mirror, TimeFinder/Clone, and TimeFinder/Snap operations, SRDF operations, Open Replicator sessions, Optimizer and Quality of Service features, as well as monitor and report alerts.
SRDF Introduction
- 18
Components of the SMC Interface

Menu Bar Menu Bar View Bar View Bar
(Properties) View (Details) (Properties) View (Details)
Navigation Tree
The major components of the SMC Interface are highlighted above. To monitor and control operations, you can select a single object or a single folder of multiple objects in the SMC Navigation Tree above. The View Bar is used to switch between five different views: 1. Properties, 2. Configuration Session, 3. Alerts, 4. Command History and 5. Replication Monitor. The current View selected will display details in the view area. The View button and corresponding view display are color coded to match. Note the Alert counter in the top right, which also selects the Alert View. Tree Selection, View Selection, and Object Selection within the View area determine the current display. The View may be split horizontally into two or three areas depending on the detail associated with the selection.
SRDF Introduction
- 19
Replication Operations Supported by SMC

SRDF
Synchronous Asynchronous Adaptive Copy SRDF Configuration
TimeFinder
Mirror Clone Snap SAVE Device Pool management
Open Replicator Quality of Service

Almost all replication technologies available on the Symmetrix Arrays can be monitored and managed via the Symmetrix Management Console (SMC) application. Note: At the present time SRDF/Star cannot be managed via SMC.
SRDF Introduction
- 20
Creating Device Groups (Replication / SRDF Control)

Device Group Management Create Device Group
SRDF Action SRDF Action - -Control Control - -Settings Settings - -Configuration Configuration
SMC Device Group creation is a multi-step process, similar to creating SYMCLI device groups using symdg, symld, and symbcv. The wizard is launched by right clicking the Device Group Folder and choosing the Device Group Management, and then the Create Device Group option. Once a Device Group has been created selecting Replication / SRDF Controls, Settings, or Config enables SMC / RDF control functionality on the selected Device Group. The above SMC01 Device Group has two R1 devices (02EF, and 02F0) paired with two RDF / R2 devices (02EF, and 02F0). The above device group is currently in synchronized, with no Alerts to report.
SRDF Introduction
- 21
SRDF Settings: Setting SRDF Mode

Replication SRDF Settings
Drop Down to select Mode Setting
SRDF Mode operations are shown in this slide. Right Click an RDF device group and choose Replication SRDF Settings. The dialog box shows the current mode and pair states. The mode can be changed by using the Set Mode pull down. When the Set Mode is pulled down, the user seea Synchronous, Asynchronous, Semi Synchronous, to mention just a few. In this example, the mode can be changed from Synchronous to Asynchronous.
SRDF Introduction
- 22
Module Summary
Key points covered in this module: Business Continuity concepts SRDF (Symmetrix Remote Data Facility) solutions SRDF solutions used to satisfy different RPO (Recovery Point Objective) needs EMCs Symmetrix Management Console (SMC)
These are the key points covered in this module. Please take a moment to review them
SRDF Introduction
- 23
SRDF Overview
Upon completion of this module, you will be able to: Describe EMC SRDF functionality and its uses Describe SRDF Link configurations Describe the concept of SRDF Group Describe SRDF swap List the characteristics of:
Concurrent SRDF Dynamic SRDF
SRDF/S (Synchronous) - 1
The objectives for this module are shown here. Please take a moment to read them.
Symmetrix Remote Data Facility (SRDF)

Facility for maintaining realtime or near-real-time physically separate mirrors of selected volumes Uses no host CPU resources
Mirroring done at the storage level
Open Systems / Mainframe
Operating system independent

Open Systems Mainframe
R1
R2
SRDF
Symmetrix Remote Data Facility (SRDF) is a Symmetrix system based business continuance, disaster recovery, restart, and data mobility solution. In the simplest terms, SRDF is a configuration of multiple Symmetrix units that maintains real time copies of logical volume data in more than one location. The Symmetrix units can be in the same room, in different buildings within the same campus, or hundreds and even thousands of miles apart.
SRDF Source and Target Volumes

Symmetrix Logical Volume types:
SRDF Source or R1 Volumes: Primary Volume with R/W access to local host SRDF Target or R2 Volumes: Backup Volume used for DS or DR Applications
The attached host is unaware of SRDF protection
RW
M1
Source
M2 M3 M4 M1
Target
M2 M3 M4
WD
This slide displays the representation of the mirror positions when both the Source and the Target SRDF Logical Volumes have local protection (RAID-1). In this diagram, the Target-R2 volume is also represented with 4 mirror positions and has local protection implemented. Three of the mirror positions are used. The first two mirror positions represent local mirrors and the third mirror is occupied by SRDF. If a BCV is established with the R2 volume, then it will occupy the next available mirror position. Under normal circumstances, the R1 volume presents a Read-Write (RW) status to the host which access it, and the R2 presents Write-Disabled (WD) to its host.
Remote Link Director (RLD)
Remote Link Director Source Remote Link Director
Remote Link Director Target Remote Link Director
A Remote Link Director is a hardware that provides communication and data paths between local and remote Symmetrix units. The Symmetrix can be configured with the following RLDs: Fibre Channel directors (RF) ESCON directors (RA) Multiprotocol Channel Directors (MPCD) available with these channel connections: FICON iSCSI for host GigE (RE) for SRDF
SRDF Groups
RDF Group 1
R1 R1 R1 R1
RDF Group 1
Remote Link Director Remote Link Director
RDF Group 1,2,3.
Remote Link Director Remote Link Director
R2 R2 R2 R2
RDF Group 1,2,3.
R1 R1 R1
RDF Group 2
R2 R2 R2
RDF Group 2
An SRDF group, also known as RDF group or RA group, logically defines relationships between Symmetrix systems. An SRDF group is a set of SRDF director port connections configured to communicate with a another set of SRDF director ports in another Symmetrix system. Logical volumes (devices) are assigned to SRDF groups. Many SRDF groups can share a physical link between the Remote Link Directors. There are two ways to create an RDF group - static and dynamic. Both share the same features and functionality, the difference between the two types is how they are created. Static RDF groups are created during the Symmetrix configuration, and almost always by EMC personnel. Dynamic RDF groups are created and deleted by users through a set of Symmetrix command line interface (SYMCLI) commands.
Prior to SE 6.3, the Symmetrix DMX supported up to 64 total RDF groups. With SE 6.4 and 5772, 1 to 250 RDF groups are supported.
SRDF Link Configuration

Uni-Directional
Symmetrix A
Source
RA Group
RA Group
Symmetrix B
Target
Bi-Directional
Symmetrix A
Source Target
RA Group
RA Group
Symmetrix B
Source Target
Dual Configuration
Symmetrix A
Source Source RA Group 1 RA Group 1 RA Group 2
Symmetrix B
Target Target
Target
Target
RA Group 2
Source
Source
SRDF offers three types of link configurations between source (local) and target (remote) Symmetrix systems: Uni-Directional, Bidirectional and Dual Configuration. SRDF Unidirectional Link Configuration If all primary (source or R1) volumes reside in one Symmetrix system and all secondary (target or R2) volumes reside in another Symmetrix system, write operations move in one direction, from primary to secondary. Data moves in the same direction over every link in the SRDF group. SRDF Bidirectional Link Configuration If an SRDF group contains both primary and secondary volumes, write operations move data in both directions over the SRDF links for that group. SRDF Dual-Directional Link Configuration With a dual-directional configuration, multiple SRDF groups are used; some groups send data in one direction, while other groups send data in the opposite direction.
SRDF Modes of Operations

Primary and Secondary Modes
Two Primary SRDF Modes
Synchronous Semi-synchronous
Secondary SRDF Mode

Adaptive Copy
Write Pending Disk Mode
Operational Modes are set on Symmetrix Logical Volume level Using GUI or CLI and can be changed dynamically SRDF/A Asynchronous Domino Mode (or attribute)
Listed are the operational modes for SRDF operations: Synchronous mode, Semi-Synchronous mode, Adaptive Copy-Write Pending mode, Adaptive Copy-Disk Copy mode, and Asynchronous mode. These operational modes are selectable based on many requirements such as RPO, bandwidth, and performance. One of the two primary SRDF modes of operations is set at the source (R1) volume during Symmetrix configuration. All source (R1) volumes are configured for either the Synchronous or Semi-Synchronous mode. These two modes are considered to be pre-determined SRDF modes, which may be altered using SymCli. Adaptive copy is the secondary mode that facilitates data sharing and migration. Asynchronous mode continually collects and sends data to the remote Symmetrix. Asynchronous mode must be set for the entire RA group. Users can set SRDF to function in a secondary or Asynchronous mode. SRDF will revert to the pre-determined primary mode if it cannot maintain the criteria to remain in the secondary mode. Domino Mode could be classified as an SRDF attribute. Not necessarily a Mode. This attribute is set or used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if the target volume become unavailable, or if all SRDF links become unavailable.
Synchronous Mode
SRDF links
Source
Target
Write I/O received from host/server at the source I/O is transmitted to the target An acknowledgment is provided by target back to the source I/O is serviced to the host
SRDF Synchronous Mode is used primarily in SRDF campus environments. In this mode of operation, Symmetrix maintains a real-time mirror image of the data of the remotely mirrored volumes. Data on the source (R1) volumes and target (R2) volumes are always fully synchronized at the completion of an I/O sequence. The sequence of operations is: A write is received from the host/server at the source. The write is transmitted to the target. An acknowledgment is provided by the target back to the source. The write is acknowledged to the Host. If step 3 never happens, the source SRDF services the I/O after a pre-determined timeout to keep the production machine running.
Semi-Synchronous Mode
SRDF links
Source
Target
Write I/O received from host/server at the source I/O is serviced to the host I/O is transmitted to target An acknowledgment provided by target back to source
2007 EMC Corporation. All rights reserved. SRDF/S (Synchronous) - 9
SRDF Semi-Synchronous Mode is used primarily in extended distance environments. Semisynchronous mode allows the primary and secondary volumes to be out of synchronization by one write I/O operation. Data must be successfully stored in the Symmetrix system containing the primary volume before an acknowledgement is sent to the local host. Semi-synchronous mode will not allow the next write operation to a primary device until a positive acknowledgement is received from the target Symmetrix system that the first write operation was received in the target Symmetrix global memory. However, any number of read operations can be performed to the primary device while awaiting acknowledgement of the first write operation. Semisynchronous mode writes data to the primary device in the source Symmetrix system, completes the I/O, and then synchronizes The data with the secondary device in the target Symmetrix. The sequence of operations is: An I/O write is received from the host/server at the source. The I/O is serviced to the host/server. The I/O is transmitted to the cache of the target. An acknowledgment is provided by the target back to the source.
Adaptive Copy Mode
SRDF links
Source
Target
Write I/O received from host/server at the source I/O accumulates in/on: - Symmetrix cache Write Pending Mode - R1 volumes Disk Mode I/O is serviced to the host I/O is transmitted to the target An acknowledgment is provided by target back to the source
SRDF Adaptive Copy Mode is used primarily for data migrations and data center moves. This operational mode is not recommended for use when mirroring for disaster recovery/restart purposes unless used with TimeFinder. This mode is very useful for initial synchronization, especially over long distances. (Used within a SRDF/Star configuration). SRDF Adaptive Copy Mode allows the source (R1) volumes and target (R2) volumes to be a out of synchronization by a number of I/Os that users can define, a skew value. There are two types of adaptive copy: Write Pending Mode and Disk Mode. Adaptive Copy data movement is handled at the track level. The target data is only usable after a full synchronization. The sequence of operations is: An I/O write is received from the host/server at the source. I/O is accumulating. I/O is serviced. The I/O is transmitted to the target. An acknowledgment is provided by the target back to the source. In Write Pending Mode, the unit of transfer across the SRDF link is the updated blocks rather than an entire track, resulting in more efficient use of SRDF link bandwidth. Data is read from global memory than from disk, thus improving overall system performance. However, the global memory is temporarily consumed by the data until it is transferred across the link. In Disk Mode, while less global memory is consumed it is typically slower to read data from disk than from global memory, additionally, more bandwidth is used because the unit of transfer is the entire track. Additionally, because it is slower to read data from disk than global memory, device resynchronization time increases. Adaptive copy disk mode should not be used if the primary volumes are not RAID protected.
Asynchronous Mode
SRDF links
Source
Target
Write I/O received from host/server at the source I/O accumulates in Source Symmetrix cache I/O is serviced to the host I/O is continually transmitted to the target I/O accumulates in Target Symmetrix cache
SRDF/A provides a long-distance replication solution with minimal impact on performance. This protection level is intended for customers requiring minimal host application impact, who need to maintain a restartable copy of data at the target site at all time. SRDF/A continually process Write I/Os in batches. The interval between batches is referred to as a cycle. The sequence of operations is: An I/O write is received from the host/server into the cache of the source. I/O is accumulating. I/O is serviced. The I/O is transmitted to the target.
Domino Mode (attribute) with SRDF/Synchronous
SRDF links
Source
Target
Write I/O received from host/server at the source I/O fails to transmit to the target Both Source and Target become unavailable
Domino Mode is used in conjunction with other SRDF modes except SRDF/A. It effectively stops all write operations to both source and target volumes if target volume become unavailable, or if all SRDF links become unavailable. User will need to manually re-enable the source volumes. While such a shutdown temporarily halts production processing, domino modes can prevent data integrity exposure that causes the inconsistent image on the target volume.
SRDF Level of Synchronization

Synchronous Mode
Source = Target
Semi Synchronous Mode

Source Target
At most, Source is 1 I/O ahead of Target, per volume
Adaptive Copy
Source Target
Source may be up to 65535 tracks per volume ahead of Target Skew value set per logical volume
Asynchronous
SRDF/A - Source is minutes ahead of Target SRDF/AR - Source is hours ahead of Target
SRDF offers considerable flexibility for various levels of synchronization. To determine the level of synchronization, one must understand the required Recovery Point Objective. This is the amount of data that can be lost in the event of a site outage. There are other factors like distance, bandwidth, and response time latency that must be considered before determining a synchronization level.
SRDF Serialization
Writes to Target volumes must happen in the same order as they are written to the Source in order to have an instance in time, consistent and recoverable copy In Synchronous, Semi-synchronous and Asynchronous modes, writes are sent to the remote Symmetrix in the order received
If the remote Symmetrix is not accessible, writes are accumulated as invalid tracks When the remote Symmetrix becomes available, invalid tracks are sent without regard to serialization
Serialization is not maintained in Adaptive Copy mode

Typically used for data migrations
Serialization maintains the order in which writes are received at the remote (target) Symmetrix. SRDF serialization must be maintained in order to have a recoverable/restartable copy of data at a target site. Through serialization, write fidelity is guaranteed. In normal operations, SRDF maintains order writes with Synchronous, Semi-synchronous, and Asynchronous modes. But when the link becomes unavailable for any reason, writes accumulate as invalid tracks which the application continues to function on the host. When the link is restored, the Adaptive Copy mode is used to propagate changes across the link. This introduces risk, since serialization is not maintained with Adaptive Copy.
Adaptive Copy: Disk Mode
Source R1
Tracks
Target R2
T1 T2 T3 T4
32k 32k 32k 32k
T1
32k 32k
T3 T4
32k 32k
SRDF / DM
T5 T6 T7 T8
32k 32k 32k 32k
T5 T6 T7
32k 32k 32k 32k
The slide shows Adaptive Copy Disk Mode during in operation. SRDF does not guarantee serialization of the tracks being transferred in this mode. In this example, track 2 and track 8 may not be present on the target volume at the time of disaster rendering the target volume useless. Therefore, the target volume will not serve as a disaster protection mechanism. The consistency of the target volume is not maintained during the replication process in Adaptive Copy Write Pending or Disk Mode. The target is consistent only after the replication has completed.
Dynamic SRDF
Enables user to dynamically define relationships between R1 and R2 volumes Provides flexibility for user to tailor SRDF configuration to their changing application requirements
001 STD 001 R1 001 R1 001 STD
054
Create pair
STD 054 R2
Establish
Connectrix(s)
054 R2
Delete pair
054 STD
Prior to Dynamic SRDF, the R1 and R2 pairings were static and defined in the configuration file (BIN File) on the Symmetrix. Any changes to SRDF device pairing required a new BIN file to be defined and loaded into the Source and Target Symmetrix. Dynamic SRDF available with 5x68 Enginuity code will provide the capability to change device pairings on the fly without requiring a BIN file configuration change to be performed by an EMC Customer Engineers.
R1/R2 Swap
001 R1
054 R2 054
Connectrix(s)
001 R2
R1
An R1/R2 personality swap (or R1/R2 swap) refers to when the RDF personality of the RDF device designations of a specified device group are swapped so that source R1 device(s) become target R2 device(s) and target R2 device(s) become source R1 device(s). Dynamic RDF swaps are available with Enginuity version 5567 or later. To perform an R1/R2 swap, you must have an SRDF license with Symmetrix 5567 microcode or higher and Dynamic RDF must be enabled in your Symmetrix configuration. Sample scenarios for R1/R2 Swap - Symmetrix Load Balancing In todays rapidly changing computing environments, it is often necessary to deploy applications and storage on a different Symmetrix without having to give up disaster protection. R1/R2 swap can enable this redeployment with minimal disruption, while offering the benefit of load balancing across two Symmetrix storage arrays. - Primary Data Center Relocation Sometimes a primary data center needs to be relocated to accommodate business practices. For example, several financial institutions in New York City routinely relocate their primary data center across the Hudson River to New Jersey as part of their disaster drills. R1/R2 swaps allow these customers to run their primary applications in their New Jersey data centers. The Manhattan data centers now act as the disaster protection site. - Post-Failover Temporary Protection Measure If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of production computing to the target site without giving up the security of remote data protection. When all problems have been solved on the local Symmetrix, you have to failover again and swap the personality of the devices to go back to the original configuration.
Concurrent SRDF
One R1 can be paired with two R2 devices, concurrently Remote BCVs can be associated with only one of the R2 mirrors
M1
M2
M3
M4
Source
M1 M2 M3 M4
Target A Target B
M1 M2 M3 M4
Connectrix(s)
Concurrent SRDF allows two remote SRDF mirrors of a single R1 device, e.g. use one remote copy for disaster recovery, and another for decision support or backup. Each Remote Link Director is assigned to an RA Group. With ESCON, only one RA group per RLD is allowed, but Fibre Channel SRDF RA Groups can be defined to the same RLD. Any mixture of SRDF modes is allowed, except for Sync and Semi-sync configuration and Async and Async configuration. A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: The SRDF IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode.
Concurrent SRDF
One R1 can be paired with two R2 devices, one in each Symmetrix, concurrently All combinations of Primary/Secondary modes for the R1R2 pairs are allowed - except one pair in Sync and the other in semi-sync, both cannot be Async Cannot restore from both R2 mirrors to the R1 simultaneously SRDF swap is not allowed. For example if the R1 is changed to an R2 one will be left with R2->R1, R2>R2@#! Remote BCVs can be associated with only one of the R2 mirrors
A BCV can only be established with one of the Target volumes, not both. In case the source is locally protected, the BCV device cannot be established with its source, because all four(4) mirror positions will be occupied 2 Synchronous remote mirrors : A write IO from the host at the primary device side cannot be returned as completed until both remote Symmetrix signal the local Symmetrix that the SRDF IO is in cache at the remote side. 1 Sync and 1 Adaptive Copy remote mirror: The SRDF IO from the secondary device operating in Synchronous mode must present ending status to the sending Symmetrix before a second host IO can be accepted. The host I/O does not wait for the secondary device operating in Adaptive Copy mode. The same general principle applies when both remote mirrors are operating in Semi-Sync mode.
Module Summary
Key points covered in this module: Overview of SRDF solutions SRDF functionality and its uses SRDF Link configurations SRDF Groups SRDF swap functionality Characteristics of Concurrent and Dynamic SRDF
SRDF Operations
Upon completion of this module, you will be able to: Identify RDF volumes with SYMCLI Configure and display properties SRDF Device Groups Display and monitor the status of a Device Group Perform the following operations using SYMCLI:
SRDF Disaster Recovery SRDF Link Control SRDF Decision Support
Create and delete Dynamic RDF pairs using SYMCLI Describe Consistency Technology and its applications
2007 EMC Corporation. All rights reserved. SRDF Operations - 1
SRDF Operations - 1
Identify Accessible SRDF Volumes

symrdf list pd displays summary information about all SRDF volumes accessible to the host symrdf list dev displays summary information about all SRDF volumes in the Symmetrix
# symrdf list pd Symmetrix ID: 000187940398 Local Device View ------------------------------------------------------------------------STATUS MODES RDF S T A T E S Sym RDF --------- ----- R1 Inv R2 Inv ---------------------Dev RDev Typ:G SA RA LNK MDA Tracks Tracks Dev RDev Pair ---- ---- ------ --------- ----- ------- ------- --- ---- ------------0190 0190 R1:11 RW RW RW S.. 0 0 RW WD Synchronized 0191 0191 R1:11 RW RW RW S.. 0 0 RW WD Synchronized 0192 0192 R1:11 RW RW RW S.. 0 0 RW WD Synchronized... Total -------- -------Track(s) 0 2 MB(s) 0.0 0.1 Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off
SRDF Operations - 2
Configuring SYMCLI SRDF Device Groups

Related devices are grouped into device groups All devices in a disk group must be in the same Symmetrix ICDA All devices must be the same type (RDF1, RDF2, Regular) A device can only belong to a single Device Group per SYMAPI database
# symdg create -type RDF1 srcdg # symld -g srcdg add pd /dev/rdsk/c8t0d0 # symld -g srcdg add pd /dev/rdsk/c8t0d1 # symld -g srcdg add pd /dev/rdsk/c8t0d2 # symld -g srcdg add pd /dev/rdsk/c8t0d3
Create Device group Add Physical Devices to device group
SRDF Operations - 3
A device group is a logical grouping of Symmetrix volumes. There are three types of device groups: regular, rdf1, and rdf2. A device group with type regular cannot contain RDF volumes. Therefore, users must create a device group with rdf1 or rdf2 for SRDF operations. The device group definition is stored in the SYMAPI database on the host where the symdg create command was executed.
SRDF Operations - 3
Displaying SYMCLI Device Groups: Part 1

symdg show displays detailed information about a device group
# symdg show srcdg Group Name: srcdg Group Type : Device Group in GNS : Valid : Symmetrix ID : Group Creation Time : Vendor ID : Application ID : Number of STD Devices in Group : Number of Associated GK's : Number of Locally-associated BCV's : Number of Locally-associated VDEV's : Number of Remotely-associated BCV's (STD RDF): Number of Remotely-associated BCV's (BCV RDF): Number of Remotely-assoc'd RBCV's (RBCV RDF) :
RDF1 (RDFA) No Yes 000187940398 Tue Mar 29 16:42:10 2005 EMC Corp SYMCLI 1 0 0 0 0 0 0
SRDF Operations - 4
SRDF Operations - 4

Standard (STD) Devices (1): -------------------------------------------------------------------Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------DEV001 /dev/rdsk/emcpower182c 01EC RW 449 Device Group RDF Information RDF Type : R1 RDF (RA) Group Number : 3 (02) Remote Symmetrix ID : 000187940371 R2 Device Is Larger Than The R1 Device : False RDF Mode : Synchronous RDF Adaptive Copy : Disabled RDF Adaptive Copy Write Pending State : N/A RDF Adaptive Copy Skew (Tracks) : 65535 RDF Device Domino : Disabled RDF Link Configuration : Fibre RDF Link Domino : Disabled Prevent Automatic RDF Link Recovery : Enabled Prevent RAs Online Upon Power ON : Enabled (RW) Device RDF Status : Ready
SRDF Operations - 5

Device Suspend State : N/A Device Consistency State : RDF R2 Not Ready If Invalid : Device RDF State : Remote Device RDF State : RDF Pair State ( R1 <===> R2 ) : Number of R1 Invalid Tracks : Number of R2 Invalid Tracks : RDFA Information: Session Number : Cycle Number : Number of Devices in the Session : Session Status : Session Consistency State : Minimum Cycle Time : Average Cycle Time : Duration of Last cycle : Session Priority : Tracks not Committed to the R2 Side: Time that R2 is behind R1 : R1 Side Percent Cache In Use : R2 Side Percent Cache In Use :
Disabled Disabled Ready Not Ready Synchronized 0 0 2 0 1 Inactive N/A 00:00:30 00:00:00 00:00:00 33 0 00:00:00 0 0
(RW) (NR)
SRDF Operations - 6
SRDF Operations - 6
SRDF SYMCLI Commands
Syntax :
symrdf -g <group> <action> [options]
Where action can be one of the following:

Establish Restore Split Failover Failback Update Suspend Resume
SRDF Operations - 7
Users can perform a number of Symmetrix SRDF operations using host-based SYMCLI commands. Major SRDF operations include: ping, control, or modify operations on a device group; composite group, device file, or on a device within a device or composite group; performs Dynamic RDF group controls to add, modify, and remove a dynamic RDF group.
Please refer to EMC Solutions Enabler Symmetrix CLI Version 6.3 COMMAND REFERENCE P/N 300-000-877 REV A08 for the complete list of capability of the symrdf command.
SRDF Operations - 7
Verifying SRDF Link Status

# export SYMCLI_DG=srcdg # symrdf ping
symrdf ping symcfg -RA all list
# symrdf ping Successfully pinged (Remotely) Symmetrix ID: 000187940371 Successfully pinged (Remotely) Symmetrix ID: 000187940399 # symcfg -RA all list Symmetrix ID: 000187940398 (Local) S Y M M E T R I X R D F D I R E C T O R S Remote Local Ident Symb Num Slot Type Attr SymmID RA Grp RF-1D 01D 49 1 RDF-BI-DIR 000187940399 11 (0A) RF-2D 02D 50 2 RDF-BI-DIR 000187940371 52 (33) 000187940371 62 (3D) 000187940371 3 (02) 000187940371 12 (0B) RF-15D 15D 63 15 RDF-BI-DIR RF-16D 16D 64 16 RDF-BI-DIR -
Remote RA Grp 11 (0A) 52 (33) 62 (3D) 4 (03) 12 (0B) -
Status Online Online
Online Online
SRDF Operations - 8
The symrdf ping command checks if RDF link communication is operational. The symcfg command lists configuration information of all remote directors.
SRDF Operations - 8
Suspend SRDF Link

Logically suspends mirror relationship between source and target volumes
# export SYMCLI_DG=srcdg # symrdf suspend DEV001 # symrdf suspend DEV001 -nop An RDF 'Suspend' operation execution is in progress for device 'DEV001' in group 'srcdg'. Please wait... Suspend RDF link(s).......................................Done. The RDF 'Suspend' operation successfully executed for device 'DEV001' in group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View MODES ------------------------------------------------------- ----- -----------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE -------------------------------- -- ------------------------ ----- -----------DEV001 01EC RW 0 0 NR 032C WD 0 0 E.. Suspended Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
SRDF Operations - 9
The symrdf suspend command stops data transfer between specified pairs. New writes to source volume accumulate as invalid tracks.
SRDF Operations - 9
Resume SRDF Link

Logically resumes mirror relationship between source and target volumes
# export SYMCLI_DG=srcdg # symrdf suspend DEV001 # symrdf resume DEV001 -nop An RDF 'Resume' operation execution is in progress for device 'DEV001' in group 'srcdg'. Please wait... Resume RDF link(s)........................................Started. Resume RDF link(s)........................................Done. The RDF 'Resume' operation successfully executed for device 'DEV001' in group 'srcdg'. # symrdf query DEV001 Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View MODES ------------------------------------------------------- ----- -----------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE -------------------------------- -- ------------------------ ----- -----------DEV001 01EC RW 0 0 RW 032C WD 0 0 S.. Synchronized Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
The symrdf resume command resumes data transfer between pair. The Invalid tracks will start to synchronize. However, remember that the serialization is not maintained during the synchronization. Note: The #export SYMCLI_DG=srcdg command at the top of the slide. This command is setting the SYMCLI_DG variable, to equal a specific device group name. Setting this enables the user to perform Symrdf commands on a default device group. Example: Performing a query on the device group srcdg without the SYMCLI_DG variable set.
# symrdf g srcdg query
Performing a query on the device group srcdg with the SYMCLI_DG variable set.
# symrdf query
Because the SYMCLI_DG variable has been set the command symrdf query DEV001 is performing a query on device DEV001 within the device group srcdg. Note For this module, from this point on, the SYMCLI_DG variable has been set.
SRDF Operations - 10
Changing SRDF Operational Mode

# export SYMCLI_DG=srcdg # symrdf set mode semi
symrdf set mode
# symrdf set mode semi nop An RDF Set 'Semi-synchronous Mode' operation execution is in progress for device group 'srcdg'. Please wait... The RDF Set 'Semi-synchronous Mode' operation successfully executed for device group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC RW 0 0 RW 032C WD 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
MODES ----- ------------
RDF Pair MDA STATE ----- -----------E.. Synchronized
The symrdf set mode command changes SRDF operation mode. The noprompt (-nop) is used to bypass a confirmation question from the command line. This switch is applicable to most operations with the symrdf command.
SRDF Volume Operational Status

# export SYMCLI_DG=srcdg # symrdf query
symrdf query
# symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View MODES ------------------------------------------------------- ----- -----------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE -------------------------------- -- ------------------------ ----- -----------DEV001 01EC RW 0 0 RW 032C WD 0 0 S.. Synchronized Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0 Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off
The name of the device group can be exported as a SYMCLI environment variable so that you have to type it in each time. The query shows the SRDF status of the device group.
SRDF Disaster Recovery Operations

Failover: symrdf failover
Make copy of data on target Symmetrix volumes (R2) available to attached hosts Disaster: host channel, Symmetrix or site failure Maintenance operation: provide data availability during host, Symmetrix, or site maintenance
Update:
symrdf update
Begins synchronization prior to resuming operations on source volumes
Failback:
symrdf failback
Resumes operation using primary host and copy of data on source volumes (R1): Saves all changes made during failover
The disaster recovery operations for SRDF devices are: Failover from the source side to the target side, switching data processing to the target side. Failback from the target side to the source side by switching data processing to the source side. Update the source side after a failover while the target side may still be operational to its local host.
SRDF Failover
# export SYMCLI_DG=srcdg # symrdf failover # symrdf failover -nop An RDF 'Failover' operation execution is in progress for device group 'srcdg'. Please wait... Write Disable device(s) on SA at source (R1)..............Done. Suspend RDF link(s).......................................Done. Read/Write Enable device(s) on RA at target (R2)..........Done. The RDF 'Failover' operation successfully executed for device group 'srcdg'. # symrdf query Device Group (DG) Name DG's Type DG's Symmetrix ID : srcdg : RDF1 : 000187940398 -----------RDF Pair STATE -----------Failed Over
symrdf failover
Source (R1) View Target (R2) View MODES ------------------------------------------------------- ----ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA -------------------------------- -- ------------------------ ----DEV001 Total MB(s) 01EC WD 0 0 NR 032C RW 0 0 S.. -------- -------0.0 0.0 -------- -------0.0 0.0
In a period of scheduled downtime for maintenance, or after a serious system problem which has rendered either the host or Symmetrix unit containing the source (R1) devices unreachable, no read/write operations can occur on the source (R1) device. In this situation, the failover operation should be initiated to make the target (R2) devices read/write enabled to their local host(s).
Updating Source Volumes
symrdf update
# symrdf update -until 1000 # export SYMCLI_DG=srcdg Execute an RDF 'Update R1' operation for devic # symrdf update until 1000 group 'srcdg' (y/[n]) ? An RDF 'Update R1' operation execution is in progress for device group 'srcdg'. Please wait... Suspend RDF link(s).......................................Done. Merge device track tables between source and target.......Started. Device: 01EC ............................................ Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Started. Resume RDF link(s)........................................Done. The RDF 'Update R1' operation successfully initiated for device group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC WD 0 0 RW 032C RW 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
MODES ----- -----------RDF Pair MDA STATE ----- -----------S.. R1 Updated
While the target (R2) device is still operational (Write Enabled to its local host(s)), an incremental data copy from the target (R2) device to the source (R1) device can be initiated in order to update the R1 mirror with changed tracks from the target (R2) device.
SRDF Failback
# symrdf failback -nop An RDF 'Failback' operation execution is in progress for device group 'srcdg'. Please wait... Write Disable device(s) on RA at target (R2)..............Done. Suspend RDF link(s).......................................Done. Merge device track tables between source and target.......Started. Device: 01EC ............................................ Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Started. Resume RDF link(s)........................................Done. Read/Write Enable device(s) on SA at source (R1)..........Done. The RDF 'Failback' operation successfully executed for device group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC RW 0 0 RW 032C WD 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
symrdf failback
MODES ----- ----------RDF Pair MDA STATE ----- ----------S.. Synchronized
A failback, or source (R1) device takeover, is performed when you are ready to resume normal SRDF operations by initiating read/write operations on the source (R1) devices, and stopping read/write operations on the target (R2) devices. The target (R2) devices become read-only to their local host(s) while the source (R1) devices are read/write enabled to their local host(s). Host activity should be stopped prior to execution: Stop Application Unmount file systems and deactivate volume groups May be executed on either source or target Symmetrix. Will abort if data integrity cannot be guaranteed.
SRDF Decision Support/Concurrent Operations

Split: symrdf split
Places the Symmetrix units in a state for concurrent access
Suspends link between source (R1) and target (R2) volumes Enables read and write operations on both source and target volumes
To re-integrate SRDF Volumes:

Save source data: symrdf establish
Resume Normal SRDF operations Preserves data on the source (R1) volumes, discarding changes to the target (R2) volumes
Save target data: symrdf restore

Resume SRDF operations Preserves data on the target (R2) volumes, discarding changes to the source (R1) volumes
The decision support operations for SRDF devices are: Establish an SRDF pair by initiating a data copy from the source side to target side. The operation can be full or incremental. Restore remote mirroring. Initiates a data copy from the target side to the source side. The operation can be full or incremental. Split an SRDF pair which stops mirroring for the SRDF pairs in a device group.
Concurrent Operations
symrdf split
# export SYMCLI_DG=srcdg # symrdf split -nop # symrdf split An RDF 'Split' operation execution is in progress for device group 'srcdg'. Please wait... Suspend RDF link(s).......................................Done. Read/Write Enable device(s) on RA at target (R2)..........Done. The RDF 'Split' operation successfully executed for device group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC RW 0 0 NR 032C RW 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
MODES ----- ------------
RDF Pair MDA STATE ----- -----------S.. Split
The command suspends the link between source (R1) and Target (R2) volumes. It also enables read and write operations on both source and target volumes. Changes to source are kept track of as R2 invalids. Changes to target are kept track of as R1 invalids.
Concurrent Operations: Saving Source Data

# symrdf establish -nop symrdf establish An RDF 'Incremental Establish' operation execution is in progress for device group 'srcdg'. Please wait... Write Disable device(s) on RA at target (R2)..............Done. Suspend RDF link(s).......................................Done. Resume RDF link(s)........................................Started. Merge device track tables between source and target.......Started. Device: 01EC ............................................ Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Done. The RDF 'Incremental Establish' operation successfully initiated for device group 'srcdg'. # symrdf query Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View MODES ------------------------------------------------------- ----- -----------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDA STATE -------------------------------- -- ------------------------ ----- -----------DEV001 01EC RW 0 0 RW 032C WD 0 0 S.. Synchronized Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
Resume SRDF operation retaining data from source and overwriting data on target.
Concurrent Operations: Saving Target Data

# symrdf restore nop
symrdf restore
An RDF 'Incremental Restore' operation execution is in progress for device group 'srcdg'. Please wait... Write Disable device(s) on SA at source (R1)..............Done. Write Disable device(s) on RA at target (R2)..............Done. Suspend RDF link(s).......................................Done. Merge device track tables between source and target.......Started. Device: 01EC ............................................ Merged. Merge device track tables between source and target.......Done. Resume RDF link(s)........................................Started. Resume RDF link(s)........................................Done. Read/Write Enable device(s) on SA at source (R1)..........Done. The RDF 'Incremental Restore' operation successfully initiated for device group 'srcdg'. # symrdf query -i 5 -c 5 Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC RW 0 0 RW 032C WD 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0. 2007 EMC Corporation. All rights reserved. 0.0
MODES ----- -----------RDF Pair MDA STATE ----- -----------S.. Synchronized

Resumes SRDF operation retaining data on target and overwriting data on source.
Continuous Monitoring
symrdf query
# export SYMCLI_DG=srcdg # symrdf query -i 5 -c 5 Device Group (DG) Name : srcdg DG's Type : RDF1 DG's Symmetrix ID : 000187940398 Source (R1) View Target (R2) View ------------------------------------------------------ST LI ST Standard A N A Logical T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks -------------------------------- -- -----------------------DEV001 01EC RW 0 0 RW 032C WD 0 0 Total -------- --------------- -------MB(s) 0.0 0.0 0.0 0.0
-i 5 -c 5
MODES ----- ------------
RDF Pair MDA STATE ----- -----------S.. Synchronized
Status of RDF Volume is a specific device group: [-i] interval [-c] count Provides summary of synchronization rate and estimated completion time.
Add / Remove Dynamic RDF Groups

# symrdf -v addgrp -label grp58 -rdfg 58 -sid 187400011 -dir 4D \ -remote_rdfg 58 -remote_sid 000187400093 -remote_dir 3C -noprompt
# symcfg list -ra 4D -sid 11 Symmetrix ID: 000187400011 S Y M M E T R I X R D F D I R E C T O R S Remote Ident Symb Num Slot Type RF-4D 04D 52 4 Attr SymmID Local Remote Status
RA Grp RA Grp
RDF-R1 -
000187400093 27 (1A) 23 (16) Online 000187400093 28 (1B) 24 (17) 000187400093 29 (1C) 25 (18) 000187400093 30 (1D) 26 (19) 000187400093 31 (1E) 27 (1A) 000187400093 58 (39) 58 (39)
# symrdf -v removegrp -label grp58 -noprompt

The symrdf addgrp command creates an empty dynamic RDF group that represents another RDF link between Symmetrix 000187400011 and Symmetrix 000187400093. It adds dynamic RDF group 58 on the local Symmetrix, and RDF group 58 on the remote Symmetrix. You must specify a group label (grp58 in this case) that can be used when modifying or deleting the group. Creation of the dynamic RDF group includes director 4D from the local Symmetrix and 3C from the remote Symmetrix as the director end points of this connection. The symrdf removegrp command deletes the dynamic group. The group must be empty to be deleted. It is important to be aware of your network topology when creating dynamic RDF groups between two Symmetrix arrays. To create a dynamic RDF link (a connection) between RA directors, the director end points must be able to see each other through the Fibre Channel fabric. For example, a dynamic RDF link can be created between local and remote directors only if the Fibre Channel zoning is set up so that the two directors can see each other through the fabric.
Dynamic RDF Pair Requirement

Make sure both Symmetrix arrays have the dynamic setting enabled
# symconfigure sid 296 list -v
Symmetrix ID Configuration Server Version Configuration Server Protocol Configuration Server Date (TRUNCATED) Dynamic RDF Configuration RAID-S support (TRUNCATED) : : : : 000187900296 5671.32.36 0x509 03.29.2005
: Enabled : Enabled
# symcfg sid 296 list -v # symcfg sid 296 list -v

Switched RDF Configuration State Concurrent RDF Configuration State Dynamic RDF Configuration State : Enabled : Enabled : Enabled
Concurrent Dynamic RDF Configuration : Enabled RDF Data Mobility Configuration State: Disabled
Since Enginuity 5568, devices can be configured to be Dynamic RDF-capable devices. Dynamic RDF functionality enables you to create, delete, and swap SRDF pairs while the Symmetrix array is in operation. Using Dynamic RDF technology, you can establish SRDF device pairs from non-configured SRDF devices, then synchronize and manage them in the same way as configured SRDF pairs. Note: Running the symcfg list command to check for concurrent RDF configuration.
Pairing Dynamic Devices

Determine source and target dynamic devices for pairing symdev list dynamic both Create a device file and list source and target pairs 09C 09D 09E 054 055 056
Issue symrdf createpair against this file Upon execution of this command, pairing information will be added to the SYMAPI database file on the host
Using Dynamic RDF technology, you can establish SRDF device pairs from non-SRDF devices using the symrdf createpair command. Once established, the new SRDF pairs can be synchronized and managed in the same way as configured SRDF pairs. Prior to Enginuity version 5568, SRDF device pairing was limited to the static SRDF pairs set at Symmetrix configuration time. Dynamic RDF enables the creation and deletion of SRDF pairs while the Symmetrix array is in operation. The symrdf deletepair command is used to cancel the Dynamic SRDF pairing.
Dynamic SRDF Options

Establish option:
Invalidates R2s, merges track tables, brings up RDF links, starts copy process from R1 to R2
symrdf createpair file file sid 01 type rdf1 -rdfg 2 establish Restore option:
Invalidates R1s, merges track tables, brings up RDF links, starts copy process from R2 to R1
symrdf createpair file file sid 01 type rdf1 -rdfg 2 restore Invalidate option:
Allows creation of dynamic SRDF pairs, but does not bring up the RDF links and initiate data copy
To perform an establish, use -invalidate r2 To perform a restore, use -invalidate r1
symrdf createpair file file sid 01 type rdf1 -rdfg 2 invalidate r2
Deleting Device Pairings

Removes the pairing information from the Symmetrix Must suspend RDF links using before issuing symrdf deletepair command (link state must be NR and pair state is Suspended, Split, or FailedOver): symrdf -f <devicefile> suspend -sid 01 -rdfg 2 symrdf deletepair -f <devicefile> -sid 01 -rdfg 2 Dynamic SRDF pairs can also be cancelled within the context of a device group symrdf deletepair g sargrp Canceling dynamic SRDF pairings changes the type of the device group from RDFx to Regular Devices in the device group are changed from DRx devices to RDFcapable standard devices and the SYMAPI database is updated
What is Consistency Technology?

Consistent Split Ensures consistency of applications in the same Symmetrix Composite Groups Ensures consistency of multiple databases and platforms across Symmetrix over distances
Composite Group
BCV UDB
SRDF
BCV UDB UDB UDB
Oracle
BCV
Oracle
SRDF
Consistent split allows you to avoid inconsistencies and restart problems that can occur if you split a database-related BCV without first quiescing the database. TimeFinder provides the capabilities to split off a consistent, restartable copy of a database with negligible impact on the production environment. A Composite Group is a user-defined group of SRDF devices that act in unison to maintain the integrity of a database distributed across multiple Symmetrix units or multiple RDF groups within a single Symmetrix. If a source R1 device in the Composite Group cannot propagate data to the target R2 device, data propagation from all R1 devices in the Composite Group is halted. This suspension is called tripping the Composite Group.
What Are Composite Groups?

A Composite Group is a user-defined group of devices that act in unison to maintain the integrity of a database distributed across multiple Symmetrix units or multiple RDF groups within a single Symmetrix It allows you to remotely mirror your largest databases and automatically split off a consistent, DBMS restartable copy of the database in seconds, with no interruption to online service Disaster restart solutions using Composite Groups provide remote restart with a short restart time and zero data loss A composite group with remote consistency protection enabled is known a Consistency Group
Composite group is another logical grouping of Symmetrix devices. There are three types of composite groups: REGULAR, RDF1, RDF2. Remote consistency protection is applicable only to the RDF type composite group. Consistency groups protect the consistency of one or more database management system (DBMS) that span RDF groups during a disaster. A Consistency Group provides synchronous disaster restart with zero data loss, even when databases span multiple hosts and Symmetrix.
Tripping the Consistency Group

If a source R1 device in the Consistency Group cannot propagate data to its corresponding target R2 device, the data propagation from all R1 devices in the Composite Group is suspended
Automatic Tripping: Occurs when one or more R1 source devices in an enabled Composite Group cannot propagate data to their corresponding R2 target devices. For example:
All RDF links between the R1 and R2 might go down R2 device might fail RDF directors on the R1 side or R2 side might fail
Manual Tripping: Occurs when you invoke the symrdf cg suspend or split command
Suspending or splitting the Consistency Group creates an on-demand, DBMS restartable copy of the database on the R2 target side. symrdf cg suspend The R2 devices are in the write-disabled state at the end of the trip and cannot be accessed by targetside hosts. (This maintains the consistency of the R2 database copy with the production copy on the R1 side.) symrdf cg split The R2 devices are enabled for both reads and writes by the target-side hosts.
Problem: Rolling Disasters
HOST
R1(D)
Data ahead of Log

Data write
3
R2(D)
Data write DBMS Log write
R1(C)
R2(C)
R2(B)
1 R2(A)
R1(B)
2 R1(A)
1. 2. 3. 4.
Rolling disaster begins Log write Dependent data write Inconsistent data
The diagram shows that the log write cannot reach the remote Symmetrix because of a problem with the SRDF link. Meanwhile, the data write has reached the remote Symmetrix via another SRDF link. This is known as data ahead of log condition. Almost all database management systems restart from this condition, without an error or any correction. The integrity of the data in the DBMS is compromised and the data is inconsistent.
Solution: Enterprise SRDF Consistency Groups

4 Suspend 5 R1/R2 Relationship DBMS 7 Restartable Copy
HOST
R1(D) ConGroup Started Task with Host Component ConGroup definition
Data write
R1(C)
R2(D)
R2(C) 1
R2(B)
DBMS
ECA PowerPath
Log write
R1(B)
2 R2(A)
3 R1(A)
1. 2. 3. 4. 5. 6. 7.
Consistency protection enabled Rolling disaster begins Log write Consistency trip Suspend R1/R2 Data write Restartable copy
Now that the composite group is defined and consistency protection enabled, our rolling disaster begins with the loss of the SRDF links from bottom source Symmetrix to the bottom target Symmetrix. A sense code is sent back stating that the data from volume A could not be propagated to its target side. The composite group started task on the mainframe or the SYMCLI/SYMAPI detects the sense code and works with ECA or PowerPath to hold the I/O. While the I/O is held, two I/Os are sent per Symmetrix. The first request sets the volumes in the Composite Group in a suspend pending state for all volumes in the composite group, and the second request suspends the relationship between the source and target volumes (R1 and R2s). The I/O is released within milliseconds. I/O continues to occur on the source host until the complete disaster happens. The DBMS or application are not aware that we held the I/O and created a Dependent Write Consistent copy or DBMS restartable copy of the data on the target side. We have simulated a local power failure at the target side at the point of the beginning of the rolling disaster. After complete failure on the source side, the target side can be restarted and the DBMS can be restarted, which provides transactional consistency.
What Happens to the Host-Based Groups

Moved into the inactive list To display the inactive SYMAPI database groups on the host: symdg list inactive symcg list -inactive To import specific inactive groups into GNS: symdg activate MyDG symcg activate MyCG To import all inactive groups: symdg activateall symcg activateall
SRDF Control RDF Split Action

Replication SRDF Control
Split option Flags Split option Flags Select the Devices to Split Select the Devices to Split
SRDF Control operations are performed by right clicking the RDF Device Group and choosing Replication SRDF Control. The user selected a Split operation in the above slide. The same dialog box can be used for the other actions like Failback, Failover, Establish etc. The SRDF Control window has 2 pages. On page 1 the action (Split for example) and the device pairs are chosen. Page 2 allows the user to choose options that relate to a specific action and to execute the action via the Finish button.
SMC SRDF RDF Split Action (Cont.)
Devices Group SMC01 In aaSplit State Devices Group SMC01 In Split State
Devices Groups Devices Groups
The above slide shows the RDF Device Group SMC01 in a split state. Selecting the SMC01 device group within the SMC Navigation Tree automatically presents the current status in the View Area window. Remember, the SMC GUI closely match Solutions Enabler CLI command, to include all basic monitoring, configuration, and control for any attached Symmetrix arrays.
Module Summary
Key points covered in this module: RDF volumes with SYMCLI SRDF Device Groups configuration and display properties Displayed and monitor the status of a Device Group SRDF Disaster Recovery operations using SYMCLI SRDF Link Control operations using SYMCLI SRDF Decision Support operations using SYMCLI Created and deleted Dynamic RDF pairs using SYMCLI Described Consistency Technology and its applications
SRDF/A (Asynchronous)
Upon completion of this module, you will be able to:
List the technical requirements of a successful SRDF/A implementation Identify all supported SRDF/A hardware platforms Describe factors that affects RPO (Recovery Point Objective) in an SRDF/A implementation Discus Transmit Idle and DSE (Delta Set Extension) List and describe Cycles within SRDF/A operations Describe SRDF/A Consistent Deactivation Describe Multi Session Consistency Describe configuration parameters that affect SRDF/A behavior
SRDF/A (Asynchronous) - 1
What is SRDF/A?
Asynchronous remote mirroring
Minimal impact to production applications Extended distance Always consistent image on R2
Efficient bandwidth usage Supports Mainframe and Open Systems Complements existing SRDF solutions
Meet a wide range of RPO and RTO service level requirements
Mixed SRDF and SRDF/A

Share links and directors
SRRD/As unique architecture delivers a remote mirroring solution that has no impact on production applications over extended distance because I/O requests from the host are acknowledged locally. Changes made to the same data blocks are periodically sent only once to the remote Symmetrix. This enables significant operational savings through reduced bandwidth requirements. Moreover, SRDF/A provides an alternative Disaster Recovery solution in addition to SRDF/S by maintaining a consistent image of RDBMS on the R2 at all timess. SRDF/A is a single solution supporting both Mainframe and Open Systems attaches. It also compliments SRDF solutions to meet mixed service level requirements. In fact, it can also share the same communication links as SRDF.
SRDF/A Supported Environments

All existing SRDF topologies
ESCON with Farpoint Fibre Channel IP
Any host
Mainframe Open System
All emulation types supported by Symmetrix, including:

FBA (Fixed Block Architecture) CKD (Count Key Data)
Key operating systems, including:

UNIX (Sun, HP, IBM) Windows (NT, 2000, 2003) IBM Mainframe (OS/390, z/OS)
Note: Support begins with Symm 5670 microcode and carries forward to future generations of Symmetrix
2007 EMC Corporation. All rights reserved. SRDF/A (Asynchronous) - 3
Industrys Traditional Write Ordering

Source Location
ALL WRITES
Target Location 3 ALL WRITES

Write #7 #4 #6 Write #3 #7 Write #5
1
1. Host I/O Written To Cache
Write #1: Write #2 Write #3 Write #4 Write #5 Write #6 Write #7
3. Writes Must Be Re-ordered Before Destaging
2
2. EVERY WRITE must be timestamped / ordered and sent to the Target side
Write #2 #5 Write #6 #3 Write #4 #2 Write #1
Cache
Cache
4
Target
Files
Traditional approaches to asynchronous mirroring have their architectural shortfalls. Uses a combination of cache and files to perform mirroring. Timestamps or sequence numbers are applied to each and every incoming write. Each write must have a timestamp applied to it before sending it to the remote side. That means every single write MUST be sent to the remote side, and because they do not necessarily arrive in order, they must be re-sequenced before being applied to disk. If you have writes number 100-200 pending at the remote side, all waiting for write number 99 to arrive, the system must resequence and wait for number 99 to arrive before committing writes 100-200. This creates a significant amount of overhead and data management activity in both the source and target systems as they scramble to time-stamp, send, re-order, wait for a dependant time-stamp, and then eventually commit the writes to disk at the target side.
SRDF/A Architecture and Ordered Cycles

SRDF/A Delta Set Begins SRDF/A Delta Set
Source Sym
Source (R1) Device
Cycle N
Capture New Write Cycle -(N)
Cycle N-1
Transmit to R2 Cycle -(N-1)
Source Host Side A

Target Sym
Target (R2) Device
Cycle N-2
Cycle N-1
Write Applied to R2 Receive Writes on R2 Cycle -(N-1) Cycle -(N-2)
Target Host Side B

SRDF/A Delta Set
When the SRDF/A cycle (N) is active on the source Symmetrix, it collects any new writes in the R1 Symmetrix cache, overwriting any duplicate tracks intended for data transfer over the link. The cycle is active for a pre-determined amount of time that can be configured on the Symmetrix at the time of the initial configuration of the SRDF/A environment; the default time is 30 seconds. After the set time has been reached, the delta set data inherits the next cycle position (N-1) and begins transferring the delta set over the link to the R2. Then, a new cycle N begins collecting new writes again for the next delta set transfer. In cycle (N-1), the delta set is temporarily collected on the R2 side for destaging. When the (N-1) cycle has finished transferring data into the R2 and the minimum cycle time has elapsed, the delta set inherits the next cycle position (N-2) and begins destaging the data to the R2 storage devices. The delta set data is considered committed to the R2 in cycle (N-2). Thus, it takes two(2) cycles for the changes from R1 to get to the R2 which make the shortest RPO of an SRDF/A environment to be twice the cycle. That is, if the cycle time is 30 seconds, the RPO is at least 60 seconds.
Dependent Write Consistency

Dependent write logic:
If A is a predecessor and B is a dependent write Any I/O B that arrives after I/O A has been acknowledged to the host, must be dependent on A Implemented by the application, such as RDBMSs
SRDF/A ensures that:

A and B are in the same Delta Set or B is in later Delta Set
All commonly used database management systems are inherently dependent write consistent. For instance, a DBMS will not perform a log write, indicating that a transaction is complete, until it has received an acknowledgement from the storage subsystem that the log data pertaining to the transaction itself was completely written to disk. Symmetrix honors this logic in SRDF/A by treating any successor I/O , which arrives after a predecessor I/O as a dependent I/O.
Verifying the Environment

List the configuration of the Symmetrix to verify that relevant System Attributes have been set
symcfg list -v sid xxx
List the Remote Adapters configured on the Symmetrix and verify their status
symcfg list ra all
List the RDF Groups that have been created on the Symmetrix
symcfg list rdfg all symrdf list rdfg RDF_Group_Number
System Attributes
DMX8HP1[ksh] # symcfg list -v -sid 35|more Symmetrix ID: 000187910035 -Output TruncatedCache Size # of Available Cache Slots Max # of System Write Pending Slots Max # of DA Write Pending Slots Max # of Device Write Pending Slots -Output TruncatedSDDF Configuration State Configuration Change State -Output TruncatedSwitched RDF Configuration State Concurrent RDF Configuration State Dynamic RDF Configuration State : Enabled : Enabled : Enabled : Enabled : Enabled : : : : : 16384 (MB) 475943 110000 55000 10340
Concurrent Dynamic RDF Configuration : Enabled -Output TruncatedSRDF/A Maximum Host Throttle (Secs) : 0
SRDF/A Maximum Cache Usage (Percent) : 94
As noted earlier, SRDF/A is supported under all topologies. However, if the attributes indicated in bold are also set, then the end user can create Dynamic RDF Groups, create dynamic pairs of SRDF devices, and perform Concurrent RDF operations. The switched RDF Configuration State must be enabled from the service processor. The other attributes can be set using Symmetrix Configuration Manager command symconfigure. The use of Host Throttle, and Maximum Cache Usage attributes are explained later in this module.
Configured Remote Adapters

DMX8HP1[ksh] # symcfg list -ra all|more Symmetrix ID: 000187910035 (Local) S Y M M E T R I X R D F D I R E C T O R S Remote Ident RF-1D RF-2D RF-15D RF-16D Symb 01D 02D 15D 16D Num 49 50 63 64 Slot 1 2 15 16 Type RDF-R1 RDF-R1 RDF-R1 RDF-R1 Attr SymmID 000187910156 000187910156 000187910156 000187910156 Local RA Grp 1 (00) 1 (00) 1 (00) 1 (00) Remote RA Grp Status
1 (00) Online 1 (00) Online 1 (00) Online 1 (00) Online
Symmetrix ID: 000187910156 (Remote) S Y M M E T R I X R D F D I R E C T O R S Remote Ident RF-1D RF-2D RF-15D RF-16D Symb 01D 02D 15D 16D Num 49 50 63 64 Slot 1 2 15 16 Type RDF-R2 RDF-R2 RDF-R2 RDF-R2 Attr SymmID 000187910035 000187910035 000187910035 000187910035 Local RA Grp 1 (00) 1 (00) 1 (00) 1 (00) Remote RA Grp Status
1 (00) Online 1 (00) Online 1 (00) Online 1 (00) Online
The output has been truncated and formatted to show just the relevant information. As displayed, the pair of Symmetrix units have four Remote Adapters each 1D, 2D, 15D, 16D. Currently all four are Online (Status).
List of All RDF Groups

DMX8HP1[ksh] # symcfg list -rdfg all Symmetrix ID : 000187910035 S Y M M E T R I X Local ------------LL RA-Grp (sec) RA-Grp SymmID T Name Remote -------------------R D F G R O U P S Group ----------------------Flags LPD Dir Cfg RDFA Info ----------------Flags CSR -----IS -IS -IS XAS Cycle time ----30 30 30 30 Pri --33 33 33 33
------------1 ( 0) 2 ( 1) 3 ( 2) 4 ( 3) 10 10 10 10
-------------------1 ( 0) 000187910156 2 ( 1) 000187910156 3 ( 2) 000187910156 4 ( 3) 000187910156
----------------------S RDFDVGR00 S RDFDVGR01 D vsrdfag3 D vsrdfag4 .X. .X. XX. XX. F-S F-S F-S F-S
-Remote Symmetrix Output Truncated-
The output shows the pairings between the local and remote RA Groups (RDF Groups). These pairings provide the logical connection between the two Symmetrix. For example, SRDF devices in RA Group 1 in sid 35 have their remote mirrors in RA Group 1 in sid 56. RA Groups 1 and 2 are of the Type Static. These were created by the CE from the service processor. RA Groups 3 and 4 are of the Type Dynamic. These were created by the user from the command line (SYMCLI). The Directors have been configured in Fibre Channel Switched mode (F-S). RA Group 4 is in SRDF/A Active state (XAS), and consistency has been enabled for this group. Legend: Group (T)ype Group Flags : : : X = Enabled, . = Disabled S = Static, D = Dynamic
Prevent Auto (L)ink Recovery
Prevent RAs Online Upon (P)ower On: X = Enabled, . = Disabled Link (D)omino Director (C)onfig : : X = Enabled, . = Disabled F-S = Fibre-Switched, F-H = Fibre-Hub G = GIGE, E = ESCON, T = T3, - = N/A RDFA Flags : - = N/A - = N/A
(C)onsistency : X = Enabled, . = Disabled, (S)tatus (R)DFA Mode : A = Active, I = Inactive,
: S = Single-session, M = MSC, - = N/A
List of an Individual RDF Group

DMX8HP1[ksh] # symrdf list -rdfg 3 Symmetrix ID: 000187910035 Local Device View ------------------------------------------------------------------------STATUS Sym Dev RDev RDF Typ:G MODES R1 Inv Tracks RDF S T A T E S
--------- ----SA RA LNK MDA
R2 Inv ---------------------Tracks Dev RDev Pair ------- --- ---- ------------0 RW 0 RW WD WD Synchronized Synchronized
---- ---000B 000B 000C 000C Total Track(s) MB(s)
------ --------- ----- ------R1:3 R1:3 RW RW RW RW RW RW S.. S.. 0 0
-------- -------0 0.0 0 0.0
Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) A(daptive Copy) : X = Enabled, . = Disabled : D = Disk Mode, W = WP Mode, . = ACp off
The list of devices in RDF Group 3 are displayed here. The two devices 00B and 00C are members of the RDF Group. They are in Synchronous mode of SRDF operations.
Transition to SRDF/A
From Synchronous
If the devices are in Synchronized state, then by definition the R2 devices have a consistent image. Enabling SRDF/A immediately provides consistent data on the R2
From Adaptive Copy Disk

Any invalid tracks owed to the R2 are synchronized. Two cycle switches after Synchronization, SRDF/A provides consistent data on the R2
From Adaptive Copy Write Pending

Write pending slots are merged into the Active SRDF/A cycles. When there are no more write pending slots, it takes an additional two cycle switches before R2 data is consistent
SRDF/A can be enabled when the device pairs are operating in any of the listed modes. In the case of Adaptive Copy to SRDF/A transitions, it takes two additional cycle switches after resynchronization of data for the R2 devices to be consistent.
Example: Synchronous to SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query Device Group (DG) Name DG's Type DG's Symmetrix ID Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : vsrdfadg3 : RDF1 : 000187910035 Target (R2) View MODES
------------------------ ----- -----------ST A T E R1 Inv Tracks R2 Inv Tracks MDA RDF Pair STATE
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
0 RW 000B WD 0 RW 000C WD
0 0
0 S.. 0 S..
Synchronized Synchronized
Total Track(s) MB(s)
-------- -------0 0.0 0 0.0
-------- -------0 0.0 0 0.0
The device pairs are operating in SRDF Synchronous mode (S..) and the pair states are Synchronized, prior to enabling SRDF/A.
Enabling SRDF/A
DMX8HP1[ksh] # symrdf -g vsrdfadg3 set mode async DMX8HP1[ksh] # symrdf g vsrdfadg3 enable DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status : 2 : 2 : Active
Tracks not Committed to the R2 Side: 64 Time that R2 is behind R1 RDFA R1 Side Percent Cache In Use RDFA R2 Side Percent Cache In Use Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : 00:00:34 : : 0 0 Target (R2) View MODES
------------------------ ----- -----------ST A T E R1 Inv Tracks R2 Inv Tracks MDAC RDF Pair STATE
-------------------------------- -- ------------------------ ----- -----------DEV001 DEV002 000B RW 000C RW 0 0 0 RW 000B WD 0 RW 000C WD 0 0 0 A..X 0 A..X Consistent Consistent
Transition to SRDF/A is immediate (A..X) and the pair state is immediately Consistent. Legend for MODES: M(ode of Operation): A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy D(omino) A(daptive Copy) : X = Enabled, . = Disabled : D = Disk Mode, W = WP Mode, . = ACp off
C(onsistency State): X = Enabled, . = Disabled, - = N/A
Example: ACP_Disk to SRDF/A

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query Device Group (DG) Name DG's Type DG's Symmetrix ID Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : vsrdfadg3 : RDF1 : 000187910035 Target (R2) View MODES
-------------------------------- -- ------------------------ ----- -----------DEV001 DEV002 Total Track(s) MB(s) 000B RW 000C RW 0 0 151 RW 000B WD 103 RW 000C WD 0 0 0 C.D 0 C.D SyncInProg SyncInProg
-------- -------0 0.0 254 7.9
-------- -------0 0.0 0 0.0
In this example, the device pairs are operating in SRDF Adaptive Copy Disk Mode (C.D). There are a number of invalid tracks owed to the R2 devices. This is governed by the skew value that has been set.
Enabling SRDF/A
DMX8HP1[ksh] # symrdf -g vsrdfadg3 set mode async DMX8HP1[ksh] # symrdf -g vsrdfadg3 enable DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status : 2 : 1 : Active
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
0 0
0 A..X 0 A..X
SyncInProg SyncInProg
Note that the transition into SRDF/A is immediate (A..X), and the group has been enabled for consistency. However, the pair state is SyncInProg. R2 device does not have consistent data until the invalid tracks owed have been resynchronized, and a further two cycle switches have occurred.
Consistent R2 Data
DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status : 2 : 3 : Active
-------------------------------- -- ------------------------ ----- -----------DEV001 DEV002 000B RW 000C RW 0 0 0 RW 000B WD 0 RW 000C WD 0 0 0 A..X 0 A..X Consistent Consistent
In this example, all invalid tracks were synchronized within the first cycle. Consequently, by the third cycle the RDF Pair STATE has become Consistent. Tracks not Committed to the R2 Side is a measure of data in Capture, Transmit, and Receive cycles at the time of the query.
Display Devices with SRDF/A Enabled

DMX8HP1[ksh] # symdev show 0b|more RDF Information: RDF Mode RDF Pair State ( RDFA Information: Session Number Cycle Number Number of Devices in the Session Session Status Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority : 2 : 8 : 2 : Active : Enabled : 00:00:30 : 00:00:29 : 00:00:29 : 33 R1 <===> R2 ) : Asynchronous : Consistent
Tracks not Committed to the R2 Side: 46 Time that R2 is behind R1 R1 Side Percent Cache In Use R2 Side Percent Cache In Use : 00:00:44 : : 0 0
When devices are in SRDF/A enabled state, the display includes their RDFA information. Session Number represents the RDF group number.
SRDF/A Consistent Deactivation from Async to Sync

5671 Environment Host Transition request
C
R1 R1
R Asyn
R2 R2 Phase 1
T C
R1 R1
A R
Switch
Phase 2
C Capture T Transmit R Receive A - Apply
Host
T C
R1 R1
Syn
R2 R2
A R
R2 R2
Switch
Phase 3
Host
A
Cache Cache
The Consistent Deactivation process works in three phases. In the first phase, SRDF/A operates normally. When the request to transition from SRDF/A to synchronous is received, at the next cycle switch, which guarantees an empty active cycle at the R1 side, a transition to the second phase occurs In the second phase, new writes at the R1 side are sent directly in synchronous mode to the R2 side, with one key exception. As these write arrive at the R2, they are kept in the inactive (receive) cycle at the R2 side. And, the inactive (transmit) cycle at the R1 side continues to send data to the inactive (receive) cycle at the R2. At the next cycle switch (two switches into the process), a transition to the third phase occurs From phase 2, when the next cycle switch is received (two cycle switches into the process), the inactive cycle at the R2 side becomes the active cycle, and the SRDF/A restore process begins and ends. At the end of the restore, when all tracks are marked write pending to the R2 devices, the Consistent Deactivation is complete.
SRDF/A Consistent Deactivation: Limitations

1. The transition is not immediate
It takes two cycle switches from request to completion
2. There will be some performance loss during the transition

To host writes done during the transition Similar to host writes done during re-sync copy
3. Cache requirements
During transition, all new writes are written to an existing inactive (receive) SRDF/A cycle at the R2 side For consistency, the data must be committed (restored) to the R2 all at once This may require an inactive (receive) cycle at the R2 which is ~2x the size as normal
4. Both Symmetrix systems must be running Enginuity 5x71

The limitations to this function are described above. Please note the additional cache which may be required at the R2 side. Because Enginuity changes are required at both R1 and R2 side Symmetrix systems, this is a 5x71 only feature. This function fails if attempted on a 5x71 to 5670 SRDF connected Symmetrix pair.
SRDF/A: Loss of Link

Temporary Loss of Link
If all links are lost for a period less than 10 seconds, SRDF/A remains in an Active state. Data continues to accumulate in cache and consequently, the cycle time is elongated. However, R2 stays consistent, and the relationship between R1-R2 is not suspended.
Permanent Loss of Link

In this event, all data in the Capture and Transmit cycles on the R1 side are changed from write-pending for the remote mirror to invalid on the remote mirror. Any new data on the R1s is also be marked as invalid on the remote mirror. Likewise, all data in the Receive cycle on the R2 side are changed to invalid on the remote mirror. The Apply cycle on the R2 side completes the commit to the local devices. The devices themselves are set to Not Ready on the link.
Data on the R2 devices is always consistent in SRDF/A, even with the loss of link. However, during resynchronization, data is temporarily inconsistent on the R2 until all the invalid tracks have been sent over to R2. For this reason, it is preferable to have a point-in-time copy on a BCV, for example, prior to starting the resynchronization process. Resynchronization can be initiated by issuing symrdf establish command. The logical connection between R1 and R2 can be lost under several conditions. Some of them are listed below: Network problems leading to loss of physical connection between source and target Symmetrix dropping the links due to link saturation User issued commands such as symrdf suspend/split
SRDF/A Reserve Capacity Transmit Idle

5772 and later versions of 5x71 support Transmit Idle After links fail
Data transmission from source to target will stop SRDF/A will remain active and the capture cycle will grow in cache Session will be suspended if cache fills up
After links are revived, cycle switching will continue Not supported with ESCON RAs Should be enabled on both Source and Target side Should be used if using Delta Set Extension (DSE - 5772)
SRDF/A Transmit Idle is a Reserve Capacity enhancement to EMCs SRDF/A feature that provides SRDF/A with the capability of dynamically and transparently extending the Capture, Transmit, and Receive phases of the SRDF/A cycle while masking the effects of an all SRDF links lost event. Without the SRDF/A Transmit Idle enhancement, an all SRDF links lost event would normally result in the abnormal termination of SRDF/A with either a CACA.20 or CACA.40 error. The SRDF/A Transmit Idle enhancement has been specifically designed to prevent this event from occurring. The versions of 5x71 which support Transmit Idle are
5671 Minimum release level 59.64 with Epack# 1017. 5771 Minimum release level of 92.99 with Epack# 1016 and
SRDF/A Reserve Capacity - Delta Set Extension

Allows offloading of SRDF/A delta sets from cache to specially configured device pools Delta Set Extension Pools Intended to make SRDF/A resilient to temporary increases in write workloads or link loss
Should be used in conjunction with Transmit Idle
SRDF/A Delta Set Extension (DSE) provides a mechanism for augmenting the cache-based Delta Set buffering mechanism of SRDF/A with a disk-based buffering ability. This extended Delta Set buffering ability may allow SRDF/A to ride through larger and/or longer SRDF/A throughput imbalances than would be possible with cache-based Delta Set buffering alone.
DSE Pools
Save Pools are designated as DSE pools at creation
Contains SAVE devices of a single emulation
CKD3390, CKD3380, FBA or AS400
Pools can be associated (shared) with multiple SRDF/A RDF groups Each SRDF/A session can be associated with zero or one DSE pool of each type (e.g. FBA, 3390)
Must have at least one DSE pool configured with a type that matches one of the device types in the SRDF/A group in order to activate DSE
Pools can optionally start automatically when SRDF/A is enabled

SRDF/A DSE Pools and Save devices are managed in the same way as TimeFinder/Snap pools A RDF group can have at most one pool of each emulation A single rdfa_dse pool can be associated with more than one RDF group, similar to snap pools shared by multiple snap sessions SRDF/A DSE Threshold sets the percentage of cache used for SRDF/A that will start offloading cache to disk DSE must be enabled on both the source and target arrays. Extension on only one side of a link would lead to failure of the SRDF/A recovery with SRDF/A dropping because the R2 side would fail to have enough cache to hold the large and extended Transmit cycle
How DSE Fits Into SRDF/A Data Flow

Traditional SRDF/A data flows continue to interact directly with the cache buffer
Cache Buffer
Separate DSE Task monitors Delta Set cache utilization and transfers Delta Set data between cache and disk Must be configured on R1 and R2 sides to be of value
and in all sessions tied together by MSC
DSE Task
Disk Buffer
SRDF/A has always buffered Delta Set data in cache. However a buffer full condition causes SRDF/A to drop. With SRDF/A DSE delta set data is offloaded to disk buffers (DSE Pools) by the DSE task. We will look at this in more details in the next few slides.
When can DSE help you?

DSE is not designed to solve any permanent and persistent problems:
Mis-configurations:
Unbalanced
cache or backend on target side is smaller/weaker than source side
Not enough cache Not enough bandwidth
Host writes consistently exceeding RDF link bandwidth Prolonged link outages
SRDF/A DSE solves abnormal and temporary problems

Unexpected host load Link bandwidth issues Temporary link loss (use with Transmit Idle)
Increase resilience of SRDF/A

SRDF/A DSE should be used with the Transmit Idle feature. Thus SRDF/A can ride through a temporary link loss, once the DSE threshold is reached, data will be paged out to the DSE Pools.
Recovering after Loss of Links

It is recommended to split a BCV copy of the R2 prior to starting resynchronization In the event of an extended loss of link, a large number of R2 invalid tracks can build up on the R1 side It is advisable to enable SRDF/A after the two sides are synchronized Resynchronization prior to enabling SRDF/A can be performed by:
Setting SRDF mode to Adaptive Copy Write Pending Setting SRDF mode to Synchronous
As noted earlier, during resynchronization R2 does not have consistent data. A BCV copy of the consistent R2 data prior to resynchronization can safeguard against unexpected failures during the resynchronization process. When the link is resumed, if there are a large number of invalid tracks owed by the R1 to its R2, it is recommended that SRDF/A not be enabled right away. Enabling SRDF/A right after link resumption causes a surge of traffic on the link due to (a) shipping of accumulated invalid tracks, and (b) the new data added to the SRDF/A cycles. This would lead to SRDF/A consuming more cache and reaching the System Write Pending limit. If this happens, SRDF/A would drop again. Like with SRDF/A, resynchronization should be performed during periods of relatively low production activity. Resynchronization in Adaptive Copy Write Pending mode minimizes the impact on the production host. New writes are buffered and these, along with the R2 invalids, are sent across the link. The time it takes to resynchronize is elongated. Resynchronization in Synchronous mode impacts the production host. New writes have to be sent preferentially across the link while the R2 invalids are also shipped. Switching to Synchronous is possible only if the distances and other factors permit. For instance, if the norm is to run in SRDF/S and toggle into SRDF/A for batch processing (due to higher bandwidth requirement). In this case, if a loss of links occurs during the batch processing, it might be possible to resynchronize in SRDF/S. In either case, R2 data is inconsistent until all the invalid tracks are sent over. Therefore, it is advisable to enable SRDF/A after the two sides are completely synchronized.
Recovery Example
DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : 2 : 0 : Inactive Target (R2) View MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
407 NR 000B NA 406 NR 000C NA
NA NA
NA A..X NA A..X
Partitioned Partitioned
In this example there is a workload on the devices in SRDF/A enabled state. A permanent loss of link place the devices in a Partitioned state. Production work continues on the R1 devices and the new writes arriving for the R1 devices are marked as invalid or owed to the R2.
Recovery Example (Cont.)

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : 2 : 0 : Inactive Target (R2) View MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
2431 NR 000B WD 2431 NR 000C WD
285 288
0 A..X 0 A..X
Suspended Suspended
When the links are active again, note that the pair state has moved to Suspended. Also note there are R2 Invalid Tracks on the R1 side AND R1 Invalid Tracks on the R2 side. In Synchronous SRDF, one would have this condition only if changes are made to both the R1s and the R2s. However, in SRDF/A data in Receive cycle during loss of link are marked as R1 invalids. The data in the Capture and Transmit cycles, and new writes after link loss, are marked as R2 invalid on the R1 side. This is one of the behavioral differences between SRDF/S and SRDF/A.
Recovery Example (Cont.)

DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status : 2 : 0 : Inactive
Source (R1) View -------------------------------ST Standard Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
3767 RW 000B WD 0 RW 000C WD
0 0
285 C.W. 0 C.W.
SyncInProg Synchronized
As per recommendation, we place the devices in SRDF Adaptive Copy Write Pending mode (C.W.) and wait for the pair states to become Synchronized. Prior to changing the mode to ACP_WP, we have to disable consistency protection via symrdf g vsrdfadg3 disable. When consistency is enabled, one cannot switch out of SRDF/A without first disabling it. Once the synchronization is complete we can then enable SRDF/A and enable consistency.
SRDF Session Recovery Tool symrecover

EMC SRDF session recovery utility is initiated by the symrecover command Runs in the background and monitors the state of SRDF/A or SRDF/S sessions If failure is detected, automatic recovery and restart is attempted based on symrecover options file (pre-configured) The symrecover command can be run from either the R1 or the R2 side
As long as all the devices making up the group being monitored are fully viewable from the host When concurrent RDF is used, this command must be run from the R1 side
Solutions Enabler 6.4 or higher must be installed
The symrecover command was available through EMC Services prior to SE 6.4. symrecover offers the ability to detect and automate restarting SRDF links in a safe and secure manner.
Failover/Failback with SRDF/A

If the primary site fails, data on R2 is consistent up to the last Apply cycle (N-2)
Partial data in the Receive cycle is discarded
SRDF failover procedure can then be executed and the workload can be started on the R2 devices
Consistency protection should be disabled prior to issuing symrdf failover without the force option
Failback procedure after the primary site has been restored is identical to Synchronous SRDF
After symrdf failback command completion, workload can be restarted on the R1 devices. SRDF/A can be enabled
Again, it is advisable to split off a BCV copy of the R2 prior to executing a failback operation. When workload is resumed on the R1 devices immediately after a failback, accumulated invalid tracks have to be synchronized from the R2 to the R1, and new writes must be shipped from the R1 to R2. If there is an interruption now, data on the R2 is not consistent. Even though SRDF/A can be enabled right after a failback, for reasons stated earlier, it should be enabled after the SRDF pairs entered the Synchronized state.
Example: Multiple Independent SRDF/A Groups

DMX8HP1[ksh] # symcfg list -ra all -sid 35|more Symmetrix ID: 000187910035
S Y M M E T R I X
R D F
D I R E C T O R S
Remote Ident Symb Num Slot Type Attr SymmID
Local RA Grp
Remote RA Grp Status
RF-1D
01D
49
RDF-R1
000187910156 000187910156 000187910156 000187910156 000187910156 000187910156 000187910156 000187910156 000187910156 000187910156
1 (00) 3 (02) 1 (00) 3 (02) 1 (00) 3 (02) 4 (03) 1 (00) 3 (02) 4 (03)
1 (00) Online 3 (02) 1 (00) Online 3 (02) 1 (00) Online 3 (02) 4 (03) 1 (00) Online 3 (02) 4 (03)
RF-2D
02D
50
RDF-R1
RF-15D
15D
63
15
RDF-R1
RF-16D
16D
64
16
RDF-R1
RA Group 3 uses all four RDF Directors 1D, 2D, 15D, and 16D. RA Group 4 uses two of the RDF Directors 15D, and 16D

(Cont.)
DMX8HP1[ksh] # symcfg list -rdfg all
Symmetrix ID : 000187910035
S Y M M E T R I X
R D F
G R O U P S
Local ------------LL RA-Grp (sec) RA-Grp
Remote --------------------
Group ----------------------Flags Dir Cfg
RDFA Info ----------------Flags CSR -----IS -IS XAS XAS Cycle time ----30 30 30 30 Pri --33 33 33 33
SymmID
Name
LPD
------------1 ( 0) 2 ( 1) 3 ( 2) 4 ( 3) 10 10 10 10
-------------------1 ( 0) 000187910156 2 ( 1) 000187910156 3 ( 2) 000187910156 4 ( 3) 000187910156
----------------------S RDFDVGR00 S RDFDVGR01 D vsrdfag3 D vsrdfag4 .X. .X. XX. XX. F-S F-S F-S F-S
Both the RA Groups 3 and 4 are in SRDF/A mode (XAS), as shown in the output above.

(Cont.)
DMX8HP1[ksh] # symrdf -g vsrdfadg3 query -rdfa RDFA Cycle Number RDFA Session Status : 2972 : Active
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
0 0
0 A..X 0 A..X
Consistent Consistent
The devices in RA Group 3 are consistent, independently of the devices in RA Group 4 (shown in the next slide).

(Cont.)
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000D RW 000E RW
0 0
0 RW 000D WD 0 RW 000E WD
0 0
0 A..X 0 A..X
Devices in RA Group 4 are in SRDF/A Active state and are consistent. Note that RA Group 4 has a different Cycle Number (62), than RA Group 3 (2972).

(Cont.)
DMX8HP1[ksh] # symrdf -g vsrdfadg4 query -rdfa RDFA Cycle Number RDFA Session Status : 0 : Inactive
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000D RW 000E RW
0 0
107 NR 000D NA 80 NR 000E NA
NA NA
NA A..X NA A..X
Loss of links on directors 15D, and 16D causes SRDF/A to drop for RA Group 4.

(Cont.)
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002
000B RW 000C RW
0 0
0 0
0 A..X 0 A..X
However, RA Group 3 still has 2 links available (1D, and 2D), and continues to be in SRDF/A Active state and the device pairs are Consistent.
What is SRDF/A Multi Session Consistency?

Manage multiple SRDF/A sessions logically as if they were a single session
RDF Daemon for Open System support Sessions can be within or across Symmetrix arrays Ensures a complete, restartable point-in-time copy on the remote side
Delta Set
Delta Set
Delta Set
Since Enginuity 5671, consistency protection for SRDF/Asynchronous devices is provided using Multi Session Consistency (MSC). If one or more source (R1) devices in an SRDF/A MSC enabled RDF consistency group cannot propagate data to their corresponding target (R2) devices. The MSC process suspends data propagation from all R1 devices in the consistency group, halting all data flow to the R2 targets SRDF/A with MSC supported by an RDF process daemon that performs cycle-switching and cache recovery operations across all SRDF/A sessions in the group. This ensures that a consistent R2 data copy of the database exists at the point-in-time any interruption occurs. A composite group must be created using the RDF consistency protection option (-rdf_consistency) and must be enabled using the symcg enable command before the RDF daemon begins monitoring and managing the MSC consistency group.
What is SRDF/A Multi Session Consistency? (Cont.)

RDF Daemon coordinates cycle switching of the SRDF/A MSC group sessions as a single entity
Responsible for detecting failure conditions that would cause data on the R2 side to become inconsistent When a failure condition is detected, the cycle switching for all SRDF/A sessions in the group are stopped in a manner that leaves the R2 side with a consistent data image
Delta Set
Delta Set
Delta Set
The RDF process daemon maintains consistency for enabled composite groups across multiple Symmetrix arrays for SRDF/A with MSC. For the MSC option (-rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host performing management operations must run an instance of the RDF daemon (storrdfd). Each host running storrdfd must also run an instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel application syscalls. Optionally, if the Group Naming Services (GNS) daemon is also running, it communicates the composite group definitions back to the RDF daemon. If the GNS daemon is not running, the composite group must be defined on each host individually.
How Does MSC Cycle Switch Work?

In single session, SRDF/A cycle switch occurs when the Transmit cycle on the R1 side AND the Apply cycle on the R2 side are both empty. The switch is controlled by Enginuity In MSC, the Transmit cycles on the R1 side of all participating sessions must be empty, and also the corresponding Apply cycles on the R2 side. The switch is coordinated and controlled by the RDF Daemon All host writes are held for the duration of the cycle switch. This ensures dependent write consistency
If one or more sessions in MSC complete their Transmit and Apply cycles ahead of other sessions, they have to wait for all sessions to complete, prior to a cycle switch.
MSC Cycle Switch

Symmetrix 1
Session 1
Symmetrix 3
Session 1
Tag a
C(1) T(1)
(N)
Tag a-1
R(1) A(1)
(N-1)
Tag a-1
(N-1)
Tag a-2
(N-2)
MSC/CG
Symmetrix 2
Session 2
Tag a
RDF Links
Symmetrix 4
Session 2
Tag a-1
RDF Daemon Host

Tag a-1
C(2) T(2)
(M)
R(2) (M-1) A(2) (M-2)

(M-1)
Tag a-2
In this illustration we have two SRDF/A sessions, 1 and 2. Each have their own cycle numbers N and M. When they are placed in MSC, the RDF Daemon assigns a Tag number to the capture cycle. This is so the cycle numbers themselves do not have to be synchronized. The Tag number is incremented at every cycle switch. It is this Tag number that is compared for recovery purposes.
SRDF/A MSC Operations

Set SYMAPI_USE_RDFD = ENABLE in options configuration file Create a Composite Group (CG) with the -rdf_consistency option
Group definition is passed to the RDF Daemon as a candidate group If the Daemon is not already running, it is started automatically
Add all of the devices in the multiple SRDF/A sessions to the CG Put all CG devices into Async mode symrdf -cg <CGname> set mode async Enable CG devices for consistency protection symcg -cg <CGname> enable
The RDF Daemon is notified that the group should now be monitored Enable command must be done after the devices are put into Async mode
When the devices become RW on the link, the RDF Daemon:

Starts performing cycle switching Actively monitors the health of the group to maintain R2 data consistency
There are three ways the RDF daemon can be started. If the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Note: Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the information stored in it to establish contact with your Symmetrix arrays. Alternatively, the daemon can be started manually via the stordaemon command line utility as follows: - stordaemon start storrdfd [-wait Seconds] Note: The stordaemon command requires a path of /usr/storapi/storbin. By default, the stordaemon command waits 30 seconds to verify that the daemon is running. To override this, use the -wait option. Additionally, the daemon can be set to start automatically every time the local host is booted using the following command line: - stordaemon install storrdfd autostart. Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load. If the daemon is stopped for some reason, it can optionally be restarted automatically by an internal Solutions Enabler watchdog mechanism. A combination of the watchdog mechanism and the auto-start option described above can be used to ensure that the daemon is always running. To stop the RDF daemon, use the following command: - stordaemon shutdown storrdfd|all [-wait Seconds] Applying the all option stops all of the daemons currently running, such as storapid, storgnsd, and storrdfd.
RDF Daemon for MSC
The RDF process daemon maintains consistency for enabled composite groups across multiple Symmetrix arrays for SRDF/A with MSC.(Multi Session Control) For the MSC option (rdf_consistency) to work in an RDF consistency-enabled environment, each locally-attached host performing management operations must run an instance of the RDF daemon (storrdfd). Each host must also be running an instance of the base daemon (storapid), which coordinates all Symmetrix locks and parallel application syscalls. Additional data about the current state of a composite group is communicated to the RDF daemon via files written to the Symmetrix file system. MSC requires that the RDF daemon exist and every attempt is made to start or restart the daemon to perform cycle switching for SRDF/A. Failure to switch SRDF/A cycles may cause all SRDF/A sessions to be dropped due to a full cache slot. If SRDF/A sessions have been dropped, the SYMAPI and RDF daemon logic determines whether to commit or discard the data accumulated in cache memory. For redundant consistency protection of RDF composite groups, multiple instances of the RDF daemon can be running at the same time on separate hosts. Each host must have a common view of the composite group being monitored. All redundant daemons run simultaneously, monitoring and switching independently of each other. If one of the redundant daemons fails, the other existing daemon(s) completes the task.
Managing RDF Daemon

Modify the SYMAPI options file
SYMAPI_USE_RDFD=ENABLE
Start Daemon
stordaemon start storrdfd
Stop Daemon
stordaemon shutdown storrdfd
Prior to starting storrdfd, ensure that your default SYMAPI configuration database is up-to-date, since storrdfd uses the stored information to establish contact with your Symmetrix arrays. There are three ways the RDF daemon can be started. First, if the RDF daemon is enabled, the daemon is started automatically by the Solutions Enabler libraries the first time they attempt to connect with it, which can cause a slight delay in performance on that initial connection while the daemon starts and builds its cache. Second, the daemon can be started manually via the stordaemon command line utility as follows: stordaemon start storrdfd [-wait Seconds] By default, the stordaemon command waits 30 seconds to verify the daemon is running. To override this, use the -wait option. Third, the daemon can be set to start automatically every time the local host is booted using the following command line: stordaemon install storrdfd -autostart Pre-starting the daemon, either manually or via the automatic option, is useful because the daemon may take a while to initially construct its cache - depending on the number of groups and Symmetrix arrays it has to load.
MSC: Example
DMX8HP1[ksh] # symcg show vsmscdg34|more RDF Consistency Enabled : Yes
Number of RDF (RA) Groups Number of STD Devices Number of Symmetrix Units (1): Number of RDF (RA) Groups (2):
: :
2 4
---------------------------------------------------------------------------Sym LdevName PdevName Dev Device Config Sts Flags CSR Cap (MB)
---------------------------------------------------------------------------DEV001 DEV002 /dev/rdsk/c14t1d3 /dev/rdsk/c14t1d4 000B RDF1+Mir 000C RDF1+Mir RW RW XAM XAM 449 449
---------------------------------------------------------------------------Sym LdevName PdevName Dev Device Config Sts Flags CSR Cap (MB)
---------------------------------------------------------------------------DEV003 DEV004
/dev/rdsk/c14t1d5 /dev/rdsk/c14t1d6
000D RDF1+Mir 000E RDF1+Mir
RW RW
XAM XAM
449 449
We wish to control RA Groups 3 and 4 as one entity. A composite group has been created and enabled using the following commands: symdg dg2cg vsrdfadg3 vsmscdg34 rdf_consistency symdg dg2cg vsrdfadg4 vsmscdg34 rdf_consistency rename symcg cg vsmscdg34 enable
Note the CSR flags, indicating that Multi-session Consistency has been enabled for these two RA Groups.
Legend: RDFA Flags: C(onsistency) : X = Enabled, . = Disabled, - = N/A
(RDFA) S(tatus) : A = Active, I = Inactive, - = N/A R(DFA Mode) : S = Single-session mode, M = MSC mode, - = N/A
MSC: Loss of Links for One RA Group

DMX8HP1[ksh] # symrdf -cg vsmscdg34 query -rdfa|more MSC Session Status Consistency State : Inactive : N/A
Source (R1) View -------------------------------ST Standard Logical Device Sym Dev A T E R1 Inv R2 Inv
Target (R2) View
MODES
------------------------- ----- -----------LI N K S Dev ST A T E R1 Inv Tracks R2 Inv Tracks MDAC RDF Pair STATE
Tracks Tracks
-------------------------------- -- ----------------------- ----- -----------DEV001 DEV002 000B RW 000C RW 0 0 76 NR 000B WD 61 NR 000C WD 0 0 0 A..X 0 A..X Suspended Suspended
DEV003 DEV004
000D RW 000E RW
0 0
79 NR 000D NA 63 NR 000E NA
NA NA
NA A..X NA A..X
Similar to the previous example (independent), we have a total loss of links for RA Group 4. The devices in this group go to a Partitioned state as before. However, note that the devices in RA Group 3 have been placed in Suspended state. Loss of links for one RA Group trips the Multi-session Consistency Group. This prevents the propagation of data for the other RA Groups in the MSC-CG, thus preserving a consistent image of data on ALL R2s at the time of link loss. As stated earlier, the storrdfd daemon is responsible for coordinating cycle switches between the RA groups, to trip the MSC-CG and perform any recovery/cache cleanup that might be necessary when the links are resumed. Recovering from this state can be accomplished as usual: symrdf cg vsmscdg34 establish Once the invalid tracks are marked, merged, and synchronized, MSC protection is automatically reinstated. i.e User does not have to issue symcg cg vsmscdg34 enable again.
MSC Recovery and Cleanup

There are three possible recovery scenarios in case of a failure in MSC:
All Receive cycles are marked as complete. In this case, the Receive cycles are committed i.e. promoted to Apply Some Receive cycles are marked as complete and others are marked as incomplete. In this case, ALL Receive cycles are discarded Some Receive cycles have been promoted to Apply, where as some of them have not. In this case, the promoted Receive cycles and those not yet promoted are committed
The first scenario is easy to understand. In this instance, the Receive cycles contain the most recent and consistent data. The second situation arises if there is a failure when some Receive cycles are complete while the others are in transit. In this case, clearly it is only the Apply cycles of all sessions that contain the consistent data. Therefore, ALL Receive cycles are discarded. To understand the third scenario, keep in mind the following: For a cycle switch to be initiated, ALL Transmits must be empty and all Applys must be empty. This means that the failure has occurred DURING the cycle switch process in this case. Receive can only be promoted to Apply on a cycle switch. In MSC, cycle switch is sent to all sessions at once. So each of them is in the process of executing a cycle switch. For example, failure occurred prior to promoting some Receives to Apply. Therefore, the Receives not yet promoted should be committed along with those that have already been promoted.
MSC Recovery and Cleanup

Recovery is automatically performed by the RDF Daemon on the R1 side, if the link to the R2 side is available If the link is unavailable (total site failure on the R1 side), then invocation of any SRDF command, such as symrdf failover or split, from the R2 side performs the automatic cleanup and recovery
SRDF/A Configuration Parameters

2 SRDF/A Symmetrix Array-wide parameters
Maximum SRDF/A Cache Usage Maximum Host Throttle Time
2 RDF Group (aka SRDF/A Session) level settings

Minimum Cycle Time Session Priority
Settable via the SYMCLI symconfigure command

Set Symmetrix Metrics for Symmetrix level attributes
set symmetrix rdfa_cache_percent = 94; set symmetrix rdfa_host_throttle_time = 0;
Set RDF Group Metrics for Group level attributes

set rdf group 3, session_priority = 33; set rdf group 3, minimum_cycle_time = 30;
SRDF/A System Configuration Parameters

rdfa_cache_percent
Defaults to 94, with a range of valid values from 0 to 100 percent It is the percentage of the Max# of System Write Pending Slots available to SRDF/A. The purpose is to ensure that other applications can utilize some of the WP limit When SRDF/A hits its WP cache limit it will be forced to drop SRDF/A sessions to free up cache Setting it lower reserves WP limit for non-SRDF/A cache usage. Setting it higher, allows SRDF/A to potentially use more of the cache WP limit, potentially creating performance problems for other applications
rdfa_host_throttle_time
Defaults to 0, with a range of valid values from 0 to 65535 If >0, this value overrides the rdfa_cache_percent and session_priority settings When the System WP Limit is reached, throttling will delay a write from the host until a cache slot becomes free The value is the number of seconds to throttle host writes before dropping SRDF/A sessions. A value of 65535 means wait forever
Each Symmetrix has an array-wide Max # of System Write Pending Slots limit (generally calculated as 80% of available cache slots). The purpose of this limit is to ensure that cache is not filled with Write Pending (WP) tracks, potentially preventing fast writes from hosts, because there is no place to put the I/O in cache. SRDF/A creates WP tracks as part of each cycle.
SRDF/A Group/Session Configuration Parameters

SRDF/A group minimum_cycle_time
Defaults to 30, with a potential range of 5 to 59 seconds
SRDF/A group session_priority

Defaults to 33, with a range of valid values from 1 to 64. The number ranks the SRDF/A session relative to other sessions to determine the order for dropping sessions should the cache WP limit be reached When SRDF/A needs to drop sessions when the cache WP limit is reached. The sessions are dropped, starting with priority values of 64. Those with a setting of 1 are last to be dropped
Monitoring SRDF/A
Using the symstat command options
symstat type cycle reptype rdfa rdfg all symstat type cache reptype rdfa rdfg all symstat type request reptype rdfa rdfg all
Using the symevent command

symevent list -error
Examples are displayed in the next several slides.
Monitoring SRDF/A
DMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5 SRDF/A Session Cache Summary Information Symmetrix Id Timestamp System Write Pending Limit : 000187910035 : 15:17:23 : 110000 (3.36 GB) (3.16 GB)
Cache Slots available for all SRDF/A sessions : 103400 Total Local Write Pending Count Total System Write Pending Count Session RA Device Group Name ----------------RaGrpNum_01 RaGrpNum_02 RaGrpNum_03 RaGrpNum_05 RaGrpNum_04 Total Slots GB
: 30690 : 57066
----------------------------Type Number Priority Status ---- ------ -------- -------RDF1 RDF1 RDF1 0 1 2 4 3 33 Inactive 33 Inactive 33 Active 33 Active 33 Active
Cache Slots In Use ----------0 0 42 56921 44 ----------57007 1.74
%Available Cache Used ---------0.0 0.0 0.0 55.0 0.0 ---------55.1
Grp --1 2 3 5 4
Note that the Cache Slots available for all SRDF/A sessions is 94% of the System Write Pending Limit (3.16/3.6). In this example, RA Group 5 is utilizing 55% of the available cache. All SRDF/A sessions have the default Priority value of 33.
Monitoring SRDF/A (Cont.)

DMX8HP1[ksh] # symstat -type cycle -reptype rdfa -rdfg all -i 5 SRDF/A Session Cycle Summary Information Symmetrix Id Timestamp : 000187910035 : 15:18:57
Session -----------------RA Device Group Name Grp ----------------- --RaGrpNum_01 RaGrpNum_02 RaGrpNum_03 RaGrpNum_05 RaGrpNum_04 1 2 3 5 4 Active Type Number Cycle# ---- ------ -----RDF1 RDF1 RDF1 0 1 2 4 3 0 0 286 34 286
Cycle Time
(sec)
Cycle Size ---------------
------------------Last Min Avg Last Switch --- --- ---- -----30 30 30 0 0 30 29 30 0 0 30 29 30 27 3 27
Active Inactive ------ -------0 0 48 6935 51 0 0 0 32589 0
Legend for the Attribute of Cycle Size: RDF1: Active = Capture RDF2: Active = Receive
Inactive = Transmit Inactive = Apply

This output was captured from the R1 side. As displayed, the Minimum cycle time for each of the SRDF/A session is at the default of 30 seconds. The Active Cycle is the Capture and the Inactive is the Transmit, as this output is from the R1 (source) perspective.
When Max Cache is used up

DMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5 SRDF/A Session Cache Summary Information Symmetrix Id Timestamp System Write Pending Limit : 000187910035 : 15:49:28 : 110000 (3.36 GB) (1.68 GB)
Cache Slots available for all SRDF/A sessions : 55000 Total Local Write Pending Count Total System Write Pending Count Session RA Device Group Name ----------------RaGrpNum_03 RaGrpNum_05 RaGrpNum_04 Grp --3 5 4 ----------------------------Type Number Priority Status ---- ------ -------- -------RDF1 RDF1 RDF1 2 4 3 33 Active 33 Active 33 Active : 37725 : 55153
Cache Slots In Use ----------37 54978 38
%Available Cache Used ---------0.1 100.0 0.1
-Next iteration of the symstat commandRaGrpNum_03 RaGrpNum_05 RaGrpNum_04

3 5 4
RDF1 RDF1 RDF1
2 4 3
33 Active 33 Inactive 33 Active
47 44308 50
0.1 80.6 0.1

For this example, the maximum available cache slots for SRDF/A has been reduced to 50% (1.68/3.36). Now, cache utilization reaches a 100% of available cache, and RDFG 5 is dropped (Inactive).
When the Session is dropped

DMX800SUN1[ksh] # symrdf -g vsrdfadg query -rdfa RDFA Session Number RDFA Cycle Number RDFA Session Status : 4 : 0 : Inactive
Target (R2) View
MODES
-------------------------------- -- ------------------------ ----- ------------
DEV001 DEV002 DEV003 DEV004 DEV005
0041 RW 0042 RW 0043 RW 0044 RW 0045 RW
0 0 0 0 0
0 NR 0041 WD 14340 NR 0042 WD 14340 NR 0043 WD 14340 NR 0044 WD 14340 NR 0045 WD
0 14340 14340 14340 14214
0 A... 0 A... 0 A... 0 A... 0 A...
Suspended Suspended Suspended Suspended Suspended
As seen earlier, the Capture and Transmit data on the R1 side are marked as R2 invalids. The Receive data on the R2 side is marked as R1 invalid. The pair state is suspended.
Monitoring SRDF/A symevent

DMX8HP1[ksh] # symevent list -error Symmetrix ID: 000187910035 Time Zone : EDT
Detection time
Dir
Src
Category
Severity
Error Num
------------------------ ------ ---- ------------ ------------ ---------Fri Aug 19 11:55:24 2005 RF-16D Symm RDF Error 0x004e
SRDF/A Session dropped, no RDF links operational.
Fri Aug 19 15:39:15 2005 RF-15D Symm RDF
Error
0x004e
SRDF/A Session dropped, no RDF links operational.
Fri Aug 26 16:29:48 2005 RF-2D
Symm RDF
Error
0x004a
SRDF/A Session dropped, write pending limit reached. Host throttling disabled.
The dropping of the SRDF/A session can also be displayed via the symevent command.
Giving Preference to Sessions via Priority Setting

DMX8HP1[ksh] # symstat -type cache -reptype rdfa -rdfg all -i 5 SRDF/A Session Cache Summary Information Symmetrix Id System Write Pending Limit : 000187910035 : 110000 (3.36 GB) (2.69 GB)
Cache Slots available for all SRDF/A sessions : 88000 Session RA Device Group Name ----------------RaGrpNum_03 RaGrpNum_05 RaGrpNum_04 Total Slots -Next iteration of the symstat commandRaGrpNum_03 RaGrpNum_05 RaGrpNum_04 3 5 4 RDF1 RDF1 RDF1 2 4 3 32 Active 31 Active 33 Inactive Grp --3 5 4 ----------------------------Type Number Priority Status ---- ------ -------- -------RDF1 RDF1 RDF1 2 4 3 32 Active 31 Active 33 Active
Cache Slots In Use ----------6664 61362 22790 ----------90816
%Available Cache Used ---------7.6 69.7 25.9 ---------103.2
9058 48098 18010
10.3 54.7 20.5
In this instance, the Cache available for SRDF/A has been set to 80% (2.69/3.36). When 100% (103.2) of this is used, we see that the session with the least priority is dropped first RA Group 4 with a priority of 33.
Factors that Affect RPO
RPO
SRDF Bandwidth
Workload
Symmetrix Cache
There are three factors that affects RPO of an SRDF/A implementation: SRDF bandwidth, Symmetrix Cache, and Workload. During a SRDF/A cycle, new changes are captured in the local Symmetrix cache before being sent via the SRDF link to the remote Symmetrix. The Symmetrix should have enough cache to accommodate these changes occurring before the cycle switch time has elapsed. At the same time, there should also be a sufficient bandwidth for SRDF link to push these changes to the remote site. If there is not enough cache or bandwidth, SRDF/A may not be able to maintain the RPO at twice the cycle time. The process to determine these three factors should involve EMC personnel and is essential to a successful SRDF/A implementation.
Industry Bandwidth Estimates

Type of Link T1 T3 OC-3 Price / Month 600 4,500 10,000 80,000
Dollars
..
OC-48
Industry Estimates of Bandwidth Pricing.
Example of Bandwidth, Cache, and Workload

Interval Length is 15 minutes Peak Workload is for 6 intervals (or 90 minutes) Non-Peak Workload results in 30 second SRDF/A Cycles
Non-Peak Workload
M y p rS c n b te e e o d
35 30 25 20 15 10 5 0
23 :4 5 :0 0 7: 45 :0 0 8: 45 :0 0 9: 45 :0 0 10 :4 5 :0 0 11 :4 5 :0 0 12 :4 5 :0 0 13 :4 5 :0 0 14 :4 5 :0 0 15 :4 5 :0 0 16 :4 5 :0 0 17 :4 4 :5 9 18 :4 5 :0 0 19 :4 5 :0 0 20 :4 5 :0 0 21 :4 5 :0 0 22 :4 5 :0 0
BANDWIDTH USAGE
Peak Workload
TIME INTERVAL
COM RE P SSE BANDWIDTH D BANDWIDTH LIM IT
In this example, there is 1 OC-3 Link (15.5 MB/s) with 2:1 Compression between the local and remote site. The workload produces 6 peak writes above the bandwidth.
Cache Requirements for One OC-3 Link

Cache Requirements are significant in order to cover the peak times
Base Cache covers SRDF/A during normal operations (below b/w) Peak Cache is the significant contributing value
SRDF/A Cache Requirements
Cache (GB) 150.00 100.00 50.00 12:45:00 14:00:00 11:30:00 15:14:59 10:14:59 16:30:00 17:44:59 19:00:00 20:15:00 21:30:00 22:45:00 7:45:00 9:00:00 TIME 0.00
Time PEAK CACHE (GB/interval) BASE CACHE (GB/30 sec cycle)
The single OC-3 bandwidth is not sufficient to push the data surge during peak writes. As a result, the data is accumulating the cache on the local Symmetrix. This drives the cache requirement up to around 80GB.
The RPO Impact with One OC-3 link

Providing you can configure enough cache, SRDF/A works perfectly fine with less than Synchronous bandwidth This, as depicted below, directly affects the RPO
SRDF/A Recovery Point Objective (RPO) R PO (Seconds) 1000.00 800.00 600.00 400.00 200.00 0.00
TIM E
7:45:00
9:00:00
10:14:59
11:30:00
12:45:00
14:00:00
15:14:59
16:30:00
17:44:59
19:00:00
20:15:00
21:30:00
Time of day RPO TIME (SECS)
If the additional cache was added, SRDF/A RPO would be almost 800 seconds with a single OC-3 link. Even though the SRDF/A cycle may have been set at 30 seconds, SRDF/A would not be able to maintain it.
22:45:00
The Effect of Adding an Extra OC-3 Link

Increasing the bandwidth to 31MB/s reduces the cache requirements
BANDWIDTH USAGE Mbyte per Second

35 30 25 20 15 10 5 0
This example explores the option if the second OC-3 link was added to the configuration. The bandwidth would be 31 MB/sec. The workload would not produce any peak writes that are above the bandwidth.
23 :4 5: 0 7: 0 45 :0 8: 0 45 :0 9: 0 45 : 10 00 :4 5: 11 00 :4 5: 12 00 :4 5: 13 00 :4 5: 14 00 :4 5: 15 00 :4 5: 16 00 :4 5: 17 00 :4 4: 18 59 :4 5: 19 00 :4 5: 20 00 :4 5: 21 00 :4 5: 22 00 :4 5: 00
TIME INTERVAL
COMPRESSE BANDWIDTH D BANDWIDTH LIMIT
Matching Cache Requirements for 2 ATM OC-3

Cache Requirements are significant in order to cover the peak times
Base Cache adequately covers SRDF/A during normal operations Peak Cache is no longer needed
Cache (GB) 2.50 2.00 1.50 1.00 0.50 0.00
:0 0 :5 9 :0 0 :5 9 :0 0 :5 9 TI M :0 0 :0 0 :0 0 :0 0 :0 0 :0 0 :3 0 21 22 :4 5 :1 4 :3 0 :1 4 :3 0 :4 4 7: 45 9: 00 :0 0 :0 0 12 10 11 15 16 17 14 19 20 :1 5 :4 5 :0 0 E
This shows that the implementation does not need additional cache to accommodate peak writes. We have eliminated the 80 GB peak cache with an extra OC-3 link.
The RPO Impact with 2 OC-3 Links

We have a reduced requirement for cache and no requirement for peak cache This, as depicted below, allows us to continuously maintain a 60 second RPO
SRDF/A Recovery Point Objective (RPO) 70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00
TI M 7: E 45 :0 9: 0 00 : 10 00 :1 4: 11 59 :3 0 12 :00 :4 5 14 :00 :0 0 15 :00 :1 4 16 :59 :3 0 17 :00 :4 4: 19 59 :0 0 20 :00 :1 5 21 :00 :3 0 22 :00 :4 5: 00
RPO (Seconds)

Therefore, SRDF/A would be able to maintain the 30-second cycle resulting in the RPO of 60 seconds.
The Bandwidth Change

If our sample customer is flexible on the RPO, but is only willing to provide limited bandwidth, say 5 T3 (4.5 MB/s each), then we can see the results We can still observe peak requirements. These directly affect the RPO and cache
BANDWIDTH USAGE
35 30 25 20 15 10 5 0
23 :4 5: 00 7: 45 :0 8: 0 45 :0 9: 0 45 : 10 00 :4 5: 11 00 :4 5: 12 00 :4 5: 13 00 :4 5: 14 00 :4 5: 15 00 :4 5: 16 00 :4 5: 17 00 :4 4: 18 59 :4 5: 19 00 :4 5: 20 00 :4 5: 21 00 :4 5: 22 00 :4 5: 00
Mbyte per Second
TIME INTERVAL
COMPRESSED BANDWIDTH BANDWIDTH LIMIT
We have examined the implementation of SRDF/A with one and two OC-3 links. Now, lets look at an alternative in terms of link bandwidth with 5 T3 links.
Cache Requirements for Five T3 Links

Cache Requirements are still high in order to cover the peak times
Base Cache adequately covers SRDF/A during normal operations Again, Peak Cache is the major factor
Cache (GB) 40.00 30.00 20.00 10.00 0.00
:5 9 :0 0 :0 0 :0 0 :5 9 :0 0 :5 9 :0 0 :0 0 TI M :0 0 :1 4 :3 0 :4 5 :0 0 :0 0 :0 0 :3 0 21 :1 4 :3 0 :4 4 7: 45 9: 00 10 11 12 14 :0 0 :1 5 15 16 17 19 20 22 :4 5 :0 0 E
With five T-3 links, SRDF/A would need 30 GB of cache to accommodate the sample workload.
RPO Impact with Five T-3 Links

SRDF/A Recovery Point Objective (RPO) 350.00 300.00 250.00 200.00 150.00 100.00 50.00 0.00 TIME 7:45:00 9:00:00 10:14:59 11:30:00 12:45:00 14:00:00 15:14:59 16:30:00 17:44:59 19:00:00 20:15:00 21:30:00 22:45:00 RPO (Seconds)
This implementation yields the RPO of 300 seconds. The balance of bandwidth and cache to achieve a desirable RPO is specific to a workload pattern as shown by the three examples. A methodical and thorough planning is crucial to a successful SRDF/A implementation.
Concurrent SRDF with SRDF/A

One source, two targets One target sync
short distance, zero data lag
One target async

longer distance, variable data lag, no performance impact
SRDF: Remote Re-synch

SRDF/S (sync) Host SRDF/A (async)
Continued protection upon source failure
In an SRDF configuration, a single source (R1) device can concurrently be remotely mirrored to two target (R2) devices. This allows you to have two identical remote copies available at any point in time. It is valuable for duplicate restarts or disaster recovery, or for increased flexibility in data mobility and migrating applications. Concurrent RDF technology can use two different RA adapters (RAs, RAFs, or RFs) in the interface link to achieve the connection between the R1 device and its two concurrent R2 mirrors. Each of the two concurrent mirrors must belong to a different RDF (RA) group Enginuity 5671 supports Concurrent SRDF with SRDF/A. Only one of the SRDF mirrors is allowed to be in Asynchronous mode, regardless if SRDF/A is active or not.
SRDF/A Restrictions
Each SRDF/A group must be unidirectional Concurrent RDF with both RDF mirrors in SRDF/A mode is not allowed A personality swap with SRDF/A running is not allowed Cannot activate empty SRDF/A group SRDF/A-capable devices that are enabled for consistency group protection must be disabled before attempting to change the mode from asynchronous Symmetrix RDF Automated Replication (SRDF/AR) control operations are not supported for SRDF/A-capable devices running in asynchronous mode
The list is not conclusive. Please refer to EMC Solutions Enabler Symmetrix SRDF Family CLI Version 6.0 PRODUCT GUIDE P/N 300-000-877 for more information.
Module Summary
Key points covered in this module:
Technical requirements of a successful SRDF/A implementation Supported SRDF/A hardware platforms Transmit Idle and DSE (Delta Set Extension) Factors that affects RPO (Recovery Point Objective) in an SRDF/A implementation Cycles within SRDF/A operations SRDF/A Consistent Deactivation Multi Session Consistency Configuration parameters that affect SRDF/A behavior
SRDF/AR (Automated Replication)

Upon completion of this module, you will be able to: Describe the benefits of integrating EMCs SRDF and TimeFinder applications List business needs and requirements using an SRDF single or multi-Hop Symmetrix configuration Describe a Clustered SRDF/AR environment List SRDF/AR configuration requirements Describe the purpose of the symreplicate options file List the replication cycle steps for a single and multi-hop environment
2007 EMC Corporation. All rights reserved. SRDF/AR (Automated Replication) - 1
SRDF/AR (Automated Replication) - 1
Symmetrix Remote Data Facility/Automated Replication
Allows business restart site to be any distance away from source Collaboration of SRDF and TimeFinder commands Minimizes network costs
R1 or STD R2
SRDF
R1/BCV Adaptive Copy
BCV
SRDF/AR allows users to automate the sequence of SRDF and TimeFinder mirror operations. The automated sequence, cycle, is performed on a user-defined interval called cycle time. The replication cycles automatically loop indefinitely or to the number of cycles specified by the users. Users perform all SRDF/AR operations, setup, start, stop, restart, and query, through the symreplicate command. Even though the SRDF link can be set to all SRDF operational mode, except Asynchronous, it is usually set to operate in Adaptive Copy mode due to the long distance between local and remote sites. This allows the users to save on network bandwidth thus minimizing the network costs without compromising the integrity of the data.
SRDF/AR Requirements Software requirements

Solutions Enabler 4.3.1 (or later, if using PowerPath for consistent split) Solutions Enabler 5.1 (or later, if using ECA for consistent split) Requires TimeFinder, SRDF, and SRDF/AR licenses
TimeFinder Consistency Options

Enginuity 5x66 or later for use with PowerPath 2.1 or later
Single server, single Symmetrix consistency
Enginuity 5567 for use with Enginuity Consistency Assist (ECA)

Multiple server, single Symmetrix consistency
Enginuity 5568, 5569 Patch Q203 for 5x67
Supported open systems hosts

Solaris, Windows NT/2000/2003, HP-UX, AIX
SRDF/Automated Replication (SRDF/AR) provides the ability to automate data copies across SRDF links, providing a restartable image of the data at the remote site in the event of a disaster at the production site. It combines both SRDF and TimeFinder to complete its operations. This slide lists the requirements for both SRDF and TimeFinder in an SRDF/AR environment. Please refer to the EMC Support Matrix for the latest information.
SRDF/AR Single-Hop Configuration

Uses 2 Symmetrix arrays Uses STD, R1-BCV, R2 and BCV device types SRDF Mode of Operation
Adaptive copy
Controlled data loss

Remote BCV can be used for disaster restart
R1 or STD
Host
R2
R1 BCV
BCV
Local
Remote
The copy path for a single-hop configuration is from the local R1/BCV pair (1) to the SRDF pair (2) to the remote BCV pair (3). The remotely associated BCV holds the DBMS restartable copy. The amount of data loss is a function of the replication cycle time (period of time between the start of one copy cycle and the start of another copy cycle). Copy cycle time is affected by distance, bandwidth, I/O update rate, and locality of reference for the updates. Update rate and locality of reference tend to equate to changed tracks. The maximum data loss would be one copy cycle, thus makes the RPO ~ One Cycle Time. Single Hop benefits include: The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target site, reducing required network bandwidth Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations
Single Hop Set-up: Create a Device Group

1. Create a device group: symdg create single type regular
100 STD 101 R2
2. Add standard device to device group: symld g single add dev 100 3. Associate R1/BCV device:
101 R1 BCV
102 BCV
symbcv g single associate dev 101 4. Associate remote BCV device: symbcv g single associate dev 102 rdf bcv
Local
Remote
Single Hop Set-up: Initialize Mirrors

1. Establish the STD and R1/BCV (Note1): symmir -g single est full This step also automatically suspends the SRDF link
100 STD 1 2 6 101 R1 BCV 3 4 101 R2 5
2. Split the STD and R1/BCV: symmir g single split -consistent 3. Resume SRDF linkNote1: symrdf g single resume -bcv 4. Establish the R2 and remote BCV (Note1): symmir g single est full rdf bcv 5. Split the R2 and remote BCV: symmir g single split full rdf bcv 6. Establish the STD and R1/ BCV: symmir -g single est
102 BCV
Local
Remote
There are 6 steps to prepare the states of all mirrors involved in SRDF/AR Single Hop setup prior to running symreplicate command. To begin a replication session in the single-hop configuration, the mirrors must be in the following states: Local BCV pair must be synchronized SRDF pair must be suspended Remote BCV pair must be split Note 1: Users must wait and check for the full synchronization after issuing the command before proceeding to the next step.
SRDF/AR Multi-Hop Configuration

Uses 3 Symmetrix arrays:
Local site, Hop1 bunker site, and Hop2 target site Uses R1, R2, R1-BCV, and BCV (optional) device types SRDF Mode of Operation
Local to Hop 1 Bunker: Synchronous Hop 1 Bunker to Hop 2 Target: Adaptive Copy
Zero data loss at Bunker site One cycle data loss at Target site
Host
R1
R2
BCV
R1 BCV
R2
Local
Bunker
Target
The copy path for a multi-hop configuration is from the local SRDF pair (1) to the remote BCV pair (2) to the remote SRDF pair (3) to the Target BCV (4). If your configuration does not include Target BCVs, the path stops at (3). Automated replication with the BCVs at Target is applicable if you want a zero data loss solution but cannot risk the loss of both the Local site and Bunker site at the same time. With this configuration, there are two possible disaster restart possibilities: If only the Local site is lost, the result is zero data loss at the Target restart site. If both the Local and Bunker site are lost, the result is a DBMS restartable copy at the Target restart site with controlled data loss. The amount of data loss is a function of the replicate copy cycle time between the Bunker site and the Target restart site. Multi-Hop benefits include: The ability to perform incremental resynchronization between the intermediate SRDF target site and the final SRDF target (Multi-Hop) site, reducing required network bandwidth Reduction in communication link cost and improved resynchronization time for long-distance SRDF implementations The ability to use the SRDF Multi-Hop site to provide disaster recovery testing, point-in-time backups, decision support operations, third-party software testing, and application upgrade testing or the testing of new applications
Multi-Hop Set-up: Create a Device Group

1. Create a device group: symdg create multi type r1
100 100 102
R1
R2
101
BCV
2. Add the R1 device to the device group: symld g multi add dev 100 3. Associate R1/BCV device: symbcv g multi associate dev 101 -rdf 4. Associate remote BCV device from the Target: symbcv g multi associate dev 102 rrdf
101
R1 BCV
Local Bunker
R2
Target
Multi-Hop Set-up: Initializing Mirrors

1. Establish the R2 and R1/BCV on Bunker (Note1 &
Note 2):
symmir -g multi est -rdf 2.

100 100 102
SRDF/S
Split the R2 and R1/BCV and establish SRDF link between Bunker and TargetNote1: symmir g multi split -remote consistent rdf Re-establish the R1/BCV and R2 on Bunker (Note 2): symmir g multi est rdf Establish the R2 and remote BCV on Target (Note1): symmir f <devfile> sid <Target> est
or
R1
R2 1 2 3
101
BCV 4 2 5
101
3. 4.
R1 BCV
Local Bunker
R2
Target
symmir g multi est rrbcv 5.
(Note 3)
Split the R2 and remote BCV on Target:
symmir f <defile> -sid <Target> split

or
symmir g multi est rrbcv
(Note 3):
There are 5 steps to prepare the states of all mirrors involved in SRDF/AR Multi Hop setup prior to running symreplicate command. To begin a replicate session in the multi-hop configuration, the mirrors must be in the following states: Local SRDF pair must be synchronized BCV pair on Bunker must be synchronized Remote SRDF link between Bunker and Target must be suspended BCV pair on Target must be split Note 1: Users must wait and check for the full synchronization after issuing the command before proceeding to the next step. Note 2: This automatically suspends the SRDF link between the R1/BCV on Bunker and R2 on the Target sites. Note 3: The BCV must associated with the device group with the rrdf option.
SRDF/AR
The continuous movement of dependent write consistent data to a remote site in an asynchronous mode
Standard to BCV/R1 => R2 to BCV Single Hop R1 => R2 to BCV/R1 => R2 [to BCV] Multi-Hop
Support for Composite Groups (CG)

CG can span multiple Symmetrix arrays A device group is limited to a single Symmetrix array Can use a CG and consistent split to create a DBMS-restartable copy of a database that spans multiple hosts and multiple Symmetrix arrays
The symreplicate Command

SYMCLI binary that integrates symmir and symrdf commands with the appropriate arguments and options depending on the configuration Used to propagate automated data copies
Incremental (changed tracks only) Behavior is based on pre-defined parameters (symreplicate options file)
When used with TimeFinder consistent split, ensures a remotely-associated BCV contains a DBMS restartable copy of data
Requires PowerPath or ECA
Used in both single-hop and multi-hop configurations Used to start, stop, restart, or query replication sessions Executed on local host
Symreplicate invokes a replicate session that generates automated, recurrent background copies of the standard data across SRDF links and cascading BCVs. You can start, stop, and restart the replicate session.
The symreplicate Options File

User created, named, and configurable text file that defines and controls replication behavior
SYMCLI_REPLICATE_HOP_TYPE=<RepType> SYMCLI_REPLICATE_CYCLE=<CycleTime> SYMCLI_REPLICATE_CYCLE_OVERFLOW=<OvfMethod> SYMCLI_REPLICATE_CYCLE_DELAY=<Delay> SYMCLI_REPLICATE_NUM_CYCLES=<NumCycles> SYMCLI_REPLICATE_USE_FINAL_BCV=<TRUE|FALSE> SYMCLI_REPLICATE_LOG_STEP=<TRUE|FALSE> SYMCLI_REPLICATE_GEN_TIME_LIMIT=<TimeLimit> SYMCLI_REPLICATE_GEN_SLEEP_TIME=<SleepTime> SYMCLI_REPLICATE_RDF_TIME_LIMIT=<TimeLimit> SYMCLI_REPLICATE_RDF_SLEEP_TIME=<SleepTime> SYMCLI_REPLICATE_BCV_TIME_LIMIT=<TimeLimit> SYMCLI_REPLICATE_BCV_SLEEP_TIME=<SleepTime> SYMCLI_REPLICATE_MAX_BCV_SLEEP_TIME_FACTOR=<Factor> SYMCLI_REPLICATE_MAX_RDF_SLEEP_TIME_FACTOR=<Factor>
The option file is a text file that contains parameters such as SRDF/AR hop type and cycle time. This file is required and is used in conjunction with the symreplicate command to start a replication session. This slide lists the variables available to be placed in the SRDF/AR options file. There are a minimum of two options that must exist for the symreplicate to be executed. The two required options are: SYMCLI_REPLICATE_HOP_TYPE and one of SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY. As of Solutions Enable 6.2 additional SYMCLI REPLICATE variables have been added.. Ref the SE product guide for a complete detailed explanation.
SYMCLI_REPLICATE_CONS_SPLIT_RETRY, SYMCLI_REPLICATE_R1_BCV_EST_TYPE, SYMCLI_REPLICATE_R1_BCV_DELAY, SYMCLI_REPLICATE_FINAL_BCV_EST_TYPE, SYMCLI_REPLICATE_FINAL_BCV_DELAY, and SYMCLI_REPLICATE_PERSISTENT_LOCKS
The symreplicate Options File (Cont.)

SYMCLI_REPLICATE_HOP_TYPE=[SINGLE|MULTI]
** Must be specified
SYMCLI_REPLICATE_CYCLE=[minutes|hh:mm]
Defines the period to wait between copy operations if more than 1 cycle Can be specified in total minutes or hours and minutes Defaults to 0; when one cycle ends another begins **Either CycleTime or Delay is a required parameter
SYMCLI_REPLICATE_CYCLE_DELAY=[minutes]
Specifies the minimum time to wait between the end of one cycle and the beginning of the next cycle Defaults to 0 ** Either CycleTime or Delay is a required variable
The two options that must exist for the symreplicate to be executed are: SYMCLI_REPLICATE_HOP_TYPE and either SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY. SYMCLI_REPLICATE_HOP_TYPE is self explanatory and is directly dependent on the configuration. SYMCLI_REPLICATE_CYCLE or SYMCLI_REPLICATE_CYCLE_DELAY requires a time interval for its value.

Time limit parameters
Control how long symreplicate retries operations
Applies only when an error occurs continuously or When no data has flowed for duration of the timer
Can set a time limit that is smaller than the actual time the operation takes Timer will not expire as long as data flows between device
EMC recommends using default settings initially. Time limit parameter changes should only be made based on recommendations from EMC.

Sleep time parameters
Specifies the minimum amount of that symreplicate sleeps:
Before checking again to see if devices have entered a specific state
or
To retry an operation when a recoverable error occurs
symreplicate must wait for prior operations to complete before moving to subsequent operations If the pair state is not reached, symreplicate sleeps for a period of time and then checks device state again When checking device state, symreplicate calculates how long to sleep based on the number of invalid tracks and the rate data is moving Value must be greater than 0
EMC recommends using default settings initially. Sleet time parameter changes should only be made based on recommendations from EMC.
Cycle, Delay and Overflow Parameters
The three parameters SYMCLI_REPLICATE_CYCLE, SYMCLI_REPLICATE_CYCLE_OVERFLOW, and SYMCLI_REPLICATE_CYCLE_DELAY significantly contribute to how the next replication cycle starts after the current cycle. The next two slides show different combinations of cycle time, delay time, and overflow behavior to achieve various results. Reference points on the cycle time lines are marked ACT (actual completion time), NCS (next cycle start), and MDT (minimum delay time). Short cycle and delay times (in minutes) were chosen for illustration purposes only. Copy cycles #1 and #2 have the same cycle time (2) and no delay time. When copy cycle #1 runs longer than two minutes, the overflow setting of Next results in a new copy cycle beginning at the four-minute mark.
Cycle, Delay and Overflow Parameters (Cont.)
Copy cycles #3 and #4 have the same cycle time (2) and the same delay time (3). When copy cycle #3 runs longer than two minutes, the system waits three minutes and then resumes copying at the next scheduled copy cycle (the six-minute mark). When copy cycle #4 runs longer than two minutes, the system waits three minutes and then resumes copying as soon as the wait period completes.
Cycle, Delay and Overflow Parameters (Cont.)
Copy cycle #5 finishes before its scheduled cycle time of three minutes, waits two minutes, and begins copying again at the next scheduled copy cycle (the six-minute mark). Copy cycle #6 performs continuous copy cycles when its cycle time is set to zero. At the end of a cycle, the system waits three minutes and then begins another copy cycle, regardless of the overflow setting.
More Options
SYMCLI_REPLICATE_USE_FINAL_BCV=[TRUE|FALSE] Specifies whether a BCV should be used in the Target site of a multihop replication SYMCLI_REPLICATE_PROTECT_BCVS=[NONE|LOCAL|REMOTE|FIRST _HOP|SECOND_HOP|BOTH] Specifies if a protected BCV establish operation should be performed. SYMCLI_REPLICATE_TF_CLONE_EMULATION=[TRUE|FALSE] Specifies if TimeFinder Clone emulation mode should be used for RAID-5 BCVs.
Protected BCV establish: NONE: normal TimeFinder operations LOCAL: protected establish of local BCVs in a single-hop replication REMOTE: protected establish of remote BCVs in a single-hop replication FIRST_HOP: protected establish of Hop-1 BCV pairs in a multi-hop replication SECOND_HOP: protected establish of Hop-2 BCV pairs in a multi-hop replication BOTH: protected establish of both Hop-1 and Hop-2 BCV pairs in a multi-hop replication, and both local and remote BCV pairs in single-hop replication Clone Emulation: Beginning with Solutions Enabler 6.0, if ALL BCVs are RAID-5 BCVs, TimeFinder/Mirror commands automatically map to corresponding TimeFinder/Clone commands. However, if there is a mixed environment, i.e. some RAID-5 BCVs and others are RAID1/unprotected, then this option must be explicitly set to TRUE. Default is FALSE.
Defining and Adjusting Parameters

EMC recommends starting with loose time constraints for cycle time parameters Adjust parameters once basic information is gathered or determined from initial cycles
Data size SRDF throughput Operation timings
Session progress can be monitored using the query argument for the symreplicate command and settings can be adjusted as required As of SE 6.1, the symreplicate stats command
It is recommended to be generous with time parameters for cycles, and adjust once more information is collected for various cycles. The times configured in the options file should be configured for worst case scenarios. Beginning with Solutions Enabler 6.1, you can display statistical information for cycle time and invalid tracks by using the symreplicate stats command. The command can be issued by device group (-g) or composite group (-cg) for a specified Symmetrix (-sid) and information can optionally be written to a specified log file (-log). The -all option is the default and will display both the cycle time and invalid tracks statistics. The following example will display both cycle time and invalid track for device group abcdg on Symmetrix 123,
symreplicate -g abcdg -sid 23 -all stats log abcdg.log
Overview of Implementation
Perform initial synchronization tasks
Devices must be in specific pair states for SRDF/AR cycle to execute
Issue the symreplicate command to begin the replication cycle

Specify the options file Specify TimeFinder consistent split operations Automatic replication continues until the session completes the number of predefined cycles or until symreplicate command issued to stop the session Sessions can be started, stopped, restarted, or queried Pre-action and post-action scripts can be applied to the data replication process
Before using the symreplicate command, perform the initial synchronization tasks. All devices must be in a specific paired state before executing the SRDF/AR cycle.
Automatic Initialization to Required Mirror States

Execute one cycle to prepare the mirror states
# symreplicate -g single setup -option /chai/test Execute a symreplicate 'Start' operation for device group 'single' (y/[n]) ? y
Prepare the mirror states and start the session

# symreplicate -g single start -option /chai/test setup -consistent Execute a symreplicate 'Start' operation for device group 'single' (y/[n]) ? y
Use ECA for TimeFinder Consistent split. ppath and vxfs could also be used
Beginning in SE 5.4, SRDF/AR supports automatic setup using: symreplicate setup symreplicate start -setup The setup command: Sets-up required pair states Options file indicates single or multi hop type setup Same options file used for setup and subsequent start command Executes one cycle, which may take some time, and then exits Setup command executes just one cycle, regardless of the number of cycles specified in the options file The start command with the -setup option: Sets-up the required pair states If successful, begins the symreplicate session Options file defines the hop type and the copy cycle parameters that you have chosen for the session
Single Hop Replication Cycle

1. Split BCV pairs (wait for any ongoing establish to complete...) 2. Establish RDF pairs 3. Suspend RDF pairs (wait for the establish operation to complete...) 4. Establish BCV pairs 5. Establish BRBCV pairs 6. Split the BRBCV pairs (wait for the establish operation to complete...)
The symreplicate command automatically executes the above steps in each cycle during a replication session. These steps are similar to those used to initializing the mirror states. It is important to understand these inner working steps because users can choose to terminate a session after the current step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must be to TRUE for symreplicate to log the information after each step to the SYMAPI log file. The location of the SYMAPI varies from one platform to another. The format of the file name, symapi-YYYYMMDD.log, is identical across platforms.
Single Hop Example: Options File

DMX800ibm1 /var/symapi/config more rep_options.txt SYMCLI_REPLICATE_HOP_TYPE=SINGLE SYMCLI_REPLICATE_CYCLE=10 SYMCLI_REPLICATE_CYCLE_OVERFLOW=NEXT SYMCLI_REPLICATE_NUM_CYCLES=6
A simple Options file specifying single-hop replication of 10 minute cycle times, to be repeated 6 times.
Single Hop Example: Setup

DMX800ibm1 / symreplicate -g single_hopdg setup -optimize -options /var/symapi/config/rep_options.txt -foreground nop
Checking for valid group configuration... Checking for valid initial group state... Setting up local BCV pairs... Optimizing Local BCV pairs... Waiting for local BCV synchronization... Splitting local BCV pairs... Incrementally establishing RDF pairs... Setting up remote BCV pairs... Optimizing remote BCV pairs... Incrementally establishing remote BCV pairs... Waiting for RDF synchronization... Waiting for remote device synchronization... Splitting remote BCV pairs... Incrementally establishing local BCV pairs... Setup complete; exiting symreplicate...
The setup command runs one cycle and exits. The foreground option provides details of each of the steps being executed.
Single Hop Example: Query and Start

DMX800ibm1 / symreplicate -g single_hopdg query Device Group (DG) Name DG's Symmetrix ID Remote Symmetrix ID Replicate Hop Type Status Step : single_hopdg : 000187910035 : 000187910156 Cycle Period Current Cycle Max Cycles
--------- --------- ----------------------------- -------- -------- -------SINGLE Completed Setting up devices 10 m 1 1
DMX800ibm1 / symreplicate -g single_hopdg start -options /var/symapi/config/rep_options.txt -consistent -nop
Checking for valid group configuration... Checking for valid initial group state... symreplicate process launched.
Query shows the setup has completed. We are now ready to start the single-hop replication cycles.
Single Hop Device States: R1/BCVs

DMX800ibm1 / symmir -g single_hopdg query Device Group (DG) Name: single_hopdg DG's Type DG's Symmetrix ID Standard Device : REGULAR : 000187910035 BCV Device State
-------------------------- ------------------------------------- -----------Inv. Logical Sym Tracks Logical Sym Inv. Tracks STD <=> BCV
-------------------------- ------------------------------------- ------------
DEV001 DEV002 DEV003 DEV004 DEV005 Total Track(s) MB(s) Legend:
001C 001D 001E 001F 0020
0 BCV001 0 BCV002 0 BCV003 0 BCV004 0 BCV005 ------0 0.0
0021 * 0022 * 0023 * 0024 * 0025 *
0 Synchronized 0 Synchronized 0 Synchronized 0 Synchronized 0 Synchronized ------0 0.0
(*): The paired BCV device is associated with this group.

At the start of a cycle, the R1/BCVs are established and synchronized with the Standard devices.
Single Hop Device States: Target BCVs

DMX800ibm1 / symmir -g single_hopdg query -bcv -rdf Device Group (DG) Name: single_hopdg DG's Type DG's Symmetrix ID Remote Symmetrix ID : REGULAR : 000187910035 : 000187910156 R E M O T E Standard Device S Y M M E T R I X BCV Device State
-------------------------- ------------------------------------- -----------Inv. Logical Sym Tracks Logical Sym Inv. Tracks STD <=> BCV
-------------------------- ------------------------------------- -----------BCV001 BCV002 BCV003 BCV004 BCV005 Total Track(s) MB(s) Legend: (*): The paired BCV device is associated with this group.
0021 0022 0023 0024 0025
0 BRBCV001 0 BRBCV002 0 BRBCV003 0 BRBCV004 0 BRBCV005 ------0 0.0
002B * 002C * 002D * 002E * 002F *
0 Split 0 Split 0 Split 0 Split 0 Split ------0 0.0
The BCVs on the Target site are split from their R2s.
Single Hop Device States: R1/BCV R2 (SRDF)

DMX800ibm1 / symrdf -g single_hopdg query -bcv Device Group (DG) Name DG's Type DG's Symmetrix ID R E M O T E Source (R1) View -------------------------------ST BCV Logical Device Dev A T E R1 Inv Tracks R2 Inv Tracks LI N K S Dev : single_hopdg : REGULAR : 000187910035 S Y M M E T R I X Target (R2) View MODES
-------------------------------- -- ------------------------ ----- -----------BCV001 BCV002 BCV003 BCV004 BCV005 Total Track(s) MB(s) 0021 NR 0022 NR 0023 NR 0024 NR 0025 NR 0 0 0 0 0 0 NR 0021 WD 0 NR 0022 WD 0 NR 0023 WD 0 NR 0024 WD 0 NR 0025 WD 0 0 0 0 0 0 S.. 0 S.. 0 S.. 0 S.. 0 S.. Suspended Suspended Suspended Suspended Suspended
-------- -------0 0.0 0 0.0
-------- -------0 0.0 0 0.0

The link between the R1/BCVs and their R2s on the Target site is Suspended.
Single Hop: Stop a Session

DMX800ibm1 / symreplicate -g single_hopdg stop Execute a symreplicate 'Stop' operation for device group 'single_hopdg' (y/[n]) ? y
Stop operation underway.
DMX800ibm1 / symreplicate -g single_hopdg query
Device Group (DG) Name DG's Symmetrix ID Remote Symmetrix ID
: single_hopdg : 000187910035 : 000187910156
Replicate Hop Type Status Step
Cycle Period
Current Cycle
Max Cycles
--------- --------- ----------------------------- -------- -------- -------SINGLE Stopped Waiting for next cycle 10 m 3 6
Single Hop: Restart a Stopped Session

DMX800ibm1 / symreplicate -g single_hopdg restart -recover Execute a symreplicate 'Restart' operation for device group 'single_hopdg' (y/[n]) ? y
symreplicate process launched.
DMX800ibm1 / symreplicate -g single_hopdg query
Device Group (DG) Name DG's Symmetrix ID Remote Symmetrix ID
: single_hopdg : 000187910035 : 000187910156
Replicate Hop Type Status Step
Cycle Period
Current Cycle
Max Cycles
--------- --------- ----------------------------- -------- -------- -------SINGLE Active Establishing RDF pairs 10 m 3 6
Failure/Recovery: Single Hop

When a failure occurs in the primary site, we can have one of the three situations on the target site
All BCVs on the target are split from their corresponding R2 devices All BCVs on the target are established with their corresponding R2 devices Some of the BCVs on the target are split from their corresponding R2 devices, while others are still established with their corresponding R2 devices
Recovery/Restart involves identifying devices with the most current and consistent copy of data
Determination of the states of the devices and deducing the cycle step using the states can be performed from a host on the target side, using appropriate device files. These operations can be performed from a different host on the source side, if the failure only affected the primary host, and the array and site are still available and accessible.
Failure/Recovery: Single Hop (Cont.)

All BCVs at target site are split:
Are R1/BCVs established with the Standard devices? Are R1/BCVs transmitting owed data to their R2 devices? Are R1/BCVs R2 pairs in a partitioned state?
All BCVs at target site are established:

R1/BCVs have completed transmission of data. The BCVs are in the process of being synchronized with their R2s
At the target site, some BCVs are split while others are still established:
R1/BCVs have completed transmission of data. The BCVs are in the process of being split from their R2s
All BCVs are split: If the host on the primary site has failed, but the array and links are available, the R1/BCVs can either be established with the Standard devices or can be split from the Standard devices and transmitting data to the R2. In the former case, both the R2 and BCV contain identical and consistent data, and restart can be done from either set of devices. In the latter case, the most recent data received by the R2 and BCV has consistent data up to the last cycle. In this case, one can wait for transmission to complete and then restart from the R2. Prior to restart it would be advisable to reestablish the BCVs, synchronize, and split to maintain a gold copy. In the third case, R1/BCVs still have tracks owed to the R2. So the consistent data resides on the BCV. One can restart using these or preferably restore the BCV data to the R2s and restart from the R2. All BCVs are established: It is clear that both R2 and BCVs have the most recent and consistent data at the end of the BCV synchronization. One can then split the BCVs (to maintain a gold copy) and restart from the R2s. Some BCVs are split while others are still established: The R2s have the most recent and consistent data. Re-establish all the BCVs with the R2s, synchronize, and split (to maintain a gold copy), prior to restarting from the R2s.
Multi-Hop Replication Cycle

1. Split the first hop BCV pairs (wait for the ongoing establish to complete) 2. Establish remote RDF pairs 3. Suspend remote RDF pairs 4. Establish the first hop BCV pairs (wait for the RDF establish to complete...) 5. Establish the second hop BCV pairs 6. Split the second hop BCV pairs (wait for the establish operation to complete...)
The symreplicate command automatically executes the above steps in each cycle during a replication session. These steps are similar to those used to initialize the mirror states. It is important to understand these inner working steps because users can choose to terminate a session after the current step instead of at the end of the cycle. The SYMCLI_REPLICATE_LOG_STEP parameter in the option file must be set to TRUE for symreplicate to log the information after each step to the SYMAPI log file. The location of the SYMAPI varies from one platform to another. The format of the file name, symapiYYYYMMDD.log, is identical across platforms.
Clustered SRDF/AR
Host A
R1
R2
BCV
Host B
SFS
R1 BCV
R2
Local
Bunker
Target
Since Enginuity 5669, Symmetrix arrays support clustered SRDF/AR environments for multiple node (host) capability. Clustered SRDF/AR provides the capability to start, stop, and restart replication sessions from any host connected to any local Symmetrix array participating in the replication session. The clustered SRDF/AR environment allows the replication log file to be written directly to the Symmetrix File System (SFS) instead of the local host directory of the node that began the session. If the primary node should fail, then any locally attached host to the Symmetrix array containing the log file would then be able to restart the SRDF/AR session from where it left off. If you begin a session and specify a user log file name (-log), you must specify the -log option for all other commands in the session sequence. To write the log file to the SFS, you must specify the ID of the Symmetrix array (-sid) where the log file is to be stored at the start of the replication session, along with a group name (-g, -cg) and an optional user log filename (-log). For example: symreplicate start -g session1 -log srdfar1.log -sid 201 Note: Not specifying the Symmetrix ID (-sid) at the start of the session, causes the log file to be written to local disk using the default SYMAPI log directory, which is not restartable from another node.
Failure/Recovery: Multi-hop
In the event of a primary site failure, R2s on the bunker site always contain the most recent and consistent data (SRDF/S between R1 R2). The state of the R1/BCVs in the bunker site dictates the restart/recovery procedures at the target site:
Are R1/BCVs established with the R2 devices? Are R1/BCVs transmitting owed data to their R2 devices at the target site i.e. All R1/BCVs are split from their R2s? Are some R1/BCVs split, while others are established?
As in the case of single-hop failure, recovery/restart involves identifying devices with the most current and consistent copy of the data, but at the bunker site
Note that BCVs at the target site are optional in a multi-hop configuration (SYMCLI_REPLICATE_USE_FINAL_BCV=<TRUE|FALSE>). In case BCVs are used at the target site, we must consider their pair states if we wish to preserve copy of data from previous cycle.
Failure/Recovery: Multi-hop (Cont.)

R1/BCVs are established with their R2s
Bunker R2s are synchronizing the most recent data with the R1/BCVs. Target R2 and optional BCV contain consistent data up to the last cycle
R1/BCVs are split from their R2s

R1/BCVs are transmitting data and optional BCVs are split from the target R2s R1/BCVs have completed data transmission, and optional BCVs are either split from or established with the target R2s
Some R1/BCVs are split while others are established with their respective R2s
Optional target BCVs are necessarily in a split state
R1/BCVs are established with their R2s: In this case we can wait for synchronization between R1/BCVs and the bunker R2s. Then split the R1/BCVs and resume the link between R1/BCVs and target R2s. The optional BCVs at the target site can be left in a split state to preserve consistent data from the previous cycle. R1/BCVs are split from their R2s: If the target BCVs are split, then one can re-establish the R1/BCVs with the bunker R2s, wait for synchronization, split, and resume the link between R1/BCVs and the target R2s. The BCVs at the target will again have consistent data up to the last cycle. If BCVs at the target site are in an established state, one can wait for or verify synchronization, split the BCVs (to preserve data from the previous cycle), then perform the re-establish, synchronization, split, and resume of the R1/BCVs. Some R1/BCVs are split while others are established with their respective R2s: In this instance, one can re-establish ALL the R1/BCVs with the bunker R2s, synchronize, split, and resume links to get the most recent and consistent data over to the target R2s.
General Recovery Procedures

Once the R2 devices on the target site are populated with the most recent and consistent data, they must be RW enabled for the host Application restart times must be factored in to the overall Recovery Time Objective Availability of necessary device file definitions at the target site can help in reducing the RTO. Device files must be updated on any configuration changes adding more Standard devices or R1s It is preferable to restart/recover from the R2 devices
Facilitates easier return home BCVs can be used to preserve previous cycles data
RW enabling of R2s can be performed via the symrdf failover command with the appropriate device file definitions.
Module Summary
Key points covered in this module: Benefits of integrating EMCs SRDF and TimeFinder applications Business needs and requirements using an SRDF single or multi-Hop Symmetrix configuration Clustered SRDF/AR environmens SRDF/AR configuration requirements The symreplicate options file Replication cycle steps for a single and multi-hop environment
These are the key points covered in this module. Please take a moment to review them.
Course Summary
Key points covered in this course: Relevancy of SRDF solutions with different (RPO) Recovery Point Objective needs SRDF concepts, terminology and functionality Use of the SYMCLI command set to perform SRDF operations SRDF host considerations and configurations within Sun Solaris, HP-UX, IBM AIX, and Windows LVM environments SRDF/A operations SRDF/A theory of operations and application Architectural components of SRDF/A SRDF/AR operations Architectural components of SRDF/AR SRDF/AR theory of operations SRDF/Star environment EMC Consistency Technology
These are the key points covered in this training. Please take a moment to review them. This concludes the training. In order to receive credit for this course, please proceed to the Course Completion slide to update your transcript and access the Assessment.
Closing Slide
This concludes, Module VI SRDF Consistency Technology for Symmetrix Business Continuity SRDF Solutions.

SRDF Solutions

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

SRDF Solutions

Enviado por

Direitos autorais:

Formatos disponíveis

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Symmetrix Business Continuity: SRDF Solutions

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Updated SE 6.3 Updated for SE 6.4/72

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

All Material is for Training Purposes Only

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Training Schedule: Day 1 & 2

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Training Schedule: Day 3

Open Lab - Activity

2007 EMC Corporation. All rights reserved.

Training Schedule for SRDF Solutions, day three.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

HPux 11.00 with 2004 patch bundle

AIX 5.2 / ML03 (Maintenance Level)

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

What is Business Continuity?

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Why Business Continuity?

Damaged Reputation Customers Suppliers Financial markets Banks Business partners

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

SRDF Recovery Point Objectives (RPO)

Periodic Replication Asynchronous Replication Synchronous Replication

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

What are SRDF Solution Sets?

Report Generation Decision Support

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Source Limited Distance

No data exposure Some performance impact Limited distance

Seconds of data exposure No performance impact Extended distance

Hours of data exposure No performance impact Unlimited distance Requires BCVs

2007 EMC Corporation. All rights reserved.

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Bunker Site (Local)

Long Distance Site

SRDF/STAR (Symmetrix Triangular Automated Replication)

"Disaster" strikes and the primary site is lost

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

SRDF Management Tools

SMC: Symmetrix Management Console

EMC Replication Manager

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Symmetrix Management Console (SMC)

Enables access, configuration, and basic operation of Symmetrix arrays

SRM Planning and Provisioning SRM Monitoring and Reporting

Supports multiple generations of Symmetrix

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Alerts and Monitoring

Copyright 2007 EMC Corporation. Do not Copy - All Rights Reserved.

Components of the SMC Interface