All,

Below is a quick write-up of Oracle RMAN backups, metadata, and how all of this is handled by both TSM and NetBackup.

The write-up is not meant be an indictment of one backup product over the other, rather to: 1. Explain Oracle RMAN backup object tracking (via metadata)

2. Explain the differences between how each backup vendor chose to interact with the Oracle RMAN 3rd-party backup API 3. Explain the importance of RMAN metadata housekeeping when backing up to TSM

Hope it helps

Understanding Oracle RMAN metadata:

Oracle RMAN tracks backup objects via metadata Oracle RMAN metadata is stored in either one of two places: 1. In an RMAN 'recovery catalog' which resides in a separate Oracle database instance (preferably on a different host) 2. In the CONTROLFILE of the Oracle instance being backed up NOTE: Oracle usually recommends the use of the 'recovery catalog' vs. the CONTROLFILE method, but most users opt to use the CONTROLFILE, due to cost/resource constraints Unlike other database software, such as Microsoft SQL, Oracle wrote their 3rd-party backup API to NOT allow the 3rd-party backup software to control retention settings, nor to determine if an RMAN backup object is eligible for deletion or not

o Rather, Oracle wrote their 3rd-party backup API to dictate that Oracle RMAN is in control of: Determining when RMAN backup objects are eligible for deletion, using the RMAN command REPORT OBSOLETE The REPORT OBSOLETE, in conjunction with several RMAN configuration settings, determine which objects are eligible as OBSOLETE All of these options and configuration settings reside in Oracle RMAN - OUTSIDE the control of the 3rd-party software Deleting backup object metadata from RMAN (and optionally from the 3rd-party backup system) via the DELETE OBSOLETE RMAN command As described below, when Oracle RMAN issues the DELETE OBSOLETE command on an Oracle backup object, TSM marks the object as INACTIVE, thus making it eligible for expunging from TSM storage

How NetBackup manages RMAN backup objects:

NetBackup took the approach of enforcing/applying its own retention policy settings, configured on the NetBackup master server, overriding Oracle RMAN settings o From the perspective of the NetBackup master server, Oracle RMAN backup objects are stored as images just like normal file system backup are As such, these images are assigned a retention policy when the image is created (during the backup process) The retention policy is set in the NetBackup Application Backup schedule settings This means that NetBackup deletes Oracle RMAN backup objects (NetBackup images), based on its own retention policies, ignoring RMAN retention policies When all things are working well, this is usually not a problem, but there are times when NetBackup will delete RMAN backup data that should not have been deleted, or that the Oracle DBA may not have want deleted

 If NetBackup deletes an image that RMAN is not aware is being deleted, this is very dangerous, from a restorability/recoverability perspective, as RMAN will have metadata for backup objects that it THINKS are available for restore, but TRULY are not o This is one reason why Oracle RMAN has the commands 'CROSSCHECK and DELETE' o These commands allows Oracle RMAN to see if the 3rd-party software still has a RMAN backup object, and if not, remove it from the RMAN metadata for that particular backup, as it is no longer available for restores The Veritas NetBackup for Oracle Administrators Guide PDF recommends automating scripts to periodically run CROSSCHECK and DELETE commands on the RMAN metadata for RMAN backups to NetBackup

How TSM manages RMAN backup objects:

Unlike NetBackup, TSM took the approach of letting Oracle RMAN determine what needs to be kept and what needs to be deleted from the backup server (TSM storage) which is what Oracle RMAN was designed to do, and wants to do TSM does not delete Oracle RMAN backups, based on retention policies configured in TSM rather, TSM relies on Oracle RMAN to: 1. 2. Define retention policies for RMAN backup data Track RMAN backup metadata

3. Notify TSM when RMAN backup data is to be deleted (via the REPORT OBSOLETE and DELETE OBSOLETE RMAN commands) This puts Oracle RMAN (and thus the Oracle DBA) in control of what should be kept, and what should not - NOT the backup administrator, who typically does not know what should/shouldn't be kept It also allows the Oracle DBA to mark certain objects to be kept longer than the default retention - useful for when an Oracle DBA knows a specific backup is an important backup, that they desire to keep longer than the default retention policy that RMAN applied to it during the backup As such, the TSM retention settings for an Oracle backup agent should always be set to: o Versions Data Exists: 1

o Versions Data Deleted: 0 o Retain Extra Versions: 0 o Retain Only Version: 0 This retention settings setup basically says: o Only keep 1 version of an object (since each Oracle RMAN backup piece should have a unique name) o Retain 0 versions of objects that are marked INACTIVE (objects get marked INACTIVE via Oracle RMAN issuing DELETE OBSOLETE commands on objects) o Keep INACTIVE extra/only versions of objects for 0 days in essence, when Oracle RMAN marks an object as INACTIVE, via the DELETE OBSOLETE command, delete the object immediately from TSM The object will be deleted from TSM storage the next time TSM runs EXPIRATION, which we run, automatically, once a day - so it may take up to 24 HRs for an INACTIVE object to truly get purged from TSM storage

Issues with housekeeping RMAN metadata in the CONTROLFILE:

When RMAN metadata is stored in the CONTROLFILE (not in an external recovery catalog Oracle database), there is a limit to how much RMAN metadata the CONTROLFILE will house o The default is 7 DAYS, or until the CONTROLFILE runs out of space to house RMAN metadata - whichever comes first o If your CONTROLFILE is not large enough to house 7 days worth of RMAN metadata, RMAN starts removing RMAN metadata entries - oldest first The Oracle DBA can do a couple of things to override these default settings, such as: o Modify the CONTROL_FILE_RECORD_KEEP_TIME setting o Expand the CONTROLFILE as needed, to ensure it has enough space to hold all RMAN metadata, so that RMAN metadata does not get deleted from the CONTROLFILE before RMAN determines the object referenced by the metadata

is deemed OBSOLETE and it has a chance to tell TSM to delete it from TSM storage Failure to do the above steps, will cause RMAN to delete RMAN backup object metadata BEFORE it can tell TSM to delete the backup data from TSM storage - thus leaving the RMAN backup data 'orphaned' in TSM storage When this happens, you have to run the TSM TDP/Oracle utility 'tdposync' to: o Find RMAN backup data that has been orphaned in TSM storage o Select the appropriate RMAN backup data objects and tell TSM to delete them from TSM storage o Failure to properly manage RMAN metadata and the CONTROLFILE, and not running tdposync will result in the orphaned RMAN objects consuming TSM storage, with the orphaned objects adding to the customers overall TSM occupancy, which we bill against NOTES regarding tdposync: The tdposync utility is a text-based, menu-based utility, it cannot be invoked via scripts, etc., therefore it is always a manual process to run the tdposync The tdposync should always be run by an Oracle DBA, not a TSM administrator, as it is only the DBA that truly knows what objects in the list are orphaned and which ones need to be kept More info on the tdposync utility can be found in the IBM Tivoli Storage Manager for Databases - Data Protection for Oracle for UNIX and Linux Installation and User's Guide PDF o Since we want to prevent orphaned objects in the first place, nor do we want to constantly have to manually run the tdposync utility, we always recommend that the customer does one of these two options: 1. Incorporates RMAN housekeeping functionality (REPORT OBSOLETE / DELETE OBSOLETE) into their daily RMAN backup scripts that we execute from TSM 2. Create separate housekeeping scripts that we execute from TSM

 Executing RMAN scripts from TSM gives us visibility into whether the scripts are running or not, and if they are running, are the completing successfully or failing When customers deny us from automating their RMAN backups/housekeeping scripts from TSM, we lose all visibility into their RMAN backup/housekeeping process

-James Morton

Senior Storage Engineer Enterprise Service Delivery SunGard Availability Services 550 E. 84th Ave Suite E-5, Thornton, CO 80229 Phone: 303.942.2813 Fax: 303.289.9155 James.Morton@sungard.com