Purging Workflows Reducing Data Proliferation

Nitin Jain

Infosys Technologies Limited

Introduction
Every workflow that is launched in Oracle creates an instance in the database which is identified uniquely by its items. Each workflow item has an itemtype and itemkey which is unique and is registered in the wf_items table as an when the workflow is launched. A typical workflow item can consist of child workflow items, parent workflow activities and notifications. This data that is inserted into the workflow tables while executing a workflow is called the runtime workflow data. Since a lot of modules make use of workflows, the amount of obsolete runtime data that is retained can grow abnormally huge, adversely affecting the performance of the system. This paper discusses an approach that can be adopted to resolve the workflow data growth issues by performing effective purging of the obsolete data.

Overview
Each workflow instance in Oracle has a persistence type that controls its status audit trail. A workflow with Synchronous persistence usually does not maintain any history while a workflow with Temporary persistence is maintained till n days after which it becomes eligible for purging. Permanent persistence workflows are required to be forced purged. After a workflow completes, all the workflow activities, child processes and notifications within the workflow item are completed and are no longer required by the process that had launched the workflow. Oracle stamps end date to workflow items/activities after they have completed. A workflow item becomes eligible for purging if it meets the following criteria: 1. The workflow item is closed i.e. its stamped end-dated. 2. All the activities within the workflow item are closed. 3. All the child workflows (including error workflows) are closed. 4. All the notifications within the workflow item are closed. The standard bde_wf_data.sql (165316.1) script can be used to identify the obsolete runtime data that is eligible for purging. The script shall help in understanding the distribution of workflow data by item type. It spools the output providing counts of workflow item data, related activities (open/closed) and notifications (Parent/Child, Open/Closed) for every item type. It should be noted that the script runs against large workflow tables and might require performing full table scans. The following are the workflow tables that can be monitored (using standard counts, histograms etc) for a check on growth of obsolete data: WF_ITEMS WF_ITEM_TYPES WF_PROCESS_ACTIVITIES WF_ACTIVITY_ATTRIBUTES WF_ACTIVITY_ATTRIBUTES_TL WF_ITEM_ACTIVITY_STATUSES WF_ITEM_ATTRIBUTE_VALUES WF_NOTIFICATIONS WF_NOTIFICATION_ATTRIBUTES

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 1

data:

Oracle offers the following two options to purge the identified obsolete workflow runtime 1. Using the Purge Obsolete Workflow runtime Data concurrent request. 2. Purge APIs (WF_PURGE PL/SQL packages).

Purge Obsolete Workflow Runtime Data (FNDWFPR): The Purge Obsolete Workflow runtime Data is a standard concurrent request provided by Oracle to purge workflow items, parent activities, child processes and related notifications. The program calls the database package WF_PURGE.total() to purge the identified eligible completed workflow items. The program has the following parameter values: Item Type Item type of the workflow to be purged. Default is blank. Item Key Item key of the workflow to be purged. Default is blank Age Minimum age of the data to purge (in days). Persistence Type Persistence type of the workflow to be purged. Either TEMP for Temporary or PERM for Permanent. Default is Temporary. Runtime Data If yes, then the program will purge only workflow runtime data. If no, then the program will purge workflow runtime, design and process definition data. Purge APIs: Oracle provides the WF_PURGE package that contains APIs to purge workflow runtime data. These APIs can be used to purge workflow items that are complete but not yet end-date due to some reasons. The most commonly used procedures are: WF_PURGE.Items() Purges all runtime data associated with completed items, their processes and notifications. WF_PURGE.Activities() Purges workflow process definition versions that are not used and are obsolete. WF_PURGE.Total() Purges both workflow item and activity data. WF_PURGE.AdhocDirectory Purge users and roles in WF_LOCAL tables whose expiration date have elapsed and are not referenced in any notification. WF_PURGE.TotalPerm() Purges obsolete runtime data associated with item types of persistence type Permanent. The standard Purge Obsolete Workflow runtime Data' (FNDWFPR) request can be used to purge the eligible obsolete workflow runtime data. The concurrent request however does not perform effective purging due to several reasons like: 1. Eligibility: A workflow item is not eligible for purging even if it is CLOSED when it does not meet the criteria mentioned earlier. 2. Data quality issues: For e.g. A PO Approval workflow that might have completed, but not end-dated, shall not be considered eligible for purging. 3. Performance issues: The concurrent request runs too slow as it performs large full table scans. 4. Memory/Table space issues: The request frequently errors as it runs into memory/table space issues when accumulating large data during purging process. Thus a combination of both the options i.e. the Purge Obsolete Workflow runtime Data request and Purge APIs is the best way to perform effective purging and thus reduce data proliferation.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 2

Approach
The below is an approach that we employed for one of our clients for performing effective workflow runtime data purging. This is by far not the only method, but definitely effective. The solution consists of the following steps: 1. Data Identification: Run the following SQL query to determine the volume of the workflow runtime data classified by item type, persistency and status. SELECT wi.item_type item_type, wit.persistence_type p_type, DECODE (wi.end_date, NULL, 'OPEN', 'CLOSED') status, COUNT (*) FROM wf_items wi, wf_item_types wit WHERE wit.NAME = wi.item_type GROUP BY item_type, wit.persistence_type, wit.persistence_days, DECODE (wi.end_date, NULL, 'OPEN', 'CLOSED') ORDER BY item_type, wit.persistence_type, status;

As seen in the above output, a workflow item with status CLOSED indicates its eligibility for purging. One may run the standard bde_wf_data.sql (165316.1) script can be used to identify the obsolete runtime data that is eligible for purging. The script shall help in understanding the distribution of workflow data by item type. 2. Closing Open Notifications/Errant Activities: Once the eligible obsolete data is identified, run the standard workflow clean worklist script bde_wf_clean_worklist.sql (255048.1) to retry all errant activities tied to the workflow items that are open. These errant activities are no longer in ERROR state and on a retry shall close. The script can be run on a weekly basis for each operating unit. For closing notifications that are assigned to the workflow system administrator, one may use the native Oracle OAM and follow the steps below to abort them: a. Login to OAM > Workflow Manager b. Navigate to throughput (at the bottom of the page) > select work items c. Select view Errored Work Items > Go

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 3

d. Select the line System Error and select the link that takes you to Work Item Activity Stages. e. Select the error types DEFAULT_ERROR or RESET_ERROR and select Abort All option. f. Once aborted the errant activities shall become eligible for purging.

3. Retry Workflows in Error: Workflow items shall error due to unknown reasons and might get corrected as well. A mass retry of such error workflows can progress them and thus complete them making them eligible for purging. One may use one of the following two scripts to retry error workflows: For workflows with notifications tied Run the standard script bde_wf_retry.sql (255047.1) to retry the errant activities. For workflows with no notifications attached Run the standard script bde_wf_proc_err.sql (255046.1). 4. Forced Purging using Purge APIs: Based on the output generated by the standard bde_wf_data.sql, identify the data eligible for purging and use the below purge APIs to perform forced purging as the standard request Purge Obsolete Workflow runtime Data' (FNDWFPR) is in-efficient runs into issues mentioned earlier.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 4

The persistence type of a workflow item can be determined from the table WF_ITEM_TYPES and its corresponding language table (_TL). SELECT wit.NAME, wit.persistence_type, wtl.display_name, wtl.description FROM wf_item_types_tl wtl, wf_item_types wit WHERE wtl.LANGUAGE = 'US' AND wtl.NAME = wit.NAME;

If the Persistence type of the workflow is PERM use the below API call for each workflow item in the identified output data: o WF_PURGE.totalperm(ITEM_TYPE, ITEM_KEY, ENDDATE, FORCE, DO-COMMIT); o Parameters: Item Type Item Type of the workflow or null for all item types. Item Key Item Key of the workflow or null for all item keys. End Date Purges all workflow processes closed after this end-date Force Forces to purge closed workflow processes even if they have child processes open. Do Commit True does a commit. o E.g. exec WF_PURGE.TOTALPERM(FAFLEXWF, NULL, SYSDATE-30, TRUE, TRUE);

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 5

If the Persistence type of the workflow is TEMP o WF_PURGE.items(ITEM_TYPE, ITEM_KEY, ENDDATE, FORCE, DO-COMMIT); o E.g. exec WF_PURGE.ITEMS('POAPPRV', '90539-251083', SYSDATE, TRUE, TRUE); One may write a routine to do the above steps like we have done. The routine can be submitted as a concurrent request like the standard purge request. It is to be noted that the request shall run for a longer time than anticipated if doing it for the first time and might error-out due to table-space issues.

5. Aborting Open WFERROR workflows: If there are WFERROR workflows that are open due to unknown reasons, they can be aborted using the abort API or using the native Oracle OAM functionality. WF_ENGINE.ABORTPROCESS(item_type, item_key); o For e.g. exec WF_ENGINE.abortprocess(WFERROR, item_key); commit; One may write a routine/concurrent request like we have done to identify such error workflows and then abort them. The Abort API can also be used to abort workflows associated with purchase orders and requisitions. 6. Run the standard Purge Obsolete Workflow runtime Data Request: After following the steps 1-5, making more no. of workflow items eligible for purging, run the standard Purge Obsolete Workflow runtime Data' (FNDWFPR) request with the following parameters: Item Type: Item type associated with the obsolete runtime data you want to delete. Leave this argument null to delete obsolete runtime data for all item types. Item Key: A string generated from the application objects primary key. The string uniquely identifies the item within an item type. If null, the program purges all items in the specified itemtype. Age: Minimum age of data to purge, in days, if x_persistence_type is set to TEMP. The default is 0. Persistence Type: Persistence type to be purged, either TEMP type for Temporary or PERM for Permanent. The default is TEMP. Core Workflow Only: Enter 'Y' to purge only obsolete runtime data associated with work items, or 'N' to purge all obsolete runtime data as well obsolete design data. The default is 'N'. Commit Frequency: Frequency at which the program needs to commit records. Default is 500.

It should be noted that several instances of the standard request Purge Obsolete Workflow runtime Data can run at the same time, however it is not recommended because of the issues mentioned earlier.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 6

For e.g. before running the above request, the count was:

After running the request the count remains same as the purging is not done as it does not meet all the conditions mentioned earlier. Hence we run the scripts mentioned in step 2, 3, 5 and then do a force purge using a custom routine mentioned in step 4.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 7

Image: 1 Custom Routine for Force Purge

Image: 2 Log of Custom Routine, doing Force Purge After a force purge, we verify the count and see 0.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 8

7. Purging Validation: One may run a count on the following tables, or run the SQL query in step-1 to determine the purging in % before and after the purging run. SELECT COUNT (*) FROM wf_item_attribute_values; SELECT COUNT (*) FROM wf_item_activity_statuses; SELECT COUNT (*) FROM wf_notification_attributes;

Conclusion
Thus a combination of both the options i.e. the Purge Obsolete Workflow runtime Data request and Purge APIs is the best way to perform effective purging and reduce data proliferation.

COLLABORATE 09

Copyright 2009 by Nitin Jain

Page 9