BDLS IN LESS THAN 2 HOURS

IMPORTANT: The methods used here are not supported by SAP.

However, they are not very complex to understand and execute.

Scenario

The BDLS run takes more than 20 hours in our quality systems post
system refresh.

Our system details:-

Database size

2 TB

SAP version

ECC 6.0 patch level

21

Database Version

Oracle 11.2.0.2.0

OS Version

RHEL 5.6

No. of Clients to run BDLS

Two (450 & 500)

The objective of this blog is to analyze and find out if there are faster ways
to perform BDLS operation, if we cross boundaries of SAP standard
methods.

Step - 1

Based on SAP Note. 1547980 we decided to use older version of BDLS

program RBDLS2LS as it has capacity to run in parallel.

We were thinking that the program RBDLS2LS itself has some inbuilt
capability to run in parallel.
We were expecting it to use some RFC groups like parallel_generators, etc
and submit conversion for each table in seperate task/ process like client
copy.

Whereas we found out that this is not the case. The RBDLS2LS does not
do anything in parallel, neither does it use use any RFC groups not it uses
more than one BGD/ DIA processes.

Parallel in this case means that multilpe executions of the report

RBDLS2LS is possible. The report do not come back complaining with an
error that there is already a running BDLS process like the new BDLS
tcode.

Based on the tips from blog http://www.sdn.sap.com/irj/scn/weblogs?

blog=/pub/wlg/4796 , we decided to run multiple jobs.

We increased number of background processes to 26.

Database was put in noarchivelog mode.

No tables were excluded from BDLS.

In client 450 > SE38 > RBDLS2LS > Generated RBDLS450

Again SE38 > RBDLS450 > Filled details as below > Executed in
Background

Continued like this for B*, C*,D*...and so on till ...Z* and finally by
excluding A* to Z*.

We also started execution on client 500 similarly in parallel.

Achievement

BDLS completed in 11 hours

Observations

We were able to find the jobs & tables taking longer

times.
For examples; Jobs converting tables C*, E*, M*, V*
and S* took more times than others.
It was time consuming to schedule 26 jobs per client

Step - 2

Taking help from an ABAPer, we could automate a BDC session to run

report RBDLS450 with A*, B*, C*...Z* as inputs to selection screen.

PART-2

Again Based on the tips from blog

http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/4796 , we wanted
to optimize it more by building indexes.

Before building indexes, its benificial to understand how BDLS works.

BDLS runs in phases:1. It will first determinies all tables that are using logical system in one
of its fields.
2. It will then find out all tables that need conversion, by running a
SELECT SINGLE command on all tables.
For example;
SELECT /*+ FIRST_ROWS(1) */ LOGSYSP FROM COEP WHERE
MANDT=500 AND LOGSYSP = 'S1Q500' AND ROWNUM <= 1;
means it has to just find one record to mark the given table as
update required (i.e. BDLS conversion required).
3. Update command is run to modify the logical system value. For e.g.
S1P500 to S1Q500.
UPDATE "MKPF" SET "AWSYS" = 'S1Q500' WHERE MANDT =
500 AND "AWSYS" = 'S1P500' AND ROWNUM <= 100000;

Phase 2 and 3 above are the most time consuming ones.

Phase 2....SELECT SINGLE has two outcomes;

b)
ROW not found.

a)

ROW found

a) When ROW is found; the response time is directly propotional to the

location of the satisfying row.
b) When ROW is not found; SELECT has to transverse through entire
TABLE or INDEX only to find no satisfying row. (FULL TABLE/ INDEX SCAN).

Unfortunately, most of the tables under BDLS falls under b), because the
LOGSYS fields have nothing (SPACE) in it.

Step - 3

Based on the tips from blog http://www.sdn.sap.com/irj/scn/weblogs?

blog=/pub/wlg/4796 , we wanted to build indexes.

To build indexes, we needed to identify all those tables contributing to

high run time on BDLS.

How to identify tables?

Reading through the code in report RBDLS2LS we could get the list of
tables BDLS would run through.

SELECT * FROM dd02l INTO

TABLE t_dd02l
FOR ALL ENTRIES IN t_field_ls
WHERE tabname = t_field_lstabname
AND as4local = tab_activ
AND tabclass = tab_class.
SELECT * FROM dd02l
APPENDING TABLE t_dd02l
FOR ALL ENTRIES IN t_field_pn
WHERE tabname = t_field_pntabname
AND as4local = tab_activ
AND tabclass = tab_class.

Equivalent Oracle SQL is:-

SELECTa.tabname, FROM dd03l

a, dd02l b
WHERE a.domname in
('LOGSYS','EDI_PARNUM')
AND a.as4local = 'A'
AND a.tabname = b.tabname
AND b.as4local = 'A'
AND b.tabclass = 'TRANSP'

Below thoughts came to our minds for selecting tables for indexing:

Top 15 tables based on size

Top 15 tables based on total number of rows

All tables of size greater than 5 GB

All tables where number of rows > 10000

We choose to go with "all tables where total number of rows > 10000",
because BDLS runs SELECT SINGLE command; and most of the tables fall
under category b) as explained above and performing a FULL TABLE
SCAN. Therefore most of the run time depends on the total number of
rows in the table.

Now we had to count number of rows in each table to finalize the tables to
build indexes.

Instead of using time consuming SELECT COUNT(*), we decided to use

data from the column NUM_ROWS in the table USER_TABLES.

We than created a seperate tablespace PSAPBDLS of 50 GB.

Created 80 indexes on client field (like MANDT) and logical system (like
LOGSYS).

NOLOGGING PARALLEL COMPUTE STATICS was used to build indexes.

An example command is given below:-

CREATE INDEX COEP_BDLS_IDX_12345 ON COEP(MANDT, LOGSYSO)

NOLOGGING PARALLEL TABLESAPCE PSAPBDLS COMPUTE
STATISTICS;

We also enabled parallelism on tables and other indexes as well.

ALTER TABLE COEP PARALLEL;
ALTER INDEX "COEP~0" PARALLEL;

We then ran RBDLS450 again in parallel (A*....Z*) as explained in Part -1,

Step - 1.

Achieveme
nt

No significant benifit. Just 1 hour

gain. BDLS completed in 10 hours

Observatio
ns

SELECT SINGLE phase completed

quickly
UPDATE phase broke down taking
very long time
Building 80 indexes was very time
consuming task taking 5 hours
80 Indexes took 50 GB of space

Step - 4

We automated by writing some sql scripts (which will be explained in later

parts):

Identifying tables where now_rows > 10000

Identifying client filed (like MANDT, CLIENT, RCLIENT, etc)

Identifying logical system field (like LOGSYS, LOGSYSP, SENDPRN,

RECVPRN, etc)

Building 80 indexes on client and logical system field in parallel

taking now 2 hours.

Step - 5

We tried with building REVERSE key index; with no significant benifit. The
run times remained more or less the same.

CREATE INDEX COEP_BDLS_IDX_12345 ON COEP(MANDT, LOGSYSO)

REVERSE NOLOGGING PARALLEL TABLESAPCE PSAPBDLS COMPUTE
STATISTICS;

Step - 6

With further studies we found that Oracle Bitmap indexes works better
than B-tree indexes for columns with only a few distinct values (just MALE,
FEMALE for example). They are compressed, very compact and fast to
scan.

MANDT contains very few distinct values and also LOGSYS realted fields
usually contains SPACE or very few distinct values. This characterstic
makes them ideal candidate for bitmap indexes.

We created bitmap indexes instead of regular indexes.

An comparions of size and access time is shown below:-

Activity on
COEP

Regular
Index

Bitmap
Index

Create index in
2 Min 25
parallel, nologging Sec

1 Min 32
Sec

Size of Index

3425 MB

53 MB

Select query run

time

16
Seconds

1.25
Seconds

Below is an example statement used to build bitmap index.

CREATE BITMAP INDEX COEP_BDLS_IDX_12345 ON COEP(MANDT,
LOGSYSO)
NOLOGGING PARALLEL TABLESAPCE PSAPBDLS COMPUTE
STATISTICS;

Achieveme 80 indexes took

nt
only 1 GB of
space
Building 80
indexes now
took 30 minutes

only with
automation
using sql scripts

PART-3

UPDATE phase will be slower with indexes; particularly those indexes

which are built on the the same fields that are being updated.

In other words, mass updates on the indexed fileds causes additional

overhead for maintainance and updating of the those indexes. Such
indexes are referred to as evil indexes.

Possible solution was to drop/ hide indexes having these fields before
performing mass updates.

Step - 6

After creating bitmap indexes; first we ran RBDLS450 in test mode

(check box - Test Run) in parallel as explained earlier in Part -1, Step - 1.

We then dropped the indexes and ran RBDLS450 in Conversion Mode,

i.e. by removing above check boxes.

Achievem
ent

BDLS completed in 8
hours. Not a great benifit
though.

Observati
ons

UPDATE phase used FULL

TABLE SCAN

From previous BDLS runs, we were able to gather below statistics: -

Table

BDLS run time in

hours

Table
size in
MB

MKPF

7.78

3253.19

SWW_CONTOB

6.28

4837.54

EDIDC

6.11

6766.91

MLHD

5.52

2786.75

RBKP

5.42

2972.29

VBRK

2.55

1029.07

VBAP

2.25

3333.66

ETXDCH

1.57

266.32

SMMAIN

1.44

96.08

SMPARAM

1.42

55.27

SMSELKRIT

1.41

234.27

We observed that these tables are not very big in size; the largest one is
EDIDC with size 6.6 GB and it take surprisingly more than 6 hours to
update.

Step - 7

With further googling around and reading several articles, we realized that
CTAS (Create table as select) is the best method to perform mass updates.

We decided to test it; we executed below commad: -

CREATE TABLE EDIDC_BDLS_BKUP

PARALLEL COMPRESS FOR OLTP NOLOGGING
STORAGE ( INITIAL 3253M NEXT 50M MINEXTENTS 1 MAXEXTENTS
UNLIMITED)
TABLESPACE PSAPNXP
AS
(
SELECT
MANDT, DOCNUM, DOCREL, STATUS, DOCTYP, DIRECT, RCVPOR,
RCVPRT,
decode(MANDT||RCVPRN,'500S1P500','S1Q500',RCVPRN) RCVPRN,
RCVSAD, RCVSMN, RCVSNA, RCVSCA, RCVSDF, RCVSLF, RCVLAD,
STD,
STDVRS, STDMES, MESCOD, MESFCT, OUTMOD, TEST, SNDPOR,
SNDPRT,
decode(MANDT||SNDPRN,'500S1P500','S1Q500',SNDPRN) SNDPRN,
SNDSAD, SNDSMN, SNDSNA, SNDSCA, SNDSDF, SNDSLF, SNDLAD,
REFINT,
REFGRP, REFMES, ARCKEY, CREDAT, CRETIM, MESTYP, IDOCTP,
CIMTYP,
RCVPFC, SNDPFC, SERIAL, EXPRSS, UPDDAT, UPDTIM, MAXSEGNUM
From EDIDC
);

You may notice the lines highlighted above in the command. The oracle's
decode function performs the conversion (UPDATE).

The newly created table EDIDC_BDLS_BKUP will have the S1Q500

replaced wherever S1P500 was found for fields RCVPRN and SNDPRN.

Another way of doing it is using CASE statement. It is more easier to build

CASE statement than DECODE.

For example, In a scenario we had to convert 4 logical systems, we use

CASE statement as below.

CREATE TABLE EDIDC_BDLS_BKUP

PARALLEL COMPRESS FOR OLTP NOLOGGING
STORAGE ( INITIAL 3253M NEXT 50M MINEXTENTS 1 MAXEXTENTS
UNLIMITED)
TABLESPACE PSAPNXP
AS
(
SELECT
MANDT, DOCNUM, DOCREL, STATUS, DOCTYP, DIRECT, RCVPOR, RCVPRT,
CASE LOGSYS
WHEN 'S1P100' THEN 'S1Q100'
WHEN 'S1P500' THEN 'S1Q500'
WHEN 'BWP200' THEN 'BWQ200'
WHEN 'PIP300' THEN 'PIQ300'
ELSE

LOGSYS

END LOGSYS,
RCVSAD, RCVSMN, RCVSNA, RCVSCA, RCVSDF, RCVSLF, RCVLAD, STD,
STDVRS, STDMES, MESCOD, MESFCT, OUTMOD, TEST, SNDPOR, SNDPRT,
decode(MANDT||SNDPRN,'500S1P500','S1Q500',SNDPRN) SNDPRN,
SNDSAD, SNDSMN, SNDSNA, SNDSCA, SNDSDF, SNDSLF, SNDLAD,
REFINT,
REFGRP, REFMES, ARCKEY, CREDAT, CRETIM, MESTYP, IDOCTP,
CIMTYP,
RCVPFC, SNDPFC, SERIAL, EXPRSS, UPDDAT, UPDTIM, MAXSEGNUM
From EDIDC
);

To our surprise this CTAS operation on EDIDC of 6.7 GB took only 12

seconds.

Then we dropped indexes on original table EDIDC.

Renamed original table EDIDC to EDIDC_ORIG.

rename EDIDC to EDIDC_ORIG;

Renamed newly created table EDIDC_BDLS_BKUP to EDIDC.

rename EDIDC_BDLS_BKUP to EDIDC;

Rebuild indexes from SE14 > EDIDC > Indexes > Create in Background

The total operation took less than 5 minutes for EDIDC , whereas BDLS
conversion took 6 hours.

Achieveme
nt

We were able to perform CTAS and conversion of

15 tables in 3 hours because CTAS command need
to be manually constructed
BDLS run on remaining tables took less than 2
hours
Total conversion time is 5 hours

Observatio
ns

Constructing CTAS command is time consuming

and prone to human error
Building indexes from SE14 needs to be triggered
manually
CTAS removed NOT NULL constrains from
converted (DECODED) fields

Looking at the significant benefit (6 hours vs. 5 minutes), we decided to

adopt CTAS for all tables where rows to be udpate > 1000.

Step - 8

The rebuilding of indexes from SE14 do not use parallel, nologging

options.
Some time can be saved if you build indexes from database level using
parallel clause and then turning off the parallelism.

Before dropping any index, you can get the DDL (SQL) Command to build
the index as below:-

set timi on
set echo off
set head off
set long 5000
set pagesize 0
set linesize 150
column DDL format A150

select DBMS_METADATA.GET_DDL('INDEX','EDIDC~0') DDL from dual;

Add clauses NOLOGGING PARALLEL COMPUTE STATISTICS the command;

Once Index is built; turn off parallelism

ALTER INDEX "EDIDC~0" NOPARALLEL;

NOT NULL constraints had to be added to table EDIDC for fields RCVPRN
and SNDPRN.
ALTER TABLE "EDIDC" MODIFY RCVPRN NOT NULL;
ALTER TABLE "EDIDC" MODIFY SNDPRN NOT NULL;

Step - 9

We left the update statistics running in the background with below

commands:-

brconnect -u / -c -f stats -t system_stats

brconnect -u / -c -f stats -t oradict_stats
brconnect -u / -c -f stats -t all -p 15 -f nocheck -f collect

Part - 4

Putting together all possible optimization and automation; below are the
brief steps we articulated:-

Put database in NOARCHIVELOG mode

Create PSAPBDLS tablespace of 5 GB

Enable parallelism for all tables invloved in

BDLS

Enable parallellism for all indexes of the

tables invloved in BDLS

Create bitmap indexes with PARALLEL

NOLOGGING COMPUTE STATITICS for tables
with total number of rows > 10000

Calculate rows to be updated by BDLS

Build CTAS command lines with decode

funtions

Perform coversion of tables where rows to

update > 1000 using CTAS

9

Save original index definitions

10 Drop original indexes

11 Rename original table to old
12 Rename newly created tables to original
13 Recreate indexes
14 Verify and Drop old table
15 Run BDLS in SAP using report RBDLS450 in
parallel (A*....Z*, etc) using BDC sessions.
16 Clean up
17 Drop bitmap indexes
18 Revert parallelism to original state for all
tables and indexes
19 Drop function created, if any
20 Drop extra tables created, if any
21 Run update statistics
22 Put database back on ARCHIVELOG mode

With automation and combining above optimization approach, BDLS was

achieved in 1 hour and 25 minutes.

Activity

Time

Create PSAPBDLS tablespace

of 5 GB

5
minutes

Bitmap indexes

30

Minutes
CTAS for 15 tables

15
Minutes

BDLS run in parallel

30
Minutes

Cleanup Bitmap indexes,

tablespace, etc

5
minutes

TOTAL 1:25
hours

You may also plan to create bitmap indexes on source system so that
these indexes are already available on target system. This will save 30
minutes.