Practical DB Essentials Mar 2010

Practical
OpenEdge RDBMS
Internals
Gus Björklund, Wizard, Progress Software

email: gus@progress.com
Progress Software User Group Finland

25.3.2010, Tallinn
Goals:
Explain selected internals to offer practical

advice, knowledge, and tips for DBA’s
Enhance DBA’s knowledge to optimize

database and solve problems
2 Practical OpenEdge RDBMS Essentials

Please ask questions if I am not explaining
something clearly or …

Allocating space for records.
the
maximum rows per block,
create limit,
toss limit
parameters

Record Data (RM) Blocks
block header
RM block header
row directory
free space
records

block header
RM block header
row directory expands
row directory
and shrinks as needed
free space free space expands

and shrinks
records

block header
RM block header
row directory expands
row directory
and shrinks as needed
free space free space expands

and shrinks
records are packed at

records the end of the block,
expand as needed

block header
RM block header
dir entry 0 RM block with 1 small record
free space
record 0

block header
RM block header
dir entry 0 RM block with 1 large record
free space
record 0

block header
RM block header
dir entry 0 RM block with 4 records
dir entry 1
dir entry 2 if maximum rows per block
dir entry 3 (rpb) is 4, then no more rows
can be added, no matter how
much free space there is
free space
record 3
record 2
record 1
record 0

Applying this stuff
block header Set rows-per-block to:

RM block header 256: never
dir entry 0
dir entry 1 128 on 8k blocks if you just
dir entry 2 want to keep it simple
dir entry 3 (10.1B and later)
free space Surprisingly, there are

dissenting opinions.
record 3
record 2
record 1
record 0

the
create and toss limits

block header
RM block header
dir entry 0 RM block with 4 records
dir entry 1
dir entry 2 create limit is minimum free
dir entry 3 space in block
create limit
purpose: provide (some) room
free space for expansion in case rows are
updated later
record 3
record 2 Must be set smaller than toss
record 1 limit
record 0

block header
RM block header
dir entry 0 RM block with 4 records, add
dir entry 1 one more
dir entry 2
dir entry 3 if ((free space - size of new row)
dir entry 4 > create limit), a new row can
create limit be added
free space
record 4
Now we have 5 rows, no space
record 3
record 2
record 1
record 0

block header
RM block header
dir entry 0 after deleting 3 records
dir entry 1 3 unused dir entries
dir entry 2
dir entry 3
dir entry 4 dir entry 4 can be removed
later if needed
free space
record 3
record 1

block header
RM block header
dir entry 0 RM block with 1 record,
lots of free space available
How can we find this block

later so we can put more
records in it?
free space
record 0

RM chain
front back
singly linked list of RM blocks with useable free space
purpose: keep track of those blocks with reasonable

amount of space so they can be filled
toss limit: minimum free space to keep block on RM

chain
RM blocks not on chain not used for adding records

RM chain
front back
When deleting, updating,

After change, if block not on chain and (free space >
toss limit) block is put on front of chain
When creating, looking for space, search RM chain
First block removed from front of chain
If (free space < create limit) or
if record does not fit and (free space <= toss limit)

Applying this stuff
•  Reduce create limit for tables (or LOB columns)

whose rows never expand after creation (less
wasted space)
•  Increase create limit for tables (or LOB columns)
that expand after creation
•  Default value is
–  75 for 1 and 2 k blocks

Applying this stuff
•  Toss limit must be set

–  higher than create limit
–  smaller than average row size
•  Default is
–  300 for 4 and 8k blocks

the
-recspacesearchdepth
parameter

-recspacesearchdepth
RM chain back
front
search for space begins at first block of RM chain
-respacesearchdepth parameter determines how many

blocks to move from front to back of chain
UNITS: tenths of cluster size, e.g. 9/10
Default setting in 10.2B is 5 (5/10 or 50%)
We believe you should not have to adjust this

the
index block
fill-percentage

Index B-Tree Terminolgy
root
Block’s DBKEY Type Chain Backup Ctr
Next DBKEY in Chain Block Update Counter
Top Bot Inde Reserved

x No.
Num Bytes Dummy Entry . . .
Entries Used
. . . Compressed Index Entries . . .
ooo
Free Space

Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter
internal
Next DBKEY in Chain Block Update Counter
Next DBKEY in Chain Block Update Counter Top Bot Index Reserved
No.
Top Bot Inde Reserved
Top Bot Index Reserved Num Bytes Dummy Entry . . .
x No.
Entries Dummy Entry
Used . . .
No. Num Bytes
Num Bytes Dummy Entry . . .Used . . . Compressed Index Entries . . .
Entries
Entries Used . . . Compressed Index Entries . . .
ooo
ooo
ooo
. . . Compressed Index Entries . . . Free Space
Free Space
Free Space

Block’s DBKEY Type Chain Backup CtrNext DBKEY in Chain Block Update Counter
Next DBKEY in Chain Block Update Counter Top Bot Index Reserved
Block’s DBKEY Type Chain Backup Ctr No.
Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Top Bot Index Reserved
Top Bot Index No.
Reserved Num Bytes Dummy Entry . . .
Block’s DBKEY Type Chain Backup Ctr Next DBKEY in Chain Block Update Counter Entries DummyUsed
leaf
Top Bot Index Reserved No. Num Bytes Entry . . .
Num Bytes Entries Dummy UsedEntry . . . . . . Compressed Index Entries . . .
No.
Next DBKEY in Chain Block Update Counter Top Bot Inde Reserved . . . Compressed Index Entries . . .
Num Bytes Entries Dummy Entry Used...
Next DBKEY in Chain Block Update Counter Top Bot Index Reserved x No.
Top Bot Index Reserved No. Num Bytes Dummy Entry Entries
... Used
Num Bytes . . . Compressed Index Entries . . . ooo
Top Bot Index Reserved No. EntriesDummy Entry ...
Used
No. Num Bytes Entries DummyUsed Entry . . . . . . Compressed Index Entries . . . ooo
Num Bytes Entries DummyUsed Entry . . . . . . Compressed Index Entries . . . ooo
Entries Used . . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . .
. . . Compressed Index Entries . . . ooo . . . Compressed Index Entries . . .
ooo . . . Compressed Index Entries . . . Free Space
Free Space
Free Space
records
Index Data (ix) Blocks
block header index blocks do not contain

index block header records
record create limit does not

compressed apply to index blocks
index
entries RM chain toss limit does not
apply to index blocks
maximum records per block

does not apply to index blocks
free space
Put indexes in different area
from table rows

block header Index entries ordered by key

index block header value. Cannot just put new
entries at end.
block splits are required when

there is not enough free space
for a new entry
compressed
index
entries

block header block header

index block header index block header
compressed compressed
index index
entries entries
free space free space
when a block is split, about half the entries are moved to new block
Applying this stuff
index block fill-percentage determines

block header minimum amount of free space to leave
index block header when
indexes are built with idxbuild
proutil foo -C idxbuild -pfactor 80
compacted with index compact
proutil foo -C idxcompact <index> 80
compressed
index purpose: leave some expansion room so
entries every new index entry addition does not
cause a split. Fewer splits are needed
when there is free space. Use 80.
Minor advantage: Set rows-per-block to 1
for areas that only contain indexes – but
min free space very fast database growth and much
wasted space if you accidentally put a
table in that area.
the
-hash
parameter

Database Buffer Pool Hash Table
buffer
header
buffer
how do we locate a buffer we want?
we use the hash table and follow the chain
-hash parameter determines number of
Buffer entries in hash table
Hash
Table entries ordered by dbkey

Database Buffer Pool Hash Chains
Buffer
Hash if hash table (-hash) small compared to
Table number of buffers (-B), chains become long
and so do searches
default size is prime number close to -B / 4
Do NOT change unless you know how
the
-lkhash
parameter

The Lock Hash Table
Free
List
Active & Queued (Waiting) Lock Entries
X SQ SQ SQ SQ
how do we find a lock?

S we use the hash table and
follow the hash chain
X -lkhash determines number
of entries in lock hash table
Lock
entries ordered by rowid
Hash
Do NOT change
Table
before-image log cluster reuse

BI Log Clusters
cluster 0
cluster 1
cluster 2
cluster 3
Identical size, linked into ring Start with 4 clusters

Composed of “bi blocks” Start filling at cluster 0
BI Log Clusters
cluster 0
cluster 1
cluster 2
cluster 3
Identical size, linked into ring Start filling at cluster 0

Composed of “bi blocks” when full, go to next in ring
Start with 4 clusters (cluster 1)
BI Log Clusters
cluster 0
cluster 1
cluster 2
cluster 3
When cluster 3 fills, all are full, we must choose:

- reuse next in ring (cluster 0), or
- insert a new one

BI Log cluster reuse rules
•  Next in ring (e.g. cluster 0) not in use by active

txn
•  All data blocks modified when notes were written
(e.g. to cluster 0) have been written to disk
•  Cluster is at least -G seconds old

BI Log Clusters - reuse 0
cluster 4
cluster 1
cluster 2
cluster 3
When cluster 0 is reused, it becomes cluster 4

BI Log Clusters
cluster 4
cluster 1
cluster 2
cluster 3
When cluster 4 fills, all are full, so we must choose:

- reuse next in ring (cluster 1), or
- insert a new one

BI Log Clusters - insert new one
cluster 4
cluster 5
cluster 1
cluster 2
cluster 3
we insert a new one - cluster 5, between 4 and 1

BI Log Clusters - cluster order
different logical and physical cluster order
logical ordering:
follow the ring
ring ordered by “bi-clock” time
leftmost cluster is oldest
rightmost cluster is newest
physical ordering:
initial clusters match logical order on disk
leftmost cluster probably not first on disk
new cluster always added after existing on disk
can be anywhere in the ring

How many BI clusters exist?
find _AreaStatus where

_AreaStatus-Areanum = 3.
find _dbStatus
display _AreaStatus-Hiwater *
_dbStatus._DbStatus-BiBlkSize /
_dbStatus-BiClSize /
1024
.
Can’t tell how many are active though

scanning the after-image log

Transaction Log Records (aka “Notes”)
transaction 1’s
T1: begin
actions
T1: action 1
T1: action 2 transaction 2’s
T2: begin
actions
T1: action 3
transaction 3’s
T3: begin
T1: action 4
actions
T3: action 1
T3: action 2
T2: action 1
T1: end
T2: action 2
T3: end
T2: action 3
T2: action 4
before-image log and after-image log get the same notes (almost)
•  Logical database operations cause physical changes

•  Log record (aka “note”) generated for every physical
change to database.
•  Each describes one change to one database block.
–  Almost - There are also log records that describe changes to
purely memory-resident data structures like the transaction table
•  Notes apply to a specific version of a block
•  Some operations require more than one change
–  Index splits, multi-block records, chaining operations
•  Notes written in same order changes are executed.
•  Notes from concurrent transactions are mixed together.

•  Each log record (aka “note”) contains:

–  Area number
–  Database block number (its “dbkey”)
–  Database block version number
–  Note type - identifies operation to do
•  10.2B has 94 note types
–  Any information needed to undo the operation

•  in case we have to roll back
–  Any information needed to redo the operation

•  in case we lose what is in memory

rfutil aimage scan verbose
•  Find what time an ‘evil’ transaction was started to

assist in knowing how far to Roll Forward
•  To find what transactions were active during an
online backup because those trx would be
reversed if the backup was restored
•  Find out what userid is causing the BI file to grow
and some details about what that process is
doing

Prerequisites:
•  After Imaging is enabled
•  V10.1A and above works best because there is a bug in
the output of aimage scan verbose on older versions of
Progress. The transaction index (Trid) overflows to a
negative number when it reaches 32767. There is a
workaround (editing the PROMSGs file) described in
Progress Solution P72241.
•  The AI logs being scanned are small enough that the
output of aimage scan verbose doesn’t use up all of your
available disk space

•  Details about Notes are not published but there

are two important Note types that we will share
here:
–  RL_TBGN = Recovery Log Trx Begin
–  RL_TEND = Recovery Log Trx End
–  Trid = Transaction Id

Trid: 17 dbkey = 0 update counter = 0 (12530)

Trid: 18 Wed Jan 27 16:54:06 2010. (2598)
Trid: 18 User Id: Dan (12531)
Trid: 18 code = RL_TBGN version = 1 (12528)
Trid: 18 code = RL_RMCR version = 2 (12528)
Trid: 18 area = 7 dbkey = 480 update counter = 5 (12529)
Trid: 18 code = RL_CXINS version = 2 (12528)
Trid: 18 Wed Jan 27 16:54:06 2010. (2598)
Trid: 18 code = RL_TEND version = 1 (12528)
Trid: 19 Wed Jan 27 16:54:06 2010. (2598
Locks, latches, and semaphores

Kinds of locks
Type Usage Duration

Schema DB data definitions Session
Table and row DB data Transaction,
locks record scope
Buffer locks Buffer pool, ai, bi Milliseconds,
buffers I/O operations
Latches Fast, low- Some
overhead shared nanoseconds
memory data locks
Semaphores used with all the above except latches

Latch tuning parameters
•  -spin
–  # retries before nap
•  Set to 10,000 or …
•  Can be changed online
•  -nap
–  Initial amount to nap (default 10 ms)
•  -napmax
–  Max amount to nap (default 5 sec)
•  -napinc and -napstep
–  Both obsolete and ignored

the
-shmsegsize
parameter

Process address space
reserved When you connect to a

database in self-serving
mode, its shared-memory
shared lib
segments must be
mapped into your
heap address space
stacks used memory
free memory
static data
code

reserved segments are mapped into

free slots in process
shared-memory segment
address space
shared lib
heap
stacks used memory

free memory
shared memory
static data
code

reserved if a segment is larger than

biggest free slot, it won’t fit
shared lib -shmsegsize limits

segment size when
heap database is started
stacks
Shared-memory segment
static data
code

the
-directio
parameter

-directio
•  Should you use –directio ?

•  Maybe – but the only way to tell is to test
your system with your application and I/O
workload
•  Warning: do not put in startup.pf
–  Things you don’t expect may be affected

-directio
•  Some older Progress documentation

states that -directio only applies to Sequent
and Data General systems. This was true
in V6 and V7 but on V8 and later, -directio
can be used on all platforms.
•  Not recommended for:
•  Windows
•  Linux (confirmed in multiple Bunker
tests)
That’s all for today, except

Questions
email: gus@progress.com

Practical DB Essentials Mar 2010

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Practical DB Essentials Mar 2010

Enviado por

Direitos autorais:

Formatos disponíveis

Practical

Gus Björklund, Wizard, Progress Software

Progress Software User Group Finland

Explain selected internals to offer practical

Enhance DBA’s knowledge to optimize

2 Practical OpenEdge RDBMS Essentials

3 Practical OpenEdge RDBMS Essentials

4 Practical OpenEdge RDBMS Essentials

5 Practical OpenEdge RDBMS Essentials

free space free space expands

6 Practical OpenEdge RDBMS Essentials

free space free space expands

records are packed at

7 Practical OpenEdge RDBMS Essentials

8 Practical OpenEdge RDBMS Essentials

9 Practical OpenEdge RDBMS Essentials

10 Practical OpenEdge RDBMS Essentials

block header Set rows-per-block to:

free space Surprisingly, there are

11 Practical OpenEdge RDBMS Essentials

12 Practical OpenEdge RDBMS Essentials

13 Practical OpenEdge RDBMS Essentials

14 Practical OpenEdge RDBMS Essentials

15 Practical OpenEdge RDBMS Essentials

How can we find this block

16 Practical OpenEdge RDBMS Essentials

singly linked list of RM blocks with useable free space

purpose: keep track of those blocks with reasonable

toss limit: minimum free space to keep block on RM

RM blocks not on chain not used for adding records

17 Practical OpenEdge RDBMS Essentials

When deleting, updating,

18 Practical OpenEdge RDBMS Essentials

• Reduce create limit for tables (or LOB columns)

19 Practical OpenEdge RDBMS Essentials

• Toss limit must be set

20 Practical OpenEdge RDBMS Essentials

21 Practical OpenEdge RDBMS Essentials

search for space begins at first block of RM chain

-respacesearchdepth parameter determines how many

UNITS: tenths of cluster size, e.g. 9/10

Default setting in 10.2B is 5 (5/10 or 50%)

We believe you should not have to adjust this

23 Practical OpenEdge RDBMS Essentials

Next DBKEY in Chain Block Update Counter

Top Bot Inde Reserved

. . . Compressed Index Entries . . .

Block’s DBKEY Type Chain Backup Ctr

Block’s DBKEY Type Chain Backup Ctr

block header index blocks do not contain

record create limit does not

maximum records per block

25 Practical OpenEdge RDBMS Essentials

block header Index entries ordered by key

block splits are required when

26 Practical OpenEdge RDBMS Essentials

block header block header

free space free space

index block fill-percentage determines

29 Practical OpenEdge RDBMS Essentials

30 Practical OpenEdge RDBMS Essentials

32 Practical OpenEdge RDBMS Essentials

how do we find a lock?

•  Reduce create limit for tables (or LOB columns)

•  Toss limit must be set

•  Next in ring (e.g. cluster 0) not in use by active

•  Logical database operations cause physical changes

•  Each log record (aka “note”) contains:

–  Any information needed to undo the operation

–  Any information needed to redo the operation

•  Find what time an ‘evil’ transaction was started to

•  Details about Notes are not published but there

•  Should you use –directio ?

•  Some older Progress documentation