1 2008 Julian Dyke

juliandyke.com

Advanced Queuing
Internals
Julian Dyke
Independent Consultant
Web Version - November 2008
2008 Julian Dyke
juliandyke.com 2
Agenda
Introduction
Single Consumer Queues
Multiple Consumer Queues
Recipients
Subscribers
Exception Queues
Array Payloads
Buffered Messages
Spilled Messages
Performance
2008 Julian Dyke
juliandyke.com 3
Introduction
Advanced Queuing
Advanced Queuing
Introduced in Oracle 8.0
Extended and enhanced in most subsequent versions
Supports Oracle Streams in Oracle 9.2 and above
Supports buffered messages in Oracle 10.2 and above

Allows messages to be enqueued and dequeued from queues that are
managed by the database

Each queue is associated with a queue table
Properties of queue table specify behaviour of associated queues

Each queue has a payload which can be:
RAW - only messages of type RAW can be enqueued
Object type- only messages of the specified type can be enqueued
ANYDATA - messages with any object type can be enqueued

2008 Julian Dyke
juliandyke.com 4
Introduction
Advanced Queuing
By default messages are dequeued in the order they are enqueued
Default behaviour can be overridden in several ways

Messages can be persistent or buffered
Persistent messages
Stored in queue table
Survive an instance restart

Buffered messages
Stored in SGA
Can be spilled to queue table
Lost during instance restart

Messages can be immediate or on-commit
Immediate messages are committed immediately when they are
enqueued/dequeued
On-commit messages are committed with the enqueuing transaction
Buffered messages can only be immediate.

2008 Julian Dyke
juliandyke.com 5
Introduction
Advanced Queuing
Queue tables can be created for single or multiple consumers
Messages in single consumer queue tables can only be dequeued once
Messages in multiple consumer queue tables can be dequeued multiple
times by multiple consumers

Multiple consumer queue tables can be associated with
Multiple recipients
Multiple subcribers

Subscribers can:
Specify rules to control which messages they dequene
Specify transformations to be performed against dequeued data

2008 Julian Dyke
juliandyke.com 6
Advanced Queuing
Introduction
Sessions can listen for messages on multiple queues
Session is notified when a message arrives on any of the target queues

Queue messages can be propagated from one queue to another
In the same database
In different databases
Propagation can be immediate or at specified intervals

Transformation converts payload from one object type to another
Queue data can be transformed when messages are:
Enqueued
Propagated
Dequeued
Messages must be transformed using DBMS_TRANSFORM API

2008 Julian Dyke
juliandyke.com 7
Introduction
Payloads
Queue payloads can be
RAW
Abstract data types
ANYDATA

Abstract data types
Maximum number of attributes is limited to 900
For example:
CREATE TYPE type1 AS OBJECT
(
c1 NUMBER,
c2 NUMBER,
c3 NUMBER
);
/
2008 Julian Dyke
juliandyke.com 8
Single Consumer Queues
Queue Tables
Every queue must be associated with a queue table
Queue table defines properties of queue
Reported in DBA_QUEUE_TABLES
Managed using DBMS_AQADM
For example:

DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');
By default single consumer queues will be created
Each queue table can contain multiple queues
Queues inherit properties of queue table
Each queue table block contains blocks for one queue
2008 Julian Dyke
juliandyke.com 9
Single Consumer Queues
Queue Table Columns (RAW Payload)
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');
Col# Intcol# Column Name Data Type
1 1 Q_NAME VARCHAR2(30)
2 2 MSG_ID RAW(16)
3 3 CORRID VARCHAR2(128)
4 4 PRIORITY NUMBER
5 5 STATE NUMBER
6 6 DELAY TIMESTAMP(6)
7 7 EXPIRATION NUMBER
8 8 TIME_MANAGER_INFO TIMESTAMP(6)
9 9 LOCAL_ORDER_NO NUMBER
10 10 CHAIN_NO NUMBER
11 11 CSCN NUMBER
12 12 DSCN NUMBER
13 13 ENQ_TIME TIMESTAMP(6)
14 14 ENQ_UID VARCHAR2(30)
15 15 ENQ_TID VARCHAR2(30)
Col# Intcol# Column Name Data Type
16 16 DEQ_TIME TIMESTAMP(6)
17 17 DEQ_UID VARCHAR2(30)
18 18 DEQ_TID VARCHAR2(30)
19 19 RETRY_COUNT NUMBER
20 20 EXCEPTION_QSCHEMA VARCHAR2(30)
21 21 EXCEPTION_QUEUE VARCHAR2(30)
22 22 STEP_NO NUMBER
23 23 RECIPIENT_KEY NUMBER
24 24 DEQUEUE_MSG_ID RAW(16)
25 25 SENDER_NAME VARCHAR2(30)
26 26 SENDER_ADDRESS VARCHAR2(1024)
27 27 SENDER_PROTOCOL NUMBER
28 28 USER_DATA BLOB
29 29 USER_PROP SYS.ANYDATA
Queue table QT1 contains the following columns:

2008 Julian Dyke
juliandyke.com 10
Single Consumer Queues
Queue Table Columns (Object Payload)
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');
Col# Intcol# Column Name Data Type
1 1 Q_NAME VARCHAR2(30)
2 2 MSG_ID RAW(16)
3 3 CORRID VARCHAR2(128)
4 4 PRIORITY NUMBER
5 5 STATE NUMBER
6 6 DELAY TIMESTAMP(6)
7 7 EXPIRATION NUMBER
8 8 TIME_MANAGER_INFO TIMESTAMP(6)
9 9 LOCAL_ORDER_NO NUMBER
10 10 CHAIN_NO NUMBER
11 11 CSCN NUMBER
12 12 DSCN NUMBER
13 13 ENQ_TIME TIMESTAMP(6)
14 14 ENQ_UID VARCHAR2(30)
15 15 ENQ_TID VARCHAR2(30)
16 16 DEQ_TIME TIMESTAMP(6)
Col# Intcol# Column Name Data Type
17 17 DEQ_UID VARCHAR2(30)
18 18 DEQ_TID VARCHAR2(30)
19 19 RETRY_COUNT NUMBER
20 20 EXCEPTION_QSCHEMA VARCHAR2(30)
21 21 EXCEPTION_QUEUE VARCHAR2(30)
22 22 STEP_NO NUMBER
23 23 RECIPIENT_KEY NUMBER
24 24 DEQUEUE_MSG_ID RAW(16)
25 25 SENDER_NAME VARCHAR2(30)
26 26 SENDER_ADDRESS VARCHAR2(1024)
27 27 SENDER_PROTOCOL NUMBER
28 28 USER_DATA TYPE1
28 29 SYS_NC00029$ NUMBER
28 30 SYS_NC00030$ NUMBER
28 31 SYS_NC00031$ NUMBER
29 32 USER_PROP SYS.ANYDATA
Queue table QT1 contains the following columns:

2008 Julian Dyke
juliandyke.com 11
Single Consumer Queues
Database Objects (RAW payload)
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','RAW');
The following objects will be created (object IDs and constraint IDs will vary):

Object ID Object Name Object Type
70581 QT1 TABLE
70582 SYS_LOB0000070581C00028$$ LOB
70583 SYS_IL0000070581C00028$$ LOB INDEX
70584 SYS_LOB0000070581C00029$$ LOB
70585 SYS_IL0000070581C00029$$ LOB INDEX
70586 SYS_C009433 INDEX
70587 AQ$_QT1_T INDEX
70588 AQ$_QT1_I INDEX
70589 QT70581_BUFFER VIEW
70590 AQ$QT1 VIEW
70591 AQ$_QT1_F VIEW
70592 AQ$_QT1_E QUEUE
LOB columns are used for USER_DATA and USER_PROP columns

2008 Julian Dyke
juliandyke.com 12
Single Consumer Queues
Database Objects (Object Payload)
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');
The following objects will be created (object IDs and constraint IDs will vary):

Object ID Object Name Object Type
70581 QT1 TABLE
70582 SYS_LOB0000070581C00032$$ LOB
70583 SYS_IL0000070581C00032$$ LOB INDEX
70584 SYS_C009433 INDEX
70585 AQ$_QT1_T INDEX
70586 AQ$_QT1_I INDEX
70587 QT70581_BUFFER VIEW
70589 AQ$QT1 VIEW
70590 AQ$_QT1_F VIEW
70591 AQ$_QT1_E QUEUE
LOB column is used for USER_PROP column

2008 Julian Dyke
juliandyke.com 13
Single Consumer Queues
Index Columns
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT1','TYPE1');
The following indexes will be created by default (constraint IDs will vary):

Index Name Column # Column Name
SYS_C009436 1 MSGID
AQ$_QT1_I 1 Q_NAME
2 STATE
3 ENQ_TIME
4 STEP_NO
5 CHAIN_NO
6 LOCAL_ORDER_NO
AQ$_QT1_T 1 TIME_MANAGER_INFO
2008 Julian Dyke
juliandyke.com 14
Single Consumer Queues
Sort Lists
The columns indexed by AQ$_QTI are determined by the SORT_LIST
parameter.
Possible values are
enq_time (default)
priority
priority,enq_time
enq_time,priority
Must be defined when queue table is created
Cannot be subsequently altered
Column # enq_time priority priority,enq_time enq_time,priority
1 Q_NAME Q_NAME Q_NAME Q_NAME
2 STATE STATE STATE STATE
3 ENQ_TIME PRIORITY PRIORITY ENQ_TIME
4 STEP_NO CHAIN_NO ENQ_TIME STEP_NO
5 CHAIN_NO LOCAL_ORDER_NO STEP_NO PRIORITY
6 LOCAL_ORDER_NO CHAIN_NO CHAIN_NO
7 LOCAL_ORDER_NO LOCAL_ORDER_NO
DBMS_AQADM.CREATE_QUEUE_TABLE
('QT1','TYPE1',SORT_LIST=>"priority,enq_time");
2008 Julian Dyke
juliandyke.com 15
Single Consumer Queues
Views
Two views are created for each queue table

For example for QT1 (object ID =70581):

QT<object_id>_BUFFER
e.g. QT70581_BUFFER
based on X$BUFFER2

AQ$_<queue_table_name>_F
e,g. AQ$_QT1_F
based on QT1 and ALL_DEQUEUE_QUEUES

2008 Julian Dyke
juliandyke.com 16
Single Consumer Queues
Views
Two views are created for each queue table

For example for QT1 (object ID =70581)
QT70581_BUFFER
AQ$_QT1_F

SELECT
"ADDR", "INDX", "INST_ID", "OBJNO", "QUEUE_ID", "MSGID", "CORRID",
"SEQUENCE_NUM", "MSG_NUM", "STATE", "PRIORITY", "EXPIRATION",
"ENQ_TIME", "ENQ_UID", "ENQ_USER_NAME", "RETRY_COUNT",
"SENDER_NAME", "SENDER_ADDRESS", "SENDER_PROTOCOL",
"DEQUEUE_MSGID", "SRCSEQUENCE_NUM", "SUBSCRIBER_ID",
"EXCEPTIONQ_SCHEMA", "EXCEPTIONQ_NAME"
FROM X$BUFFER2
WHERE objno = 70581;
QT70581_BUFFER is defined as follows:

2008 Julian Dyke
juliandyke.com 17
Single Consumer Queues
Views
AQ$_QT1_F is defined as follows:

SELECT
qt.q_name Q_NAME, qt.rowid ROW_ID, qt.msgid MSGID, qt.corrid CORRID,
qt.priority PRIORITY, qt.state STATE, qt.delay DELAY, qt.expiration EXPIRATION,
qt.enq_time ENQ_TIME, qt.enq_uid ENQ_UID, qt.enq_tid ENQ_TID,
qt.deq_time DEQ_TIME, qt.deq_uid DEQ_UID, qt.deq_tid DEQ_TID,
qt.retry_count RETRY_COUNT, qt.exception_qschema EXCEPTION_QSCHEMA,
qt.exception_queue EXCEPTION_QUEUE, qt.cscn CSCN, qt.dscn DSCN,
qt.chain_no CHAIN_NO, qt.local_order_no LOCAL_ORDER_NO,
qt.time_manager_info TIME_MANAGER_INFO, qt.step_no STEP_NO,
qt.user_data USER_DATA , qt.sender_name SENDER_NAME,
qt.sender_address SENDER_ADDRESS, qt.sender_protocol SENDER_PROTOCOL,
qt.dequeue_msgid DEQUEUE_MSGID, 'PERSISTENT' DELIVERY_MODE,
0 SEQUENCE_NUM, 0 MSG_NUM, qo.qid QUEUE_ID,
qt.user_prop USER_PROP
FROM
"QT1" qt,
ALL_DEQUEUE_QUEUES qo
WHERE qt.q_name = qo.name
AND qo.owner = 'US01'
WITH READ ONLY;
2008 Julian Dyke
juliandyke.com 18
Single Consumer Queues
Queues
Every queue must be associated with a queue table
Queue table must exist before queue can be created


Every queue has a type which can be:
NORMAL (default)
EXCEPTION
NON PERSISTENT

Non persistent queues are deprecated in Oracle 10.2
Use buffered messages instead
DBMS_AQADM.CREATE_QUEUE
(queue_name => 'Q1',queue_table => 'QT1');
This statement might create the following object:
Object ID Object Name Object Type
70793 Q1 QUEUE
2008 Julian Dyke
juliandyke.com 19
Single Consumer Queues
Enqueue
The following code enqueues a message of TYPE1 on a single consumer
queue:
DECLARE
message TYPE1;
msgprop dbms_aq.message_properties_t;
enqopt dbms_aq.enqueue_options_t;
enq_msgid RAW(16);
BEGIN
message := new TYPE1 (10001,20001,30001);
msgprop.expiration :=DBMS_AQ.NEVER
dbms_aq.enqueue
(
queue_name => 'Q1',
enqueue_options => enqopt,
message_properties => msgprop,
payload => message,
msgid => enq_msgid
);
END;
2008 Julian Dyke
juliandyke.com 20
Single Consumer Queues
Enqueue
The enqueue processes executes the following recursive statement:
insert into "US01"."QT1" (q_name, msgid, corrid, priority, state, delay, expiration,
time_manager_info, local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid,
retry_count, exception_qschema, exception_queue, recipient_key, dequeue_msgid,
user_data, sender_name, sender_address, sender_protocol, user_prop, cscn, dscn)
values (:1, :2, :3, :4, :5, :6, :7, :8, :9, :10, :11, :12, :13, :14, 0, :15,:16, :17, :18, :19, :20, :21,
:22, :23, :24, :25)
In Oracle 11.1 this statement uses the LOAD TABLE CONVENTIONAL
operation
STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL (cr=1 pr=5 pw=5
time=0 us)')
For this statement the following objects are modified
Object ID Object Name Object Type
70581 QT1 TABLE
70586 SYS_C009433 INDEX
70588 AQ$_QT1_1 INDEX
2008 Julian Dyke
juliandyke.com 21
Single Consumer Queues
Dequeue
The following code dequeues a message of TYPE1 from a single consumer
queue:
DECLARE
message TYPE1;
msgprop dbms_aq.message_properties_t;
deqopt dbms_aq.dequeue_options_t;
deq_msgid RAW(16);
BEGIN
dbms_aq.dequeue
(
queue_name => 'Q1',
dequeue_options => deqopt,
message_properties => msgprop,
payload => message,
msgid => deq_msgid
);
END;
2008 Julian Dyke
juliandyke.com 22
Single Consumer Queues
Dequeue
The dequeue processes executes the following recursive statement:
select /*+ FIRST_ROWS(1) */ tab.rowid, tab.msgid, tab.corrid, tab.priority, tab.delay,
tab.expiration, tab.retry_count, tab.exception_qschema, tab.exception_queue, tab.chain_no,
tab.local_order_no, tab.enq_time, tab.time_manager_info, tab.state, tab.enq_tid, tab.step_no,
tab.sender_name, tab.sender_address, tab.sender_protocol, tab.dequeue_msgid,
tab.user_prop, tab.user_data
from "US01"."QT1" tab
where q_name = :1 and (state = :2 )
order by q_name, state, enq_time, step_no, chain_no, local_order_no
for update skip locked
The statement selects all rows in the queue specified by :1 with a state of :2
The FIRST_ROWS(1) hint is used to optimize the plan
The statement locks any rows to be deleted
This will generate undo/redo
The statement uses the FOR UPDATE SKIP LOCKED clause to skip any rows
still locked by ongoing transactions
2008 Julian Dyke
juliandyke.com 23
Single Consumer Queues
Dequeue
Execution plan for SELECT FOR UPDATE statement is:
STAT #3 id=1 cnt=1 pid=0 pos=1 obj=0 op='FOR UPDATE (cr=7 pr=2 pw=2 time=0 us)'
STAT #3 id=2 cnt=1 pid=1 pos=1 obj=0 op='SORT ORDER BY (cr=7 pr=0 pw=0 time=0 us
cost=4 size=2759 card=1)'
STAT #3 id=3 cnt=1 pid=2 pos=1 obj=70581 op='TABLE ACCESS FULL QT1 (cr=7 pr=0 pw=0
time=0 us cost=3 size=2759 card=1)'
As queue grows, object statistics must be gathered to
ensure AQ$_QT1_I index is used
prevent full table scans on QT1

Rows identified by SELECT FOR UPDATE are deleted using:
delete /*+ CACHE_CB("QT9") */ from "US01"."QT1" where rowid = :1
Execution plan for DELETE statement is:
STAT #7 id=1 cnt=1 pid=0 pos=1 obj=0 op='DELETE QT1 (cr=1 pr=2 pw=2 time=0 us)'
STAT #7 id=2 cnt=1 pid=1 pos=1 obj=70581 op='TABLE ACCESS BY USER ROWID QT1
(cr=1 pr=0)'
2008 Julian Dyke
juliandyke.com 24
Queues
Exception Queues
A default exception queue is created for each queue table
Exception messages will be moved to default queue unless a user-
defined exception queue has been specified when the message is
enqueued

For example to create a user-defined exception queue
DBMS_AQADM.CREATE_QUEUE
(queue_name => 'Q1',queue_table => 'QT1');
DBMS_AQADM.CREATE_QUEUE
(queue_name => 'Q1E',queue_table => 'QT1'
queue_type => DBMS_AQADM.EXCEPTION_QUEUE);
Object ID Object Name Object Type Queue Type
70793 Q1 QUEUE NORMAL
70794 Q1E QUEUE EXCEPTION
To check number of rows in each queue:
SELECT q_name, COUNT(*)
FROM qt1
GROUP BY q_name;
2008 Julian Dyke
juliandyke.com 25
Queues
Exception Queues
Exceptions will be written to user-defined exception queue if it is specified
during enqueue operation
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_enqopt dbms_aq.enqueue_options_t;
l_enq_msgid RAW(16);
BEGIN
l_payload := new TYPE1 (10001,20001,30001);
l_msgprop.expiration := 60;
l_msgprop.exception_queue := 'Q1E';
dbms_aq.enqueue
(
queue_name => 'Q1',
enqueue_options => l_enqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_enq_msgid
);
END;
Message will expire after 60 seconds
Expired message will be move to exception queue Q1E by queue monitor
2008 Julian Dyke
juliandyke.com 26
Multiple Consumer Queues
Introduction
There are two ways to use multiple consumer queues
Multiple Recipients
Multiple Subscribers
BEGIN
dbms_aqadm.create_queue_table
('QT3','TYPE1',multiple_consumers=>TRUE)
dbms_aqadm.create_queue ('Q3','QT3');
dbms_aqadm.start_queue ('Q3');
END;
/
The same queue definitions are used for both examples:
2008 Julian Dyke
juliandyke.com 27
Multiple Consumer Queues
Database Objects
DBMS_AQADM.CREATE_QUEUE_TABLE
('QT1','TYPE1',MULTIPLE_CONSUMERS=>TRUE);
Object ID Object Name Object Type
70756 QT1 TABLE
70757 SYS_LOB0000070581C00032$$ LOB
70758 SYS_IL0000070581C00032$$ LOB INDEX
70759 SYS_C009457 INDEX
70760 AQ$_QT1_S TABLE
70761 SYS_C009460 INDEX
70762 AQ$_QT1_N SEQUENCE
70763 AQ$QT1_S VIEW
70764 AQ$_QT1_V EVAL CTXT
70765 AQ$_QT1_T TABLE
70766 SYS_IOT_TOP_70765 INDEX
Object ID Object Name Object Type
70767 AQ$_QT1_H TABLE
70768 SYS_IOT_TOP_70767 INDEX
70769 AQ$_QT1_G TABLE
70770 SYS_IOT_OVER_70769 TABLE
70771 SYS_IOT_TOP_70769 INDEX
70772 AQ$_QT1_I TABLE
70773 SYS_IOT_TOP_70772 INDEX
70774 QT70756_BUFFER VIEW
70775 AQ$QT1 VIEW
70776 AQ$_QT1_F VIEW
70777 AQ$_QT1_E QUEUE
The following objects will be created (object IDs and constraint IDs will vary):

2008 Julian Dyke
juliandyke.com 28
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_T e.g AQ$_QT3_T
IOT used queue monitor to manage timed operations
Single consumer queues use TIME_MANAGER_INFO column only

AQ$_<queue_table_name>_I
IOT that maintains state for dequeue operations
One row per message per recipient/subscriber

AQ$_<queue_table_name>_S
Heap table containing information about subscribers

AQ$_<queue_table_name>_H
IOT used to store dequeue history
One row per message per recipient/subscriber

AQ$_<queue_table_name>_G
IOT correlating messages to subscriber signatures


2008 Julian Dyke
juliandyke.com 29
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_T
IOT used queue monitor to manage timed operations
e.g. AQ$_QT3_T
Column Name Data Type
NEXT_DATE TIMESTAMP
TXN_ID VARCHAR2(30)
MSGID RAW(16)
ACTION NUMBER
First 3 columns form primary key

Values for the ACTION column include:
0 - delay
1 - expiration
2 - delay

Single consumer queues use TIME_MANAGER_INFO column only
2008 Julian Dyke
juliandyke.com 30
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_I
IOT that maintains state for dequeue operations



Column Name Data Type
SUBSCRIBER NUMBER
NAME VARCHAR2(30)
QUEUE# NUMBER
MSG_ENQ_TIME TIMESTAMP
MSG_STEP_NO NUMBER
MSG_CHAIN_NO NUMBER
MSG_LOCAL_ORDER_NO NUMBER
MSG_ID RAW(16)
HINT ROWID
SPARE RAW(16)
First eight columns form primary key
HINT and SPARE columns are stored in IOT overflow segment
2008 Julian Dyke
juliandyke.com 31
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_S
Heap table containing information about subscribers

Column Name Data Type
SUBSCRIBER_ID NUMBER
QUEUE_NAME VARCHAR2(30)
NAME VARCHAR2(30)
ADDRESS VARCHAR2(1024)
PROTOCOL NUMBER
SUBSCRIBER_TYPE NUMBER
RULE_NAME VARCHAR2(30)
TRANS_NAME VARCHAR2(65)
RULESET_NAME VARCHAR2(65)
NEGATIVE_RULESET_NAME VARCHAR2(65)
CREATION_TIME TIMESTAMP(6)
MODIFICATION_TIME TIMESTAMP(6)
DELETION_TIME TIMESTAMP(6)
SCN_AT_REMOVE NUMBER
2008 Julian Dyke
juliandyke.com 32
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_H
IOT used to store dequeue history


Column Name Data Type
MSGID RAW(16)
SUBSCRIBER# NUMBER
NAME VARCHAR2(30)
ADDRESS# NUMBER
DEQUEUE_TIME TIMESTAMP
TRANSACTION_ID VARCHAR2(30)
DEQUEUE_USER VARCHAR2(30)
PROPAGATED_MSGID RAW(16)
RETRY_COUNT NUMBER
HINT ROWID
SPARE RAW(16)
First four columns form primary key
No IOT overflow segment

2008 Julian Dyke
juliandyke.com 33
Multiple Consumer Queues
Tables
AQ$_<queue_table_name>_G
IOT correlating messages to subscriber signatures
Column Name Data Type
NAME VARCHAR2(30)
ADDRESS# NUMBER
SIGN SYS.AQ$_SIG_PROP
DBS_SIGN SYS.AQ$_SIG_PROP
All columns form primary key
2008 Julian Dyke
juliandyke.com 34
Multiple Consumer Queues
Indexes
By default six indexes are created for each queue table. For example:
Index Name Index Type # Columns Table Name
SYS_C009457 NORMAL 1 QT3
SYS_C009460 NORMAL 1 AQ$_QT3_S
SYS_IOT_TOP_70765 IOT 3 AQ$_QT3_T
SYS_IOT_TOP_70767 IOT 4 AQ$_QT3_H
SYS_IOT_TOP_70769 IOT 4 AQ$_QT3_G
SYS_IOT_TOP_70772 IOT 8 AQ$_QT3_I
Index Name Column # Column Name
SYS_C009457 1 MSGID
SYS_C009460 1 SUBSCRIBER_ID
Index columns for NORMAL indexes are:
Index columns for IOT indexes are shown on previous slides
2008 Julian Dyke
juliandyke.com 35
Multiple Consumer Queues
Views
Three views are created for each queue table
For example for QT3 (object ID = 70756)
QT70756_BUFFER
AQ$_QT3_F
AQ$QT3

<queue_object_Id>_BUFFER e.g QT70756_BUFFER
Similar for single and multiple consumers

AQ$_<queue_table_name>_F e.g AQ$_QT3_F
Similar for single and multiple consumers

AQ$<queue_table_name> views e.g. AQ$QT3
Based on:
Queue table (QT3)
History IOT (AQ$_QT3_H)
Subscriber table (AQ$_QT3_S)
2008 Julian Dyke
juliandyke.com 36
Multiple Consumer Queues
Views
AQ$<queue_table_name> views (AQ$QT3) are based on:
Queue table (QT3)
History IOT (AQ$_QT3_H)
Subscriber table (AQ$_QT3_S)

Abbreviated definition is as follows:
SELECT
<column_list>
FROM
"QT8" qt,
"AQ$_QT8_H" h,
"AQ$_QT8_S" s
WHERE qt.msgid = h.msgid
AND ((h.subscriber# != 0 AND h.subscriber# = s.subscriber_id)
OR (h.subscriber# = 0 AND h.address# = s.subscriber_id))
AND (qt.state != 7 OR qt.state != 9)
WITH READ ONLY;
Best view to understand current state of queue for all subscribers
2008 Julian Dyke
juliandyke.com 37
Multiple Consumer Queues
Views
AQ$QT3 contains the following columns
Column Name Data Type
QUEUE VARCHAR2(30)
MSG_ID RAW(16)
CORR_ID VARCHAR2(128)
MSG_PRIORITY NUMBER
MSG_STATE VARCHAR2(16)
DELAY DATE
DELAY_TIMESTAMP TIMESTAMP(6)
EXPIRATION NUMBER
ENQ_TIME DATE
ENQ_TIMESTAMP TIMESTAMP(6)
ENQ_USER_ID VARCHAR2(30)
ENQ_TXN_ID VARCHAR2(30)
DEQ_TIME DATE
DEQ_TIMESTAMP TIMESTAMP(6)
DEQ_USER_ID VARCHAR2(30)
DEQ_TXN_ID VARCHAR2(30)
Column Name Data Type
RETRY_COUNT NUMBER
EXCEPTION_QUEUE_OWNER VARCHAR2(30)
EXCEPTION_QUEUE VARCHAR2(30)
USER_DATA TYPE1
PROPAGATED_MSGID RAW(16)
SENDER_NAME VARCHAR2(30)
SENDER_ADDRESS VARCHAR2(1024)
SENDER_PROTOCOL NUMBER
ORIGINAL_MSGID RAW(16)
ORIGINAL_QUEUE_NAME VARCHAR2(30)
ORIGINAL_QUEUE_OWNER VARCHAR2(30)
EXPIRATION_REASON VARCHAR2(31)
CONSUMER_NAME VARCHAR2(30)
ADDRESS VARCHAR2(1024)
PROTOCOL NUMBER
2008 Julian Dyke
juliandyke.com 38
Multiple Consumer Queues
Recipients
The following code enqueues a message for three named recipients
DECLARE
l_payload type1;
l_msgprop dbms_aq.message_properties_t;
l_enqopt dbms_aq.enqueue_options_t;
l_enq_msgid RAW(16);
l_recipient_list dbms_aq.aq$_recipient_list_t;
BEGIN
l_recipient_list(1) := sys.aq$_agent ('CONSUMER1',NULL,NULL);
l_recipient_list(2) := sys.aq$_agent ('CONSUMER2',NULL,NULL);
l_recipient_list(3) := sys.aq$_agent ('CONSUMER3',NULL,NULL);
l_msgprop.recipient_list := l_recipient_list;
l_msgprop.expiration := DBMS_AQ.NEVER;
l_payload := new TYPE1 (10001,20001,30001);
dbms_aq.enqueue
(
queue_name => 'Q3',
enqueue_options => l_enqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_enq_msgid
);
END;
2008 Julian Dyke
juliandyke.com 39
Multiple Consumer Queues
Recipients
A recipient list is constructed using AQ$_AGENT objects
In the example all recipients are in the local database

The enqueue operation performs the following actions:
Inserts one row in the queue table (QT3)
Inserts three rows in the queue status table (AQ$_QT3_I)
Inserts three rows in the queue history table (AQ$_QT3_H)

2008 Julian Dyke
juliandyke.com 40
Multiple Consumer Queues
Recipients
The following code dequeues a message for one of the named recipients
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_deqopt dbms_aq.dequeue_options_t;
l_deq_msgid RAW(16);
BEGIN
l_deqopt.consumer_name := 'CONSUMER2';
dbms_aq.dequeue
(
queue_name => 'Q3',
dequeue_options => l_deqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_deq_msgid
);
END;
Notes
A consumer name MUST be specified
The message must have been enqueued specifically for that consumer
2008 Julian Dyke
juliandyke.com 41
Multiple Consumer Queues
Recipients
The dequeue operation performs the following actions
Deletes one row from the queue status IOT (AQ$_QT3_I)
Updates the following columns in one row of the queue history IOT
(AQ$_QT3_H)
DEQUEUE_TIME
TRANSACTION_ID
DEQUEUE_USER
Inserts one row into the queue timer table (AQ$_QT3_T)

The queue monitor (QMNC) process asynchronously checks the timer table
(AQ$_QT3_T) for actions
If any actions are found these are sent to the queue monitor slaves (Q001,
Q002 etc)

When last recipient has dequeued message, queue monitor slaves perform
the following actions
Delete all rows for message in queue history table (AQ$_QT3_H)
Delete row in queue table (QT3) for message
2008 Julian Dyke
juliandyke.com 42
Multiple Consumer Queues
Subscribers
Subscribers must exist for the queue before messages can be enqueued
The following code creates two subscribers for queue Q3
DECLARE
l_subscriber sys.aq$_agent;
BEGIN
l_subscriber := sys.aq$_agent ('SUBSCRIBER1',NULL,NULL);
DBMS_AQADM.ADD_SUBSCRIBER
(
queue_name => 'Q3',
subscriber => l_subscriber
);
l_subscriber := sys.aq$_agent ('SUBSCRIBER2',NULL,NULL);
DBMS_AQADM.ADD_SUBSCRIBER
(
queue_name => 'Q3',
subscriber => l_subscriber
);
END;
Creating a subscriber inserts one row in the AQ$_QT3_S table
2008 Julian Dyke
juliandyke.com 43
Multiple Consumer Queues
Subscribers
The following code enqueues ten messages on Q3
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_enqopt dbms_aq.enqueue_options_t;
l_enq_msgid RAW(16);
BEGIN
FOR f IN 1..10
LOOP
l_payload := new TYPE1 (10000 + f,20000 + f,30000 + f);
l_msgprop.expiration := DBMS_AQ.NEVER;
dbms_aq.enqueue
(
queue_name => 'Q3',
enqueue_options => l_enqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_enq_msgid
);
END LOOP;
END;
2008 Julian Dyke
juliandyke.com 44
Multiple Consumer Queues
Subscribers
The following code dequeues a message from Q3 for SUBSCRIBER1
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_deqopt dbms_aq.dequeue_options_t;
l_deq_msgid RAW(16);
BEGIN
l_deqopt.consumer_name := 'SUBSCRIBER1';
dbms_aq.dequeue
(
queue_name => 'Q3',
dequeue_options => l_deqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_deq_msgid
);
DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1));
DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2));
DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3));
END;
2008 Julian Dyke
juliandyke.com 45
Multiple Consumer Queues
Subscribers
Subscribers can subsequently be added and deleted dynamically
DECLARE
l_subscriber sys.aq$_agent;
BEGIN
l_subscriber := sys.aq$_agent ('SUBSCRIBER3',NULL,NULL);
DBMS_AQADM.ADD_SUBSCRIBER
(
queue_name => 'Q3',
subscriber => l_subscriber
);
DBMS_AQADM.REMOVE_SUBSCRIBER
(
queue_name => 'Q3',
subscriber => l_subscriber
);
END;
New subscribers will only be allowed to dequeue messages that have been
enqueued after the subscriber was added
2008 Julian Dyke
juliandyke.com 46
Array Payloads
Introduction
Payload of a queue can optionally be a VARRAY of object types
For example:
CREATE OR REPLACE TYPE type2 AS VARRAY (10) OF type1;
/
CREATE OR REPLACE TYPE type3 AS OBJECT (c1 type2);
/
Queue table can be created with a TYPE3 payload
It is not possible to create a queue table with a TYPE2 payload
For example:
DBMS_AQADM.CREATE_QUEUE_TABLE ('QT3','TYPE3');
DBMS_AQADM.CREATE_QUEUE ('Q3','QT3');
DBMS_AQADM.START_QUEUE ('Q3');
2008 Julian Dyke
juliandyke.com 47
Array Payloads
Enqueue
DECLARE
l_payload TYPE3;
msgprop dbms_aq.message_properties_t;
enqopt dbms_aq.enqueue_options_t;
enq_msgid RAW(16);
BEGIN
l_payload := new TYPE3 (TYPE2 (
TYPE1 (10001,20001,30001),
TYPE1 (10002,20002,30002),
TYPE1 (10003,20003,30003),
TYPE1 (10004,20004,30004)
));
msgprop.expiration := DBMS_AQ.NEVER;
dbms_aq.enqueue
(
queue_name => 'Q4',
enqueue_options => enqopt,
message_properties => msgprop,
payload => l_payload,
msgid => enq_msgid
);
END;
2008 Julian Dyke
juliandyke.com 48
Array Payloads
Dequeue
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE3;
msgprop dbms_aq.message_properties_t;
deqopt dbms_aq.dequeue_options_t;
deq_msgid RAW(16);
BEGIN
dbms_aq.dequeue
(
queue_name => 'Q4',
dequeue_options => deqopt,
message_properties => msgprop,
payload => l_payload,
msgid => deq_msgid
);
FOR i IN 1..message.c1.COUNT
LOOP
DBMS_OUTPUT.PUT ('C1 = '||TO_CHAR (l_payload.c1(i).c1)||' ');
DBMS_OUTPUT.PUT ('C2 = '||TO_CHAR (l_payload.c1(i).c2)||' ');
DBMS_OUTPUT.PUT ('C3 = '||TO_CHAR (l_payload.c1(i).c3));
DBMS_OUTPUT.NEW_LINE ();
END LOOP;
END;
2008 Julian Dyke
juliandyke.com 49
Buffered Messages
Introduction
In Oracle 10.2 and above messages can be buffered in the SGA
Messages will not be written to database immediately
Messages are spillled to database if:
Number of messages exceeds threshold value
Messages not dequeued within 10 minutes

Buffered messages
Are much faster than persistent queues
Do not guarantee reliability
Cannot form part of a transaction
Do not support (Oracle 11.1)
Message retention / delay
Transaction grouping
Array enqueue / dequeue
Message export / import
2008 Julian Dyke
juliandyke.com 50
Buffered Messages
Introduction
Buffering is specified at message level
Queues can contain both persistent and buffered messages
Payload can be ADT, XML, ANYDATA or RAW
Support for LOB payloads is restricted

BEGIN
dbms_aqadm.create_queue_table ('QT1','TYPE1')
dbms_aqadm.create_queue ('Q1','QT1');
dbms_aqadm.start_queue ('Q1');
END;
Note that all queue tables support buffered messages
No additional attributes are specified for the queue or the queue table


The following definitions are used with the examples in this section

2008 Julian Dyke
juliandyke.com 51
Buffered Messages
Enqueue
The following code enqueues a buffered message
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_enqopt dbms_aq.enqueue_options_t;
l_enq_msgid RAW(16);
BEGIN
l_payload := new TYPE1 (10001,20001,30001);
l_msgprop.expiration := DBMS_AQ.NEVER;
l_enqopt.visibility := DBMS_AQ.IMMEDIATE;
l_enqopt.delivery_mode := DBMS_AQ.BUFFERED;
dbms_aq.enqueue
(
queue_name => 'Q1',
enqueue_options => l_enqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_enq_msgid
);
END;
2008 Julian Dyke
juliandyke.com 52
Buffered Messages
Dequeue
The following code dequeues a buffered message:
SET SERVEROUTPUT ON
DECLARE
l_payload TYPE1;
l_msgprop dbms_aq.message_properties_t;
l_deqopt dbms_aq.dequeue_options_t;
l_deq_msgid RAW(16);
BEGIN
l_msgprop.expiration := DBMS_AQ.NEVER;
l_deqopt.visibility := DBMS_AQ.IMMEDIATE;
l_deqopt.delivery_mode := DBMS_AQ.BUFFERED;
dbms_aq.dequeue
(
queue_name => 'Q1',
dequeue_options => l_deqopt,
message_properties => l_msgprop,
payload => l_payload,
msgid => l_deq_msgid
);
DBMS_OUTPUT.PUT_LINE ('C1 = '||TO_CHAR (l_payload.c1));
DBMS_OUTPUT.PUT_LINE ('C2 = '||TO_CHAR (l_payload.c2));
DBMS_OUTPUT.PUT_LINE ('C3 = '||TO_CHAR (l_payload.c3));
END;
2008 Julian Dyke
juliandyke.com 53
Buffered Messages
Memory Usage
Memory is allocated from the Streams Pool
The following table shows the amount of streams pool memory required to
enqueue 5101 messages with the TYPE1 payload:
Before After
kodpaih3 0 10,324,448
kwqbsinfy:mpr 480 2,448,480
image handles 84 428,512
kwqbsinfy:bms 72 387,692
kggmem_fl_1 44 224,444
kggbt_alloc_block 2,072 88,060
Sender info 14,140 19,796
recov_kgqbtctx 12,288 16,384
kwqbcqini:spilledovermsgs 2,952 3,936
kwqbsinfy:bqg 1,236 1,648
recov_kggmctx 924 1,232
Before After
recov_kgqmsub 336 504
kwqbsinfy:cco 332 332
kwqbsinfy:sta 208 312
spilled:kwqbl 216 288
fixed allocation callback 256 256
kgqmsub 144 216
deqtree_kgqmctx 136 192
substree_kgqmctx 120 160
time manager index 120 160
msgtree_kgqmctx 120 160
name_kgqmsub 32 48
2008 Julian Dyke
juliandyke.com 54
Buffered Messages
Database Objects
Additional database objects are created the first time a buffered message is
enqueued on a queue table
This will cause elapsed time of first enqueue operation to be high

For example the following objects might be created
Object ID Object Name Object Type
72638 AQ$_QT3_P TABLE
72639 SYS_LOB0000072638C00032$$ LOB
72640 SYS_IL0000072638C00032$$ LOB INDEX
72641 SYS_C0010003 INDEX
The enqueuing session also creates a service for the queue
For example SYS$US01.Q3.TEST where
US01 is the queue owner
Q3 is the queue name
TEST is the database name
2008 Julian Dyke
juliandyke.com 55
Buffered Messages
Database Objects
AQ$_<table_queue_name>_P contains the following columns
Column Name Data Type
Q_NAME VARCHAR2(30)
MSGID RAW(16)
CORRID VARCHAR2(128)
PRIORITY NUMBER
STATE VARCHAR2(16)
DELAY DATE
EXPIRATION NUMBER
TIME_MANAGER_INFO TIMESTAMP(6)
LOCAL_ORDER_NO NUMBER
CHAIN_NO NUMBER
CSCN NUMBER
DSCN NUMBER
ENQ_TIME DATE
ENQ_UID VARCHAR2(30)
ENQ_TID VARCHAR2(30)
Column Name Data Type
DEQ_TIME DATE
DEQ_UID VARCHAR2(30)
DEQ_TID VARCHAR2(30)
RETRY_COUNT NUMBER
EXCEPTION_QSCHEMA VARCHAR2(30)
EXCEPTION_QUEUE VARCHAR2(30)
STEP_NO NUMBER
RECIPIENT_KEY NUMBER
DEQUEUE_MSGID RAW(16)
SENDER_NAME VARCHAR2(30)
SENDER_ADDRESS VARCHAR2(1024)
SENDER_PROTOCOL NUMBER
USER_DATA TYPE1
USER_PROP SYS.ANYDATA
2008 Julian Dyke
juliandyke.com 56
Buffered Messages
Database Objects
The AQ$_<queue_table_name>_P table has one primary key index on
Q_NAME
MSGID

Two view definitions are also updated when the first buffered message is
enqueued:
AQ$<queue_table_name>
e.g. AQ$QT3
reports all messages in persistent and buffered queues
AQ$_<queue_table_name>_F
e.g. AQ$_QT3_F
reports all messages that have not yet been dequeued in both
persistent and buffered queues

2008 Julian Dyke
juliandyke.com 57
Buffered Messages
Database Objects
The queue monitor slaves write spilled messages to
AQ$_<queue_table_name>_P
Rows are inserted individually; no array operation is used
For example
INSERT INTO "us01"."aq$_qt3_p"
(
q_name, msgid, corrid, priority,state, delay, expiration, time_manager_info,
local_order_no, chain_no, enq_time, step_no, enq_uid, enq_tid, retry_count,
exception_qschema, exception_queue, recipient_key, dequeue_msgid,
user_data, sender_name, sender_address, sender_protocol, dscn, cscn
)
VALUES
(:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,0,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24)
Messages are asynchronously deleted from AQ$_<queue_table_name>_P by
queue monitor slaves
Messages are deleted using an array size of 32
For example
DELETE FROM us01.aq$_qt24_p WHERE q_name = :1 AND msgid = :2
2008 Julian Dyke
juliandyke.com 58
Buffered Messages
Database Objects
AQ$_<table_queue_name>_D contains the following columns
Column Name Data Type
OID NUMBER
MSGNUM NUMBER
MSGID RAW(16)
SUB NUMBER
SEQNUM NUMBER
RSUBS SYS.AQ$_RECIPIENTS
The RSUBS column is stored as a LOB

2008 Julian Dyke
juliandyke.com 59
Buffered Messages
Spillage
If flow control is enabled then number of buffered messages that can be
enqueued on any queue is limited
Subsequent attempts to enqueue messages will be rejected

Set _BUFQ_STOP_FLOW_CONTROL parameter to TRUE to disable flow
control completely

Limited to
5000 buffered messages
15000 captured messages

Can be overridden in 10.2.0.3 by applying Patch 5093060 and setting
Event 10867 for buffered messages (level is # messages)
Event 10868 for captured messages (level is # messages)

Can be fixed in 10.2.0.4 onwards by setting:
_BUFFERED_PUBLISHER_FLOW_CONTROL_THRESHOLD
_CAPTURED_PUBLISHER_FLOW_CONTROL_THRESHOLD

2008 Julian Dyke
juliandyke.com 60
Buffered Messages
Database Objects
For a multiple consumer queue the following objects will be created when the
first buffered message is enqueued:

Object ID Object Name Object Type
72638 AQ$_QT3_P TABLE
72639 SYS_LOB0000072638C00032$$ LOB
72640 SYS_IL0000072638C00032$$ LOB INDEX
72641 SYS_C0010003 INDEX
72642 AQ$_QT3_D TABLE
72643 SYS_IOT_OVER_72642 TABLE
72644 SYS_LOB0000072642C00006$$ LOB
72645 SYS_IL0000072642C00006$$ INDEX
72646 SYS_IOT_TOP_72642 INDEX
2008 Julian Dyke
juliandyke.com 61
Performance
Elapsed Times
Enqueue Dequeue
No Commit Commit No Commit Commit
Single Consumer
PERSISTENT
ON COMMIT
4.77 10.99 5.75 9.62
Single Consumer
PERSISTENT
IMMEDIATE
10.80 11.41 8.77 9.78
Single Consumer
BUFFERED
IMMEDIATE
2.32 2.60 1.53 2.13
Single Consumer
PERSISTENT
ON COMMIT
VARRAY(10) OF TYPE1
0.66 1.38 1.00 1.35
Multi Consumer
PERSISTENT
ON COMMIT
2 recipients
6.40 14.45 6.36 11.20
Multi Consumer
PERSISTENT
ON COMMIT
2 subscribers
6.02 14.59 6.54 11.40
10000 TYPE1 messages enqueued then 10000 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5
x86
2008 Julian Dyke
juliandyke.com 62
Performance
Redo Generation
Enqueue Dequeue
No Commit Commit No Commit Commit
Single Consumer
PERSISTENT
ON COMMIT
9223 15491 10806 15521
Single Consumer
PERSISTENT
IMMEDIATE
15165 15485 14936 14904
Single Consumer
BUFFERED
IMMEDIATE
0 0 0 0
Single Consumer
PERSISTENT
ON COMMIT
VARRAY(10) OF TYPE1
1211 1831 1381 1832
Multi Consumer
PERSISTENT
ON COMMIT
2 recipients
16459 23324 8102 12521
Multi Consumer
PERSISTENT
ON COMMIT
2 subscribers
15832 23404 7934 12953
10 TYPE1 messages enqueued then 10 messages dequeued. Average of 5 runs. Oracle 10.2 on RHEL4.5 x86
2008 Julian Dyke
juliandyke.com 63
Conclusion
Several single queues may be more efficient than
Multiple recipients
Multiple subscribers

Use ON_COMMIT visibility where possible
No transaction overhead for queuing operations
Reduces undo / redo generation
IMMEDIATE is much more expensive

Buffered messages give best performance
Provided they do not spill regularly

Array payloads are very efficient
Message overhead is reduced


2008 Julian Dyke
juliandyke.com 64
Thank you for your interest
info@juliandyke.com