MORE THAN JUST SOME BITS

BASEL BERN LAUSANNE ZRICH DSSELDORF FRANKFURT A.M.

Data Warehousing and Bitmap Indexes

Dani Schnider, Trivadis AG

Oracle Open World 2011, San Francisco

FREIBURG I.BR.

HAMBURG

MNCHEN

STUTTGART

WIEN

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Dani Schnider Principal Consultant and DWH/BI Lead Architect at Trivadis, Switzerland Teacher for courses about Data Warehousing, SQL Optimization and Oracle Warehouse Builder Co-author of book Data Warehousing mit Oracle

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

About Trivadis
Swiss IT Consulting Company Main focus on databases

Vienna Freiburg Basel Bern Lausanne Munich

Hamburg

Dusseldorf

~180 employees

Frankfurt

Infrastructure Engineering Application Development Business Intelligence Managed Services Business Integration Services Training

Stuttgart

Zurich

~20 employees

More than 600 employees in 11 locations in Switzerland, Germany and Austria http://www.trivadis.com

~350 employees

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

 SOME BITS ABOUT BITMAP INDEXES AND THEIR USAGE IN DATA WAREHOUSES

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes Concept

Oracle Database Concepts 11g Release 2 (11.2)

Bitmap indexes are primarily designed for data warehousing or environments in which queries reference many columns in an ad hoc fashion. Situations that may call for a bitmap index include: > The indexed columns have low cardinality, that is, the number of distinct values is small compared to the number of table rows. > The indexed table is either read-only or not subject to significant modification by DML statements.

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes Concept

FIRSTNAME GENDER MARITAL STATUS --------- ------- ------John M M David M S Mary F S Harry M D Jane F M ...

Bitmap GENDER:
ROWID F M A 0 1 B 0 1 C 1 0 D 0 1 E 1 0 ...

Bitmap MARITALSTATUS:
ROWID D M S A 0 1 0 A 0 0 0 B 0 0 1 B 1 0 0 C 0 0 1 C 1 1 1 D 1 0 0 D 0 0 0 E 0 1 0 E 0 1 0 ... ...

WHERE gender = 'F' AND maritalstatus = 'S'

ROWID S AND F Result

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes Example

CREATE BITMAP INDEX cust_gender_bix ON customers (gender) Demo

CREATE BITMAP INDEX cust_marital_bix ON customers (maritalstatus) SELECT FROM WHERE AND firstname, lastname, gender, maritalstatus customers gender = 'F' maritalstatus = 'S'

-------------------------------------------------------------| Id | Operation | Name | -------------------------------------------------------------| 0 | SELECT STATEMENT | | | 1 | TABLE ACCESS BY INDEX ROWID | CUSTOMERS | | 2 | BITMAP CONVERSION TO ROWIDS| | | 3 | BITMAP AND | | |* 4 | BITMAP INDEX SINGLE VALUE| CUST_MARITAL_BIX | |* 5 | BITMAP INDEX SINGLE VALUE| CUST_GENDER_BIX | -------------------------------------------------------------7
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Comparison B-tree Index / Bitmap Index

WHERE Condition = <, <=, >, >= BETWEEN LIKE IN NOT IN !=, <> IS NULL IS NOT NULL
1 2

B-tree Index

Bitmap Index

()1

()2 ()2 ()2

Only in concatenated b-tree indexes Only for combinations of bitmap indexes

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Comparison B-tree Index / Bitmap Index

Usage / Feature Primary/Unique Key Concatenated Index Combine in one query Star Transformation Row Locking Oracle Standard Edition Oracle Enterprise Edition B-tree Index Bitmap Index

()1

Possible, but not recommended

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Concatenated Bitmap Indexes

Bitmap Indexes on two or more columns are possible, but not useful

CREATE BITMAP INDEX cust_gender_marital_bix ON customers (gender, maritalstatus)

Why? Concatenated bitmap index can be used only when all columns are filtered Two or more separate bitmap indexes are more flexible and can be used for all combinations

10

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Index Strategy on Star Schema

Dimension Tables:
Unique b-tree index on primary key column Bitmap indexes on additional columns (optional)

Fact Tables:
Bitmap indexes on all dimension key (foreign key) columns Bitmap join indexes on often used filter columns (optional) Usually no primary key on fact tables

11

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Index Strategy on Star Schema

Dimension 1
Bitmap Index DIM_1_ID Bitmap Index DIM_3_ID

Dimension 3

Fact Table

Bitmap Index DIM_2_ID

Bitmap Index DIM_4_ID

Dimension 2
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Dimension 4

12

Query Optimization on Star Schemas

Typical Queries:
Filter criteria on (multiple) dimension tables Facts are selected by join with dimensions 1

Problem:
Tables with restrictions should be read first Only two tables can be joined at a time No relationships between dimension tables

3 3

13

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Star Transformation Concept

Bitmap Index SALES_PROD_BIX

Products

AND

Sales (Fact Table)

Customers
Bitmap Index SALES_CUST_BIX
14
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Star Transformation Example

What is the total revenue for product Shorts for all customers in Germany, grouped by male and female customers?
SELECT , , FROM JOIN JOIN WHERE AND GROUP p.prod_desc c.cust_gender sum(s.amount_sold) sales s products p ON (s.prod_id = p.prod_id) customers c ON (s.cust_id = c.cust_id) p.prod_name = 'Shorts' c.country_id = 'DE' BY p.prod_desc, c.cust_gender

15

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Star Transformation Example

---------------------------------------------------------------| Id | Operation | Name Demo | ---------------------------------------------------------------| 0 | SELECT STATEMENT | | | 1 | HASH GROUP BY | | |* 2 | HASH JOIN | | |* 3 | HASH JOIN | | |* 4 | TABLE ACCESS FULL | PRODUCTS | | 5 | PARTITION RANGE ALL | | | 6 | TABLE ACCESS BY LOCAL INDEX ROWID| SALES | | 7 | 3 BITMAP CONVERSION TO ROWIDS | | | 8 | BITMAP AND | | | 9 | BITMAP MERGE | | | 10 | BITMAP KEY ITERATION | | | 11 | BUFFER SORT | | 1 |* 12 | TABLE ACCESS FULL | PRODUCTS | |* 13 | BITMAP INDEX RANGE SCAN | SALES_PROD_BIX | | 14 | BITMAP MERGE | | | 15 | BITMAP KEY ITERATION | | | 16 | BUFFER SORT | | 2 |* 17 | TABLE ACCESS FULL | CUSTOMERS | |* 18 | BITMAP INDEX RANGE SCAN | SALES_CUST_BIX | |* 19 | TABLE ACCESS FULL | CUSTOMERS | ---------------------------------------------------------------16
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Join Index Concept

Bitmap index is defined on fact table Indexed column is part of dimension table Possible for star and snowflake schemas

17

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Join Index Example Demo

Same example with Bitmap Join Indexes
-----------------------------------------------------------------------| Id | Operation | Name | -----------------------------------------------------------------------| 0 | SELECT STATEMENT | | | 1 | HASH GROUP BY | | | 2 | NESTED LOOPS | | | 3 | NESTED LOOPS | | |* 4 | HASH JOIN | | | 5 | MERGE JOIN CARTESIAN | | |* 6 | TABLE ACCESS FULL | COUNTRIES | | 7 | BUFFER SORT | | |* 8 | TABLE ACCESS FULL | PRODUCTS | | 9 | PARTITION RANGE ALL | | | 10 | TABLE ACCESS BY LOCAL INDEX ROWID| SALES | | 11 | BITMAP CONVERSION TO ROWIDS | | 3 | 12 | BITMAP AND | | |* 13 | BITMAP INDEX SINGLE VALUE | SALES_PRODNAME_BJI | 1 |* 14 | BITMAP INDEX SINGLE VALUE | SALES_COUNTRYNAME_BJI | |* 15 | INDEX | CUSTOMERS_PK | 2 UNIQUE SCAN |* 16 | TABLE ACCESS BY INDEX ROWID | CUSTOMERS | -----------------------------------------------------------------------18
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes and ETL

Bitmap indexes are efficient for set-based ETL
Index maintenance is deferred until end of each DML operation

Efficient index maintenance for Parallel DML

ALTER SESSION ENABLE PARALLEL DML; INSERT /*+ parallel(f, 4) */ INTO fct_sales f SELECT /*+ parallel(s, 4) */ * FROM stg_sales s;

Avoid hand-made parallel ETL with multiple sessions

Locking behavior of bitmap indexes

For large data loads: Rebuild indexes after ETL operation

Set all indexes to UNUSABLE Load data into table Rebuild all indexes
19
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes and Partitioning

Bitmap Indexes must always be LOCAL
CREATE BITMAP INDEX sales_prod_bix ON fct_sales (prod_id); ORA-25122: Only LOCAL bitmap indexes are permitted on partitioned tables

CREATE BITMAP INDEX sales_prod_bix ON fct_sales (prod_id) LOCAL; Index created.

Local indexes are recommended in Data Warehouse

Only purpose of global indexes in DWH: unique indexes without partition key Restriction of local bitmap indexes is not a problem
20
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes and Compression

Bitmap Indexes are always compressed
Zeros are compressed, Ones are not compressed Bitmap contains number of zeros to next 1 bit
00000000 00000000 00000000 00000000 00000000 00000010 00000000 00000000 00000000 00000000 01000000 00000000

46 zeros x2E 33 zeros x21

Bitmap Indexes need less disk space than B-tree Indexes

Even for columns with high number of distinct keys

Detail description see Bitmap Index Internals, Julian Dyke

http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt

21

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes and Compression Demo

Number of distinct keys has impact on size of index
Bitmap index is usually smaller than b-tree index For high number of keys, size of bitmap index increases Size of b-tree index is more or less constant Sort order affects size of bitmap index, but not of b-tree index
350 300 250 200 150 100 50 0 10 100 Bitmap (scattered)
2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

1000 Bitmap (sorted)

10000 B-tree index

100000

22

Bitmap Indexes and Compression Demo

Enabling table compression
When a table is enabled for compression, all bitmap indexes must be disabled
CREATE TABLE customers () NOCOMPRESS; CREATE BITMAP INDEX cust_gender_bix ON customer (gender); ALTER TABLE customers COMPRESS; ORA-14646: Specified alter table operation involving compression cannot be performed in the presence of usable bitmap indexes

Reason: Hakan factor of table

Maximum number of rows that can be stored in a table block Hakan factor changes when table is enabled for compression (only first time)

23

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

Bitmap Indexes Conclusion

Bitmap Indexes are a powerful Data Warehouse feature
Can be combined in ad hoc queries Star Transformation Useful for selective and non-selective columns

Bitmap Indexes are useful for large databases

Efficient for set-based ETL and Parallel DML Local bitmap indexes on partitioned tables Compression of bitmap indexes

There are almost no reasons to use b-tree indexes in a Data Warehouse

B-tree indexes are mainly used for Primary Key or Unique Key constraints

24

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

References
Oracle Database Data Warehousing Guide 11g Release 2 (11.2)
Oracle Corporation, http://www.oracle.com/pls/db112/

Oracle Database Concepts 11g Release 2 (11.2)

Oracle Corporation, http://www.oracle.com/pls/db112/

Christian Antognini: Troubleshooting Oracle Performance

Apress, http://www.apress.com/9781590599174

Jonathan Lewis: Cost-Based Oracle Fundamentals

Apress, http://www.apress.com/9781590596364

Julian Dyke: Bitmap Index Internals

http://www.juliandyke.com/Presentations/BitmapIndexInternals.ppt

25

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011

THANK YOU.

Trivadis AG Dani Schnider Europa-Strasse 5 CH-8152 Glattbrugg/Zrich Switzerland Tel. + 41 44 808 70 20 Fax + 41 44 808 70 21 info@trivadis.com www.trivadis.com

BASEL

BERN

LAUSANNE

ZRICH

DSSELDORF

FRANKFURT A.M.

FREIBURG I.BR.

HAMBURG

MNCHEN

STUTTGART

WIEN

26

2011 Trivadis Data Warehousing & Bitmap Indexes 02.10.2011