Presented By

Quontra Solutions
Online Training & Placement Support
Call Us:+ 404-900-9988
Email:info@quontrasolutions.com
Web:www.quontrasolutions.com
INCREMENTAL LOAD
Is sometimes called
Incremental Load
Differential Load
Delta Load
INCREMENTAL LOAD
Goal:
Load only the new or the changed records from the
database. The rest should already be available, one
way or another.
COMMENTS ON BUFFER LOAD
Buffer (Incremental) Load
is a solution only for Log files (text files), but not for
DBs.
Buffer (Stale after 7 days) Select
is not a good solution. It makes a full Load after 7
days. And nothing in between
INCREMENTAL LOAD
Load new data from
Database table (slow,
but few records)
Load old data from
QVD file (many
records, but fast)
Create new QVD file
Procedure must be
repeated for each
table
DIFFERENT DB-CHANGES
If source allows
1) Append only. (Logfiles)
2) Insert only. (No Update or Delete)
3) Insert and Update. (No Delete)
4) Insert, Update and Delete.
1) APPEND ONLY
Must be Log file
Loads records
added in the end of
the file
1) APPEND ONLY


Buffer (Incremental)
Load * From LogFile.txt
(ansi, txt, delimiter is '\t', embedded labels);

Done!
But it should be renamed to
Buffer (Append) Load
2) INSERT ONLY
Can be any DB
Loads INSERTed
records
Needs the field
ModificationDate
2) INSERT ONLY


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

2) INSERT ONLY


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;

2) INSERT ONLY


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;

STORE QV_Table INTO File.QVD;


Almost done
But there is a small chance
that a record gets loaded twice
2) INSERT ONLY


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#
AND ModificationTime < #$(BeginningThisExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD;

STORE QV_Table INTO File.QVD;


3) INSERT AND UPDATE
Can be any DB
Loads INSERTed and
UPDATEd records
Needs the fields
ModificationDate and
PrimaryKey

3) INSERT AND UPDATE


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT Exists(PrimaryKey);

STORE QV_Table INTO File.QVD;


4) INSERT, UPDATE AND DELETE
Can be any DB
Loads INSERTed and
UPDATEd records
Removes DELETEd
records
Needs the fields
ModificationDate and
PrimaryKey
Tricky to implement
4) INSERT, UPDATE AND DELETE


QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT EXISTS(PrimaryKey);

Inner Join
SQL SELECT PrimaryKey FROM DB_TABLE;

STORE QV_Table INTO File.QVD;


4) INSERT, UPDATE AND DELETE
ListOfDeletedEntries:
SQL SELECT PrimaryKey AS Deleted FROM DB_TABLE
WHERE DeletionFlag = 1
and ModificationTime >= #$(LastExecTime)#;

QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate
LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT Exists(PrimaryKey)
AND NOT Exists(Deleted,PrimaryKey);

Drop Table ListOfDeletedEntries;

STORE QV_Table INTO File.QVD;


OK, but needs
a DeletionFlag
LASTEXECUTIONTIME & ERROR HANDLING
Let ThisExecTime = Now();

{ Load sequence }

If ScriptErrorCount = 0 then
Let LastExecTime = ThisExecTime;
End If
FINAL SCRIPT
Let ThisExecTime = Now();

QV_Table:
SQL SELECT PrimaryKey, X, Y FROM DB_TABLE
WHERE ModificationTime >= #$(LastExecTime)#
AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD
WHERE NOT EXISTS(PrimaryKey);

Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;

If ScriptErrorCount = 0 then
STORE QV_Table INTO File.QVD;
Let LastExecTime = ThisExecTime;
End If
SUMMARY 1
Incremental Load possible for
Append only. (Logfiles) Yes!
Insert only. (No Update or Delete) Yes!
Insert and Update. (No Delete) Yes!
Insert, Update and Delete. Slow,
or demands
DeletionFlag
SUMMARY 2
Incremental Load normally not equivalent to
Buffer (Incremental) Load