Escolar Documentos
Profissional Documentos
Cultura Documentos
White Paper
2009
Abstract: An overview is given about Lilith 6.x technology. The HCR database is described and
the main competitive advantages are highlighted. The main database objects and their theoretical
foundations are explained. Some important features like virtual cubes are described in deep. The
typical architectural implementation on different environments is shown. The main front-end
techniques are exposed, including block-operations and apportionments. The charting techniques
and the profiling methods are reviewed.
0. Introduction.......................................................................................................2
1. Back end............................................................................................................2
1.1 Data Structure........................................................................................2
1.2 Data Objects...........................................................................................3
1.3 Virtual Cubes.........................................................................................7
1.4 Typical Architectures.............................................................................9
1.5 The Licensing System..........................................................................12
2. Front end.........................................................................................................13
2.1 Architecture..........................................................................................13
2.1 Multidimensional Operations..............................................................14
2.2 Semantics.............................................................................................16
2.3 Charts...................................................................................................19
WP_Lilith6 B4
In order to allow the multidimensional analysis with virtually no waiting time (in the
range of milliseconds, independent of the size of the data repository), an own internal
engine was created, totally integrated in the software. All data are therefore saved in
an optimized way, in order to be ready for exploration. A brand new set of data may be
derived easily at any time, based on actual data. Any further hierarchical level may be
added at any time. This is very useful in comparisons among actual, budged, revised
and what-if simulation data.
All analysis may be performed on Lilith Client or on Lilith Webserver, in both cases
with write-back capabilities.
This White Paper examines Lilith Back-end and Front-end, and gives information on
how data are extracted, transferred and loaded, how whole sets of data are stored and
rearranged according to Block Operations, Normalization, Apportionment, Formula
applications, and how automatic script are created and launched.
The White Paper examines also the different application modules, features and
privileges, and the different customization in the company IT structure.
In a single suite, sound and independent of external libraries, created in C++ language
and therefore multi-platform, Lilith 6 can simplify the use, interpretation and share of
data throughout a company or organization. This makes the product suitable for
business control and for for marketing simulation over future activities and top-down
or bottom-up budget negotiations. It is suitable besides for bank and insurance
performance and trend analysis, and for the storage and analysis of multimedia and
physics data.
1. Back end
Lilith software version 6.x implements a unique and native database layer, which is Independent
Database
directly based on its own file system and independent on external third-party database
engines.
This database layer is named 'HCR' with reference to the capabilities of the database
engine (Hierarchical, Cartesian and Relational).
Data are stored in the HCR database in the most natural and technologically efficient
way, allowing for comparably better performance and improved ease of use and
navigation.
The HCR database layer is also scalable from individual desktop to enterprise usage
(Users can store repositories on hard disks, Usb disks or even Usb pens!), thereby Scalable
storing teradata and serving hundreds of users. Multiplatform
Legacy
RDBMS
.csv
Clie nt,
File s
Web Flat
ODBC File s
ETL
HCR Repository
File system
The objects implemented by the database layer are: Tables, Cubes, Lists, Views,
Charts and Macros.
The first two objects (Tables and Cubes) are used to store data into the HCR
repository. The other ones are used by the front-end to manage and report data.
Tables are used to store metadata. Thanks to the unique indexing feature called b- Metadata
tree, tables can store many millions of elements for each dimension of the cube B-Tree
(allowing billions of storage cells).
• The relationships are stored by relating direct binary pointers to corresponding Pointers,
table rows, instead of using the typical methods of relating identical keys on Integrity,
two columns to be matched during the query. This allows a very high Semantic
performance while resolving relationships, with no need to define explicit
indexes (since the pointer itself is an index). Furthermore, the referential
integrity is always intact, allowing the users to change any key without
breaking the existing relationships.
• The database stores relationships in a semantic manner. The database knows all
the links among the tables and facilitates the users during the queries. So SQL
is abandoned and the query techniques don't need to specify any 'join' or 'group
by' clause, because they are implicit in the data structures.
Many levels
Hierarchies
Formulas
Columns
y
an
Parallel hierarchies
-m
to
1-
Metadata
Rows
Formulas
Totals
The Lilith 6 tables utilize both Relational and Hierarchical capabilities of the HCR
repository. Hierarchies can be defined on many levels, and many alternate views are
allowed (parallel hierarchies).
Tables also store formulas and easily allow users to calculate sub-totals and roll-ups.
Multi-cube
Cubes, instead, store bulk data. By crossing together many tables, users can easily Model,
build a cube (Cartesian capability in the HCR repository). No redundancy
Data redundancy is annulled; metadata, formulas, keys are defined or imported once
and then used by the whole system.
Shared table
Metadata subset used
Cube 2
Cube 1
The internal cube architecture is quite complex, even if the performance for the final
Many Gigas,
user is so efficient. A large cube can store many gigabytes of data. The way the data Direct Access
are stored on disk is optimized for multidimensional data. The direct access via
dimensional decoding makes the fetch time independent of the cube size (for a fluent
navigation of charts and grids).
Sparse data sets are stored in special cubes allowing to leverage the Relational Sparse
capabilities of the repository (many-to-may relationships) without losing the Cubes
advantages the multidimensional model has with dense dimensions. The system may
have an automatic shift towards a more relational behavior, in extremely sparse DB
conditions (cells density <= 5%).
Multi-type cubes are available, allowing the physical storage into the same cube of
Multi Type
any kind of data: numbers, strings, dates, flags, links to tables, links to files (for
multimedia and documents indexing).
Cubes can be obviously fed with incremental data, but one of the unique features No Rebuild
Lilith 6 implements is the incredibly efficient behavior when metadata change.
The cube can be reduced or expanded without rebuilding any of the existing data,
allowing high performances and easy maintenance.
Pilot table
Float
String
Date
Flag
Totals can be nested on many levels and for any dimension. The totals definition
easily comes from the links among tables. It allows users to automatically create totals
rows in the right positions.
Totals are precalculated for efficiency and allow 'drill-down' in any situation. Nested Totals,
Roll-up,
On the fly roll-ups are allowed for not predefined totals. Semi-additive
The aggregation methods are: Sum, Product, Count, Min, Max, Average, First, Last.
Semi-additive aggregations are allowed (i.e. use 'Sum' along one dimension, 'Last' on
another dimension, etc.).
Metadata hierarchies
Leaves
Sub-total
Total
Semi-additivity
Las t
s
on
vs . Las t ye ar
si
en
Margin
im
rd
Alert
e
Conditional formula
th
O
Switch(Margin<0, Warning, Ok)
Year s
1.3 Virtual Cubes
Virtual cubes are a unique feature in the Lilith 6 back-end architecture, being a sort of
“Adapters” in calculations among other cubes having different rank.
Logical Views
To fully describe them many pages would be needed, here we will explain the
fundamentals about this peculiar way to organize data.
Virtual cubes are the solution for a series of typical modeling problems. Generalization
Each cube type we created after the genesis of the HCR engine comes from a peculiar and
Integration
need: its generalization and integration with other virtual cube types and other
methods (like Block Operations, Normalizations, Apportionments etc.) allow to
pursue any kind of data modeling.
- Type I simply allows to reduce the elements of a cube dimension (the virtual cube is
a subset of the physical cube).
- Type II increases the rank of the cube by adding virtual dimensions. The data are
(virtually) repeated along these new dimensions.
- Type III increases the cardinality of one or more dimensions. This operation is
performed using the hierarchical relationships; for example the Areas dimension can
be transformed into the Plants dimension (being Plants a lower hierarchy of Areas).
The data are repeated by reading the figures belonging to the parent element (i.e.
Plants will receive the values from the corresponding Area).
- Type IV reduces the rank of the physical cube by putting together two or more The Four Types
of Virtual
Cubes
All right reserved © 2009 Hicare Research Page 7 of 20 www.hicare.com
dimensions. The new cube will have one new dimension where the 'cartesian product'
(all the possible combinations) of the merged dimensions is exposed. The data will
flow from the physical cube to the virtual one according to the metadata semantics.
→
d
an
p
Ex
Type I Type II
C
1-to-many
BxC
B
A
Type III Type IV
The use of the four types can be combined to elegantly solve complex problems.
For example the Third type is often used to reshape the currency rates while Combination of
converting into a single currency the balance sheets from many plants working in Types
different currencies (the proper rate is derived from the hierarchy Plants→
Countries→ Currencies).
The Fourth type is useful if a multi-year time series is needed (for example in charts)
instead of viewing data in the classical two-entry Months/Years table.
If the model involves a currency convention over a multi-year time span, the resulting
virtual cube will be a mix between Third and Fourth type.
Finally, virtual cubes are, under any point of view, isomorphic to physical cube. This
means that you can perform any activity allowed on physical cubes: calculations,
charts, block operations etc.
Furthermore virtual cubes can even be used in the write mode. Therefore you can Read & Write
write values into the physical cube using the virtual cube as a filter (i.e. write a figure
in the Plant to set the Area value).
This side-effects, i.e. the usage of generalist features beyond the original scope,
revealed to be one of the main points of strength of the self-consistent and coherent
Lilith 6 architecture.
Lilith 6 is a complete and flexible platform, powerful and easy in the deployment in a
large range of application fields.
Its flexibility allows a wide range of architectural setups. The following paragraphs
have the goal to describe the most common scenarios.
This entry level environment allows a lean hardware setup and a quick and simple Minimal
architecture. No server is needed.
The super user uses the Lilith Designer to create the database on its local disk, import
data, draw charts and reports, and save them in a shared folder. The final users simply
The super user can also produce static outputs using PDF's or PPT's.
Super user
Lilith
Lilith Designer Application M ngr
Final users
Lilith
LAN Application M ngr
Local Lilith
Disk Application M ngr
HCR DB
Data sources
b) Middle environment: development team (two/three modelers), some local users, Local
some intranet Web users. &
Web
Here we have a small development team; we need a Master, managing metadata and
permissions via Lilith Designer, and a couple of modelers allowed to create cubes,
charts and macros using the Model Manager.
The local users connect to the LAN as in a) scenario. A basic Web Server is needed to
serve the intranet Web users.
Master
LAN users
Lilith
Lilith Designer LAN
Application M ngr
Web users
Server Lilith
disk Web server Web browser
Data sources
Lilith
M ode l M anager
Modelers
In the Enterprise scenario we usually have one (or more) development environment
(where the model is updated, when needed), and one (or more) production
environment. When updated, the development environment is 'released' by just
copying the database on the production server.
The development team receives different access levels on the different models
according the access list defined by the administrator.
Since we are in a protected network, we can use the two-tiers architecture where the
Lilith Web Server is acting, at the same time, like an Application Server and a Web
server.
DB Admin
Lilith Lilith
Designers Web server Web browser
&
Model Mngrs
Access
Development Production
List
Environment Release Environment Web users
Developers
d) Enterprise environment: development team, many public Web users (three-tier Secure
architecture).
This environment is very similar to point c), but due to the public access required, a
three-tiers architecture is recommended.
In this scenario Lilith Web Server acts as a pure Application Server and
communicates, vie Servlet, with an Apache HTTP Server for maximum security.
Web users
Developers
Lilith has a unique floating licensing system, with all modules accounting for a
specific resource usage, in terms of “Lilith Tokens” (LTK). Upon the reach of the
number of Tokens the company has purchased, the system inhibits the access of
further users. If a user exits the software, or after a timeout period, the Tokens are
released, and the system allows the access of a new user.
Only the user profiling policies determine the access rights, without limitations
regarding to where the software is installed. A central server administers the use of
resources.
This licensing scheme gives some advantages: licenses are not bound to a specific
computer, and there is a great flexibility in assigning them to a specific user (the
number of licenses sold to the company is therefore reduced); upon Token release, the
license is immediately available to a new user.
2.1 Architecture
The Lilith 6 front-end is a very powerful and flexible tool for the complete and visual Binary
interaction with the HCR back-end. Communication
Front-end and back-end are fully integrated. They exchange data as binary values,
assuring a very fast in-memory communication.
The back-end implements a high level of abstraction on the data. So any front-end Abstraction
operation deals with cubes, blocks, cells and never with records, fields, indexes.
It also uses a powerful caching system to store the latest request from the front-end as
well as the results from data manipulations. Caching
This minimizes the disk access and can even, if enough memory is installed, allow the
system to work with the whole cube in memory.
All front-end actions can be performed in a visual way, using the GUI (Guided User
Interface) and without programming. Vice versa any GUI action can be recorded into GUI & Recorder
a script (macro) for later execution or editing.
The macro language is very close to the Basic language. It allows subroutines,
IDE
structured programming (if then, for next, while, etc.), and multi type variables.
Provider
The Provider is the Communication layer between front and back
end
Lilith Back-end The back-end reads bulk data from the cache and implements the
HCR data structures, the load and the calculations.
Lilith Caching System The cache stores in memory the disk data and the results of data
manipulation. It minimizes the disk access.
The system file server is used to read/ write binary data from/ to
File system the disk device.
• The Desktop Installation, where all modules lay on the same computer, in a
single process. This installation is very quick (five-minutes setup) and is Quick Install
suitable for small offices or individual usage. The file system can be hosted on
a server to allow many users to share they data models, procedures and charts
(this option requires a broadband network, for higher efficiency).
• The Terminal Installation allows the users to exploit the full features of the Development
local front-end GUI (including IDE) without renouncing to leverage a Install
powerful server for data load calculations. It requires the installation of a
terminal server system and is suitable for medium/ large development teams.
It can be associated to the Web Installation model (see below).
• The Web Installation is suitable for large number of final, non developing
Web Install
users (since IDE is not available via web). A Lilith Web Server is required (and
Data Entry
optionally a HTTP server). No software installation is required on remote Remote Scripts
locations (nor plug-ins, applets etc.). The front-end acts as a renderer for
reports and charts. Then it supports web data entry and remote procedures.
Pure dynamic html is used to build a nice and interactive interface.
User side
Front-end Front-end
Provide r
Back-end
Front-end
Cache Provide r Provider
Back-end Back-end
Cache Cache
A powerful and efficient data manipulation engine is the main differentiation key of a
Business Intelligence system.
This means that the user will never have to carry out any 'loop' on data (like in No Loops
traditional programming). All the loops are performed in the most efficient way by the
engine. So the user will be brought to think using abstract object as cubes in a very
natural way, without caring about the technicalities of implementation.
All the cube 'methods' in Lilith work according to this spirit: dimensional calculations,
block operations, normalizations, apportionments, etc.
To explain how this works let's concentrate on block operations and apportionments.
Block operations, in the most general acceptation, are data manipulations involving Block
two multidimensional data set (in some case just one set is used, which is both source Operations
and target).
A formula is specified during the operation. The result of the formula is stored in the
target data set (target block), the operands of the formula can be scalar values
(constants), the source block values, or the target block itself.
The blocks can belong to two different cubes or to the same cube:
The two blocks are subsets of the source and target cubes, of course they must be
somehow 'compatible' so that the operation is possible. The compatibility is given by
the rank (the number of dimensions) and by the cardinality (the numbers of elements
on dimensions).
The method allows a good flexibility. About ranks, for instance, if the source block is
Flexibility
All right reserved © 2009 Hicare Research Page 15 of 20 www.hicare.com
'over-ranked' the user will be asked to select just one element among the exceeding
dimensions. If the block is 'under-ranked' identical values will be repeated for the
exceeding dimensions on target cube. About the cardinality, instead, the user is
supported by the database semantics (see next paragraph).
A more complex formula can use a test to copy data from source only where the target
block is empty:
t = Switch( IsNull(t) , s )
The apportionment is an operation involving three cubes. The data stored in the
Apportionment
source cube, usually very synthetic, are shared on the cells of the target cube
according to the driver cube data. Thanks to the information stored in the metadata,
Lilith 6 can automatically normalize the driver cube: the semantics about the one-to-
many relationships perform the apportionment: the user only needs to specify the
names of the three cubes!
Driver cube
2.2 Semantics
The high semantic content of the HCR database allows the front-end to continuously
assist the user in his task to extract information from data.
Here is a list of the main data manipulation features leveraging the semantic
capabilities of the database.
In a standard relational database this is done during the query, specifying the tables
involved (join) and then the operation to be performed; for instance the GROUP BY
clause to perform an aggregation or the WHERE clause to filter data.
In Lilith the information about the linked tables are semantic, i.e immediately
available at any time.
When using the 'selection lists' (a filtering method very similar to the WHERE clause)
the usage of embedded semantics dramatically reduce the complexity of the formula. Filtering
AREA.AREA=”EAST”
Semantics in Block Operations: While performing a block operation you can control
the cardinality (the elements of the dimension) of the blocks involved by simply 'IDEM'
choosing IDEM (a Latin word for 'ditto') for the target block.
If you use IDEM this means that the blocks, even if they belong to different cubes, will
be 'synchronized' according to the metadata.
So any figure from the source block will land in the corresponding cell in the target
block, even if the cubes are sorted in a different way or have different occurrences.
In this way you can perform block operations both in a semantic way (using IDEM) or
in a non-semantic way, specifying in a temporary list the rows you want use (for
example to copy actual data over budget data, or last year data over this year data).
2.3 Charts
Lilith 6 implements more than twenty different chart types. Each chart was designed
to fit to the most varied analytic environments. For example the Radar chart allows
several different scales in the same chart and is useful when we need to show different > 20 Chart Types
different units (money, percentages, indexes, etc.) in the same time lapse. The Delta
Drill and Delta Share charts are specific for studying sensitivities, the Cluster charts
for large populations, the Bubble charts for marketing analysis, and so on...
Choosing the right chart to show all business or physical phenomena is a sort of 'art'
related to the capacity of the modeler to represent the figures in the right way. The
large choice of chart types offered by Lilith 6 is the best way to ease this activity,
eliminating all technology difficulties, and leaving to the users only the model
conceptual activity.
Under the technical point of view, the Lilith 6 charts are not simple 'outputs' but real
charting objects. Not only because they can be saved as database objects in the HCR
repository, but mainly because they are logical objects, living in memory and in one
(or more) cube.
When a cube is opened by the user a logical cube object is created in memory. This
Chart Object
object interacts with the back-end, by means of the cache, to expose to the user the Architecture
whole data set. On top of this cube object the user builds the chart object. The cube
dimensions are assigned to the chart details (legend, axes). The chart object lets the
user navigate and asks the cube object the data.
All this activities are performed in binary format, in memory (using the Lilith 6 cache)
resulting in an incredible browsing speed even on gigabytes cubes.
Device
context Hi Quality, Pixel
Logical Cube Object
Precise
Chart Object
Display, Printer
Disc row data
Vector
graphics
PDF, WMF
Load Browse
Data Data Renderer Raster
graphics
PNG, JPG
The vector graphics allow to produce device independent outputs with 'infinite' Vector Graphics
resolution. The PDF format allows the production or ready-to-use booklets, to be
printed, e-mailed or downloaded. The EWMF (Enhanced Windows Meta File) format
is good for dynamically embedding charts into Office documents.
The raster graphics rendering is used by the Lilith Web Server (JPGs) and for high Raster
definition (typographical printouts) or middle definition output (web embedding) via Graphics
PNG format.
A single Panel (Dashboard, Cockpit) may be created with information from different
cubes. Navigation is possible along all dimensions. If two graphs or tables in the same
panel have one or more dimensions in common, these navigate at the same time. All
dashboards or graphs in a dashboard may be stored and personalized at any time, with
drag and drop capabilities.2.4 Profiling
Lilith 6 profiling module allows the database administrator to give access to users or Report Level and
groups. There are two profiling levels: the report level and the metadata level. Metadata Level
The report profiling applies a filter to the report list. It gives to the users the capability
to access (or not) a specific report (cube or chart). If the user can access the report
then the metadata filter is applied. This means that the administrator can specify for
each user, and for each dimension, which rows will be visible on the reports (this is
very convenient because of the lower time to create a single form for all company,
with a selected view according to the specific privileges)
Metadata filter
Group and users can be defined manually, or imported from the customer LDAP LDAP
environment. The same authorization setup will used by the web and LAN clients.