Escolar Documentos
Profissional Documentos
Cultura Documentos
Target Audience
Database Administrators
All Levels
Query Developers
All Levels
Hardware Configuration Creating the Database IQ Server Settings Connectivity Maintenance Tasks Memory Usage
Volume Manager
Not a necessary or recommended component for IQ Can use MPXIO on Sun Platform to provide controller failover, dynamic multi-pathing, and world wide naming convention Additional overhead and software that has no value add to an IQ installation Acceptable for an IQ single node operation where hardware does not have the ability to apply RAID to the disk devices
RAID Level
Recommend using RAID 5 as a blend between performance, protection, and cost
Recommend using raw devices (a must for multi-node IQ) For details on drive array specifications and configurations see the Sun Reference Architecture Whitepapers and EMC Whitepapers on array configurations with IQ
Disk Controllers
1 fiber controller per 5 write CPUs 1 fiber controller per 10 read CPUs In mixed mode operation use the 5:1 ratio Recommend a minimum of 2 controllers
Creation Options
Block Size and Page Size Minimum should be 128K page size (8K block) Use 256K page for larger databases (>200 GB or so) Larger the page size the more RAM that will be required CASE IGNORE vs. CASE RESPECT Use CASE RESPECT whenever possible Removes case comparison steps and improves performance Java in the Database Install if Java will be needed in the database No impact if installed and not used
10
Use relative links makes it easier to relocate files Use symbolic links for IQ MAIN and IQ TEMP
/IQM_devs/IQ_MAIN_00 /IQM_devs/IQ_TEMP_00
Place the transaction log and transaction log mirror on significantly large file system
Recommend 5 GB of filesystem space per IQ node for database related storage Transaction logs, IQ MSG, scripts, catalog file, etc
11
Filename rules
Catalog: DATABASE_NAME.db Transaction log: DATABASE_NAME.tlog IQ MSG: DATABASE_NAME.iqmsg IQ Main Store: DATABASE_NAME_iqmain_000 000 would be an incrementing number IQ Temp Store: DATABASE_NAME_iqtmp_000 000 would be an incrementing number IQ Configuration File: params.cfg (default)
12
Use for writer and readers (multi-node configuration) to communicate IQ 12.4.3 and earlier
Must be shared across all hosts Generally done via NFS, but some customers use cluster based filesystems (Veritas Cluster Manager)
IQ 12.5
No longer a shared filesystem! Should be local filesystem on each host that is protected accordingly The name and path can be unique for each node No need for NFS or cluster based filesystems
13
FORCE_NO_SCROLL_CURSORS
Should always be set to ON Very few applications require this to be OFF Can improve query performance
14
Query Plan Settings to Provide Optimal Query Information to DBAs and Sybase Engineering
set temporary option query_plan='on'; set temporary option query_plan_after_run='on'; set temporary option query_timing='on'; set temporary option query_detail='on'; set temporary option dml_options10='on'; set temporary option query_plan_as_html='on'; set temporary option Query_Name = Query Name
Should not be set globally as the IQ MSG file will grow rapidly
15
Append_Load
Can be used to improve load performance Will not reuse Row IDs or the space occupied by those Row IDs Great for systems where large, contiguous chunks of data are deleted May not be good if random rows are deleted as it can lead to fragmentation and allocated, but unused space
16
Disk_Striping
If ON, IQ will stripe writes to all available devices If OFF, the first device must be full before the next is used
Disk_Striping_Packed
If ON, it forces better space usage and less fragmentation Fragmentation is indicated when out of space messages are returned but the main dbspace used is less than 100% full Allows Adaptive Server IQ to better utilize small pieces of unused space that remain after compression
17
LOAD_MEMORY_MB
Set to 0 (default) on systems with enough RAM Set to <= 500 on systems where RAM is tight or there are simultaneous loads taking place Rough rule of thumb for load memory requirements Run: select tablewidth( tablename ) Divide that value by 2 New number is rough calculation of MB needed to load that table Remember that each load requires separate memory
18
When configuring a TCP/IP port for IQ to listen on, use something other than the default of 2638 ASA and IQ use port 2638 as a broadcast listener Multiple servers on the same host share the default port for broadcast listening, but not for client network traffic Broadcast listening takes precedence over client network traffic as it is the first port started Easier and safer to assume that 2638 is in use already by ASA/IQ and to use another port for client network traffic Many things must be considered if an IQ server is to use port 2638 for client network traffic on a host with more than one IQ server (server start order, multiplex synchronization, etc) See http://my.sybase.com/detail?id=1012955 for details
19
Different internal environments and settings for ODBC, JDBC, and Open Client connections JDBC and ODBC are recommended
JConnect is Sybases implementation of JDBC Use the JDBC 2 driver (JConnect 5.2/5.5) com.sybase.jdbc2.jdbc.SybDriver
Open Client
Use with caution Most applications written using Open Client expect an ASE server
20
See Chapter 32 of the ASA Users Manual as well as the ASA Reference Section Transact-SQL and SQL/92 compatibility options for complete list of differences If writing stored procedures or embedded application code, make sure to explicitly make settings for compatibility as these options will get set to different values for Open Client vs. ODBC connections
21
ODBC and JDBC connectivity support all IQ datatypes Open Client does not support all IQ datatypes
Unsigned integer Depending on version of Open Client, bigint and unsigned bigint should be OK (OC 12.5 and later)
22
Packet Sizes
Larger packet sizes will help with large data retrieval Use p option in IQ configuration file to increase size Use CommBufferSize parameter in ODBC connection string
23
Network Speed
Data retrieval will be as fast as the network 100 MB of data will take 80 seconds on a 10 Mbit LAN 100 MB of data will take 8 seconds on a 100 Mbit LAN 100 MB of data will take 0.8 seconds on a Gigabit LAN LAN speed may be the performance bottleneck for queries that return large amounts of data The faster the network cards and LAN the better off concurrency will be: more available bandwidth per user Adding multiple network cards to the IQ node will help with network
24
25
26
IQ 12.4.3
The routine to check the database for potential corruptions is SP_IQCHECKDB There are no run-time options for this command, however the DBCC options control the behavior of the SP_IQCHECKDB command Recommendations Run Level 1 (DBCC_OPTION=1) DBCCs every 1 to 4 weeks (will process 1-2 GB per second) Run Level 0 (DBCC_OPTION=0) DBCCs every 1 to 3 months (will process 50-100 MB per second) Use DBCC_TEAM_SIZE option to control parallel execution
27
IQ 12.5
New syntax for IQ 12.5 See the Reference Manual Recommendations Perform CHECK every 1 to 4 weeks Perform VERIFY every 1 to 3 months Perform REPAIR if errors are reported from VERIFY Use RESOURCE_PERCENT parameter to control CPU usage during DBCC.
28
Recommendation
Create no more indexes in a batch than you have CPU's Note: Two indexes on same column will be serial
29
Adjust allocations from here based on monitoring Number of HG indexes in a table may alter this plan for loading data Make sure not to starve other operations that need RAM (OS, LOAD_MEMORY_MB, other applications, etc)
30
IQ Monitor is a diagnostic tool for DBAs It collects and reports internal counters from the IQ Buffer Caches
Main Cache interaction with the IQ Store Temporary Cache interaction with the IQ Temp Store
IQ Monitor offers a series of views of the counters to showing differing aspects of the server and buffer cache workload
31
You must specify an option when you start the monitor to specify what view to monitor One Monitor may be running for a cache
May have one for each (Main & Temp)
32
Solaris and AIX (64-bit) limited to RAM + swap space with no kernel changes necessary AIX (32-bit) limited to 2.5 GB using iqsmem to achieve this HP-UX limited to kernel value of maxdsiz_64bit
maxdsiz_64bit should never be set to exceed RAM + swap space
33
Keep in mind that more memory may mean tuning the O/S On Solaris, a system with 288 GB spent a tremendous amount of time managing memory with standard configuration
Changed following kernel parameters to help (/etc/system): set autoup=900 set tune_t_fsflushr=1 set lotsfree=0x10000 set bufhwm=8000 set maxphys=1048576 set maxusers=2048 set rlim_fd_max=2048 Keep in mind that above options were for a limited test and were the result of site specific analysis performed by Sun and should be used as reference only
Assess whats running and how much memory it uses (exclusive of IQ) Typically these include
Operating System OLAP Servers Middleware Other applications
35
Managing Memory
Deduct that total from RAM Allow 20% for File System Cache
If using file systems
Deduct another 10% (just in case) The rest is for IQ and IQ Caches A picture tells this story
36
Start with the O/S and other applications Determine what the IQ server needs (normal load) IQ Overhead (Heap for loading) Allocate IQ Caches
40% IQ Main Cache 60% IQ Temp Cache
Operating System + All Other Apps IQ Server IQ Overhead IQ Main Cache IQ Temp cache
37
38
Use the smallest datatypes possible for data Be aware of all datatypes in IQ there may be more than you know If hour, minute and second information is not necessary, use DATE instead of DATETIME If the data will fit within a TINYINT or SMALLINT datatype use that rather than INTEGER or BIGINT Allows the engine to store data in smaller units (1-byte TINYINT or 2-byte SMALLINT versus 4-byte INTEGER or 8-byte BIGINT Dont over allocate storage when defining NUMERIC() or DECIMAL() as it can be costly for data that doesnt need all that space
39
40
Consider using Minimize_Storage option in Sybase IQ 12.5. This will place an IQ UNIQUE(255) on every column for every table created and removes the need to use IQ UNIQUE If the value is <= 255 then IQ will place a 1-byte FP index on the column 1 byte of storage per row If the value is > 255 but <= 65536 then IQ will place a 2-byte FP index on the column 2 bytes of storage per row May slightly hinder data loads, but improve query speeds May incur onetime slight load slowdown while 2-byte FP is converted to flat FP, but this usually happens during the first load
41
Allows the optimizer a better guess at join criteria NULL data does not save space on the database page as it would in ASE Will be compressed out when stored on disk
42
Use unsigned datatypes where possible Use for surrogate keys and join columns Unsigned data comparisons are quicker The caveat to this is that Open Client may misinterpret the value if it is too large as it does not understand large unsigned data
Can convert to signed integer, numeric, or decimal if returning data to an Open Client application This caveat applies to moving data between IQ servers with INSERT FROM LOCATION
43
Can be used to store moderate amounts of text or binary data VARCHAR() or VARBINARY() datatypes
Maximum width is 32K (64K ascii hex for VARBINARY())
The WORD index is the only index allowed on VARCHAR() data wider than 255 bytes Storage will be allocated in 256 byte chunks
A 257 byte string will require 512 bytes of storage Much less than the 2K requirement in ASE TEXT and IMAGE columns
44
Can be used to store binary or text based objects Extends the long binary datatype from a maximum size of 64K to an unlimited size!!! FP index is the only viable index Cannot be searched, only stored and retrieved Special function to return the size of an object (byte_length64) Special function to return portions of the object, not the entire contents (byte_substr64)
45
No change to load table statement, so how is data loaded in the LOB? Through a load table source data file:
1|/tmp/lob_1.txt| 2|/tmp/lob_2.txt| 3|/tmp/lob_3.txt|
Load engine will place the contents of the file specified in the source data file (/tmp/lob_1.txt) into the long binary column for that row Can also extract contents of a binary object to individual files with the BFILE() function
46
Use CHAR() whenever possible All storage in IQ is fixed width VARCHAR() types only add overhead
A VARCHAR(100) columns will require 101 bytes of storage 100 bytes for data 1 byte for the size of data
47
If uncertain, place an LF or HG index on the column Use Primary Key, Unique Constraint, or UNIQUE HG indexes where appropriate
48
A column with an HNG, CMP, or WD index should have a corresponding LF or HG index (very rare circumstances will negate this) Indexes are not needed on columns whose data is ONLY returned to the client (projected) Remove HNG indexes on date/time/datetime columns. Replace them with DATE, TIME, or DTTM indexes Investigate the use of HNG indexes. In certain circumstances having HNG indexes on columns can hurt performance (mostly in aggregations).
49
Ask yourself these simple questions about each column to determine which indexes belong
Is the cardinality greater than 1500-2000? Yes: place an HG index on this column No: place an LF index on this column Does this column contain date, time, datetime, or timestamp data? Yes: place a DATE, TIME, or DTTM index on this column (these should be used in lieu of the HNG index recommended in IQM 12.4.3 and earlier) Will this column be used in range searches or aggregations? Yes: place an HNG index on this column (if the aggregation contains more than just the column, an HNG may not be appropriate) Will this column be used for word searching? Yes: place a WD index on this column Will this column be compared to another column in the same table? Yes: place a CMP index on the two columns being compared
50
Currently, only HG, UNIQUE HG, UNIQUE CONSTRAINT, and PRIMARY KEY indexes support multiple columns HG inserts are the most expensive in IQ Try to guarantee that inserts will happen at the end of the index
Place generally incrementing data at the beginning of the index list (sequential data) For instance, a transaction date or batch number Something that will try to guarantee a sequential key
51
Prefer joining integer datatypes (unsigned if possible) Integer comparisons are quicker than character comparisons Keep the datatypes as narrow as possible to improve join performance by reducing disk I/O and memory requirements Prefer using HG index on join columns rather than cardinality appropriate index (LF or HG)
52
Multi-column primary keys should have an additional LF or HG index placed on each individual column
Must be done manually
UNIQUE CONSTRAINT, UNIQUE HG, and Primary Key are identical structures: HG index with no G-Array Use when possible
Helps optimizer make more informed query path decisions even if index is not used in joins or searches You get an HG Index created automatically This HG index has no G-Array (uses less space)
53
Should be used with IQ 12.5 and later to aid in query join performance Use when possible
Helps optimizer make more informed query path decisions even if index is not used in joins or searches You get an HG Index created automatically on the foreign key column in the current table Requires that a primary key exist on referenced table
54
55
Temporary tables are available at the current level (parent) and all of its children A parent cannot see a child's temporary table(s)
56
57
IQ (ASA) is not 100% T-SQL Compatible, but very close Recommend using Watcom SQL
All system procedures written with it Many more code examples and more IQ people versed in it Dynamic SQL Better Loop control Full cursor movement rather than just read next Cannot mix T-SQL with Watcom SQL
58
Global variables Variable Names CALL FOR ASA requires variables to be declared immediately after a BEGIN
Use transaction control around logical units of work, even read only queries Should commit before a read/write batch is started to ensure latest version of data is available Should issue commit and rollback after batch completion to release all query resources Rollback will free memory resources in use by previous operations For systems with high number of connected users, freeing memory resources can aid in query performance
60
Custom functions can be written in either SQL or Java Great way to encapsulate business logic for transforming data Can have a significant performance impact on queries Functions are executed in the catalog portion of the engine
All result rows may need to be moved to ASA Can be time consuming for large result sets
Turn on query plans to see what impact the functions have on effective query plans
61
T-SQL Outer Join: *= or =* ANSI Standard/Watcom SQL Outer Join: [left|right|full] outer join Be careful of outer join syntax The T-SQL syntax can be very ambiguous or non-deterministic for IQ to translate
All T-SQL outer joins must be converted to ANSI outer joins and then processed
Use the ANSI standard instead as they are not ambiguous and are always clear in their meaning Visit http://my.sybase.com/detail?id=1017447 for more details
62
63
Can use FILLER() clause with a delimiter or byte count Better performance achieved by casting the date or datetime formats rather than letting IQ guess If possible, issue a single load table with multiple files rather than 1 load table per file to be loaded into a table
Only important for tables with HG, Unique HG, or Primary Key indexes
64
Great way to move data from any Open Client source Syntax:
insert into TABLE() location SERVERNAME.DBNAME { select statement };
IQ username and login must match on remote system Interfaces entry must match the SERVERNAME Can also be used to move data quickly from an ASA table to an IQ in the same server See section on Open Client datatype limitations
65
Avoid at all costs for large data manipulation operations Different from single statement operations that modify many rows Individual INSERT VALUES() will be slower than bulk load operations
Expect no more than 5,000 to 20,000 operations per hour
66
If multiple single row inserts are needed there is a method to improve performance Consider this code (1 insert per row):
insert into #my_table values ( 1, 1 ) insert into #my_table values ( 2, 1 ) insert into #my_table values ( 3, 1 )
Named pipes can be faster no disk I/O How to make a named pipe on most flavors of UNIX:
mknod PIPE_NAME p
Can use BCP, GZIP, UNCOMPRESS, or applications to push IQ formatted data into named pipe LOAD TABLE command can read from named pipe Can also fast extract data to named pipes so that they can be read by another application or even compressed and stored
68
Can induce fragmentation and overly large space consumption if not watched Row IDs from deleted data are not reused during a partial width insert operation Space from delete data is not reused (because Row IDs are not reused) Partial width inserts are analogous to APPEND_LOAD=on in terms of Row ID and space behavior Only becomes a problem if partial width inserts are a way of life for a table
69
70
71
72
Data Extraction
For fastest data unloads use the TEMP_EXTRACT options Can unload data in ASCII or BINARY format
Recommend BINARY format as it is faster to reload into IQ
Great way to archive portions of the database Can unload to one or more files, serially Avoid using ISQL or DBISQL redirection to a file
Much slower than fast unloads
73
Th
Questions?
74