TechTarget Enterprise Applications Media

Bolstering business intelligence with a sound data integration plan

to be effective, trusted and adopted by organizations. Since most BI initiatives require BI professionals designing data integration processes must address difficult questions: so data lineage can be traced back to the originating source systems? The issues are considerations for BI programs and learn about new tools and methods that may help, trends in the data integration market. for data integration and get a top analyst firms take on the leading vendors and right data integration style for your BI initiative. Find out specific data integration challenging, but not insurmountable. In this E-Book, learn more about choosing the and/or BI system? How often must this occur? Does the system require audit trails, information from multiple source systems, data integration is a critical consideration. How will data from multiple sources be reconciled and loaded into the data warehouse Business intelligence (BI) systems must have a solid data management infrastructure

E-Book

such as real-time data integration. Read more about the pros and cons of hand-coding

Sponsored By:

Bolstering business intelligence with a sound data integration plan Table of Contents

Bolstering business intelligence with a sound data integration plan

Table of Contents:
The real deal on data integration for business intelligence The three paths to real-time data integration Gartner data integration Magic Quadrant 2007: Platforms, market expand Data integration software vs. hand coding: Balancing costs and benefits Resources from SAS

E-Book

Sponsored by:

Page 2 of 15

Bolstering business intelligence with a sound data integration plan The real deal on data integration for business intelligence

The real deal on data integration for business intelligence

By Jeff Kelly, SearchDataManagement.com News Editor There are various ways to sabotage a business intelligence (BI) initiative, but perhaps none is more effective than ignoring key data-integration considerations at the outset. Data integrationconnecting data sources with data consumerslies at the very foundation of BI, according to

other obstacles, see to that. BI done right requires carefully selected data sources, particular attention to data quality, and an understanding of the ultimate business uses of data.

experts. But successful data integration is no easy task. Poorly labeled data fields and siloed data sources, among

semantic and other culturaland very real[obstacles] to success.

tive BI without good data integration or else youll be running off different sets of books and encountering various Ensuring a successful BI initiative requires attention to some key data integration considerations, according to

management at East Hanover, N.J.-based consulting firm Conversion Services International. You cant have effec-

Business intelligence is all about data integration, said William McKnight, senior vice president of information

McKnight and Rick Sherman, founder of Stow, Mass.-based business intelligence consulting firm Athena IT Solutions.

Get a grip on data granularities

BI users often want to drill down deep into data to gain better insights on the business. The first step in any BI tions, McKnight said. That way, when users delve into the data, they are all working off a common framework. data integration project is to understand the granularities of an organizations data in order to agree upon definiIn a retail company, for instance, what exactly defines a customer? Is it a single person or the entire household? If

said. How they are answered will largely depend on a companys business model and corporate culture.

hold customer account? Questions like these must be addressed before integrating data into a BI system, McKnight

another member of that household makes a purchase, is he or she a new customer or part of the existing house-

Not all data sources are created equal

tant to realize that different departments often have their own internal databases and spreadsheets that contain A lot of companies have grown up in silos and now are looking to integrate data to get a holistic view of the The next step is to take an inventory of data sourcesdatabases, data warehouses, spreadsheets, etc. It is impor-

overlapping information, which can lead to problems with data quality.

these various data sources must be reconciled.

times with their own IT departments or their own rogue IT departments. Before data can be integrated, he said,

business, McKnight said. That means there are different cuts of the data taken off in different departments, some-

Sponsored by:

Page 3 of 15

Bolstering business intelligence with a sound data integration plan The real deal on data integration for business intelligence

Athenas Sherman also stressed the importance of data quality in any integration initiative. Prior to the implementation of a BI system, he said, users in separate departments probably routinely make changes to data to suit their purposeschanges that are not relayed to other business units. Thats fine when the data never leaves a given They massage the data to make up for data quality issues, or gaps in the data, or things they dont think are department, but can wreak havoc on BI systems that access enterprise-wide data.

data, from across different systems. Thats when you start to see data quality issues.

right in the data, Sherman said. Once you [undertake] a BI initiative, then you want to look at more detailed

Foster buy-in from BI stakeholders

how they plan to use the resulting BI reports, analytics and dashboards. stand the choices that they have to pick from within the organization. In conjunction with addressing data quality issues, executives and analysts should also be consulted to determine

You need to pick data sources appropriately for the stakeholders of BI, McKnight said. And they should underUnderstanding how data will be used is essential to successful BI data integration, Sherman agrees, but underThe lifecycle of your data is very closely related to your business processes, Sherman said. Data used by a

standing data lifecycles is equally important.

manufacturing company, for instance, undergoes very different processes throughout its lifetime than data used integrate data for BI purposes during its lifecycle is a key success factor.

by a retailer. Even within industries, data lifecycles often vary. Both Sherman and McKnight said knowing when to Only after data source options are clear, data quality issues have been addressed, and BI business uses determined, can an organization actually begin integrating data with BI systems, McKnight and Sherman agreed. The reason why some BI projects take so long, sometimes nine months to a year or longer, is due to getting

data governance rules, Sherman said. Business intelligence really raises the bar for data integration.

business users to determine how theyre going to define and use the data, plus assuring data quality and writing

Sponsored by:

Page 4 of 15

Gophers burrow through life without seeing the havoc they create.
They cant help having tunnel vision. But you can. With proven business intelligence and analytic software from SAS.
www.sas.com/gophers

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 2008 SAS Institute Inc. All rights reserved. 00000US.0408

Bolstering business intelligence with a sound data integration plan

The three paths to real-time data integration

The three paths to real-time data integration

By Jeff Kelly, SearchDataManagement.com News Editor nate the scourge of out-of-date data. Real-time data integration can be achieved through a variety of methods, but Companies are increasingly turning from traditional batch-oriented techniques to real-time data integration to elimi-

also warn that it is not a methodology to adopt lightly.

better-informed business-critical decisions. Experts agree that real-time data integration is gaining popularity but

the goal is the same: to communicate accurate, timely data from point A to point B in real time so users can make

with Stamford, Conn.-based Gartner Inc. Any given company is going to have data integration requirements that oriented mode, and there [are] going to be other things where real-time data integration really does have value.

Recognize that the world is not a black-and-white place, said Ted Friedman, vice president and distinguished analyst

span the latency spectrum. There are going to be pieces that are best suited to be delivered in a high-latency, batch-

Real-time data integration options

The most common real-time data integration method is change data capture (CDC), also called data replication. transmit the change to a given target. CDC tools and technologies recognize when an important change has occurred in one data source and, in real time,

Bloor Researchs Philip Howard explains: As a change is made to a database record in your transactional system, whatever youve got running, so its ready to answer real-time queries.

for instance, its also actively captured and fed through to your data warehouse or business intelligence system, or

CDC is used most often to synchronize operational applications and for real-time business intelligence (BI) purposes, according to Gartners Friedman. Indeed, business intelligence is a major driver of real-time data integration adoption, he said, especially among businesses that require BI reports at a moments notice.

BI application in more of a real-time fashion.

is performing, [for example] Friedman said, then you need to be delivering data from some data sources to your

If youve got some type of short-cycle business and you need up-to-the-second analysis of how your supply chain

CDC is less ideal, however, if the goal is a comprehensive, real-time view of a single entity via data housed in multiple sources. For that, users more often turn to data federation, sometimes called enterprise information integration. Data federation is better suited to people that are looking at a more narrow slice of the data landscape, employeeas opposed to somebody whos doing historical trending in the data warehouse.

Friedman said. They want to get a complete view of a single instance of an entitya customer, a product, an

[data] needs to be in real-time, Friedman said.

ogy to search multiple data sources to obtain a comprehensive view of that customer while still on the call. That

For example, an insurance agent on a customer call might use an application supported by data federation technol-

Sponsored by:

Page 6 of 15

Bolstering business intelligence with a sound data integration plan

The three paths to real-time data integration

Both the CDC and data federation markets are well established, Howard said, having already gone through the players like Teradata and GoldenGate Software, offer a variety of solid CDC and data federation real-time data integration tools, he said.

consolidation phase that you tend to get once products start to mature. Large vendors like IBMwhich acquired

data integration specialist DataMirror last yearand Oraclewhich scooped up Sunopsis in 2006as well as smaller

Friedman also identified a third approach, what he calls the messaging-middleware method, in which real-time data integration is achieved through middleware technologies that connect applications. Think of IBM WebSphere MQ and Microsoft BizTalk Server, and products like that, that are really meant to do

application in a low-latency fashion. So its sort of like the data replication idea, but working at the application layer as opposed to the database layer. This middleware approach is ideal for inter-enterprise scenarios, when theres a need for real-time data integration among organizations that may not have access to one anothers data sources, Friedman said. A vendor might communicate an important data change to a supplier in real time using this method, for instance.

of something meaningful that happened, and these technologies propagate that message to another system or

granular, message-oriented propagation of data, Friedman said. An application on one end spits out a message

Real-time data integration raises data quality concerns

Both Howard and Friedman noted, however, that while there are many benefits to real-time data integration, there are numerous drawbacks as wellfirst among them, poor data quality. In more traditional, batch-oriented data real-time data integration, regardless of the method. integration processes, there is ample time to scrub and cleanse data before it reaches its destination. Not so with In the middle of that process [batch-oriented data integration], youve got a chance to actually analyze and

cleanse that data, Friedman said. In the world of real-time data integration, theres less opportunity to apply integrated in real time will be of poorer quality, incorrect or misleading.

very sophisticated files for analyzing the quality and cleansing the data. There is a higher risk, then, that data

Friedman said current real-time data integration tools are better at data transformation and cleansing than theyve time data integration quality could be achieved, he said, as the problem is more technological than conceptual. been in the past, but there is still plenty of room for improvement. It is possible that someday near-perfect real-

organizations, and in some cases may even prove detrimental. Friedman advises users to match their data integrabasis, for example, would in that case have no need for real-time data integration, which could actually cause more harm than good, partly because of the already mentioned data quality concerns. tion method to their latency requirements. An organization that routinely analyzes certain data sets on a weekly

Both analysts said it is also important to recognize that real-time data integration isnt ideal for all companies and

Sponsored by:

Page 7 of 15

Bolstering business intelligence with a sound data integration plan

The three paths to real-time data integration

Organizational structure and corporate politics also play a role in determining the appropriateness of real-time data integration, Friedman said. If users arent ready to accept and use real-time data, theres little point in integrating data in real time in the first place.

decisions, the culture and the politics of the organization are not set up for them to act on real-time information, Friedman said. I think thats a limiting factor for many organizations today. Howard agreed, pointing to what he called decision-making latency. have a meeting with five other people and it takes two days to arrange that, or even two hours to arrange that, then you dont need real-time [data integration], Howard said.

Frankly, I know some companies that if they had real-time BI it wouldnt matter at all because the way they make

How soon can you as a human being make a decision based on new information that youre given? If you have to

then thats where real-time decision making becomes important.

He added: If you can make a decision instantly Ah, this has happened, therefore I know to do such-and-such

Sponsored by:

Page 8 of 15

Chickens are hypnotized by drawing a line in the dirt over and over.
They find it hard to step past the obvious. But you can. With proven business intelligence and analytic software from SAS.

www.sas.com/chickens

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 2007 SAS Institute Inc. All rights reserved. 00000US.04076

Bolstering business intelligence with a sound data integration plan Gartner data integration Magic Quadrant 2007: Platforms, market expand

Gartner data integration Magic Quadrant 2007: Platforms, market expand

By Hannah Smalltree, SearchDataManagement.com Senior Site Editor Data integration buyers now have many choices of vendors and platformsbut theyre increasingly interested in one method in particular, according to a co-author of Gartner Inc.s Magic Quadrant. The market continues to experience a dichotomy, with a changing cast of vendors delivering data integration platof Cognos and SAPs acquisition of Business Objects. But other, smaller, pure-play vendors are sprouting up, too. sion criteria but still added to the markets growth.

forms, according to Ted Friedman, vice president and distinguished analyst with the Stamford, Conn.-based analyst

firm and co-author of the study. Big vendors are getting bigger through acquisition, as evidenced by IBMs purchase There are additions to the study this yearWoodcliff Lake, N.J.-based Syncsort Inc. and Hummingbird Connectivity, a division of Waterloo, Ontario-based Open Text Corp., as well as a host of other vendors that didnt meet the incluBuyers are demanding data integration platforms that support a range of different styles of data delivery, a trend

support ETL and at least one other integration style, like real-time data replication or data federation, Friedman tooling, metadata and runtime architectures, according to the study.

noted in last years study and mirrored in this years inclusion criteria. The study looked only at platforms that

said. Platforms are converging at a deeper technology level as well. That means theyre based on common design

capture (CDC), Friedman said. CDC tools help companies recognize and capture important changes that are occurit acquired Data Mirror, and Friedman has often been asked about it recently. ring in one data source and then propagate that changed data to another target. IBM talked up that feature when

The data integration method garnering more interest now is real-time data integration, particularly change data

The terminology thats emerging is event-driven approaches to data integration, Friedman said. Its growing of synchronizing data across different systems and databases.

more than that, people are interested in the technology because it enables a more time-effective, low-latency way Theres increased interest in data federation, too, though Friedman said inquiries about it are a distant third to stantial use of open source data integration tools on an enterprise-level scale, he said, if the freely downloaded which in the longer term could affect the market in a fairly dramatic way, he said. Though hes not seeing sub-

fairly substantially in popularity. Real-time, operational business intelligence is pushing some of that interestbut

questions about ETL or event-driven approaches. And buyers are also asking about open-source data integration,

tools mature to offer more functionality, commercial vendors may begin to feel some pricing pressure.

Sponsored by:

Page 10 of 15

Bolstering business intelligence with a sound data integration plan Gartner data integration Magic Quadrant 2007: Platforms, market expand

Enterprise data integration tool vendor rankings 2007: Leaders lead, others jockey for position
Informatica. Both were in positions similar to those of last year, thanks to continuing to meet market demands with The leaders quadrant of the study, for vendors with high marks for vision and execution, hosted IBM and

versatile platforms, Friedman said.

Business Objects, Microsoft and Oracle all appeared in the challengers quadrant, for vendors with less vision but said, these vendors need to do more to articulate a comprehensive vision.

strong customer execution. The latter two have vision challenges, Friedman said. While their position as challengers is largely a result of increased customer adoption of SQL Server Integration Services and Oracle Data Integrator, he

its breadth of integration capabilities and vision, particularly around the role of data integration in service-oriented architectures, Friedman said. SAS Institute was also in the visionaries quadrant with a strong, versatile platform.

this year. iWay, a subsidiary of business intelligence vendor Information Builders, made it into this section thanks to

The visionaries quadrant, for vendors with good vision but not as much execution experience, had just two vendors

from its position in the challengers section last year. Pervasive has a strong product with a mix of capabilities, and the cost-to-functionality ratio is good, Friedman said, but the vendor didnt have enough enterprise-class implementations or execution experience to be in the challengers quadrant this year. Others in the niche quadrant were Cognos, Friedman is interested to see how SAPs acquisition of Business Objects affects the space, since both vendors have Sybase, Tibco Software, SAP, ETI, Sun, Pitney Bowes Group 1 Software, Syncsort, and Hummingbird Connectivity.

still met the studys inclusion criteria. Near the top of the section was Pervasive Software, which dropped back a bit

Most vendors were clustered in the niche quadrant, reserved for those not as strong on vision and execution but which

integration products and little is known yet about how they may rationalize these capabilities.

Advice for enterprise data integration platform purchasers

Buyers should definitely look at data integration platforms with breadth or many integration capabilities, Friedman saidand should think beyond ETL for a data warehouse. You can get a lot of value out of these tools for a range of different scenarios, he said, not just ETL for the ware-

house, but consolidations, conversions, application integration and more.

Organizations should also look for strong metadata management and modeling functions, Friedman said. Basic intethat can help them truly understand data assets, relationships between data, and how data changes over time. tures (SOA), Friedman said. gration capabilities may become commoditized in the long term, he said, so organizations should align with vendors

Finally, buyers should seriously think about the importance of data and data integration in service-oriented architec-

find thats a concept that too few organizations are thinking about today.

Consider how these types of tools can be used to help facilitate and add value in your SOA work, he said. We

Sponsored by:

Page 11 of 15

Need to turn your information stampede into a logical path for success? You can.
Find your direction with assets like this using the SAS Platform Pathfinder. www.sas.com/pathfinder1

Bolstering business intelligence with a sound data integration plan Data integration software vs. hand coding: Balancing costs and benefits

Data integration software vs. hand coding: Balancing costs and benefits
By Hannah Smalltree, SearchDataManagement.com Senior Site Editor As organizations consider investing in data integration software, they encounter a familiar question: When and why does it make sense to buy technology to do something that you could do manually? Organizations have long relied on in-house developers to hand-code scripts for data integration projects. Yet, for Canadian retailer Home Hardware Stores Ltd. tion platforms. But many companies still use hand coding for data integration projectsincluding, until recently,

almost 20 years, theyve had alternatives in the form of extract, transform and load (ETL) tools and data integra-

pany to consider data integration software, according to Will Buddell, senior systems analyst with the St. Jacobs, was no small integration task.

Two years ago, when Home Hardware was upgrading its merchandising system, a consultant encouraged the comOnt.-based retailer. The hardware, building materials and furniture retailer had to merge its product database with

an acquired companys product database. With about 100,000 different products, combining two Oracle databases

from Austin, Texas-based Pervasive Software Inc. But first, Home Hardware ran a comparison test.

would provide productivity gains, according to the consultant. The team zeroed in on the Data Integrator software

Though Home Hardware had the in-house skills necessary to manually hand-code scripts, data integration software

We had an internal development team that produced a test XML file. Time on the books for them to produce it was used that as the basis to go forward, because it showed that we could reduce our development time drastically. Hand coding wouldnt have been difficult, Buddell saidjust time-consuming. In contrast, it took only a day or so to target mapping made it especially user-friendly, he said. When it was actually time to merge the data for phase one not as long as hand coding would have taken, Buddell said. Plus, time is money. Home Hardware estimates that the of the project, it only took about two hours. Integration for other phases of the project sometimes took longer, but set up the Data Integrator software. The pre-built database connectors and a drag-and-drop interface for source-toabout 20 hours, Buddell said. With [Data Integrator], we produced the same XML file in about four hours. We

software helped it save anywhere from $200,000 to $500,000 (Canadian dollars) during the project, he said.

Why use data integration software?

founder of Stow, Mass.-based consulting firm Athena IT Solutions. In addition to saving time, many common, Productivity gains are a big reason that many companies use data integration software, according to Rick Sherman,

repeatable functions are coded into the software, so that companies dont have to reinvent the wheel every time they do an integration project. Thats not all.

Sponsored by:

Page 13 of 15

Bolstering business intelligence with a sound data integration plan Data integration software vs. hand coding: Balancing costs and benefits

aged for version control, Sherman said. With hand coding, you have code all over the place, little management and little documentation.

An ETL tool or data integration platform means that you have all of the code in one place, documented and man-

And then there are issues like auditability and transparency, since regulations like SOX or HIPAA demand that companies be able to trace data back to their original source systems. Data integration software helps by automatically documenting information about data sources and transformations, making for an easier audit trail.

Thats a problem when companies need to replicate integration procedures, troubleshoot or change code, he said.

Many still dont see the benefits

However, many companies still dont use data integration software, Sherman said. His experiences as a consultant common for them to revert to hand coding for data marts, operational data stores or other projects. indicate that while Fortune 500 companies often use data integration software for a large data warehouse, its

robust integration platform can be a six-figure proposition for a large company, he explained. Thats because the Add this all up and many companies see in-house development as the path of least resistance.

The top inhibitor is likely cost, Sherman said. While available ETL tools run the functionality and pricing gamut, a

scale of the project affects the price. Data integration software is often either priced by concurrent CPU usage or by the number of source and target systems. Beyond licensing costs, training is often required for platforms, he said.

an ETL tool.

mer leaveswhos going to maintain that persons code? If you worry about what you would do, you probably need

Theres seemingly no cost to coding, just the person you have doing it, Sherman said. But, when that program-

Justifying the cost of data integration software

Certain systems may justify the use of data integration software more than others, Sherman added. While a one-time data aggregation project may not require software, any system updated or accessed frequently is a good over time to encompass more data sources, may also benefit. Additionally, any sort of financial reporting system, especially those supporting SOX compliance, has good reasons to rely on data integration software. The cost justification for data integration software is often more qualitative than quantitative, Sherman said. Theres

productivity gains of software. Systems accessed by many business users, or applications that will be expanded

candidate. For example, a business intelligence system, with data loaded on a regular basis, could benefit from the

not a straightforward ROI equation. But the soft benefitsespecially around auditability and transparencyare often compelling enough on their own to convince CFOs and CIOs to sign off on purchases, Sherman said.

Sponsored by:

Page 14 of 15

Bolstering business intelligence with a sound data integration plan Resources from SAS

Resources from SAS

Report: Competing on Analytics

successful 2007 Competing on Analytics event tour, this compilation features analytics expert, Thomas H. Davenport and highlights his groundbreaking findings. Pick up best practices and experiences from executives on how their organizations are building effective competitive strategies around data driven insights and trumping their rivals. Online Assessment: Platform Pathfinder ages and uses information. Your journey includes: organization.

Want to learn how specific organizations are using analytics? You can. Derived from summary reports of the hugely

Do you want to turn your information stampede into a logical path for success? You can. SAS invites you to visit our Platform Pathfinderan innovative Web site that provides an opportunity to determine how your organization man A five-minute Business Intelligence Assessment survey

An objective look at the current roles of analytics, business intelligence and data integration in your demos. Learning resources tailored to your survey responses, including white papers, Webcasts and product

Discover more about the organizations who use SAS. You can. SAS customers represent many of the most Global 500 List.

SAS Success Stories

innovative and successful organizations in the world, including 96 of the top 100 companies on the 2007 FORTUNE

About SAS
software to improve performance through insight from data, resulting in faster, more accurate business decisions; SAS is the leader in business intelligence and analytical software and services. Customers at 43,000 sites use SAS

more profitable relationships with customers and suppliers; compliance with governmental regulations; research business intelligence applications within a comprehensive enterprise intelligence platform. Since 1976, SAS has been giving customers around the world The Power to Know. http://www.sas.com/

breakthroughs; and better products and processes. Only SAS offers leading data integration, storage, analytics and

Sponsored by:

Page 15 of 15