Creating Trustworthy Master Data Using SAP BusinessObjects Data Services

Asif Pradhan

[ SYLVIE GAUTHIER
ASUG INSTALLATION MEMBER MEMBER SINCE: 1999

[ RAECHAL MARTIN
ASUG INSTALLATION MEMBER MEMBER SINCE: 2004

[ CHARAN MARWAH
ASUG INSTALLATION MEMBER MEMBER SINCE: 2003

[ Learning Points
Master Data and its challenges SAP BusinessObjects Data Services and SAP NetWeaver Master Data Management (MDM) Steps to create trustworthy data using SAP BusinessObjects Data Services Profiling Address Cleansing Data Cleansing Matching

Real Experience. Real Advantage.

[ Heart of Every Organization - Data

Data - All organizations, partners, individuals and systems depend on it

Real Experience. Real Advantage.

[ What is Master Data?

It is the practice of defining and maintaining consistent definitions of data, then sharing them via integration techniques across multiple IT systems within an enterprise. TDWI MDM research Every company has master data Master data supports critical business processes across the enterprise Master data is a strategic corporate asset

Real Experience. Real Advantage.

[Master Data is Crucial to Your Business

SRM
Part: 8975 VENDOR: ABC123

But Every Department has a Different Version of it

Master data is data about your customers, products, suppliers etc.
Logistics
VENDOR: XYZ456

Trading partner data introduces yet another version of data Inaccurate data leads to $30+ billion cost on supply chains

ERP
199, 3rd Street Palo Alto, CA

Jane Peters

Part: B7521

Call Center
Jane Smith 4418 N. Str. Chicago, IL 60611 Part: 2574

Real Experience. Real Advantage.

[Master Data Management - Common Questions

Which employee should we assign to? Do I have the right product? Who is my best supplier? Who is my customer?

ERP Jane Peters 199, 3rd St Palo Alto, CA

Call Center

Jane Smith 4418 N. Str. Chicago, IL 60611

Real Experience. Real Advantage.

[Master Data Communication - Data Consolidation and

Flow Control
Manufacturer Distributor Company

Data travels in multiple paths. This way you receive the same record multiple times. It is not easy to re-consolidate the records into one record. How do I handle complex data flows?
Real Experience. Real Advantage.
7

[External Driven Master Data Updates - Periodical

Data Synchronization
SAP

Real-world Updates
Neurother Strasse vs. Business Objects vs.

Dietmar Hopp Allee SAP BusinessObjects

Your Your Company Data Storage Sites Company Data Storage Sites

Data travels in multiple paths. This way you receive the same record multiple times. It is not easy to re-consolidate the records into one record. How do I keep track of external data changes?
Real Experience. Real Advantage.
8

[Master Data Creation and Maintenance - Ensuring

Data Quality
My name is Bob Builder. I would like to Bob Bilder Constreet 2

?
I cant find you in the system. But I can create a new record to help you.

Bob Builder Con Street 2 Bob Builder Renovation 6

Records are created due to unavailable or inaccurate information. The quality of the record content is often not ensured. This often causes disruption in business process flows. How do I ensure data quality from the beginning?
Real Experience. Real Advantage.
9

[Enterprise Master Data Pain Points

Reduced Sales Effectiveness Lack of consolidated view of the customer across channels Customer transactions are compromised due to limited view of the facts Sub-optimal Procurement Decisions Lack of transparency into suppliers and products High costs due to insufficient supplier selection and rationalization Failed Marketing Plans Failure to capitalize on market opportunity when introducing new products Lack of real-time collaboration with front office, thus hindering customer services

Real Experience. Real Advantage.

10

[Enterprise Master Data Pain Points contd

Ineffective Business Decisions Lack of consistent information to support innovation and growth Compromised business dynamics with negative impact on managing resources effectively Not Actively Addressing Data Quality Garbage in, garbage out Distrust of data in the system Overwhelming loading exceptions - lack of transformation and normalization of data

Real Experience. Real Advantage.

11

[ Impact of Poorly Managed Master Data on Processes

Quality of master data dramatically impacts transactional and analytical processes What is worse, youll only notice it after the damage is already done

Real Experience. Real Advantage.

12

[Results of Poorly Managed Data - Negative impact on

businesses in all industries
40% of orders blocked due to master data problems. EMEA High Tech Company 90% of upper level management feel they dont have the necessary information for critical business decisions; 56% of them are afraid they are making poor decision because of it. Economist Higher performing companies are 50% more likely to use analytical information strategically. Competing on Analytics,Thomas Davenport Through 2010, Global 1000 enterprises will incur more costs, due to poor-quality data, than the benefits they will gain from implementations of customer relationship management, enterprise risk management and business intelligence applications. Key Issues for Data Management and Integration Initiatives 2009, Gartner

Real Experience. Real Advantage.

13

[Typical Causes of Loading Messy Data

A critical component of your master data project is understanding your source systems and recognizing any errors, inconsistencies or omissions in the data. Data content errors Errors in migration (ETL)

Missing data Invalid data Data domain outliers Data inconsistencies

Normalization inconsistencies Duplicate or lost data

Multiple formats for same data elements Different meanings for the same code value Multiple code values with the same meaning Field overuse: used for unintended purpose. Data in filler

Data structure problems Referential integrity problems

e.g. your ORDER DETAILS table contains part IDs that do not exist in the PARTS table.

Real Experience. Real Advantage.

14

[ SAP BusinessObjects Data Services and SAP NetWeaver

Master Data Management (MDM)
Master data is the ultimate goal of Enterprise Information Management
Ensure data remains trustworthy with Master Data Management Make data trustworthy with Data Services (Data Quality)

Load integrated data into the right place with Data Services (Data Integrator)

SAP BusinessObjects Data Services + SAP NetWeaver MDM

Real Experience. Real Advantage.
15

[SAP BusinessObjects Data Services

Integrate and Improve the Quality of All Data

Market-leading, unified solution for enterprise-class data integration and data quality
Single, easy-to-use user interface to build, test and deploy projects Connect, transform and make the information available from virtually any sources Comprehensive data quality solution for cleansing and enriching all types of data Embed data quality steps directly into the ETL process to build a data warehouse

Having all this functionality as part of one application makes it easy to select from the different data transforms provided the software and run data through them quickly and efficiently. Richard West President, Peachtree Data Inc.
Real Experience. Real Advantage.
16

[SAP BusinessObjects Data Services

Data Integrator: Enterprise-Wide Data Access

Support for Structured and Unstructured Data Broad connectivity to databases, applications, legacy and file formats
Databases Oracle IBM DB2 Microsoft SQL Server Sybase ASE & IQ Informix Teradata Netezza HP NeoView MySQL ODBC Applications JD Edwards Oracle EBS PeopleSoft Siebel Salesforce.com SAP NetWeaver BW & BWA SAP ERP & R/3 ABAP BAPI IDoc Files/Transport Text delimited Text fixed width EBCDIC XML Cobol Excel HTTP JMS SOAP (Web Services) Mainframe
(with Partners)

Unstructured Data Any text file type 32 languages

ADABAS ISAM VSAM Enscribe IMS/DB RMS Both direct and change data

Real Experience. Real Advantage.

17

[Data Services Designer

Data Flows move, convert, validate, and enhance data

DQ-Transform

Real Experience. Real Advantage.

18

[SAP BusinessObjects Data Services capabilities

Cleanse Validate

Deliver Access

Profile

Real Experience. Real Advantage.

19

[ SAP NetWeaver Master Data Management (MDM)

Deliver a Single View of Business Information
Consolidate, harmonize, and centrally manage master data of SAP and heterogeneous environments
Deliver trusted product, customer, supplier master data and global data synchronization Pre-integrated with SAP BusinessObjects Data Services for market leading data quality and data integration Pre-packaged IT and business usage scenarios Compose cross-application processes in SOA with consistent master data

Prior to using SAP NetWeaver MDM and SAP BusinessObjects IM solutions, our heterogeneous IT landscape made it difficult for us to view our total product line, what our customers were buying, and how we could serve those customers better. Joe Young Senior Manager, IT, Lexmark International, Inc.
Real Experience. Real Advantage.
20

[SAP NetWeaver MDM - Manage master data in your

industry-specific business processes

MDM

TAKE ORDER

VERIFY AVAILABILITY

IDENTIFY SUPPLIER

MANAGE CUSTOMER

Which employee should we assign to?

Do I have the right product?

Who is my best supplier?

Who is my customer?

Real Experience. Real Advantage.

21

[The Power of our Combined Product Offering -

SAP NetWeaver MDM and SAP BusinessObjects Data Services

Real Experience. Real Advantage.

22

[ Comparing Data Services (DQ and DI focus) with MDM

Data Services Capabilities Hierarchy Management Governance and Stewardship Workflow Management Flexible Data Modeling Authoring Master Data Match / Merge Universal Data Cleanse Address Cleansing Data Enhancement Bulk Data Movement Complex Transformations Broad sources / targets support
Real Experience. Real Advantage.

MDM

DQ

DI

23

[ Comparing Data Services with MDM

Real Experience. Real Advantage.

24

[Creating Trustworthy Data with SAP BusinessObjects

Data Services
Measure and analyze data through data assessment and continuous monitoring Cleanse and enhance customer and operational data anywhere across the enterprise Match and consolidate data at multiple levels within a single pass for individuals, households, or corporations

Real Experience. Real Advantage.

25

[ Creating Trustworthy Data Profiling

Real Experience. Real Advantage.

26

[ Data Profiling/Assessment measure and analyze

Inspecting data Measuring the data defects Analyzing the cause and impact of those defects Reporting the results of the analysis to key stakeholders

Real Experience. Real Advantage.

27

[ Data profiling - Analyzes Contents, Quality, Structure and

Relationships
1. Analysis of data beyond viewing
Frequency distribution Distinct values Null values Minimum/Maximum values Data Patterns (e.g. Xxx Xxxx99, 99-Xxx)

2. Comparison of values between data sets to determine fit 3. Can drill down to view specific records
Real Experience. Real Advantage.
28

[ Creating Trustworthy Data Cleansing

Real Experience. Real Advantage.

29

[Creating Trustworthy Data Address Cleansing

Worldwide Address Coverage

Statistics
City Level Validation Street and House Level Validation Country specific = 36 All world (countries & territories) > 240

Real Experience. Real Advantage.

30

[ Creating Trustworthy Data Address Cleansing

Worldwide Address Coverage

Real Experience. Real Advantage.

31

[ Standardization Example: United States Postal Service (USPS)

address preferences ADVANCE MOVERS

Recipient Line

1500 E MAIN AVE STE 201

Address Line

SPRINGFIELD VA 22162-1010

Last Line

Real Experience. Real Advantage.

32

[ Address components
Name Data (non-address data) Firm Data

Line 1 Line 2 Line 3 Line 4

Primary Range Locality

Jack Schneider Business Objects, an SAP company 332 Front St. South Postfix La Crosse, WI 54601-4023
Primary Type Region Primary Name Postcode

Real Experience. Real Advantage.

33

[ Prepare your input data

Before you start address cleansing, you must decide which kind of address line format you will input. Both the USA Regulatory Address Cleanse transform and the Global Address Cleanse transform accept input data in the same way. Address line formats:
Discrete Multiline Multiline Hybrid

Real Experience. Real Advantage.

34

[ Discrete Format
In discrete record format, data appears consistently in the exact same field arrangement in every record.
RECORD 1 Firm = Micro Elektronic Ges.m.b.H. Address1 = Baumschulengasse 250 Lastline = 1010 Wein Country = Germany RECORD 2 Address1 = Harborview Locality1 = La Crosse Region1 = WI Postcode = 54601 Country = USA

Real Experience. Real Advantage.

35

[ Multiline Format
In multiline format, data is not consistently located in the same arrangement in all records. That is, data items "float" among fields.
RECORD 1 Multiline1 = Oxford Publishing Multiline2 = Wetherby House Multiline3 = 20 Oakly Rd. Multiline4 Multiline5 = Harlow Multiline6 = Essex Multiline7 = CM19 5AE Multiline8 = United Kingdom
Real Experience. Real Advantage.

RECORD 2 Multiline1 = Gran Via de Carles S.A. Multiline2 = Multiline3 = Ibarra Campillo 16-3-A Multiline4 = 48010 Bilbao Multiline5 = Spain

36

[ Multiline Hybrid Format

In multiline hybrid format, data is in both discrete and multiline floating format.
RECORD 1 Multiline1 = Jonas Lancaster Multiline1 = Woodworks, Ltd. Multiline2 = Wetherby House Multiline3 = 20 Oakly Road Multiline5 = Locality2 = Harlow Locality1 = Region1 = Essex Postcode = CM19 5AE Country = United Kingdom RECORD 2 Multiline1 = Multiline2 = Akzo Chemicals Multiline3 = Multiline4 = Moreelsepk 24 Multiline5 = Multiline6 = Multiline7 = Lastline = 3511 EP Utrecht Country = Netherlands

Real Experience. Real Advantage.

37

[Creating Trustworthy Data Data Cleansing

Data Quality Data Flow

Real Experience. Real Advantage.

38

[SAP BusinessObjects Data Services

Data Cleanse Transform
Data Cleanse Dictionary

Parsing Rules

Real Experience. Real Advantage.

39

[ Creating Trustworthy Data Data Cleansing

Parse Data

Real Experience. Real Advantage.

40

[ Creating Trustworthy Data Data Cleansing

Standardize Data
Assign gender and prenames Create personalized greetings Create separate data for each person

Real Experience. Real Advantage.

41

[ Creating Trustworthy Data Data Cleansing

Prepare Records for Matching

Real Experience. Real Advantage.

42

[ Creating Trustworthy Data Data Cleansing

Parse, Standardize, and Correct
Input record
Maggie.kline@future_electronics.com Maggie Smith-Kline phd FUTURE Electronics 5/23/03 101 6th ave manhattan ny 10012 001124367

Output record
Salutation: Ms. First name: Maggie Last name: Smith-Kline Postname: Ph. D. Match standards: Margaret, Magdalena, Magnolia Company name: Future Electronics Address 1: 101 Avenue of the Americas City: New York State: NY ZIP+4: 10013-1933 Email: maggie.kline@future_electronics.com SSN: 001-12-4367 Date: May 23, 2003
43

Real Experience. Real Advantage.

[SAP BusinessObjects Data Services - Data Cleansing

Data Input Standardized Output

Real Experience. Real Advantage.

44

[ Creating Trustworthy Data Enhancement

Real Experience. Real Advantage.

45

[ Data Enhancement
Completes records with directory information by appending name, address, code data and more Provides geocoding capabilities for geographic and demographic marketing initiatives Provides geo-spatial assignment of customer addresses for tax jurisdictions, insurance rating territories, and insurance hazards, etc.
Real Experience. Real Advantage.
46

[ Data Enhancement
Directory information Geocoding information Geo-spatial information
Margaret Smith-Kline, Ph.D. Future Electronics 101 Avenue of the Americas-1933 New York, NY 10013 (222) 922-9922

Centroid Latitude: 40.723195 Centroid Longitude: -74.004977 Address Latitude: 40.723175 Address Longitude: -74.004970 FIPS State Code: 36 New York FIPS County Code: 061 New York FIPS Place Code: 51000 New York MCD Code: 44919 BSA Code: 35620 Metro Code: 5600 Section Code: 0051001012

Real Experience. Real Advantage.

47

[Cleansing and Enhancement Capabilities in Data Services

Extract, cleanse, normalize, correct, and validate data using predefined extractors Configure and execute rules to identify records with data quality concerns
Multi-line Input Record michael.schmidt@hypertech.com Michael Schmidt phd HYPERtech, Inc. 5/23/03 330 east 63 avenue # 10b Manhattan ny 10021 001124367 Output Record Enhancements, Casing and Standardization

Data Parsed into Individual Components

Salutation: Mr. First name: Michael Last name: Schmidt Postname: Ph. D. Match standards: Michael, Mike, Mick Gender: Strong Male Company name: HyperTech, Inc. Address 1: 330 E 63rd St Ste 10B City: New York Corrections State: NY ZIP+4: 10065-7706 Email: michael.schmidt@hypertech.com SSN: 001-12-4367 Date: May 23, 2003
48

Real Experience. Real Advantage.

[ Creating Trustworthy Data Matching

Real Experience. Real Advantage.

49

[ Creating Trustworthy Data Matching

Break Groups
Break Groups provide the ability to control the number of record comparisons in the match process. This is important for a couple of reasons:
Speed Optimal processing time is achieved with many small break groups; however, valid matches may not be identified if break groups are too small. Match quality Optimal match quality is achieved with fewer and larger break groups; however, larger break groups require more comparisons and processing time.

Real Experience. Real Advantage.

50

[ Break keys and matching

Creating Break keys All records Each record belongs to a postal code Set the break key on the first three digits of the postal code Records that contain 809 as the first three digits, form the break group A. Records that contain 981 as the first three digits form the break group B. Records in A1 are compared to records in A2, but never to records in B1 or B2. Records in B1 are compared to records in B2, but never to records in A1 or A2.
Real Experience. Real Advantage.

80920 A1 A

80909 A2

98146 B

98144

51

[ Break group records

Real Experience. Real Advantage.

52

[ Creating Trustworthy Data Matching and Consolidating

Unlocks the relationships between distinctly different sets of data by: Householding data Creating a panoramic single best record Providing identity resolution to uncover non-obvious relationships for fraud detection

Real Experience. Real Advantage.

53

[ Creating Trustworthy Data Matching and Consolidating

Input records
M. Bernard Martin 1, place des Saisons 92083 Paris-la-Dfense Cedex France

Consolidated record

B MARTIN AXA France Assurance Tour AXA 1, place des Saisons 92083 Courbevoie FRANCE M. MARTIN AXA 1, place des Saisons 92083 Paris-la-Dfense +33 (0)1 47 74 10 01 23/7/2003

Nom: M. Bernard MARTIN Societe: AXA France Assurance Date dachat: 23 Juillet 2003 Adresse: 1, place des Saisons Courbevoie 92083 Paris-la-Dfense Code Postal: 92083 Telephone: +33 (0)1 47 74 10 01 Adresse e-mail: bernard.martin@axa.com

Real Experience. Real Advantage.

54

[Creating Trustworthy Data Matching Records

Duplicate records often exist in one or more source systems. The goal of matching is to determine whether records refer to the same entity. This involves evaluating how well the individual fields, or attributes, of records match each other. SAP BusinessObjects Data Services employs powerful matching algorithms to account for data entry errors, character transposition, and other data errors to match records. These three records have been deemed matching records based off of the business rules you define in the matching process with SAP BusinessObjects Data Services

Real Experience. Real Advantage.

55

[Creating Trustworthy Data Populate missing values

Once matches have been identified, data from these match groups can be salvaged and posted to form a single best record or posted to update all matching records.

Master records or fields can be defined with the consolidation process to align with your business rules. For example, you could decide to post the most current phone number to all three records.
Real Experience. Real Advantage.
56

[Data Quality: Matching & Merging - Using Cleansed Data

to Identify Potential Duplicate Records
Input record variations
Mr Michael Schmidt Ph.D. HyperTech Inc * 330 East 63 Avenue Suite 10B New York NY 10021-7706 michael.schmidt@hypertech.com May 23, 2003; E3 Stamping Machine * Mike Schmidt Hypertech Corp 330 S 63rd St # 1000 Manhattan, NY 10012 michael.schmidt@hypertech.com 001-12-4367 30-6-2005; Fabrication Facility class C Mr. Mick Schmidt Hype_Tech 330 E 63rd Road * New York NY 10065 001-12-4367 (222) 922-9922 10/21/04; Victory Injection Molder
Real Experience. Real Advantage.

Complete/Consolidated master record

Name: Mr. Michael Schmidt Ph.D. Company name: HyperTech Inc SSN: 001-12-4367 Address: 330 E 63rd St Ste 1000 City, State, ZIP: New York, NY 10065-7706 Latitude: 40.722970 Longitude: -74.005035 Phone: (222) 922-9922 Email: michael.schmidt@hypertech.com Purchase history: 5/23/03; E3 Stamper, $1,300,000 10/21/04; A1 Injector, $520,000 6/30/05; C2 Fabricator, $23,000,000

57

[ Process Flow for Integrating Master Data - Leveraging

SAP BusinessObjects Data Services
Using SAP BusinessObjects ETL and Data Quality capabilities
5

MDM

MDM Server

Sequence of this process 1

Extract master data from remote systems using SAP BusinessObjects Data Services Cleanse data, match duplicates and deliver to MDM Import Manager

4
MDM Data Manager

Import

MDM Import Manager

2 3 4 5

SAP Business Objects Data Services

Map data using MDM Import Manager and send to MDM Server Persist data in MDM repository Access data using MDM Data Manager for post-processing (or SAP NetWeaver Portal)

Third party source systems

Real Experience. Real Advantage.
58

[ Integrating SAP BusinessObjects Data Services with

SAP NetWeaver MDM
Prior to loading data into SAP MDM: ERP ORCL Extract master data from client systems using predefined extractors Analyze the quality of the extracted data Cleanse and correct master data Identify duplicate records Find = Generate best record based on duplicate Duplicates records and survivorship rules Use the SAP MDM Import Manager to load Generate Best Record the best record and link the duplicate Best Best Record records based on the matching ID = + Record
ABC12 ABC12 Margret Margret SmithSmithKlein Klein ABC12 ABC12 Margret Margret SmithSmithKlein Klein ERP ERP ABC12 ABC12 Match Match ID 2 ID 2 Margaret Margaret Smith-Klein Smith-Klein

SEBL
678DE 678DE Peg Peg Klein Klein

12345 12345 Maggie Maggie Smith Smith

12345 12345 Maggie Maggie Smith Smith

678DE 678DE Peg Peg Klein Klein

ORCL ORCL 12345 12345 Match Match ID 2 ID 2

SEBL SEBL 678DE 678DE Match Match ID 2 ID 2

1
MDMID 789 MDMID 2 Match ID 789 Match MargaretID 2 Margaret Smith-Klein Smith-Klein

Import into MDM based on Match ID

MDMID 789 -> ERP ABC12 MDMID 789 -> ORCL 12345 MDMID 789 -> SEBL 678DE
59

Real Experience. Real Advantage.

[Added Value of Data Services with MDM

MASTER DATA CONSOLIDATION / INITIAL LOAD: Customers want an efficient, reusable mechanism to load pristine data into MDM

Data Services enables cleansing, matching & consolidation for initial and delta loads to MDM Data Services supports rule-based automerge and survivorship

CONTROLLED DATA GOVERNANCE AND DATA QUALITY REPORTING: Customers want to prevent duplicate entries from entering their systems, at the source

Data Services provides data cleansing and matching services for data maintenance on MDM Business rules are checked within Portal iViews before records are committed

PERIODIC CLEANSE: Customers recognize that a periodic comprehensive cleanse of their data may be necessary, to catch hidden incomplete entries and to update all with the most recent address information
Real Experience. Real Advantage.

Data Services supports periodic cleanse, match and consolidate of the MDM repository Export, process and re-import the MDM repository via the enrichment adapter

60

[Trustworthy data with SAP BusinessObjects Data Services

Conclusion: Data Services is the foundation for trustworthy master data. Value
4. Data GOVERNANCE 3. Data CONSOLIDATION 2. Data INTEGRATION & CLEANSING 1. Data READINESS

MDG MDM Data Integrator Data Quality

MDM Data Integrator Data Quality

Data Integrator Data Quality

Data Quality

People & Process Maturity

Paths to Master Data

Real Experience. Real Advantage.
61

[ Key Learnings
Data is spread throughout systems and applications in the enterprise Data quality issues are everywhere in the information supply chain SAP BusinessObjects Data Services should be used to create trustworthy master data Make sure you understand your data well by profiling your source data Standardize, cleanse, and enrich data before loading into target applications Avoid duplicate data - match and consolidate to get one golden record

Real Experience. Real Advantage.

62

Thank you for participating.

Please remember to complete and return your evaluation form following this session. For ongoing education on this area of focus, visit the Year-Round Community page at www.asug.com/yrc

]
63

SESSION CODE: 307

Real Experience. Real Advantage.