Learn

Collaborate

Succeed

Network

WebSphere Transformation Extender ( formerly Datastage TX ) Integration with WAS, WPS and WESB
Shahryar Sedghi Senior Consulting IT Specialist ssedghi@us.ibm.com

2006 IBM Corporation Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.

Objectives
Learn about WTX and How it solves Integration Problems Why we need to Integrate it with other WebSphere products Integration techniques Usage Patterns Best practices of Integration

WTX History
Formerly Mercator, 3-5% Market Share in the BI sector Unlike all the others, a major player on Z Apps (CICS, Batch) Acquired by Ascential to improve the ETL solutions Ascential was acquired by SWG DB2 Davison for Its ETL solutions WTX transferred to AIM to improve our WebSphere Integration story
3

Data Integration Terms

Parser
Parses (De-serialize) an Input Stream (aka. Byte Array) to a tree or a hierarchical Object Serialize the tree or hierarchical object to an output stream Apache Xerces is an XML parser

Mapper

Converts from one tree or object to another XSLT is a standard mapper for XML Adapter Deals with different Source and Targets File , Database, MQ, JMS,.

Broker

Runs a flow of various activities, involve Adapters, Parsing, Mapping and Decision Making Brokers implement ESB (Enterprise Services Bus)
4

WTX Components
Parser for various Data Formats Tag Delimited Data (Industry standard, aka. X12, HL7, HIPPA), Already has many off the shelf formats Fixed Width (COBOL) XML (Basic and SOAP) Legacy Parsers implement lazy parsing Mapper Adapter File, Database, SAP, PeopleSoft,. Broker Event Server Tooling Windows Based (Not Eclipse)
5

WTX Integration Patterns Standalone

WPS WESB WAS SIB

Input Stream

WTX
WebSphere MQ

Pros

All adapter capabilities Better Performance for large Messages Cons MQ get and Put overhead Managing two environments
6

WTX Integration Patterns Integrated

WPG,WPS,WESB, WBIMB, WAS SIB
Input Stream WTX Output Stream

Pros

Managing a single environment A simpler model Cons No use of adapters

7

WTX Integrated solutions - What we need to Learn

Design Studio Type Designer

Creating Types with Importers Creating Type Manually

Map Designer
Create Maps between Input and Output Types

Testing, Debugging Optionally, Development Toolkit Java, RMI and JCA Adapter API

A WTX Mapping Scenario A Sample BO

Type Designer XSD Input

<?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:bons0="http://JMSLibrary" targetNamespace="http://JMSLibrary"> <xsd:include schemaLocation="School.xsd"/> <xsd:complexType name="Student"> <xsd:sequence> <xsd:element minOccurs="0" name="firstName" type="xsd:string"/> <xsd:element minOccurs="0" name="lastName" type="xsd:string"/> <xsd:element minOccurs="0" name="address1" type="xsd:string"/> <xsd:element minOccurs="0" name="address2" type="xsd:string"/> <xsd:element minOccurs="0" name="schools" type="bons0:School" maxOccurs="unbounded"/> </xsd:sequence> </xsd:complexType> <xsd:element name="Student" type="bons0:Student"></xsd:element> Does not exist in WPS/WESB BOs </xsd:schema> <?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://JMSLibrary"> <xsd:complexType name="School"> <xsd:sequence> <xsd:element minOccurs="0" name="name" type="xsd:string"/> <xsd:element minOccurs="0" name="address" type="xsd:string"/> </xsd:sequence> </xsd:complexType> </xsd:schema>

10

Type Designer Type Tree from XML

11

Type Designer COBOL Input

01 Student. 02 name PIC X(20). 02 lastName PIC X(20). 02 address1 PIC X(20). 02 address2 PIC X(20). 02 numOfSchools PIC 99. 02 schools OCCURS 0 TO 99 TIMES DEPENDING ON numOfSchools. 03 schoolName PIC X(20). 03 schoolAddress PIC X(20). 02 FILLER PIC XX.
12

Type Designer COBOL Type Tree

13

Map Designer

Submap
14

WTX Map Artifacts

Files Type Tree .mtt Map Source File .mms Compiled map .mmc Map Source file requires Type Trees to build compiled maps Compiled maps do not require any artifact, unless a card uses a Xerces Parser (V8.0 or above)

15

WPS/WESB + WTX ?
WPS/WESB natively support XML COBOL Copybook and C Header files( CICS and IMS adapter tooling) WTX is required for: Industry Standard Formats (aka. Tag Delimited Data) Content based routing for Non-XML data

16

Inbound Data De-Serialization (Parsing)

Edge SCA Export Bytes Stream

Data Binding

SDO

Middle Java Component Bytes Stream Wrapped in SDO

POJO

SDO

17

Oubound Data Serialization

Edge
SCA Import Byte Stream

SDO

Data Binding

Center
Java Component Bytes Stream Wrapped in SDO

SDO

POJO

18

WTX, WPS and WESB ISSW Offerings

WTXJMSDataBinding Can be used in SCA JMS Import/Export binding in WPS/WESB Fully parameterized and configurable through WID WTXMedaition Complements the WPS/WESB solution Normally is not required if best practices followed WTXMaprunner A Wrapper for WTX Java API
Currently supports one Input and One output card which is sufficient for WPS/WESB Designed for WebSphere Environment. Loads Maps from Java classpath Is not a perfect solution yet (No Caching,..)
19

ISSW Offering - JMS Enablement

JMS Export
JMSData Binding For WTX

WPS Process Or WESB Mediation

JMS Import
JMSData Binding For WTX

Outbound

Inbound

WTX Medaition

SIBus

WTX Medaition

Input Queue
Incoming Data

Output Queue
Outgoing Data

Either JMS Data Binding for WTX or WTX Mediation Can be used not both together, on the same destination
WTX Mediation helps using WTX without modifying Existing Processes

and Mediations Non-JMS WMQ Applications can take advantage of the mediation
20

WTXJMSDataBinding - Outbound
WTXOutMap JMS Property

WPS Process
JMS
Export

Or WESB Mediation

JMS
Import

WTXOutMap JMS Property Legacy

Legacy

SDO

EMF XML Parser

XML Data

Map

Byte Stream

21

WTXJMSDataBinding - Inbound
WPS Process
WTXInMap JMS Property Legacy

JMS
Export

Or WESB Mediation

JMS
Import WTXInMap

JMS Property
Legacy

Byte Stream

Map

XML Data

EMF XML Parser

SDO

22

WTX SIBus Mediation

JMS Text or Bytes Message

Bus

WTXMedaition

Map Name Message Format And Other Context Properties

JMS Text or Bytes Message

Destination

Inspect Message

Byte Stream

Map

Byte Stream

Generate Message

23

WTXMedaition Code
SIMessageContext siCtx = (SIMessageContext)ctx; SIMessage message = siCtx.getSIMessage(); String messageFormat = (String)ctx.getProperty("Message Format"); String mapName = (String)ctx.getProperty("Map Name"); byte[] inBytes = message.getDataGraphAsBytes(); byte[] outBytes = new WTXMapRunner().runMap(mapName, inBytes); DataGraph newDataGraph = SIDataGraphFactory.getInstance().createDataGraph(outBytes, messageFormat); message.setDataGraph(newDataGraph, messageFormat); for(Iterator it = ctx.getPropertyNames();it.hasNext();) { String propName = (String) it.next(); message.setUserProperty(propName, (Serializable)ctx.getProperty(propName) ); // Optional }
24

WTX Non-JMS Inbound, Edge

Flat File Custom Data Binding Adapter Export

WPS POJO WESB Custom Mediation

SCA
Import

File

Byte Stream

Map

XML Data

EMF XML Parser

SDO

25

WTX Non-JMS Inbound, Middle

Flat File Adapter Export

WPS POJO WESB Custom Mediation

SCA
Import

File

Bytes Stream Wrapped in SDO

Extract Bytes

Bytes Stream

Map

XML Data

EMF XML Parser

SDO

26

WTX Non-JMS Outbound, Edge

WPS Process
SCA
Export

Or WESB Mediation

Flat File Adapter Import

Custom Data Binding

File

SDO

EMF XML Parser

XML Data

Map

Byte Stream

27

WTX Non-JMS Outbound, Middle

SCA
Export

WPS POJO WESB Custom Mediation

Flat File Adapter Import

File

SDO

EMF XML Parser

Bytes Stream XML Data Map Byte Stream Wrapped in SDO

28

Usage Patterns
ESB

Legacy to Web Service Data Integration Content Based Routing Process Integration Legacy to Business Processes

29

Legacy to Web Service Lab 1

JMS Client
1 7

Mediation Module
JMS Export 6
In

Mediation
3
XSLT Out

5
Out XSLT In

WS Import

Currency Exchange Service

Internet
30

Data Integration Lab 2

JMS Client is for demonstration only

Typically Message Producers and Message Consumers are different

JMS Client
1 5

WPS Module
4 2 6 JMS Export
JMSInInterface

Interface Mediation
3
Data Map

JMSOutInterface

JMS Import

31

Content Based Routing

Component
Generic Interface

WTXJMSDatabinding

JMS Export

Router Component
Component

Can be a POJO or a Custom Selector

Component

Event Agnostic

Event Aware

32

Generic Interface

33

Legacy to Process Lab 3 (simulated)

POJO WTXMapRunner

Flat File Adapter Export

BPEL
HTM

File JMS Import

34

Deployment Prerequisites
Environment WPS/WESB interact with WTX Java API Java API invokes Shared Libraries through JNI layer WTX Jar (dstxpi.jar) must be in the classpath Shared Libraries (aka. DLLs) must be in the system PATH (Path for WAS user profile in UNIX/Linux) Complied Maps must be accessible to the code Compiled map needs XSDs for validation if: XML is inbound to the map Xerces parser is used (8.0 and above)

35

Deployment Typical practice

Add dstxpi.jar to the system classpath Add WTX Install root to system PATH Move Compiled Map and all XSDs to a specific directory In a WAS cluster repeat step 1-3 for every Server Anytime a map changes replace the map in every server

36

Deployment Best Practices

Use WAS Shared Library to add dstxpi.jar to the classpath Add WTX Install root to system PATH WAS shared library support for native libraries does not work for WTX ( DLLs calling other DLLs) Load the map from classpath, rather than a folder Allows deploying maps with process EAR file Follow XML Best practices
No need to copy maps to each serve

37

WTX and XML

WTX validates every incoming data including XML WTX V8.0 supports Xerces Parser for XML Xerces requires XSD at runtime for validation Xerces is faster for large XML documents Classic XML parser does not require XSDs and Type Trees for inbound XML Mapping outbound XML is very difficult with classic parser
38

XML Best practices

Use Classic parser for inbound XML Eliminates the need for copying XSDs to each server Use Xerces parser for outgoing XML Outgoing XML does not need XSDs for validation Use Xerces if Classic parser is very slow for large documents Copy XSDs to the map working directory
WTXMapRunner needs a System Property XSDs change less frequently than maps

39

Links
Lab Docs (3MB):

ftp://pokgsa.ibm.com/pokgsah1/01/ssedghi/web/public/WTX/LabDocuments.zip
Solutions Folder without Workspaces (228 KB):

ftp://pokgsa.ibm.com/pokgsah1/01/ssedghi/web/public/WTX/Solutions_without_workspace.zip
Solutions Folder with workspaces (30 MB)

ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/Solutions.zip
VMWare (5.2 GB) ftp://pokgsa.ibm.com/pokgsa-h1/01/ssedghi/web/public/WTX/base_vm.zip
40

Questions

41