Binder 1

IBMs Web 2.
0 Strategy:
Products And Themes
Raymond Josef Edward A. Lara
Regional Technical Team, ASEAN SW Lab Services
2008 IBM Corporation
Page 1
WebSphere Technical Conference 2008
Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore
What Is Web 2.0
Term coined by Tim OReilly for the next generation
of web applications and software in general
Shows a paradigm shift where applications are not
dependent on technology but by the social
factorthe American Idol phenomenon
Web applications, since their inception, has been one
big proof-of-concept showing that the technology
works, Web 2.0 is about what the technology is used
for: to serve content and incite participation
Page 2
Concepts Behind Web 2.0
Embrace the long tail
Current web applications just expose the head
Web 2.0 apps reaches back as far as needed to
get content
Content focus
Web 2.0 hinges on content, lots of it
Critical mass needs to be achieved to bring out
the worthy 20% but the other still 80% needs to
be there
Power to the users
Web 2.0 is what the Internet is all about: choice
Users must be given freedom to determine what
clicks and this will lead others to follow
How many times have you heard: Did you see
that new video in YouTube?
Webmasters demoted to administrators
Webmasters are delegated to just managing the
infrastructure and for technical support
Content control and even presentation is entirely
up to the users
The inmates have taken over the
asylum
The more freedom users have the more they
can enrich the application
Some rights may have to be given up to the user
Constant refinement
There is no such thing as a final version in
Web 2.0 web apps
As long as users keep coming back the web
application is in perpetual beta
Cooperate, dont control
Work with the users, build a framework for them
to operate with but dont restrict
See On The Lot and Google Gadgets
Multi-channel
The browser is not the only platform to present
Web 2.0
Mobile Phones, PDAs, Web Kiosk Terminals,
and even desktop applications
Page 3
IBMs Web 2.0 Strategy
Technology
AJAX
REST
JSON
RSS
ATOM
Techniques/Services
Mashups
Composite Applications
Tagging
Folksonomy
Blogging
Wikis
Page 4
AJAX
Stands for Asynchronous JavaScript and XML
AJAX is not a new technology or programming language, but a new
way to use existing standards and mechanisms
With AJAX you can create better, more dynamic, and more user-
friendly web applications that incorporates features that were
previously only seen in desktop applications (e.g. drag and drop)
Dojo Toolkit Widely supported AJAX framework
Page 5
Traditional Web Applications
Client
Application Server
User interaction with page
triggers HTTP request
Request processed and HTML page generated
Time
Client refreshes whole page
based on new HTML
HTTP request
HTTP response
User interaction
Page 6
AJAX Web Applications
Client
Application Server
User interaction with page
triggers AJAX action
Request processed and XML data or
HTML page fragment generated
Time
Client refreshes part of a page
based on new XML data or HTML
JavaScript
JavaScript invoked to
handle request
HTTP request
HTTP response
User interaction
Page 7
REST
Stands for Representational State Transfer
Conceptualized by Roy Fielding in his doctoral dissertation which
became the architectural model for HTTP (and therefore the World-
Wide Web)
Characteristics:
Client-Server Uses a request-response type of interaction
Stateless The request must hold all information necessary to accomplish the
interaction and should not rely on any server context
Uniform Interface The request makes use of set methods to determine the
type of access (GET, PUT, POST, DELETE)
Named Resources The request must name the resource to be accessed from
the server
Works well with AJAX
Page 8
More On REST
REST is not a standard
W3C will not put out a specification for REST
Vendors will not sell a toolkit for REST
REST is a design pattern
It is not packaged, it is implemented
REST uses standards
HTTP
URI/URL
XML, HTML, GIF, JPEG (resource representations)
text/xml, text/html, image/gif, image/jpeg (MIME types)
Page 9
REST In Action
This is an example of how REST can be used in an application such
as a customer relationship management system
Atom GET ../crm/customer/cust1225/item Retrieve a customers purchases
PUT
GET
POST
GET
GET
Method
../crm/customer/cust1225/item
../crm/customer/cust1225/item/it301
../crm/customer/cust1225
../crm/customer/cust1225
../crm/customer
URI
Enter a customers new purchase
Retrieve a customers purchase
Update a customers information
Retrieve a customer
Retrieve all customers
Action
JSON
XML
JSON
XML
Atom
Representation
Page 10
REST vs. RPC
The URI is the method The URI calls the method
Designed for HTTP Protocol neutral
Unique URI per resource All resources use the same URI
Document oriented Data oriented
REST RPC
Page 11
What About Web Services?
Web services is an excellent technology
For systems integration and business processes
Generally, for computer-to-computer solutions
Web services is not appropriate for all situations
Not easily consumable (cant code by hand)
Generally, it is not good for user-to-computer solutions
We only need this
so whats all this!!!
Page 12
JSON
Stands for JavaScript Object Notation
A specification for the simple and lightweight representation of data
JSON is text-based and easy to parse (compared to XML)
JSON is not tied to a particular programming language
Works well with REST services
{
"accountNo": "12250301",
"name": "Jack B. Nimble",
"acctType": "CA",
"balance": 1000
}
Page 13
RSS
Stands for Rich Site Summary (or Really Simple Syndication)
A protocol used for subscription and delivery of content, called
syndication
Returns an XML document summarizing the updated content
Page 14
Atom
A syndication protocol similar to RSS but with a richer and more
strictly defined message format (works well with REST)
Request
GET /reilly/ HTTP/1.1
Host: example.org
Content-Type: application/atom+xml
Response
HTTP/1.1 200 Ok
Content-Type: application/atom+xml
Content-Length: nnnn
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom ">
<title>My Submitted Expenses</title>
<link href="http://example.org/reilly/"/>
<entry>
<title>Lotusphere 2006</title>
<link rel=alternate href="http://example.org/reilly/3.html"/>
<link rel=edit href=http://example.org/reilly/3.html/>
<updated>2006-12-05T12:29:29</updated>
<content type="XHTML" xml:lang="en-us">
<p xmlns="...">I am a speaker and am staffing a stand in the innovation lab</p>
</content>
</entry>
</feed>
Page 15
Mashups And Composite Applications
A mashup is
A collection of two or more components bound
together through content
An application derived by seamlessly combining
content from more than one data source
A composite application is
A rapidly created application to address an
immediate need
A typically (but not necessarily) short-lived
solution
Informal, good enough (lacks product quality)
Page 16
Tagging
Tagging is the collaborative effort to organize information in which
users are encouraged to assign freely chosen keywords to identify
web sites (i.e. bookmarks) in ways that are meaningful to
themselves
Tagging results in a new way of classifying data known as a
folksonomy
Gives rise to tag clouds
Its not what you know
its not who you know
its who knows what you need to know!
Page 17
Blogs And Wikis
Page 18
What Does Web 2.0 Mean For My Company?
Syndication can be used to disseminate information quickly both
internally (e.g. meeting recordings/podcast) and externally (e.g.
product updates)
Companies can allow mashups and composite applications for
specialized functions (e.g. CRM, proposal generation) to increase
user productivity
Search engines can offer searching based on folksonomies as an
alternative to standard taxonomy-based searches
Companies can derive ideas for new products by deploying a blog
for its customer community
Product documentation can be refined and made more complete
through wikis, errors in documentation can be corrected quickly
Page 19
What Happened To SOA?
WEB 2.0
Componentized
Interoperable
Modular
Scaleable
Page 20
Where Is The Web 2.0 In WebSphere?
WebSphere sMash
Based on the Project Zero public incubator
REST services platform
Supports JSON and ATOM
Supports PHP and Groovy scripting
WAS 6.1 Feature Pack for Web 2.0
AJAX proxy with support for REST connection
JSON messaging capability (JSON-RPC)
ATOM support
Dojo toolkit + IBM extensions
Thank you
larara@ph.ibm.com
Speaker Name YOGESH BHATIA
Title PORTAL AND WEMP CLUSTERING CONCEPTS
Page 1
AGENDA
INTRODUCTION TO WAS
CLUSTERING CONCEPTS
WAS CLUSTERING STEPS
INTROUDUCTION TO PORTAL
WHY WAS STEPS NOT APPLICABLE IN PORTAL
CONCERNS IN PORTAL CLUSTERING
STEPS FOR PORTAL CLUSTERING
INTRODUCTION TO WEMP
WHY PORTAL STEPS NOT APPLICABLE IN WEMP
Page 2
WAS CORE CONCEPTS
An application server is a Java Virtual Machine (JVM) that is running
user applications. The application server collaborates with the Web
server to return a dynamic, customized response to a client request.
Application code, including servlets, JavaServer Pages (JSP) files,
enterprise beans and their supporting classes, runs in an application
server. Conforming to the Java 2 platform, Enterprise Edition (J2EE)
component architecture, servlets and JSP files run in a Web
container, and enterprise beans run in an Enterprise JavaBeans
(EJB) container.
Page 3
Cluster Concept
Clusters are groups of servers that are managed together and
participate in workload management. A cluster can contain nodes or
individual application servers. A node is usually a physical computer
system with a distinct host IP address that is running one or more
application servers. Clusters can be grouped under the configuration
of a cell, which logically associates many servers and clusters with
different configurations and applications with one another depending
on the discretion of the administrator and what makes sense in their
organizational environments. Clusters are responsible for balancing
workload among servers. Servers that are a part of a cluster are
called cluster members. When you install an application on a cluster,
the application is automatically installed on each cluster member.
Page 4
Vertical Scaling and Horizontal Scaling
Page 5
Page 6
FROM STAND ALONE TO FEDERATED
Page 7
REPOSITORY STRUCTURE IN CELL
Page 8
PORTAL SERVER OVERVIEW
Product of WebSphere family for displaying portlets ,web site and
personalization
From WAS point of view , application sitting on top of WAS
Upon Installation , its give option to use existing WAS set up or
install new WAS
Its huge application with lots of libraries and create its own folder
structure and config repository
Page 9
FUNDAMENTAL CHARACTERISTICS
Information aggregation
Targeted and personalized information
Accessibility
Single sign-on
Page 10
Typical Portal Set up
Page 11
PORTAL AND WAS CLUSTERING
TRADEOFF
In WAS no importance of default applications
Portal has set of applications required for its functionality
ANY SOLUTIONS ?
Page 12
WAS CLUSTERING
DEPLOYMENT
MANAGER
NODE 1
SERVER 1
NODE 2
SERVER 1
addNode
addNode
Page 13
CLUSTERED WEBSPHERE PORTAL
Page 14
PORTAL CLUSTERING
DEPLOYMENT
MANAGER
NODE 1
SERVER 1
NODE 2
SERVER 1
addNode -
includeApps
addNode
PRIMARY NODE
SECONDARY NODE
Page 15
STEPS FOR PORTAL CLUSTER SET UP
Install Portal on both nodes
Install Deployment Manager
Migrate Cloudescape to Db2 of primary node
Enable LDAP security on primary node
Add Primary node with includeapps
Add Secondary node
Create Cluster
Page 16
INTRODUCTION TO WEMP
Page 17
It is an extension to WebSphere Portal
Enables device specific markup capability
Extends Portal navigation to mobile devices
Adds mobile administration portlets
Provides a framework that can be used to create device independent portlet applications
(XDIME Portlets)
XDIME Aggregator displays navigation nodes which are configured and stored in Portal.
XDIME Portlets render applications that generate XDIME.
Mutli Channel Server (MCS) Integration converts XDIME to device-specific markup.
Easily supports new devices with mobile device update service
It takes advantage of many Portal Features
Configuration (xml access)
Navigation (Aggregator/Model API)
Security (e.g. TAI/ACL)
Scalability (cluster)
Virtual Portal
It includes an extensive tooling suite built on IBMs Eclipse-based Rational Software
Development Platform
What is WebSphere Everyplace Mobile Portal?
Page 18
With unlimited applications and thousands of devices, the challenge
becomes how to solve an expanding "M x N" matrix
N devices
M applications...
Page 19
The solution is to define a Mobile Platform that dynamically
and intelligently adapts to the specific device
N devices
M applications...
IBM WebSphere Everyplace Mobile Portal software
Page 20
WebSphere Application Server
WebSphere Everyplace Mobile Portal Overview
WebSphere Portal
MCS
runtime
MCS
Repository
Layout
Theme
Components
Device
XDIME
Portlets
XDIME
Aggregator
Devices
Page 21
Runtime Component Interaction
XDIME
Aggregator
Portlet Filters (IBM
Portlets only)
XHTML
MP
Portlet Container
XDIME
(XDIME)
XHTML
Basic
WML
PortletRequest
PortletRequest
PortletResponse
PortletResponse
Manage Mobile
Pages
Portal
Navigation and
Content Model
ServletRequest
ServletResponse
Multi -
Channel
Server
MCS Policy
Repository
XDIME
Device -
specific
6
XHTML
Basic
XHTML MP
WML
WebSphere Portal (WP)
7
5
4
3
2
1
MCS
Servlet
Filter
MCS components
WEMP extensions to WP Legend
XDIME
XDIME
Page 22
REQUIREMENT FOR WEMP INSTALLATION
MCS to be installed after Portal
Problem is in secondary node as well , MCS needs to be installed
and it requires database
ANY SOLUTIONS ?
Page 23
SOLUTION
INSTALL BOTH PORTAL NODES
MIGRATE PRIMARY NODE
CONNECT SECONDARY NODE WITH SAFE MODE
USE SAME DATABASE FOR SECONDARY NODE
Page 24
VARIOUS STRATEGIES FOR WEMP
CLUSTERING
Install WEMP on federated Portal Node
Install WEMP on node not managed by DMGR ( discussed )
Install WEMP to a Portal Cluster
Page 25
INSTALL WEMP ON FEDERATEDE NODE
Install WebSphere Portal on WP01
Create the Installing and configuring a WebSphere Portal cluster .
Verify that WebSphere Portal works with the Web server
configuration.
Install the mcs Multi-Channel Server for DB2 or Install Multi-Channel
Server for Oracle.
Install the Everyplace Mobile Portal extensions.
Replace the ./WPSconfig.sh init mcs-deploy-mcs command with:
./WPSconfig.sh init mcs-config-cluster-secondary-node
-DMcsRepositoryType=odbc
-DMcsDbVendor=<db_vendor>
-DMcsDbHost=<hostname>
-DMcsDbPort=<port_number>
-DMcsDbSource=<db_name>
-DMcsDbUser=<db_instance>
-DMcsDbPassword=<db_instance_password>
-DMcsProject=mobile-portalUse the same Multi-Channel Server
database information used by the WP01 instance.
Page 26
INSTALL WEMP NOT MANAGED BY DMGR
Install WebSphere Portal on this instance following steps in Installing WP02 in
Installing WebSphere Portal on a separate WebSphere Application Server
node not managed by the deployment manager.
Follow the remaining required steps and actions in the topic Before you install
Everyplace Mobile Portal.
Install WebSphere Portal on this instance and then follow the remaining
required steps and actions in the topic Before you install Everyplace Mobile
Portal.
Before reconfiguring the database from Cloudscape to the database used by
the WP01 instance:
Follow the procedures in the topic Installing Multi-Channel Server for DB2 or
Installing Multi-Channel Server for Oracle and use the following information:
Page 27
.continued
Use the same Multi-Channel Server database information used by the WP01
instance. Ensure that the following parameters have identical values for both
WP01 and WP02:
DMcsRepositoryType,
DMcsDbVendor,
DMcsDbHost,
DMcsDbPort,
DMcsDbSource,
DMcsDbUser,
DMcsDbPassword Do not run the following command since the default device
policies were imported when you installed Multi-Channel Server on WP01:
./WPSconfig.sh init mcs-import-devices -DMcsProject=mobile-portal
When installing the Everyplace Mobile Portal extensions, do not run the
following commands since the Everyplace Mobile Portal extensions policies
were imported when you installed the Everyplace Mobile Portal extensions
on WP01:
./WPSconfig.sh mwp-import-policies
./WPSconfig.sh mwp-import-portlet-policies
Page 28
INSTALLING WEMP TO PORTAL CLUSTER
Perform the recommended steps and actions in the topic Before you
install Everyplace Mobile Portal.
Verify that WebSphere Portal works properly.
Follow the procedures in the topic Installing Multi-Channel Server for
DB2 or Installing Multi-Channel Server for Oracle.
Follow the procedures in the topic Installing Everyplace Mobile
Portal extensions and replace the ./WPSconfig.sh mwp-config or
./WPSconfig.sh mwp-config-all command with ./WPSconfig.sh mwp-
config-cluster-primary-node
Page 29
.CONTINUED
Following steps should be performed after installing WP02:
Load the Administrative Console from DM01 and perform the following
steps:
Run Full Resynchronize on both nodes.
Restart the cluster.
Configure the Everyplace Mobile Portal extensions for cluster using one of
the following options:
Option 1 - To install and configure all of the Everyplace Mobile Portal
extensions, run the following command:
./WPSconfig.sh mwp-config-cluster-allOption 2 - To install and configure all
of the Mobile Portal extensions, except for the sample XDIME portlets and
the sample XDIME pages, run the following command:
./WPSconfig.sh mwp-config-clusterTo activate the portlets, run the following
command:
./WPSconfig.sh activate-portlets
Page 30
THANK YOU
QUESTIONS ???
IBM Software Group
WSTC Workshop
Collaborative Development Using Rational Team
Concert
WSTC 2008
Lab Overview
2
Agenda
Introduction to Rational Team Concert
Lab Overview
Module 1 Setting up the Team
Module 2 Planning Your Work
Module 3 Keeping Track of All Our Work
Module 4 Performing and Sharing Your Work
Module 5 Remembering Well Known SCM Configurations
Module 6 Users View of Build
Module 7 Endgame and a Tightened Process
Session Summary
WSTC 2008
Objectives
Describe development and delivery challenges with distributed teams
Explore how Team Concert can
Enable development teams to collaborate in real time in the context of the work they are
doing, especially in globally diverse environments
Enable projects to be managed more effectively by providing visibility into accurate project
health information drawn directly from actual work
Automate traceability and auditability by managing artifacts and their inter-relationships
across the lifecycle empowering teams to deliver more value
Provide customizable process design and enactment through rule-based process
guidance, automation and definable checkpoints
Provide a hands on experience using Team Concert to automate the software
delivery process
Collaborative Development Using Rational Team Concert 3
IBM Software Group
WSTC Workshop
WSTC 2008
2008 IBM Corporation Collaborative Development Using Rational Team Concert 5
Global development and delivery challenges
50% of outsourced projects are expected to underperform
Mismatched & Misunderstood
Process
Communication Issues
Cultural Issues
Decreased Productivity
Increased Rework
Mistakes in Work-Transfer
Higher Co-ordination Costs
Political Issues
Lack of Security IP Protection
Lack of Project Visibility, Agility and
Control
Lack of Project Metrics Unable to
Measure Success
Geographically
Distributed
Development
Creates
Gartner survey of 219 clients who outsourced projects offshore & domestically
half are expecting to fail to deliver anticipated savings.
Acquisitions &
Mergers
Application
Backlogs
Budget cuts/
Cost Reductions
Variable
Staffing Model
Increased
flexibility to adapt
quickly
Business Drivers Customer Pain Points
WSTC 2008
Trends toward SOA & geographic distribution
Impact of growing modularity
More granular service functionality in
composite business applications
Large number of projects & assets
including custom, outsourced & packaged
Impact of accelerating change
Asset will be impacted by frequent updates
and changing interdependencies
Effective cross-organizational visibility and
synchronization becomes an imperative!
Drives the need for enabling better performance of organizationally
diverse people & assets across the software supply chain
Business management
Business-IT alignment
Development
Software delivery
Operations
Service management
SOA enabled
Business Flexibility
WSTC 2008
Customers speak: Whats needed to break through
Whats needed is a collaborative application lifecycle management
approach that gets business and technology to work as a team.
Something that doesnt get in
the way and doesnt add
overhead.
Something that gives both the top down
view and the low level view and links back to
configuration, bug tracking, and build trending
Development
Lead
Something that helps me plan iterations and balance work loads
across my distributed teams and onboard team members and
projects in a day instead of a month.
Development
Manager
Something that automates my development process with short iteration
cycles so were able to quickly fulfill business requests.
Program
Manager
A more transparent view into my development teams
allowing me to make better decision and to deliver on time
to my executive team
CIO
Developer
WSTC 2008
Whats different today?
2008 and beyond: Shifts in software delivery
These shifts are setting the agenda for a
new approach to software delivery...
What were seeing
2/3 of our projects span multiple
business units
Our architecture has to be simpler...
we cant afford this cost model anymore
Weve done two acquisitions in six months
and we cant lose customers
Our last ERP upgrade took six months
that isnt going to fly next time
We have to go where the best talent is,
but we have IP and compliance realities
What were hearing
Increasingly diverse and cross
organizational global project teams
Interest in replicating the proven
models of open, community-
governed software delivery models
Popularity of Agile development
and social networking models
SOA, Web-centric and Web 2.0
enabling new business models
WSTC 2008
Enable team transparency
of who, what, when, why
Build team cohesion and presence
Automate hand-offs so nothing
falls through the cracks
Automate team workflow improving
productivity
Automate data collection eliminating
administrative overhead
Real time reporting and alerts
reduces project risk
Dynamic provisioning
of projects and teams
Real-time iteration planning
and workload balancing
Unify teams with tools choice
Real-time integration of people, process and projects across the lifecycle
Collaborate
in Context
Right-size
Governance
Day One
Productivity
Robust, extensible and scaleable
Globally distributed, fluid and dynamic
Community-based and open at Jazz.net
A open technology initiative
to transform how people work
together to deliver greater
value and performance from
their software investments
Envisioning a platform that can transform software delivery
WSTC 2008
How Jazz is different
Dynamic integration of
People
Desktop/database integration Global integration
Function first Team first
Pre-determined tool function Dynamically extensible
Rigid rules Creative collaboration
Manual tasks Process aware and transparent
Proprietary infrastructure Internet standards
Lengthy project onboarding Day one productivity
Hindsight analysis Continuous steering
Status guesstimates Real-time insight
...across the software lifecycle
Process
Projects
WSTC 2008
Open Lifecycle Service Integrations
Jazz platform architecture
JAZZ TEAM SERVER
Search
and Query
In Context
Collaboration
Team Awareness
Dashboards Events
Notification
Security
Server Integrations
Eclipse
Web 2.0
Visual Studio (4Q)
Others to come
Client Integrations
Rational
Team
Concert
Rational
Quality
Manager
Rational
Requirements
Composer
Enterprise
Reporting
Project
Management
Others
Business
Partner & IBM
Offerings
Best Practice Processes
Rational ClearCase 7.1
Rational ClearQuest 7.1
Rational Build Forge 7.1
Rational Asset Manager 7.1
Subversion
WSTC 2008
Software innovation through collaboration
Real time, in-context team collaboration
Make software development more automated,
transparent and predictive
"Think and work in unison"
Integrated source control, work item and build
management
Assess real-time project health
Capture data automatically and unobstrusively
Automate best practices
Dynamic processes accelerate team workflow
Out-of-the-box or custom processes
Unify software teams
Integrate a broad array of tools and clients
Extend the value of ClearQuest and ClearCase
Support for System z and System i servers (Beta 3Q)
IBM Rational Team Concert
transparent integrated presence
wikis OPEN real-time reporting
chat automated hand-offs Web 2.0
custom dashboards automated data gathering
EXTENSIBILITY Eclipse plug-ins services
architecture FREEDOM TO CREATE
Open and extensible on
Collaborate in context
Right-size governance
Day one productivity
WSTC 2008
Delivering customer value with Rational Team Concert
The evolution of software delivery
Supporting the software economics of an increasingly global,
dynamic and web-based technology environment
Project initiation in days rather than weeks
Reduction in time to first demonstrable release
Improved cycle times through automated and reusable
patterns, services and architectures
Less overhead to stay compliant with standards
Earlier detection and accelerated resolution of defects
Improved predictability of project outcomes
Fewer meetings and status reports
Reduction in late scrap and rework
Reduced attrition of skills and greater flexibility to
leverage top technical and business talent
Day one productivity
Right-size governance
Collaborate in context
WSTC 2008
Rational Team Concert: A Deeper Look
Team advisor for defining / refining rules
and enabling continuous improvement
Process enactment and enforcement
In-context collaboration shows team members
and status of their work
Customizable project dashboard
Real-time reporting and queries
Integrated iteration planning and execution
Real-time project status window
Single structure for project related artifacts
World-class team on-boarding / offboarding
including team membership, sub-teams and
project inheritance
Role-based operational control for flexible
definition of process and capabilities
Jazz Team Server
Flow browser shows developer
to team flow
Integrated stream management
Component level baselines
Server-based sandboxes
Identifies component in streams
and available baselines
ClearCase connector
SCM Work Items
Defects, enhancements
and conversations
Query results view and share
queries with team or member
Support for approvals and
discussions
ClearQuest connector
Query editor interface
Work item and change
set traceability
Local or remote build servers
Supports ant and command
line tools
Integration with build forge
Build definitions for team
and private builds
Build
WSTC 2008
Unify distributed teams in mid-sized or enterprise settings
A complete, standalone solution for
departmental and medium-sized distributed teams
Supports departmental satellite or agile projects optionally integrated with existing
enterprise-proven Rational ALM infrastructure offerings
Enhances existing ALM offerings with new collaborative & process capabilities
Bi-directional
auto-synchronization
Project Management
Europe
Development
China
Config & Build China
Team Concert
Development
Australia
Eclipse
TM
, Linux
, Microsoft
Windows
,
UNIX
, IBM z/OS
, IBM i5/OS
Rational ClearQuest
Rational Enterprise-proven ALM Infrastructure
Rational Build Forge
Rational
ReqPro
Rational ClearCase
Define Build Test Deploy
New York City
North Carolina
Testing
Ireland
WSTC 2008
Delivering global team collaboration and visibility
Allows individuals to focus on what they do best
Deployment Manager
Developer
Business Stakeholder
Architect
Build Engineer
Know exactly what to fix when
something breaks
Create great software,
unencumbered by tedious
status reports
Build design reviews into your
software lifecycle-no more chasing
after rogue development streams.
Provision a new team member
across the world in a day
Track progress on the updates that
your customers care about.
Make confident go/no
decisions based on facts,
not estimates.
Project Manager
WSTC 2008
We eat our own dog-food!
Collaborative Development Using Rational Team Concert
2-way Xeon Server running application
server (WAS) and another running DB2
Jazz Project Using bi-weekly iteration
builds
Jazz and Rational Team Concert
self hosting since 4Q06
Global team in 7 locations in
NA and Europe
~100 developers, plus jazz.net web
client access
Repo > 10G, 66K files, 43K work items
Over 10 internal Rational Development
Team leveraging RTC
WSTC 2008
Customer Feedback
Rational Team Concert not only brings the communications channel
inside the development platform, but it integrates everything. The
integration with the complete workflow for the life cycle ... is a big
improvement.
-- Major health care insurance provider
"Features provided by Rational Team Concert are equal or more
sophisticated than current used tool development environment at RB
(JIRA, CVS and Release Engine).
-- Global supplier of industrial, consumer and building technology
"I found that it was straightforward to install, which made a nice change.
Starting up the server and getting the clients talking to it was simple as
well.
-- Leading UK financial services organization
WSTC 2008
WSTC 2008
Lab Overview
20
Agenda
Lab Overview
Session Summary
WSTC 2008
Scenario for PoT Labs
You are joining an exciting new project called Squawk that has recently been
started in your company.
You will be using Rational Team Concert as the projects collaborative
development environment.
The Squawk project is run according to the Eclipse Way, a development process
based on the wildly successful Eclipse project (www.eclipse.org).
You have joined the project at the start of Milestone 2 and have a chance to make
a real difference to this project.
Lab Overview
21
WSTC 2008
Scenario for PoT Labs
Squawk is a (simple) program that will print out different sounds
depending on who squawks. The Dog squawker goes bark,
the Cat squawker goes meow, etc. Your main task is to create
a new squawker, along with tests and documentation.
At the same time as creating new squawkers, you will get to participate in some
planning activities, interact with your fellow team members, deliver your work to the
project, trigger automated builds and various tasks typical for project teams
everywhere.
The project team structure mimics the four major components:
Core Library
Documentation
User Interface
Release Engineering (build)
You are assigned to the Core Library and Documentation teams with a team leader
(one of the instructors). Welcome to the team!
Lab Overview
22
WSTC 2008
Eclipse Overview
Lab Overview
23
Current Perspective
Change Perspective
A View
All screen
elements below
the button bar
are called
Views
Add new Views
using the
Window-
>Show View
menu
Menu
Button bar
View tabs
This View
has
different
areas
accessed
via the
View tabs
WSTC 2008
Team Concert Terminology
Jazz artifacts are stored in a
repository.
The repository contains project areas,
which are the system's representation
of a software projects.
Each project area has an associated
process, which governs how the
project is run.
Project Areas are decomposed into a
set of team areas, which describe the
teams that work on the project.
Teams use a stream to store the
master copy of projects files.
Team Members use a personal
repository workspace to work on
project files.
Lab Overview
24
WSTC 2008
Squawk Project in Team Concert
Lab Overview
25
Project Area
Streams
Repository
Workspaces
Team Areas
Process Definition
WSTC 2008
Team Lead Team Lead
(instructor) (instructor)
Team Team
Member Member
(student) (student)
Lab 2 Lab 1
Chat with the
Team
Chat with the
Team
Create Work
Items
Create Work
Items
Examine
Iteration Plan
Examine
Iteration Plan
Accept Team
Invite
Accept Team
Invite
Create &
Distribute
Iteration Plan
Create &
Distribute
Iteration Plan
Sequence of events Lab 1 & 2
Lab Overview
26
WSTC 2008
Team Lead Team Lead
Team Team
Member Member
(student) (student)
Lab 4 Lab 3
Project Status
with Team
Central View
Project Status
with Team
Central View
Track your
Work with My
Work View
Track your
Work with My
Work View
Create &
Deliver Work
Create &
Deliver Work
Lab Overview
27
Accept all
Changes
Accept all
Changes
WSTC 2008
Team Lead Team Lead
Team Team
Member Member
(student) (student)
Lab 5
Create &
Promote
Workspace
Snapshots
Create &
Promote
Workspace
Snapshots
Accept New
Baselines
Accept New
Baselines
Create
Component
Baselines
Create
Component
Baselines
Explore
Snapshots
Explore
Snapshots
Replace
contents of
Workspace
Replace
contents of
Workspace
Sequence of events Lab 5
Lab Overview
28
WSTC 2008
Team Lead Team Lead
Lab 6
Request
Personal Build
Request
Personal Build
Request
Integration
Build
Request
Integration
Build
Explore Build
Explore Build
Lab Overview
29
Lab 8
Change to
Development
Iteraton
Change to
Development
Iteraton
Experience
Process
Enactment
Experience
Process
Enactment
Change to the
Endgame
Iteration
Change to the
Endgame
Iteration
Team Team
Member Member
(student) (student)
WSTC 2008
Lab Conventions
The hostname used to connect to the Jazz Team Server is jazz-server
Each student is assigned a unique user id of the form student<N> based on their
student number, e.g. student1
Examples in the lab workbooks use student1, you will need to adjust per your assigned id
Every student creates their own unique Squawker.
Examples in the lab workbooks use Lion
Students can choose any squawker they want but should include your student id in
the name
<squawker>_<student id>, e.g. Lion_student1
Work items created should include the full squawker name in the summary title
<squawker name> Implementation and <squawker name> Documentation e.g.
Lion_student1 Implementation
Lab Overview
30
WSTC 2008
2008 IBM Corporation Lab Overview
31
WSTC 2008
Lab Overview
32
Agenda
Lab Overview
Session Summary
WSTC 2008
2008 IBM Corporation Session Summary
33
Session summary
We have described current collaboration challenges with distributed teams
We have explored how Team Concert can
Enable development teams to collaborate in real time in the context of the work they are
doing, especially in globally diverse environments
Enable projects to be managed more effectively by providing visibility into accurate
project health information drawn directly from actual work
Automate traceability and auditability by managing artifacts and their inter-relationships
across the lifecycle empowering teams to deliver more value
Provide customizable process design and enactment through rule-based process
guidance, automation and definable checkpoints
We have provided a hands on experience using Team Concert to automate the
software delivery process
WSTC 2008
34
Next steps
Engage your local Rational team
Provide a customized demo for your team
Conduct a targeted proof of concept
Register on jazz.net and explore learning tutorials and videos
http://www.ibm.com/developerworks/spaces/jazz
WSTC 2008
35
Additional resources
Learn more about and download free trials of Rational Team Concert at
http://ibm.com/rational/rtc
Explore Rational Team Concert tutorials, demos and other developer learning
resources
http://ibm.com/developerworks/spaces/jazz
Participate in the open commercial development of Jazz by joining the community
http://jazz.net
Learn more about the Jazz technology and the future IBM Rational product roadmap
http://ibm.com/rational/jazz/roadmap
WSTC 2008
36
WSTC 2008
37
We appreciate your feedback in order
to improve this educational event.
Please fill out the survey form.
Melvin Cheng
Advisory IT Specialist
Page 1
Topics to Cover
Pain Points
Typical Software Delivery Challenges
Overview of Build Forge
Page 2
Pain Points: Spaghetti Infrastructure Issues
Lack of consistency and/or repeatability of
configuration changes
Staff bogged down in keeping environments
synchronized
No change history / audit trail for WebSphere
configuration
Costly automation of configuration changes and
deployments
Requires custom coding
Time consuming to deploy without a framework
Page 3
Best Practices for WebSphere Administration
There are many Best Practice procedures
recommended by WebSphere but not
implemented within the product
Backup current version of application before updating with
new build
Use source control to manage & control WebSphere
configuration settings
Embed build id in J2EE EAR file Manifest.mf file as it is
built
Fall back to previous version of the application if the
update fails or is unacceptable
Store a record of who updates the application and when
(and why)
Page 4
Test
Environment
Deploy
Source
Code
Repository
Typical Software Delivery Lifecycle Roles
Source
Build
Scripts
Build
Results
Deploy
Package
+ =
Why I am I
always the
last to know
that the
build broke?
I wish I knew
immediately when
a build breaks and
could restart it
from there!!
Manually configure the server,
the test framework and deploy
the build what environments
on how many servers?
Hello! I need to get
some testing done!!!
Tester Deployer Build Engineer Developer
Page 5
Problem: Silos between development,
build, test and deployment
Inconsistent policies and
processes cannot be practiced
among all stakeholders
Silos of information contribute to
reducing team productivity
Difficult to facilitate
communication, visibility,
sharing, and leveraging globally
distributed teams without mass
chaos
Limited to no traceability among
the artifacts
Minimal reuse of artifacts and
controls
Difficulty tracking status of
multiple branches and multiple
builds
Labor Intensive
Tribal Knowledge
Error Prone Handoffs
Implement
& Iterate
Build
& Stage
Test
& Validate
Software)
Build Test
Build
Assets
Development
Assets
Test
Assets
Develop Test
Environment
Deploy
Page 6
Application deployment challenges
Inability to ensure the test
environment matches production
results in quality problems and slow
problem resolution
Customer
Dissatisfaction
Lost Revenue or
Productivity
Deployment errors result in roll-backs and
unplanned down-time of business critical
applications
Lack of an auditable deployment
process hinders ability to comply with
external regulations
Penalties,
Shareholder
Value
Manually configuring complex
applications to deployment servers
results in inefficient use of IT resources
High Labor
Costs
Page 7
System Deployments become increasingly complex
A Composite Web Application, Involving J2EE, Integration Middleware, and Legacy Systems
Page 8
Build & deployment motivators
Compliance
Requires the ability to track software from development to production and
back again
Repeatability & Reliability
How to reliably deploy software through multiple test phases and ultimately
to production.
Process a need to apply a standardized set of steps in an organization
Server Configurations production server configurations dont always
match test lab configurations, which leads to errors in deployment
Change updates to existing systems often cause failure due to a lack of
understanding of how the application is currently deployed, and what
impact the new application will have on the existing system
Security authentication and electronic signature approvals
Lifecycle Automation
Be able to queue multiple automation efforts into a single automated
process
Build -> Deploy ->Test as a single automated process
Page 9
Can you trace your development artifacts?
D
e
v
e
l
o
p
m
e
n
t
O
p
e
r
a
t
i
o
n
s
Release
Build
Artifacts
Source
Code
Environment
Changes
Implemented with
this
Built using these
In this
Executable
This
Deployment
Artifacts
Which deployed
these
System
on this
Application
Of this
Tests
Test Results
Where these
Produced these
Build Scripts
Which required
these
Requirements
Approval
Implemented these
Approved by
Page 10
Development Matters: Can you pass this audit?
What is your deployment process ?
Who approves the deployment of applications into test and production
environments. Where is the audit trail?
What build artifacts are associated with this release of the application?
What version of build artifacts are deployed in the Test Environment? The
Production Environment?
What version of source is associated with this build artifact?
Who deployed the code on this server? How is that controlled?
How are approvers authenticated & how do they sign off?
Page 11
Deployment Questions
How can I create and enforce a repeatable process for deploying applications
through a variety of test environments prior to going into production
How can I ensure that the build that was approved during testing is the same build
deployed into production?
How can I prevent unauthorized tampering of the build artifacts?
Can I recreate the build that is currently deployed in a specific environment?
What build artifacts are loaded on Testing Servers? The Production Servers?
How can we load different machines with the same application version
simultaneously?
Page 12
Software Delivery Lifecycle Challenges
Manual, error-prone processes
Proprietary, internally-developed
systems
Inconsistent processes for different
products and platforms
Dependence on the build team for
execution, status, and troubleshooting
Separation of developer environments
from production systems
Difficult and time consuming to resolve
problems
Difficulty proving adherence with
compliance requirements
Delays and missed deadlines for
application releases/updates
Unpredictable product release cycles
Costly systems to support, troubleshoot
and maintain with knowledge held by
few
More work requires more people
limited repeatability or portability for
new projects and platforms
Burdens staff and requires increased
headcount to take on new projects
Requires better tracking and auditing
of application lifecycle
Current Conditions Typical Challenges Business Impact
Page 13
Build Forge is a process execution framework
Automates any set of tasks that can be executed from command line or scripts
Eliminates manual handoffs between code, build, test, package, and deploy activities
Self documenting processes that record all results in one auditable location
Manages the activities in the software development
lifecycle
Application build and release process management
Runtime customization and application deployment
The conveyor belt analogy for a software assembly line
Automating Software Delivery with Build Forge
Page 14
Implement
& Iterate
Build
& Stage
Test
& Validate
Software
Test
Environment
Build Forge Streamlines Software Delivery
Automate, Accelerate, Coordinate, Track, Analyze
Reduces global development
cost by automating processes,
handoffs, and communication
Accelerates development cycles
by enabling rapid iterations and
rapid feedback
Improves quality by managing
product assembly tasks for
consistent results each time
Practical solution that leverages
existing scripts and tools for rapid
implementation and results
Provides traceability and control
for compliance management and
better software governance
Develop Test
IBM Rational Clear Case - Source Control
Deploy
IBM Rational Clear Quest - Defect Tracking
Build
IBM Rational Testing Suite - Testing Tools
IBM Rational Build Forge
Development
Assets
Build
Assets
Test
Assets
Page 15
In
Production
Source
Build &
Deploy
Artifacts
Implement
Build
& Stage
Provision
& Validate
Server(s)
Function
Test
Approve
Defects
E
-
S
IG
Integration
Production
Project Manager Tester Deployer Build Engineer Developer
Automation Orchestrates The Roles
Requirements
Page 16
Pushing the Quality Agenda Upstream
Why Customers Choose Build Forge
Hold developers to a higher level of
quality
Identify & resolve quality issues faster
Enable agile development
Meeting Compliance Requirements
Avoid the dreaded failed application
audit
Get audit control and traceability
Secure the build environment
Achieving The Software Factory
Software production as a service
Standardize and reuse across teams
Remove obstacles to distributed
development
Time-to-Market Advantages
Remove the software production
bottlenecks
Automate the assembly process
Remove manual handoffs
Page 17
Build Forge Extensive Tool Integration
Deployment & Provisioning
Tools
Tivoli Provisioning Manager
WebSphere Admin (wsadmin)
WebSphere Service Registry
VMWare, VMLogix
Ascendant WICA
Software Quality Tools
Mecury Quality Center,
LoadRunner, TestDirector
WinRunner, Junit
Rational Test Lab Manager,
Rational Functional, Performance
& Manual Tester
Agitar
Monitor, Record, Report
Automate and manage
lifecycle processes
IBM Rational Build Forge
Source Control & Change
Management Tools
Rational ClearCase
Rational ClearQuest
StarTeam, Perforce
CVS, PVCS, VSS
Synergy
Subversion
Remedy, Bugzilla
Endevor, ChangeMan
Development Tools
Rational Application Developer
Eclipse, VisualStudio
Rational Automated Software Analyst
Rational Team Concert (Jazz)
Klocwork, Black Duck
Compile & Assemble Tools
Compilers (any)
Linkers (all)
Ant, Make, NMake
Maven, Rational ClearMake
Jar, rpm, zip,
Governance Tools
Rational Method Composer
Rational ClearQuest
WebLayers
Rational Asset Manager
Page 18
Process Automation
Automated, Distributed, Repeatable, Consistent Software Delivery Lifecycle
Solution Overview
Python, VBscript, Batch, Perl, KSH Scripting
Source
Control
Product
Build
Quality
Assurance
Package Release Development
Control Acceleration Environment Notification Scheduling Log Analysis Tracking Analysis
Platforms UNIX, Windows Mac, Linux, zSeries, iSeries, Proprietary
ClearCase
StarTeam
Perforce
CVS
PVCS
VSS
Synergy
Subversion
etc.
ClearMake
Ant
NAnt
Make
GNUMake
NMake
Open Source
CruiseControl
Maven
etc.
Java
C
C++
C#
etc.
Source Control Languages Build Tools
TestManager
Performance
Functional
Robot
LoadRunner
TestDirector
WinRunner
Junit
etc.
Test Tools
Tivoli
WebSphere
WebLogic
etc.
Release
RAD
Eclipse
Visual .NET
IDEs
ClearQuest
Remedy
ChangeMan
DevTrack
Bugzilla
etc.
Change Mgmt
Wise
InstallShield
jar
RPM
Package
etc.
Package Tools
Management Console
Centralized Web-based Management, Collaborative Distributed Access, Role-Based Security
IDE Plug-Ins
Developer Self-Service, Role-Based Security
SOFTWARE DELIVERY AUTOMATION FRAMEWORK
Page 19
Example Build Forge Automated Process
Build Forge
GO
Rational
Clear
Case
Rational
Clear
Quest
RAM
1.
Extract
Source
Code
6.
Update
Defect
Tracking
Aggregated
Reports
across entire
Process
2.
Extract
Asset
Metadata
3.
Enforce SOA
Development
Policies
4.
Build
Source
Code
5.
Package
Build
Results
7.
Deploy
Build
Results
8.
Execute
Tests
Effect of Class Sharing on Memory Footprint in WAS V6.1
126
245
363
481
600
719
140
233
328
422
516
609
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6
Number of Running App Servers
M
e
g
a
b
yte
s
WAS 6.02 WAS 6.1
Systemconfiguration
Linux: IBMX-Series 335, 2x 3.2 GHz Intel Xeon, 8.00 GB RAM, Suse Enterprise Linux 9 SP3
Rational
ClearCase
Adaptor
Rational
Asset
Manager
Adaptor
WebLayers
Policy-based
Enforcement
Adaptor
Rational
ClearQuest
Adaptor
Mercury
Quality
Center
Adaptor
WICA
Adaptor
Page 20
Functional Architecture
Build Forge Master Database
Projects and Steps
Server Configurations
Environment Configurations
User/Permission Information
Build Statistics
System Master Log
Build Forge Engine
3 Tier Architecture
Centrally Managed
Orchestrates BuildForge Tasks
Server Pools
Manage
- Users
- Resources
Author
- Capture Processes and Steps
Execute
- Rune/Schedule Jobs
- View Real-Time Status
- Troubleshoot Build Issues
Analysis
- Reporting & Analysis
Linux HP-UX zSeries Mac Windows Solaris
AIX Linux Windows Windows Windows
Integration
Command Line
Adaptors
API
Developer Self Service
DB2
Oracle
MySQL
SQL Server
Sybase
Linux HP-UX AIX Mac
Windows Solaris
Quality Assurance
Build Systems
Linux HP-UX zSeries
Windows Solaris
Production
Build Forge
Management Console
Build Forge
IDE Plugin
Build Forge
Server
Build Forge
Agents
Agents
Real Time Environment Configuration
Execute Project Steps
Return Output / Logs to Mgmt Server
ACCESS EXECUTE IMPLEMENT
Source Code
Repositories
Test Suites
Deployment
Tools
SCM
Applications
iSeries
Database
Build Stats
BOM data
Engine
Page 21
Master Database
Admin & Server Configurations
Environment Configurations
User/Permission Information
All project record data
Server Pool
Linux Windows Solaris AIX Linux Windows Windows Windows
Linux HP-UX AIX
Mac Windows Solaris
Braintree
Bangalore
Dallas
Build Log Database
Build Statistics
BOM data
Build Log Database
Build Statistics
BOM data
Console / Engine
Console / Engine
Console / Engine
Console / Engine
Build Log Database
Build Statistics
BOM data
Log
DB
Log
DB
Log
DB
Master
DB
GDD features focuses on performance
improvements for remote agents writing to the
database
Architected for Centralized Management
Across Geographically Distributed Development
Page 22
Building Blocks
Containers that hold a series of steps necessary to execute a process.
A project can contain a series of diverse tasks beyond the compile, including
automated tests, running installers, updating databases, or a website update.
Project
Individual tasks performed as part of a project. Each contains one or
more command line statements to be executed on a server machine. A
step can run a script, invoke another application, move or copy files,
export a project for backup, and more.
Steps
Environment
Collections of environment variables which can be maintained separate
from projects and steps, then assigned to projects and steps as needed.
1. RELEASE=Release_1.1
2. JAVA_HOME=C:\Program Files\Java\jdk1.5.0_06
3. PATH=C:\Program Files\Java\jdk1.5.0_06\bin
4.
Build Test Package Deploy Source
1a. cleartool mkview -snapshot -tag $BF_TAG -vws

$VIEW_STG\$BF_TAG.vws $BF_SERVER_ROOT\$BF_TAG
1b. cleartool setcs -tag $BF_TAG config.spec
2a. gcc main.c -o main.o
2b. gcc main.c -o ui.o
2c. make Release
3a. testscript.sh run r $RELEASE module HelloWorld.exe
Page 23
Build Forge project
Page 24
Build Forge project
Threading: Threading allows for multiple Steps to run in parallel,
thereby increasing the speed of your build.
Multiple, adjacent, Threaded steps are thought to be a Thread Block
Threaded Steps are indented on the Project Page
Page 25
Build Forge projects - Threading
Get Src
Compile Solaris
Compile Linux
Test Package Deploy Init
Compile Windows
Get Src
Compile Solaris Compile Linux
Compile Windows
WIthout ThreadIng
WIth ThreadIng
Page 26
Build Forge environments
The BuildForge system allows you to manage environment variables
separately from your Projects, Servers, and Steps
You can define Environment Groups
Containing one or more Environment Variables
You can assign Environment Groups to Servers, Projects, and Steps
At runtime, the system assembles the final working environment for a Step from all of the
relevant Groups assigned to the Server, Project, and Step
Page 27
Build Forge environments
Page 28
Build Forge security
Concepts in the BuildForge security system:
Users A user in the BuildForge environment corresponds to a single, named
person who interacts with the system in some way.
Access Groups An access group is a collection of users who are assigned
permissions to perform specific tasks within the BuildForge system.
An Access Group is usually associated with a logical Role within an organization
the .NET User Interface development team, for example.
Permissions Permissions are typically set on system level functionality and are
assigned to Access Groups only not individual Users.
Ownership Specific resources Projects, Servers, Steps, etc. in the
BuildForge system have a single, named Access Group as Owner.
Ownership over an entity allows for full edit/delete/execute privileges over that resource.
Page 29
Role-based system
Access Groups represents a role a User has in organization
Roles have Permissions and Ownership
User access is determined by the union of the Permissions of all
groups the user belongs to
Combination of Permissions and Ownership define what a group can
do and/or see
To edit a give Project, a User would have to have Ownership on the Project AND
Permissions to Edit Projects
Example
User who is a member of the Guest group (and no other groups) sees only Projects
which have the Guest group assigned as their Access property
User can only launch projects with Guest access
Page 30
Page 31
Build Forge servers
A Server is a logical resource in the BuildForge environment that
represents any physical system running the BuildForge Agent.
A Server is defined within the BuildForge system with certain properties, such as a
default path.
A Server Pool is a group of Servers with similar build properties
(running the same operating system and with the same compilers
available, for example).
The BuildForge system can run a Project on a different Server in the same Pool
when the default Server for the Project is busy.
Page 32
Build Forge servers
Page 33
Build Forge project execution
Example Schedule
Page 34
Project/Process
Putting it all together
Environment
Get Src
Compile Solaris
Compile Linux
Compile Windows
(main)
Reusable Steps
Compile Windows
(ui)
Link Windows
(main, ui)
Control
Environment
Notification Scheduling Log Analysis Tracking
Analysis
Pooling Threading
Error/Cleanup
Page 35
Project/Process
Putting it all together
Environment
Get Src
Compile Solaris
Compile Linux
Deploy Init
Compile z/OS
(main)
Reusable Steps
Test Package
Compile z/OS
(cobol, cics)
Link
(main, cics)
Control
Environment
Notification Scheduling Log Analysis Tracking
Analysis
Pooling Threading
Page 36
Ensure Traceability Among the Artifacts
Create an automated, documented audit trail
Release
Build
Artifacts
Source
Code
Environment
Changes
Implemented with
this
Built using
these
In this
Executable
This
Deployment
Artifacts
Which
deployed these
System
on this
Application
Of this
Tests
Test Results
Where these
Produced
these
Build Scripts
Which required
these
Requirements
Implemented
these
Approved by
Approval
Development Pre-Prod
Transparent
process for
turning source
code into software
Records of how
software was
released into
production
Page 37
Summary
The Build Forge architecture provides a robust, flexible foundation
for build (and more) process automation.
We have discussed
Build Forge Architecture
Projects
Environments
Security
Servers
Project Execution
In the following labs we will dive into a Build Forge environment and
see how to use these concepts.
Page 38
Java Garbage Collection
Java Garbage Collection
Best Practices for Sizing and Tuning the Java Heap Best Practices for Sizing and Tuning the Java Heap
22
Conference materials may not be reproduced in whole or in part without the prior written permission of IBM.
Objectives
Objectives
Overview
Selecting the Correct GC Policy
Sizing the Java heap
Questions/Answers
33
Garbage Collection Performance
Garbage Collection Performance
GC performance issues can take many forms
Definition of a performance problem is user centric
User requirement may be for:
Very short GC pause times
Maximum throughput
A balance of both
First step is ensure that the correct GC policy has been
selected for the workload type
Helpful to have an understanding of GC mechanisms
Second step is to ensure heap sizing is correct
Third step us to look for specific performance issues
55
Understanding Garbage Collection
Understanding Garbage Collection
Responsible for allocation and freeing of:
Java objects, Array objects and Java classes
Allocates objects using a contiguous section of Java heap
Ensures the object remains as long as it is in use or live
Determination based on a reference from another live object or from
outside of the Heap
Reclaims objects that are no longer referenced
Ensures that any finalize method is run before the object is
reclaimed
66
Object Allocation
Object Allocation
Requires a contiguous area of Java heap
Driven by requests from:
The Java application
JNI code
Most allocations take place in Thread Local Heaps (TLHs)
Threads reserve a chunk of free heap to allocate from
Reduces contention on allocation lock
Keeps code running in a straight line (fewer failures)
Meant to be fast
Available for objects < 512 bytes in size
Larger allocates take place under a global heap lock
These allocations are one time costs out of line allocate
Multiple threads allocating larger objects at the same time will contend
77
Object Reclamation (Garbage Collection)
Object Reclamation (Garbage Collection)
Occurs under two scenarios:
An allocation failure
An object allocation is requested and not enough contiguous memory is available
A programmatically requested garbage collection cycle
call is made to System.GC() or Runtime.GC()
the Distributed Garbage Collector is running
call to JVMPI/TI is made
Two main technologies used to remove the garbage:
Mark Sweep Collector
Copy Collector
IBM uses a mark sweep collector
or a combination for generational
88
Global Collection Policies
Global Collection Policies
Garbage Collection can be broken down into 2 (3) steps
Mark: Find all live objects in the system
Sweep: Reclaim unused heap memory to the free list
Compact: Reduce fragmentation within the free list
All steps are in a single stop-the-world (STW) phase
Application pauses whilst garbage collection is done
Each step is performed as a parallel task within itself
Four GC Policies, optimized for different scenarios
-Xgcpolicy:optthruput optimized for batch type applications
-Xgcpolicy:optavgpause optimized for applications with responsiveness
criteria
-Xgcpolicy:gencon optimized for highly transactional workloads
-Xgcpolicy:subpools optimized for large systems with allocation
contention
99
Parallel Mark Sweep Collector, with compaction avoidance
Created to make use of additional processors on server systems
Designed to increase performance for SMP and not degrade
performance for uni-processor systems
Optimized for Throughput
Best policy for batch type applications
Consists of a single flat Java heap:
Parallel GC (
Parallel GC (
optthruput
optthruput
)
)
0 GB 2 GB
Heap Base
Heap Limit
Heap Size
LOA
10 10
Parallelism achieved through the use of GC Helper Threads
Parked set of threads that wake to share GC work
Main GC thread generates the root set of objects
Helper threads share the work for the rest of the phases
Number of helpers is one less than the number of processing units
So helper threads and main GC thread equals the number of
processing units
Configurable using -Xgcthreads
GC Helper Threads
GC Helper Threads
11 11
Parallel Mark/Parallel Sweep view of GC
Parallel Mark/Parallel Sweep view of GC
Avolo Compactlon
MARK SW
Parallel MARK
SW
-Parallel SWLLP
MARK SW COMPACT Stanoaro GC
M S
M
12 12
Reduces and makes more consistent the time spent inside
Stop the World GC
Reduction usually between 90 and 95%
Achieved by carrying out some of the STW work whilst
application is running
1.4.2: Concurrent Marking
5.0: Concurrent Marking and Concurrent Sweeping
Slight overhead on thruput for greatly reduced STW times
Policy is ideal for systems with responsiveness criteria
eg. Portal applications
Concurrent GC (
Concurrent GC (
optavgpause
optavgpause
)
)
13 13
Parallel and Concurrent Mark/Sweep
Parallel and Concurrent Mark/Sweep
-Parallel MARK/SWLLP
-Concurrent MARK
-any lole cpu tlme
-Concurrent Sweep
-any lole cpu tlme
M S
MS
M
Concurrent Kickoff
14 14
Concurrent Mark
Concurrent Mark
hidden object issue

hidden object issue
Higher heap usage
15 15
Higher heap usage
because not all garbage removed
Concurrent Mark
Concurrent Mark
hidden object issue

hidden object issue
Dangling pointer!
16 16
Similar in concept to that used by Sun and HP
Parallel copy and concurrent global collects by default
Motivation: Objects die young so focus collection efforts
on recently created objects
Divide the heap up into a two areas: new and old
Perform allocates from the new area
Collections focus on the new area
Objects that survive a number of collects in new area are
promoted to old area (tenured)
Ideal for transactional and high data throughput
workloads
Generational and Concurrent GC (
Generational and Concurrent GC (
gencon
gencon
)
)
0 GB 2 GB
Heap Base
Heap Limit
Heap Size
LOA
Nursery (new) Space Tenured (old) Space
Allocate Survivor
17 17
Allocate Space Survivor Space
Nursery is split into two spaces (semi-spaces)
Only one contains live objects and is available for allocation
Minor collections (Scavenges) move objects between spaces
Role of spaces is reversed
Nursery/Young Generation
Survivor Space Allocate Space
Movement results in implicit compaction
Nursery (new) Space Copy Collection
Nursery (new) Space Copy Collection
18 18
Subpooling
Subpooling
(
(
subpool
subpool
)
)
Goals:
Reduce allocation lock contention by distributing free memory into multiple lists
Reduce allocation contention through use of atomic operations instead of a
heap lock
Prevent premature garbage collections by using a best fit (or closer to best fit)
policy instead of address ordered
Ideal for very large SMP systems where large amounts data
is being allocated
where there is heap lock contention
19 19
Looking for Heap Lock Contention
Looking for Heap Lock Contention
All locks can be profiled using Java Lock Analyzer (JLA)
http://www.alphaworks.ibm.com/tech/jla (AlphaWorks)
Provides time accounting and contention statistics for
Java and JVM locks
Functionality includes:
Counters associated with contended locks
Total number of successful acquires
Recursive acquires times a thread acquires a lock it already owns
Number of times a thread blocks because a monitor is already owned
Cumulative time the monitor was held.
20 20
JLA Sample Report
JLA Sample Report
System (Registered) Monitors
%MISS GETS NONREC SLOW REC TIER2 TIER3 %UTIL AVER-HTM MON-NAME
87 5273 5273 4572 0 710708 18487 1 95408 JITC Global_Compile lock
9 6870 6869 631 1 113420 2976 0 11807 Heap lock
5 1123 1123 51 0 11098 286 1 248385 Binclass lock
0 1153 1147 5 6 1307 33 0 47974 Monitor Cache lock
0 46149 45877 134 272 36961 877 1 6558 JITC CHA lock
0 33734 23483 19 10251 6544 150 1 17083 Thread queue lock
0 5 5 0 0 0 0 0 9309689 JNI Global Reference lock
0 5 5 0 0 0 0 0 9283000 JNI Pinning lock
0 5 5 0 0 0 0 0 9442968 Sleep lock
0 1 1 0 0 0 0 0 0 Monitor Registry lock
0 0 0 0 0 0 0 0 0 Evacuation Region lock
0 0 0 0 0 0 0 0 0 Method trace lock
0 0 0 0 0 0 0 0 0 Classloader lock
0 0 0 0 0 0 0 0 0 Heap Promotion lock
Java (Inflated) Monitors
%MISS GETS NONREC SLOW REC TIER2 TIER3 %UTIL AVER-HTM MON-NAME
15 68 68 10 0 2204 56 2 11936405 test.lock.testlock1@A09410/A09418
2 42 42 1 0 186 5 0 300478 test.lock.testlock2@D31358/D31360
0 70 70 0 0 41 1 0 7617 java.lang.ref.ReferenceQueue$Lock@920628/920630
21 21
JLA: Fields in the report
JLA: Fields in the report
22 22
Choosing the Right GC Policy
Choosing the Right GC Policy
Four GC Policies, optimized for different scenarios
-Xgcpolicy:optthruput optimized for batch type applications
-Xgcpolicy:optavgpause optimized for applications with
responsiveness criteria
-Xgcpolicy:gencon optimized for highly transactional
workloads
-Xgcpolicy:subpools optimized for large systems with allocation
contention
How do I know whether to use optavgpause or gencon?
Monitor GC activity
Look for certain characteristics
23 23
Monitoring GC Activity
Monitoring GC Activity
Use of Verbose GC logging
only data that is required for GC performance tuning
Graph Verbose GC output using GC and Memory Visualizer (GCMV)
from ISA
Activated using command line options
-verbose:gc
-Xverbosegclog:[DIR_PATH][FILE_NAME],X,Y
where:
[DIR_PATH] is the directory where the file should be written
[FILE_NAME] is the name of the file to write the logging to
X is the number of files to
Y is the number of GC cycles a file should contain
Performance Cost:
(very) basic testing shows a 2% overhead for GC duration of 200ms
eg. if application GC overhead is 5%, it would become 5.1%
24 24
Important Characteristics for Choosing
Important Characteristics for Choosing
GC Policy
GC Policy
Rate of Garbage Collection
High rates of object burn point to large numbers of transitional objects, and therefore
the application may well benefit from the use of gencon
Large Object Allocations?
The allocation of very large objects adversely affects gencon unless the nursery is
sufficiently large enough. The application may well benefit from optavgpuse
Large heap usage variations
The optavgpause algorithms are best suited to consistent allocation profiles
Where large variations occur, gencon may be better suited
Rule of thumb: if GC overhead is > 10%, youve most likely chosen the wrong one
25 25
optavgpause gencon
Gencon could handle a higher rate of garbage collection
Completing the test quicker
Gencon had a smaller percentage of time in garbage collection
Gencon had a shorter maximum pause time
26 26
Gencon provides less frequent long Garbage Collection cycles
Gencon provides a shorter longest Garbage Collection cycle
27 27
Large Object Allocations
Large Object Allocations
(Very) Large Object allocations affects the gencon GC policy
If object is larger than the Nursery size, the object is immediately tenured
Removes the benefit of generational heaps
Still has the additional overhead of running generational
If object is fits in the nursery but fills it, frequent nursery collects will have to occur
Too frequent nursery collects mean objects are likely to survive and need copying
Copying is an expensive process
If (Very) Large Objects are being used, a sufficiently large enough nursery is
required
Sizing the Java Heap
29 29
Maximum possible Java heap sizes
The correct Java heap size
Fixed heap sizes vs. Variable heap sizes
Heap Sizing for Generational GC
30 30
Maximum Possible Heap Size
Maximum Possible Heap Size
32 bit Java processes have maximum possible heap size
Varies according to the OS and platform used
Determined by the process memory layout
64 bit processes do not have this limit
Limit exists, but is so large it can be effectively ignored
Addressability usually between 2^44 and 2^64
Which is 16+ TeraBytes
31 31
An Operating System process like any other application:
Subject to OS and architecture restrictions
32bit architecture has an addressable range of:
2^32 which is 0x00000000 0xFFFFFFFF
which is 4GB
Not all addressable space is available to the application
The operating system needs memory for:
The kernel
The runtime support libraries
Varies according to Operating System
How much memory is needed and where that memory is located
0 GB 4 GB
0x0 0xFFFFFFFF
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Java Process Memory Layout
Java Process Memory Layout
32 32
Memory Available to the Java Process
Memory Available to the Java Process
On Windows:
On AIX:
0 GB 4 GB
0x0 0xFFFFFFFF
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Operating System Space
Libraries
0 GB 4 GB
0x0 0xFFFFFFFF
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Kernel Libraries
33 33
Java Process Restrictions
Java Process Restrictions
Not all Java Process space is available to the Java
application
The Java Runtime needs memory for:
The Java Virtual Machine
Backing resources for some Java objects
This memory area as well as some other allocations, is part of the Native
Heap
Memory not allocated to the Java Heap is available to the native heap
Available memory space Java heap = native heap
Effectively, the Java process maintains two memory pools
34 34
The
The
Native
Native
Heap
Heap
Allocated using malloc() and therefore subject to memory
management by the OS
Used for Virtual Machine resources, eg:
Execution engine
Class Loader
Garbage Collector infrastructure
Used to underpin Java objects:
Threads, Classes, AWT objects, ZipFiles
Used for allocations by JNI code
35 35
Native Heap available to Application
Native Heap available to Application
On Windows
On AIX (1.4.2 with small heaps)
0 GB 4 GB
0x0
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Operating System Space
Libraries
Java Heap
0xFFFFFFFF
VM Resources
Native Heap
0 GB 4 GB
0x0 0xFFFFFFFF
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Kernel Libraries Java Heap
VM Resources
Native Heap
36 36
Layout with Large Java Heaps on AIX
Layout with Large Java Heaps on AIX
Applies to heaps > 1GB in size and Java 5.0
Java heap becomes allocated using mmap()
Segments used start at 0xC and work downwards
understanding memory layout important for monitoring
0 GB 4 GB
0x0 0xFFFFFFFF
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Kernel Libraries Java Heap
VM Resources
0x7
Native Heap
0xD 0x3
37 37
Linux:
z/OS:
Memory Layout for Linux
Memory Layout for Linux
0 GB 4 GB
0x0
2 GB
0x80000000
1 GB 3 GB
0x40000000 0xC0000000
Kernel Java Heap
0xFFFFFFFF
VM Resources
Native Heap
PAGE_OFFSET TASK_SIZE
0 GB
0x0
2 GB
0x7FFFFFFF
1 GB
0x40000000
Java Heap
VM Resources
38 38
Theoretical and Advised Max Heap Sizes
Theoretical and Advised Max Heap Sizes
The larger the Java heap, the more constrained the native heap
Advised limits to prevent native heap from becoming overly
restricted, leading to OutOfMemoryErrors
Exceeding advised limits possible, but should be done only when
native heap usage is understood
Native heap usage can be measured using OS tools:
Svmon (AIX), PerfMon (Windows), RMF (zOS) etc
1.8GB 1.8GB /3GB
2.5GB 3 GB Hugemem Kernel
Advised Maximum Maximum Possible Additional Options Platform
1.3GB 1.7GB z/OS
1.5GB 1.8GB Windows
1.5GB 2 GB Linux
2.5GB 3.25 GB automatic AIX
39 39
Moving to 64bit
Moving to 64bit
Moving to 64bit remove the Java heap size limit
However, ability to use more memory is not free
64bit applications perform slower
More data has to be manipulated
Cache performance is reduced
64bit applications require more memory
Java Object references are larger
Internal pointers are larger
Major improvements to this in Java 6.0 due to compressed
pointers
40 40
The
The
correct
correct
Java heap size

Java heap size
GC will adapt heap size to keep occupancy between 40% and 70%
Heap occupancy over 70% causes frequent GC cycles
Which generally means reduced performance
Heap occupancy below 40% means infrequent GC cycles, but cycles longer
than they needs to be
Which means longer pause times that necessary
Which generally means reduced performance
The maximum heap size setting should therefore be 43% larger than the
maximum occupancy of the application
Maximum occupancy + 43% means occupancy at 70% of total heap
Eg. For 70MB occupancy, 100MB Max heap required, which is 70MB + 43% of 70MB
41 41
The
The
correct
correct
Java heap size

Java heap size
Long Garbage Collection Cycles
To Frequent Garbage Collection
M
e
m
o
r
y
Time
70%
40%
Heap Occupancy
Heap Size
42 42
Should the heap size be fixed?
ie. Minimum heap size (-Xms) = Maximum heap size (-Xmx)?
Each option has advantages and disadvantages
As for most performance tuning, you must select which is right for the
particular application
Variable Heap Sizes
GC will adapt heap size to keep occupancy between 40% and 70%
Expands and Shrinks the Java heap
Allows for scenario where usage varies over time
Where variations would take usage outside of the 40-70% window
Fixed Heap Sizes
Does not expand or shrink the Java heap
43 43
Heap Expansion and Shrinkage
Heap Expansion and Shrinkage
Act of heap expansion and shrinkage is relatively cheap
However, a compaction of the Java heap is sometimes
required
Expansion: for some expansions, GC may have already compacted
to try to allocate the object before expansion
Shrinkage: GC may need to compact to move objects from the area
of the heap being shrunk
Whilst expansion and shrinkage optimizes heap occupancy,
it (usually) does so at the cost of compaction cycles
44 44
Conditions for Heap Expansion
Conditions for Heap Expansion
Not enough free space available for object allocation after
GC has complete
Occurs after a compaction cycle
Typically occurs where there is fragmentation or during rapid
occupancy growth (ie, application startup)
Heap occupancy is over 70%
Compaction unlikely
More than 13% of time is spent in GC
Compaction unlikely
45 45
Conditions for Heap Shrinkage
Conditions for Heap Shrinkage
Heap occupancy is under 40%
And the following is not true:
Heap has been recently expanded (last 3 cycles)
GC is a result of a System.GC() call
Compaction occurs if:
An object exists in the area being shrunk
GC did not shrink on the previous cycle
Compaction is therefore likely to occur
46 46
Introduction to
Introduction to
Xmaxf
Xmaxf
and
and
Xminf
Xminf
The Xmaxf and Xminf settings control the 40% and 70%
occupancy bounds
-Xmaxf: the maximum heap space free before shrinkage (default is
0.6 for 60%)
-Xminf: the minimum heap space before expansion (default is 0.3 for
70%)
Can be used to move optimum occupancy window if
required by the application
eg. Lower heap utilization required for more infrequenct GC cycles
Can be used to prevent shrinkage
-Xmaxf1.0 would mean shrinkage only when heap is 100% free
Would completely remove shrinkage capability
47 47
Introduction to
Introduction to
Xmaxe
Xmaxe
and
and
-
-
Xmine
Xmine
The Xmaxe and Xmine settings control the bounds of the
size of each expansion step
-Xmaxe: the maximum amount of memory to add to the heap size in
the case of expansion (default is unlimited)
-Xmine: the minimum amount of memory to add to the heap size in
the case of expansion (default is 1MB)
Can be used to reduce/prevent compaction due to expansion
Reduce expansions by setting a large -Xmine
48 48
GC Managed Heap Sizing
GC Managed Heap Sizing
Long Garbage Collection Cycles
To Frequent Garbage Collection
M
e
m
o
r
y
Time
-Xminf
-Xmaxf
Heap Occupancy
Heap Size
Expansion (>= -Xmine)
49 49
Fixed or Variable??
Fixed or Variable??
Again, dependent on application
For flat memory usage, use fixed
For widely varying memory usage, consider variable
Variable provides more flexibility and ability to avoid
OutOfMemoryErrors
Some of the disadvantages can be avoided:
-Xms set to lowest steady state memory usage prevents expansion at
startup
-Xmaxf1 will remove shrinkage
-Xminf can be used to prevent compaction before expansion
-Xmine can be used to reduce expansions
50 50
-Xmosm -Xmoxm
A C B
-Xmnsm -Xmnxm
-Xmom
Nursery Tenured
0 0
-Xmnm
A B
Fix both nursery and tenured space
Allow them to expand/contract
General Advice:
Fix the new space size
Size the tenured space as you would for a flat heap
Options are:
51 51
Sizing the Nursery
Sizing the Nursery
Copying from Allocate to Survivor or to Tenured space is
expensive
Physical data is copied (similar to compaction with is also expensive
Ideally survival rates should be as low as possible
Less data needs to be copied
Less tenured/global collects that will occur
The larger the nursery:
the greater the time between collects
the less objects that should survive
However, the longer a copy can potentially take
Recommendation is to have a nursery as large as possible
Whilst not being so large that nursery collect times affect the
application responsiveness
52 52
Summary
Summary
GC Policy should be chosen according to application
scenario
Java heap should ideally be sized for between 40 and 70%
occupancy
Min=Max heap size is right for some applications, but not for
others
53 53
Questions
Questions
Please remind students to complete
their session evaluation form
Thank you!!
D02 D02 A Behind the Scenes Look at the Performance Improvements in A Behind the Scenes Look at the Performance Improvements in
WPS/WESB v6.1 WPS/WESB v6.1
Speaker Name Vijay K Peachimuthu/India/IBM Vijay K Peachimuthu/India/IBM
E-Mail vpeachim@in.ibm.com vpeachim@in.ibm.com
Title Consulting IT Architect Consulting IT Architect
Agenda
Recent WPS Performance Improvement Themes
Key Performance Enhancements in Previous WPS/WESB Versions
B i P E i (BPE) Business Process Engine (BPE)
SCA Performance Improvements
Overall WPS Code Path length
WESB S ifi P f I t WESB Specific Performance Improvements
WBM v6.1 Performance Improvement Highlights
Key Performance Enhancements in WPS/WESB v6.1
Business Process Engine (BPE)
General WPS Runtime
Areas of Investigation
Gain new insights, knowledge and skills | 19-22 August 2008 Suntec Singapore Page 2
Recent WPS Performance Themes
Core SCA invocation improvements
M h li i ti Message hop elimination
Eliminate over-reliance on messaging
Reduce SCA overhead on system services
Improve efficiency of generated code Improve efficiency of generated code
Optimize expensive interactions with EMF
Bread and butter profile-driven performance analysis based
improvements
p
WPS v6.02 BPE Improvements
Exploiting Materialized Views
The BPC data model (schema) is optimized for create, retrieve, and
d t ti d t id d li ti f d t i th d t b update operations and to avoid duplication of data in the database
Navigation is very fast but queries against the model often require
access to multiple tables
Materialized Views enables single table performance levels for
queries that access data from multiple tables
Materialized Views
With materialized views, complex queries become
simple queries that are executed against a single simple queries that are executed against a single
table only
MV_1
WorkItem
Process
Instance
Task
Task
Description
Query
Properties
Exploiting Materialized Views with HTM
Human task queries benefit the most from materialized views since human task
queries are complex and materialized views improves complex database queries
Starting with WPS v6.0.2.1 g
Requires materialized views support from associated DB provider
Steps to configure
C t th t l t bl f th t i li d i Create the control table for the materialized views
Create the table spaces for the materialized views
Give the user that accesses Business Process Choreographer database (<BPEDB>) the rights that
are required to administer the materialized views
Define the materialized views Define the materialized views
Activate the changes by restarting the WPS server/cluster
WPS Information
http://www ibm com/support/docview wss?rs=2307&uid=swg27009623 http://www.ibm.com/support/docview.wss?rs=2307&uid=swg27009623
DB2 Materialized Views Information (DB2 Infocenter)
http://public.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=/com.ibm.websphere.ii.db2udb.
nav.doc/navcontainers/iiypnavc tuning mqt.html
nav.doc/navcontainers/iiypnavc_tuning_mqt.html
WPS v6.02 Key Enhancements
Benefits fromMaterialized Views Benefits from Materialized Views
14000
14000
Query Response Time [ms]
97
90
100
DB Server CPU Utilization [%]
6000
8000
10000
12000
40
50
60
70
80
90
No optimizations
[ms]
6
0
2000
4000
No optimizations
Utilization
[%]
22.5
0
10
20
30
No optimizations
With materialized views
No optimizations
With materialized views
Additional details can be found in the paper Performance Tuning of Human Workflows Using Materialized Views
Additional details can be found in the paper Performance Tuning of Human Workflows Using Materialized Views
(see http://www-1.ibm.com/support/docview.wss?rs=2307&uid=swg27009623 )
WPS v6.02 SCA Improvements
MQ and MQJMS Bindings
Direct access to MQ queues from SCA based imports and exports
MQ bindings use MQ APIs to access MQ Series hosted queues
MQJMS bindings use the version of the JMS API supported by MQ
Series to access MQ Series hosted queues Series to access MQ Series hosted queues
2 to 2.2 times faster than MQ Link based bindings
MQ Link Benchmark Topology
JMS
Consumer
JMS
Producer
MQ Queue Manager
JMS
Consumer
JMS
Producer
MQ Queue Manager
JMS
Queue
JMS
Import
Mediation
JMS
Export
JMS
Queue
JMS
Queue
JMS
Queue
JMS
Import
JMS
Import
Mediation Mediation
JMS
Export
JMS
Export
JMS
Queue
JMS
Queue
DB2 DB2
MQ J MS Benchmark Topology
MQ Queue Manager
JMS
Consumer
JMS
Producer
MQ Queue Manager
JMS
Consumer
JMS
Producer
MQ JMS
Import
Mediation
MQ JMS
Export
MQ JMS
Import
MQ JMS
Import
Mediation Mediation
MQ JMS
Export
MQ JMS
Export po t po t po t
WPS v6.02 Performance Enhancements
Web Services Bindings Web Services Bindings
Local Web Services Optimization
If the target of the Web Services import binding is hosted locally in the same
application server the performance can be further improved by exploiting the application server, the performance can be further improved by exploiting the
optimized communication path provided by the Web container. Normally
requests from the Web Services clients are sent through the network
connection between the client and the service provider. For local Web
Services calls, however, WAS offers a direct communication channel , ,
bypassing the network layer completely.
Steps to enable
Set Web Container custom propery enableInProcessConnections to true
Do not use a wildcard for the host name of the Web Container port Use the hostname or IP Do not use a wildcard for the host name of the Web Container port. Use the hostname or IP
address instead
Use localhost instead of host name in the Web Services client binding
Make sure there is not an entry for your server hostname and IP address in your servers
hosts file for name resolution
Contact Manager on Windows
Local versus Remote Bindings Local versus Remote Bindings
Messages
Local synchronous SCA bindings are
3 1 ti f t th t
ContactManager - Windows 2000
Local versus Remote Bindings
3.1 times faster than remote
synchronous SCA bindings
Optimized ocal Web Services bindings
are faster than remote Web Services
bindings
Throughput Results
Local versus Remote Bindings
100%
200
250
Throughput Results
198 BTPS local SCA binding
~64 BTPS remote SCA bindings
110 BTPS local optimized Web
Services
100%
100%
100%
100
150
B
T
P
S
1 WPS JVM (WS
Opt)
1 WPS JVM
Services
96 BTPS local Web Services (no
optimization)
88 BTPS remote Web Services
bindings
Benchmark Details
100%
0
50
2 WPS JVMs
Benchmark Details
Windows Host System
1 CPUs @ 2.8 GHz
1
SCA Sync WebServices
Copyright IBM Corporation 2005, 2008. All rights reserved WPS System CPU Utilization Above Each Bar
J MS Binding Improvements since v6.01
Invocation Hop Elimination
JMS Import intra-module invocation hop elimination
JMS Export intra-module invocation of downstream component hop
elimination
SCA J MS Binding One-way
WPS 6 0 1 WPS 6.0.1
M1
SCA
M2
MDB
ServiceSIBusMessageBean
SCA
Runtime
SCA Runtime
MDB
Export.E1EIS
MDB
ServiceSIBusMessageBean
J2C Adapter
sca/M1/ActivationSpec
J2C Adapter
(ActivationSpec)
J2C Adapter
sca/M/ActivationSpec
g
sca/M1/import/I1* sca/M1
Forward
Routing
Path
SIBus
SCA.SYSTEM.<cellname>.Bus
JMS Queue
sca/M
Forward
Routing
Path
sca/M/export/E1*
SIBus
SCA.SYSTEM.<cellname>.Bus
Hop Reduction: SCA J MS Binding One-way
WPS 6 0 1 1 WPS 6.0.1.1
M1
SCA
M2
SCA
Runtime
SCA Runtime
MDB
Export.E1EIS
J2C Adapter
(ActivationSpec)
JMS Queue
WESB v6.02 Enhancements
Deferred Parsing
Deferred Parsing enables WESB to avoid SDO inflation
(deserializing) in cases where it is not required ( g) q
Assuming incoming SDO is in a serial form that SCA understands
No benefit if you convert from an incoming custom format to SDO then serialize
using SCA serialization functionality. If you get to a fully de-serialized SDO, pass
the de-serialized SDO to the mediation module using synchronous bindings
Benefits flows where the primitives do not explicitly or implicitly
access the data via the SDO API
Primitives that benefit from deferred parsing
Route on header
XSLT Transformation
Endpoint Lookup without Xpath user properties
Primitives that require full parsing q p g
Custom mediation
Route on body
Any mediation with an Xpath expression against the message body
RPC Encoded WSDL
RPC Encoded WSDL
Note: Deferred parsing is achieved by using WID generated WSDL
WBM v6.1 Performance Improvements Highlights
Significant throughput improvements in WPS v6.1 illustrated by the core workload
benchmarks
SOABench automated mode improvements up to 132% and SOABench manual mode improvements up to 82%,
relative to WPS v6.0.2
Banking throughput improved by 33% relative to WPS v6 0 2 and 3 9 times higher than WPS 6 0 0 0 Banking throughput improved by 33% relative to WPS v6.0.2 and 3.9 times higher than WPS 6.0.0.0
Contact Manager throughput improved by up to 69% relative to WPS v6.0.2 and 6.2 times higher than WPS
v6.0.0.0
WPS v6.1 SMP scaling data that demonstrates outstanding vertical scaling; SOABench
automated mode achieved 7 1x scaling on an 8-way AIX system and 13 8x scaling on a 16- automated mode achieved 7.1x scaling on an 8-way AIX system and 13.8x scaling on a 16-
way AIX system
WPS v6.1 clustering data that shows near-linear horizontal scaling with up to 8 nodes;
Contact Manager achieved 7.3x scaling on an 8 node topology
WESB v6.1 using Web Services bindings delivers throughput improvements from 94% to
258% relative to WESB v6.0.2
WESB v6.1 using JMS bindings delivers throughput improvements from 72% to 198%
relative to WPS v6 0 2 relative to WPS v6.0.2
WESB v6.1 clustering data shows linear horizontal scaling up to 4 nodes; the Transform
Value workload delivered 4x scaling on a 4 node topology
WBM v6.1 Performance Improvements Highlights
WPS and WESB v6.1 dramatically improved large message/object capability
across all scenarios; WESB achieved up to an 8 times improvement and WPS
achieved up to a 5 times improvement
Performance data for i5/OS is introduced in this report.
WESB v6.1 demonstrates the ability to process up to 1,400 messages per second
WPS v6.1 delivers throughput up to 418 transactions per second with the SOABench automated
mode benchmark on a 4-way system
WebSphere Adapters
The WA SAP Adapter achieves a 2 times improvement in throughput over v6.0.2 when using the
BAPI module for request processing
The WA SAP Adapter achieves a 45% improvement over v6.0.2 when using the ALE module for The WA SAP Adapter achieves a 45% improvement over v6.0.2 when using the ALE module for
event delivery
The WA Flat File Adapter achieves a 2 times improvement in throughput over v6.0.2 for event
delivery.
The WA Flat File Adapter was able to successfully process a 32 GB file when using SplitBySize
capability capability
WebSphere Business Monitor
WBM v6.1 achieved almost a 2 times improvement in throughput over v6.0.2
WBM v6.1 achieved a event processing rate of almost 2,000 events per second while sustaining
WBM v6.1 achieved a event processing rate of almost 2,000 events per second while sustaining
100,000 active process instances
WID Performance Improvements
Significant improvement in authoring productivity
since WID V6.0.2.0
Build response time reduced by 65%
Build memory usage reduced by 60%
Publish response time reduced by 40%
Significantly reduced memory leaks in WID g y y
Key Enhancements in WPS v6.1
Business Process Engine
G l WPS E h t General WPS Enhancements
WPS v6.1 Performance Enhancements
Business Process Engine Business Process Engine
Key BPE Enhancements
New High Speed Business Process Navigation Engine for Long Running g p g g g g
Processes
Previous engine relied on JMS messages
Improved engine leverages WAS async beans
Improved engine also caches data and leverages server affinity to associate to Improved engine also caches data and leverages server affinity to associate to
the server that holds the cache
Data compression during serialization
Significantly reduces the size of XML business data stored and retrieved from the g y
BPC database
Materialized views for Task List Queries
Introduced in WPS v6.021
WPS v6.1 Key Performance Enhancements
Business Process Engine (BPE) Business Process Engine (BPE)
Performance of the Business Process Engine (BPE) is significantly improved in WPS v6.1.0.1
14
Business Process Performance
8
10
12
6 1 (hi speed)
2
4
6
SCA async
EJB
Web service
JMS
POJO
6.0.2
6.1 (classic)
6.1 (hi-speed)
0
2
General Enhancements to the Common Runtime General Enhancements to the Common Runtime
Exploitation of WAS v6.1
Usage of Java 5
Improvements in Synchronous JCA performance p y p
58% improvement in WPS v6.1 over WPS v6.02 for the ContactManager benchmark
Improvements in Synchronous SCA performance
Over 60% improvement in WPS v6.1 over WPS v6.02 for the ContactManager
benchmark
SMP Scaling Optimizations
Illustrated by Travel Booking 8-way results
SIBus File Store
Use the file system instead of a database to hold messages Use the file system instead of a database to hold messages
Configuration used for the Banking benchmark results in the WPS v6.1 Performance
Report
Memory and Path Length Optimization
Reduce memory usage to support larger objects and messages Reduce memory usage to support larger objects and messages
Additional BPC and SCA path length reductions
WID Enhancements WID, Enhancements
Utilized shared resource management system for all of the
SCA component builders and validators p
Enhancements to the WID Indexer and Resolver
Caching of widgets used by the WID Indexer
Reduced number of file copies that the Web Services Reduced number of file copies that the Web Services
deployment builder performed
Leverage Java 5 and the GENCON garbage collector
WID v6.02 used Java 1.4.2 J9
WESB v6.1 Performance Enhancements
Native Form Re-use
Incoming native data streams (non-SCA streams) will be re-used if
the associated mediation does not perform data or object
modifications
Applies to routing only mediations pp g y
Not applicable to custom mediations even if the custom mediation
does not modify the data/object
WebSphere Business Monitor v6.1
Monitor will re-order CBE that arrive out of order so more binding options with
Monitor are enabled
Improvements in the data representation of BPEL headers and application data in p p pp
CBEs
Support for previous formats preserved
Over 2 times improvement using this new data representation
New multi-processor mode called Serial Multi-Threaded (SerialMT) that achieves
around an improvement based on the number of processors beyond 1 when
compared with the previous SerialST implementation
Support for previous operation preserved
SerialMT is heavier per transaction so SerialST will still be faster on single processor host systems
Remote Model Logic removes CPU contention between the model logic function
and the moderator function when the two functions are run on different host
systems systems
Required Events Only improves BPEL throughput
Monitor throughput is reduced because of additional load on WPM
Areas of Investigation in SOA Performance
Remaining Pain Points
The cost of dynamicity
Dynamic invocations (SCA) y ( )
Dynamic data (SDO)
Process Engine (BPEL)
More specifically p y
Transformations to/from wire format (XML e.g.) to object format (SDO)
Interpreted BPEL engine navigation
BPEL engine state management for long-running flows g g g g
Side effects of programming model
Pass by value across SCA modules
Summary
Recent WPS Performance Improvement Themes
Key Performance Enhancements in Previous WPS/WESB Versions
B i P E i (BPE) Business Process Engine (BPE)
SCA Performance Improvements
Overall WPS Code Path length
WESB S ifi P f I t WESB Specific Performance Improvements
WBM v6.1 Performance Improvement Highlights
Key Performance Enhancements in WPS/WESB v6.1
Business Process Engine (BPE)
General WPS Runtime
Areas of Investigation
Questions
their session evaluation form
Thank you!!
Credits
Gary Hunt - gthunt@us.ibm.com
Hindi
Thai
Gracias
Ob i d
Russian Spanish
Traditional Chinese
Thank You
Obrigado
D k
Brazilian Portuguese
Arabic
Merci
Grazie
Dank
e
French
German
Italian
Simplified Chinese
Tamil
Korean
Japanese
Tuning WebSphere App Server with What
Tuning WebSphere App Server with What
You Already Have
You Already Have
Session Number: D03
Session Number: D03
Kevin Tobin Kevin Tobin kevintob@au1.ibm.com kevintob@au1.ibm.com
2
Objectives
Objectives
To demonstrate you do not need high-end,
expensive tools to adequately tune
WebSphere.
Learn how to make the WebSphere Application Server perform to its
potential without additional IT budget for tools.
Learn about different performance testing scenarios including
performance data collection and analysis.
Learn about common causes of performance issues.
Learn WebSphere tuning and service tips.
Learn issues that can be caused due to load distribution, caching and
different infrastructures.
3
Agenda
Agenda
Intro to tools
OS tools, Application tools, Free Tools.
Scenarios
Performance testing scenarios, collecting and analysing data, common
causes of performance issues.
Tuning
JVM memory tuning, Connect pool tuning,
DB problem analysis.
Concerns
Load distribution, Caching, Infrastructure.
4
Intro to tools
Intro to tools
Intro to tools
OS tools, Application tools, Free Tools
Scenarios
Performance testing scenarios, collecting and analysing data, common
causes
Tuning
JVM memory tuning, Connect pool tuning, DB problem analysis
Concerns
Load distribution, Caching concerns, Infrastructure concerns.
5
Tools
Tools
Operating System
Operating System
Network monitoring
ipconfig / netstat / iostat / Management Console
snoop / iptrace / tcpdump / nettl / Network Monitor
Process monitoring
top / ps / Task Manager
free / vmstat / Management Console
Disk Monitoring
df / Disk Manager
6
Tools
Tools
JVM Analysis
JVM Analysis
PMAT (Pattern Modeling Analysis Tool)
IBM HeapAnalyzer / HAT (Heap Analysis Tool)
IBM Thread and Monitor Dump Analyzer
IBM Request and Trace Analyzer
7
JVM
JVM
-
-
PMAT
PMAT
Pattern Modeling Analysis Tool developed by
Jinwoo Hwang from WAS technical support.
Features:
GC analysis
GC table view
Allocation failure summary
GC usage summary
GC duration summary
GC graph view
GC pattern analysis
Recommends key configurations based on pattern modeling of Java heap usage
Download http://www.alphaworks.ibm.com/tech/pmat
8
Sample Verbose GC Output
Sample Verbose GC Output
AF[1]: Allocation Failure. need 1544 bytes, 0 ms since last AF>
<AF[1]: managing allocation failure, action=1 (0/3983128) (209640/209640)>
<GC(1): GC cycle started Tue Apr 24 10:49:58 2001>
<GC(1): freed 1421672 bytes, 38% free (1631312/4192768), in 9 ms>
<GC(1): mark: 8 ms, sweep: 1 ms, compact: 0 ms>
<GC(1): refs: soft 0 (age >= 32), weak 17, final 16, phantom 0>
<AF[1]: completed in 10 ms>
9
JVM
JVM
PMAT
PMAT
10
JVM
JVM
HeapAnalyzer
HeapAnalyzer (1) (1)
HeapAnalyzer provides:
List of Java heap leak suspects.
Recommendation of the size of kCluster.
List of gaps among allocated objects/classes/arrays.
Java objects/classes/arrays search engine.
List of objects/classes/arrays by:
Type name
Object name
Address
Size ( including children )
Frequency
List of available heap spaces by size.
Tree view of Java heap dump.
Supports persistence of processed heap dumps.
JRE 1.3.1, or above ( including IBM & SUN).
11
JVM
JVM
HeapAnalyzer
12
JVM
JVM
HeapAnalyzer
13
JVM
JVM
HeapAnalyzer
14
JVM
JVM
HeapAnalyzer
15
JVM
JVM
Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer (1) (1)
Java Core Dump Analyzer V2.5 (WAS 5.1 & up) JRE 5
Identifies hang suspects ( HP, SOLARIS, IBM 1.3.1 6.0 JVMs)
Can report on multiple dump files.
Analyzes each thread and provides diagnostic information, such as:
Current thread information,
The signal that caused the javacore
Maximum Java heap size
Initial Java heap size
Garbage collector & Allocation failure counter
Free and allocated Java heap size
Number of runnable threads, total number of threads, number of monitors
locked, and deadlock information.
Compares all monitor information in javacore and detects deadlock and
resource contention or monitors bottlenecks, if there are any.
16
JVM
JVM

Thread States
17
JVM
JVM

18
JVM
JVM

19
JVM
JVM

20
JVM
JVM

21
JVM
JVM
-
-
Trace and Request Analyzer (TRA)
This tool processes WebSphere Application Server JVM trace files
Allows you to find delays and possible hangs in the application server.
The tool reads HTTP plug-in trace files.
User definable thresholds.
The tool parses method call tree, time stamp, process ID, thread ID,
trace entry, trace exit, method entry and method exit information.
Parsing information is customizable.
TRA
Calculates actual times spent in each method and trace.
Finds delays longer than the threshold.
Sorts the delay list by delay time.
Displays a list of clocked trace entries.
22
JVM
JVM
-
-
Trace method analysis
23
JVM
JVM
-
-
HTTP request analysis
24
JVM
JVM
-
-
Trace gap analysis
25
Other analysis tools:
WireShark
Open source, network monitoring /packet sniffing
ktmatu Relax
Webalizer Xtended
AWStats
Network & HTTP Log Tools
Network & HTTP Log Tools
26
WireShark
WireShark
Popular network protocol analyzer
Live capture and offline analysis.
Standard three-pane packet browser.
Multi-platform: Runs on Windows, Linux, OS X, Solaris,and many
others.
Captured network data can be browsed.
Read/write many different capture file formats.
Capture files compressed with gzip.
Live data can be read from Ethernet, IEEE 802.11, PPP/HDLC, ATM,
Bluetooth, USB, Token Ring, Frame Relay, FDDI, and others.
Decryption support for many protocols, including IPsec, ISAKMP,
Kerberos, SNMPv3, SSL/TLS, WEP, and WPA/WPA2.
Output can be exported to XML, PostScript, CSV, or plain text.
Download from http://www.wireshark.org/
27
WireShark
WireShark
configuration
configuration
28
WireShark
WireShark
Capture Options
Capture Options
29
ktmatu
ktmatu
Relax
Relax
Relax is a free specialized web server
log analysis tool for referrer information
processing written in PERL. http://ktmatu.com/software/relax/
It answers the question: "Which search engines, are
directing traffic to this site?".
Relax reports the number of users you are getting from the
leading search engines.
Relax generates lists of URLs directing to your pages with
a count of the number of times each referring URL was
detected.
30
WEBALIZER XTENDED
WEBALIZER XTENDED
HTTP logs analyzer
Statistics for all hits, number of requests, percentage requests and "HTTP
404 Not Found - error URL's.
Automatic sorting and incremental or full log file processing.
Additional response code pie chart.
Fully user definable colors for all statistics.
User definable table and chart borders (3D-effects).
HTTP response codes with additional percentage hits.
31
WEBALIZER XTENDED
WEBALIZER XTENDED
Example output
32
WEBALIZER XTENDED
WEBALIZER XTENDED
Example output
33
AWSTATS
AWSTATS
AWStats is a feature rich,
HTTP log analyzer.
AWStats generates web, streaming, ftp or mail server
statistics, graphically.
The log analyzer works as a CGI or from command line and
shows you all possible information your log contains, in few
graphical web pages.
34
AWSTATS
AWSTATS
Number of visits, and number of unique visitors.
Visit durations and last visits.
Authenticated users, and last authenticated visits.
Days of week and rush hours:
pages,
hits,
KB for each hour and day of week.
Hosts list, last visits and unresolved IP addresses list.
Most viewed, entry and exit pages.
Files types.
Web compression statistics (for mod_gzip or mod_deflate).
35
AWSTATS ( continued )
OS used (pages, hits, KB for each OS).
Browsers used (pages, hits, KB by browser, version).
Visits of robots.
Worms attacks.
HTTP errors (e.g. Page Not Found with last referrer).
Other personalized reports based on URL.
Number of times a site is added to favorites.
36
37
Apache JMeter
Apache JMeter
Popular load test performance tester
Open source Java desktop application.
Originally developed by Stefano Mazzocchi.
Designed for functional/load/performance/stress testing.
Extensible write your own test component.
Simulate heavy load (application, server and network).
Gives instant visual feedback.
Distributed testing support.
Various protocols - HTTP, FTP, JDBC, JMS, LDAP, SOAP.
Multi-platform.
Full multi-threading framework.
Caching and offline analysis/replaying of test results.
Download from http://jakarta.apache.org/jmeter/
38
Apache JMeter
Apache JMeter
Application Overview
GUI, command line interface.
Tests can be run and analyzed interactively, or run in batch mode and
analyzed offline.
Test plans consist of:
Thread groups: organize threads of execution.
Samplers: sends requests to a server.
Logical controllers : control test plan flow (loops, conditionals,ordering, etc.).
Listeners: record, summarize and display record and response data.
Timers: introduce delays in test plan.
Assertions: assert facts about responses, for functional testing.
Configuration elements.
Pre-processors and post-processors.
39
Apache JMeter
Apache JMeter
Testing Process
40
Apache JMeter
Apache JMeter
Test Plan Elements

Test Plan Elements
41
Scenarios
Scenarios
Intro to tools
OS tools, Application tools, Free Tools.
Scenarios
Performance testing scenarios, collecting and analysing data,
common causes.
Tuning
JVM memory tuning, Connect pool tuning, DB problem analysis.
Concerns
42
Scenarios
Scenarios
Scenarios
What you should know before you start.
Performance testing scenarios.
Collecting performance test data.
Analysis of performance test data.
Common causes of performance issues.
43
Scenarios
Scenarios
Pre
Pre
-
-
knowledge
knowledge
What are the business policies?
Security policies
Access restrictions?
Sniffer policy?
Privacy policy?
Change control
Who has access?
Lead times?
Exceptions?
Can you deploy investigation tools?
Monitoring
What is being monitored?
How are issues responded to?
44
Scenarios
Scenarios
-
-
Performance testing
Performance testing
Load generation tools can assist (JMeter).
Remember to monitor:
Network and Network Interfaces.
Application Interfaces.
Database.
Requires scheduled time and special data on a production system
Use the same mix of transactions as seen in production.
45
Scenarios
Scenarios
-
-
Performance testing (cont)
Performance testing (cont)
Soak testing (JMeter)
Good for finding leaks (PMAT).
Detecting external system events
that potentially impact performance.
Code path testing (JMeter)
Targets a particular operation in isolation.
Can find operational optimizations.
Remember to monitor
The network,
Application interfaces,
Database
Should only be done in a test environment.
46
Scenarios
Scenarios
-
-
Collecting data
Collecting data
Logs
Ensure logging is appropriate and without errors.
Tracing
Be aware of any tracing enabled during
performance tests.
Dumps (thread and heap)
CPU stays low even with heavy loading.
Tool such as heap and thread analyzers help.
May not be permitted on production systems.
47
Scenarios
Scenarios
-
-
Collecting data (cont)
Collecting data (cont)
Monitoring
Is monitoring enabled?
What can be derived from
what is being monitored?
Configuration
Backup the current system configuration before any testing.
Look for abnormal settings (heap, pools, cache, etc.)
Time
Ensure time is synchronised on all systems.
Manually check even is NTP is in use.
Ensure you know the exact time of tests.
48
Scenarios
Scenarios
-
-
Analysis of data
Analysis of data
Memory Analysis
Log Analysis
Heap / Thread Analysis
49
Scenarios
Scenarios
-
-
Memory Analysis
Memory Analysis
What type of JVM is in use?
Java 1.4 (Flat( IBM) or Generational (Sun))
Java 1.5 (Generational)
Sun HotSpot (Generational +)
Analyze VerboseGC (PMAT)
How often is a compact being done?
How often is a sweep being done?
Is there a lot of pinned and dosed objects?
What is the steady state heap size after startup?
Add flag to remove explicit garbage collects
Explicit GC always performs a compact.
Java flag (-XX:+DisableExplicitGC)
50
Scenarios
Scenarios
-
-
Log Analysis
Log Analysis
What logs are available?
SystemOut.log
Are there warnings or errors?
SystemErr.log
Should be small. What errors are being reported?.
System_stdout.log and System_stderr.log
Should be empty
trace.log (is this production?)
activity.log
Requires LogAnalyzer to read
Is it being monitored (at all)?
FFDC
Number?
Cleanup?
51
Scenarios
Scenarios
-
-
Heap Analysis
Limited value for WAS tuning.
Look for p & k objects outside the designated clusters.
Thread Analysis
Look for multiple threads waiting for a single resource.
Look for large numbers of idle threads.
52
Scenarios
Scenarios
Common issues
Common issues
Network
Too narrow a pipe (ifconfig / iostat)
Intervening devices like firewalls ( nmap )
53
Scenarios
Scenarios
Common issues (cont.)

Database
Connection pool
Check using TPV
Poorly constructed queries
DB2 analyser
Non-indexed data
Cleanup not being performed
Runstats
Connection Pool
Too large (Server swamped) TPV
Too small (Server starved) TPV
54
Scenarios
Scenarios

Memory
Initial heap too low (slow startup) (-verbosgec / PMAT)
Max heap too large (long gcs) (-verbosgec / PMAT)
Keep-alive
Too short (multiple re-connects from same client)
Too long (reduces max system throughput)
DynaCache
Check if its being used and enquire if not.
Not warmed on startup (CacheMonitor)
Too small (CacheMonitor)
Invalidations (all, timed)
Prepared Statement Cache
Not tuned (TPV)
55
Tuning
Tuning
Intro to tools
Scenarios
common causes
Tuning
Concerns
56
Tuning
Tuning
Tuning
JVM memory tuning
Connection pool tuning
DB communication problem analysis
57
Tuning
Tuning
-
-
JVM memory
JVM memory
Requires a JVM memory tool
-verboseGC
Tivoli Performance Viewer
Other PMI tool, (ITCAM , etc..)
PMAT
HeapAnalyzer
What to look for:
Long garbage collects (max heap too large).
Memory leaks (available heap shrinks).
Frequent garbage compacts (heap too small).
Explicit garbage collects (should be discouraged).
Initial startup size (guide for initial heap size).
Average heap size after mark-sweep (guide for max heap).
k and p cluster sizes (tune iteratively).
58
Tuning
Tuning
-
-
Connection pools
Connection pools
Tools required
TPV
Application logging
Netstat
Web Container pools
Number of concurrent requests.
Average request response time.
Average data volume returned.
EJB Container pools
Number of concurrent operations.
Average operation response time.
Average data volume communicated.
DB Connections (covered next slide)
59
Tuning
Tuning
-
-
DB communications
DB communications
Tools required
TPV
SQL logs
netstat
Required Information
Expected peak operations (guide to max connections)
Expected ramp-up and ramp-down (guide to reap time)
Set min connections to zero
Firewalls !!!
OK for slow ramp-up
Only set non-zero in exceptional circumstances
SQL optimization (long requests, indexing)
60
Concerns
Concerns
Intro to tools
Scenarios
common causes
Tuning
Concerns
61
Concerns
Concerns
Concerns
Load distribution
Infrastructure concerns
Caching concerns
62
Concerns
Concerns
-
-
Load distribution
Load distribution
Active-active
Active-passive
Cell topology - covered next
63
Concerns
Concerns
-
-
Cell Topology - Gold Standard?
Multiple Data Centres
Wide Area Communication
Backup and Recovery
64
Concerns
Concerns
-
-
Caching
Caching
Startup
Size in memory
Disk offload speed
Invalidation
Cache Distribution
65
References
References
Sources
66
Summary
Summary
We discussed existing and free tools that can
be used to help tune WebSphere.
We discussed different performance testing
scenarios.
We discussed common causes of
performance issues.
We discussed WebSphere tuning and service
tips.
We discussed possible issues caused by
load distribution, caching and infrastructures.
You do not need high-end, expensive
tools to adequately tune
WebSphere.
67
Questions
Questions
Please complete your session evaluation!
68
Weak/Soft references
Weak/Soft references
Weak references
A weak reference, simply put, is a reference that isn't strong enough to force an object to remain in memory. Weak references allow you to leverage the
garbage collector's ability to determine reachability for you, so you don't have to do it yourself. You create a weak reference like this:
WeakReference weakWidget = new WeakReference(widget); and then elsewhere in the code you can use weakWidget.get() to get the actual Widget
object. Of course the weak reference isn't strong enough to prevent garbage collection, so you may find (if there are no strong references to the widget)
that weakWidget.get() suddenly starts returning null.
To solve the "widget serial number" problem above, the easiest thing to do is use the built-in WeakHashMap class. WeakHashMap works exactly like
HashMap, except that the keys (not the values!) are referred to using weak references. If a WeakHashMap key becomes garbage, its entry is removed
automatically. This avoids the pitfalls I described and requires no changes other than the switch from HashMap to a WeakHashMap. If you're following
the standard convention of referring to your maps via the Map interface, no other code needs to even be aware of the change.
Reference queues
Once a WeakReference starts returning null, the object it pointed to has become garbage and the WeakReference object is pretty much useless. This
generally means that some sort of cleanup is required; WeakHashMap, for example, has to remove such defunct entries to avoid holding onto an ever-
increasing number of dead WeakReferences.
The ReferenceQueue class makes it easy to keep track of dead references. If you pass a ReferenceQueue into a weak reference's constructor, the
reference object will be automatically inserted into the reference queue when the object to which it pointed becomes garbage. You can then, at some
regular interval, process the ReferenceQueue and perform whatever cleanup is needed for dead references.
Different degrees of weakness
Up to this point I've just been referring to "weak references", but there are actually four different degrees of reference strength: strong, soft, weak, and
phantom, in order from strongest to weakest. We've already discussed strong and weak references, so let's take a look at the other two.
Soft references
A soft reference is exactly like a weak reference, except that it is less eager to throw away the object to which it refers. An object which is only weakly
reachable (the strongest references to it are WeakReferences) will be discarded at the next garbage collection cycle, but an object which is softly
reachable will generally stick around for a while.
SoftReferences aren't required to behave any differently than WeakReferences, but in practice softly reachable objects are generally retained as long
as memory is in plentiful supply. This makes them an excellent foundation for a cache, such as the image cache described above, since you can let the
garbage collector worry about both how reachable the objects are (a strongly reachable object will never be removed from the cache) and how badly it
needs the memory they are consuming.
Phantom references
A phantom reference is quite different than either SoftReference or WeakReference. Its grip on its object is so tenuous that you can't even retrieve the
object -- its get() method always returns null. The only use for such a reference is keeping track of when it gets enqueued into a ReferenceQueue, as at
that point you know the object to which it pointed is dead. How is that different from WeakReference, though?
The difference is in exactly when the enqueuing happens. WeakReferences are enqueued as soon as the object to which they point becomes weakly
reachable. This is before finalization or garbage collection has actually happened; in theory the object could even be "resurrected" by an unorthodox
finalize() method, but the WeakReference would remain dead. PhantomReferences are enqueued only when the object is physically removed from
memory, and the get() method always returns null specifically to prevent you from being able to "resurrect" an almost-dead object.
What good are PhantomReferences? I'm only aware of two serious cases for them: first, they allow you to determine exactly when an object was
removed from memory. They are in fact the only way to determine that. This isn't generally that useful, but might come in handy in certain very specific
circumstances like manipulating large images: if you know for sure that an image should be garbage collected, you can wait until it actually is before
attempting to load the next image, and therefore make the dreaded OutOfMemoryError less likely.
Yogesh Bhatia
SDP Introduction and Architecture
Page 1
AGENDA
SDP INTRODUCTION
CURRENT MARKET REQUIREMENTS
SDP RELEVANCE
IBM COMPONENTS
IBM SOLUTION
ARCHITECTURE
COMPONENTS DISCUSSION
Page 2
Motivation
Subscribes: 2M to 100M ARPU: $30 to $7
Page 3
The Opportunity
Page 4
Service Delivery Platform - Introduction
In a compettve envronment, operators must be abIe to do the IoIIowng:
1. Create, DepIoy, and DeIver servces.
2. IncIudng servces Irom thrd-party content provders.
3. Respond to marRet qucRIy and eIIcentIy.
4. Ease oI managng and extendng these servces
The most basc deInton oI an SDP s:
A common architecture for the creation and delivery of services across multiple
network types and service enabling platforms
Ths new servces modeI has drven the concept oI the servce deIvery pIatIorm, but no common,
standardzed deInton oI an SDP.
There s IttIe common agreement wthn the teIecom ndustry on how to deIne an SDP;
The deInton vares wdeIy dependng on whom you are speaRng wth and how they are
addressng the marRet.
Page 5
Market Context Service Creation
Current
Process
Months 2-3
Months
6-8 Months 12-14
Months
Commercial
Launch
Soft Launch Trials
Proof of
concept
Ideation
18 24
Months
Today
As Is
Service
1
8

m
o
n
t
h
s

t
o

2

y
e
a
r
s
Ideas source:
Internal
idea
idea
idea
Long testing
and
integration
period due to
proprietary
solutions
Little user
interaction for
market
feedback
Services
developed in
silos
Little reuse
Leads to delays
in launch time
Page 6
Market Context Service Creation
Innovation
Process
Hours/Days Days Days/Weeks Weeks
Commercial
Launch
Soft Launch Trials
Proof of
concept
Ideation
Month
Service
Ideas source:
Internal, partners and customers
W
e
e
k
s

t
o

m
o
n
t
h
s
Breadth
Tomorrow To Be
idea
idea
idea
idea
idea
idea
idea
idea
idea
idea
idea
idea
idea
Modelling and Prototype
designed for iterative
development
2
Use of exposed common
capabilities to enable rapid
service assembly
3
Ideation and Product
Inception form the expanding
source for innovation
Use Collaborative tools to
solicit interactive feedback.
1
Service Service
Page 7
Rapid Service Creation depends on the common building
blocks available in order to create new services quickly
Service
enabler
Service
enabler
Our Service Innovation proposition provides
operators with the ability to productionise
service assembly and deployment
Telecom services can be created from common building blocks a similar concept
to that used in manufacturing
Service
enabler
Service
enabler
Service
enabler
Service
enabler
C
o
m
m
o
n

C
a
p
a
b
i
l
i
t
i
e
s

P
a
l
l
e
t
Service
enabler
Service
enabler
Service 1
Service
enabler
Service 2
Service
enabler
Service 3
Service
enabler
Service
enabler
Service
enabler
Service 4
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Service
enabler
Page 8
Convergence is driving TSPs towards service based models
To attract and serve an expanded customer base, Service Providers are becoming more
customer-centric and moving to services that target the long tail.
Telecom
Biz Model
Web 2.0
Model
C
e
n
t
r
a
l
i
z
e
d
C
u
s
t
o
m
e
r

d
e
f
i
n
e
d
F
e
w
e
r
,

s
l
o
w
e
r
M
o
r
e
,

f
a
s
t
e
r
Technology
Focused
Customer
Focused
Network-centric Marketing-centric
Network Investments Business Investments
Rapid Market Response
Any service at anytime
Customer self-management
Broad partner ecosystem
Tomorrow
Today
Situational Applications
1. Industry Specific
2. Ad hoc group specific
Situational Applications
1. Cross Industry
2. Cross Community
Legacy
Network
Services
New Innovations / New Services / New Markets
C
a
p
e
x

,
O
p
e
x

a
n
d

T
i
m
e

t
o

M
a
r
k
e
t
Number of New
Products
Services
& Markets
Legacy
Telecom
Partner
Ecosystem
Customer
Participation
Reduced Cost
Effort & Time
Voice
SMS
Voice Mail
Subscription
Billing
Location
Services
Mobile
Video
MVNO
Content
Services
Virtual
Worlds
Community
User
Shared
Content
Blogs
Wikis
Forums
Tagging
Web 2.0
Mashups
Video
Broadcasts
This shift to a customer-centric model requires the adoption
of new sets of customer-driven data sources, tools and
business processes for a more thorough understanding of
customer behavior.
Services will become more targeted, focused, personalized, and
based on reusable components in many combinations. Service
Providers need to re-invent how new services are created, launched
and supported.
Page 9
What SDP provides
A SDP (Service Delivery Platform) solution is a service
implementation layer across a diverse set of capabilities
This layer allows Marketing, Network, and IT to create and implement new
services faster, more efficiently and with lesser cost
Provides a unified service creation and control environment where integrated
value added services can be defined, deployed and controlled
Service
Creation
Order
Capture
Provisioning
Activation
Service
Delivery
Billing
Service
Retirement
SDP
SDP architecture provides an end-to-end control and enables
Selection of a Product or a Service
Placement of an Order
Provisioning of a Service or a Product
Activation of a Service or a Product
Delivering a Service or a Product
Billing for a Service or a Product
Disconnecting a Service or a Product
Page 10
SDP Context
Page 11
Page 12
Deploying Value Added Service (with SDP)
Page 13
SDP Services in Scope
Content Gateway
Portal Management System
Content Management System
Digital Rights Management
Transcoding
Directory Services
Messaging Gateway
A2P and P2A messaging for SMS, MMS
Unified API(s) for applications such as televoting
Subscription Engine for contents such as alerts, joke of the
day, horoscope, etc
Bulk Messaging capabilities for promotion, campaigns
Application Gateway
Enterprise Framework
3rd Party Framework
Service Creation Environment
Service Management
Service Creation Gateway
Open access for various applications
Enterprise Framework
Unified API(s) - Parlay compliant for developing
various applications such as Virtual PBX
Consumer - User Domain
Mobility Users
Infotel Users
Business User Domain
Content Providers
Application Providers
Developer Community
1
2
4
5
3
Page 14
IBM SDP Architecture - Solutions To given Problem Statements
Portal Solution (IBM WebSphere Portal Server + WebSphere Everyplace Mobile
Portal))
Broadband Portal Solution - (IBM WebSphere Portal Server)
Web services Access Gateway Solution (IBM Telecom Web Services Server)
Process Server and Workflow Engine Solution (IBM WebSphere Process Server)
ITCAM Solution for System Management & Performance Monitoring (IBM Tivoli
Composite Application Manager)
Content Gateway Solution (MPower by Alcatel/Lucent earlier Mobilitec) +
(CoreMedia DRM) + (Adamind MediaSphire)
Messaging Gateway Solution - (First Hop)
Network Directory Services for SDP Solution (Apertio One NDS)
Real Time Charging Solution (Differential Billing Platform)
Application Gateway Solution (Aepona Causeway)
Campaign Management Solution (CMS + CIT Systems)
SDP Integrated Solution
SDP Integrated NFR
SDP Integrated Deployment Solution
Page 15
SDP Functional Architecture
Page 16
Content Gateway: Problem Statement
Page 17
CMS Introduction
Centralized Content Management
Common platform, for content ingestion,
categorization, publishing. (Flexible workflow based)
Content Partner Management (Flexible workflow
based)
Content Rating
Content Transcoding
Content Protection
Device & Media Management
Merchandising
Dynamic segmentation and profiling
Content Promotions and Notifications
Recommendations
Personalized discovery and reliable delivery (Device, Locale,
LOB,BU etc)
Subscription Management
Subscriber Management & Customer Care
Page 18
Content Gateway - Functions
Page 19
Page 20
Portal Architecture Context Diagram
Page 21
High Level Security Architecture of Portal
Page 22
Page 23
Core SDP Engine WPS/WESB
Page 24
The Logical Topolgy Gold Standard
IHS
Plug-in
IHS
DMGR
WebSphere Process Server 3
Support
Cluster
WebSphere Process Server 1 WebSphere Process Server 2
Messaging
Engine
Cluster
Active Messaging Engine
SCA Queues
BPC Queues
Active Messaging Engine
SCA Queues
BPC Queues
CEI
BRM
CEI
BRM
Modules are run in ActiveActive config
Queues in Active-Active Config
This is a node and not a physical machine
WebSphere
Process
Server
Cluster
BPC Container
HTM Container
BPC Container
HTM Container
Page 25
WebSphere Process Server V1
WPS LPAR 1
WPS LPAR 2
WPS LPAR 3
DB2 LPAR 1
WPS LPAR 4
WPS LPAR 5
WPS LPAR 6
DB2 LPAR 2
Web Servers WebSphere MQ FileSystem
Active- Active Cluster
Two instances per LPAR
HA Configuration
HACMP based with shared disk
Page 26
WPS Deployment Diagram
WPS
CEI
AME
WPS
WPS
AME
WPS
WPS
AME
WPS
WPS
WPS
WPS
WPS
WPS
Active Messaging Cluster
Arbor
Mediation Workload Cluster
All Other HTTP and MQ based Workloads Cluster
Each Green Box is a physical LPAR
Each box within the green box is a WAS node
Groups of WAS nodes are marked for function and the cluster is indicated
Summary : 4 Clusters + 1 CEI; Single Cell
Service creation
Subscription Mgmt
Selfcare
ADC
Portal
PACS/BI
Page 27
Page 28
TWSS - Architecture
Enable 3rd Party Framework
The third-party framework allows:
Rapidly creating and
Offering new services
Extends the various levels of functionality such as MB, CMS, DRM, Identity and Access Management, Service Creation and
Composition, Security, etc.
The Third Party Enabling Framework enables:
The interaction with third party applications.
The Third Party framework also:
Authenticates and Authorizes 3rd parties applications and
Provides access to the services offered by Bharti Airtel.
This framework also provide the customize interfaces to the components existing both in IT and network.
TWSS will address A2P Actual Messaging from 3
rd
Party / CP applications Phase - 1
Page 29
TWSS HA
Deployment
(w/ SIP)
Page 30
Overview of TWSS Internal Components
CMS/Third party
Applications
Transaction
Recorder
Policy and
Subscription
Network
statistics
Service
Authorization
SLA
Enforcement
Service
Invocation
SMS Service
Implementation
MMS Service
Implementation
WAP Service
Implementation
MM7 Connector SMPP Connect or
SPM
Partner
Portal
TWSS CORE
Page 31
Page 32
Network Directory Service
Architect, Design, Implement and Deploy Directory for:
Providing one view for Bharti Subscribers for SDP services.
Flexible and extensible schema for future to accommodate other network grade services and functions
Flow through provisioning of Bharti Subscribers for SDP and its service platforms (CMS, MB)
Prepaid & Postpaid
Open access to data
directory protocols such as LDAP
executable software procedures with in NDS architecture
Simplified integration of SDP components:
For read operations over direct LDAP i/f from individual component
For all write operation over Process Server (single point for writes)
Profiles to be supported
Subscriber Partners
Service Device
Packages, Add-ons, Bundles, Plans
References like Subscriber to device Subscriber to language, Subscriber to geo etc..
Visibility of all subscription and service data in the directory to all other hosted applications for query and update on a
controlled and secure basis
Page 33
SDP View of Using NDS As Common Data Repository
NDS
CMS
WPS
First Hop
Portal
WEB/Mobile
Integration
Hub
TWSS
SDP A
SDP D
CRM
Billing
SC
IN
PACS
Provisioning & Profiling Mgmt
Subscriber Profile
Service Profile
Partners Profile
Device Profiles
SDP Service Packages & Bundles
CRBT
Voice
Portal
Broadband
Subs
Tele
voting
ADC
Page 34
High Level SDP Deployment Architecture View
OSS/BSS
Nodes
CRM
Rating
Billing
IN PACS
RA
BI
SMSC Node
Core Network
Nodes
STP
HLR
GGSN
SGSN
MMSC Node
Streaming Server
Node
WAP G/W Node
IVR
CRBT
D
M
Z
F
I
R
E
W
A
L
L
N
O
D
E
External
Environment
DMZ
I
N
T
R
A
N
E
T
F
I
R
E
W
A
L
L
N
O
D
E
Internal
Network
E
N
T
E
R
P
R
I
S
E
F
I
R
E
W
A
L
L
N
O
D
E
Enterprise/
Core
Network
Content
Management
System
Node
Message
Broker
Node
TWSS
Node
Parlay
Gateway
Node
Portal Server
Node
I
n
t
e
r
n
e
t
DRM
Node
Transcoder
Node
TAM
Node
W
P
S
N
O
D
E
Web
Server
Content
Partner
Client
Node
Subscribers
PC Node
Inranet Network
Admin
Node
Content
Partner
Host
Node
Subscribers
Mobile Client
Node
CMS
DB
Node
MB
DB Node
Portal
DB Node
NDS
Node
Page 35
Conclusion
Business Value
Reduced Concept to Market to Revenue timeframe for new innovative products &
Services
Enables Innovative business models reaching out to 3
rd
party, developers
Standardized and repeatable IT cost across service deployments
IT Value
Optimized and converged workflow and business processes
Shared operating infrastructure and resources
Scalability for over 200 million subscribers and 300+ devices
High performance/ availability platforms
Provides key reconciliation infrastructure between OSS/BSS systems and down stream
network application and media environments
Network Value
Creates a Service Control Layer to the Network
This layer operates independently from the access network (wire line, wireless, broadband,
IP, 3G) and OSS-BSS systems.
Page 36
A few key points to take
away
A SDP solution is a service implementation
layer across a diverse set of capabilities
Drastically reduces time to market of new
services
SDP is a complex System Integration exercise
No two SDP implementations are the same
Page 37
THANK YOU
QUESTIONS ?
Reach me at +9198184870626
ybhatia1@in.ibm.com
Randy Tan, Rational IT Specialist
tanrkk@my.ibm.com
Session D05: Using Rational
Performance Tester v7.0.1
Page 2
Agenda
Introduction
Platforms and Protocols
Workload Scheduling
Greater Performance & Flexibility
SOA Capabilities
Debugging Tests
Page 3
Create realistic user scenarios
that exercise key business
transactions
Identify the root cause of
problems
Validate Performance with IBM Rational Performance
Tester
Powerful workload
scheduler
simplifies the
creation of realistic
user workloads
Advanced Data
Correlation features
significantly reduce
test development
effort
Easily identify
parameters and
enable scenarios for
data-driven testing
Quickly identify poor-
performing
transactions
Breakdown response
times across
application tiers and
drill down to source
Correlate response
times with system
resource
measurements
Test
Authoring
and Control
System
Under Test
Load Generators
Is your application ready for
production deployment?
Page 4
Whats new in RPT
v7.0.2?
Page 5
Agenda
Introduction
Workload Scheduling
SOA Capabilities
Debugging Tests
Page 6
New Platforms and Protocols
Socket (telnet, ftp, etc)
Siebel 8.0
Citrix Performance Server 4.5
Windows 2003 SP2
Red Hat Enterprise Linux Version 5
Page 7
What type of environments are supported?
Web-based applications
HTTP/HTTPS with client-side digital certificates
Web 2.0/Ajax
Entrust TruePass security
SOA
SOAP over HTTP/HHTPS
SOAP over MQ
SOAP over JMS
ERP environments
mySAP, SAP GUI and SAP-JCO
Siebel-web (7.7 and 8.0)
Lawson System Foundations
Session Initiation Protocol (SIP)
Citrix Presentation Server (Citrix ICA)
Partner Integrations
BSD Oracle eBusiness Suite
Page 8
Line Speed Control
Emulate WAN connection speeds for HTTP to determine the effect on
response times and throughput.
Control by User Group.
Specify upload or download speed.
Wide range of speed options available
56KB Modem to 1GB LAN
Custom values
Page 9
Example Enabling Line Speed Control
Page 10
Time-based Loops
New options for how loops behave in schedules and tests
Execute specified number of iterations
Infinite loops controlled by schedule stage time or manual stop
Time-based loops execute for the time you specify
Page 11
Agenda
Introduction
Workload Scheduling
SOA Capabilities
Debugging Tests
Page 12
Staged User Scheduling
Ramp Up Virtual Users
Ramp Down Virtual Users
Virtual User Load Preview Graph
Automatic Time Ranges
Automatic Compare Report
Page 13
Agenda
Introduction
Workload Scheduling
SOA Capabilities
Debugging Tests
Page 14
Greater Performance
Comment Actions
Disable Secondary Requests
Custom Percentile Reports
Random Selector in Test
Test Log Data Scalability
Playback Engine CPU Efficiency
Page 15
Comment Actions
Possible to comment out actions
Temporarily disables action
execution
Right-click and choose Disable
Grays out and marks action with //
Page 16
Disable Secondary Requests
HTTP
At playback GETs only primary
request
Skips gifs, jpegs, etc.
Page 17
Custom Percentile Reports
Defaults are 85, 90, 95
Possible to change or enlarge this set
Windows > Preferences > Test > Performance Test Reports > Percentile
Reports
Page 18
Random Selector in Test
Already available for schedule elements
Run test elements in random order
Page 19
Test Log Scalability
Disk Paging System
Collect Test Log data up to disk size
Just because you candont
Many users logging at a high level can take a lot of disk space
Data must be transferred and loaded after the run which takes time
Not much user feedback during the transfer process
When should I turn the knob to 11?
Data needed for debugging incorrect test operation
Very small number of users running
Page 20
Playback Engine Efficiency (Internal Benchmark)
HTTP Performance Metrics by Release
0
2
4
6
8
10
12
6.1.1 6.1.2 7.0 7.0.1 7.0.2
Release
%
C
P
U
/
M
B
0
200
400
600
800
1000
1200
K
B
/
u
s
e
r
CPU (%/MB) Memory (KB/user)
Page 21
Agenda
Introduction
Workload Scheduling
SOA Capabilities
Debugging Tests
Page 22
IBM Rational Tester for SOA Quality
Code free testing of GUI-less services
Simplify testing of GUI-less services and ensure
interoperability
Automated generation of web service test client
Create, comprehend, modify, and execute a
functional test
Graphical test editor enables both high-level and deeper
detail views
Support Web services standards
Simplify service integration testing
Automated test creation from Web Services Business
Process Execution Language ( WS-BPEL) resources
Accelerate test creation and execution
Automated data correlation and data driven testing through
a code-free test environment
Supporting a wide variety of users on broad range of
platforms
Test creation WS-
BPEL
Test log details
Page 23
New & Noteworthy in 7.0.2
Support for the latest Web services standards
SOAP 1.2, MTOM attachments, and XML type derivation
Support for asynchronous communication models
Support for WS-Addressing, WS-Notification
Enhanced functional test report
HTML-based comprehensive functional report
5 pre-defined report design templates with different level of details
Highly customizable through the creation of user-defined design template
Improved usability
New Web services test creation wizard
Automatic import of multiple WSDL/XSD files
SOAP 1.2
7.0.1 7.0.1 7.0.1 XML
7.0.1 7.0 7.0 SOAP 1.1
MQ JMS HTTP
Page 24
HTML-based comprehensive functional report
Page 25
Functional Report Generation
1. In the Test
Navigator, select
the runs
2. Right-click
Generate.
3. Select the design
Page 26
Agenda
Introduction
Workload Scheduling
SOA Capabilities
Debugging Tests
Page 27
Debugging Tests Data Correlation
Enhanced support for WebSphere Portal
Test Data Sources View
Separate Custom Code, Datapool, references, and candidates
Help find missed correlations
Page 28
Real-time Browser View
Observe test request and response
behavior in real time
Capture and compare request and
responses between record and
playback
Pause and navigate live rendering to
review browser and events during
playback
Note: implemented for stand-alone test
run only. Future update will integrate
at the schedule level
Real-time Browser results for test
validation and debugging
Page 29
RPT 7.0.2 Documentation Improvements
Posted external Help system (information center)
on http://publib.boulder.ibm.com/infocenter/rpthelp/v7r0m0/index.jsp
Addressed customer comments in 7.0.1.1 beta. New topics include
Changing look of a report
Deleting a file
Section on error messages
New SAP and Citrix tutorialsentire walkthroughs
Schedule UI has changed. Help topics will include not only how to use the
new UI, but how to do things the old way.
More context sensitive help just press F1
Page 30
Plus More!
Change order of Custom Code arguments
New counters for bytes sent and bytes received
Automated correlation of XML data
Add or Remove users button
Automatic compare report
Remove users incrementally
Page 31
Page 32
Lab Agenda
Six lab exercises to build expertise with
Rational Performance Tester:
Introduction to RPT
Lab 1: Basic Capture/Replay
Lab 2: Input Data Variation
Lab 3: Data Validation
Lab 4: Custom Coding & Data Correlation
Lab 5: Workload Generation
Lab 6: Performance Analysis
Page 33
Page 34
Lets Begin!!!
Page 35
Session F02: XML Solutions with
Session F02: XML Solutions with
WebSphere Transformation Extender
2008 WebSphere Technical Conference
22
Conference Speaker
Conference Speaker
XML Solutions with WebSphere
XML Solutions with WebSphere
Transformation Extender
Session Number: F02
Session Number: F02
Ernese Ernese Norelus Norelus
ernese@sg.ibm.com ernese@sg.ibm.com
Senior Managing Consultant Senior Managing Consultant
IBM Software Group | ASEAN Software Lab Services IBM Software Group | ASEAN Software Lab Services
33
WebSphere Transformation Extender Sessions
WebSphere Transformation Extender Sessions
F08 WebSphere Transformation Extender 8.2 Design
Tools
Tuesday 14:30 Elmer Senson and Dinesh G. Shetty
F02 - XML Solutions with WebSphere Transformation
Extender
Tuesday 16:45 Ernese Norelus
F02 - XML Solutions with WebSphere Transformation
Extender
Thursday 10:45 Ernese Norelus
F01 Building Financial Solutions with WebSphere
Friday 9:00 - Elmer Senson
B04 WPS/WESB 6.1 Integration with WTX, Support and
Best Practices
Friday 14:45 - Ernese Norelus
44
Objectives
Objectives
Gain a basic understanding of WebSphere
Transformation Extender (WTX) concepts
What is WTX?
Key Concepts
Methods of Execution
Connectivity
Packs
WTX Components
Demos
55
WebSphere TX Vision
WebSphere TX Vision
Collaborative, extensible and
ever-simplifying environment
The seamless conversion of all
information
Ubiquitous, plug and play engine
66
WebSphere TX Vision
WebSphere TX Vision
-
-
Simplified
Simplified
Everyone Transforms
Transform Anything
Transform Anywhere
77
What Kinds of Problems Does
What Kinds of Problems Does
Transformation Extender Solve?
Transformation Extender Solve?
A powerful, portable, transformation engine
Simplified Complexity Simplified
Simplified
Complexity
Complexity
~ No coding
~ Consistent approach to multiple
types of integration
~ Reusable objects
Adaptability Adaptability
Adaptability
~ Leverages existing IT infrastructure
~ Fully support existing IT investments
such as databases and messaging
middleware
~ Non-invasive integration
~ Application and industry packs
Speed Speed
Speed
~ Easy design and implementation
~ Fast maintenance without coding
~ Rapid adaptation to change
Interoperability Interoperability
Interoperability
~ Connect applications, databases,
processes within the enterprise
~ Partners, processes across
enterprises
~ Powerful transformation and
routing
88
WebSphere TX Product Strategy
WebSphere TX Product Strategy
The best-in-class transformation engine that is seamlessly
leveraged across the WebSphere platform
The best-of-breed transformation engine that is easily
deployed in any enterprise infrastructure
99
Transform, validate, and enrich any document, message or complex data
Deliver trustworthy information for critical business initiatives
Meet regulatory compliance requirements
Codeless development; universal reuse and deployment
WebSphere TX: Universal Transformation
WebSphere TX: Universal Transformation
& Reuse
& Reuse
Make Advanced ESB
More Advanced
Embedded
Standalone
Event driven
MAP
TRANSFORM
Advanced ESB ESB
J 2EE
App. Server
Process Server
Standalone
System z (native)
Windows, UNIX, Linux
Multi-In Multi-Out
10 10
For Batch
Environments
(as a standalone engine)
For WebSphere
Portfolio
For ETL Solutions
WebSphere
TX
WebSphere
TX
MAP
TRANSFORM
Unifying
Enterprise Data
Transformation
Unifying
Enterprise Data
Transformation
WebSphere Transformation Extender: A Universal
WebSphere Transformation Extender: A Universal
Transformation Environment
Transformation Environment
Manage The Complexity!

Manage The Complexity!
ONE Transformation
Engine for the
Enterprise!
Powerful transform
capabilities without
coding
Natively handles any
data type
Solves really hard
transformation
problems
Complements
existing portfolio of
products
For Event Driven
Environments
For 3
rd
Party
Integration Products
(as an embeddable engine)
11 11
and the IBM Platforms
and the IBM Platforms
WebSphere
Application
Server
WebSphere
ESB
WebSphere
Process
Server
WebSphere
Message
Broker
z/OS
(CICS, MVS, IMS
Batch, USS)
Leveraging common tooling and runtime, for multiple, interoperable,
deployment options
WebSphere
DataPower
12 12
WTX Industry Accelerator Packs
Adding Value to the WebSphere Portfolio

Adding Value to the WebSphere Portfolio
X12
EDIFACT
TRADACOMS
ODETTE
EANCOM
SWIFT Net FIN
SWIFT Net Funds
*SEPA
*ACORD
*NACHA
*FIX
HIPAA
HL7
NCPDP
WTX (any deployment option)
WTX Industry Packs
A Pack is not an adapter - a Pack contains meta data, data
validation, usage rules, transformation maps and additional
functionality
13 13
WTX (any of the server types)
WTX Enterprise Application PACKs
SAP R/3
SAP XI
Siebel
PeopleSoft
WTX Enterprise Application Accelerator
WTX Enterprise Application Accelerator
Packs
Packs
Includes BAPI, ALE and IDOC
integration capabilities
Runs inside SAP App Server
Includes BAPI, ALE and IDOC
integration capabilities
validation, usage rules , transformation maps and additional
functionality (you can sell an adapter and a Pack)
14 14
* $4.6T is based on available, tangible measurements.
On average, over US $4.6T* in transactions flows
through WebSphere Transformation Extender on a daily
basis
4 Major Stock exchanges
9 of 10 Major US Financial Firms
6 of 6 Top Global Banks
6 of 10 Top US Banks, 4 of 5 Top UK Banks
75% of Blue Cross / Blue Shield claims
6 of 10 Major Insurance Firms
3 of 3 Top US Pharmacy Benefits Providers
Major Global and National Telecoms
Over 95% of all US Department of Defense logistic
transactions pass through TX
and on and on and on.
WebSphere Transformation Extender is a
WebSphere Transformation Extender is a
strategic solution for our customers
strategic solution for our customers
15 15
Unique Values of WebSphere
Unique Values of WebSphere
Handles all data natively
Adapters are communication vehicles only
No internal format all work happens in the engine, under the control of the
designer
Significantly reduces complexity of product, and opportunity to leverage
customer technology investment
Can process interdependent inputs simultaneously
High performance data validation and processing
Handles Complex IT processing / transformation challenges without code
Can be leveraged in all styles of integration
Batch, Asynchronous, Synchronous, Real-Time and Web Service
environments
16 16
across WebSphere
across WebSphere
WTX is the WebSphere Universal Transformer
Any kind of data, any kind of transform
Is uniquely capable of MANY to MANY transforms
WTX is a value add on top of the WebSphere Platform Products
WTX runs natively in CICS, MVS and IMS Batch
Unique offering across IBM
Data Processing Capability without Code
Same Maps can be reused across Mainframe, App Server,
Windows, Unix and ESB, Message Broker
One Tool and One Engine, able to be leveraged in Any
Architecture, for Any Transformation Demand, in Any
Transformation Stylea true Universal Transformation Engine
17 17
WebSphere TX v8.2 Release Features
-
-
Tooling
Tooling
Design Tools
Map Designer and Type Designer have become
Eclipse plug-ins.
They may be hosted by a stand-alone Design Studio,
or they may be hosted in other Eclipse design
environments, such as the WebSphere Message
Broker Toolkit, or WebSphere Integration Developer.
No significant change in functionality or design
Integration Flow Designer and its Management
Tools have not been ported to Eclipse.
Maps are still MMS files and most Type Trees
are still MTT files
Native support added for XML Schemas (details
to follow)
18 18
WebSphere TX: Key Constructs
WebSphere TX: Key Constructs
Type trees
Graphical data dictionary
Maps
Complete definitions of data objects and rules for their transformation
A subroutine implemented as a map is called a Functional Map
Cards
Inputs to or outputs from maps
Systems
Collections of Maps configuration to run inside the Launcher
19 19
WebSphere TX: Working with Type Designer
WebSphere TX: Working with Type Designer
Type Designer
Used to create and edit type trees
Importer Collection
Launched from the Type Designer.
Used to generate type trees from external meta data sources
Database Interface Designer
Used to build type tress from Database tables
Generates Database Query (MDQ) files in additions to type trees
20 20
WebSphere TX: Working with Maps
WebSphere TX: Working with Maps
Map Designer
Used to create, edit and compile maps.
Host the Profiler and Debugger
Can launch the Type Designer.
Resource Registry
Defines runtime parameters which may be referenced
Command Server
Used to configure and execute maps
21 21
Map Designer
Map Designer
22 22
Type Designer
Type Designer
23 23
-
-
Tooling
Tooling
Classic Importers
The classic importers are now Eclipse importers, which
may be invoked from the new Type Designer.
No change in behavior from v8.1.
They will be joined by a new set of EMD importers common
to many IBM Eclipse-based design tools.
Text File Importer improvements
Group members no longer must be all on the same line
Types organized by Category
Improved usability
New PL/1 importer coming with 8.2.0.2
24 24
Importing a Type Tree
Importing a Type Tree
-
-
Example
Example
25 25
XML Schemas
XML Schemas
vs
vs
Type Trees
Type Trees
Schemas appear as regular type trees
In addition to .mtt files, the new Map Designer supports .xsd files
There is actually no need to import schemas anymore, but the
XML importers are retained in V8.2
Schemas may be edited using any schema editor IBM or other.
You do not need to use Type Designer to edit them
Expect performance improvements compared to previous
versions
26 26
Working with XML Schema
Working with XML Schema
-
-
Example
Example
Create a map
Create a card
Choose a schema
Choose a root
Output cards work too
Map away !
It looks like a Type Tree
27 27
WTX demo
WTX demo
Create a WTX Project
Create a Type Tree
Create a Map
Load a WTX example
WTX Importers
Any-2-XML
28 28
-
-
XML
XML
Native support added for XML Schemas
Restart and reject logic is added for XML Type Trees
With restart enabled, the XML validation log file now reports all
validation errors, it doesnt stop at the first error.
Schema location setting for Cards
Card Metadata Location parameter
Allows you to specify the location of your schema (as a file or url)
Prior to v8.2 the information was only in XML Type Trees
(Intent/Validate As/Location on the Doc group)
Schema validation can be disabled on a per Card basis
via a new Card property option
Document Verification / Well Formed
Support has been added for the XSD Union Type and xsi:type
29 29
XML document as map source
XML document as map source
1. Generate type tree from schema or DTD
2. Modify Doc component list to identify root element when
possible
3. Create map with schema type tree on input card
4. Select document validation option on input card
5. Expand components
6. Enter map rules based on input and output type properties
7. Create functional maps or use INDEX to map objects that
occur more than once
8. Build and run the map
30 30
Any
Any
-
-
to
to
-
-
XML utility
XML utility
Creates an XML schema from a type tree
Generates a Xerces-based type tree from the
XML schema
Optionally creates a new map source file and
map that transforms original input to XML
output
Input card references the reference type tree and
input file
Output card references new XML type tree and
schema to create output file
31 31
-
-
Connectivity
Connectivity
A subset of IBM WebSphere Adapters are offered with WTX v8.2
WebSphere Adapters are JCA based
Their legacy counterparts are still included, but have been deprecated.
The specific adapters included are:
Email
Includes support for multiple attachments, and headers
Adds a listener interface
FTP
Allows remote files to be renamed once they have been read
Adds a listener interface
SAP *
Adds support for the Query Interface for SAP Software (QISS)
PeopleSoft *
Adds a listener interface for Component Interfaces
Siebel *
Adds a listener interface for Business Objects
IMS TM (coming in 8.2.0.2)
*Note: SAP, PeopleSoft and Siebel are available only in the Enterprise Packs
32 32
Importers
Importers
Each JCA adapter includes a new EMD * importer, which may
be used to generate:
XML schemas in lieu of Type Trees
Service Definitions similar to MDQ files
Adapter Command Lines
At the point where the adapter is bound to the card, a Command
Line Wizard is available.
* Enterprise Metadata Discovery
33 33
WebSphere
WebSphere TX TX
v8.2 Release Features
v8.2 Release Features
-
-
Editions
Editions
WTX the edition for imbedding is renamed for clarity of purpose.
WebSphere Transformation Extender for Application Programming
(for z/OS)
The Extender for Message Broker is now packaged with a new
Extender in a single Edition *.
WebSphere Transformation Extender for Integration Servers
Integration Servers name chosen as an inclusive term for ESB and
BPM solutions:
Extender for WebSphere Message Broker
Extender for WebSphere Process Server / WebSphere ESB
A single License instance permits use of one or the other.
Note: The Extender for Message Broker remains at V8.1 in terms of functionality
but will be significantly enhanced in 1H08.
34 34
WebSphere TX for Integration Servers
WebSphere TX for Integration Servers
-
-
extender for WPS/WESB
extender for WPS/WESB
WebSphere TX can be selected as a Data Binding on JMS, MQ, HTTP or EIS
Imports or Exports
Maps typically convert between external data formats and Business Objects
Map Generator is a new Eclipse based Wizard
simplifies development of maps
Business Objects are converted to Type Trees (technically schemas)
Template Maps are created
Projects (including their maps) are easily deployable using standard WPS
and WID mechanisms
The Wizard integrates WTX Design Studio with WebSphere Integration
Developer
35 35
Map Generator Wizard
Map Generator Wizard
Example
Example
36 36
-
-
Industry Packs
Industry Packs
WTX Packs for EDI v2.7.1.1
WTX Pack for X12
Released for WTX V8.2 in 1Q08
WTX Pack for EDIFACT
Released for WTX V8.2 in 1Q08
WTX Pack for EANCOM
No change
WTX Pack for TRADACOMS
No change
WTX Pack for ODETTE
Bug Fixes only
37 37
EDI
EDI
Earlier Releases
Earlier Releases
Earlier Releases
Released Sept. 2007 to work with WTX v8.1
WTX Pack for X12 v2.7.1.0
Support for ANSI X12 5040
Majority of changes in Transportation/Shipping
Also includes a new transaction for Customs and Border
Protection
WTX Pack for EDIFACT v2.7.1.0
Support for UN/EDIFACT D.06A, D.06B
No new messages added; only a few changed.
38 38
-
-
Platforms
Platforms
Development
Windows
Run time 32 bit
Windows
IBM AIX
Red Hat Enterprise
Linux for x86 platforms
SUSE Linux Enterprise
HP-UX 11i (PA-RISC) or
11i V2, V3 (Itanium)
Sun Solaris
z/OS 1.4 or later (batch,
USS, IMS, CICS)
Run time 64 bit tolerance
new
IBM AIX
Red Hat Enterprise Linux
for x86 platforms
SUSE Linux Enterprise
HP-UX 11i (PA-RISC) or 11i
V2, V3 (Itanium)
Sun Solaris
Linux on System z new
39 39
-
-
Upgrading
Upgrading
Design Studio
All existing maps and type
trees will automatically open in
v8.2
No changes to Database
Interface Designer &
Integration Flow Designer
WebSphere TX for Message
Broker
Existing flows that use the
v8.1 Node will seamlessly
migrate to the v8.2 node
Existing flows that use the
v8.1 Parser will need to be
recreated
WebSphere TX Runtimes
All compiled maps from
WebSphere TX v5.0 and
higher will run without
recompiling
If a map will be recompiled
with Version 8.2, all type trees
must be reanalyzed with
Version 8.2
Use of the new Native XML
features instead of type trees
will require maps to be
rewritten. Existing XML type
trees and maps will remain
valid.
Like previous versions, TX
can be installed side by side
on the same machine
40 40
Summary
Summary
What is WebSphere TX ?
WebSphere TX is IBMs Universal Transformation Engine
Key Concepts
Maps, Cards, Type Trees, Functional Maps
Methods of Execution
Command Server, Launcher, Extenders for WebSphere Portfolio, SDK
Connectivity
Adapters, Importers
Demos
Importers, XML output
41 41
Questions
Questions
Please complete your session evaluation
Thank you!!
Dinesh G Shetty
IBM Certified IT Specialist
F03 - Driving WebSphere Message Broker
using WSRR
Page 1
Agenda
Brief Review of WSRR
WMB usage patterns
Implementing patterns
Page 2
Agenda
What is it?
How WMB interacts
WMB usage patterns
Page 3
What is WSRR?
Repository? Registry?
Stores Artifacts / entities
WSDL
XSD
Other XML documents
SCDL
etc.
Hold Meta-data
Describes the service
artifacts
Allows you to search for
artifacts in the repository
Page 4
How WMB Interacts with WSRR
Two nodes that query an instance of WSRR
Result placed into LocalEnvironment
Payload is not modified
Results of query propagated to Out terminal
Out
Failure
NoMatch
In
Out
Failure
NoMatch
In
EndpointLookup Node
Focused on the retrieval of endpoints defined
within WSDL
Retrieves EPR and Metadata, not whole
artifact
Able to search port type names within WSDL
imported into WSRR
Use endpoint for dynamic routing at runtime
RegistryLookup Node
Generic Node for retrieval of any WSRR
entity
Able to search entity names
Retrieve entire contents of entity into the
Broker
Use entity information at runtime
Page 5
Agenda
WMB usage patterns
General patterns
Sample patterns where WSRR fits
Page 6
WMB Usage patterns
Service Enablement
Service Virtualization
OR
OR
OR
Message Enablement
Message Brokering
File Processing
Event Processing
A10
Page 7
Notes
We have identified six key usage patterns for Message Broker. You may find that all or only some of these patterns apply to your environment.
The extent to which all patterns apply will probably depend on the number, age and maturity of your IT systems. It is also important to say
these classifications are not rigid. As with any classification system there will be anomalies and overlaps. Indeed some implementations of
Message Broker will sit in more than one usage pattern and this happens in a couple of cases that we will look at.
Service enablement. This is essentially taking a varied collection of applications and presenting aspects of their functionality through a service
orientated interface. This pattern represents the transition from more traditional notions of EAI into service orientated architectures.
Service virtualization. This pattern is concerned with providing true loose coupling between services by providing additional levels of
indirection through an ESB. This addresses the requirements of mediation between services in a services orientated model.
Message enablement. This is concerned with applying a message based architecture to applications, which were not necessarily designed to
use one by using an Enterprise Service Bus (ESB).
Message brokering. This covers extending an existing messaging infrastructure by providing an environment for building and deploying
"infrastructure level" message-based "applications. Examples of such applications would include routing, logging and transformation
services.
File processing. This usage pattern is about facilitating the integration of file based applications and providing a managed execution
environment for the processing of files. It is also about providing a bridge between batch-orientated applications and messaging models of
execution including transactional processing.
Event processing. This is about providing a managed execution environment for the processing of events including distribution of events in
real time, processing of events from physical devices, simple information distribution and detection of temporally orientated event patterns.
Page 8
Service Enablement
Message Enablement
Message Brokering
WMB Usage patterns
OR
OR
OR
File Processing
Event Processing
A10
Page 9
Service Virtualization Service Proxy
request message contains Service Name or rather a Service ID
ESB retrieves endpoint information for that particular service from WSRR
hiding the actual service providers true endpoint
ESB
ESB
Provider
Service B
Provider
Service B
Requestor
Requestor
Provider
Service A
Provider
Service A
Service A
WSRR
WSRR
Page 10
Service Virtualization Service Selector
(Environment)
Multiple implementations of a service interface. Example:
request is sent to a certain ESB environment (e.g., TEST / PROD)
depending on which environment the ESB is running on, the message is sent to the corresponding service
providers environment
ESB (PROD)
ESB (PROD)
Provider
Service A (PROD)
Provider
Service A (PROD)
Requestor
Requestor
Provider
Service A (TEST)
Provider
Service A (TEST)
Service A
WSRR
WSRR
Page 11
Service Virtualization Service Selector (Fault
Handling)
retrieval of a collection of service endpoints providing the same function
message is sent to one of the services taking into contrast the services
providers availability, workload, performance etc.
ESB
ESB
Provider
Service A
Provider
Service A
Requestor
Requestor
Provider
Service A
Provider
Service A
Service A
Fault
WSRR
WSRR
Page 12
Service Virtualization Service Normalizer
mapping information for the mapping between the service requestors
interface and the service providers interface is retrieved from WSRR
message content is adapted before the message is sent to the provider
ESB
ESB
Provider
Provider
Requestor
Requestor
A
B
C
C
AB
D
WSRR
WSRR
Page 13
Service Virtualization Service Translator
(Binding)
ESB retrieves the binding information from WSRR
message is then sent to the Service Provider using the adequate protocol
and transport for that service implementation
ESB
ESB
Provider
Provider
Requestor
Requestor
WSRR
WSRR
Page 14
Service Virtualization Policy Enforcement
request message is signed and / or encrypted
ESB checks against WSRR if the service policy is met for a particular
service
ESB
ESB
Provider
Provider
Requestor
Requestor
WSRR
WSRR
Page 15
Agenda
WMB usage patterns
Service Selector (Environment)
Service Proxy & Translator (Binding)
Page 16
(Environment)
What is it?
What problem does it solve?
How to define in WSRR?
How to build WMB solution?
Demo
OR
OR
OR
ESB Mediation: Service selector
Service
requestor
Service
provider
Service
proxy
Service Registry
Service
provider
Service
provider
Content
Based
Router
endpoint reference
Service
requestor
Service
provider
Service
proxy
Service Registry
Service
provider
Service
provider
Content
Based
Router
endpoint reference
Service Selector
Page 17
(Environment)
What is it?
Enables invocation of endpoint based on runtime environment
For example:
Development
Test
Production
Service
requestor
Service
provider
Service
proxy
Service Registry
Service
provider
Service
provider
Content
Based
Router
endpoint reference
Dev
Dev
Test
Test
Prod
Prod
Page 18
(Environment)
What problem does it solve?
Empowers customers to manage runtime behavior via central SOA
repository
Minimizes preparing deployment artifacts per environment
Minimizes revisiting development artifacts per environment
Page 19
(Environment)
1. Follow recommended practice for Endpoint Management
2. Follow recommended practice for multi-file wsdl
Separate service port from the interface
(May have separate binding or include binding with service port)
3. Load WSDL and Schema to WSRR
4. Classify ports by Environment
Page 20
<?xml version="1.0" encoding="UTF-8"?>
<schema targetNamespace="http://demo.sr.eis.ibm.com
<element name="updateCustomer">
<complexType>
<sequence>
<element name="str" nillable="true" type="xsd:string"/>
</sequence>
</complexType>
</element>
<element name="updateCustomerResponse">
<complexType>
<sequence>
<element name="updateCustomerReturn" nillable="true" type="xsd:string"/>
</sequence>
</complexType>
</element>
<element name="updateCustomerAlternateInfo">
<complexType>
<sequence>
<element name="customer" nillable="true" type="impl:CustomerInfo"/>
</sequence>
</complexType>
</element>
<complexType name="CustomerInfo">
<sequence>
<element name="firstName" nillable="true" type="xsd:string"/>
<element name="lastName" nillable="true" type="xsd:string"/>
<element name="middleName" nillable="true" type="xsd:string"/>
<element name="addresses" nillable="true" type="impl:ArrayOfAddressInfo"/>
</sequence>
</complexType>
<complexType name="ArrayOfAddressInfo">
<sequence>
<element maxOccurs="unbounded" minOccurs="0" name="AddressInfo"
</sequence>
</complexType>
</schema>
schema (xsd) wsdl
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<sequence>
</sequence>
</complexType>
<sequence>
</sequence>
</complexType>
</schema>
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<complexType>
<sequence>
</sequence>
</complexType>
</element>
<sequence>
</sequence>
</complexType>
<sequence>
</sequence>
</complexType>
</schema>
2. Follow recommended practice for multi-file wsdl
(Environment)
Page 21
(Environment)
3. Load WSDL files and Schema files to WSRR
Page 22
(Environment)
Page 23
(Environment)
Page 24
(Environment)
Page 25
(Environment)
In
Set
Query
Endpoint
Lookup
Map
Map Reply Request
Page 26
(Environment)
Set
Query
Endpoint
Lookup
Page 27
(Environment)
NOTE: For demo purposes the above code
uses Execution Group to represent the runtime Environment.
DECLARE ThisBroker CHAR BrokerName;
DECLARE ThisEG CHAR ExecutionGroupLabel;
DECLARE ClassificationString CHAR
'http://www.ibm.com/xmlns/prod/serviceregistry/6/1/GovernanceProfileTaxonomy#';
-- Copy HTTP Request
SET OutputLocalEnvironment.Destination.HTTP.RequestIdentifier =
InputLocalEnvironment.Destination.HTTP.RequestIdentifier;
-- Set Service Registry Lookup elements
SET OutputLocalEnvironment.ServiceRegistryLookupProperties.Classification = ClassificationString || ThisEG;
Page 28
(Environment)
Page 29
(Environment)
Demo
1. Scenario Overview
2. Define WSDL
3. Configure WSRR
4. Construct Solution
5. Test
Page 30
(Environment)
Demo scenario begin
Service
Requester
Service
Provider
Sub-flow
Page 31
Service Virtualization Service Selector (Environment)
Demo scenario
Reply message is string:
Updated customer via WMB env: <Broker Name>.<EG Name>
deployed to Development using URL:
http://localhost:7080/DemoCustomerMB/services/Dev/DemoCustomer
deployed to Test using URL:
http://localhost:7080/DemoCustomerMB/services/Test/DemoCustomer
Service Provider
Page 32
Demo scenario
Begin challenge: static URL
Moving from Development to Test env requires modifying bar file
property for the HTTP Request node url
from Development URL:
http://localhost:7080/DemoCustomerMB/services/Dev/DemoCustomer
to Test URL:
http://localhost:7080/DemoCustomerMB/services/Test/DemoCustomer
Service Requester
Page 33
Demo scenario
Using VMWare WMB v6.1 fp1, WSRR v6.1 fp2
Single Broker: WBRK61_DEFAULT_BROKER
ServiceRegistry pre-configured to broker
Two EGs:
Development
Test
Page 34
(Environment)
Demo scenario end
Service
Requester
Service
Provider
Sub-flow
Page 35
(Environment)
Demo link
Page 36
Service Virtualization Service Proxy &
Translator (Binding)
What is it?
What problems does it solve?
Demo
OR
OR
OR
ESB Mediation: Service Normalizer
Service
requestor
Service
provider
Service
proxy
Service Registry
Service
provider
Service
provider
Content
Based
Router
Service
proxy
Service
proxy
Message
translator
Message
translator
Message
translator
Service Translator
Page 37
What is it?
decoupling between service requestors and service providers by introducing an additional level
of indirection to the service invocation
technical details about the services implementation are not exposed to the service requestors
Service Normalizer
Service
requestor
Service
provider
Service
proxy
Service Registry
Service
provider
Service
provider
Content
Based
Router
Service
proxy
Service
proxy
Message
translator
Message
translator
Message
translator
Page 38
What problems does it solve?
single point of access
service implementation details can change without the need to inform
the clients
further advantages with the implementation of policy enforcement,
autiting etc.
Page 39
Assumptions / Constraints
service requestor knows the service names
service interfaces / message content not validated in the flows
definition of a SOAP binding for a service suggests that the service endpoint must
be defined as a SOAP address (regardless of the transport used for that endpoint)
simple functional error-handling
no technical error handling
communication ESB WSRR and ESB Services has been established
service information is present and can be retrieved from WSRR
service responses (processing of requests) are hard-coded
ServiceRegistry pre-configured to broker
Page 40
followed recommended practice for multi-file WSDL (Top-Down Approach)
problems with the recommended practice for WSDL service versioning
SOAP Nodes do not support different target namespaces within the single files
internal service provider: AccountBalanceService (SOAP / WMQ)
external service provider: StockQuoteService (SOAP / HTTP)
Page 41
StockQuoteService WSDL (Binding and Port)
Binding
Port
Page 42
AccountBalanceService WSDL (Binding and
Port)
Binding
Port
Page 43
Service Endpoint representation in WSRR
SOAP protocol can be used with different transports (e.g. HTTP, WMQ)
for the WMQ service refer to the IRI specification which defines the
addresses and additional messaging properties of WMQ resources
(SupportPack MA93)
Page 44
Service Binding representation in WSRR
the different transports are specified on the SOAP binding
native MQ binding as defined in supportpack MA93 is not supported yet
refer to REDP-4350-00 Cataloging WebSphere MQ applications in WSDL for reuse in SOA
for a workaround on that
Page 45
Page 46
Considering the use of the new SOAP Nodes
LocalEnvironment overrides are
available for a couple of runtime
properties, e.g. endpoints
other information about the
service interface and binding are
set with a particular WSDL during
buildtime
these information cannot be
changed during runtime
for a Service Proxy you would
need the same number of SOAP
Request Nodes defined at
buildtime as the number of
services you want to access at
runtime
Page 47
Example Services (Broker Message Flows)
StockQuoteService
AccountBalanceService
1. get the request message
2. construct the response message
3. send back the response message
Page 48
Request
to ESB
Request
to ESB
WSRR
Lookup
WSRR
Lookup
Transport
Selection
Transport
Selection
Response
from ESB
Response
from ESB
Protocol
Conversion
Protocol
Conversion
Service
Request
Service
Request
Page 49
Enterprise Service Bus Message Flow
Request to ESB / WSRR-Lookup
Request to ESB / WSRR-Lookup
Protocol Conversion / Transport Routing
Protocol Conversion / Transport Routing
Service Request / Protocol Conversion
Service Request / Protocol Conversion
Response from ESB
Response from ESB
Functional Error Handling
Functional Error Handling
Page 50
Trade-Off between WSRR Nodes and Java
Compute Node
WSRR Nodes
supports WSRR API v6.0.x
cache feature
WSDL extension elements are not
supported
custom relationships are not supported
whole WSDL can be retrieved and parsed
in broker
Java Compute Node
any WSRR API (e.g. v6.1) can be used
cache has to be build on someones own
take into contrast the complexity of cache
updates (relationships between artifacts)
Page 51
WSRR Lookup and Response Processing
dummy value on the ServiceRegistryLookUp Node
modifying the lookup properties at runtime
SET OutputLocalEnvironment.ServiceRegistryLookupProperties.Name =
InputRoot.XMLNSC.RequestData.Service;
Page 52
Evaluating the results from WSRR
SOAPAdress element suggests
the service uses a SOAP binding
determines the further route in
the flow
location element contains the
service endpoint
Page 53
Accessing fields in the LocalEnvironment
results of a WSRR lookup are written to the LocalEnvironment
initial problems with accessing particular fields in the tree
InputLocalEnvironment.ServiceRegistry.Entity.ports.SOAPAddress
trace output of the ServiceRegsitryLookupNode showed that
namespaces were put into the LocalEnvironment
for some of the fields you have to use the correct namespace
DECLARE sdoNS NAMESPACE 'http://www.ibm.com/xmlns/prod/serviceregistry/6/0/sdo';
DECLARE soapRef REFERENCE TO
InputLocalEnvironment.ServiceRegistry.Entity.sdoNS:ports.sdoNS:SOAPAddress;
Page 54
Routing to the appropriate service request part
continue with routing the message to the corresponding service request part
depending on the transport information that are available from the
environment
for the HTTP request the endpoint can be taken as is
http://192.168.19.20:7801/banking/StockQuote
for the WMQ request the endpoint has to be decomposed into the request
queue and the reply to queue part
wmq:/msg/queue/ACC.BALANCE.REQUEST?ReplyTo=msg/queue/ACC.BALANCE.RESPONSE
create a standard SOAP Envelope before the message is sent further
Page 55
Service Requests
request is send out with either a HTTP Request or WMQ Output node
for the HTTP service request the SOAP Request Node should be avoided in
a generic Service Proxy Pattern
SOAP envelope is removed before the message is sent back to the
requestor
Page 56
Future Considerations?
SOAP Node support for multi-file / multi-namespace WSDLs
SOAP Node support for JMS / WMQ
Support for dynamic changing interface / binding information at
runtime
Support for dynamic transformation resolution
Support in WSRR nodes to dynamically set query (avoid need for
another compute node to set query)
WSRR Node support for WSDL extension elements
WSRR Node support for custom relationships
IBM Confidential
Page 57
Demo Link
Page 58
WSRR and WMB References
IA9Q: WebSphere MB v6 Client for WebSphere Service Registry and Repository
http://www.ibm.com/support/docview.wss?uid=swg24014652
Publishing a Web Service to WebSphere Service Registry and Repository from CICS
Transaction Server v.3.1:
http://www.ibm.com/developerworks/websphere/library/techarticles/0610_millwood/0610_millwood.html
Introducing WebSphere Service Registry and Repository - Part 1:
http://ltsbwass001.sby.ibm.com/cms/developerworks/websphere/library/techarticles/0609_mckee/0609_mckee.html
Introducing WebSphere Service Registry and Repository - Part 2:
http://www.ibm.com/developerworks/websphere/library/techarticles/0609_mckee2/0609_mckee2.html
Introducing WebSphere Service Registry and Repository APIs:
http://www.ibm.com/developerworks/websphere/library/techarticles/0611_baldwin/0611_baldwin.html
Build flexible ESB mediations with MessageBroker and WebSphere Service Registry
and Repository:
http://www.ibm.com/developerworks/websphere/library/techarticles/0610_patten/0610_patten.html
WSRR Info Center
http://publib.boulder.ibm.com/infocenter/sr/v6r0/index.jsp
WSRR Product Page
http://www.ibm.com/software/integration/wsrr/index.html
Page 59
Questions
Questions?
Please complete the session evaluation form
Thank you!!
Page 60
Dinesh G Shetty
IBM Certified IT Specialist
F04 - Best Practices in Modeling Complex
Messages with WMB 6.1
Page 1
Agenda
Introduction to message modeling
A reminder of the key concepts
Key tasks when modeling non-XML messages
Understanding the physical format
Understanding the logical structure
Relating the physical format to the logical structure
Text message formats
A look at text messages in more detail
Scenario
COBOL application
Comma Separated Values (CSV) application
Best practices
Diagnosis of modeling problems
XML parsing
Page 2
Notes - Agenda
This session assumes some prior knowledge of message
modeling and parsing using WebSphere Message Broker.
However, dont panic if you arent that familiar as well start
with an introduction of the main concepts.
We will introduce the key steps involved when modeling
messages using the MRM domain, looking at binary and text
message formats.
We will concentrate on text message formats as they are
usually the hardest to model, but the techniques introduced
are applicable to all message styles you are likely to
encounter.
We will conclude with a scenario to illustrate these
techniques.
Introduction to message
modeling
Page 4
<Person Age=32 Height=172.5>
<Name>Joe Bloggs</Name>
</Person>
What is message parsing?
Input message
bit stream
Output logical
message tree
Input logical
message tree
Output message
bit stream
Transformation
Parser Parser
032 1725 Joe Bloggs
Model Model
Page 5
Notes What is message parsing?
Typically a WebSphere Message Broker message flow will receive messages in a defined
format, transform them, and output them in a different format. The example shows a COBOL
Person data structure being transformed into an XML Person document. You can see that
each of the Person messages contains Name, Age and Height fields.
The COBOL Person message arrives in the form of an input bit stream. Before it can be
processed by the message flow, it must be converted into something called an input logical
message tree for transformation by the message flow. This is a broker data structure that
reflects the logical structure of the message. All of the brokers processing nodes work with
the logical message tree, regardless of whether your transformation logic is expressed in
Java or ESQL or Mapping. In the example, some logic is applied which transforms the
message and creates a new, output logical message tree. Finally this is converted into an
output bit stream that represents the message as an XML document.
The component of the broker responsible for converting a bit stream into a logical message
tree (parsing), and vice versa (serializing or writing), is called a parser. The parser must
understand both the physical format of the bit stream and its logical structure in order to
create the logical message tree. In the example you can see two parsers, one that
understands COBOL data structures and another that understands XML.
Note that it is only the parser that needs to understand the physical format of the message.
The logical message tree is independent of the physical format of the message bit stream. This
decoupling of physical format from transformation logic is a key architectural feature of
broker message flows.
A parser may, or may not, use a model when parsing. You will see the advantages provided
by having such a model, later on.
Page 6
Parsers and domains
Each message is associated with a domain
A domain is suited to a particular class of messages and has a parser
Broker supplies several domains
Some domains use a model-driven parser
XMLNSC domain for XML documents (6.1)
MRM domain for binary and text data (also handles XML)
SOAP domain for SOAP-based web services (6.1)
DataObject domain for WebSphere Adapters business objects (6.1)
Others do not use a model
BLOB domain for opaque data
XMLNSC (pre-6.1) or XMLNS domain for XML documents
MIME domain for MIME messages
You can also write your own
Page 7
Notes Parsers and domains
Each message that is to be processed by a message flow must be associated with a domain. A domain
determines the parser that is used when parsing and serializing the message. Each domain is suited to
a particular class of messages. Some domains even support several different classes of message. The
domain to use for an input message is typically specified on the input node of the message flow, but
can also be specified by an MQRFH2 header or JMS header. The domain to use for an output message
is specified when the message logical tree is created by a transformation node within the message
flow. To keep things simple, the parser for a domain has the same name as the domain effectively
domain and parser mean the same thing.
The recommended domain for XML documents is XMLNSC. This has high performance and creates a
compact message tree. From 6.1 it is optionally model-driven and so can validate XML documents
according to XML Schema 1.0 rules.
The recommended domain for non-XML messages is MRM which has a general purpose parser capable
of parsing a wide range of binary and text formats. It is model-driven and so can validate messages. It
also understands XML documents, so it is truly general purpose!
The recommended domain for SOAP based web services from 6.1 is SOAP. This also handles SOAP
with Attachments (SwA) and MTOM encodings and understands WS-Security and WS-Addressing
headers.
When communicating with the new JCA-based WebSphere Adapters from 6.1, use the DataObject
domain to parse and write the business objects used by the adapters.
The BLOB domain can be used when the message is being treated as opaque, and just being routed.
Effectively the message is not parsed at all.
If you are parsing XML documents and need the message tree to conform exactly to the XML infoset
(and so be fully XPath compliant) you can use the XMLNS domain. This does not use a model and so
can not validate.
Use the MIME domain for MIME messages other than SwA (eg, RosettaNet).
Alternatively you can write your own parser for use by your own user-defined domain.
Page 8
1. Most messages are not self-defining
Often a model is needed to parse the message bit stream
2. Validation
A model is needed to check message correctness when parsing
3. Speeds development of transformations
Provides source and target for graphical mappings
Provides assistance when editing transformation programs
4. Version control
By storing in a central shared repository
5. Instant documentation
For programmers, analysts and integration specialists
Runtime use of
model by parser
Design time use of
model by toolkit
Why model messages?
Page 9
Notes Why model messages?
There are several reasons why you might need to model your messages. The first two
are uses of the model by a parser at runtime. The others are uses of the model at design
time, and are independent of whether the model is actually used by a parser.
1. The majority of messages are not what is termed self-defining. Think of a message
created by a C or COBOL program its just a stream of binary data. Without some
intelligence to interpret it, it is meaningless, and thats where the model comes in. XML
is the opposite it is so verbose that an XML parser can parse any XML document
without using a model at all.
2. If you want to validate that your messages are correctly structured you need a model.
An XML parser can parse any XML document but it can only check it is correct if it has
a model to guide it.
3. A model can speed up the development of transformations enormously. For example,
graphical mapping from a source message to a target message is not possible without
a model. If you are transforming XML documents in this manner, you need a model
even if the XML parser chooses not to use a model at runtime.
4. Models provide a good way of tracking different versions of your messages. A COBOL
programmer would typically create a new version of a copybook each time he made a
change, using a configuration control system. The same principle applies to message
models.
5. A model provides documentation for a message that can be shared between
programmers, business analysts and integration specialists.
Page 10
Creating & using message sets
Generators
Message
set
WSDL
Message Message
Brokers Brokers
Toolkit Toolkit
Message Message
Broker Broker
Runtime Runtime
Mapper
Model Model
MRM
Parser
XMLNSC
Parser
Domain: MRM
Set: Person Defs
Type: Person
Format: COBOL1
Domain: XMLNSC
Set: Person Defs
Type:
Format:
Importers
COBOL C XSD WSDL
Adapter
ESQL
Message
dictionary
WSDLs
Zip of
XSDs and
WSDLs
Message
dictionary
XSDs
Page 11
Notes Creating & using message sets
Message models live in a container called a message set. A message set is a
special folder inside a message set project within the Message Brokers Toolkit
workspace. The typical sequence of events when creating and using a message
set is as follows.
First you create a message set project and message set using the wizard. This will ask you what kinds of message
format you are working with, and set the message domain of the message set for you (which you can change later if
you want).
Next you import application message formats described by XML DTD, XML Schema, WSDL types, C structures or
COBOL structures, which create and populate message definition files (.mxsd). These are annotated XML
Schema files more on this later.
Alternatively, you can create an empty message definition file and create your message model using just the editor.
If the message models in the message set are to be referenced by the Mapping
editor or the ESQL editor, you must set the message flow project so that it
references the message set project. This is done using the File -> Properties
menu.
If the message models in the message set are to be used by other tools to create
Web Services, you can generate WSDL from the message set using a wizard.
If the message models in the message set are to be used by a parser at runtime,
you must deploy them to the broker. It is the message set that is the unit of
deployment. You do this by adding the message set to a broker archive (.bar) file.
If the message set is to be used by the MRM parser, it is generated into a compact form called a message dictionary.
If the message set is to be used by the XMLNSC, SOAP or DataObject parsers, it is generated into pure XML
Schema (xsd) files which are zipped.
If the message set is to be used by the SOAP parser, WSDLs are added to the zip.
Page 12
Element
Element
Substitution Group
Message
1
Element content
n
n
Attribute
Group
n
Attribute
1
n
Attribute content
Wildcard
Attribute
Wildcard
n
1
1
Value
Constraint
n
Simple Type
Complex
Type
Group
n
n
1
Element
1
Reference
Contains
Logical Message Model
Page 13
Notes Logical Message Model #1
It was noted earlier that a message definition file is actually an annotated XML Schema file.
Thats because the WebSphere Message Brokers message model is XML Schema, with a few
extensions and restrictions. It turns out that XML Schema is great at expressing the logical
structure of all kinds of data, not just XML, and of course its an industry standard. The
following is a quick summary of the key features of the logical message model:
Each data value in a message must be modeled by a Simple Type. XML Schema supplies many built-in Simple Types and you can
create your own by restricting a built-in Simple Type using Value Constraints.
To refer to a data value in the message tree, you need to give it a name. Thats what an Element provides, a way of naming a data
value. An Element therefore refers to a Simple Type.
As well as data values, a message might have named structures. Elements therefore can alternatively refer to a Complex Type that
defines the structure.
The actual content of a Complex Type is described by a Group. A Group typically contains several Elements. A Group can also contain
other Groups directly.
In WebSphere Message Broker, we need to add an additional concept that is not in XML Schema, that of a Message. A Message is
nothing more than an Element which describes an entire message.
Simple Types, Complex Types, Elements and Groups can be local or global. A global object
can be reused from many places. A local object can only be used at the point it is defined.
Elements always have names, whether they are local or global. Simple Types, Complex
Types and Groups only have names if they are global.
To use a global Element or global Group within a Group, an Element Reference or Group
Reference is used. These refer to the Element or Group by name.
A local Element, a local Group, an Element Reference and a Group Reference can be defined
as mandatory, optional or repeating, using minOccurs and maxOccurs properties.
The content of a Group can either be an ordered list of objects (sequence), an unordered list
of objects (all or unordered), or a choice of one of a list of objects (choice), as described by the
Groups composition property.
Page 14
Notes Logical Message Model #2
If you are modeling XML data, there are some additional objects and concepts that you might
require.
An XML attribute is modeled by an Attribute which must refer to a Simple Type. Attributes can also be gathered together into Attribute
Groups
Both Attributes and Attribute Groups are used directly by a Complex Type, that is, they are not part of the Group that defines the main
content of the Complex Type.
A Group can also contain Element Wildcards. These are used when the content of the Group is not known at design time.
A Complex Type can also contain Attribute Wildcards. These are used when the attributes are not known at design time.
Elements can belong to something called a Substitution Group. This is an advanced XML Schema concept and will not be described
further here.
Simple Types of variety list and union are supported when modeling XML.
XML namespaces are supported.
XML Schema Unique, Key and KeyRef objects are imported but ignored.
Objects in one mxsd file can refer to objects in another mxsd file in the same message set
using XML Schema Include or Import. Redefine is not supported.
Page 15
MRM Domain
Flexible, model-driven parser
Supports binary messages
C, COBOL, PL/1, etc, structures
Supports text messages
Industry standards such as SWIFT, X12, HL7, FIX
Common text formats such as CSV
Supports XML messages
Use to generate equivalent XML document
MRM needs both logical and physical
information
Add a physical format to a message set
Provides a rich set of properties capable of representing most kinds of
data
Held as annotations to logical model objects
<Person>
<Age>32</Age>
<Height>172.5</Height>
<Name>Joe Bloggs</Name>
</Person>
032 1725 Joe Bloggs
32,172.5,Joe Bloggs
{A:32;H:172.5;N:Joe Bloggs}
CWF
XML
TDS
Page 16
Notes - MRM domain
The most flexible domain is called MRM which has a general purpose model-driven parser.
For example, the MRM parser can perform runtime validation of messages against the
model.
The MRM domain supports modeling binary messages from applications written in C,
COBOL, PL/1 and other languages. This support includes the ability to create a message
model directly from a C header file or COBOL copybook.
The MRM domain supports modeling text messages, perhaps with field content identified by
tags or separated by specific delimiters or both. This includes industry standards such as
SWIFT, EDIFACT, X12, HL7 and FIX, and commonly used text messages such as Comma
Separated Values (CSV).
The MRM domain supports modeling XML messages, including those that exploit XML
namespaces. From 6.1, the improvements to the XMLNSC domain in terms of performance
and XML Schema 1.0 validation mean the primary use case for MRM XML is to generate an
XML equivalent of a CWF or TDS message.
The MRM parser needs to know not only the logical structure of a message, as defined by
the logical model, but also how to create that logical structure from a physical stream of
bytes. You do this by adding a physical format to a message set. A physical format provides
a rich set of properties that can be set to represent most kinds of non-XML and XML data.
Binary messages are modeled using a physical format called Custom Wire Format (CWF).
Text messages are modeled using a physical format called Tagged/Delimited String Format (TDS).
XML messages are modeled using a physical format called XML Wire Format (XML).
Physical format properties are held as XML Schema annotations to logical model objects.
In the remainder of this session, we will explore how to model non-XML messages using
MRM domain. We will ignore XML, as these days the majority of XML modelers will create
their model in XML Schema (xsd) form and simply import into a message set.
Modeling MRM messages
Page 18
Approaching Message Modeling
Message modeling is like programming
You can read up on the theory
You can learn how to use the message editor
The hard part is knowing how to structure your message model
Knowledge
Knowledge
Wisdom
Wisdom
X
Page 19
Notes Approaching Message Modeling
Message modeling has a strong analogy with programming. Lets say you want to learn a new
programming language and write a program to solve a business problem. Take Java as an
example. You buy a Java book and read up on the language theory. You get hold of a good Java
editor and learn how to use it. But the hardest part is taking your business problem and working
out how to structure the program that will solve it. Very often you will look at examples created by
other programmers in order to get a head start.
With message modeling its the same. You can learn the theory and you can learn how to use the
message editor. But the hardest part is looking at an actual message and working out how to go
about modeling it.
If you are lucky the problem is solved in whole or in part by already having a model of the
message in one format or another (metadata). If you are even luckier that format will be one for
which an importer exists, which will create the message model for you automatically. Well cover
importers in this session.
The real fun starts when you have no model to reference. All you have is one or more example
messages. This most frequently occurs for formatted text messages, such as Comma Separated
Values (CSV) messages. Well show how to analyse messages in order to understand the
message structure and create the corresponding message model, using CSV as our example.
Page 20
Three Key Tasks
2. Understand the logical
structure
Enables creation of the logical model
3. Relate the physical format to the logical
structure
Enables setting of the model physical properties
01 PERSON.
05 AGE PIC 999;
05 HEIGHT PIC 999V9;
05 NAME PIC X(48);
032 1725 Joe Bloggs
1. Understand the physical format
COBOL, C, CSV, SWIFT etc
CWF or TDS
Msg
Person
Type
Person
Element
Age
Element
Height
Element
Name
Type
INTEGER
Type
DECIMAL
Type
STRING
32,172.5,Joe Bloggs
{A:32;H:172.5;N:Joe Bloggs}
Page 21
Notes Three Key Tasks
The first task is to understand the physical format of the message.
Where does this message come from? Does it conform to an
industry standard, or perhaps a well-known format like CSV? Is it
from a C or COBOL application? Which of the MRMs physical
formats should be used to describe it CWF or TDS?
The second task is to understand the logical structure of your
message. This determines the shape of your logical model, which in
turn dictates the shape of the brokers logical message tree.
The third task is the hardest one; relating the physical format to the
logical model. It is this task that determines the physical model and
provides values for all the CWF, TDS or XML physical properties.
Page 22
1. Understanding the Physical Format
For messages with just binary data use the CWF
format
Examples are COBOL, C, PL/1
Importers are available for COBOL and C structures
These automatically populate CWF format properties
For messages with text markup use the TDS format
Examples are SWIFT, HL7, CSV
IBM sells message sets for several industry standard formats
SWIFT, EDIFACT, X12, FIX, TLOG, HL7
Samples provided for CSV messages
For mixed binary and text messages use the TDS
format
From 6.1, TDS also handles binary data and binary markup
Page 23
2. Understanding the Logical Structure
1. Identify complex structures
Provides your
Complex Types
Groups
Complex Elements
2. Identify simple items
Provides your
Simple Types
Simple Elements
Attributes
3. Identify structure ordering
Provides your
composition (Sequence, Choice, All, Unordered)
4. Identify structures and items that repeat
Provides your
Element minOccurs & maxOccurs
Page 24
Notes Understanding the Logical Structure
There are four stages to understanding the logical structure of your message
1. Identify the complex structures. These correspond to the Complex Types, and the Groups used
within these Complex Types. There will be an overall Complex Type for the message itself. If the
message contains sub-structures, there will be a Complex Type per sub-structure. For example,
each parent level in a COBOL copybook, or each different row in a CSV message, corresponds
to an Element of Complex Type.
2. Identify the simple items within each Complex Type and Group and their logical data type. These
correspond to simple Elements and Attributes. For example each field in a COBOL copybook
with a PIC clause, or each comma separated value in a CSV message, corresponds to an
Element of Simple Type.
3. Identify the ordering rules of the items within each complex structure. This provides the value of
the composition property for the Complex Type or Group. Typically the composition is just a
sequence of one or more items (sequence). But sometimes the rule is that only one of several
items can occur (choice), examples being C unions and COBOL REDEFINES. Perhaps the
composition is a sequence of items but on closer examination the order of the items is not fixed
(all or unorderedSet)?
4. Identify which complex structures and which simple items repeat. This provides the cardinality
rules for your Elements, that is, the values for the minOccurs and maxOccurs logical properties.
Is an Element mandatory (minOccurs != 0) or optional (minOccurs = 0). Are there a fixed number
of occurrences of the Element (minOccurs = maxOccurs) or can the number vary (minOccurs !=
maxOccurs)? If the number can vary can there be an unlimited number (maxOccurs = -1)?
Page 25
3a. Relating CWF data to the Logical Model
For simple Elements
Identify the physical characteristics of each data item
Physical type is the key property
Strings: Fixed length, Null terminated, Length encoded
Numbers: Binary integer, Packed decimal, External (zoned) decimal, IEEE Float
Length count and Length units
Justification and Padding character
Signed, Sign orientation, Virtual decimal point
For simple and complex Elements
Byte alignment
Repeat reference
For COBOL and C the importers create both logical and physical
models
For PL/1 consider mapping to equivalent C structure then fixing up
Page 26
Notes Relating CWF data to the Logical
Model
For a CWF message there is nothing in the message data itself that identifies the individual
fields or even the structures to which these fields belong. Fortunately most CWF messages you
will encounter in the real world will be described by external metadata, for example a C header
file, a COBOL copybook or PL/1 include file.
For each Element of simple type in the logical model there is an allowable set of CWF Physical
types. The key task is to identify this physical type as it also determines what other CWF
properties need to be set. With COBOL for example, :
PIC X(20) is a fixed length string, left-justified with a length of 20 bytes.
PIC S9(9) is an extended (zoned) decimal, right-justified with a length of 9 bytes and signed.
Because compilers align fields of certain data types on specified memory boundaries, it is
important to also set correctly the CWF Byte alignment rules for each Element and Attribute.
This applies to both simple and complex Elements.
For CWF the number of repeats must be known in advance, the only exception being when the
number of repeats is specified by a field earlier in the message. This is called a Repeat
reference and can be specified instead of the logical maxOccurs property.
The good news is that for the most commonly encountered CWF messages, in practice from
COBOL and C applications, importers are provided that automatically create the logical and
physical models, so you dont have to do any of this .
For PL/1, there is no importer (yet), so consider creating the equivalent C or COBOL structure,
importing that, and fixing it up post-import using the editor. Specifically, check Byte alignment
and change Physical type of variable length strings to Length encoded string 2.
Page 27
3b. Relating TDS to the Logical Model
For Complex Types and Groups
Data element separation of each structure is key property
Determine this by looking at the markup in the message
Does each data item have a Tag?
If so is the Tag fixed length or is there a Tag data separator?
Are the data items fixed length or separated by a Delimiter?
Does the structure itself have a Group indicator (GI) or terminator (GT)?
For simple Elements
Identify the physical characteristics of each data item
Length if used
Tag if used
Justification and Padding character
Sign orientation, Virtual decimal point, Precision
For simple and complex Elements
Repeat reference, Repeating element delimiter (RED)
Page 28
Notes Relating TDS to the Logical Model
TDS messages vary widely. There may, or may not, be something in the message data itself that identifies
the individual fields. There may, or may not, be external metadata that describes the structure of the
message (such as a specification). Well work on the assumption that metadata is not available, and that all
we have to go on are example instances of the messages we want to model.
The key step is to identify the TDS Data element separation property. To answer this question you have to
look at the markup in the message. Markup is extra message content that assists with the identification or
parsing of the data. Does each data item have a Tag ? If so is the Tag fixed length or is there a Tag data
separator ? Are the data items fixed length or separated by a Delimiter ? Does the structure itself have a
Group indicator (GI) or Group terminator (GT) ?
For TDS each Element and Attribute of simple type in the logical model is physically a string in the
message. So there is not the same concept of physical type like there is with CWF. But there are still TDS
properties to set that describe how these physical strings appear in the message data.
Certain Data element separations need to know the number of repeats in advance, while others do not.
Those that do need to know use maxOccurs and minOccurs directly or a Repeat reference if the number of
repeats is specified by a field earlier in the message. Additionally, where Delimiters are involved without
Tags, an additional Repeating element delimiter property is needed in order to parse repeating items.
The good news is that for the some of the commonly encountered industry text standards, IBM supplies
pre-packaged message sets which contain the logical and physical models, so you dont have to do any of
this . Theres also a set of samples for Comma-Separated Value messages (CSV) which you can use as
a starting point for your own CSV messages. For non-standard formats you have to create the model
yourself using either the message definition editor or SupportPac IA7D Importer for formatted text
message models.
Text message formats
Page 30
TDS Messages - Example #1
{A:32,H:172.5,N:Joe Bloggs}
A N 172.5 Joe Bloggs { } : , : : H 32 ,
Group
Terminator
Group
Indicator
Data
Tag
Tag Data
Separator
Delimiter
?
Tagged
Delimited
Tagged
Delimited
Page 31
32,172,Joe Bloggs
CSV
CSV
Data
Delimiter
Joe Bloggs 172.5 , 32 ,
?
All Elements
Delimited
All Elements
Delimited
Page 32
32,172.5,Joe
Bloggs<CRLF>
111,104.0,Bilbo Baggins
CSV
CSV
111 Bilbo Baggins 104.0 , ,
Joe Bloggs 172.5 , 32 , CRLF
Repeating
Element
Delimiter
All Elements
Delimited
All Elements
Delimited
How many
different
complex
types?
How many
different
complex
types?
2
2
Page 33
Notes - TDS Messages - Examples
The previous foils show two different example TDS Person Complex Types.
The first example contains data, Tags and Delimiters. The Tags in this example are separated from the data by
a Tag data separator. We would expect to use a Data element separation of Tagged Delimited to model such a
Complex Type. Note that the Person child data is wrapped by a GI and a GT. The use of these is optional. In
fact GI and GT can be used with all Data element separations.
The second example contains just data and Delimiters. It is an example of a Comma Separated Values (CSV)
message. This is a widely used format, for example it is an export format from databases and spreadsheets. The
data is considered to be a row and each simple value is a column. We would expect to use a Data element
separation of All Elements Delimited to model such a Complex Type.
The third example shows the most typical example of a CSV message, where the row structure repeats multiple
times, each instance separated by <CRLF>. This separator is called a Repeating element delimiter (RED).
It is important to notice that there are two complex structures in the third example! The first is the Person
Complex Type itself, the second is the People Complex Type for the overall message, containing two
occurrences of the Person Complex Type. We would expect to use a Data element separation of All Elements
Delimited to model the overall message Complex Type as well.
Also notice that <CRLF> has been explicitly shown, but you would not see this quite so obviously in a text editor!
Always make sure that you understand the white space in your text messages!
We will look at several different CSV messages in more detail later and see how they can be modeled using
TDS.
Page 34
TDS Algorithm
Tags? Unique?
Y
All
Elements
Delimited
All
Elements
Delimited
Y
All
fields?
Y
Tagged
Delimited
Tagged
Delimited
Y
Tagged
Fixed
Length
Tagged
Fixed
Length
Y
Tagged
Encoded
Length
Tagged
Encoded
Length
Y
Var Len
Elements
Delimited
Var Len
Elements
Delimited
Y
Fixed
Length
Fixed
Length
Y
Delimited?
Y
Fixed
lengths?
N
Encoded
lengths?
N
Use Data
Pattern
Use Data
Pattern
N
N
N Variable
fields?
N
Fixed
lengths?
N
N
GI
Delimited?
N
A:32,H:172.5,N:Joe Bloggs
A:03032H:05172.5N:10Joe Bloggs
A:032H:172.5N:Joe Bloggs
32,172.5,Joe Bloggs
032172.5Joe Bloggs
032172.5,Joe Bloggs
Page 35
Notes - TDS Algorithm
This foil provides an algorithm which can be applied to the physical modeling of TDS complex
structures in order to deduce the Data element separation.
1. Start off looking for Tags. If the data contains Tags and the Tags are unique for each field, then
the chances are it can be modeled using one of the Tagged Data element separations. If the
data is separated by Delimiters use Tagged Delimited, if the data is fixed length use Tagged
Fixed Length or if the data contains an embedded length use Tagged Encoded Length.
2. If the Tags are not unique, then consider whether they could be treated as Group indicators.
Remember that GIs can be used with any Data element separation.
3. If the data does not contain unique Tags, then see whether the data is separated by Delimiters.
If not and the data is fixed length use Fixed Length.
4. If all the fields are delimited use All Elements Delimited, if just varying length fields are delimited
and the remainder are fixed length use Variable Length Elements Delimited.
5. If none of the above quite manages to model the structure, then you should fall back to Use Data
Pattern. A data pattern is a regular expression which is dynamically matched against the data in
the message to determine what it is. It is often the only method of modeling certain types of
structure. For example, a numeric field might be variable length, but the end is only known
because the next field is alphabetic. A regular expression can be created that stops when all
digits have been consumed. Well also see data patterns when we look at CSV messages in
more detail later on.
Tip: If you encounter a stray piece of markup at the end of a structure, you may be able to treat
this as a Group terminator (GT). One common application of GTs is in a delimited structure
when the delimiters are post-fix as opposed to the more normal in-fix. The final delimiter can
often be modeled as a GT. Similarly, you can often treat a pre-fix delimiter as a Group indicator
(GI).
Scenario
Page 37
Scenario
COBOL app
COBOL app
01 CUSTOMERS.
05 CUSTOMER-DETAILS OCCURS 5 TIMES.
10 CUS-FIRST-NAME PIC X(10).
10 CUS-LAST-NAME PIC X(10).
10 CUS-STREET-ADDR PIC X(25).
10 CUS-CITY-NAME PIC X(20).
10 CUS-STATE-CD PIC X(2).
10 CUS-ZIP-CD PIC X(5).
10 CUS-POLICY-KEY PIC X(10).
10 CUS-LUPD-CNT PIC S9(4).
10 CUS-MR-FROM PIC S9(9).
10 CUS-MR-TO PIC S9(9).
01 CUSTOMERS.
CSV app
CSV app
Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686
Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432
Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456
Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045
Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686
Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686
Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432
Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456
Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045
Message
Broker
Message
Broker
CWF
TDS
Page 38
Notes Scenario
To provide some examples of message modeling we are going
to consider a simple integration scenario.
We have a COBOL application which emits a CUSTOMERS
message consisting of a number of customer details records.
We will model this message using CWF.
There is a requirement to take these messages and forward
them to an application that understands CSV messages,
perhaps to display them in a spreadsheet. We will model this
message using TDS.
This scenario enables us to demonstrate:
Creation of logical and CWF models using the COBOL importer
Creation of TDS models for CSV messages using the editor
Several variations of CSV message and how to go about modeling them using
TDS facilities
Page 39
COBOL message
Create a model using the COBOL copybook importer
01 CUSTOMERS.
01 CUSTOMERS.
Understand physical
Understand logical
Relate physical to logical
Understand physical
Understand logical
Page 40
Notes COBOL message
We will use the COBOL importer to create a message model from the COBOL copybook shown.
We can see from the metadata that the logical structure of the CUSTOMERS message is a
Complex Type that contains a single child Element called CUSTOMER-DETAILS which repeats
exactly five times. CUSTOMER-DETAILS is itself complex, and its Complex Type contains a
sequence of ten simple Elements, being a mixture of strings and integers, none of which repeat.
Firstly we create a message set project and message set to hold the model, using the New
Message Set wizard. This asks you for the kind of data you will be modeling, so select binary
data. This automatically creates a CWF physical format. This ensures that the importer will create
a physical model as well as a logical model.
The COBOL importer is invoked via the New Message Definition File From COBOL wizard. Once
the COBOL copybook and target message set have been selected, a pre-import check takes
place to ensure that the copybook is syntactically correct and to analyze the copybook for level 01
structures. You then select the structures you want to import. By default a level 01 structure is
imported as a Complex Type. You can choose which of your selected structures to make into
actual Messages. The wizard also lets you choose some extra options, such as which compiler
originally compiled the copybook, to ensure that your created model exactly matches the real-
world message.
Page 41
Notes COBOL message
Once the import is complete, a new message definition file called Customers will have
been created. Opening this shows the message model. You can see how the logical
structure mirrors the original copybook, and you can examine the Element CWF property
values and relate them back to the COBOL field definition PIC clauses.
A command line version of the importer is also supplied, called mqsicreatemsgdefs. This is
capable of bulk importing several copybooks at once.
The C importer is very similar in operation.
You will notice that the wizard lets you choose a target namespace for your message
definition file. Why might we want to specify a target namespace for a COBOL message?
The MRM parser, when parsing messages that are defined in a message definition file that
has a target namespace, produces a logical message tree containing both name and
namespace information. It does this regardless of the physical format of the message. For
CWF and TDS messages, the namespace is obtained from the message definition file. If
you will be transforming a COBOL message into a namespace-aware XML message, the
transform must map the logical message tree created for the COBOL message to a logical
message tree that matches the XML message. If the COBOL message tree does not
contain namespace information then each mapping from a COBOL field to an XML element
must set the namespace for the XML element. However, if the COBOL message tree
already contains the same namespace information then this mapping is greatly simplified.
Page 42
CSV message
Create an equivalent CSV physical model
Re-use the same logical structure
Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF>
Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF>
Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF>
Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF>
Delimiter
Repeating
Element
Delimiter
All
Elements
Delimited
All
Elements
Delimited
Occurs=5
Understand physical
Understand logical
Understand physical
Understand logical
Page 43
Notes CSV message
To exchange data with the CSV application we need an equivalent CSV message. We dont have any metadata to
describe this, but we do have example instances of what the messages must look like.
We know that we will model CSV using TDS. We can also see that the logical structure of the CSV message is the
same as the COBOL message. This simplifies our task as we can simply add a TDS physical format to our
existing message model and then populate its properties.
Being a CSV message we can see the expected Delimiters and also Repeating element delimiters. Note that if we
were to view this data in an editor we would not see <CRLF>, so it is easy to ignore this markup accidentally!
Because only data, Delimiters and Repeating element delimiters are involved, the appropriate TDS Data element
separation for a CSV row is All Elements Delimited so we can set the CUSTOMER-DETAILS Complex Type TDS
property accordingly. We can also set the Complex Type TDS Delimiter property to be comma, and we can set the
CUSTOMER-DETAILS Element TDS Repeating Element Delimiter property to <CR><LF> (TDS supplies
mnemonics for special characters).
Similarly the appropriate Data element separation for the overall CSV message is All Elements Delimited so we
can set the CUSTOMERS Complex Type TDS property accordingly. We can also set the CUSTOMERS Complex
Type TDS Delimiter property to be <CR><LF>. Interestingly, because the CUSTOMERS Complex Type has only
one child Element, and Delimiters in CSV are in-fix (meaning they only occur between Elements) there is no
example of the CUSTOMERS Delimiter in the message data. But because this is a CSV message we know that
the Delimiter should be <CR><LF> (well see this on the next foil).
In 6.1, setting the TDS message set property Messaging Standard to CSV will automatically set many of the
properties mentioned above.
Page 44
CSV message #2
Add Header and Trailer CSV
rows
#CUST1,2005-06-01<CRLF>
Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF>
5,349634
#CUST1,2005-06-01<CRLF>
5,349634
Delimiter
Delimiter
All
Elements
Delimited
Occurs=5
CLASH
Understand physical
Understand logical
Understand physical
Understand logical
Delimiter
Delimiter
All
Elements
Delimited
All
Elements
Delimited
All
Elements
Delimited
Why is the
delimiter
clash not a
problem?
Why is the
delimiter
clash not a
problem?
Page 45
Notes CSV message #2
The CSV message we have been looking at consists of a single row that repeats five times.
Commonly CSV messages start with a Header row and finish with a Trailer row so lets add these
to our message.
Clearly adding a Header and a Trailer changes the logical model. We need to create a Complex
Type for the Header and a Complex Type for the Trailer, then create Header and Trailer Elements
that use those Complex Types and insert them into the appropriate place in the overall
CUSTOMER Complex Type. In fact, rather than disturb the original CUSTOMERS Complex Type,
we will create a new Complex Type called CUSTOMERS2, to consist of the Header Element, the
CUSTOMER_DETAILS Element (repeating five times) and the Trailer Element.
Because the Header and Trailer Complex Types are just examples of CSV rows we will model
them in TDS in exactly the same way we did for CUSTOMER_DETAILS, namely All Elements
Delimited with a Delimiter of comma. The overall CUSTOMERS2 Complex Type is modeled in
TDS as All Elements Delimited with a Delimiter of <CR><LF>, exactly like CUSTOMERS.
In the message data we can now see <CRLF> in use as both a Delimiter and a Repeating
element delimiter. We call this a delimiter clash. Most text based messaging standards
deliberately avoid clashes like this for clarity reasons, but with CSV it is unavoidable. In the
message shown the clash is not a problem, because the MRM parser knows to expect exactly
one Header, exactly five CUSTOMER_DETAILS, and exactly one Trailer. In other words, the
logical Max occurs property enables the MRM parser to unambiguously interpret each <CRLF> it
encounters.
Page 46
CSV message #3
Number of CUSTOMER-DETAILS records can
vary
Clash is now a problem use Tags
Header,#CUST1,2005-06-01<CRLF>
Cust,Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF>
Cust,Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF>
Cust,Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF>
Cust,Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF>
Cust,Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF>
Trailer,5,349634
Header,#CUST1,2005-06-01<CRLF>
Cust,Bob,Dylan,1 Desolation Row,Orlando,FL,32802,HYT-457-AX,340,67542,97686<CRLF>
Cust,Jimmy,Page,1002 Misty Mountain Drive,New York,NY,10001,ADU-239-BX,7652,238,432<CRLF>
Cust,Paul,McCartney,67 Penny Lane,Las Vegas,NV,89125,JUI-854-CF,1673,-99678,-23456<CRLF>
Cust,Elton,John,3 Yellow Brick Road,Chicago,IL,60699,GGY-118-AS,443,123,-100045<CRLF>
Cust,Eric,Clapton,461 Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,97686<CRLF>
Trailer,5,349634
Unbounded
Tag
Tag Data
Separator
Tagged
Delimited
CLASH
Understand physical
Understand logical
Understand physical
Understand logical
All
Elements
Delimited
All
Elements
Delimited
All
Elements
Delimited
Page 47
The Delimiter clash becomes a problem when the value of minOccurs and
maxOccurs for CUSTOMER_DETAILS are no longer the same. If we change the
message so the repeats of CUSTOMER_DETAILS become unbounded, the MRM
parser is no longer able to tell whether <CRLF> means the end of an individual
repeat or the end of the entire Complex Type .
One way to solve this is to see whether the CSV row itself gives us a clue as to what
type it is. In the message we can see that each row has now been appended with a
new first column value, which appears to be fixed for a given type of row. If we
choose to interpret this fixed value as a TDS Tag, instead of as data, then the MRM
parser can use this information to know which Complex Type it has encountered.
Note also that we interpret the comma after the fixed value as a Tag data separator
and not as a Delimiter.
We model this in TDS by giving each of the Header, CUSTOMER_DETAILS and
Trailer Elements an appropriate Tag property, and by changing the CUSTOMERS2
Complex Type to have a Data element separation of Tagged Delimited and a Tag
data separator property of comma. The Header, CUSTOMER_DETAILS and Trailer
Complex Types do not change; their Data element separations remain All Elements
Delimited, and no new Element is needed for the first column value because it is
being treated as markup and not as data.
Page 48
CSV message #4
Number of CUSTOMER-DETAILS records can
vary
Clash is now a problem use Data Patterns
#CUST1,2005-06-01<CRLF>
5,349634
#CUST1,2005-06-01<CRLF>
5,349634
Unbounded
CLASH
[0-9].*
[^0-9].*\r\n
#.*\r\n
No
Tags!
Understand physical
Understand logical
Understand physical
Understand logical
All
Elements
Delimited
All
Elements
Delimited
All
Elements
Delimited
Use Data
Pattern
Group
Terminator
Group
Terminator
Page 49
If we cannot use Tags in this manner, because all the CSV rows do not contain a fixed
value in their first column, then we have to use data patterns.
To use this technique, we must look at the message data in the Header row, the
CUSTOMER_DETAILS row and the Trailer row and work out a regular expression for
each that a) distinguishes the row apart, and b) covers all the data and markup in the
row.
For Header we can use a regular expression of #.*\r\n which means any sequence of characters that starts with # and
ends with <CRLF>.
For Trailer we can use a regular expression of [0-9].* which means any sequence of characters that starts with a
digit.
For CUSTOMER_DETAILS we can use a regular expression of [^0-9].*\r\n which means any sequence of characters
that does not start with a digit and end with <CRLF>.
This works because we know the CUSTOMER_DETAILS data can never start with #
and the Trailer data can never start with something other than a digit.
We model this in TDS by giving each of the Header, CUSTOMER_DETAILS and Trailer
Elements the appropriate TDS Data pattern property, and by changing the
CUSTOMERS2 Complex Type to have a Data element separation of Use Data Pattern.
Note also that a side-effect is that the <CR><LF> is longer treated as a delimiter, and
must therefore be modeled as a Group terminator. So while the Header,
CUSTOMER_DETAILS and Trailer Complex Types do not change their Data element
separations (they remain All Elements Delimited), the Header and
CUSTOMER_DETAILS Complex Types must have Group terminator set to <CR><LF>.
It should be noted that when using data patterns to parse a message, the MRM parser
does not use maxOccurs, it continues to consume the message until the regular
expression fails to match.
Page 50
Notes CSV message #4 - Variation
If we look again at our example message, the data pattern technique is only actually
needed to distinguish Trailer from CUSTOMER_DETAILS. An alternative logical model is
shown below.
Here, the overall CUSTOMERS2 Complex Type remains with Data element separation All
Elements Delimited, but we have changed the logical structure by inserting a local
anonymous sequence with a Data element separation of Use Data Pattern purely to parse
CUSTOMER_DETAILS and Trailer. This is a good idea if there are several other child
Elements prior to CUSTOMER_DETAILS in the CUSTOMERS Complex Type, as it
avoids having to think up unique regular expressions for all the other child Elements
(which might not be possible). Its also a good idea from a performance point of view,
because Use Data Pattern is the slowest Data element separation in TDS.
This is just one example of inserting a Group to scope and solve a TDS parsing problem.
Its a very useful and powerful technique, can be used with any TDS Data element
separation, has no effect on the brokers logical message tree, and is sometimes the only
way to model the messageas we shall see on the next foil.
Use Data
Pattern
All
Elements
Delimited
All
Elements
Delimited
All
Elements
Delimited
All
Elements
Delimited
Page 51
CSV message #5
What if a field contains the Delimiter as part of the data?
6.1: Handled automatically if TDS Quote Character selected
Pre 6.1: Use Data Pattern to parse the field
#CUST1,2005-06-01<CRLF>
Eric,Clapton,461, Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF>
5,349634
#CUST1,2005-06-01<CRLF>
Eric,Clapton,461, Ocean Boulevard,Miami,FL,33101,LOP-212-TY,2134,67542,976865<CRLF>
5,349634
([^]*) | ([^,\r\n]*)
Use Data
Pattern
Understand physical
Understand logical
Understand physical
Understand logical
Page 52
Lets look at one final variation on the CSV theme. It can arise that a column value validly contains a
comma. When this happens the usual action is for the value to be placed in double quotes, which acts
as an escape mechanism to prevent the interpretation of the embedded comma as a Delimiter. In our
message, this has happened with the CUS_STREET_ADDR Element in the final repeat of
CUSTOMER_DETAILS.
TDS provides an escape scheme for escaping individual characters. In 6.1 this allows quotes to be
designated as the escape scheme, so in 6.1 the fact that CSV data is quoted is not a problem and the
parser handles it seamlessly. However, prior to 6.1, escape schemes did not handle quotes. To solve
this problem we once again use a Data pattern.
We need to provide a regular expression that accepts either an unquoted value that contains no
comma, or a quoted value that can contain a comma. Such a regular expression would be ([^]*) |
([^,\r\n]*) which means either any sequence of characters except quote preceded and followed by a
quote, or any sequence of characters that does not include the comma or <CRLF>.
The CUSTOMER_DETAILS Complex Type remains with Data element separation All Elements
Delimited, but we change the logical structure by inserting a local anonymous sequence with a Data
element separation of Use Data Pattern purely to parse CUS_STREET_ADDR, which is given the
Data pattern above.
The model shown enables the parsing of quoted and unquoted values, but it does not cope with extra
quotes that validly occur in data values nor does it strip off the quotes before inserting the value in the
message tree. Both these can be achieved by using a further embedded group and more complex
regular expressions combined with a GI and a GT.
Finally, the Samples Gallery from 6.0 onwards contains a sample message set that illustrates many of
the CSV variations described here.
Best Practices :
Diagnosing modeling
problems
Page 54
Diagnosis tools
Debugger
Allows single-stepping through nodes and ESQL
Displays the message tree at a breakpoint
Warning: Message tree may not be an exact representation of the real tree
Trace node
Provides a 100% accurate representation of the message tree
Can write to file or user trace
User Trace
Shows the full text of all error messages
Logs TDS parser activity
Can contain Trace node output
Page 55
TDS user trace output
Debugging TDS parsing problems
Can be quite difficult to debug a complex TDS model
What was the parser doing before it failed?
In v6.1 you can find out
User Trace ( debug level )
TDS parser issues user trace messages at important stages of the parse
UserTrace BIP5612I: ELEM1:', ending at byte 5', was matched as
the tag for '/myStructure/myElement'.
UserTrace BIP5609I: '#' has been matched as a delimiter for
'/myStructure' at byte '17'.
Page 56
Diagnosis Best Practices
Use the debugger as a first line of defence only
Debugger only tells you what happened
User Trace should be a standard diagnosis tool
User trace tells you why it happened
Be wary of the message tree displayed by the debugger
Use a Trace node if you want to be sure of an accurate picture
Trace nodes write to User Trace, so they make a good pair
Page 57
Best Practices : XML Parsing
XMLNSC
XMLNSC is the preferred XML parser for new flows
Validation is now available in the XMLNSC domain
Performance of XMLNSC is ~100% better than any other XML parser
( and sometimes better than that )
XMLNSC validation is far less costly in CPU than MRM validation
Opaque parsing
Performance feature now available in XMLNSC
Configurable via the input node properties
XML domain
XML domain is now deprecated
All existing message flows will continue to work in future versions.
( No function is actually being withdrawn )
Page 58
TDS enhancements in v6.1
New CSV Messaging Standard
Includes support for CSV-style quoting of strings
Tags / delimiters can contain raw byte values
Support for a wide range of CWF physical types
Repeat reference
Exactly like the CWF support
Automatic trimming of over-size Fixed Length data
Also added to CWF
Support for SAP text Idocs
IBM-supplied message for CC/DD message
C importer populates TDS physical format(!)
Note : v6.1 TDS is not a replacement for CWF
no support for alignment/padding of structures
Not all physical types are supported
Page 59
Summary
Understand the physical format
CWF or TDS
When you already have metadata
Use an importer if possible
When you dont or cant use an importer
Understand the logical model
Structures, simple items, composition, repeats
Relate the physical format to the logical model
For CWF, Physical type property is key
For TDS, markup is key to deciding Data element separation property
Be smart about diagnosis
Use User Trace / Trace node as well as debugger
Don't forget to
complete your
evaluation !
Session xxx
SundarGopal Venkatraman
sundaven@in.ibm.com
Session Number: Q30 Session Number: Q30
WebSphere Message Broker Administration Best Practices
Page 1
Objectives
This session discusses the common administration tasks and best practices for WebSphere
Message Broker.
Section at end suitable for advanced users
General Knowledge of WebSphere Message Broker is assumed
All statements will apply to V6 and V6.1 unless otherwise stated
Page 2
Acknowledgements
Matt Lucas (WMB Development)
Page 3
Administration Architecture
Common administrative actions and
how best to accomplish them
Power users section
Agenda
Page 4
Message
Broker Toolkit
(Admin
Perspective)
Message
Broker
Explorer (IS02)
Command line
Configuration Manager Proxy
Configuration
Manager Broker
Page 5
Message Broker Explorer (IS02)
Alternative MQ Explorer based administration
Simplifies administration of MQ,MB networks in single Explorer
console
Uses MQ Eclipse extension points to provide seamless
experience
Comprehensive administration facilities
All features in Broker Administration (excluding
publish/subscribe)
Also includes new features such as multi Execution Group
deploy
Offload WS-Security to Data Power appliances
IS02 Cat 3 SupportPac
Fully supported in production
Performance Monitor
Easily view CPU, IO and other metrics
Requires APAR IZ07803
Page 6
Comprehensive Java API for administering broker domains:
import com.ibm.broker.config.proxy.*;
public class CreateBrokerReference {
public static void main(String[] args) {
ConfigManagerProxy cmp =
ConfigManagerProxy.getInstance();
TopologyProxy topology = cmp.getTopology();
topology.createBroker(MYBROKER, QMGR);
}
}
Used for administering brokers, execution groups, message flows, dictionaries (and other deployed message
flow dependencies), subscriptions, the publish/subscribe topology, collectives, the event log, topics hierarchy
and the Configuration Manager
The Message Broker Toolkit Administration perspective, IS02 and various commandline utilities are
visualizations of this API.
Powerful samples available (e.g. Configuration Manager Proxy Exerciser)
Page 7
Configuration Manager Proxy API Exerciser
Page 8
Commands three types
Commands that perform direct administration
Tend to contain an unflagged component name parameter (e.g. mqsilist BROKER)
Run locally on the machine on which the component exists
Examples: mqsilist, mqsicreatebroker
Commands run in Eclipse headless mode
Parameters to specify workspace or project location
Examples: mqsicreatebar, mqsicreatemsgdefs
Commands based on the Configuration Manager Proxy
Parameters to specify location of Configuration Manager
Examples: mqsideploy, mqsicreateexecutiongroup
Page 9
Common Administrative Actions
Create/Delete components
Controlling access to resources
Versioning of resources
Configuring broker archive files
Deploying broker archive files
Operational Tweaks
Page 10
Naming Standards
Configuration Manager
Functional Domain name
Prefix/Suffix of QMGR Name ?
Prefix/Suffix CMGR ?
Broker
Functional Project/Application Name
Prefix/Suffix of QMGR Name ?
Prefix/Suffix BROKER ?
Execution Group
Application/Project Name
Prefix/Suffix of EG ?
Message Flow
Mediation/Application Name
Prefix/Suffix of FLOW ?
Ensure consistent naming convention
Page 11
How many
Configuration
Managers do you
need ?
Configuration Manager
How many brokers/environments and their complexity?
Are there different classes/types of applications
deployed?
Security and Isolation
Ease of administration
If each environment has multiple brokers, it is
recommended to have separate Configmgrs.
e.g. DEV, TEST, etc ..
Page 12
http://www.ibm.com/developerworks/websphere/library/techarticles/0706_lucas/0706_lucas.html
..Or Google Config Manager Secrets
Page 13
How many Execution
Groups per Broker
can you have ?
Complexity of message flows
Message sizes that are expected to be processed
Environment limitations e.g. hardware, operating
system
Are there different classes/types of applications
deployed ?
Ease of administration
Broker/Execution group/Message flow
How many Message
flows per execution
group can you have ?
Page 14
http://www-128.ibm.com/developerworks/websphere/library/techarticles/0311_dunn/dunn.html ..Or Google Message
Flow copies
Applicable to all
Broker versions
Page 15
MQ Configuration
Control access to the Configuration Manager and brokers queues and queue manager:
Who should have access to the Configuration Managers queues and queue manager?
Who should have access to the brokers queue manager? And broker system queues?
Use SSL between MBT and CM (and between Config Manager and brokers if necessary)
Consider using the same queue manager for both broker and Config Manager
This will minimize MQ administration, but having multiple CMs may increase administration cost
Configuration Manager ACLs
Runtime authentication / authorization / identity mapping with LDAP or TFIM (MB v6.1 only)
Message
Brokers
Toolkit
Configuration
Manager
Broker
EG
Page 16
Runtime versioning
Various pieces of information gathered at deployment time
The time the object was deployed
The time the object was last saved in the broker archive
The name of the broker archive file that deployed the object
Other user-defined keywords and their values
For instance, Author -> Prasad Imandi, Release name -> S000
Viewable in Message Broker Toolkit, mqsireadbar and CMP applications
e.g:
Page 17
Runtime versioning Embedding keywords
The Configuration Manager scans through each file looking for a predefined eye catcher that describes
each keyword and its value
Each time one of these strings appears in the file being deployed, it is associated with the Configuration
Managers record for that file
However, message flows and dictionaries have a freeform Version property, which means that for
simple versioning scenarios you dont need to remember the $MQSI..MQSI$ syntax:
$MQSI keyword = value MQSI$
Page 18
Embedding Version Information How to examples
Message flows:
Use the string field of a passthrough node
Use ESQL strings:
Set target = $MQSI target = production MQSI$
When using subflows, use a different version keyword in each subflow; this allows you to see version information for each
constituent part
Dictionaries
Use Version property only
Stylesheet
In a comment or unused field
<?xml version=1.0 encoding=UTF-8>

<! Binding - specifies persistence, message formats, and correlation style -->
<binding name="mq_insurance_bindings" type=tns:portType1">
<wmqservice:binding/>
<wmqservice:persistence>MQPER_NON_PERSISTENT</wmqservice:persistence>
<wmqservice:format>MQSTR</wmqservice:format>
<wmqservice:reportOptions>MQRO_PASS_MSG_ID</wmqservice:reportOptions>
<operation name=getQuote">
<input>
<wmqservice:body/>
</input>
<output>
<wmqservice:body/>
</output>
</operation>
</binding>
<service name=InsuranceServices">
<port name=mq_ports" binding=tns:mq_insurance_bindings">
<wmqservice:address location=" wmq:/msg/queue/INS.QUOTE.REQUEST"/>
<wmqservice:replyTo>msg/queue/INS.QUOTE.REPLY</wmqservice:replyTo>
</port>
</service>
</definitions>
2
1
3
Page 52
Example Native WMQ Request-Response MEP (WSDL)
This example shows a native MQ request/response application which processes an insurance quote request and
place a response on a reply queue
The service Interface (portType) and Message definitions are not shown in this example because the MQ Service
Definition specification does not extend or change WSDL's intrinsic XML Schema-based type system.
Note, this means that WMQ messages encoded in non-XML data e.g. COBOL copybooks must be described in WSDL
using XML schema. A standard set of schema annotations and rules - called Data Format Description Language (DFDL) - is
being developed to extend XML schema to non-XML formats and should be used once ratified, but until it is ratified,
proprietary schema annotation can be used (e.g. the ones used by WebSphere Message Broker).
Looking at the WSDL in more detail:
The service definition (1) contains the addresses of the request queue
(wmq:/msg/queue/INS.QUOTE.REQUEST) and response queue (msg/queue/INS.QUOTE.REPLY) and
defines the bindings to use (mq_insurance_bindings). The bindings section (2) is identified as an MQ
transport binding by the presence of the <wmqservice:binding/> element. It specifies that the application must be
accessed using the qualities of service defined by the elements shown in (3) i.e. Persistent messages, with an
MQ String format. It also specifies that the responding MQ Service Application correlates the response message
to the request message by copying the request MsgId into the Response MsgId (MQRO_PASS_MSG_ID)
The Application can either be described using the WSDL shown on the previous page, or alternatively it can be
completely described using the following IRI.
wmq:/msg/queue/INS.QUOTE.REQUEST?
&replyTo=msg/queue/INS.QUOTE.REPLY
&format=MQSTR
&persistence=MQPER_NON_PERSISTENT
&reportOptions=MQRO_PASS_MSG_ID
The IRI definition is a more compact version of the WSDL definition, but they are equivalent.
The presence of the replyTo parameter tells the invoking implementation to expect a response (In the IRI this
MEP could also be indicated by a msgType=MQMT_REQUEST parameter, in which case the replyTo parameter
could be omitted and the implementation would be expected to provide information about the replyTo destination
perhaps by using a temporary queue)
The format and persistence parameters specify that the messages are non-persistent text messages.
The reportOptions parameter specifies that the responding application should set the messageID of the response
to the messageID of the request message (in addition to the default behaviour of copying the request messageId
to the correlId of the response).
N
O
T
E
S
Page 53
Summary MQ, the Web, and Web Services
MQ can be used as a Web 2.0 enabler
Extending the reach of Enterprise applications to the Web
MQ HTTP can be used for Zero Footprint Clients
Server, Client , and Device applications can communicate over HTTP
No need to deploy client libraries
Extends MQ to many more client environments
MQ can be used as a Web services transport
Using SOAP over JMS standard, or by sending SOAP messages in raw MQ messages
As a integrated custom channel in the Windows Communication Foundation
WMQ based applications can be described and re-used in Service
Oriented Architectures
Describe WMQ applications in a standard way like other Web service assets using WSDL and
IRIs
Store in Service Registry, and use tools to access
Elmer Senson (sensonel@ph.ibm.com)
Original Slides by: Nancy Rosett (nrosett@us.ibm.com)
Session F08: WebSphere Transformation
Extender 8.2 Design Tools Lab
Page 1
Objectives
Overview of Websphere Transformation Extender 8.2
Setting up the WTX 8.2 environment
Defining the data
Creating a map
Objective 1:
Overview of Websphere Transformation Extender 8.2
Page 3
What is WebSphere Transformation Extender?
WebSphere Transformation Extender (WTX)is a unique
data integration technology for use in Application to
Application and Business to Business enablement.
What makes WTX different from all other types of
integration products on the market is its ability to speak
directly to native interfaces and APIs without the need for
coding or re-tooling.
Page 4
What does WebSphere TX do?
Its a complex transformation engine that extends the value of WebSphere Enterprise
It takes any kind of data from its native form
Processes them together, natively, with no code
And outputs them into their native target formats
Page 5
Complex Integration Challenges Addressed by WTX
Data Enhancement
High Throughput Lookups
Data Logic and Routing
Data Validation
Transaction Repair
Dependent Data and Usage Rules
Many to Many Transformation
Single Pass of Data Sources conversions and logic
Mixed Data and Source/Target Types
Dependent Result Sets, Nested Structure Dependencies
Complex Data Transformation
Nested, Semi-structured And Hierarchical Data Types
Dependent Inputs And Outputs
Type Conversions, Auto-Format Conversions
Binary, Packed, EBCDIC, ASCII, Mixed Character Data
Page 6
Successful Transactional
Data Integration Requires:
WebSphere Transformation
Extender Provides:
Quick-response, high-
performance data integration
Event-driven, real-time data
synchronization
Transformation of complex
data from multiple sources to
multiple targets
Automated transformation of
complex, hierarchical nested
data across many interfaces
Automatic routing of data
based on complex business
rules
Routing based on content
anywhere in multiple data
sources
Interoperability across
operational data systems
Codeless synchronization of
legacy, custom and off-the-
shelf applications
Real-time screening of key
transactions
Data content validation
against business rules
New integration power
without replacement of
existing IT infrastructures
Loosely coupled
components for rapid
embedding (SOA) and
seamless interoperability
Compliance with industry
standards and regulatory
mandates
Out-of-the-box solutions for
industry standards and
regulatory compliance

The WebSphere TX Solution
Page 7
Design Studio
Platform-specific transformation engines for
production environment
Adapters to integrate with specific types of data
sources and targets
Databases
Messaging systems
Packaged applications
Business integration utilities
Client
Development System
Server
Production System
Integration Flow Designer to model and manage
transformation workflows
Type Designer to define and import data objects
Database Interface Designer to generate data
definitions from metadata imported from database
Map Designer to define data transformation and
routing rules
Transformation engines to develop, test and run
transformations in development environment
Page 8
One Engine, Multiple Deployment Options
WebSphere TX is the one transformation engine that can be reused across all
applications and infrastructures in the enterprise.
Deploy the same WebSphere transformation asset across:
Unix
Linux
Windows
z/OS
Batch
CICS
IMS
USS
ESB
SOA
Web Services
SOAP
EJB
HTTP
Event driven
Batch
Java
C/C++
COBOL
30+ Adapters
ETL
Data quality
Enterprise Applications
WebSphere Message Broker
WebSphere ESB
WebSphere Process Server
WebSphere Partner Gateway
WebSphere DataStage
WebSphere DataPower
Page 9
validation, usage rules , transformation maps and additional
functionality
X12 EDI
EDIFACT
TRADACOMS
EANCOM
ODETTE
SWIFT Net FIN
SWIFT Net Funds
SEPA
ACORD
NACHA
FIX
HIPAA
HL7
NCPDP
SWIFT Net FIN
SWIFT Net Funds
SEPA
ACORD
NACHA
FIX
HIPAA
HL7
NCPDP
X12 EDI
EDIFACT
TRADACOMS
EANCOM
ODETTE
SWIFT Net FIN
SWIFT Net Funds
SEPA
ACORD
NACHA
FIX
HIPAA
HL7
NCPDP
Objective 2:
Setting up the WTX 8.2 environment
Page 11
Workspaces
WTX 8.2 stores its artifacts in workspaces which are physical directories on your
development Windows PC
Each time you start Design Studio you may be asked to choose your Workspace.
Page 12
Projects
Workspaces are further subdivided into projects.
You can create
new artifacts or
import them into
your project.
Page 13
Views
You will navigate using views.
Extender Navigator
Navigator
Outline
Composition, etc.
You will also use views to support editors.
Extender Properties
Functions
Component View
Audit, etc.
Page 14
Opening Views
To open a view, Open WindowShow Views
Page 15
View Properties
Views have their own menus. To
open the menu for a view, click the
icon at the left end of the view's
title bar.
Some views also have their own
toolbars. The actions represented
by buttons on view toolbars only
affect the items within that view.
To close a view, click the X on the
right side of its title tab.
Page 16
Sample Workspace
You may want to
leave a few views
open most of the time.
Heres one possible
configuration with
Extender Navigator,
Navigator, Outline,
Composition,
Extender Properties
and Functions views
open.
Page 17
Getting help
Welcome page
Opens the first time you open Design Studio
Can always open from help menu
Includes:
Get Started
Samples
Returning Users
Web Resources
Info center
Open from Get Started on Welcome page
Open from Help menu
Help menu
Objective 3:
Defining the data
Page 19
Type Trees
A type tree is a data dictionary that defines how types are classified.
The Type Designer is used to create and maintain type trees, although type
trees can also be created using the Type Tree Maker, the Importer Wizard,
and the Database Interface Designer.
Use the Type Designer to define:
Properties for data structures.
Data validation rules.
Data as text or binary.
Different character sets.
Page 20
Creating type trees
From scratch
Define a type for each data
object
Specify the properties of each
type
Importers
Import data structure from native
application
Accessed from Import menu
Packs (See slide 12.)
Page 21
Extender Properties
Use the Extender Properties view to create and edit the properties of
each data object
Page 22
Type names
Each type has a name
Case sensitive
Can contain numbers and special characters
Use an underscore instead of a space
Cannot be Design Studio reserved words
Description
Optional
Page 23
Type class
Each type has a class
Category
Logical construct
Does not describe a physical data object
Group
Describes complex object in the data
Contains components
Item
Describes simple object in the data
Does not contain components
Click in the Value area to select the types class from drop-down list
Page 24
Item subclass properties
Subclass
Number
Text
Date & Time
Syntax
Interpret as
Character
Binary
Size
Miniumum and maximum
In bytes or characters
Pad
Restriction
Optional feature you can use to specify valid or
invalid data objects for an item type
Can be defined for values, characters or ranges
Page 25
Group subclass properties
Group subclass and format
Describe how to distinguish one component of that group from another
component of that group.
Three subclasses
Sequence
A partially-ordered or sequenced group of data objects. Each component of a
Sequence group is validated sequentially.
A partially-ordered or sequenced group of data objects. Each component of a
Sequence group is validated sequentially.
Choice
Defines a selection from a group of components
validated as only one of its components
Unordered
Page 26
Group subclass properties (continued)
Partitioned
Use to distinguish the difference between data objects based on values in the data or differences in the
syntax
Method of subdividing objects into mutually exclusive subtypes.
Component syntax
Sequential group
Fixed
Delimited
No syntax
Choice group
None
White space
Unordered group
None
Delimited
White space
Page 27
Type syntax
Initiator
Terminator
Release
Empty
Page 28
Components
Objects a group contains
To define
Double-click group
Drag and drop
Specify range
Page 29
Component rules
Used for data validation
Statement about component that determines if data is valid
Evaluate business rules
Evaluate to TRUE or FALSE
NextReviewDate must be later
than LastReviewDate
NextReviewDate must be later
than LastReviewDate
Page 30
Component rule references
Rule can reference
Component it applies to
Any component earlier in same component list
Can reference itself (NextReviewDate)
and anything earlier in component list
Can reference itself (NextReviewDate)
and anything earlier in component list
Objective 4:
Creating a map
Page 32
Overview
To create maps in the Transformation Extender Development perspective:
1. Create a map source file. The file name extension of map source files is
.mms.
2. Create required executable map(s) in the map source file (.mms).
3. Create an input card that represents your input data.
4. Create an output card that represents your desired output.
5. In the output card map rules, specify the logic necessary to transform the
input data to the desired output data.
6. Build the map. This creates a compiled map file (.mmc).
7. Run the map.
Page 33
Cards
Input and output cards represent data objects. Each card represents one data object, which is
defined as a particular type.
An input card contains the complete definition of an input for the map including information such
as source identification, retrieval specifics, and the behavior that should occur during processing.
An output card contains the complete definition of an output for the map including information such
as target identification, destination specifics, and the behavior that should occur during processing.
Input cards display in the From window; output cards in the To window.
Page 34
Map rules
Describe how to build an output data object
Can be any combination of
Hard-coded values (literals)
WTX functions
Object names
Concatenation of multiple objects
Functional map references
Always start with an equal (=) sign
Are required for every output object
Enter =NONE for objects that do not require output
Page 35
Entering map rules
Map rules entered in rule bar
Colors distinguish functions from arguments, object names and other map
rule components
Key words can be configured with different colors
Invalid rule can be highlighted with background color
Always press ENTER to accept changes
Always drag-and-drop object names from cards
Page 36
Mapping items
If output object is an item type, enter map rule at item
level.
Page 37
Mapping groups occurring once
If output object is a group that occurs exactly once, enter map rule at group OR item
level NOT both!
Enter rule at
group level
Enter rule at
group level
Or item level
Or item level
Page 38
Mapping groups in a series
If a group or an item can occur more than once, decide how many
occurrences are to be built
If there is a known number of outputs, use Indexing
If the number of outputs depend upon a variable number of inputs, use
a Functional Map
Page 39
What is a functional map?
Like a subroutine
Maps portion of data at a time
Number of output produced based on some variable number of
corresponding input
Page 40
Defining a functional map
1. Determine need for a functional map
2. Determine input argument(s)
3. Create functional map call
4. Create new map
Create map (Map > New)
Create input card(s)
Create output card
Enter map rules
Page 41
Determining the need
Create a functional map when:
Output group object can occur more than once (component
range is > 1)
AND
Some unknown number of objects are created in output
(component range is s)
AND
Output group type is different from input group type
Page 42
Determining the argument(s)
Functional map syntax is used to
Name the map
Specify data to pass
Functional maps only have access to data passed to them
through function arguments
What objects are required to produce output?
Data needed must be passed from calling map as an argument to the
function
One argument is thought of as a trigger
I want one output object for each ...
=FunctionalMapName(Arg1,Arg2,,ArgN)
=FunctionalMapName(Arg1,Arg2,,ArgN)
Page 43
Creating the functional map call
Common practice to precede functional map name with f_ to easily
distinguish functional maps
Functional map name
Functional map name
Argument
Argument
Page 44
Creating functional map
1. Create a new map (Map > New)
2. Enter new map name EXACTLY as in functional map
reference
3. Add input card(s) (Card > New)
4. Add output card (Card > New)
5. Enter map rules into functional map
Page 45
Using the Functional Map Wizard
To edit specific card, select the
card then click Edit
To edit specific card, select the
card then click Edit
To create functional
map, click Create
To create functional
map, click Create
Return to initial settings
before Edit
Return to initial settings
before Edit
Page 46
Using the Functional Map Wizard
Functional map is created in the current map source file (.mms) using
the parameters specified in the Functional Map Wizard dialog
New functional map
is created with name
entered in map rule
New functional map
is created with name
entered in map rule
Page 47
Functional map example
f_MakeEachEmp
functional map
f_MakeEachEmp
functional map
Page 48
Evaluating functional maps
Evaluated once for each combination of occurrences of
each argument passed
To conditionally control input to functional map or output of
object, use functions and expressions within functional map
call
If any argument evaluates to NONE, functional map will not
be called for that combination of arguments
Functional maps build output, functions do not
Page 49
References
WSTC Resources
WTX Forum
WTX Tech Sales Wiki
WTX Support Site
AIM IC Repository
Sales Kits
Technical Examples
Page 50
Questions
Please remember to complete your session
evaluation form
Thank you!!
Session F08: WebSphere Transformation
Extender 8.2 Design Tools Lab
Session F12: Building a Highly
Session F12: Building a Highly
Available SOA Infrastructure featuring
Available SOA Infrastructure featuring
HACMP, MQ, and WMB
HACMP, MQ, and WMB
22
Conference Speaker
Conference Speaker
Building a Highly Available SOA
Building a Highly Available SOA
Infrastructure featuring HACMP, MQ,
Infrastructure featuring HACMP, MQ,
and WMB
and WMB
Session Number: F12
Session Number: F12
33
Objectives
Objectives
This presentation is designed to provide a basic overview of
HA concepts that can be obtained with WMQ and WMQI.
We will discuss features that are part of the products or the
operating systems, on distributed and z/OS platforms. We
will compare these capabilities, and show how these can be
effectively used together for extremely reliable environments.
After completing this session, you should be able to:
Have a general knowledge of HA concepts
Distinguish differences between WMQ clusters and HA fail
over
Have a better understanding of the various platform HA
specifics and capabilities
Assist a customer in designing and implementing a HA
environment
44
Topics
Topics
What is High Availability?
WMQ Clusters
WMQ for z/OS Sysplex Support
Restart/Recovery Considerations
GDPS (Geographically Dispersed Parallel Sysplex)
Failover
Active/Passive
Active/Active
WMQ & HA Clustering
Support Pac for HA
MSCS
HACMP
Sun & Veritas Cluster
HP MC/ServiceGuard
OS/400 CRS
Compaq TruCluster
Related Publications & Resources
Q & A
55
What is High Availability (HA)?
The objective is to achieve 24x7 processing of all
messages
High Availability refers to a system that is available most
of the time, i.e., minimal downtime
Applications should be processing messages
continuously, regardless of any failures in hardware or
software
This presentation concentrates on WMQ HA, but it's not
the only area to consider
Availability is measured as a percentage of uptime.
Its not always achievable, but we can get close
99.9% (system can be down for 526 minutes / year
(8.7 hours) )
99.99% (system can be down for 53 minutes / year)
99.999% (system can be down for 5 minutes / year)
66
Potential outage types:
80% Scheduled downtime (new software release, upgrades,
maintenance)
20% Unscheduled downtime (source: Gartner Group)
40% Operator error
40% Application error
20% Other (network failures, disk crashes, power outage etc.)
Single points of failure are eliminated with a High Availability solution
77
Unplanned
outages
Planned
outages
Unplanned
outages
Planned
outages
Full availability
Reduced capacity
(off-peak)
Reduced capacity
Outage
Do not do this
Availability
Availability
-
-
two servers are better than
two servers are better than
one!
one!
88
Availability Terms
Availability Terms
Number of nines
An expression for the percentage of time a system,
subsystem, or application is available.
5 99.999 Five
53 99.99 Four
526 99.9 Three
5256 99 Two
Downtime
min/year
Percent Number of
nines
99
Availability Terms
Availability Terms
Planned Outages
Sometimes called the maintenance window. A
planned down time when system, sub-system,
and application maintenance are normally applied.
Unplanned Outages
Also known as the emergency maintenance
window. Any unplanned outage of a component
that impacts availability goals.
10 10
Availability Definitions
Low availability
Anything Less Than "HA"
High Availability
Application cannot undergo an unplanned outage for more than
a few seconds/minutes at a time, but may do so as often as
necessary, or may be down for a few hours for scheduled
maintenance.
Likely achievable in WAS V5 (and later) without the "gold standard
Prior versions of WAS may have required the "gold standard" for this
service level.
Continuous Availability
Application cannot go down at all (the five 9's).
Will likely require the "gold standard" to achieve this service
level
11 11
Request Distribution
Incoming Requests are Distributed Across Multiple Identical
Server Instances
Distribution Agent Typically Employees an Algorithm
Work Load Management
Balances Client Requests Across Servers
Active Monitoring
Response Time
Capacity
Request Distribution Based on Workload
Request Redirection
There is an attempt to contact a server to make a request, when
the request fails, the request is redirected to another server
Also Known as Failover
o Assumes That There are Multiple Identical Servers (a cluster)
Multiple Instance Also Improve Scalability
12 12
Peer Standby
An identical Instance is Ready to Accept Work from a
Failed Server
May or May Not Be Performing Other Work While Waiting
Process Recovery
Also know as process failover, once a failure is
detected then a restart of the process is performed
o Usually on another machine
Hardware Clustering Products Provide This
o e. g. Veritas Server Cluster, HACMP, SunCluster, MSCS,
MC/Serviceguard
13 13
Disaster Recovery
The reconstruction of your physical production site in an
alternate physical site, occurring after the loss of your primary
data center.
The process of bringing up servers and applications, in priority
order, to support the business from the alternate site (data
center).
Normally this environment is substantially smaller than the
entire production environment, as only a subset of production
applications demand DR (normally based on justifiable business
need).
14 14
Availability Terms
Availability Terms
Continuous Availability (CA)
The system user experiences no discernable downtime. There
may be rare failures that occur, but the end user does not see
them.
High Availability (HA)
The system user may experience some very temporary
downtime, the system automatically detects failures and reacts
to them in an appropriate fashion.
Single Point of Failure (SPOF)
A single vital component of a system. If this component
(hardware, software or human) is down, the whole system is
impacted.
Service Level Agreement (SLA)
A contracted quality of service, often including response time
and availability goals.
15 15
Availability Components
Hardware availability
Everything from the CPU to the storage systems
System availability
LPAR can be a Single Point of Failure (SPOF)
Subsystem availability
WMQ, WMB, CICS, IMS, etc.
Application availability
CICS and IMS transactions
WMB Message flows
Batch programs
Data availability
WMQ messages
DB2 data
Other data stores (VSAM, etc.)
16 16
Connectivity
Network availability
Security
WMQ Channels
Staff
Having the right people to plan, design, implement, document,
manage, monitor and support the HA infrastructure and
applications is vital.
Reserve Capacity
Includes processing power, storage media, network capacity,
connection allocations, etc.
Sufficient reserve capacity affects the ability to respond to
unexpected volume spikes and both planned and unplanned
outages.
17 17
Availability
Availability
Other comments
Other comments
Availability does not come free
Availability should not be confused with disaster recovery
Some of the techniques are the same, but there are many more
considerations for DR
Building the HA infrastructure is only part of a solution
Application modification may be necessary
Enterprises can evolve towards an HA or CA environment
An all at once philosophy can be difficult and too costly
Most enterprises have hybrid implementations
Services are typically at different availability levels within an enterprise
18 18
Common Terms
Common Terms
Fault Tolerance
The system is able to cope with almost any type
of failure
There is no downtime apparent to applications;
no connections are lost
Hardware may be hot-plugged (Tandem)
High Availability
Some limited downtime is permissible
Service may be interrupted, but reappears
quickly with no manual intervention
19 19
Common Terms
Common Terms
Scalability
The ability to add extra 'servers' and get additional throughput
Often achieved by running cloned servers and applications
Workload Balancing / Workload Distribution
Different algorithms, but aimed at maximizing throughput
Most common round-robin (workload distribution)
A work load manager will load balance based upon
machine utilization
20 20
Common Terms
Common Terms
Standby Terminology
A Cold standby is inactive, and will need to be made active before it
can perform any work; sometimes referred to as active/passive
Requires a high degree of hardware redundancy
A Hot standby is active and ready to perform work, or already
performing work; sometimes referred to as active/active
The failover process is quicker than with a cold standby
Clusters
Can have different meanings HACMP & Veritas vs. WMQ, we'll
cover some later
21 21
Questions Which Need Answers :
Questions Which Need Answers :
The question is How long can my MQ applications and
systems be unavailable to my customers?
What is acceptable?
What is practical?
What hardware, software and people resources are
available?
Usually driven from customers Service Level Agreements
(SLAs)
Time is money
22 22
FAILOVER Scenario
FAILOVER Scenario
s Explained
s Explained
23 23
Failover
Failover
HA solutions use a cluster of resources to run a system
This is not to be confused with WMQ Clusters
A cluster consists of multiple servers, disks, network interface cards,
etc. working together as one unit
This is known as a Resource Group
If failure occurs on one resource, then a backup resource
automatically takes over responsibilities for the failed resource
Cluster management software manages a Resource Group
Monitors both hardware and software components
Restarts software applications if process failure detected
Migrates applications to backup server if hardware failure detected
Examples: Microsoft Cluster Server, Sun Cluster, Veritas Cluster
Server, HACMP, HP MC/ServiceGuard, etc.
24 24
Failover
Failover
Active/Passive
Active/Passive
25 25
Failover
Failover
Active/Passive
Active/Passive
26 26
Failover
Failover
Active/Passive
Active/Passive
27 27
Failover
Failover
Active/Passive
Active/Passive
Only one node runs applications and services while the other
node waits in "standby" mode
Redundant hardware can get expensive and may not be
available
The network identity (e.g. IP Address) must also be movable
to the standby node
The standby node could require a sufficient boot time to
accept the workload performed by the failing active node.
10-20 minutes is common
28 28
Failover
Failover
Active/Passive
Active/Passive
Desirable to maintain node equivalence
Common software levels
Executables are not shared and are local to each
machine
Common environments (e.g. paths, profiles, kernels)
Common security models, etc.
Also consider performance/capacity
Sufficient capacity to handle maximum workload
Standby machines need to be equivalent in terms of CPU,
memory, etc., capacity as the failing node
29 29
Failover
Failover
Active/Passive
Active/Passive
This illustration shows "shared" disks for transferring state
information about applications from one node to another
In most systems the disks are not accessed concurrently by
both nodes, but are accessible from either node, which
take turns to "own" each disk or set of disks
In other systems the disks are concurrently visible to both
(all) nodes, and lock management software is used to
arbitrate read/write access
Alternatively, disk mirroring can be used instead of shared
disks
An advantage is increased geographical separation, but
latency limits the distance that can be achieved
30 30
Failover
Failover
Active/Passive
Active/Passive
WMQ with HA Clusters

WMQ with HA Clusters
Separate disks for data files and logs
While not essential for HA reasons, it is recommended for
performance
Failover is exactly the same as a normal queue manager
restart
Non-persistent messages will disappear
Sender channels will be automatically restarted providing
they are triggered
May need to use custom scripts or services to restart things
like
Trigger monitors
Command server
31 31
Failover
Failover
Active/Active
Active/Active
32 32
Failover
Failover
Active/Active
Active/Active
33 33
Failover
Failover
Active/Active
Active/Active
Consider capacity of the failover machine to handle
maximum workload
The throughput of each queue manager may degrade
(depending on how heavily loaded they run) but at least work
is getting done
Failback can either be automatic or manual, your choice
Manual is strongly recommend so that applications which
have already connected to the running queue manager do
not have their connections broken arbitrarily
Carefully monitor the workload and only failback when
there's not too much work that's going to be disrupted
during the failback
34 34
Failover
Failover
Active/Active
Active/Active
A variety of active/passive and active/active variations are
possible
The availability of these options will depend on the
hardware facilities available, cluster software and workload
capacity
Coupling WMQ clustering capabilities in concert with a HA
failover clustering solution provides a very high degree of
continuous availability
35 35
Combining WMQ Clustering and HA
Combining WMQ Clustering and HA
Clustering
Clustering
36 36
Disaster Recovery
Disaster Recovery
Disaster recovery not included in standard HA configuration
Requires two geographically distinct sites to back up one another
Requires additional site clustering software such as IBM High
Availability Geographic Cluster (HAGEO) or Veritas Global Cluster
Manager
Node
#1
Node
#2
Site #1
Node
#1
Node
#2
Site #2
Node
#1
Node
#2
Site #1
Node
#1
Node
#2
Site #2
37 37
WMQ Clusters
WMQ Clusters
Clusters can be heterogeneous, available on all V5.1 and
upwards platforms
To avoid a single point of failure within the cluster, whenever
possible, use 2 full repositories
Full repositories should reside on highly available machines
Partial repositories publish and subscribe to a full repository
for information it needs to exchange messages
Each queue manager is an independent entity, but they
share configuration and topology information via a full
repository which uses WMQ messages stored in the
SYSTEM.CLUSTER.REPOSITORY.QUEUE for its integrity
38 38
From occasionally around to always there
From occasionally around to always there
-
-
What is a WMQ cluster and how does it apply?
What is a WMQ cluster and how does it apply?
QMG2
QMG3
QMG1
QMG4
CLUSTER
39 39
WMQ Clusters
WMQ Clusters
Because each queue manager is
independent you get linear scaling
Simplifies administration
Automatic definition of channels and
automatic discovery of queues
New nodes "learn" from repositories
If a destination queue manager fails:
Messages will be re-routed to an available
queue manager
Messages are not available until the
queue manager is restarted (orphaned
messages)
40 40
WMQ Clusters
WMQ Clusters
Need to ensure that 'equivalent' services are
available to take advantage of clustering.
Provides workload balancing (workload distribution)
Multiple instances of a cluster queue can be
defined on any participating clustered queue
manager
Automatic rerouting of messages when a target
machine is unavailable
The WMQ workload management algorithm
selects the local queue manager as the
destination whenever possible
If the clustered queue is local, then all
messages are put there
If the clustered queue is not local, an algorithm
(round-robin) determines which destinations are
available
41 41
WMQ Clusters
WMQ Clusters
42 42
z/OS High Availability Options
z/OS High Availability Options
z/OS
there is a lot of life left
in this Critter :
Queue Sharing Groups
Clustering
Geographically Dispersed Parallel Sysplex
43 43
Up to 32 queue managers in a queue-
sharing group can be part of a
Coupling Facility
A single Coupling Facility list structure
can hold up to 512 shared queues
Supports both persistent and non-
persistent messages
Supports peer recovery
Multiple queue managers can access
the same shared queues
Pull workload balancing
Scalable capacity
Provides the highest degree of
availability
44 44
Clusters and queue
Clusters and queue
-
-
sharing groups
sharing groups
Users in the network see the shared queue as being
hosted by each queue manager
a queue-sharing group puts messages to the same shared
queue manager within the queue-sharing group
Clients can start sessions with any members of the cluster
45 45
Very Low High Application
Medium Low Disk Subsystem microcode
Very Low Low Linux
Low Low z/VM
Low Very Low LPAR
High Very Low zSeries hardware
Cost to fix Probability of Failure Single Point of Failure
zSeries hardware upgrades requiring POR.
LPAR configuration changes requiring reboot of the LPAR.
z/VM maintenance.
Linux kernel maintenance that requires reboot.
Application maintenance
Besides hardware and software failures, what else can cause down time for
the application?
So what architecture eliminates most of these SPoFs for the least cost?
High Availability considerations for System z
High Availability considerations for System z
46 46
One z/VM LPAR with many Linux virtual servers enables resource sharing
z/OS LPAR z/VM Production LPAR
Router
WebSphere Cluster
HTTP
Server
HTTP
Server
WAS
Server
WAS
Server
WAS
Dmgr
All Linux virtual
servers draw from a
common pool of
memory and IFLs.
Resources from a
failed server flow to
surviving servers
Small application
clusters (Just enough
nodes for failover)
Smaller cluster
reduces failure
points
Remaining SPoFs
include System z
hardware, LPAR,
z/VM.
Leveraging Virtualization to Reduce Points of
Leveraging Virtualization to Reduce Points of
Failure
Failure
47 47
3 Linux LPARs. This solution eliminates most failure points, for a reasonable cost.
z/OS LPAR z/VM Production LPAR 1
LPAR weight = 35%
z/VM Production LPAR 2
LPAR weight = 35%
Router
WebSphere Cluster
HTTP
Server
HTTP
Server
WAS
Server
WAS
Server
WAS
Dmgr
z/VM Test/Development LPAR
LPAR weight = 30%
Test Test Test
Dev
Dev
Dev
Dev
Dev
All LPARs can share
a pool of IFLs.
Two LPARs run
production workload.
Applications run in
clusters split
between the prod
LPARs.
One LPAR runs
Test/Dev workload.
Only SPoF
remaining is zSeries
hardware.
Most Efficient HA Solution for
Most Efficient HA Solution for
zSeries
zSeries
Linux
Linux
48 48
HTTP
Server
Node 1
SA
Node 2
SA
z/OS LPAR z/VM LPAR 1
z/VM LPAR 2
Router
1: IBM Tivoli System Automation Monitors for Outage
S
e
r
v
i
c
e

I
P
H
e
a
r
t
b
e
a
t
Reference Architecture: Active / Cold
Standby Cluster
Standby Cluster
49 49
Node 2
SA
z/OS LPAR z/VM LPAR 1
z/VM LPAR 2
Router
2: IBM Tivoli System Automation Detects Outage and Automates Failover
S
e
r
v
i
c
e

I
P
HTTP
Server
Node 1
Standby Cluster
Standby Cluster
50 50
51 51
Systems WILL go down - either planned or
unplanned
Restart performance is important for availability
The longer the restart, the longer the downtime
Log files are key recovery resource
z/OS supports dual logging
Other platforms recommend RAID/mirroring
designs
After an abend, recovery time is dependent on .
Length of outstanding UoW
Time since last checkpoint
52 52
z/OS Automatic Restart Manger (ARM)
If a queue manager or a channel initiator has failed,
ARM restarts it on the same z/OS image
If z/OS fails, everything fails, ARM can restart all the
failed systems automatically in a predefined order on
another z/OS image within the sysplex
ARM provides a higher degree of availability even if you
are not using queue-sharing groups
53 53
-
-
GDPS
GDPS
Automation shell to manage applications and data availability in
and across sites
Monitors systems, disk & tape subsystems
Builds on (multi-site) sysplex and data mirroring technologies
Manages planned and unplanned exception conditions
System maintenance/failure
Site maintenance/failure
GDPS supports 2 data mirroring technologies
Peer to Peer Remote Copy (PPRC)
Synchronous data mirroring
Distance between sites up to 40Km (Fiber)
Disaster Recovery solution
eXtended Remote Copy (XRC)
Asynchronous data mirroring
Limited user involvement
Supports any distance
Disaster Recovery solution
54 54
Distributed HA Support
Distributed HA Support
55 55
Microsoft Cluster Server (MSCS)
Microsoft Cluster Server (MSCS)
Integrated in WMQ V5.3
Originally shipped as SupportPac MC74, now withdrawn
Monitors health of nodes & resources
Can perform local restarts (if desired)
Can perform failovers
Can combine local restarts and failover
Synchronizes registry entries
Administration from command line or GUI
The time taken to migrate a queue manager from one node to
another depends on a number of factors, but it's typically on
the order of 60 seconds
56 56
Unix Systems
Unix Systems
This is a Category 2 support pac MC91
Include scripts to:
Create, start, stop and monitor HA queue managers
Restarts services on a standby machine
2 disk partitions per queue manager
Each on shared disk, independently defined and failed over
Ignore advice to create /var/mqm, /var/mqm/log filesystems
before installing WMQ
Can be configured for either Active/Passive or Active/Active
configurations
The failover units should contain queue manager, disks and
IP 'service address, etc.
57 57
HACMP
HACMP
High Availability Cluster Multi-Processing (HACMP)
SupportPac MC63 now withdrawn for consolidated HA
Support Pac
Use 'smit' on AIX to configure HACMP & HACMP/ES
HACMP/ES (Enhanced Scalability) required to monitor
queue managers
Scripts installed in /MQHA/bin
Unit of failover is called a resource group
58 58
Sun Cluster
Sun Cluster
SupportPac MC69 now withdrawn for consolidated Support
Pac
Scripts installed in /opt/SUNWcluster/ha/mqm
Unit of failover is called a Sun Cluster
59 59
Veritas Cluster
Veritas Cluster
SupportPac MC6A now withdrawn for consolidated Support
Pac
Scripts installed in /opt/VRTSvcs/bin/MQM
Unit of failover is called a service group
60 60
HP
HP
-
-
UX MultiComputer ServiceGuard
UX MultiComputer ServiceGuard
SupportPac MC6B withdrawn for consolidated Support Pac
Scripts installed in /MQHA/bin
Unit of failover for MC/ServiceGuard is called a package
61 61
OS/400 CRS
OS/400 CRS
OS/400 Cluster Resource Services (CRS)
Support Pac MC41, Apr 27, 2004
IASPs - Independent auxiliary storage pool
A collection of disk units that can be brought online or
taken off-line independent of the rest of the storage on a
system, and can be switched between systems
Only one system can access the data at a time
Scripts and programs are saved in a file on each of the
cluster nodes
From the saved file, restore the MQHA library onto the
system
Unit of failover is called a Cluster Resource Group (CRG)
62 62
Compaq TruCluster
Compaq TruCluster
SupportPac MC68, Feb 10, 2004
Scripts installed in /var/cluster/caa/script
Unit of failover is called a service
63 63
AIX, HACMP, MQSeries, MQSeries Support Pac, WMQI
and DB2 should already be installed, using the normal
procedures onto internal disks on each of the nodes.
Windows, MSCS, MQSeries, WMQI and DB2 should
already be installed, using the normal procedures onto
internal disks on each of the nodes.
It is important that under normal operating conditions
you are running identical versions of all the software on
all cluster nodes.
The only exception to this is during a rolling upgrade
Platforms with current Scripts
Platforms with current Scripts
64 64
Broker-related commands and methods
The hamqsicreatebroker and
hamqsicreateusernameserver scripts used during the
creation and configuration of a broker create an
HACMP/ES Application Monitor for that broker.
These application monitors are saved in /MQHA/bin and
are called hamqsi_applmon.<broker>, where <broker> is
the name of the broker to be monitored.
General files and utilities
General files and utilities
65 65
Related Publications
Related Publications
MQSeries Manuals -
http://www-4.ibm.com/software/ts/mqseries/library/manualsa
MQSeries SupportPacs -
http://www-
4.ibm.com/software/ts/mqseries/txppacs/txpm2.html
MQSeries Fixes
http://www-4.ibm.com/software/ts/mqseries/support/fixes
MQSeries Platforms
http://www-4.ibm.com/software/ts/mqseries/platforms
IBM Redbooks
http://www.redbooks.ibm.com
* Suggested reading HA WBI Solutions
66 66
Related Resources
Related Resources
MSCS
http://www.microsoft.com/ntserver/support/faqs/Clustering_fa
q.asp
MSCS general questions
AIX HACMP
http://www-
1.ibm.com/servers/eserver/pseries/library/hacmp_docs.html
HACMP for AIX home page
Sun Cluster
http://www.sun.com/software/cluster/index.html
Sun Cluster home page
Veritas Cluster Server
http://support.veritas.com/
Veritas Cluster Server home page
67 67
Related Resources
Related Resources
HP MC/ServiceGuard
http://h71028.www7.hp.com/enterprise/cache/4174-0-0-0-
121.aspx
High availability solutions for HP-UX 11i home page
OS/400 Cluster Resource Services (CRS)
http://publib.boulder.ibm.com/iseries/v5r2/ic2924/info/rzaig/rzai
g.pdf
iSeries Clusters
http://w3-
itswonderland.ath.gr.ibm.com/documents/platform/iseries/High
%20Availability%20Services%20for%20iSeries.pdf
High Availability Services for iSeries
Compaq TruCluster
http://h18000.www1.hp.com/
Compaq home page
68 68
Summary of Failover HA Environments
Summary of Failover HA Environments
z/OS
WMQ uses Automatic Restart Manager (ARM)
Windows MSCS
Included in WMQ
AIX HACMP/ES
Solaris
HP - MC/ServiceGuard
iSeries
Compaq Tru64 TruCluster
69 69
Comparison of Technologies
Comparison of Technologies
70 70
Session Summary
Session Summary
Now that you have completed this session, you should be
able to:
Have a better understanding of HA concepts
Distinguish differences between WMQ clusters and HA
failover
Understand the various platform HA specifics and
capabilities
Assist a customer in designing and implementing a HA
environment
71 71
Questions
Questions
Thank you!!
SundarGopal Venkatraman
Session Number: W01 Session Number: W01
IBM IBM WebSphere WebSphere Application Server Application Server
Introduction To High Availability Introduction To High Availability
Page 1
Agenda
HA Vocabulary
WLM in WebSphere Application Server V6.x
HA in WebSphere Application Server V6.x
Application Availability
Management Availability
HTTP Session Failover
Failover Topologies with WebSphere
Page 2
Low availability
Anything Less Than "HA"
High Availability
Application cannot undergo an unplanned outage for more than a few seconds/minutes at a time,
but may do so as often as necessary, or may be down for a few hours for scheduled maintenance.
Likely achievable in WAS V5 (and later) without the "gold standard
Prior versions of WAS may have required the "gold standard" for this service level.
Application cannot go down at all (the five 9's).
Will likely require the "gold standard" to achieve this service level
Page 3
HA Gold Standard
More On This Later..
Page 4
Request Distribution
Incoming Requests are Distributed Across Multiple Identical Server Instances
Distribution Agent Typically Employees an Algorithm
Work Load Management
Balances Client Requests Across Servers
Active Monitoring
Response Time
Capacity
Request Distribution Based on Workload
Request Redirection
There is an attempt to contact a server to make a request, when the request fails, the request is
redirected to another server
Also Known as Failover
o Assumes That There are Multiple Identical Servers (a cluster)
Multiple Instance Also Improve Scalability
Page 5
Peer Standby
An identical Instance is Ready to Accept Work from a Failed Server
May or May Not Be Performing Other Work While Waiting
Process Recovery
Also know as process failover, once a failure is detected then a restart of the process is
performed
o Usually on another machine
Hardware Clustering Products Provide This
o e. g. Veritas Server Cluster, HACMP, SunCluster, MSCS, MC/Serviceguard
Page 6
Disaster Recovery
The reconstruction of your physical production site in an alternate physical site, occurring after the
loss of your primary data center.
The process of bringing up servers and applications, in priority order, to support the business from
the alternate site (data center).
Normally this environment is substantially smaller than the entire production environment, as only a
subset of production applications demand DR (normally based on justifiable business need).
Page 7
Agenda
HA Vocabulary
Page 8
WAS-ND Web Client Requests
App Server
Web
Container
HTTP
Server
Plug-in
Servlet
Requests
HTTP(S) Protocol Traffic
App Server
Web
Container
HTTP Server Plug-in
Detects Failure
Marks Container as Unavailable
Tries Next Server in ServerCluster
Round Robin or Random for Web Server
Maintains Client Affinity As Appropriate
Page 9
WAS-ND IIOP Requests
IIOP
Traffic
App Server
Web
Container
App Server
EJB
Container
App Server
EJB
Container
Java
Client
IIOP
Traffic
Client ORB Plug-in
Detects Failure
Marks Container as Unavailable
Tries Next Server in ServerCluster
Weighted Round Robin Request Distribution
Maintains Client Affinity As Appropriate
Page 10
WAS-ND Web services (1/2)
Leverages WS-Addressing
WAS will generate the endpoint reference for
clustered applications.
EndpointReferenceManager.createEndpointReference(QName
serviceName, String endpointName) for Stateless Requests
EndpointReferenceManager.createEndpointReference(QName
serviceName, String endpointName, java.rmi.Remote
statefulSessionBean) for Stateful Requests
Web Service Clients
WS Client Direct Connection
No WLM/Failover
WAS Application Server in a Cell
WLM/Failover
Stateless and Stateful Requests
Page 11
WAS-ND Web services (2/2)
Web Service Clients (continued)
WS Client to WAS Proxy
WLM/Failover
Stateless and Stateful Requests
WS Client to HTTP Server
WLM/Failover
HTTP Session Affinity Honored
Stateless Requests
Stateful Requires Intermediate Proxy Server
Page 12
WVE V6.x WLM
Stock
Trading
Prioritization and
Flow Control
Routing and
Load Balancing
Classification
Operational Policy
Node Groups
ODR
WebSphere XD
Decision Makers
Account
Mngmt
Financial
Advice
High
Importance
Medium
Importance
Low
Importance
AM FA
Node
5
AM ST
Node
2
FA ST
Node
3
AM FA
Node
4
AM ST
Node
1
Provisioning
Executions
Stock
Trading
Account
Mngmt
Financial
Advice
Application Demand
Resource State
Provisioning
Decisions
Application Placement
Page 13
Agenda
HA Vocabulary
Page 14
V6.x Application Server HA Architecture
Application Servers Independent of Mgmt Runtime
V6.x Application Servers Load From Local File System
V6.xJNDI Lookups
Each Application Server Has Its Own JNDI Service
CORBA object URL Available
In Addition to IIOP
More Flexible
Multiple Bootstrap Servers Can/Should Be Specified
Eliminating SPOF
Randomize the List to Avoid Overloading a Server
V6.x Security
Each Application Server Has Its Own Security Server
V6.x Transactions
Each Application Server Logs Distributed Transactions
V6.x Systems Management
Each Application Server Has Its Own JMX MBean Server
**Above Applies to WAS V5.x as Well
Page 15
WebSphere V6.x HA Architecture
WebSphere V6 Provides
Peer Recovery Model with Active Hot Standbys*
For Persistent Services
Transactions
Messaging
If a JVM fails then any Singletons Running in that JVM are Restarted on a Peer Once the Failure is Detected
Starting on an Already Running Peer Eliminates the Start Up Time of a New Process Which Could Take Minutes
WAS HA dependency services; Transaction Manager, WLM routing, PMI aggregation, JMS failover, DRS (HTTP Session, SFSB,
Dynacache), ODC
Planned fail over takes a few seconds
Design Criteria <1 second.
This Low Fail Over Time Means WAS can tolerate many failures without exceeding the 5.5 minute yearly maximum outage dictated by
99.999%
99.999% Can Only Be Achieved When All Components Are Engineered for This Availability
Level
WebSphere 6.x Without an Overall 99.999 % Infrastructure Will Not Assure 99.999% Availability
* This in addition to Client Request Redirection As Previously Noted
Page 16
WAS-ND V6.1 HA Components
Provides services that leverage distributed
agreement:
Highly Available Singleton Services
Low-level Replication Abstraction
Shared State Bulletin Board
Several other runtime features depend on HA
Manager services
HA Manager runs in every single JVM in the cell
HA Manager services are only available between
JVMs that are part of the same core group
High Availability Manager
Distribution and Consistency
Services (DCS)
Reliable Multicast Messaging
(RMM)
Transaction Service
Workload
Management (WLM)
Data Replication
Services (DRS)
Messaging
Engine
On-Demand
Configuration (ODC)
WAS-ND JVM
Page 17
HA: Transaction Log Peer Standby
Allows Failover of In-flight 2 PC transactions
WebSphere V6 Can Be Configured to Store Transaction Logs For Each Server on a NAS (Network
Attached Storage) Shared File System
Allows All Peers to See All Transaction Logs
Automatic HAManager Recovery requires that the distributed file in use provide the necessary file locking semantics,
o Specifically that file locks are released when a server fails.
When a V6 Cluster Member Fails, a Peer is Elected to Recover the Transaction Log From the Failed
Server
In Doubt Transactions From a Failed Server Are Recovered Very Quickly
Huge Improvement Over WAS V5
o Recovery Took Minutes
o Required OS Clustering & Shared Disks
Not Enabled By Default For a Cluster
Page 18
Example Hardware Clustering vs. WAS HA
Failure Scenario:
Client calls EJB which updates database using 2PC
transactions
Failure during in-process transaction (after the prepare
statement) database record is locked until transaction is
recovered (committed or rolled back)
Hardware Clustering
Cluster Manager detects failure
Server Process is restarted/moved to recover the
transaction log
Could be up to 5 minutes
Other applications could be locked out if they required the same record
WAS HA
HA Manager detects the failure
Failover to a peer server which recovers the transaction
log (shared on a NAS) from the failed server
Recovery in a few seconds
Design Criteria < 1 sec.
Page 19
Agenda
HA Vocabulary
Page 20
WAS-ND Application Update
Rollout Update Option for Application Updates on Clusters
Following steps Will Be Applied to Each Cluster Member in Turn:
Stop Cluster Member(s) on a Node
Distribute Update to Node
Re-start Cluster Member(s) on a Node
Plug-in detects Server Outage and Can Then Select Another Cluster Member
Additional Effort May Be Required for Uninterrupted Service
Superior over Manual & Scripted Approaches Using Stop Server, etc
Employs Application Update Function for Correct Event Registration and Synchronization
wsadmin updateAppOnCluster command
Page 21
WVE Application Editions
Interruption-free update of application on existing deployment targets (e.g. dynamic cluster)
Workload quiesced on and diverted from each server or cluster as edition swap is performed
Group Rollout
Old edition is replaced with new edition one server at a time or batches
Atomic Rollout
Old edition completely offline before new edition is available
Application requests arriving in the window are queued by on-demand router
Edition back-out
Ability to undo an edition rollout
Simply use edition rollout capability to rollout a previous edition
Validation
Hosting of new edition in production environment on clone of original deployment targets
Use routing policy to control edition visibility e.g. only test personnel
Page 22
Application Editions Group Rollout
quiesce
& stop
Edition 1.0
Edition 1.0
Edition 1.0
On-demand
routers
Dynamic cluster
Edition 2.0
restart
application
requests
Page 23
Application Editions Atomic Rollout
quiesce
& stop
Edition 1.0
Edition 1.0
Edition 1.0
On-demand
routers
Dynamic cluster
Edition 1.0
application
requests
Edition 2.0
Edition 2.0
quiesce
& stop
Edition 2.0
Edition 2.0
request
request
request
restart
restart
Page 24
Agenda
HA Vocabulary
Page 25
WebSphere V6.x HA of Dmgr and Node Agent
Dmgr
Dmgr Still Requires a Shared File System or Shared Drives With External Hardware Clustering Software to be
Highly Available.
No Longer a Single Point of Failure (SPOF) for IIOP WLM Routing Table Maintenance
The Dmgr is Only for Configuration Changes and JMX Routing.
Eliminating the Need for HA of the DMgr
Node Agent
Node Agent HA Requires SFS and Hardware Clustering
Node agent in V6 No Longer Critical for Transaction Log Recovery
Use NAS for Transaction Logs or Remote DB for Message Queues
Continue to Use OS Process Nanny to Insure Location Service Daemon Availability
Still Only Runs Inside the Node Agent
Page 26
WAS-ND Beer Budget Deployment Manager HA
1. Take Regular Backups for Cell Configuration with backupConfig - Located in
<wasroot>\DeploymentManager\bin
2. After Failure, Install ND on Backup/Alternate
ServerSpecify Node Name, IP Address and Cell Name from Original Server when creating profile
3. Restore Configuration on Backup Server using restoreConfig
Located in <wasroot>\DeploymentManager\bin
4. Change DNS alias or IP Address on Backup Server to Match Original Server
5. Start Deployment Manager
Total Time ~ 15 minutes **
Depending on Server, Network Speed and Size of Configuration
Note: Make Sure Your Backups are Available!
** Likely Less Than 5 Minutes with Preinstall of DMgr and Scripting
Page 27
WVE Deployment Manager HA
Each deployment manager on a separate machine
Only one is active
Others are standby
Shared file system required for dmgrs to share configuration repository
File system with recoverable locks required - e.g. SAN FS,NFS v4
JMX traffic proxied through XD On-demand Router (ODR)
SOAP connector only
HA ODRs recommended
(theyre recommended for production XD configurations anyway)
hadmgrConfig command line utility provided to perform configuration
Page 28
On-Demand
Routers
SAN FS
wsadmin
scripting
admin client
administrative
console
active
dmgr
standby
dmgr
Warm-standby Model
Page 29
SAN FS
wsadmin
scripting
admin client
administrative
console
Standby
dmgr
Standby
dmgr
On-Demand
Routers
Active
dmgr
Take-over after primary failure
Page 30
Agenda
HA Vocabulary
Page 31
WAS-ND Distributed Sessions
Failover Options
Local Sessions
No Failover (Technically not a Distributed Session)
Relies on Affinity
Database
Utilize Database HA
WebSphere Distributed Replication Service (DRS )
Simplified Configuration and Terminology in V6
DRS Runs in a each server instance in a replication domain
Build on Small, Fast 'pub/sub' Engine
Failover
Transparent to Application
State is Automatically Retrieved From Remote Copy if Not Present in Local Copy
Page 32
WAS-ND Distributed Sessions Updates Occur to Remote Copy
At the End of a Specified Time Interval
V6.x Default
Time Interval Defaults to 10 Seconds
Recommended Tuning for Performance and Failover
At End of the Servlet Service Method
Manually
Requires Use of IBM Extension to HttpSession
Session Manager is Distribution Client
No Application Visibility to DB or Replicator/Session Outage
WAS V6.02 (and above)
Updates Only Occur in Local Copy During Replication/DB Outage
Messages In Logs and Administration Client During Outage
Distributed Session Updated Once DB/Replicator is Restored
At Prescribed Replication Interval
Loss of Application Server During Outage
Loss of Application State
Application Could Move Backward in State
Page 33
WebSphere eXtreme Scale
HTTP Sessions Can Be Replicated Across Servers Using ObjectGrid
Implemented as a Servlet Filter that Enables Session Replication
Can Be Inserted Into Any Web Application
Script is run to Modify the Web Application Deployment Descriptor
Session Distribution is Independent of the WebSphere cell infrastructure
Can Be Used by
Older Versions of WebSphere Products
Other Middleware Servers
High speed, local access to ObjectGrid
Session Filter Hosts Session State in the Local JVM with Remote Replicas
Ability to leverage all the dynamic shard placement and migration
Embedded in WAS
External stand-alone ObjectGrid servers
Automatic Distributed State Retrieval
State is automatically retrieved from remote servers if the data does not exist locally.
Page 34
WebSphere Distributed Sessions DRS Peer to Peer (Both)
Peer to Peer Configuration
Page 35
WebSphere Distributed Sessions DRS Client Server
Client Server Configuration
Page 36
WebSphere Distributed Sessions Database Failover
Page 37
WebSphere Distributed Sessions
Database or DRS
Which Should I Use?
Both Perform Essentially the Same!!
Object Serialization/Deserialization is 95% of Performance Cost
Database Will Actually Persist the Data (to disk)
Amount of Data Only Limited By Disk Capacity
A HA Database Server Can Survive a Cascading Failure
Application Servers as Session Stores and Replicators May Not.
Memory to Memory
There Can Only Be a Single Replicator Common to the Two Cells
This is a Single Point of Failure
Replication Between Cells Only Available in V5.x
WAS Maintenance May Require an Outage
On Server Restart data may not be available immediately
Gold Standard" (two identical cells/domains),
An HA Database Can Assure Session Failover Between Domains
Note That while sharing sessions across cells is supported, this is not generally recommended.
Sharing state between cells it makes it more difficult to independently upgrade components
(application and WAS) in the two cells.
Page 38
Agenda
HA Vocabulary
Page 39
High Availability for Non-WAS Components
Firewall
IP Sprayer
WebSphere MQ
Security Registry
Database Server
SOA Appliance (DataPower)
File System
Make all HA !
Via hardware clustering or software clustering
Page 40
NAS, NFS/CIFS and HA File Systems
A NAS is a shared file system
It provides a remote file system using either NFS or CIFS protocols to Linux/Unix/Windows machines
NFS/CIFS servers can be used if a file system flush to disk works correctly all the
way through to the DASD
High end NAS support safe asynchronous writes and perform much better by exploiting NV (non volatile)
memory
WebSphere 6.x HA is designed to leverage shared file systems like IBM SAN FS,
Windows CIFS, and NFS v4
File System Locking Requirements
The file system product must release file locks when the process that owns the locks fails.
" However, not all file systems provide the necessary file locking semantics, specifically that file locks are released when
a server fails. "
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/cjta_hacons_log.html
Page 41
HA By Layer
Page 42
HA Gold Standard
Recall Recommendation Not to Share Sessions Between Cells
Page 43
HA Gold Standard
Two (or More) Domains
Hardware Isolation
Software Isolation
Planned Maintenance
Insurance Against Catastrophic Outage
May Require More Administrative Effort
Dont Forget Rule of 3
With 2 of Everything, an Outage (Planned or Unplanned) Reduces Capacity by 50
Is No Longer Fault Tolerant
Page 44
Do You Really Require the Gold Standard ?
Node
Agent
Node
Agent
IHS
IHS
WES
WAS
App
WAS
App
WAS
App
WAS
App
WAS
App
WAS
App
Applicaton,
Session &
LDAP
Deployment
Manager
MQ QUE
Clustering
For Most A Single Hardware Redundant Domain Will Suffice
Unless You Require Non-Stop Service
Reminder More Than 2 Provides Additional Redundancy/Capacity
Page 45
WAS HA vs. HA architecture
WAS and the WebSphere stacked products
Provide HA Features and Function
Architected for HA
Can Be Employed in an HA Architecture
A HA Deployment Architecture is Not a Product Feature.
The Appropriate Environment Varies by Customer
One Size Does *Not* Fit All
Correctly Leveraging WAS HA Capabilities into the Suitable Deployment
Requires In-depth Understanding
Of Environment
Of Applications
Of Operational Requirements
Architectural Advice May Require ISSW Assistance
Product Support Resolves Defects
Not Architectural Advice
Page 46
Indispensable for High Availability
Page 47
Questions
Please complete the session evaluation form
Thank you!!
Page 48
Additional Resources
Redbooks
WebSphere Application Server V6 Scalability and Performance Handbook SG24-6392-00
WebSphere Application Server V6.1: Planning and Design SG24-7305-00
WAS Info Center
Updating LDAP Binding Information
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/tsec_updateldap_bi
nd.html
Page 49
Additional Resources
Whitepapers
Maintain continuous availability while updating WebSphere Application Server enterprise applications
http://www-128.ibm.com/developerworks/websphere/techjournal/0412_vansickel/0412_vansickel.html
A Highly Available & Scalable LDAP Cluster in an IBM AIX Environment
http://www-1.ibm.com/servers/esdd/articles/ldap/index.html
Implementing a Highly Available Infrastructure for WebSphere Application Server Network Deployment, Version 5.0 without Clustering
http://www-128.ibm.com/developerworks/websphere/library/techarticles/0304_alcott/alcott.html
Product Support & Downloads
Downloadable IBM File System Locking Protocol Test Tool for WAS-ND
http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&dc=D400&q1=file+lock&uid=swg24010222&loc=en_US&cs=utf-8&lang=en
Architecting Solutions With The
Default Messaging Provider
Page 1
WebSphere Application Server Default Messaging
Provider
As part of the J2EE specification, all application
servers must have a messaging system to support
message-driven EJB (MDB)
The WebSphere Application Server (WAS) has
extended this feature to allow for full messaging
capability including point-to-point and publish-
subscribe
The main component for the default messaging
provider is the Service Integration Bus or SIBus
Page 2
WAS Messaging Topology Elements
Bus (or SIBus)
A group of connected servers used for messaging
A connection point for client applications to send
messages
WAS cells can host multiple buses
Bus Member
A server or cluster that hosts messaging capability
Messaging Engine (ME)
A component of the bus member
Hosts the messaging point
Destination
The actual messaging point that client applications
send messages to
Either a queue (point-to-point) or a topic (publish-
subscribe)
Data Store
A persistent storage that implements the
destination
Page 3
Messaging In Action
Applications (either on the same server or on a different runtime) can
connect to the bus and access the destination (queue) either as a
producer or consumer
Mediation is a process that is independent of the producer and
consumer that modifies the messaging behavior as it goes through
the queue
Available in the ESB
implementations of WAS
(e.g. WESB and WPS)
Page 4
How Many Buses?
Buses are scoped to a WAS cell
Multiple buses can be put in a single cell
Effectively segregates client applications wanting
to connect to the bus
Creates separate namespaces for applications to
use (bus names must be unique across cells)
Enables different messaging engine
configurations within the same topology
General philosophy
Use buses to segregate the messaging
infrastructure
Dont use any more buses than you have to
Not all servers should be part of a single bus
Cell
BusA
BusB
Server1 Server2
ME
ME ME
Page 5
Client Connection
Easy to configure
Applications connect using APIs
Scalable
All messages are concentrated on the queue
Cell
Bus A
Server1
Messaging
Engine P C
Page 6
Foreign Bus Link
Requires the setup of an SIBus link between buses on the same or
separate cell(s)
A foreign queue is defined in the remote bus that targets a queue in
the local bus
Provides store-and-forward capability (automatic)
Remote get is not possible (remote means accessing a queue on
remote bus through a local bus)
Cell
Bus A
Server1
Messaging
Engine P
Cell
Bus B
Server2
Messaging
Engine C
SIBus Link
Page 7
Messaging Engine Placement
Foreign bus links are ideal for enterprise-wide asynchronous
messaging
Not good for pseudo-synchronous operations
Not good for servers in the same cell that are geographically co-located
Always keep in mind that client connections require less
configuration
Standard strategy is to put the consumer local to the bus, the
producer(s) can store-and-forward (bus link) or use client connection
Cell
Bus A
Server1
Messaging
Engine
P
Cell
Bus B
Server2
Messaging
Engine
C
SIBus Link
P
P
Page 8
Clustered Messaging Engines
Clustered MEs are automatically created when a WAS cluster is
made a bus member
A choice must be made whether to use high availability or workload
management (cannot be both or mixed)
The high availability option is default (no configuration required)
Cell
Bus
Cluster
Server1 Server2
ME
Page 9
Clustering For High Availability
Only one ME is active at a given time
If the active ME fails, the system fails over to a stand-by ME
The fail over process is not instantaneous and should be considered
whether high availability option will be used or not
The data store is a point of failure
Page 10
Clustering For Workload Management
Requires that the queue be partitioned by
associating it to multiple MEs (each ME
increments the number of partitions)
All MEs in the cluster are active at any
given time
Consumers get messages in a partitioned
fashion regardless of remote or local
connection
Order of messages is not guaranteed
Page 11
Deciding On Message Persistence
Non-persistent messages will be lost when the system restarts or
when clustered MEs fail over
Does it matter?
Message persistence adds overhead to the system
If not used, messages cannot be recovered
Check if the requirements really need message persistence
Use the message persistence qualities of service to fine-tune
performance vs. reliability
Page 12
Message Persistence Qualities Of Service
Page 13
Transactional Messaging
In the default platform messaging infrastructure, send and receive
message operations can be bound in transactions
These can be coordinated with a resource manager and can
participate in a two-phase commit type of transaction
Make sure that messages are persistent if transactions will be used
For MDBs transactions must be container managed (CMT) to ensure
that the message receive operation and the onMessage() method
are on the same transaction boundary
Page 14
Impact Of Queue Depth
Shallow queues optimize performance by increasing the chances of
in-memory caching
It is ideal to maintain the depth of a queue to up to half-full
Study requirements and consistently monitor queues to determine
the actual size of the queue to maintain optimal depth
Use message expiration techniques to prevent build-up and maintain
optimal depth
Page 15
Impact Of Message Size
Message size has a huge impact on system performance
Based on BEST_EFFORT_NONPERSISTENT settings
1 KB ~ 24,000 msgs/sec
10 KB ~ 6,000 msgs/sec
100 KB ~ 650 msgs/sec
1 MB ~ 100 msgs/sec
Use claim check pattern
Cell
Bus A
Server1
Messaging
Engine P C
RDBMS
Page 16
WebSphere MQ Interoperability
MQ Link
Allows WAS messaging to connect to a WebSphere MQ infrastructure
Allows interconnection to WebSphere Message Broker to provide mediation
services
MQ Client Link
Allows MQ clients to connect to WAS messaging as if it were an MQ server
WebSphere MQ network
Cell
Bus
Server Server
Messaging
Engine
Messaging
Engine
Queue
Manager
Queue
Manager
C MQ Link
P
Page 17
Enterprise Service Bus (ESB)
Page 18
An ESB Is Like
If power is needed, we just need
to find a power outlet
The electricity can come
from any power source
Page 19
3
rd
Party Tools
SIBus Performance Tools (in AlphaWorks)
SIBus Explorer (in AlphaWorks)
IBM Tivoli Composite Application Manager (ITCAM) for WebSphere
PMI/JMX interfaces for custom solutions
Page 20
In Conclusion
Maximize the use of the default messaging providerits more than
just for MDBs
Utilize for web servicesmuch more efficient than HTTP
Use as a starting ESB implementationstart small etc.
It is not MQ!
Thank you
larara@ph.ibm.com
WebSphere Application Server:
Placing Tigers Stripes On Web
Services
Page 1
Service Oriented Architecture (SOA)
What is SOA?
A new approach to building IT systems that allows businesses to leverage
existing assets and easily enable the inevitable changes required to support the
business.
SOA is as much a business approach as it is an IT methodology.
SOA is focused on
Re-use of existing assets
Responsiveness to change
Connecting isolated domains
Use of open standards
Page 2
What Is A Service?
a useful action or task performed for the benefit of others.
(dictionary)
In SOA, a service is a software component that is
Aligned with an organizations scope of work or domain (i.e. banking)
Re-usable for both past and future requirements
Implemented using open standards
Loosely-coupled
Scalable
Page 3
Web Services Basics
Web Services Definition Language (WSDL)
XML document that defines the structure of the web service
Simple Object Access Protocol (SOAP)
XML document that defines the messages to and from the web service
SOAP
Message
WSDL
Service Response
Service Request
Page 4
Sample SOAP Message
Service Name
Parameter
Page 5
Sample WSDL File
Data Types
Parameters
Method
Protocol
Address
The structure is similar to a Java Interface
Page 6
WS-* Standards
Implementation neutral as long as the specifications are followed.
Apache Axis
JAX-RPC
SCA
Page 7
Java API for XML Remote Procedure Call (JAX-RPC)
Used to implement Web Services for J2EE (WSEE)
Main components
Service Implementation Bean A JavaBean (or Stateless Session EJB) that
implements the business logic of the web service through its methods and
implements the SEI interface
Service Endpoint Interface (SEI) A Java interface derived from the service
implementation bean methods used as an abstraction for the web service
Service Implementation Either an EJB and/or a Servlet used to implement the
web service
WSDL document to define the web service
Supports SOAP message handlers for fine control of web service
requests and responses
Page 8
JAX-RPC Components
LibraryService
<<Interface>>
LibraryServiceImpl
<<Stateless Session EJB>>
or
<<JavaBean>>
implements
<<EJB>>
and/or
<<Servlet>>
wsdl
is defined by
uses
Application Server
service
implementation
service implementation
bean
service endpoint
interface
Page 9
JAX-RPC Client Programming
Obtain the service object using InitialContext (JNDI) call and cast to
the SEI interface
The Service object allows 3 types of access:
Stub Client is developed using the full details of the WSDL, which is not
needed anymore at runtime
Dynamic Proxy Client is developed from partial WSDL details and the full
WSDL is needed at runtime
Dynamic Invocation Interface (DII) Client is developed dynamically at runtime
using the WSDL
Managed clients are clients that run in the same server as the web
service, unmanaged clients run external to the server.
Page 10
Asynchronous Web Services
Uses Java Message Service (JMS) transport for the SOAP protocol
Makes the request and response of the web service independent
from one another
Asynchronous Send A fire-and-forget process where the invoker does not
have to wait for the response
Asynchronous Receive Considered a separate event from the send, which the
client has to make preparations for
Brings advantages of messaging systems (JMS) into web services
Reliable/guaranteed delivery
Message persistence
Support for message broadcasting (publish-subscribe)
Page 11
WSDL Declaring SOAP/JMS
Page 12
Service Integration Bus (SIBus)
WebSphere Application Server (WAS)
provides a JMS messaging system to
support Message-Driven Beans in the EJB
specification
This has been extended to provide a full
messaging capability for applications that
might need it
The default messaging provider builds its
messaging capability around a Service
Integration Bus (SIBus) which is linked to
the features of the WAS infrastructure
Cell topology
Clustering
Linkages to WebSphere MQ
Cell
BusA
BusB
Cluster
Server1 Server2
ME
ME ME
Page 13
Asynchronous Operations Through JMS
Page 14
Bottom-Up Development: Servlet Implementation
Page 15
Top-Down Development: Servlet Implementation
Page 16
Bottom-Up Development: EJB Implementation
Page 17
Top-Down Development: EJB Implementation
Page 18
Client Development: JAX-RPC Client Application
Page 19
Client Development: J2EE Client Application
Page 20
The Future Of JAX-RPC
Next iteration will be named Java API for XML Web Services (JAX-
WS)
Shift to a messaging format rather than an RPC style
Extensive use of
Java API for XML Binding (JAXB) Java-XML data conversion
WS-Addressing In-message addressing
WS-Notification Publish and subscribe
WS-Resource Stateful services
Message Transmission Optimization Mechanism (MTOM) For attachments
Page 21
Enterprise Service Bus (ESB)
Page 22
An ESB Is Like
If power is needed, we just need
to find a power outlet
The electricity can come
from any power source
Page 23
Service Component Architecture (SCA)
An alternative to JAX-RPC in implementing web services
Implementations can be packaged into components that can be
accessed as web services
Focus is in wiring services together creating composite services
Standard service implementation for IBMs SOA solutions
WebSphere Enterprise Service Bus
WebSphere Integration Developer
Page 24
WebSphere Application Server V6.1
Uses the IBM Java 1.5 VM as runtime
Supports J2EE 1.4 and WSEE 1.1
WAS 6.1 Feature Pack for Web Services
WS-Reliable Messaging
WS-Addressing
MTOM
WS-Secure Conversation
JAX-WS
JAXB
SAAJ
StAX
Uses RAD 7.0 as development environment
Full complement of web services development tools/wizards (server and client)
Top-down and bottom-up development
Web Services Explorer test client and TCP/IP Monitor
Page 25
Advantages Of Web Services
Easiest way to implement distributed systems
Most efficient way to integrate systems
Most efficient way to expose legacy systems
Most efficient way to implement business processes
Leverages open standards
Loosely-coupled/re-usable
Interoperability (WS-* standards)
Used to implement SOA
Thank you
larara@ph.ibm.com
Wayne Toh
ISSW Consultant
Page 1
Introduction and Benefits of Middleware
Distributed Systems and Middleware
Middleware Patterns
IBM and Middleware
SOA
Page 2
What is Middleware?
Commonly used buzz word with unclear scope
Page 3
Earliest Citation
1968 NATO Software Engineering conference
Page 4
Distributed Systems
A distributed system is an information-processing system
that contains a number of independent computers that
cooperate with one another over a communications
network in order to achieve a specific objective.
Attains a common goal
Page 5
Structure of a Distributed System
Page 6
Distribution applications
The pervasiveness of networking technology both locally (e.g., local
area networks) and globally (e.g., the Internet) enables the
proliferation of distributed applications.
Page 7
Benefits of Distributed Systems
Distributed systems offer a variety of advantages
compared to centrally organized mainframes.
Availability
Speed
Page 8
Disadvantages of Distributed Systems
Dependency among Distributed Components
Heterogeneity of Components
Security vulnerabilities
Page 9
Comparision with Centralized System
Low High Security
Heterogenous Homogeneous Technology
Good Poor Scalability
Difficult Simple Consistency
High Low Complexity
High Low Availability
Economics
Criteria
High Low
Distributed
System
Centralized
System
Page 10
Definition of Middleware
It sits "in the middle" between application
software working on different operating systems.
It is similar to the middle layer of a three-tier
single system architecture, except that it is
stretched across multiple systems or
applications.
Page 11
Middleware for Distributed Systems
Application 1 Application 2
Application 5 Application 4
Application 3
Middleware
Page 12
The Stack
Web Sites
e-Commerce
Supply Chain
HR
Personal Productivity
Engineering and Design
CRM
Business intelligence
Systems Management
Application and Transaction Servers
Collaboration and Messaging
Database
Memory
Processor
Operating System
Networking
Storage
Applications Software
Middleware Software
Systes
Page 13
Tasks of Middleware
Object model support
Operational interaction
Remote interaction
Distribution transparency
Technological independence
Page 14
Middleware is Plumbing
Page 15
1. It is invisible
You dont generally see much of the plumbing in
your house.
Page 16
2. It provides a standard way of doing things.
If you wanted to build your own plumbing from
scratch, you could. But its much easier to just
buy plumbing fixtures.
Page 17
3. It ties together parts of complex systems.
In your house, you have kitchens, heating
systems, bathrooms, washing machines, garden
faucets, etc. Each plays an important part in
making your house livable.
Page 18
4. It lets you worry about other things.
When you put an addition onto a house, what do
you worry about?
Page 19
Types of Middleware
Message Oriented Middleware
Object Middleware
RPC Middleware
Database Middleware
Transaction Middleware
Portals
Page 20
Remoting Styles
1. Use the metaphor of a remote procedure call
(RPC)
2. Use the metaphor of posting and receiving
messages
3. Use a shared repository
4. Use continuous streams of data
Page 21
Remote Procedure Calls
Client
Application
Server
Application
Client
Object
Server
Object
invoke operation
return result
Page 22
More of RPC
Nowadays there are many popular OO-RPC middleware systems
specifically designed for distributed object communication. Examples
are:
Common Object Request Broker Architecture (CORBA)
Microsoft's .NET Remoting
Web Services
Microsoft's DCOM
Sun's Java RMI
Page 23
Message Passing
Client
Application
Server
Application
Queue
Queue
Message
Reply
Reply
Message
Page 24
Shared Repository
Client
Application
Shared
Repository
Client
Application
Client
Application
retrieve add
delete
Page 25
Streaming
Server
Application
Client
Application
stream
Client
Application
Client
Application
stream
stream
Page 26
Message Queue
WebSphere MQ provides simplified communication
between applications running on different hardware
platforms and operating systems, implemented using
different programming languages, or running within
different hardware and software environments.
Page 27
A Closer Look at Message Queue
Page 28
If there was no Message Queue?
Developers have to deal with low-level networking details
Are usually not easy to scale
Are typically rather cumbersome and error prone to use
for developers
Are hard to maintain and change
Do not provide transparency of distributed
communication
Page 29
If there was no Message Queue?
Without a secure, flexible, and reliable message queuing infrastructure, the
application programmer might need to apply significant extra development to the
following issues:
A remote service might be temporarily unavailable at the time the local application
needs to connect.
A communications link between the local application and a remote service might be
unavailable or might fail part way through transmission.
A remote service might be running on a different hardware platform to that of the local
application, with fundamental differences in character and numeric data formats.
The route to a remote service might have changed since the application was first
written.
There might be multiple instances of a service in the network, and the application
might need to choose to which instance of this service to connect.
The action performed by a particular service might be sensitive, and as such, the
service must only be invoked by authorized applications.
The data being sent over a communications link to a remote service might be
sensitive and require protection. The application might need to perform encryption of
this data.
A failure in communication with a particular service might affect the validity of other
actions that have already been performed by the application.
Current capacity requirements might change over time, typically increasing.
Page 30
E-Business
With the increase in popularity of the Internet, and indeed
E-Commerce, Middleware has become indispensable in
the development of web-based applications.
Page 31
Vendors of Middleware
IBM,
Red Hat
Oracle
SAP, TIBCO, Mercator Software, Crossflo, Vitria
and webMethods
32
Windows Linux Solaris z/OS AIX HP-UX OS/400
Security Management
Ensure compliance to identity and
access control policies for IT resources
and services.
Stay ahead of outsider and insider
threats to data, systems and
applications
IT Operations
Control cost and quality of service
delivery through process automation
and optimization.
Optimize infrastructure utilization and
service availability while moving from
re-active to pro-active management.
Deliver a web-services infrastructure
that is agile, high-performing and
secure.
Service Provider Operations
Ensure critical services are performing
to the highest standards.
Storage Operations
Create highly resilient storage
infrastructures, protect valuable
information assets and comply with data
protection policies.
Enterprise Operations
Focus is on assets in a service context.
Maximize the performance and lifetime
value of all business assets across the
enterprise.
Application and transaction
infrastructure
Build, deploy and run applications in
a proven, secure and flexible
environment.
Application Integration
Linking applications and services for
operational efficiency and cost
savings
Business Process Management
Improve business responsiveness
with end-to-end integration across
and beyond the enterprise.
Commerce
Automate and integrate online
marketing and sales processes
across multiple channels.
Mobile and speech middleware
Extend access to business processes,
applications and information to
anyone anywhere.
Portals
Deliver a point of personalized
interaction with applications, content,
processes and people.
Database Servers
Build the foundation for information
on demand to meet the full range of
your business needs.
Data Management Tools
Enhance the performance of IBM IMS
and DB2 using tools designed to
meet your needs.
Enterprise Content Management
Make better decisions faster by
managing content, optimizing
associated business processes and
enabling compliance through an
integrated information infrastructure.
Enterprise Data Management
Deliver data on demand to optimize
business usage and governance
across your enterprise
Dynamic Warehousing and Business
Intelligence
Gather, manage & analyze data,
generating information to develop
insight for informed decisions.
Information Platform and Solutions
Leverage information for innovation
combining market-leading
Information Integration and Master
Data Management technologies
Appl. Design and Development
Increase user productivity and business
flexibility with custom solutions for
Lotus software.
Dashboard and Business Solutions
Deploy role-based work environments
and speed time-to-value with
dashboards, scorecards and composite
applications.
E-Mail, Calendaring and Collaborative
Applications
Provide more than just messaging with
Lotus Notes and Domino built-in
collaboration tools and application
platform.
Instant Messaging, Web Conferencing
Collaborate real-time with integrated
Unified Communications and
Collaboration (UC) software and
solutions.
Social Software
Gather and exchange information
through professional networks and
build communities of experts to help
execute tasks faster.
Team Collaboration, Content Mgmt and
e-forms
Create, organize, share, and manage
business content to provide the right
information to the right people -- fast.
Mobile and Wireless Products
Extend enterprise business processes,
applications and information to mobile
and wireless devices and clients.
MWV May, 2008
Architecture Management
Model, design and rapidly build
resilient architectures for SOA,
systems and applications
Quality Management
Ensure software
functionality,reliability,and
performance production
Process and Portfolio Management
Align business goals, best
practices and projects for
improved productivity and
predictability
Change and Release Management
Improve effectiveness with
process automation, build
management, reporting and
traceability
Host Tools/Integration, Languages
and Compliers
OS/390
Integration and Application
Infrastructure
IT Service Management Integrating Data and Content Collaboration and Access
Software and System
Development
33
Windows Linux Solaris z/OS AIX HP-UX OS/400
Security Management
IBM Tivoli Identity Manager
IBM Tivoli Federated Identity Manager
IBM Tivoli Access Manager Family
IBM Tivoli Security Compliance Manager
IBM Tivoli Directory Integrator
IBM Tivoli Directory Server
IBM Tivoli Security Operations Manager
IBM Tivoli zSecure Suite
IT Operations
Tivoli Monitoring Family
Tivoli OMEGAMON XE Family
Tivoli Netcool OMNIbus
Tivoli Network Manager Family
Tivoli Provisioning Manager Family
Tivoli Intelligent Orchestrator
Tivoli Workload Scheduler Family
Tivoli Netcool Family
Tivoli System Automation Family
Tivoli Composite Application Manager Family
Tivoli Business Service Manager
Tivoli Service Level Advisor
Tivoli License Manager Family
Tivoli Financial Manager Family
Tivoli Change/Configuration Mgmt Database
Tivoli Unified Process Composer
Tivoli Application Dependency Discovery Mgr
Tivoli Process Manager family
Tivoli Impact
Service Provider Solutions
Tivoli Netcool Service Quality Manager
Tivoli Netcool Performance Manager for
Wireless
Tivoli Netcool Performance Mangager
Tivoli Netcool OMNIbus
Tivoli Network Manager
Tivoli Impact
Storage Operations
Tivoli Continuous Data Protection for Files
Tivoli Storage Manager Family
TotalStorage Family
TotalStorage SAN Family
Enterprise Operations
Maximo Asset Management Family
Tivoli License Manager Family
Tivoli Usage and Compliance Manager
Tivoli Asset Manager for IT
Integration and Application
Infrastructure
IT Service Management Integrating Data and Content Collaboration and Access
Database Servers
DB2 Family
IMS
Informix Family
U2 Family
Data Management Tools
DB2 Tools for Multiple Platforms
DB2 Extenders Family
DB2 Connect Family
IMS Tools Family
WebSphere Replication Server
WebSphere DataStage
WebSphere QualityStage
WebSphere ProfileStage
WebSphere Federation Server
WebSphere Information Analyzer
Enterprise Content Management
IBM FileNet Content Manager Family
IBM FileNet Image Family
IBM FileNet Capture Family
IBM FileNet Content Federation Services
IBM FileNet Connectors for SharePoint
IBM FileNet Business Process Manager Family
IBM FileNet Records Manager
IBM FileNet Records Crawler
IBM FileNet Email Manager
IBM Content Manager Family
IBM Content Manager OnDemand Family
WebSphere Information Integrator Content Edition
IBM Document Manager
IBM Records Manager
IBM CommonStore Family and eMail Search
OmniFind Family
IBM Classification Module
Enterprise Data Management
IBM Optim Solutions
IBM Data Studio
Dynamic Data Warehousing and
Business Intelligence
Cognos
BM Data Warehouse Editions (DWE)
Business Intelligence on Systems z
OmniFind Analytics Edition
OmniFind Discovery for Business Intelligence
Information Platform and Solutions
IBM Information Server
IBM InfoSphere
WebSphere Product Center
WebSphere Customer Center
WebSphere RFID Information Center
IBM Industry Models
IBM Global Name Recognition Products
IBM Identity Resolution
IBM Relationship Resolution
IBM Anonymous Resolution
Appl. Design and Development
IBM Lotus Domino Designer
IBM Lotus Enterprise Integrator for Domino
IBM Lotus Connector for SAP solutions
IBM Lotus Workflow
IBM Lotus Expeditor
Dashboard and Business Solutions
IBM Lotus ActiveInsight
IBM Workplace for Business Controls and
Reporting
IBM Workplace for SAP Software
IBM Lotus Workforce Management
IBM Workplace Solutions
E-mail, Calendaring and Collaborative
Applications
IBM Lotus Domino
IBM Lotus Notes and Domino Express
IBM Lotus Notes
IBM Lotus Domino Web Access
Instant Messaging, Web Conferencing
IBM Lotus Sametime Standard
IBM Lotus Sametime Entry
IBM Lotus Sametime Unyte
Social Software
IBM Lotus Connections
Team Collaboration, Content Mgmt and
e-forms
IBM Lotus Quickr
IBM Workplace Web Content Management
IBM Lotus Forms
IBM Lotus Forms Express
IBM Lotus Domino Document Manager
IBM Lotus Quickr Content Integration
Mobile and Wireless
IBM Lotus Expeditor
IBM Lotus Mobile Connect
IBM Lotus Domino Unified Communications
(See WebSphere Brand Section for WebSphere
Portal Products)
MWV May, 2008
Software and System
Development
Architecture Management
Rational Software Architect
Rational Business Developer Extension
Rational Systems Developer
Rational Data Architect
Rational Rose Family
Rational Software Modeler
Rational Asset Manager
Telelogic System Architect
Telelogic Rhapsody
Quality Management
Rational ClearQuest
Rational Tester for SOA Quality
Rational Functional Tester
Rational Performance Tester
Rational Manual Tester
Rational PurifyPlus
Rational Test RealTime
Rational AppScan Family
Rational Policy Tester Family
Telelogic Logiscope
Telelogic Tester
Process and Portfolio Mgmt.
Rational Portfolio Manager
Rational Method Composer (includes Rational
Unified Process)
Rational Team Unifying Platform
Telelogic Focal Point
Telelogic Harmony
Change and Release Mgmt.
Rational RequisitePro
Rational ClearCase
Rational ClearCase Multisite
Rational ClearCase Change Mgmt Solution
Enterprise Edition
Rational ClearQuest
Rational ClearQuest Multisite
Rational Build Forge
IBM SCLM Advanced Editor for z/OS
Host Tools/Integration,
Languages and Compilers
WebSphere Host Integration Solution Family
WebSphere Host Access Transformation Services
WebSphere Host On Demand
IBM Host Access Client Package Family
OS/390
Application and Transaction
Infrastructure
WebSphere Application Server (WAS) Family
including WAS Community Edition
WebSphere Extended Deployment
CICS Transaction Server
Application Intregration
WebSphere Services Registry & Repository
WebSphere MQ Family
WebSphere DataPower Appliances
WebSphere Adapters
WebSphere Business Services Fabric
WebSphere Business Modeler
Commerce
WebSphere Commerce Family
Mobile and Speech
WebSphere Everyplace Family
WebSphere Voice Response
WebSphere Voice Server Family
WebSphere Translation Server for Multiplatforms
IBM embedded Via Voice
Unified Messaging for WebSphere Voice
Response
Portals
WebSphere Portal Server
WebSphere Portal Enable
WebSphere Portal Enable for z/OS
WebSphere Portal Extend
WebSphere Portal Express
WebSphere Portlet Factory
Accelerators for WebSphere Portal
WebSphere Dashboard Framework
WebSphere Everyplace Mobile Portal Enable
Use the Rational Rainbow for high-level
product overviews. Search XL with
keywords: Rational Rainbow
Page 34
Application and Transaction Infrastructure
WebSphere Application Server (WAS) Family including WAS
Community Edition
CICS Transaction Server
Page 35
Application Integration
WebSphere Services Registry & Repository
WebSphere MQ Family
WebSphere DataPower Appliances
WebSphere Adapters
Page 36
WebSphere Business Services Fabric
Page 37
Commerce and Portals
Commerce
WebSphere Commerce Family
Mobile and Speech
WebSphere Everyplace Family
WebSphere Voice Response
WebSphere Voice Server Family
WebSphere Translation Server for Multiplatforms
IBM embedded Via Voice
Unified Messaging for WebSphere Voice Response
Portals
WebSphere Portal Server
WebSphere Portal Enable
WebSphere Portal Enable for z/OS
WebSphere Portal Extend
WebSphere Portal Express
Accelerators for WebSphere Portal
WebSphere Dashboard Framework
WebSphere Everyplace Mobile Portal Enable
Page 38
Future of Middleware
Client/server computing gave way to networked
computing
More users. More Devices. More transactions.
More demand for ways to integrate applications,
processes, systems, users and institutions.
SOA is a natural progression for middleware
Page 39
Services
(SOA)
Monolithic
Architectures
Pre 1950s
To 1960s
1970s to
mid 1980s
Mid 1990s to
early 2000s
Today
Late
1990s
Sub-routines
/Remote
Procedure
Calls
Remote
Object
Invocation
Message
Processing
Enterprise
Application
Integration
(EAI)
1980s to
mid 1990s
Increasing Modularity to Achieve Flexibility
IT Architectural Evolution: Making IT More Responsive
Page 40
The Next Step in the Connectivity Evolution
Page 41
Service Oriented Architecture
SOA is a way of implementing services that decouple application
logic components and thereby facilitate rearrangement and reuse of
software modules or objects. Once the software components have
been decoupled, they need to be reconnected using messaging that
passes transactions between the SOA components.
Page 42
Flexible IT
Composable IT Services
Flexible Business Model
5ervice Oriented Architecture
Composable Processes
Flexible Business Requires Flexible IT
A service-oriented architecture (SOA) is an application framework that
takes everyday business applications and breaks them down into
individual business functions and processes, called services. An SOA lets
you build, deploy and integrate these services independent of
applications and the computing platforms on which they run.
Page 43
A
p
p
s

&

I
n
f
o

A
s
s
e
t
s
Business Innovation & Optimization Services
D
e
v
e
l
o
p
m
e
n
t
S
e
r
v
i
c
e
s
Interaction Services Process Services Information Services
Partner Services Business App Services Access Services
Integrated
environment
for design
and creation
of solution
assets
Monitor,
manage
and secure
services,
applications
&
resources
Facilitates better decision-making
with real-time business information
Enables collaboration
between people,
process & information
Orchestrate and
automate business
processes
Connect with trading
partners
Build on a robust,
scaleable, and secure
services environment
Facilitates interactions
with existing information
and application assets
ESB
Facilitates communication between services
I
T

S
e
r
v
i
c
e
M
a
n
a
g
e
m
e
n
t
Infrastructure Services
Optimizes throughput,
availability and performance
Manages diverse
data and content in a
unified manner
SOA Reference Architecture
44
8
5
0
+

A
p
p
s

&

I
n
f
o

A
s
s
e
t
s
Business Innovation & Optimization Services
D
e
v
e
l
o
p
m
e
n
t

S
e
r
v
i
c
e
s
Interaction
Services
Process
Services
Information
Services
Partner
Services
Business App
Services
Access
Services
Enterprise Service Bus
I
T

S
e
r
v
i
c
e

M
a
n
a
g
e
m
e
n
t
Infrastructure Services
Rational Software Architect
Rational Data Architect
WebSphere Enterprise Developer
Rational ClearCase
IBM Workplace Designer
Lotus Domino Designer
WebSphere ESB
WebSphere DataPower
WebSphere II Classic Federation
WebSphere Adapters
WebSphere Everyplace Deployment
WebSphere Portal
IBM Workplace
Collaboration Services
DB2, IMS. CICS, SAP,
Oracle, Siebel, etc
Information Integration Services
WebSphere Information Integrator
WebSphere Information Analyzer
WebSphere QualityStage
WebSphere DataStage
WebSphere II Event Publisher
WebSphere II Replication
WebSphere Information Server
WebSphere Metadata Server
WebSphere Data Integration Suite
Data Services
DB2 Universal Database
DB2 UDB Data Warehouse Edition
Informix, IMS, Cloudscape, U2
Information Accelerators
Industry Data Warehouses
(BDW, CIW, IIW, TDW, etc.)
DB2 Entity Analytics
DB2 Alphablox
Content & Discovery Services
DB2 Content Management
WebSphere II Content Ed
WebSphere Content Discovery Server
WebSphere II OmniFind Ed
IT Process Management
Tivoli Availability Process Manager
Tivoli Release Process Manager
Tivoli Storage Process Manager
IT Service Management Platform
Tivoli Change & Configuration
Management Database
Storage Management
Tivoli Storage Manager
Tivoli Continuous Data Protection
for Files
TotalStorage Productivity Center
Security Management
Tivoli Access Manager
Tivoli Identity Manager
Tivoli Federated Identity Manager
Tivoli Directory Server
Tivoli Directory Integrator
Tivoli Security Compliance Manager
Netcool for Security Management
Business Application Management
Products (not listed)
Server, Network & Device Mgmt
Products (not listed)
SOA IBM Product Mapping
Page 45
SOA Sessions
SOA Top 10 Client Questions and IBMs Answers Rachel Rachel
Reinitz
Managing Assets in SOA Governance with RAM and WSRR
(Repeat Available) - Randy KK Tan
Case Study: An SOA Deployment Inside IBM - Hannah Parker
Page 46
Questions and Answers
Yak Yew Cheng
IBM WebSphere Application Server
Advanced Performance: Whats new for 64-bit performance
IBM Confidentiality Statement
IBM Confidentiality Statement
This presentation is IBM confidential
All statements regarding IBM's future plans, direction and intent are subject to change or withdrawal
without notice, and represent goals and objectives only. Such statements do not represent a commitment
of future availability, content, performance or function of any products or features.
3
Roadmap of todays presentation
WAS and the 64-bit platforms supported
Key Advantages of WAS on 64-bit platforms
Downside of the 64-bit platforms
Scalability and other useful information about 64-bit platforms
How to decide if 64-bit is for you?
New features in WAS V7 64-bit
Summary
Resources
Q&A
4
Roadmap
V6.0.1, V6.0.2 and V6.1 (and V7)
64-bit platforms: POWER, x86-64, SPARC
Summary
Resources
5
IBM WebSphere Application Server (WAS) 6.0.x introduces support for
64-bit platforms
V6.0.x
Linux on x86-64 platform, on POWERplatform (LoP) and on IBM System z
Windows Server 2003 Enterprise x64 Edition on x86-64 platform
AIX on POWER platform
HP-UX on Itanium 2
V6.1
All the V6.0.x platforms
Solaris 10 on SPARCand x86-64
All IBM platforms now run IBM J9 (5.0) JVM for both 32-bit and 64-bit
V7
All of the above platforms
All IBM platforms will run IBM J9 (6.0) JVM
6
WAS V6 and 64-bit platforms
POWER
Performance Optimization With Enhanced RISC
Supports both 32-bit and 64-bit computing at full native performance (not an emulation
layer)
Simultaneous multithreading (SMT), large number of registers (120 GPRs, 120 FPRs)
Large caches, very high chip-memory bandwidth
Virtualization, micro-partitioning
x86-64
In 2003, AMD introduced the x86-64 architecture as an enhancement to the Intel x86-32
in their Athlon64and Opteron processors
Adopted by Intel under the name (EM64T) in newer Pentiumand XEONprocessors.
Full native performance and compatibility for 32-bit x86 applications
Additional hardware registers available in 64-bit mode
64-bit wide computing for faster high precision computations via double precision integer
support and Instruction set extensions
SUN SPARC
Scalable Processor Architecture
Newer processors support high level of multithreading (8 cores/chip x 8 threads/core)
All Cores connected to memory and I/O subsystems through high bandwidth interconnect
Smaller caches but with high associativity
7
Roadmap
WAS V6 and the 64-bit platforms supported
Leveraging large heaps
Leveraging high precision computations and extra registers
Summary
Resources
8
WAS V6 and 64-bit platforms: Key Advantages
64-bit enabled WAS products
leverage 64-bit performance
features
Capability of Javaheaps much larger
than the ~2Gb limits of the 32-bit
platforms
Transparent IBM Just-in-Time (JIT)
compiler enhancements that generate
optimized machine code to directly
leverage 64-bit platform performance
features
Usage of extra registers (x86-64)
Extended machine instructions and
precision computations
Max. Java Heap available on 32 and 64-bit platforms
2.6
8
0
1
2
3
4
5
6
7
8
9
32-bit platform 64-bit platform
GB
Max Heap tested on 64-
bit in the labs. In reality,
the max heap on 64-bit
is limited only by
physical memory and
OS limitations
9
Large heaps can yield significant performance
WAS 64-bit is able to allocate a heap large enough to cache the entire
dataset
WAS 32-bit cannot, causing repeated access to the database to fetch large
objects over the network
This example shows a best-case scenario for large heap usage by a
WebSphere application.
Applications that require cache entry invalidations will typically not see performance gains
that are this dramatic
Dynamic data is not generally 100% cache-able
88
12735
0
2000
4000
6000
8000
10000
12000
14000
32 bit 64-bit
r
e
q
s
/
s
e
c
o
n
d
32-bit versus 64-bit Performance Comparison
Big Cache Scenario
System Configuration
Workload: PingJDBCRead Modif ied f or Big Cache requirements, port 9080, 50 clients
SUT: AMD Opteron, 4 x 1.8 GHz , 8GB RAM, SLES 9.0 x86_64
10
IBM JVM 64-bit code generation optimizations
The IBM JVM and Just-in-Time (JIT) compiler
automatically take advantage of 64-bit
capabilities for all supported platforms.
JVM supports large heap allocation (no artificial limit below
OS/Platform capabilities)
IBM JIT generates machine code to take advantage of 64-bit
instruction extensions and high precision computational features
IBM JIT leverages extra registers available in 64-bit mode on
x86-64, reducing register spills, memory loads and stores,
improving the bus utilization and overall memory performance
11
Roadmap
Implications of memory growth on the 64-bit platforms
Summary
Resources
12
WAS and 64-bit performance : Downside
The reality of WAS 64-bit performance is a
mixed bag
Applications capable of taking advantage of WAS 64-bit
features can see significant performance gains
The downside for WAS 64-bit applications is a significantly
greater memory footprint that can lead to performance loss
All address references are 64-bit wide, twice the size of
address references in 32-bit deployments
This results in an increased memory footprint reducing
hardware cache efficiency and thus performance
4-byte ptr address ref (in 32-bit)
8-byte ptr address ref (in 64-bit)
13
Primitive fields are NOT references (4 bytes)
Downside of 64-bit platforms
Public Class Foo
{
private int A;
private char[] B;
private HashMap C;
}
14
Array fields are references (8 bytes), compared to 4
bytes on 32-bit.
Public Class Foo
{
private int A;
private char[] B;
private HashMap C;
}
15
Object fields are references (8 bytes), compared to 4
bytes on 32-bit.
Public Class Foo
{
private int A;
private char[] B;
private HashMap C;
}
On 32-bit platform
On 64-bit platform
Same code when
migrated from 32-bit to
64-bit platform uses
~67% extra memory
16
WAS and 64-bit memory growth
In IBM Java for 64-bit all address references are 64-bit wide, twice the size of
address references in 32-bit deployments
This results in a significantly increased memory footprint
POWER yields 60% memory growth for Trade6 on WAS 6.1
x86-64 yields 70% for Trade6 on WAS 6.1
Can reduce hardware cache efficiency, increasing CPI and reducing application performance
WAS 64-bit Memory growth% for Trade6
0
10
20
30
40
50
60
70
80
90
100
Power64 x64
17
Processor cache size impact on 64-bit performance
General performance expectation for applications that do not specifically leverage 64-
bit features is a degradation of 10-35% when run on 64-bit compared to 32-bit
platforms on WAS 6.1
Processor cache size influences relative performance of 32-bit and 64-bit
applications.
Generally speaking bigger processor caches help 64-bit performance
From a price/performance perspective, bigger hardware cache can buy you better
performance
18
You can re-engineer your applications for 64-bit
Many existing applications may not improve by moving to 64-bit, but
you might be able to re-engineer the applications to use the additional
memory and significantly improve performance.
Example: Use co-located memory caches (Dynacache or ObjectGrid) for database
type of workloads
Applications can be moved from WAS 32-bit to WAS 64-bit without
much effort.
Native code (JNI) or any 3
rd
party libraries (which uses native code/JNI) would
require porting/recompilation for 64-bit OS.
19
Roadmap
Scalability and other useful information about 64-
bit platforms
Scalability of WAS 64-bit on multi-core platforms
Running WAS 32-bit on 64-bit platforms
Impact of large heaps on GC Pause times
Summary
Resources
20
WAS 64-bit Scalability on multi-core platforms
Chart above shows scaling on an Intel 2P/8C (quad-core) system running WAS V6.1 32-bit
and 64-bit.
The overall scaling of 64-bit is slightly lower than that of 32-bit on this hardware. Potential
reasons include higher impact on caches and bus utilization in the 64-bit mode.
21
WAS 32-bit on 64-bit platform support
Many customers want to
standardize on 64-bit OS in
their infrastructure. They
should run WAS 32-bit if their
apps dont leverage WAS 64-
bit features.
Support for WAS 32-bit on
64-bit platforms is shown in
the chart.
Running WAS 32bit and
WAS 64-bit on the same
64-bit platform is supported
These instances can be
part of same cell as well
(mixed mode support) with
either a 32/64 bit DMgr
Very few 64-bit platforms
dont support running WAS
32-bit
HW Platform 64-bit OS platforms
WAS 6.02 WAS 6.1.
32-bit 64-bit 32-bit 64-bit
POWER AIX 5L 5.2
POWER AIX 5L 5.3
POWER Red Hat AS 4.0
POWER SLES 9
POWER SLES 10
SPARC Sun Solaris 8
SPARC Sun Solaris 9
SPARC Sun Solaris 10
x86-64 Sun Solaris 10
x86-64 Red Hat AS 4.0 V6.1.0.11
x86-64 SLES 9 V6.1.0.11
x86-64 SLES 10 V6.1.0.11
x86-64 Windows Server 2003 V6.1.0.11
zSeries zLinux RH AS 4.0**
zSeries zLinux SLES 9**
zSeries zLinux SLES 10**
zSeries z/OS 1.6**
PA-RISC HP-UX 11iv2
Itanium 2 HP-UX 11iv2
Supported
Not Supported
** WAS is 31-bit or 64-bit on zLinux and z/OS
22
Running WAS 32-bit on 64-bit platforms
WAS is available in both 32 and 64-bit
editions for many supported 64-bit
platforms
32 and 64-bit WAS installs can co-exist on
these 64-bit platforms
Applications can be deployed on both 32 and
64 bit to compare performance
Provides a simplified migration path for 32-bit
applications
32-bit applications run natively (full
performance is contingent on OS
implementation)
Customers can choose the preferred edition
for optimal performance
WAS 32-bit on 64-bit operating systems can
provide incrementally greater heap size
~3Gb heaps are possible on Linux
Linux Process
Virtual Address Space
0x00000000
0x08048000
0x40000000
(1GB)
0xC0000000
Kernel Text
User Text
User Data
Shared Libs
Java Heap
Kernel Virtual Mem
On Linux the Address where shared
libs are loaded can be moved
leaving more room for the Java
heap
echo "0x10000000" > /proc/self/mapped_base
The above command moves it to 256M
23
Impact of large heaps on GC Pause times
Java does not set an artificial
limit on maximum heap size.
But what is the effect on GC with
very large heaps?
IBM Concurrent GC technology
(optavgpause, gencon) options
in large heap environments.
Gencon (Generational GC) can
outperform flat-heap collection while
providing smaller pause times.
But this requires tuning effort, out-of-
box defaults will not give best results
in most cases.
Example application in Lab, tuned to run with
either 1024MB or 6144MB heap. What is the
effect of GC frequency and pause time?
1024MB
Average GC Pause Time: 239 ms
Average GC Mark Time: 224 ms
Frequency: 2000ms
6144MB
Average GC Mark Time: 368 ms
Frequency: 19000ms
6144MB with IBM Concurrent GC (optavgpause)
Frequency: 22000ms
24
Roadmap
The Checklist
Summary
Resources
25
Is 64-bit for me ? The Checklist
Does your application need a Java heap much greater than ~2GB
for higher performance? If Yes WAS 64-bit
Does your application use computationally intensive algorithms for
statistics, security, encryption, etc which can benefit from high
precision computation support? If Yes WAS 64-bit
Does your application need a Java heap just a little more than
~2GB i.e. 2~3GB ? If Yes WAS 32-bit on 64-bit platforms
Do you need to support 64-bit OS although your application doesnt
leverage 64-bit features? If Yes WAS 32-bit on 64-bit platforms
If you answered No to all above questions, then WAS 32-bit on
32-bit platforms is better suited for you
26
Roadmap of todays presentation
Scalability, Cross platform comparisons and other useful information about 64-bit
platforms
Summary
Resources
WAS 64-bit Memory growth% for Trade6
0
10
20
30
40
50
60
70
80
90
100
Power64 x64
Effect of Processor Caches on relative performance of WAS 32-bit and 64-bit
0
500
1000
1500
2000
2500
3000
3500
Intel Paxville - 2ML2 per core AMD Rev E - 1ML2 per core Intel Clovertowns x3550 - 4M
L2 per dual-core segment on
the quadcore die
Intel Tulsa - 2ML2 per core +
16ML3 per chip
Various x86-64 multi-core systems (8 cores per system)
T
h
r
o
u
g
h
p
u
t
32-bit 64-bit
35%
27%
21%
11%
27
Selling to the CIO
We want a full 64-bit stack. We have 64-bit hardware running 64-bit OS, so
we want 64-bit WAS, it would be super-fast! duh!!
We dont want mixed mode of deployment, I dont want to bother with twice as many WAS
installs in my infrastructure
This increases our infrastructure costs more management, patching, licenses, etc
Another J2EE vendor claims they dont have this problem!!
Potential Loss of Sale!!
32
bits
32
bits
64 bits
0x 00000000 00000000
0x 00000000 FFFFFFFF
Compressing the 64-bit bloat
Compressed references: allows a
64-bit JVM to use a pointer smaller
than 64-bits wide to reference
memory (only on IBM platforms)
When pointer compression technology is
used with the 64-bit JVM it optimizes
memory references to be just 32-bits wide
(i.e. compresses all the memory
references from 64-bit to 32-bit)
Allows to run 64-bit deployments
with minimal performance or
memory penalty for applications
which dont exploit 64-bit features.
28
4GB
Memory Layout
Planned for V7
If you are running a 64-bit JVM with a
max heap of 4GB, then all you need
is 4-bytes (32-bits) to reference this
entire heap.
Extending the benefits of compressed references
The benefits of compressed
references dont end at 4GB
We plan to use it up to 32GB
We need 35 bits to reference 32G, but
have just 32 bits (4 bytes)
All objects are 8 byte aligned, which
means lowermost 3 bits of all pointer
addresses are zero
So we right-shift by 3 and store the
address in 32 bits w.o loosing any useful
bits
29
32 GB
Memory Layout
OS and Kernel libs
~4GB
Contiguous
space available
for Java Heap
~28GB
Object A (9 bytes)
Object B (8 bytes)
PAD
8 bytes
00000000
00001000
00010000
00011000
Planned for V7
Provide you usable java heap of
~28GB (maximum)
This would provide improved
performance and memory utilization
at larger heap settings than possible
with current 64-bit JVMs.
Automatic enabling of compressed references in WAS V7
In V7, on the 64-bit deployments, if WAS detects
that the max heap (-Xmx) requested is less than
25GB, it will automatically enable pointer
compression as the default mode.
All our current customers run with heaps less than 25G.
This will enable them to get the advantages of pointer
compression right out of the box.
Overriding the above behavior is possible through command line
switches.
30
Planned for V7
Switch on Compressed references
IBM Java6 will ship a single-image 64-bit
JVM which will support both the
compressed references (64C) mode and
the uncompressed references (64U)
mode.
Defaults to uncompressed mode
You can enable compressed references
explicitly by using the java command line
switch -Xcompressedrefs.
Similarly you can explicitly disable the
compressed references by using the java
command line switch -
Xnocompressedrefs. This would switch
to the default uncompressed mode.
31
Planned for V7
Planned for V7
Performance of 64-bit (compressed) on Linux
Charts above show comparison of WAS 32, 64C and 64U running
Daytrader on Linux at various heap sizes.
Overall with compressed references, the throughput of 64-bit is ~95% of 32-bit across
the heap sizes possible with 32-bit (10-20% improvement over Java5 64U)
64C provides a big memory benefit (~60-70% over Java5 64U) across all heap sizes
and is equivalent to 32-bit in memory requirement
32
Planned for V7
Performance of 64-bit (compressed) on Windows
Daytrader on Windows at various heap sizes
Results similar to those observed on Linux
64C has throughput results ~95% of 32-bit
64C has memory utilization similar to that of 32-bit
33
Planned for V7
Performance of 64-bit (compressed) on AIX
Daytrader on Windows at various heap sizes
The gap between 64U and 32 bit is smaller compared to that seen on Intel platform
due to difference in POWER architecture
64C still provides a performance benefit (~2-5%) and is ~95% of 32-bit
64C provides a big memory benefit (50-60% over Java5 64U) and is equivalent to
32-bit in memory requirement
34
Planned for V7
Improvements in 64-bit uncompressed
There are some base optimizations in the IBM Java6 JVM which apply to both the
compressed and uncompressed modes
Charts above show the impact of these optimizations on uncompressed mode-
JIT optimizations which improve performance of 64-bit uncompressed (~3%)
Java6 improves over Java5 - (a) 32-bit by 5-9% (b) 64-bit (uncomp) by ~12%
Memory allocation and GC algorithm improvements which decrease the memory requirement (~15-
18%)
35
36
Roadmap
platforms
Summary
Resources
37
Summary
Version 6.02 onwards WAS extends support to various 64-bit platforms
WAS 64-bit versions leverage 64-bit platform features providing the capability to have Java
heaps much larger than the ~2Gb limitations of 32-bit platforms
The IBM JVM automatically generates optimized code leveraging 64-bit CPU performance
features
Applications that leverage these features can see significant performance gains when
deployed on a WAS 64-bit install
Applications can see a significantly increased memory footprint requirement on 64-bit
versus 32-bit. This can result in reduced hardware cache efficiency and performance
Features in V7 64-bit (would hopefully) make the above discussion moot
Compressed references reduces the size of the addresses and makes more efficient use of
the heap, it improves processor cache and bus utilization thus improving performance of all
workloads run on it
It would significantly improve the performance of applications running on 64-bit platforms but
not leveraging the 64-bit features
It would provide a single image solution on the 64-bit platforms for both applications that
leverage and those that dont leverage the 64-bit platforms
38
Roadmap
platforms
Summary
Resources
39
Resources
WebSphere Application Server Performance website
http://www.ibm.com/software/webservers/appserv/performance.html
Provides performance articles, best practices, Trade Benchmark etc.
WebSphere Application Server Infocenter
http://www.ibm.com/software/webservers/appserv/was/library/
Provides install/admin guides, software/hardware pre-requisites, OS related tuning info
Redbooks
http://www.redbooks.ibm.com
Best Practices for High-Volume Web Sites
DB2 UDB/WebSphere Performance Tuning Guide
IBM WebSphere V6.1 Performance, Scalability, and High Availability WebSphere Handbook
WebSphere Version 6 Application Development Handbook
Questions
Questions
Please complete your session evaluation form
Thank you!!
H01 Service integration bus security
H01 Service integration bus security
or How to get onto the security bus
or How to get onto the security bus
2008 WebSphere Services Technical Conference
2
2
Plumbing
Plumbing
Summer 2008 the UK had record
rainfall causing massive floods
A contributing factor was that the
storm drains could not cope.
In other words the plumbing
could not cope.
3
3
Objectives
Objectives
Equip the WebSphere Application
Server, WebSphere ESB or WebSphere
Process Server practitioner with the
skills required to secure a Service
Integration Bus and its clients
4
4
Agenda
Agenda
Basic concepts
Securing a bus
Securing a messaging engine
Connecting to a bus
Destination authorization
Topic authorization
What is new in WAS v7
5
5
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
6
6
IBM Confidential
Basic concepts
Basic concepts
cell
srv1
srv2
bus
app
me1
me2 q1
ts1
q2
7
7
IBM Confidential
Before you begin
Before you begin
Ensure there are no in-doubt transactions
Stop the bus
8
8
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
9
9
IBM Confidential
Enabling global security
Enabling global security
Authentication uses the user
registry
10
10
Enabling bus security
Enabling bus security
Enabled by default
If admin security is enabled, clients
connecting to the bus will be authenticated and
authorization checks performed.
Does not prevent unencrypted messaging traffic
Does not protect message store
11
11
IBM Confidential
Securing intra
Securing intra
-
-
bus comms
bus comms
me2
me1
supertedCellManager02/interEngineAlias
sibUser/sibPassword

12
12
2006, 2008 IBM Corporation
IBM Confidential
Securing intra
Securing intra
-
-
bus comms
bus comms
Encrypt traffic between
messaging engines
Set inter-engine transport chain
Must be available to all bus members
Explicitly to InboundSecureMessaging
InboundSecureMessaging default for
secure bus
6.0
6.1
13
13
IBM Confidential
Bus security panel
Bus security panel
6.1
14
14
IBM Confidential
DCS encryption
DCS encryption
Messaging resource locations
shared via DCS
By default DCS does not use SSL
15
15
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
16
16
IBM Confidential
Securing the data store
Securing the data store
Messaging engine can specify
credentials when connecting to
the database
Stored on the datastore as an auth alais
Do not set component
alias on data source
17
17
IBM Confidential
Disabling insecure transports
Enabling bus security does not
prevent the use of insecure
transports
InboundBasicMessaging
should be disabled or
deleted
Not possible for secure
and insecure buses to share a server
6.0
18
18
IBM Confidential
Permitted chains
Any defined chain
Chains protected by SSL
Chain with a specified name
Secure and insecure buses can share a
server
18
6.1
19
19
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
19
20
20
IBM Confidential
Connecting to the bus
Connecting to the bus
lo
o
k
u
p
cell
srv1
me1@srv1
srv2
bus
client
me1
c
o
n
n
e
c
t
q1
21
21
Connection lifecycle
Connection lifecycle
Connections can be long lived
Updates to authorization do not force a
disconnect
User to group assignments refreshed from
repository periodically
Every two hours (based on LTPA token timeout)
22
22
Authorization to connect
Authorization to connect
Client must be authorized
Specified using bus connector role
Must use wsadmin
Can use admin console
6.1
6.0
23
23
IBM Confidential
Specifying connection credentials
Specifying connection credentials
Authentication mode set in
resource reference res-auth
Application
Credentials passed on createConnection call
credentials hard-coded into application
Component-managed authentication alias
server based applications with JNDI access share
credentials
Container
Credentials provided by container
Authentication alias set on resource-reference
only this application can use the credentials
24
24
IBM Confidential
Using container managed authentication Using container managed authentication
Alias defined in admin console
has a prefix
Alias defined in wsadmin does not
Specify on resource reference
25
25
IBM Confidential
Specify an XA recovery alias
Specify an XA recovery alias
Credentials defined on
resource-reference are not
available during transaction recovery
Specify XA Recovery
alias on ConnectionFactory
Alias is used during
recovery
User only needs bus
connector role
25
26
26
IBM Confidential
Message
Message
-
-
driven bean credentials
driven bean credentials
Define in the message-driven
bean binding
Used for initial connection and XA recovery
27
27
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
28
28
IBM Confidential
Role
Role
-
-
based authorization
based authorization
Role-based policy applies to bus
Role Types
Special groups: AllAuthenticated, Everyone,
Server
6.1
sender receiver browser
creator
29
29
IBM Confidential
Role applicability
Role applicability
Destinations
Queue Topic
Space
Alias Foreign Web
service
Port Temporary
destination
prefix
Role
Type
sender

Receiver

Browser

Creator

30
30
Policy Enforcement Point
JMS
Application
Security Destination
createProducer
checkAccess
send
send
31
31
Default authorization policy
Default authorization policy
Groups that can connect
AllAuthenticated
Server
All connected users have full access
Not suitable if application isolation is needed
6.1
6.0
32
32
Default roles
Default roles
destination 1
default
destination 2
default
access roles
for
destinations
destination
with no access
roles
destination
with access
roles and
using default
33
33
Administration
Administration
Only via wsadmin
AdminTask.help(SIBAdminBusSecurityCommands)
Anatomy of addGroupToDestinationRole
Parameter Description
type The type of the destination, one of: Queue, Port, Webservice, TopicSpace,
ForeignDestination, Alias
bus The bus name
destination The destination name
role The role type, one of: Sender, Receiver, Browser, Creator
group The group name
foreignBus The name of a foreign bus the destination is defined to. Mandatory for
foreign destinations, optional for alias destinations.
34
34
IBM Confidential
Temporary destinations
Temporary destinations
Names are generated on creation
Security configured
using just the prefix
Prefix set on JMS
Connection Factory
<uh1que va1ue> <pef1x>
35
35
IBM Confidential
Temporary destination authorization
Temporary destination authorization
Only two roles relevant
Example:
AdminTask.addGroupToDestinationRole('[ -type Queue
bus isswbus -destination isswTmpDest -role Creator
-group isswUsers]')
Resultant queue name
A334C30000001 1ssWTmp0esf
36
36
IBM Confidential
Default prefix
Default prefix
Default prefix is the empty string
Security can be applied to default:
AdminTask.addGroupToDestinationRole('[ -type Queue
bus isswbus -destination -role Creator
-group isswUsers]')
Prevent it being used:
AdminTask.setInheritDefaultsForDestination('[
-type Queue -bus isswbus
-destination -inherit false]')
37
37
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
38
38
Topics
Topics
sports
baseball
football
soccer rugby American
basketball
Default.Topic.Space
39
39
Topics (2)
Topics (2)
Topics are pre-declared
Child topics inherit roles from parents
sports
baseball
football
basketball
secured
topic
40
40
Topics (3)
Topics (3)
Applicable Roles
sports
baseball
football
basketball
41
41
IBM Confidential
Topics (4)
Topics (4)
Topics inherit role assignments
from their parent
Inheritance can be blocked
sports
baseball
football
basketball
block
inheritance
42
42
JMS
Application
Security Destination
createConsumer
checkAccess
receive
checkAccess
43
43
Examples
Examples
baseball
soccer rugby
sports
football
American
basketball
alice
bob
keys
simon
bill
44
44
Examples (1)
Examples (1)
Who can access?
sports
baseball
football
basketball
alice
bob
keys
simon
bill
alice
bob
keys
spofs/baskefba11
45
45
Examples (2)
Examples (2)
Who can access?
sports
baseball
football
basketball
alice
bob
keys
simon
bill
alice
bob
simon
spofs/foofba11/socce
46
46
IBM Confidential
Examples (3)
Examples (3)
Who can access?
sports
baseball
football
basketball
alice
bob
keys
simon
bill
bill
spofs/foofba11/Ame1cah
47
47
IBM Confidential
Topic space root
Topic space root
What is to stop a games
hierarchy being added?
sports
baseball
football
basketball
root
games
48
48
IBM Confidential
Agenda
Agenda
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
What is new in WAS v7 Beta Insights
48
49
49
Disclaimer
Disclaimer
THE FOLLOWING INFORAMTION ON
POTENTIAL CAPABILITIES OR PRACTICES
FOR THE FUTURE RELEASE OF WAS 7.0
IS FOR ILLUSTRATIVE PURPOSES ONLY.
THE PRESCENCE OF THE INFORMATION
IN THIS PRESENATION IS BASED ON
CAPABILITIES OF THE WAS 7.0 BETA AND
IBM MAY CHOOSE TO CHANGE AND
MODIFY THIS BEFORE THE FINAL GA
RELEASE.
50
50
IBM Confidential
Highlights
Highlights
Admin console support for
authorizations
Multi domain
Audit
Simplified admin
WebSphere MQ
7.0
51
51
IBM Confidential
Authorization administration
Authorization administration
Authorization model can be
fully configured using admin
console
Accessed via bus
security panel
51
7.0
52
52
IBM Confidential
Administer defaults
Administer defaults
allows access to
be allowed or
disallowed
wizard for adding
new users or
groups to role
7.0
53
53
Add wizard
Add wizard
search
pattern
search
groups
or users
limit the
number of
results
7.0
54
54
Select groups
Select groups
select the
groups of
interest
7.0
55
55
IBM Confidential
Assign roles
Assign roles
assign the
relevant
roles
7.0
56
56
IBM Confidential
Results
Results
7.0
57
57
Multi domain
Multi domain
A bus can be in its own domain
All messaging engines use one
domain.
will read
global
domain when
released
will read
global
domain when
released
7.0
58
58
Audit
Audit
Authentication and authorization
Using new audit support for WAS
Can be disabled for a bus
Can be disabled for a topic space
7.0
59
59
Securing intra
Securing intra
-
-
bus comms
bus comms
me2 me1
engine
identity
LTPA
No longer
used in v7
buses
No longer
used in v7
buses
Engines
mutually
authenticate
Engines
mutually
authenticate
7.0
60
60
XA recovery alias
XA recovery alias
No longer required
Unless contacting a bus in another
cell
Even this is possible if LTPA keys are shared
Server group must be in bus connector role
7.0
61
61
Permitted chains
Permitted chains
Chains that are not permitted
will not be started
Ports will not be open
Chains started or stopped when policy
changes
7.0
62
62
Client SSL authentication
Client SSL authentication
Supported
Identity in client certificate used for
authorization
6.x clients are supported
Simply configure the client to use mutual SSL
and provide no, or null credentials
7.0
63
63
WebSphere MQ
WebSphere MQ
Major Revamp of JMS integration
JMS provider uses JVM wide
keystore and trust store settings
Unreliable as WAS changed these
PK24431 added ssl.repertoire for per
connection factory key and trust store
7.0
64
64
WebSphere MQ SSL
WebSphere MQ SSL
Per connection factory SSL
configuration supported
7.0
65
65
WebSphere MQ SSL (2)
WebSphere MQ SSL (2)
Centrally managed SSL
configuration
7.0
66
66
WebSphere MQ security
WebSphere MQ security
Additional SSL properties still
supported
Admin console support for security exits
7.0
67
67
Summary
Summary
Basic concepts
Securing a bus
Connecting to a bus
Topic authorization
68
68
References
References
WebSphere Application Server v6.1
infocenter
http://tinyurl.hursley.ibm.com/213
WebSphere Application Server V6.1: JMS
Problem Determination
Connect an application to the default
message provider using the run-as identity
WebSphere Application Server v7.0 open
beta
69
69
Questions
Questions
their session evaluation form H01
Thank you!!
Session F02: Using Tivoli Federated
Session F02: Using Tivoli Federated
Identity Manager with WebSphere
Identity Manager with WebSphere
Application Server (SS0)
Application Server (SS0)
22
Conference Speaker
Conference Speaker
Using Tivoli Federated Identity
Using Tivoli Federated Identity
Manager with WebSphere Application
Manager with WebSphere Application
Server (SSO)
Server (SSO)
Session Number: H02
Session Number: H02
33
Agenda
Agenda
Provide an overview of the basic concepts of Federated
Single Sign-On
Outline new Point of Contact callback interface in TFIM 6.2
Describe use of WebSphere Application Server as Point of
Contact for TFIM
Identity and Service Provider perspectives
Extending SSO to non-WebSphere applications
Outline installation, deployment and management of TFIM
Outline new entry-level entitlement to TFIM for WebSphere
6.1/7.0 customers
Provide an overview of Tivoli Federated Identity Manager
(TFIM) and highlight how it leverages Tivoli Access Manager
(TAM) to provide drop-in federation for applications.
44
Federated Single Sign
Federated Single Sign
-
-
On
On
Source Web Site
www.ibm.com
Destination Web Site
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernese@ibm.com
.
0
.

R
e
q
u
e
s
t

S
S
O
55
IBM
IBM
s Identity Flow Solution

s Identity Flow Solution
IBM Tivoli Federated Identity Manager (TFIM)
IBM Tivoli Federated Identity Manager (TFIM)
TFIM provides Single Sign-On (SSO)
across domains :
between Web Applications
between Services in SOA
to Legacy applications
TFIM is like a multi-purpose adapter
for passing identity across domains
66
Terminology
Terminology
Confidentiality
Only authorized actors or security token owners should be able to view
the data
Authentication
The identity of the party should be validated based on some credentials
Authorization
The party should have the permissions to access the requested
resource.
Integrity
There should be a guarantee that information is not modified in transit
Non-repudiation
The sender and receiver are able to provide legal proof to a third-party
that the sender sent the information and receiver received the
information.
77
Terminology
Terminology
Federated Identity
Existence of an agreement between providers on a set of identifiers
and/or attributes to use to refer to a principal
Asserting party (SAML Authority)
An entity that produces SAML assertions
Identity Provider
An entity that creates, maintains, and manages information for
principals and provides principal authentication to other Service
Providers
Relying party
An entity that decides to take an action based on information from
another system entity
Service Provider
An entity that provides services to principals or other entities
88
Identity Federation
Identity Federation
A users identity is said to be federated between a set of providers when
there is an agreement between the providers on a set of identifiers and/or
attributes to use when referring to a user.
Different forms of Identity Federation:
Single Sign-On with Out-of-Band Account Linking
Userids may differ between sites (out-of-band resolution)
SAML, WS-Federation
Attribute Federation
Attributes of the user (in the assertion) are used to link the account
SAML, WS-Federation
Persistent Federation
A persistent name identifier is used to identify each user
SAML 2.0, Liberty ID-FF 1.1/1.2
Transient Federation
Supports role-based and anonymous identity mapping
SAML 2.0, Liberty ID-FF 1.2
99
Federated identity
Federated identity
Federated identity technology is used for creating a globally interoperable online
business identity, driving relationships or affinity driven business models between
companies.
a passport is a global identity credential that vouches for one's identity in a
country
an ATM card is a credential that vouches for one's bank account
10 10
Identity Federation
Identity Federation
SSO with OOB Acct Linking

Simple case same user id on
both sides
Source Web Site
www.ibm.com
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernese
.
ernese|
ernese|
Assertion
11 11
Identity Federation
Identity Federation

(cont)
(cont)
Mapping between identities is not
defined by the specifications.
Source Web Site
www.ibm.com
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernie
.
ernie|
ernese|
Assertion
?
12 12
ernie
ernese@ibm.com
Identity Federation
Identity Federation
Identity mapping based on some
shared attribute
Source Web Site
www.ibm.com
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernese|ernese@ibm.com|
ernese|ernese@ibm.com|
Assertion
13 13
1234
.
Identity Federation
Identity Federation
Federation based on name
identifiers
Separate user-driven account
linking operation to establish name
identifiers
Source Web Site
www.ibm.com
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernie|1234|
ernese|1234|
Assertion
14 14
Establishing Persistent Account Linking
Establishing Persistent Account Linking
SP IdP
Account Linkage Trigger
Response to client
Link
Accounts
Generate
and store
Alias
Send
Assertion
Indicate
Account
Linkage
required
Consent + Authentication
SSO Exchange (requiring account linkage)
Authentication
Local
Authentication
15 15
1234
.
Identity Federation
Identity Federation
Federation based on name
identifiers
Name identifiers not stored only
used for one operation
Source Web Site
www.ibm.com
my.travel.com
1
.

A
u
t
h
e
n
t
i
c
a
t
e
3
.

A
c
c
e
s
s

R
e
s
o
u
r
c
e
Identity Provider
Service Provider
2
.

A
s
s
e
r
t

I
d
e
n
t
i
t
y
ernese|
guest|
Assertion
16 16
Single Logout (SLO)
Single Logout (SLO)
Allows session logout for all of the users Web SSO sessions
No logout of sessions the user has established through other protocols
No logout of sessions the user has established through other federations
Identity Provider and Service Provider initiated forms exist
However, the Identity Provider coordinates the SLO messages
(If SP initiated) Service Provider sends SLO request to relevant Identity Provider for
processing.
Identity provider sends messages to all (other) Service Providers in the federation
which have received an authentication assertion for the current user session
Once replies received, Identity Provider logs out
(If SP initiated) Service Provider that initiated the request logs out
SLO is supported by SAML 2.0, WS-Federation and Liberty ID-FF 1.1/1.2
Safest option is to close all browser sessions.
17 17
Mobile Device Single Sign
Mobile Device Single Sign
-
-
On
On
Liberty ID-FF 1.1/1.2
SAML 2.0
Source Web Site
www.ibm.com
my.travel.com
Enhanced Proxy
www.telco.com
Adoption ??
18 18
Attribute Query
Attribute Query
An Attribute Statement may be included with an Authentication Statement
in a Web SSO Assertion
However, there are times that a Service Provider may wish to request
some additional attributes
E.g.. For Just-in-Time provisioning at the Service Provider.
www.ibm.com
www.travel.con
Return the requested
Attributes for this Subject
(SAML Assertion)
19 19
Authorization Decision Query
Authorization Decision Query
Similarly, the Service Provider may request an authorization decision
from the Identity Provider using SAML 2.0.
www.ibm.com
www.travel.con
Should these actions on this
resource be allowed for this
subject given this evidence ?
(SAML Assertion
containing Decision)
From SAML 2.0 specifications:
Note: The <AuthzDecisionStatement> feature has been frozen as of SAML V2.0, with no
future enhancements planned. Users who require additional functionality may want to
consider the eXtensible Access Control Markup Language [XACML], which offers
enhanced authorization decision features.
20 20
TFIM and Federated SSO
TFIM and Federated SSO
Traditionally:
To achieve (browser based) Federated Single
Sign-On, TFIM is integrated with WebSEAL
Loose coupling with applications
Support for comprehensive set of web applications
Low impact solution for TAMeB customers
However:
TAMeB is not for everyone
Not well suited to entry-level deployments
Customers may have deployed a solution based on a
TAMeb competitor (or custom solution)
21 21
WebSEAL handles user authentication, authorization, session tracking
and can makes attributes available to applications
interface to WebSEAL is External Auth Interface (EAI)
WebSphere
FIM Runtime
User-1
Login/Session
WebSEAL
point of contact
Users Groups: Attributes
User-1 Group-1 e-mail
TAM Registry
Identity/Attributes
Applications
authentication (IdP)
cred/attributes (IdP)
identity mapping rules
some attribute source
W
e
b
S
E
A
L

E
A
I

I
n
t
e
r
f
a
c
e
Token
Identity/Attributes
TFIM with
TFIM with
WebSEAL
WebSEAL
as
as
Point of Contact
Point of Contact

22 22
Extending the Reach of TFIM
Extending the Reach of TFIM
In TFIM 6.2
Interface to point of contact server opened up
API for custom solutions
During beta program, this API was used to develop a
solution for GetAccess
Integration with other SSO products possible
Solution for WebSphere Application Server 6.1
as point of contact server
Modules for consumption of LTPA token by:
Microsoft IIS version 6
IBM HTTP Server 6.1/Apache 2.2.4
23 23
Identity Transfer to/from TFIM
Provide identity & attributes to TFIM Runtime (IdP)
Accept identity & attributes from TFIM Runtime (SP)
Access Control to Protocol End-Points
SSO protocols define end-points i.e. URLs associated with various
protocol operations
PoC must provide a mechanism to enforce the appropriate URL
access policy
A SOAP end-point will likely require a separate authentication policy
Basic Auth or Client-side Certificate
Session Termination
Some SSO Protocols provide Single Logout (Liberty and SAML 2.0)
PoC must accept session termination notification from TFIM
Identity/Attribute Transfer to Applications (SP)
PoC should also provide a mechanism to transfer user identity and
attributes to other application environments
Requirements for a Point of Contact
Requirements for a Point of Contact
24 24
The Point-of-Contact interface is encapsulated into callback plug-ins
that reside in TFIMs SSO Protocol Service (SPS)
Custom point-of-contact servers can be supported by providing plug-ins
that adapt to that PoC
Point of Contact Server
Authentication
Services
SAML 1.x WS-Federation Liberty SAML 2.0 CardSpace
SSO Protocol Layer
PoC Profile
Identity Callbacks
Authenticate
End User
Get Local
Identity
Local
Token
class
for PoC Sign-in to PoC
Authenticate User
Callbacks
Authenticate User
Callbacks
Sign-in Callbacks
Sign-in Callbacks
Sign-out Callbacks
Sign-out of PoC
Point of Contact Callback Interface
Point of Contact Callback Interface
25 25
WebSphere security handles user authentication, authorization, session
tracking and makes attributes available to WebSphere applications
WebSphere 6.1
FIM Runtime
User-1
Login/Session
Users Groups:
User-1 Group-1
WebSphere
Registry
I
d
e
n
t
i
t
y
/
A
t
t
r
i
b
u
t
e
s
Applications
authentication (IdP)
WAS credential
identity mapping rules
some attribute source
WebSphere
Security
FIM JAAS
LOGIN
(SP)
Identity/Attributes
Subject
(IdP)
WebSphere as Point of Contact for TFIM
WebSphere as Point of Contact for TFIM
26 26
Federated SSO protocols require authentication for some URL endpoints
and unauthenticated for others
To simplify deployment, TFIM defines a J2EE security constraint for only
URL endppint
/sps/wasauth requires J2EE role FIMAnyAuthenticated
TFIM re-directs the user to this authenticated URL when authentication is
required
FIMRuntime - J2EE Application
SSOProtocolServiceWeb Web Module
c
o
n
t
e
x
t
-
r
o
o
t

=

S
P
S
S
S
O
P
S
S
e
r
v
l
e
t
/sps/<fed name>//login
This servlet (called the SPS) handles
all federated-SSO requests.
All URLs start with /sps.
All URLs are public except for
/sps/wasauth.
/sps/wasauth requires an authenticated
WebSphere user
In general, requests to the
SPS are public no J2EE
role restriction
/sps/wasauth
This URL is the exception.
Access to /sps/wasauth
requires authentication to
WebSphere
WebSphere
WebSphere
PoC
PoC
J 2EE URL Authorization

J 2EE URL Authorization
27 27
Browser
TFIM SPS Servlet WebSphere Security
/sps/<fed name>//login?TARGET=
allow access to
unauthenticated
user
302 re-direct /sps/wasauth
/sps/wasauth
authenticated
user required
Authenticate User with WAS registry
/sps/wasauth
re-direct to
original SSO URL
set LTPA cookie
WAS UserId
302 re-direct /sps/<fed name>//login?TARGET=
pass
authenticated
identity
Java Subject now
available,
perform SSO
processing
302 re-direct to value of TARGET= along with:
WAS UserId
SAML-Assertion/Artifact
LTPA Cookie
Java Subject
required but not
available, redirect
to /sps/wasauth
Original SSO Request
saved in session data
WebSphere
WebSphere
PoC
PoC
Identity Provider Flow

Identity Provider Flow
28 28
Recall the following sub-flow of the Identity Provider
flow from the previous slide
In order for this to work, WebSphere must
propagate the users identity for an unprotected
URL
Browser
WAS UserId
Java Subject now
available,
perform SSO
processing
pass
authenticated
identity
LTPA Cookie
WebSphere
WebSphere
PoC
PoC
Identity Propagation
29 29
TFIM Installer enables this option for Embedded WebSphere
Must be configured manually for installation to an existing
WebSphere server
WebSphere
WebSphere
PoC
PoC
Config
Config
.
.
30 30
Two supported options for user authentication
customizable login form
desktop SSO via WebSphere SPNEGO TAI
Browser
/sps/wasauth
authenticated
user required
Authenticate User with WAS registry
/sps/wasauth
re-direct to
original SSO URL
set LTPA cookie
WAS UserId
This user authentication can be done via a
customizable form or WebSphere
SPNEGO TAI
WebSphere
WebSphere
PoC
PoC
User Authentication
User Authentication
31 31
Browser
allow access to
unauthenticated
user
TFIM JAAS
login module
Create login
context for SP
userid
set LTPA cookie
302 re-direct to value of TARGET
Validate identity
assertion and
determine Service
Provider userid
WebSphere
WebSphere
PoC
PoC
Service Provider Flow

Service Provider Flow
Set userid and attributes
SAML-Assertion/Artifact SAML-Assertion/Artifact
32 32
TFIM 6.1: TAM groups can be added by mapping rules
TFIM 6.2 also allows dynamic group membership in
WebSphere credential
Map Identity to
Partner-X
Add Groups:
Finance
GoldLevel
Identity
Provider
Identity
Provider
Token
WAS Credential
UserID: Partner-X
Groups: Group-X
WAS Registry
UserID: Partner-X
Groups: Group-X
Finance
GoldLevel
SP Mapping Rules
Map Identity to
Partner-X
Add Groups:
Finance
GoldLevel
Identity
Provider
Identity
Provider
Token
TAM Credential
UserID: Partner-X
Groups: Group-X
TAM Registry
UserID: Partner-X
Groups: Group-X
Finance
GoldLevel
SP Mapping Rules
WebSphere
WebSphere
PoC
PoC
Dynamic Group Membership

Dynamic Group Membership
33 33
Inclusion of attributes in LTPA cookie controlled by custom property
(ssoAttributeNames) on TFIM JAAS Login Module
* : include all attributes
attr-1, attr-2 : explicit list of attributes to include (no wild-cards) separated
by commas
Maximum size of attributes allowed in LTPA Cookie is approx. 3K
Browser maximum cookie size is 4K, WebSphere attributes use up
approximately 1K
WebSphere Registry
WebSphere Security
SSO Protocol Service (SPS)
in FIM Runtime
User-1
Users Groups:
User-1 Group-1
WAS credential
FIM JAAS
LOGIN Module
Identity/Attributes
When FIM is SP
LTPA Cookie
1
2
3
ssoAttributeNames=*
Custom Property
WebSphere
WebSphere
PoC
PoC
Placing Attributes into LTPA

Placing Attributes into LTPA
34 34
TFIM provides a plug-in for .Net and Apache/HIS to allow SSO
to be extended to these application environments
TFIM Service Provider creates LTPA cookie with identity and
attributes
TFIM Plug-in extracts identity and attributes from LTPA cookie and
provides to application via HTTP Headers
.NET
WebSpere
FIM Service Provider
SSO Token
Identity in
LTPA Cookie
Federated SSO
IIS 6
FIM Plug-in
Other
Applications
IHS/Apache
Identity in
LTPA Cookie
Identity in
LTPA Cookie
Identity in
LTPA Cookie
FIM Plug-in
WebSphere
WebSphere
PoC
PoC
SSO to non
SSO to non
-
-
WebSphere Apps
WebSphere Apps
35 35
The application must be able to accept user identity (and
optionally attributes) via an HTTP header
Service Provider must be configured to use WebSphere as the
point of contact server
or, a custom Point of Contact that creates an LTPA cookie with the
necessary attributes
The userids must exist in both the WebSphere registry and
the applications registry
The TFIM web plug-in must be installed co-resident with the
application
TFIM Service Provider and the application must reside in the
same DNS domain
LTPA cookie must be configured as a domain cookie
LTPA key file and password must be loaded into the TFIM
web plug-in
WebSphere
WebSphere
PoC
PoC
SSO to non
SSO to non
-
-
WebSphere Apps
WebSphere Apps
Points to Note Points to Note
36 36
Federated Single Sign-On
Integration with IBM Tivoli Access Manager
Supported Protocols:
SAML 1.0 / 1.1 / 2.0
WS-Federation
Liberty ID-FF 1.1 / 1.2
Federated Web Services
Identity Propagation in SOA
Passing identity between domains what userid and how is it passed
Authorization (via calls to TAM)
Audit logging
Integration with IBMs Enterprise Service Bus offerings
Datapower XI50
Integration with WebSphere Application Server (and therefore WebSphere Portal, )
Legacy system integration via JAAS login module
RACF/CICS and other legacy applications (via JCA)
JDBC
Enables SAML 2.0 token handling for WebSphere Application Server, WESB, WMB, ..
Federated Provisioning
Provides linking of local provisioning systems
Supported Protocol: WS-Provisioning
37 37
Federated SSO
Federated SSO
--
--
Classic Deployment
Classic Deployment
Model
Model
IdP
Partner A
Identity Provider
Partner B
User
Heavyweight
SSO everywhere
Partner
certificate
management at
every endpoint
38 38
TFIM Model for Service Provider
TFIM Model for Service Provider
Heavyweight SSO
only where needed
Partner certificate
management now
more centralized
Faster deployment
of applications at
SP
Partner A
Identity Provider
Partner B
User
TAM/TFIM
TAM/TFIM
ESM
39 39
TFIM Model for Identity Provider
TFIM Model for Identity Provider
TFIM
Partner A
Identity Provider
Partner B
User
TAM
40 40
TFIM
TFIM
Application Development Perspective

Application Development Perspective
Application is shielded from complexity of Federated SSO protocol(s)
(Lightweight) SSO to application container, and/or
Application can access user identity and attributes from HTTP Headers
Concurrent Federated SSO protocol support
New versions can be supported without changes to the applications
No proprietary APIs
Faster time to deploy new application to Federated SSO environment
Centralized management of partners metadata and certificates
Provided by TFIM not part of application development scope (save time/$)
Drop-in strong authentication or Desktop SSO (via SPNEGO)
Large number of application integration solutions documented
41 41
Managing TFIM
Managing TFIM
TFIM is managed via the ISC
i.e. it is integrated into the WebSphere Administration
Console
TAM is also included in the ISC from version 6.1
42 42
ISC
TFIM
Console
Domain 1
WebSphere
Application Server
TFIM Runtime
TFIM
Management
Service
TFIM Config
Domain 1
TAM
WebSphere
Application Server
TFIM Runtime
TFIM
Management
Service
TFIM Config
Domain 2
TAM
Domain 2
TFIM Console can Manage Multiple Domains
TFIM Console can Manage Multiple Domains
43 43
TFIM Deployment Platform
TFIM Deployment Platform
TFIM is a WebSphere application that runs on:
Embedded WebSphere server
Standalone WebSphere server
WebSphere ND Cluster
In a clustered environment:
TFIM Management Server runs on the Deployment
Manager
TFIM Runtime runs on managed nodes
Shared configuration across TFIM instances
Shared session information across TFIM instances
44 44
Deployment
Manager
Managed Node
Managed Node
Cluster
TFIM
Management
Service
TFIM FSSO
Installer
ISC
TFIM
Console
Installing TFIM
Installing TFIM
45 45
ISC
TFIM
Console
Deployment
Manager
TFIM
Management
Service
TFIM Config
Domain 1
Managed Node
Managed Node
Cluster
In TFIM Console:
Create Domain
TFIM Config
TFIM Config
Domain 1
Creating TFIM Domain
Creating TFIM Domain
46 46
ISC
TFIM
Console
Domain 1
Deployment
Manager
TFIM
Management
Service
TFIM Config
Domain 1
Managed Node
TFIM Runtime Managed Node
TFIM Runtime
Cluster
In TFIM Console:
Deploy TFIM Runtime
TFIM Config
TFIM Config
Deploying TFIM to Managed Nodes
Deploying TFIM to Managed Nodes
47 47
ISC
TFIM
Console
Domain 1
Deployment
Manager
TFIM
Management
Service
TFIM Config
Domain 1
Managed Node
TFIM Runtime
TFIM Config
Managed Node
TFIM Runtime
TFIM Config
Cluster
In TFIM Console:
Configure TFIM Nodes
TAM
TAM
Configuring TFIM Runtime
Configuring TFIM Runtime
48 48
WebSphere
Application Server
TFIM Runtime
TFIM
Management
Service
TFIM Config
Domain 1
ISC
TFIM
Console
Domain 1
Deploying TFIM on a Standalone Server
Deploying TFIM on a Standalone Server
49 49
What is SAML 2.0
What is SAML 2.0
Definition from SAML 2.0 Specification:
The Security Assertion Markup Language (SAML) defines the syntax and
processing semantics of assertions made about a subject by a system
entity.
This specification defines both the structure of SAML assertions, and an
associated set of protocols, in addition to the processing rules involved in
managing a SAML system.
SAML is built around the concept of asserting identity
Similar to drivers licenses, passports, ATM cards,
Avoids need for credential vaults, password synchronization,
http://www.oasis-open.org/specs/index.php#samlv2.0
50 50
Profiles
Combinations of protocols, bindings and assertions to
support defined use cases
Bindings
Mappings of (abstract) SAML protocols to (concrete) messaging
or communications protocols
Protocols
Message exchanges (encoded in an XML schema)
for dealing with Assertions and related concepts
SAML 2.0 Contents
SAML 2.0 Contents
Assertions
Authentication, attribute and entitlement
information
Authentication
Context
Types and strengths
of authentication
Metadata
Used to
exchange
configuration
data between
partners
51 51
Assertion
Anatomy of a SAML 2.0 Assertion
Anatomy of a SAML 2.0 Assertion
Issuer
Signature
Subject
Conditions
Statement(s)
Authentication Statement
Authorization Decision Statement
Attribute Statement
52 52
Evolution of Federated SSO Standards
Evolution of Federated SSO Standards
SAML 1.0
Completed: May 2002
OASIS Std: Nov 2002
Liberty ID-FF 1.1
Completed: Jan 2003
Shibboleth 1.0
Completed: Jun 2003
SAML 1.1
Completed: May 2003
OASIS Std: Sep 2003
Liberty ID-FF 1.2
Completed: Oct 2003
Shibboleth 1.1
Completed: Aug 2003
SAML 2.0
Completed: Jan 2005
OASIS Std: Mar 2005
WS_Federation
(Submitted to OASIS)
*
*
OpenID
InfoCard/
CardSpace
Limited Support for Shibboleth
*
53 53
Grouping Partners into Federations
Grouping Partners into Federations
For Federated SSO, TFIM groups sets of partners
into Federations
Only one protocol per federation
Only one role for self per federation
Same set of URL endpoints per federation
TFIM Configuration
Federation 1
SAML 2.0, self role=IdP
Self Partner 1 Partner 2
Federation 2
InfoCard, self role=RP
Self Partner 1 Partner 2
54 54
Entitlement to TFIM for WAS Customers
Entitlement to TFIM for WAS Customers
WebSphere Application Server ND 6.1
Entitled to use TFIM in a restricted manner
Limited to SAML protocols
Limited to 2 federations, with one partner in each
Available at GA of TFIM 6.2
Scheduled for end-June
WebSphere Application Server ND 7.0
(same entitlement as for WAS 6.1)
Available at GA of WAS 7.0
Paper restrictions
No enforcement by TFIM code
Migration to full TFIM license is a paperwork exercise
55 55
Standard TFIM package
Federated SSO
SOA Identity Services
TAM for eBusiness
User-based licensing (and high-end processor based licensing)

TFIM Business Gateway (TFIM-BG)
This has nothing to do with gateways
Entry-level TFIM for Federated SSO
Targeted at partners of TFIM customers
Processor-based licensing
TFIM for SOA
Pre-approved discount offering
Restricted license
No TAMeB or Federated SSO
Targeted at co-selling with IBM ESB products
Processor-based licensing
Fed SSO SOA Identity
Console TAMeB
Fed SSO SOA Identity
Console TAMeB
Fed SSO
Console
How is TFIM Packaged ?
How is TFIM Packaged ?
56 56
Summary
Summary
TFIM traditionally uses WebSEAL to provide Point of Contact for
Federated SSO
Scaleable solution
Comprehensive support for heterogeneous web applications
Broad support for stronger authentication schemes
TFIM 6.2 also provides a Point of Contact callback interface for TAM-
less deployment
WebSphere Application Server as Point of Contact for TFIM
Custom interfaces
Eg for integration with competitive product to TAMeB
As an entry-level solution, WebSphere 6.1/7.0 includes a limited
entitlement for TFIM
Limited to SAML protocols
Limited to 2 federations, 1 partner in each
By implementing SSO to TAM, which in turn provides SSO to the local
applications, TFIM provides:
Faster application integration into a Federated SSO environment
More efficient partner/self management
Improved performance and user experience
57 57
Questions
Questions
Thank you!!
WebSphere Application Server Security:
WebSphere Application Server Security:
Key/Certificate/SSL Management
Key/Certificate/SSL Management
Session Number: H03
Session Number: H03
Kevin Tobin Kevin Tobin
kevintob@au1.ibm.com kevintob@au1.ibm.com
Page 2
Materials may not be reproduced in whole or in part without the prior written permission of IBM
WebSphere Security Presentation Series
This presentation is part of the WebSphere Security Presentation Series
led by Keys Botzum with help from so many others
Available internally at
http://pokgsa.ibm.com/~keys/documents/securitySeries
Related presentations
We assume youve seen or are familiar with
Core Concepts
WAS Security Introduction
You may be interested in
Hardening
Operations
Page 3
Change is the Only Constant
This presentation reflects
My current opinions regarding WAS security
The product itself continues to evolve (even in PTFs)
Presentation is based on V6.1.x w/ some future speculation
This will be revised as we learn more
Your thoughts and ideas are welcome
Page 4
WARNING
This presentation assumes you have critical APARs applied
More discussion later
WARNING
Page 5
Agenda
Basic design
Implicit Key Store Management
SSL Selection and Scope
Creating and Managing KeyStores
Signer Imports
Expiration Management
Key Sets, Key Groups, and LTPA Key Management
Challenges
Page 6
WAS V6.1 Key and SSL Management Highlights
SSL Configuration Management
Pluggable key and trust manager support
Support for certificate revocation checking using PKIX trust manager
Multiple SSL configuration selection types with precedence rules
KeyStore and Certificate Management
No longer using the dummy certificate!
Self-signed certificate created per profile (properties to define values).
Advanced IKeyMan-like certificate management capabilities built into Admin Console.
Removed JVM system property requirement for SOAP and URL connections using SSL
More detailed error messages for SSL handshake failures
KeyStore and Certificate Management (server-side)
Federation of Base AppServer exchanges signer with Deployment Manager
Common TrustStore used by default in Cell
Webserver plugin CMS (.kdb) key store automatically maintained in config repository.
Expiration monitoring with notification and auto-replacement of expiring self-signed certificates
Hierarchically scoped SSL and KeyStore configurations
KeyStore and Certificate Management (client-side)
Signer exchange prompt for easy exchange (browser-like, can disable prompt)
New SSL configuration properties file, ssl.client.props, supports multiple SSL configurations
Same SSL configuration support as server-side, including pluggable trust/key managers
LTPA key versioning support
Warning: Many of the new certificate management features apply only to newly created cells.
Cells and nodes that are migrated from previous versions will not benefit from many of these
new features.
Page 7
Key Stores
Key stores & trust stores contain
Signers there are usually (CMS files still have all the CAs) no default signers except
for compatibility signers (dummy signers for earlier versions of WAS)
Personal certificates certificate + corresponding private key
Personal certificate requests
By convention trust stores contain only signers while key stores contain only personal
certificates
Storage types are
JCEKS, JKS Java file formats
PKCS12 standard file format (.p12). Microsoft may call this PFX.
PKCS11 hardware device (in this case path is actually the location of the shared
library to load)
CMSKS older format used by IHS and plugin (.kdb)
WAS provides all the needed key stores by default
You can create your own if needed E.g., a key store just for LDAP or the Web
container
Page 8
Key Store
Page 9
Key Stores are First Class Configuration Objects
Key Stores and Trust Stores (even KDB files) are managed via the
WAS admin console and stored in the configuration repository. Some
example defaults:
CellDefaultTrustStore is in
${CONFIG_ROOT}/cells/keysbotzumCell01/trust.p12
A NodeDefaultKeyStore is in
${CONFIG_ROOT}/cells/keysbotzumCell01/nodes/keysbotzumNode02/key
.p12
Note: NodeDefaultTrustStore is not used by default (Nodes all use the
CellDefaultTrustStore)
A Web Servers KDB file is in
${CONFIG_ROOT}/config/cells/keysbotzumCell01/nodes/webserverJan30/
servers/secondserverJan30/plugin-key.kdb
CellLTPAKeys is in ${CONFIG_ROOT}/cells/keysbotzumCell01/ltpa.jceks
Used to encrypt/decrypt LTPA tokens
Changes are automatically replicated throughout the cell
Many SSL components can reload information dynamically
Page 10
SSL Configurations
SSL Configurations contain/specify
Key store private key info
Certificate aliases which private key/cert pair should be used from this store
Trust store trusted signers
Trust manager manages validating certificates from peer
Key manager manages getting keys as needed
Quality of Protection SSL strength, etc.
IBM provides two trust managers
IbmX509 (the default) checks certificate signature and expiration. Optionally can
require hostname verification for certificates on URL connections.
IbmPKIX same plus more
CRL distribution point processing contact CA for revocation based on info in
certificate
Optionally supports Online Certificate Status Protocol (OCSP)
You can write your own trust manager if you want
You could implement your own arbitrary complex validation rules
IBM provides one key manager
IbmX509
You can write your own key manager if you want
Page 11
Node SSL Configuration
Page 12
SSL Quality of Protection
Page 13
Default SSL Configurations
WAS creates a number of default SSL configurations
Each node has (by default) a node specific SSL configuration:
NodeDefaultSSLSettings
The cell has a cell default SSL configuration: CellDefaultSSLSettings
You can create your own SSL configurations if needed
Perhaps youll dedicate an SSL configuration for the LDAP server, or a
Web Container, a just a particular application server
Page 14
Agenda
Basic design
Signer Imports
Challenges
Page 15
Key Stores and Topology
Each managed node has its own key store and trust store with its own self-
signed personal certificate
The initial node and Dmgr share one personal certificate if configured together
A single cell level trust store (but not key store) is replicated to all nodes
Contains all trusted signers for cell
When nodes are federated into the cell, the cell level trust store is updated to add
the new nodes signer
Nodes (including the Dmgr) can then validate each others certificates
SSL interoperability among managed nodes is assured automatically
Unmanaged nodes (e.g., Web servers) are a bit trickier
Each Web server has its own KDB file and self-signed personal certificate
When created (as of 6.1.0.5) the signer is placed in the cell trust store
automatically
When a new managed node is added to the cell, all Web server KDB key stores are
automatically updated with the new nodes signer (as of 6.1.0.5)
You must then copy the updated KDB file to the web server machine
Note: your Web server is likely an unmanaged node because putting a managed node
in the DMZ is a very bad idea
Page 16
Automatic Key Store Management
We are now going to show you what happens when
A web server is added to the cell
The web server gets a self-signed certificate
The cell trust store is updated to trust the new web servers signer
A node is added to the cell
The cell trust store is updated to trust the new node
The web server trust store (KDB file) is updated to trust the new node
Page 17
Default CellDefaultTrustStore
Page 18
New Web Server - Key Store Personal Certificate
Note: the certificate says localhost because I configured the node
name to be localhost when I created it. I should have chosen my real
hostname.
Page 19
CellDefaultTrustStore After Adding Web Server
Page 20
Web Server Key Store Signers
Page 21
Add a New Managed Node Its Certificate
Notice that the CN value (the hostname) is wrong. This is a good example of why
hostname verification is off by default.
Page 22
CellDefaultTrustStore After Adding Node
Page 23
Web Server Key Store Signers After Adding Node
Page 24
Agenda
Basic design
Signer Imports
Challenges
Page 25
SSL Configurations
There are numerous SSL configurations in a cell
You can define as many as you like in addition to the default SSL
Configurations mentioned earlier
Configurations are scoped (this is new in V6.1) at the level of cell, node,
server, cluster, and node group
At the lowest level configurations can be scoped by port
Which configuration is used is based on several rules that we will
discuss in a moment
Page 26
Top Level SSL Configuration View
Page 27
SSL Configuration Node, Server, and Port
Page 28
Scoped Information
Specify whether to inherit or override
Page 29
Scope Related Items
You can click on the related items to
see them
Youll see items of that type defined
at that scope and higher
You can define items at that scope
Page 30
SSL Configurations by Scope
Click on SSL Configurations and youll see
Page 31
Key Stores by Scope
Click on Key Store and Certificates and youll see
Page 32
Dynamic Endpoints for SSL by Scope
Specify what SSL configuration should be used based upon the URL of the target
(outbound only)
Matches based on comma separated tuple: protocol, host, port
E.g., https, ibm.com, 443
Wildcards can be used for entire element. E.g., https, ibm.com, *
Page 33
SSL Configuration Selection
A cell has numerous SSL configurations and associated key stores
and trust stores when are they used?
SSL selection is based on one of several criteria (in precedence order)
thread-based selection -- Properties have been specified on the thread
programmatically by an application that is directly using the IBM
JSSEHelper API. Later implicit usage of JSSE (e.g., opening an HTTPS
connection with a URL provider) implicitly uses the thread specific settings
set by the JSSEHelper API (not shown here).
dynamic outbound selection -- A dynamic outbound configuration has
been found that matches this particular outbound connection information.
This applies for both application code and WebSphere Application Servers
own usage.
direct selection -- A WebSphere Application Server SSL configuration has
been directly specified for this end point. This can be done administratively
for WebSphere Application Server SSL usage. Application programs can
also reference SSL configuration directly using the JSSEHelper API.
scoped selection -- None of the previous apply, so the topology-based
SSL configuration selection method applies.
Page 34
Agenda
Basic design
Signer Imports
Challenges
Page 35
Create/Manage Certs
All function is now in console no more ikeyman!
You can update the key store (which might be a trust store)
Create self signed personal certificates, Add signers
Create personal certificate requests (for a CA) and Receive them later
Import personal certificates (including private key) from a key file
Delete signers, personal certificates
Retrieve signers from a port
Replace existing personal certificates
You can export stuff from the store
Extract signing certificates
Extract personal certificates (just the signer)
Export personal certificates (signer and private key)
You can also see information about the certificates
Expiration, fingerprint, subject, issuer, serial number, etc
Page 36
Signer Management
Page 37
Personal Certificate Management
Page 38
Retrieve (Certificate) from Port
Sometimes it is a pain to get the signer cert from the other guy
Now, you can just import the cert directly from the server (making SSL work)
The SSL protocol automatically returns the endpoints certificate as part of the
handshake, so just import it
Downside is you are
importing the certificate,
not the signing certificate
If the same CA issues a
new certificate, well need
to import again!
Best to use this only for
self-signed certificates
Page 39
Agenda
Basic design
Signer Imports
Challenges
Page 40
Client SSL and Signers
SSL configuration information for clients is stored (by default) in
<profile>/etc/ssl.clients.props
The default trust store and key store are in <profile>/etc/trust.p12 (or
key.p12)
The trusted signers are of course in trust.p12
With all of the unique certificates in a WAS cell, many SSL clients
(including admin tools) cant connect to the WAS servers because they
dont trust the signers
This also impacts cross cell connectivity
The client runtimes support automatic certificate imports
Runtime will prompt end user do you want to trust this signer?
User is presented with certificate fingerprint which they should
verify!
If they say yes, the trust store will be permanently updated (just like
SSH)
Can be disabled to improve security
Can also bulk import signers into trust store using retreiveSigners.bat
Page 41
Client SSL Signer Auto Import
$ ./wsadmin.bat
*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host localhost is not found in trust store
C:/IBM/WebSphere/AppServer/profiles/AppSrv02/etc/trust.p12.
Here is the signer information (verify the digest value matches what is displayed
at the server):
Subject DN: CN=keysbotzum, O=IBM, C=US
Issuer DN: CN=keysbotzum, O=IBM, C=US
Serial number: 1151337276
Expires: Tue Jun 26 11:54:36 EDT 2007
SHA-1 Digest: 53:43:75:86:A8:C3:55:15:98:35:54:E7:49:B7:15:AF:16:A9:53:6F
MD5 Digest: 29:36:B1:9C:22:5A:36:AD:78:B3:7E:FD:D3:B1:B4:19
Add signer to the trust store now? (y/n)y
WASX7209I: Connected to process "dmgr" on node keysbotzumCellManager01 using SOAP
connector; The type of process is: DeploymentManager
WASX7029I: For help, enter: "$Help help"
$ ./wsadmin.bat
WASX7209I: Connected to process "dmgr" on node keysbotzumCellManager01 using SOAP
connector; The type of process is: DeploymentManager
WASX7029I: For help, enter: "$Help help"
wsadmin>
Page 42
Bulk Import of Signers For Clients
$ ./retrieveSigners.bat CellDefaultTrustStore ClientDefaultTrustStore -host
localhost
*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host localhost is not found in trust store
C:/IBM/WebSphere/AppServer/profiles/AppSrv02/etc/trust.p12.
Here is the signer information (verify the digest value matches what is displayed
at the server):
Subject DN: CN=keysbotzum, O=IBM, C=US
Issuer DN: CN=keysbotzum, O=IBM, C=US
Serial number: 1151337276
Expires: Tue Jun 26 11:54:36 EDT 2007
SHA-1 Digest: 53:43:75:86:A8:C3:55:15:98:35:54:E7:49:B7:15:AF:16:A9:53:6F
MD5 Digest: 29:36:B1:9C:22:5A:36:AD:78:B3:7E:FD:D3:B1:B4:19
Add signer to the trust store now? (y/n) y
CWPKI0308I: Adding signer alias "dummyclientsigner" to local keystore
"ClientDefaultTrustStore" with the following SHA digest:
0B:3F:C9:E0:70:54:58:F7:FD:81:80:70:83:A6:D0:92:38:7A:54:CD
CWPKI0308I: Adding signer alias "dummyserversigner" to local keystore
FB:38:FE:E6:CF:89:BA:01:67:8F:C2:30:74:84:E2:40:2C:B4:B5:65
CWPKI0308I: Adding signer alias "default_3" to local keystore
18:42:AB:E5:D6:59:4F:FF:BE:C0:A2:E7:FB:E6:41:F1:AB:4B:58:F8
D6:34:A0:76:91:C3:8D:14:52:4E:76:06:82:B2:77:B9:B4:57:8A:17
68:87:9A:41:E5:25:42:0D:CB:C4:51:A5:54:C3:C7:B9:08:58:57:09
Page 43
Agenda
Basic design
Signer Imports
Challenges
Page 44
Expiration
Certificates expire
Once they have expired, SSL communication that depends on them will fail
If Im doing the authentication and my personal certificate has expired, I
cant authenticate
If Im verifying the other partys certificate and the signer has expired, I
cant authenticate them
The default certificates expire after 15 years IF you applied APAR
PK34093 BEFORE configuring the profile
Self signed certificates make this worse management issues
If a CA issued certificate expires, you issue a new personal certificate the
signers are still good
If a self-signed certificate expires, the personal certificate and signer are no
good
Of course when the CAs own certificate expires, very bad things happen,
but this is rare
Page 45
You can see when certificates will expire by looking at all key stores using the admin console
Remember, key stores are scoped, so you have to look at every scope!!!
WAS automatically (by default) scans all key stores every two 2 weeks looking for certificates
that will soon expire
A serious event will be issued for any that will expire in the next
com.ibm.ws.security.expirationMonitorNotificationPeriod (security property) + expiration notification
(in GUI) days (90 + 15 by default)
An email can be optionally sent as well
Any self-signed certificates that will expire in the next expiration notification days will be replaced
New certificates generated
Changes replicated to all managed nodes in the cell
Warning: if you have disabled automatic node synchronization, an outage will result
SSL configurations reloaded dynamically although some transient problems should be
expected
Its safer to manually replace the self-signed certificates before WAS replaces them
Must manually update systems that arent part of managed WAS cell e.g., unmanaged web server
plugins!!!!!
If WAS does the replacement for you, your unmanaged web servers will stop working!!!
Updates might also break communication with other servers MQ, other cells, CICS, etc.
Can control when transient problems will occur
Consider disabling automatic certificate replacement just make sure you pay attention to those
serious events!!
Page 46
Expiration Management Configuration
When to replace certificates
This is after APAR
PK34093
The GUI descriptive
text was not updated,
so it is now misleading
Page 47
Agenda
Basic design
Signer Imports
Challenges
Page 48
Key Sets and Key Set Groups
Key stores of course contain cryptographic keys
Normally a key store contains one copy of a key for a particular purpose
Key sets are an abstraction that supports multiple versions of a
particular key in the same key store
The key store can contain multiple key versions (identified by a Prefix_N
where N is the version number)
Key sets also include function for generating new versions of these keys
(using a user defined key generator)
The key set limits the number of key versions stored
There are programming APIs for accessing key sets
Get a particular version, the latest version, all versions, etc
Key Set Groups
Manage collections of key sets as a single unit
Intent is automatically update multiple related keys (in multiple key sets) in
a single operation
Including time based automatic updates
Page 49
Key Set Configuration
Prefix used for keys
Key generator class
Number of versions to keep
Use this key store
Page 50
Key Set Group Configuration
Key sets managed
Page 51
Manual Key Set Group Key Generation
Page 52
LTPA Key Automatic Generation
LTPA encryption keys leverage key sets and key set groups
By default the LTPA keys are regenerated every 12 weeks
Two versions are kept so existing LTPA tokens can still be decrypted
Slight performance cost since both decryption keys must be checked
(newest first)
No longer have to worry about danger of using same encryption key
forever!!
Potentially breaks cross cell SSO
Have to use same LTPA keys, so auto generation will cause cells to go out
of sync
Instead, disable automatic generation in both cells and then copy the keys
from one to the other
Every two months manually regenerate the keys in one cell and then copy
again
Page 53
Agenda
Basic design
Signer Imports
Challenges
Page 54
Of Course, Everything isnt Perfect ...
Automatic Certificate Generation
Doesnt update components that arent managed parts of the WAS cell (e.g., non-
WAS systems in your environment)
Without careful planning outages will result!
If you dont apply the patches (next slide) outages will result without warning!!
NEVER disable auto node synchronization if automatic certificate generation is
enabled
WAS would replace the certs and the dmgr and then the rest of the cell would
stop working because they wont see the new certs!
Recommendation: disable automatic certificate generation in production systems
make sure warnings are configured properly
Scoping
All views of key stores, SSL configurations, etc, are scoped which seems good
Unfortunately there is no way to see all of the objects of a particular type in the admin
console
You have to search scope by scope
Should be possible to do something reasonable with scripting
Cross cell SSO configuration is now more complicated
See earlier slides for information on importing SSL certificates and disabling LTPA
auto generation
When nodes are removed, their certificates often arent
If you make a lot of topology changes, your trust files might get big keep an eye on
this
Page 55
WARNING
This presentation assumed you applied three critical APARs
APAR PK48659 prevents corruption of cert configurations during automatic updates.
Should be part of 6.1.0.15. Note that APAR PK36869 is not a proper fix as it requires
creating a new profile from scratch.
APAR PK29246 part of WAS V6.1.0.5
Ensures that web server key files are updated when new nodes are added
Ensures that cell trust files are updated when new web servers are added
APAR PK34093 targeted for WAS V6.1.0.7
Changes default certificate lifetime to 15 years
Fixed serious problem with certificate expiration and warning notifications to
provide appropriate warnings before certificates are automatically replaced
Without this APAR applied, changes will occur without warning, which will likely
result in a production outage.
What if I dont apply those patches???
Read on
WARNING
Page 56
WAS V6.1 SSL
Issues in 6.1.0.0
Certs expire 12 months
after profile creation (not
true on z/os)
When certs are
regenerated, unmanaged
systems (like the web
server plugin) may lose
connectivity
If a node is added to the
cell, existing web servers
will not be able to talk to
any of the app servers on
that node
If client cert authn is
enabled on the web
container then web servers
will be unable to talk to the
web container because
web server cert isnt in trust
store
Page 57
What to do
Install new environments by installing WAS, applying at least patch level 6.1.0.11
(other SSL related bugs fixed), and THEN creating initial configuration
DO NOT create initial configuration using installer (this applies to Process Server as
well)
For existing cells built before patches where recreating the cell is not an option
Apply latest patch level
This will fix bugs except that cert lifetime will remain 1 year by default
Manually replace certificates with longer lifetimes
For any environment
WAS will warn you before replacing expiring certs make sure you look for those
messages!!!
Disable Automatically replace expiring self-signed certificates as it is buggy
Then youll get warnings about 90 days before they expire. Manually act on those
warnings.
If adding nodes you may still encounter web server KDB file bugs
The existing KDB file isnt always updated
Create a new web server node that will have all the signers in it and copy that
Page 58
Watch For Warning Messages
Long before WAS tries to change the certificates, it will warn you. Watch for those messages and
act!!
[10/1/07 10:22:35:359 EDT] 0000000a WSNotifier I CWPKI0037I: Expiration monitor reports the following information:
**** Subject: Expiration Monitor ****
*** CERTIFICATES WITHIN THE 90 DAY PRE-NOTIFICATION THRESHOLD (MAY BE REPLACED WITHIN 90 DAYS) ***
The expiration monitor has recently run and discovered that the certificates listed below will be replaced within the next 90 days based upon the
configured policy to automatically replace expiring self-signed certificates 60 days prior to expiration. This notification is warning you that
problems may arise when this automatic replace occurs.
In some cases, automatic updates to the self-signed certificates can cause outages for WebServer plug-ins operating on unmanaged nodes. In such
a situation, the plugin will be unable to contact the application servers over HTTPS because it will be using signers for certificates that have been
replaced by the automatic replacement process.
To prevent what may be and serious outage you should act before the scheduled replacement date and replace the expiring certificates and update
the plugin kdb to use the new signers.
Self-signed personal certificate "mycert" in the "CellDefaultTrustStore" key store is due to be replaced on November 29, 2007.
Signer certificate "default_5" in the "CellDefaultTrustStore" key store is due to expire on January 29, 2008.
Signer certificate "default_4" in the "CMSKeyStore" key store is due to expire on January 29, 2008.
Signer certificate "default_3" in the "CMSKeyStore" key store is due to expire on January 29, 2008.
*** CERTIFICATES THAT ARE BEYOND THE EXPIRATION THRESHOLD ***
Checking for expired certificate and certificates about to expire
CWPKI0680I: Personal certificate alias "default" in KeyStore " CellDefaultKeyStore " expired on June 26, 2007.
CWPKI0679I: Signer certificate alias "default_2" in KeyStore "CellDefaultTrustStore" expired on August 13, 2007.
CWPKI0679I: Signer certificate alias "default_1" in KeyStore "CellDefaultTrustStore" expired on July 20, 2007.
CWPKI0680I: Personal certificate alias "default" in KeyStore " CMSKeyStore " expired on July 13, 2007.
*** CERTIFICATES THAT ARE BEYOND THE EXPIRATION THRESHOLD AND HAVE BEEN REPLACED ***
CWPKI0645I: Personal certificate alias " default " in KeyStore "CellDefaultKeyStore" was REPLACED.
CWPKI0645I: Personal certificate alias " default " in KeyStore "CMSKeyStore" was REPLACED.
Waited too long. WAS
did it for you (hopefully
correctly).
Page 59
Futures
IBM's plans, directions and intent are subject to change or withdrawal
We are looking at including a mini certificate authority inside WAS
Will make updating personal certificates used by WAS components simpler
Page 60
References
WAS V6.1 SSL - http://www-
128.ibm.com/developerworks/websphere/techjournal/0612_birk/0612_
birk.html?ca=drs-
Other security papers and presentations
Internal - http://pokgsa.ibm.com/~keys/
Public http://www.keysbotzum.com
WSDD author page http://www-
128.ibm.com/developerworks/websphere/community/authors/keysbotzum
_bio.html
Peter Birks presentation and feature documentation on these
enhancements
Page 61
Appendix
Page 62
Need an Example
Examples of SSL selection??
For *EVERY* SSL selection, all possible rules apply, they are
selected based on precedence. If a piece of code opens a JSSE
connection, WAS will first check for a thread scoped selection,
then a dynamic selection, then direct selection (if specified), and
then a scoped selection. Direct selection would not occur if the
code doesn't specify an SSL configuration. And, yes, dynamic &
thread do override direct. At least, that's how it's supposed to
work. I haven't had time to test this in code.
Warning: Although it may be confusing at times, this approach gives you
tremendous flexibility. One item to be very conscious of is the fact that
dynamic end point specification overrides both direct end point
specification and scoped selection. This gives the administrator
tremendous ability to change how SSL is used, but can lead to
surprising results.
Page 63
Scripting
For WAS V6.0 and earlier with ikeyman
http://www.ibm.com/developerworks/java/jdk/security/142/secguides/keytoo
lDocs/KeyToolUserGuide-142.html
WAS V6.1 (or later) has certificate management built into the WAS
administration runtime so wsadmin could be used, though I suppose
you could also use the keytool there as well
Page 64
Ikeyman Hints
IBMs legacy ool for managing certificates and private keys
Can create/manage key databases in numerous formats
CMS/KDB older format used by plugin and IHS
JKS standard Java format used by WAS
JCEKS Another Java key store
PKCS12 standard format
Extension is usually P12 or PFX (Microsoft)
To import a signer
Go to signing certificates and click Add
Page 65
Using Ikeyman Personal Certificates
Self signed
Go to personal certificates and then click new self-signed
Getting a signed cert from a CA
On menu, click Create->New Certificate Request
Send ARM file to CA
When you get back the response, go to Personal Certificates and click Receive
Importing a complete certificate with private key
Go to personal certificates
Get the key file (e.g., PKCS12)
Click Import
Need the signers!
Ikeyman will not import a personal certificate if the signer is not either in the key db you
are importing from or already in the ikeyman key db.
It will issue some meaningless error message in this case
This is the most common ikeyman problem
Page 66
Ikeyman Trouble Shooting
Starting ikeyman with tracing
<gsk install root>/bin/gsk7ikm -Dkeyman.debug=true -
Dkeyman.jnitracing=YES
If an import fails, try opening the key file directly
E.g., If someone gives you a PKCS12 file (p12) open that in ikeyman
Look at the cert to see the issuer DN. Then look at the kdb file and see if any
of the certs in the kdb have a subject DN that matches the issuer DN from the
import DB. If you don't find one then ikeyman will not import the cert.
You might also try opening in firefox or IE as they are more forgiving
Page 67
How do I Create My Own Certificates?
IBMs JDK includes documented APIs for this
http://www-
128.ibm.com/developerworks/java/jdk/security/50/secguides/keycertmgmtD
ocs/api/
Better still with WAS V6.1, you can ask WAS to create and manage
certificates for you
Using scripts to create certs:
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/topic/com.ibm.websp
here.nd.doc/info/ae/ae/txml_managecert.html
Page 68
Certificate Key Length Issues
Ikeyman and WAS itself cant import or generate keys with long key
lengths doesnt (by default) support that key strength
http://www-
1.ibm.com/support/docview.wss?uid=swg21201170&loc=en_US&cs=utf-
8&lang=en
You can update the security policy to make it work as indicated
Once this is done the WAS runtime will be able to import and understand
longer key lengths
However it still wont be able to generate longer lengths
This should be fixed in WAS V7 via defect #421110
Page 69
Advanced SSL Debugging
Prior to WAS V6.1 low level SSL errors can be very hard to debug
because the error information is not clear
You can get better error information by getting a special version of the
JDK SSL JAR files enabled for debugging
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21162961
Page 70
Legal
Copyright IBM Corporation 2007, 2008. All rights reserved.
IBM, the IBM logo, the e-business logo and other IBM products and services are
trademarks or registered trademarks of the International Business Machines
Corporation, in the United States, other countries or both. References in this
publication to IBM products, programs, or services do not imply that they will be
available in all countries in which IBM operates.
Product release dates and/or capabilities referenced in this publication may change
at any time at IBMs sole discretion based on market opportunities or other
factors, and are not intended to be a commitment to future product or feature
availability in any way.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in
the United States, other countries or both.
Microsoft, Windows, Windows NT and the Windows logo are trademarks of
Microsoft Corporation in the United States, other countries or both.
All other trademarks, company, products or service names may be trademarks,
registered trademarks or service marks of others.
More of What Hackers Don
t Want You to
t Want You to
Know
Know
Session Number: H05
Session Number: H05
Kevin Tobin, Senior IT Specialist Kevin Tobin, Senior IT Specialist
(kevintob@au1.ibm.com) (kevintob@au1.ibm.com)
Objectives
Objectives
By the end of this session,
you will be able to:
Think critically about new technologies and
the security risks they introduce
Identify some of the latest security risks
Anticipate future security vulnerabilities
Be better informed on appropriate security
countermeasures
t Want
t Want
You to Know
You to Know
Jeff Crume,
Jeff Crume, CISSP CISSP
Executive IT Security Architect Executive IT Security Architect
http://extranet1.lotus.com/crume http://extranet1.lotus.com/crume
ISS
ISS
-
-
210
210
-
-
G
G
Agenda
Agenda
Intro
Vulnerabilities in:
Phishing
Passwords
Biometrics
RFID
Bluetooth
Web 2.0
Open Source
Conclusion
More technology
More technology
more security
more security
Myth: My system is secure
because it has ______
technology.
Maginot Line
the Great Wall of France
If you entrench yourself behind strong
fortifications, you compel the enemy to
seek a solution elsewhere.
-- Carl von Clausewitz
People are the weakest link
phishing schemes
opening email attachments
Bagle / Netsky encrypted zip files
Phishing defenses aren

Phishing defenses aren
t working
t working
Study by Harvard and MIT researchers
found:*
100% ignored the absence of HTTPS
97% ignored the absence of site-authn
image
53% ignored invalid cert msg from browser
Whats really broken here?
The technology or the user?
Man-in-the-middle phishing
compounds the problem
gives the illusion that everything is working
* Study: Users ignore bank security features, ComputerWorld, Feb 5, 2007
#1 call to most help desks
2 out of 3 Web users use < 5 passwords for
all access to electronic information*
15% use a single password
71% of people at Victoria Station (London)
station gave out passwords for an Easter
egg**
PC sunflower
Keystroke loggers
Hidden passwords
Passwords have reached the end of their
useful life. Today, they only work for low-
security applications. Bruce Schneier***
Passwords don
Passwords don
t work
t work
*Security fears daunt online shoppers, Dawn
Kawamoto, CNET News.com, 14 Feb 2005.
**Password protection no match for Easter egg
lovers, searchSecurity.com, 20 April 2004
*** The Curse of the Secret Question,
ComputerWorld, 9 Feb 2005
Biometrics are broken

Biometrics are broken
* Ponemon Institute as reported by Information Week, Feb 6, 2007
Myth: Biometrics are far more
secure.
69% of US citizens want banks,
credit cards, health care and govt to
use biometrics*
Be careful what you ask for
false positives, false negatives
man-in-the-middle attacks
replay attacks
expensive technology
Gummy Fingers
Gummy Fingers
http://cryptome.org/gummy.htm
Another broken biometric
Another broken biometric
Unintended consequences/risks of biometrics
Unintended consequences/risks of biometrics
How do you know if a remote
reader is trustworthy?
What happens if your fingerprint is
compromised?
If the thing being protected is
extremely valuable, what happens
to an uncooperative biometric
owner?
Myth: Privacy is threatened.
Biometrics work best when:
Used for convenience rather than
security
Combined with other authentication
factors
RFID tags wireless transmit data
Wireless eavesdropping
Wireless impersonation
at a safe distance
1997 ExxonMobile SpeedPass
RFID key fob to pay for gas at the pump
3 grad students impersonated the system
and filled up for free
UMass professor skimmed credit card
numbers, exp. dates, names
Never physically possessed the card
Reader built for $150 using off-the-shelf
components
RFID means I can see you but you can

RFID means I can see you but you can
t see me
t see me
Bluetooth is a hacker
Bluetooth is a hacker
s best friend
s best friend
Pervasive technology
Mobile phones, PDAs, laptops, etc.
All of which hold more and more
sensitive info
Many (most?) bluetooth devices
remain in discoverable mode
Colorful names for various
attacks:
Bluejacking, Bluesnarfing
Bluebugging, Blueprinting
Bluesniper Rifle
Range > 1 mile
Web 2.0 called participatory Web
Wikis, Blogs, Mash-ups, Tagging, Social Networking,
etc.
Heavy reliance upon client-side scripting
AJAX, JavaScript, etc.
Long history of overlooked security vulnerabilities
Code is automatically downloaded and run on your
system without your knowledge or permission by merely
visiting a Web site
Examples:
MySpace password stealing worm
Samy: MySpace worm which automatically added the
author to visitors profiles
giving the illusion that he was the most popular person on
the site
Web 2.0 = Hack 2.0

Web 2.0 = Hack 2.0
Other Web 2.0 risks
Other Web 2.0 risks
Information leaks
Folksonomies and tagging tell others a lot about you and
the work you are doing
Confidential information may be inferred from examining
your activities
Even more info may be gleaned if taken for an entire dept
in aggregate
Remember: not all the bad guys are out there
Attacks on information
Wikiality Colbert attack
A reality where, if enough of the right people agree,
it becomes the truth.
If trusted sources are compromised then
who can you trust?
Conclusion: Web 2.0 represents a new minefield of
security vulnerabilities
Open source means open season

Open source means open season
Myth: Open source software is far more
secure than proprietary.
Counter-myth: security by obscurity
Open source software may be more secure
has the potential to be more secure
Linux vs. Windows
Window vs. z/OS
Myth: A million eyes guarantees greater
security.
assumes massive numbers of code reviewers
who are sufficiently skilled and motivated
who know what they are looking for
works well for crypto algorithms
may not scale for OSs
NO system is secure
NO system is secure
No operational system is
completely secure
Despite what
modern-day
Titanic builders
tell you
Ex: Oracle's unbreakable security
Risk Mgmt vs. Risk Avoidance
Summary
Summary
New technologies and the security risks they
introduce
Phishing
Passwords
Biometrics
RFID
Bluetooth
Web 2.0
Open Source
Latest security risks
Future security vulnerabilities
Appropriate security countermeasures
Concluding Thought
Concluding Thought
"It ain't so much the things you don't know that get you
in trouble. It's the things you know that just ain't so."
-- Artimus Ward, 1834-1867
Questions
Questions
Please complete session evaluation
form
Thank you!!
WebSphere Application Server V6 Deployment on Z/OS
&
Best Practices
Gaurav Bhagat
gauravbhagat@in.ibm.com
Objectives
Understanding components of WAS on z/OS.
Different type of configurations
Before you start the deployment.
The deployment process.
SMP/E Install
Configuration
Planning
Prerequisite for Configuration
Running Installation Clist
Running Configuration Jobs
Starting the Server
Best Practices during Deployment.
Components of WAS
Server is the basic unit of the
configuration, the server is where
applications execute.
Nodes is the collection of servers
on a given MVS image.
Node Agents is a special purpose
server which acts on behalf of the
administrative application to make
changes in the node.
Deployment Manager is a special
purpose server which runs
administrative application.
Cell is a collection of nodes
managed by the deployment
manager.
Daemon they provide whats
known as the "Location Service" for
the cell
Network Deployment Cell
Stand Alone Configuration
Standalone As Building Block of ND
The Basic Component - Appl. Server
Where Configuration is kept ?
Before You Start
Before you Start .
Review the Following
Target System Requirements( e.g. Z/OS 1.7 for WAS 6.1).
Review the current Java Version (JDK 1.4 for WAS 6.1)
Unix System Services(OMVS) is active.
TCP/IP Configured with DNS name.
Enough DASD Storage is available.
RRS is active.
Enough real memory and page datasets (to avoid auxiliary storage shortage).
Delivery method you want to order (CBPDO, Server Pac or System Pac).
Plan your configuration.
The Deployment Process
SMP/E Install
Server Pac, System Pac or CPBDO
Follow the Program Directory shipped with the code.
or
Follow the Installing your Order Guide shipped with Server Pac delivery Method.
Plan for the following
HLQ of TLIBs & DLIBs.
The USS Mount Point
The zones where the FMID (e.g. H28W610) will be installed.
Volumes for SMPTLIB, TLIBs, DLIBs, HFS Datasets.
Check the current maintenance level available
WAS 6.1.0.17 (as on 1
st
July 2008)
WAS 6.0.2.27 (as on 1
st
July 2008)
The Deployment Process.
Planning for configurations
Final Configuration (Stand Alone Server or Network Deployment)
Cell Name
Decide on two characters cell identifier (for e.g. P1 for Production 1)
LPAR Qualifier.
The LPAR Qualifier will be required when cell spans on multiple Z/OS images.
SMP/E Install Directory Path (a part of System Locations)
To use path of the installed directory (/usr/lpp/zWebSphere/V6R1) or to use symbolic Links
(/wasv601config/mycell/mydmn_smpe)
Note : More to be discussed during Best Practices.
TCP/IP Configurations
TCP/IP Host Name.
Ports.
Stand Alone Server 18 Ports ( approx.)
Network Deployment Cell (with one Server) - 34 Ports ( approx.)
The Deployment Process.
RACF Configurations
Common User & Group Configuration
WAS Configuration Group
WAS Configuration Owner
WAS Configuration Servant Group
WAS Local User Group
Home Directory - The USS home directory of the users.
Components (DMGR, Server..) User & Group Configuration
Started Task User id Configurations
UIDs of users & GIDs of groups (Not applicable if using AUTO GID/UID option).
OMVS Mount points
Where Configurations will be located
Mount Points are required for each components of cell (DMGR, Node Agent,
Server).
Name for the Components (Short and Long)
Short and Long Name Concept
Short Names and Long Names need to be configured for the following Components
Deployment Manager (long name dmgr , cannot be changed)
Node Agent (long name nodeagent , cannot be changed)
Application Servers
(Sample Short & Long Names for Standalone Server configuration)
mynodea MYNODEA Node Name
mysr01a MYSR01A Server Name
mybasea MYBASEA Cell Name
Long Name Short Name Component Name
Job Names
Each Component (DMGR, Node Agent,Server) needs a job name for its
Controller Region (limited to 7 Characters)
Servant region
Daemon
Equal to Short Name for Controller Region
Servant Region Job Name is equal to Short Name + S
(e.g. MYSR01A is a CR job name & MYSR01AS is a SR job name)
Procedure Names
Need to be specified for both CR & SR.
Called for starting the controller region.
Security
Security Domain Identifier
Its purpose is to provide granularity on a cell-by-cell basis.
Is an Option.
Easy to enable in V6.1
Profiles are defined in EJBROLE RACF Class.
SECDMN.Administrator
SECDMN.Operator
SECDMN.Configurator
SECDMN.Monitor
Administrator Full Authority
Operator Monitor + ability to start & stop servers
Configurator Monitor + ability to change WebSphere Configurations
Monitor view the configuration
Target Datasets
The following Datasets are required for each component to contain customized jobs , when we do a generate jobs during ISPF
Customization panels.
HLQ.CNTL Contains JCL & Instruction Member
HLQ.DATA Contains variable Length Scripts.
Another Dataset should be created to save variables defined during
ISPF Customization panel.
HLQ.SDFG or HLQ.SAVECFG Contains the Variables defined during ISPF customization panel.
Plan for the HLQ of datasets & the volume which will contain these datasets
Note : Some recommendations will be given during best practices.
Configuration Planning Spread Sheet
Download WebSphere for z/OS Version 6 - Configuration Planning Spreadsheet
from TechDocs Library.
By capturing a few key variables, this spreadsheet does all the rest -- it calculates and allocates TCP ports, UID and
GID values; it constructs names based on root character strings ... in short, it makes things a whole lot easier.
Following is the link for the planning spreadsheet
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1331
It has the spreadsheet for following versions
Pre V6.0.2
V6.0.2
V6.1
Running the installation CLIST
TSO EXEC SYS1.WAS610.SBBOCLIB(BBOWSTRT)
* SYS1.WAS610 is the HLQ of TLIBS created from SMP/E Install
Enter the Planned Variables (like Server Configurations, Ports, Hostname .)
Generate Customization jobs.
Save Variables.
Flow Common to All Customizations
ISPF Customization DialogsDMGR
Creating a "Standalone Server"
Submit the Customized Jobs
Follow the instructions (generated by the Clist) & submit the customized jobs.
Some Contents of the **.CNTL Data Set for DMGR Configuration
You Start Controller, Rest Done For You
Best Practices During Deployment
Naming Conventions (Short Names)
RACF Configurations
Common User & Group Configuration
WAS Configuration Group - Cell Identifier + CFG (e.g. MYCFG)
WAS Configuration Owner - Cell Identifier + ADMIN (e.g. MYADMIN)
WAS Configuration Servant Group - Cell Identifier + SRG (e.g. MYSRG)
WAS Local User Group Information Cell Identifier + GUESTG
(e.g. MYGUESTG)
Components (DMGR, Server..) User & Group Configuration
Used in Started Class Profiles as STDATA parameter
User id = Cell Prefix + LPAR Qualifier + CR/SR + U
e.g.
Controller Region Userid - MYACRU
Servant Region Userid - MYASRU
Home Directory - The USS home directory of the users.
Create separate home directory from other users & groups for WAS users & groups (e.g. /wasv601config/users/home).
UIDs of users & GIDs of groups.
All Userid & Groups should have a OMVS segment
which requires UID or GID number (Root has a UID 0)
- Block a range of number UIDs & range of numbers for GIDs
for e.g. 2400 2450 for UIDs
2500 2550 for GIDs
Contd ..
USS Directory Structure
USS Path needs to be configured, it is the path where configuration will be located.
An HFS or ZFS Dataset needs to be mounted at the USS Path defined.
Path name should have the Version + Cell Name + Component Name (if not using shared HFS/ZFS)
e.g. /wasv610config/mycell/mydmgr
Prefer to use separate HFS for each Node & DMGR (as shared HFS with nodes on
separate images does not give good performance).
Summary
Types of WAS Configuration
Network Deployment
Stand Alone
Basic Components of WAS
Deployment Manager
Nodes
Servers
Node Agent
Daemons
Steps involved in deployment of WAS.
SMP/E
Planning
Concept of Short & Long Names
Running Install Clist
Running Customized Jobs
Starting WebSphere Application Server
Best Practices during WAS Deployment.
Lab and Questions
Lets Configure our Standalone Server , refer to the Lab Instructions.
Please complete your session evaluation after the labs.
Ask any questions during the lab.
THANK YOU
WebSphere Process Server V6.1 for z/OS
Configuration & Troubleshooting
Gaurav Bhagat
Page 2
Configuration Topics
Pre-Reqs & Product Packaging on z/OS
Task Overview
Configuration scripts
DB2 Configuration
Security Setup
Configuration Scenarios
Standalone with Cloudscape
Standalone Using DB2
Network Deployment (with DB2)
Trouble-Shooting & Tools
Page 3
WPS/WESB V.6.1 for z/OS Software
Prerequisites
z/OS Version 1.7 +
Communication Server with TCP/IP
Unix Systems Services with HFS or zFS
SAF/RACF,
System SSL if using SSL
WLM in goal mode,
RRS, Logger, ARM, and many more ....
WAS for z/OS Version 6.1.0.15
DB2 for z/OS Version 8.1, 9.1 (V 7 with limitations)
DB2 Stored Procedure Builder (DSNTPSMP) reqd for WPS Relationships
DB2V8+ needed for Business Rules and Selectors
Should have a TEMP DB in the DB2 Subsystem.
WLM Application Environment should be setup (WLMENV5 by default).
See http://www.ibm.com/software/integration/wps/sysreqs
http://www.ibm.com/support/docview.wss?rs=2307&uid=swg27009841
Page 4
Packaging & Structure for WPS/ESB for z/OS
H28W610
HWPS610
Configured on top
hlq.SBBOxxxx
.
hlq.SBBOHFS
/usr/lpp/zWebSphere/V6R1
hlq.SBPZHFS
/shared/zWPS/V6R1 or
/shared/zWESB/V6R1
HESB610
WESB
WPS or
WAS
(WAS 6.1.0.15
required)
FMIDs
Page 5
WPS V6.1 Whats New?
WAS 6.1 Enhancements
New JVM (5.0) + 64-bit JVM option
z/OS security enabled by default
Sample jobs to allocate zFS or HFS for product code
WPS/WESB Configuration
Installation & Configuration faster
zWPSConfig.sh response files (dont use your old .rsp files)
Scripts: createDB.sh can generate and run .sql files for all DBs
(fixWPSvars.jacl not used for WPS 6.1)
Performance
Runtime scenarios improved (Event Sequencing, Larger Business objects)
Page 6
Configuration options
Stand-alone AppServer
Limited to a Single Node, Single Server
Useful for a first practice configuration, but not for production.
Two Database options:
1. Derby (Cloudscape)
2. DB2
Network Deployment (ND)
Multiple Servers, Nodes, LPARs, Clustering available
Recommended for Production (DB2 Only)
Implementation Steps:
1. Configure Deployment manager
2. Configure Stand-alone node OR empty node
3. Federate node into Deployment Manager cell
4. Post-configuration tasks needed
Page 7
Configuration task overview
1. Configure WebSphere Application Server
2. Run zSMPInstall.sh script on each Node
Install WebSphere Process Server or WebSphere Enterprise Service Bus
function into node profiles
3. Run zWPSConfig.sh script on each Node
Augment profiles with WPS function
Create resources (data sources, queues, SI Buses)
Install applications, profiles, scripts, DDL (sql), etc.
4. Complete post-configuration tasks, if needed
Page 8
1. zSMPInstall.sh script in
<wps_smpe_root>/zos.config/bin/
zSMPInstall.sh -smproot <wps_smpe_root> \
-runtime <app_server_root> -install
Installs WPS into your profile using many ant scripts.
Creates symLinks in <app_server_root> to <wps_smpe_root>
Updates the administration console for WPS/WESB
Runs applyPTF as well.
Prefer to use Symbolic link for wps_smpe_root.
First script provided to Install WPS
- Run against each node (AppServer, DMgr):
Page 9
WPS/WESB SMP/E directory configuration
hlq.SBPZHFS
wps_smpe_root= /shared/zWPS/V6R1
Page 10
<app_server_root> after install
New directories links within existing directories
hlq.SBPZHFS
wps_smpe_root=
/shared/zWPS/V6R1
Page 11
2. zWPSConfig.sh script in <app_server_root>/bin/
zWPSConfig.sh -augment -response <response_file>
Reads installation-specific parm variables from response file.
Augments profiles to use the WPS functions.
Creates resource definitions (data sources, databases, queues,)
Installs applications, profiles, scripts, DDL (sql), etc.
Second scripts provided toConfigure WPS
- Run against each node (AppServer, DMgr):
Page 12
Response Files Used by zWPSConfig.sh
Use Samples found in: /usr/lpp/zWPS/V6R1/zos.config/
standAloneProfile.rsp (Cloudscape only)
standAloneProfileDB2.rsp
DmgrDB2.rsp
ManagedDB2.rsp
Dont use your WPS/WESB V.6.0.x rsp files!
Leave templatePath set to default.wbiserver
Copy to a working directory, edit for your env.:
Common augment keyword, server name, etc.
WBI Properties SCA, AppScheduler Config
Database Configuration - Cloudscape or DB2 specs
BPC Properties (stand-alone only not for ND)
CEI (stand-alone only not for ND)
Run zWPSConfig.sh with Admins UserID
Messages:
INSTCONFSUCCESS: Profile augmentation succeeded. . . .
JMSUSER=f6admin
JMSPASS=xxxxxxx
CONFIGSERVER=dmgr
DBPRODUCT=DB2UDBOS390_V8_1
DBLOCATION=WSCDBP0
DBHOME=/shared/db2810/jcc
DBPROPERTIES=/u/hutch/wps61
DBJDBCCLASSPATH=/shared/db2810/jcc/classes
DBUSER=sysadm1
DBPASS=sysadm1
DBHOSTNAME=wsc2.washington.ibm.com
DBSERVERPORT=446
SQLID=F6CELL
SQLDB=F6CELLDB
SQLSTO=F6DBSTO
######### End Global Properties
augment
cellName=f6cell
nodeName=f6dmnode
profileName=default
profilePath=/wasv61config/f6cell/f6dmnode/DeploymentManager/profiles/defaul
templatePath=/wasv61config/f6cell/f6dmnode/DeploymentManager/profileTe
serverName=$CONFIGSERVER
dbCreateNew=false
dbDelayConfig=true
Page 13
Response file common properties
augment
cellName=f6basea
nodeName=f6nodea
profileName=default
profilePath=/wasv61/f6cell/f6nodea/AppServer/profiles
/default
templatePath=/wasv61/f6cell/f6nodea/AppServer/profile
Templates/default.wbiserver
serverName=f6sr01a
. . .
Leave templatePath set to dmgr.wbiserver
/WebSphere/V6R1/DeploymentManager/profileTemplates/dmgr.wbicore
/WebSphere/V6R1/DeploymentManager/profileTemplates/dmgr.bfm
/WebSphere/V6R1/DeploymentManager/profileTemplates/dmgr.wbiserver
/WebSphere/V6R1/DeploymentManager/profileTemplates/dmgr.*
Long names!!
Page 14
Response file database configuration parms
DB2 parameters
DBPRODUCT=DB2UDBOS390_V8_1
DBLOCATION=WSCDBP0
DBHOME=/shared/db2810/jcc
DBPROPERTIES=/u/hutch/wps61
DBJDBCCLASSPATH=/shared/db2810/jcc/classes
DBUSER=sysadm1
DBPASS=xxxxxxx
dbCreateNew=false
Determines whether databases will be created during augmentation
dbDelayConfig=true (default)
Determines whether databases are configured during augmentation
If set to false, the tailored SQL will be run during augmentation
Must have authority to run the SQL
Databases & Storage Groups must already exist!
DB2 defaults
Page 15
DB2: JDBC Data Sources & DBs used by WPS
WebSphere Process Server databases:
WPSCommon Relationship services, Mediation, Recovery, Business rules, Selector,
Scheduler (26 tables)
Includes 2 tables for Logging ESB mediation events (new in V6.1)
BPE - Business Process Container & Human Task Manager (221)
CEI - Common Event Infrastructure - EVENT & EVENTCAT (38)
SIBs (4) Messaging Engines (SCA, APP, BPC, CEI) (11x4=44 tables)
Determine your naming conventions
Talk to your DB2 Administrator
Scripts may be used to Generate sample .sql / .ddl:
zWPSConfig.sh
bpeconfig.jacl
config_event_database
sibDDLGenerator.sh
or use createDB.sh script to generate for all (Recommended!)
Page 16
Creating Databases & Tables Overview:
Three steps:
1. Plan your DB2 definitions (Talk to your DB2 administrator!)
One Database / Storage Group, or Multiple
Naming conventions for Databases, StoGroups, & Schema qualifiers
Database & Storage Group names Needed to run zWPSConfig.sh
2. Create the .sql (ascii) or .ddl (ebcdic) definition files
Use zWPSConfig.sh, along with other scripts, or
Use createDB.sh scripts (allows tables to be created during augmentation.)
(other components have specific scripts (WPS, BPC, CEI, SIB)
3. Run (execute) the .sql/.ddl files
First, create the Databases & StoGroups manually. . .
zWPSConfig.sh scripts will NOT create Database or Storage Groups.
Use DBUtility.sh script to run tailored SQL, or
Use createDB.sh script, or
Convert to EBCDIC, Copy to MVS datasets & Use batch jobs or SPUFI
Page 17
DB2 databases Sample naming
considerations
xxWPS xxCELL xxEVTSTO xxEVTDB Event Catalog
xxWPS xxCELL xxEVTSTO xxEVTDB CEI Event
xxWPS xxS1B xxSIBSTO xxSIBBDB SIB-BPD
xxWPS xxS1C xxSIBSTO xxSIBCDB SIB-CEI
xxWPS xxS1A xxSIBSTO xxSIBADB SIB-APP
xxWPS xxS1S xxSIBSTO xxSIBSDB SIB-SCA
xxWPS xxCELL xxBPESTO xxBPEDB BPE
xxWPS xxCELL xxWPSSTO xxWPSDB WPS
VCAT DSN-
hlq
Schema
owner (2)
Storage Group (1) Dbase Name (1) Database
where xx=cell identifier (for instance, W6CELL)
1. Another possibility is to use a single Database & StoGroup.
2. SIB databases have common tables names, so need unique schema
Page 18
DB2 Database creation & GRANTs
Work with your DB2 administrator!
Need information about database and storage group.
Scripts will NOT create database or storage groups.
Sample SQL for creation (using one Database/StoGroup):
DROP DATABASE xxCELL;
DROP STOGROUP xxDBSTO;
COMMIT;
CREATE STOGROUP xxDBSTO VOLUMES (*) VCAT xxWPS;
CREATE DATABASE xxCELL
STOGROUP xxDBSTO
BUFFERPOOL BP0
CCSID UNICODE
INDEXBP BP0;
COMMIT;
GRANT USE OF STOGROUP xxDBSTO TO xxCELL WITH GRANT OPTION;
GRANT DBADM ON DATABASE xxCELL TO xxCELL;
GRANT USE OF BUFFERPOOL BP0 TO xxCELL;
GRANT USE OF BUFFERPOOL BP1 TO xxCELL;
GRANT USE OF BUFFERPOOL BP8K0 TO xxCELL;
Page 19
createDB.sh - Generate sql files for DB2
Tables
New Script shipped in {wps_smpe_root}/zos.config/samples/
Documented in help (type createDB.sh ?) and Techdoc WP101209
Copy createDB.sh into your working directory.
Edit this copy to specify your Names, Locations, and desired actions:
Generate .sql files for DB2 Tables - WPS, BPC, CEI, and SIBs
Execute the .sql to Create the Tables.
.sql files created in {profile_root}/dbscripts/CommonDB/DB2zOSV8/<CELL_NAME>/
createDatabaseX.sql creates database, storage groups
bpc.sql for configuring BPC
ceidb.sql for configuring CEI
common.sql required
sibAPP.sql required, SCA Application Bus
sibBPC.sql for configuring BPC (Process Choreographer Bus)
sibCEI.sql for configuring CEI (CommonEventInfrastructure Bus)
sibSCA.sql required, SCA System Bus
Page 20
createDB.sh - Create sql files for DB2 Tables
Assumptions:
One database and Storage Group for all Tables
To create separate databases, run createDB separately for each one.
SI Bus Tables require unique Schema Names (or unique Databases)
createdDB.sh uses the vi editor, so requires a vi-capable session
telnet works fine, OMVS does not.
Various Options:
1. Let the createDB.sh script run them
2. Use the DBUtility.sh script (in the Servers bin directory)
3. Convert to EBCDIC, Copy to MVS files, and use SPUFI or Batch jobs.
Page 21
Security: EJBROLE Profiles for WPS
RACF Profiles for BPC, HTM & Explorer:
<domain>.BPEAPIUser
<domain>.BPESystemAdministrator
<domain>.BPESystemMonitor
<domain>.TaskApiUser
<domain>.TaskSystemAdministrator
<domain>.TaskSystemMonitor
<domain>.WebClientUser
<domain>.JMSAPIUser APPLDATA(xxADMIN)
<domain>.EscalationUser APPLDATA(xxADMIN)
Profiles for CEI:
<domain>.catalogAdministrator
<domain>.catalogReader
<domain>.eventAdministrator
<domain>.eventConsumer
<domain>.eventCreator
<domain>.eventUpdater
Instance-based roles for Human Task People Assingments:
instance creator, owner, originator, starter, etc.
See WPS InfoCenter
for details
Page 22
Create WebSphere Standalone AppServer, Create WebSphere Standalone AppServer, Start, Test & Stop it
1. Run zSMPInstall.sh.
2. Tailor StandAloneProfile.rsp file for Config.sh
3. Run zWPSConfig.sh
or zWESBConfig.sh.
4. Add RACF Profiles & Permissions
5. Start Server, Install Apps & Test . . .
(No DB2 database configuration to worry about!)
Cell
Daemon
CR
Node
App
Server
CR SR
Page 23
Create WAS Standalone AppServer,
Start, test, stop server, & back it up...
1. Run zSMPInstall.sh (same as prev.)
2. Create Database(s) & Storage Group(s)
3. Tailor standAloneProfileDB2.rsp file
4. Run zWPSConfig.sh
dbDelayConfig=false to
Create Tables for WPS & BPC
5. RACF Profiles
Extra DB2 work:
6. Fix Data Source and Data Stores.
7. Create DB2 Tables.
Start up Server, Install Apps & Test . . .
Cell
Daemon
CR
Node
App
Server
CR SR
DB2
Tables
Page 24
Four Steps . . .
1. Configure Deployment Manager
2. Configure Empty Node or Standalone Server
3. Federate Empty Node or Standalone Server
4. Perform post-configuration tasks
CR SR
AppServer
CR
Node Agnt
CR
Daemon
CR SR
DMGR
MVS System Image
CR SR
AppServer
CR
Node Agnt
CR
Daemon
MVS System Image
Page 25
Two different ND Configuration Routes:
Method B
1. Configure Deployment Mgr
2. Configure Standalone Server:
WPS, SCA, BPC, & CEI configured
Create tables for WPS, BPC, CEI, SIBs
Fix Datasources & Data Stores for SIBs
3. Federate Standalone Server
-includebuses -includeapps
4. Post-Install Tasks
Method A
2. Configure Empty Node:
Base WPS functions configured
(not SCA Buses, BPC, or CEI)
Tables created for WPS (optl)
3. Federate Empty Node
Create Server (or Cluster)
Configure SCA, BPC, & CEI
Create tables BPC, CEI & SIBs
Techdoc WP101209
Techdoc TD104335
Page 26
1. Configure Deployment Manager Cell
a) Create Deployment Manager. Start it up and then stop it.
b) Run zSMPInstall.sh
c) Tailor response file from <WPS_smpe_root>/
zos.config/DmgrDB2.rsp
augment
profileName=default
profilePath=/was62/w6cell/DeploymentManager/profiles/default
templatePath=/was61/w6cell/DeploymentManager/profileTemplates/dmgr.wbiserver
cellName=w6cell
nodeName=w6dmnode
enableAdminSecurity=true
. . .
d) Run zWPSConfig.sh.
e) Start up DMgr, look for errors, & logon to the AdminConsole.
Daemon
Dmgr Cell
DMgr
CR SR
Page 27
2. Configure Empty Node
a) Create Empty Node, but do not Federate
b) Run zSMPInstall.sh
c) Tailor response file from <WPS_smpe_root>/
zos.config/ManagedDB2.rsp
augment
cellName=f6cell
nodeName=f6nodeb
profileName=default
profilePath=/was61/w6cell/w6nodeb/AppServer/profiles/default
templatePath=/was61/w6cell/w6nodeb/AppServer/profileTemplates/managed.wbiserver
dbCreateNew=false
dbDelayConfig=true
enableAdminSecurity=true
. . .
d) Run zWPSConfig.sh.
Daemon
Empty Node
Page 28
3. Federate augmented empty node
Run the BBOWMNAN job to federate the empty
managed node
addNode.sh <DM_HOST> <DM_PORT> -includeapps -includebuses
-nodegroupname DefaultNodeGroup -username xxADMIN
-password xxx -nodeagentshortname xxAGNT1 replacelog
-trace (to create a large trace file)
(You may have to turn Security off to help.)
Configuration is now ready for WPS function,
but needs a server or cluster added next . . .
Page 29
Define new Server & tailor to installation
standards
Create application server in the
federated empty managed node
Application Servers->New
or Clusters->New
Select defaultProcessServerZOS template
Specify SHORT NAMES, Next, Save
Correct the ports in new server:
Server > Ports
Server > Web container transport chain >
WSInboundAdmin, WSInboundAdminSecure
WSInboundDefault, WSInboundDefaultSecure
Virtual Hosts > default host >Host Aliases
Cell
MVS System or LPAR
DM
C
R
S
R
Daemon
C
R
MVS System or LPAR
Node
Node
Agent
CR SR
CRA
Server
Page 30
4a. Configure Service Component Architecture
Standalone Server: SCA configured by zWPSConfig.sh
Response file parms:
configureScaSecurity=true
scaSecurityUserId=$JMSUSER
scaSecurityPassword=$JMSPASS
ND Server:
SCA Configuration Wizard in the Administrative Console:
Add support for service component architecture components
Application servers > serverName > Service Component Architecture
-or-
Clusters > clusterName > Service Component Architecture
Found under
Business Integration
Page 31
4b. Configure service component architecture
Application servers > serverName > Service Component Architecture
Configures service integration
buses and messaging engines
DB Location
Different Schema names
Uncheck!
Page 32
4c. Configure Business Process & Human Task
Containers
Standalone Server: configured automatically by zWPSConfig.sh
based on response file parms:
configureBPC=true
bpcDbName=$SQLDB
bpcDbStorageGroup=$SQLSTO
bpcDbSqlId=$SQLID
bpcDbProduct=$DBPRODUCT
bpcDbConnectionLocation=$DBLOCATION
bpcDbJDBCClasspath=$DBJDBCCLASSPATH
bpcDbUser=$DBUSER
bpcDbPassword=$DBPASS
ND Server:
Configured by bpeconfig.jacl script, or . . .
BPC and HTM Configuration Wizard which are part of
the Administrative Console:
Page 33
Configure Business Process Container
Business Flow
Manager
Using the Administrative Console:
Add support for business processes and human tasks
Application servers > serverName > Business Flow Manager
-or-
Clusters > clusterName > Business Flow Manager
Page 34
Configure BPC & HTM Containers with the Install
Wizard
- No separate HTM wizard in 6.1
Uncheck to configure
Same look as SCA config
BPC Bus
BPC datasource
BPC Security
Page 35
Install BPC Explorer, Observer, & Event Collector
Applications
Need DB2 tables configured
for event collector application
CEI required first!
(see next foil)
Page 36
Configure Common Event Infrastructure (CEI)
Standalone Server: configured automatically by
zWPSConfig.sh based on response file parms:
ceiDbProduct=CEI_DB_DB2_ZOS
ceiDbConnectionLocation. . .
ND Server: Configured by the Administrative Console:
Application servers > serverName > Common Event Infrastructure Server
Clusters > clusterName > Common Event Infrastructure Server
Check box to enable
EVENT Database
CEI Bus ME Database
Page 37
Clean-up and start your new Server
Add EJBROLE profiles for BPC, HTM, CEI security.
Create a Cluster containing this Server (optional)
Start your new Cluster Member.
Look in the Server and DMgr Logs
(SYSPRINT on spool)
Config logs
{profile_root}/logs/ directory
Login to the AdminConsole
Install and Test some applications.
TechDoc WP101218 Installation Verification for WPS on z/OS V6.1
Look for Errors, and fix what breaks
SYSA
Daemon
Cell
NAgent
f6sr01b
CR SR
NodeA
DMgr
CR SR
Page 38
ND Configuration Steps Method B
Method B
2. Configure Standalone Server:
WPS, SCA, BPC, & CEI configured
Create tables for CEI, SIBs
Fix Datasources & Data Stores for SIBs
3. Federate Standalone Server
Specify -includebuses -includeapps
Method A
2. Configure Empty Node:
Base WPS functions configured
Tables created for WPS (optl)
(not SCA Buses, BPC, or CEI)
3. Federate Empty Node
Create Server (or Cluster)
Configure SCA, BPC, & CEI
Create tables BPC, CEI & SIBs
techdoc WP101209
See techdoc TD104335
Page 39
Federate Standalone Augmented Server
a) Configure Deployment Manager for WPS & Start it
b) Configure standalone server for WPS
c) Federate server into DMgr Cell:
Use ISPF dialog : Federate an existing stand-alone
application server
Run BBOWMNAN job to federate stand-alone augmented
node (-includebuses and includeapps required)
Only Initial Federation supported
After the first augmented node is federated, only
empty nodes can be federated
Cannot back this out!
Daemon
Dmgr Cell
DMgr
CR SR
standalone Cell
Daemon
server
CR SR
NodeA
Page 40
Federation of augmented stand-alone node
Naming of resources based on stand-alone node cellname
Bus names
BPC.xxbase.Bus
CommonEventInfrastructure_Bus
SCA.APPLICATION.xxbase.Bus
SCA.SYSTEM.xxbase.Bus
Messaging engines
xxnodea.xxsr01a-BPC.xxbase.Bus
xxnodea.xxsr01a-CommonEventInfrastructure_Bus
xxnodea.xxsr01a-SCA.APPLICATION.xxbase.Bus
xxnodea.xxsr01a-SCA.SYSTEM.xxbase.Bus
Applications:
BPEContainer_xxnodea_xxsr01a
BPCExplorer_xxnodea_xxsr01a
TaskContainer_xxnodea_xxsr01a
May not want to inherit stand-alone nodes Configuration
Page 41
Common Problems:
Spellling, TYping, & Following directions
AppServer up (when it shouldn't be)
Scripts not run in the correct order
Scripts interrupted
Space exhausted in configuration HFS (& /tmp), or DB2 volumes
Authorization problems (wrong UserID)
File/Directory - Ownership/Permissions
UNIX env. setup: MAXFILEPROC, MAXTHREADS, MAXSOCKETS
Skills Required:
UNIX commands, Scripts, & Tools - Finding & using Logs & Traces
DB2 Administration
Experience, Contacts & Patience.
Page 42
Troubleshooting zSMPInstall.sh
Knowing where to look:
Log files (ASCII)
{app_server_root}/logs/wbi/install/installconfig.log Look here
for first indication of an error.
Find SEVERE to determine error
Find Buildfile previous to see what .ant task was running . . .
Look in corresponding .ant.log file(s) for more information:
{app_server_root}/logs/wbi/ - xxx.ant.log files:
2503 Jan 4 15:06 100SCleanOSGICache.ant.log
496360 Jan 4 15:06 94SDeployCoreAdminConsolePlugins.ant.log
4339 Jan 4 15:06 zSMPInstall.trace
3748 Jan 4 14:57 93SDeployWBICommonAdminConsolePlugins.ant.log
4165 Jan 4 14:57 93SDeployServerAdminConsolePlugins.ant.log
3540 Jan 4 14:57 93SDeployBPCAdminConsolePlugins.ant.log
4635 Jan 4 14:57 85SConfigNoProfileFirstStepsWBI.ant.log
2777 Jan 4 14:57 90SCleanDeployTool.ant.log
2919 Jan 4 14:57 90SConfigWBIMigrationScript.ant.log
3054 Jan 4 14:57 90SConfigureWSProfileForWBI.ant.log
2726 Jan 4 14:57 91SConfigNoProfileFirstStepsCharset.ant.log
2591 Jan 4 14:57 80SCopyInstallValidatorLog.ant.log
Page 43
Troubleshooting Knowing where to look:
Log files from zWPSConfig.sh
High-level View:
{app_server_root}/logs/wbi/zWPSConfig.trace
Default log - Start here:
{app_server_root}/logs/manageprofiles/default_augment.log
{app_server_root}/logs/manageprofiles/default/ (64 .ant.logs)
and
{app_server_root}/profiles/default/logs/ (14 more .log files)
Page 44
Troubleshooting zWPSConfig.sh example
1.default_augment.log go to end
<message>Returning with return code: INSTCONFFAILED</message>
2. find SEVERE
<logger>com.ibm.ws.install.configmanager.actionengine.ant.utils.ANTLogToCmtLogAdapter</logger>
<level>SEVERE</level> <class>com.ibm.ws.install.configmanager.actionengine.ant.utils.ANTLogToCmtLogAdapter</class>
<message>Failed to execute: CREATE TABLESPACE SCHEDTS LOCKSIZ ROW CCSID UNICODE BUFFERPOOL BP0 IN SSCELLDB USING STOGROUP
SSDBSTO</message>
3. default_augment.log find Buildfile PREV
<class>com.ibm.ws.install.configmanager.actionengine.ant.utils.InProcessAntRunner</class>
<message>Buildfile:
. . ./profileTemplates/default.wbiserver/actions/configAppSchedulerDBTables.ant</message>
4.{app_server_root}/logs/manageprofiles/default/
configAppSchedulerDBTables.ant.log
[sql] Executing file:
/etc/sscell/ssnode1/AppServerNode1/profiles/default/dbscripts/CommonDB/DB2zOSV8/SSCELLDB/createTable_AppScheduler.sql
[sql] Failed to execute: CREATE TABLESPACE SCHEDTS LOCKSIZE ROW CCSID UNICODE BUFFERPOOL BP0 IN SSCELLDB USING STOGROUP
SSDBSTO
[sql] com.ibm.db2.jcc.t2zos.y: [IBM/DB2][T2zos/2.10.59]T2zosStatement.readExecuteImmediate:nativeExecuteImmediate:807:DB2
engine SQL error, SQLCODE = -601, SQLSTATE = 42710,
error tokens = SSCELLDB.SCHEDTS;TABLESPACE
Page 45
Troubleshooting Tracing:
zSMPInstall.sh & zWPSConfig.sh scripts
Add this parm: trace *=all=enabled
wsadmin.sh scripts
Update {profile_root}/properties/wsadmin.properties file - uncomment
#com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
Trace directed to .
com.ibm.ws.scripting.traceFile={profile_root}/logs/wsadmin.traceout
Runtime servers:
MVS Command (dynamic!) :
modify <server>,tracejava=com.ibm.bpe.*=all=enabled
Reset with:
modify <server>,traceinit
Page 46
Handy tools that will help you . . .
createDB.sh script to create .sql files (and run them)
Shipped in /shared/zWPS/V6R1/zos.config/samples/
AdminConsole
Enhancements available in WAS 6.1
Other Tools:
SDSF, RMF, ITCAM
Monitors, Sniffers, Drivers
DB2 Admin Utility: Database 2 Administration Tool - 5655-D52
Backup Config HFS / zFS
DFDSS
Pax
For WPS 6.0, useful shell & jacl scripts in Techdoc articles:
fixWPSvars.jacl, Sample scripts, & Sql/DDL in PRS2520
Page 47
More Tools - Running Shell Scripts
1. Running shell scripts as a batch job:
//AUGMNT JOB 1,AUG,USER=F6ADMIN,PASSWORD=,TIME=NOLIMIT,REGION=0M
//AUGMNT EXEC PGM=IKJEFT01,REGION=0M
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
bpxbatch sh +
cd /wasv61config/f6cell/f6nodeb/AppServer/bin; +
export PATH=.:$PATH; +
export LIBPATH=/shared/db2810/jcc/lib:$LIBPATH; +
./zWPSConfig.sh -augment -response +
/u/hutch/wps61/F6standAloneProfileDB2.rsp +
1> /tmp/f6Aug_0104.out 2>&1
//COPYO EXEC PGM=IKJEFT01,REGION=0M
//SYSEXEC DD DISP=SHR,DSN=WAS600.WAS.SBBOEXEC
//SYSTSIN DD *
BBOHFSWR '/tmp/f6Aug_0104.out'
//SYSTSPRT DD SYSOUT=*
2. Use workstation text editor to compose long UNIX commands, then
copy & paste into UNIX command line.
Page 48
Summary
Configuring standalone server with Cloudscape is simple!
Configuring standalone server with DB2 requires manual work to
create some Tables (scripts, batch jobs or SPUFI.)
Configuring Network Deployment cell requires additional steps to
configure clustered servers and create DB2 tables.
Federating augmented standalone server V6.1 may be easier than a
empty node.
Use the InfoCenter to understand each of the configuration steps for
your scenario. http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r1/index.jsp
See Additional References for more valuable help.
Page 49
References
Product WebSite:
Product Websites http://www.ibm.com/software/integration/wps or /wsesb
Critical fixes: http://www.ibm.com/support/docview.wss?uid=swg27011675
InfoCenters:
WPS V6.1 for z/OS InfoCenter at
http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r1mx/index.jsp?topic=/com.ibm.web
sphere.wps.z.610.doc/welcome_wpsz.html
IBM Education Assistant http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp
(Look under WebSphere Process Server, WebSphere Integration Developer V6 - V6.1 for z/OS)
Techdocs articles at http://www.ibm.com/support/techdocs -
WP101209 WPS/WESB v6.1 for z/OS: ND Configuration Lab
TD104335 WPS for z/OS V6.0.2 Easy ND Configuration
Education: see IBM Education Catalog at ibm.com/services/learning/us/
WPS V6 for z/OS Implementation Workshop (Wildfire class ZWPS6)
WPS V6 for z/OS Implementation Workshop (OZ820) coming soon!
Introduction to WebSphere Replication Server V9 for
z/OS - Q Replication
Gaurav Bhagat
AGENDA
Types of Replication.
Architecture of Q-replication.
Implementation Scenarios of Q-Replication.
Use of Q-Replication in different scenarios.
Tools for Replication (Implementation and Administration)
SQL Replication
Capture program reads changed data from the transaction log
Stages changed data in tables (CD, or changed data, tables)
Changes pulled (or pushed) to target tables by Apply program
All done through SQL and database client-server infrastructure
Source
SOURCE2
SOURCE1
CD2
CD1 CD1
DB2 Log
Capture
Target
Apply
CCD
HISTORY
REPLICA
UCOPY
DB2
SQL Replication (aka. Data Propagator)
Source server
Staging Tables
Non-DB2
Apply
DB2
Information
Integrator
Informix
Oracle
Sybase
SQL Server
Teradata
database
recovery
log
Capture
SQL
Apply
DB2
Apply
DB2
DB2
Apply
z/OS
iSeries
UDB LUW
Capture and Apply are SQL applications
Staging is in relational tables
Control, Monitoring, and Troubleshooting information also in
relational tables
Publish once for N subscribers. A pull model
Transport is over a database connection
Target
servers
Control tables
Non-DB2
Informix
Oracle
Sybase
SQL Server
DB2 connections
Sample SQL Replication Scenarios
CD1 SOURCE
CD1 CD
UCOPY UCOPY UCOPY
Data Distribution (1:many)
CD1 SOURCE
CD1 CD
CD1 SOURCE
CD1 CD
CD1 SOURCE
CD1 CD
UCOPY
Data Consolidation (many:1)
CD1 SOURCE
CD1 CD
CD1 STAGING CD1 STAGING
UCOPY UCOPY
UCOPY UCOPY
Multi-Tier Staging
CD1 SOURCE
CD1 CD
REPLICA
CD1 CD
REPLICA
CD1 CD
REPLICA
CD1 CD
Hub & Spoke Update Anywhere
CD1 SOURCE
CD1 CD
Transformation
HISTORY
SQL-Replication Strengths
1. Data Distribution Capabilities in huge mobile Environments
More efficient fan-out to 100s of target servers
2. Multi-Vendor Sources (Oracle, Sybase, Informix, MS SQL Server)
Trigger-based Capture
3. iSeries support
Not yet available with Q Replication
4. Join Views supported as Sources
5. Many Target Table Types supported
Point In Time (PIT) - User copy + source change timestamp
Aggregates - e.g., maintain a sum in the target table (little used feature)
Why did we develop Q Replication?
Performance
Increase Throughput
Lower Latency
Reduce Overhead (e.g., less MIPS
and DB2 logging)
New Capabilities
Event Publishing
Significantly improve multi-directional
Replication Support
Q Replication
WebSphere MQ
Source
SOURCE2
SOURCE1
DB2 Log
Capture
Target
Apply
CCD
UCOPY
Capture program reads changed data from the transaction log
Puts data directly into WebSphere MQ queues no staging
MQ delivers data to system where Apply program runs
Apply pulls data from queues and applies to target tables
IBM Q Replication
Staging and transport in MQSeries persistent queue
High-throughput. Low-latency.
Tens of 1000s rows/second over 1000s of miles, with sub-second latency
Highly parallel Apply program,. Can handle Referential Integrity and Unique constraints at the target,
reorder transactions, if and needed.
Apply can perform initial table loading, also in parallel, without need to suspend
user workload
DB2
Apply
z/OS
UDB LUW
Apply
Apply
WebSphere MQ
Queue Manager
(or client - LUW)
Admin queue
Restart queue
Send queues
Control Tables
WebSphere
MQSeries Queue
Manager (or
client)
Source server Target servers
Replication
Admin
DB2
database
recovery
log
DB2 z/OS
DB2 LUW
Control tables
Capture
Data Warehouse
ETL
Event Publisher
subscription
Non-DB2
Apply
Informix
Oracle
Sybase
SQL Server
Teradata
Apply
Apply
Control Tables
WebSphere MQ
Queue Manager
(or client)
DB2
DB2
Information
Integrator
Q Capture Processing (as of V9.1)
ADMIN
Thread PRUNING
Thread
CAPMON
Prunes Monitor
Trace,
Signal Tables
Initial Thread
asnqccmd IPC Commands
asnqcap
(internal) wait Q
DB2
database
recovery
log
TransMgr
(MAX_MEMORY)
SOURCE2
SOURCE1
PUBLISH
Thread
(COMMIT_INT
ERVAL)
HOLD
Thread
Notifications
LOGRD
Thread
(SLEEP_INTE
RVAL)
Memory stats
Latency stats
Transaction states
SIGNAL
Log Reader thread (LOGRD)
Calls database log read API. If EOL sleeps for SLEEP_INTERVAL
Accumulates complete transactions in transmgr until MAX_MEMORY is reached (spill large trans if needed)
Send a notification to the publish thread for each transaction captured
if MAX_MEMORY reached and >= 1 trans to publish, sleep until worker has published this trans
Publisher thread (PUBLISH) never waits more than time remaining in
COMMIT_INTERVAL
Predictable latency: COMMIT_INTERVAL always respected .
No two-phase commit, restartQ is used for Capture restart information
WebSphere Queue
Manager (or client)
Admin queue
Restart queue
Send queues
Q Replication Architecture
Each message represents a transaction.
Highly parallel apply process
Differentiated conflict detection and resolution
Integrated infrastructure for replication and
publishing
Staged availability of heterogeneous support
Control Control
Federation
Engine
Log based
Source
Admin
WebSphere MQ
Capture
Target
Apply
Utilities
Why MQ?
Take advantage of MQ strengths
Guaranteed one-time delivery of data
SSL and security options for encryption, digital signatures, etc
Provides a secure method to move data between companies and firewalls
Many large shops have standardized on MQ
An outage at one site doesnt prevent progress on the other, eg:
Work queued up for target - no need to restart at source when outage ends
WebSphere MQ
Source
SOURCE2
SOURCE1
DB2 Log
Capture < system down >
A Few Other Points About Q Replication
Transactions are rebuilt in memory
Only committed transactions put on queues
Each transaction is a separate MQ message
TX1: INSERT S1
TX2: INSERT S2
TX3: ROLLBACK
TX1: COMMIT
TX1: UPDATE S1
TX3: DELETE S1
DB2 Log
TX1: INSERT S1
TX1: COMMIT
TX1: UPDATE S1
Q Capture In-Memory Transactions
TX3: DELETE S1
TX3: ROLLBACK
TX2: INSERT S2
Zapped at Abort
Never makes it to queue
Transaction is still in-flight
Nothing MQ put yet.
MQ Put when
Commit found
ADMINISTRATION
Q Replication Q Subscription Process
Replication
Monitor
Replication
Center
TGT3
TARGET
TGT1
Q Apply
Browser
Apply Agent
Apply Agent
Apply Agent
TGT2
METADATA
SOURCE
SOURCE2
SOURCE1
METADATA
DB2 Log
Q
Capture
Q Replication Subscription Types (and Topologies)
Primary
Secondary
Unidirectional - no conflict resolution
1:N N:1 topologies Distribution and Consolidation
Changes can be filtered and transformed
Multi-Unidirectional V9 FP1
Avoids recapture with the IGNORE TRANS option
Minimal conflict detection - not foolproof, even with 2 nodes!
Bidirectional master/slave conflict resolution
Conflicts detected by comparing data values:
Conflict rules: Check key, changed cols only, or all columns
Conflict resolution: Designated master wins
Conflict action: Force, ignore, merge change
Peer to peer timestamp based conflict resolution
Conflicts resolved by using most recent version
Very robust conflict detection - handles out of order arrivals (e.g.,
delete before insert)
Requires 2 extra columns and triggers for each replicated table
Source
Target(s)
Peers
Multi-Tier Staging with Q
TARGET2
TARGET1
Combines Q and SQL
Need to offload production system before
distributing
TARGET2
TARGET1
TARGET2
TARGET1
Staging System
Target 1 Target 2
Target n
SOURCE2
SOURCE1
Production System
Q Replication
SQL Replication
C C D
C C D
SQL Replication
Some Rules When to choose What
Low-Latency, High-Volume
Replication
Source and Target Tables of
similar Structure
Bi-directional Replication
(e.g. for Hot-Standby Purposes)
Peer-to-Peer Split Workload
Huge Number of Tables (e.g. Siebel)
Replication across DB2-Family
Replication from DB2 to federated
Targets
Fan-Out to huge Number of Targets
Multi-Tier Staging via CCD
Source and Target Tables not of the
same Structure (which requires Source
Views, Joins or SQL Expressions)
Replication from federated Sources
Q Replication
CD2
CD1 CD1
Strengths
Proven SQL Architecture
Benefits from new Architecture
SQL Replication well suited for a lot of Requirements
SQL- and Q-Replication in parallel
SQL Replication Architecture
Addresses High-Volume and Low-Latency Requirements
Robust Conflict Detection and Resolution for splitting workload
Database changes for replicated Tables read from DB2 Log
Changes immediately published to Message Queges
Major Throughput and Performance Improvements
New Q Replication Architecture
How Customers Use the Products
1. Live Reporting - Our major market today (around 90% of
customers)
Offload production/OLTP server
2. Data Distribution/Consolidation
Multiple Points of Sales (POS)
Often bidirectional (with assumption of no conflicts)
3. Continuous (High) Availability
Workload Balancing or Live Reporting + Disaster Recovery
4. Application Integration
Publishing changes to reporting applications (growing market for event publishing)
5. Feed the Warehouse
Source Target + ETL (growing market for event publishing)
Two notes:
- Most enterprises use replication for several of these scenarios
- Sometimes replaces home-grown, semi-automated processes
Updating applications work against primary system
Read-only applications can use either primary or backup (workload distribution)
On fail-over, data can be captured on backup for replication to primary
Read/Write Applications
Read Only Applications
Connection Available for Failover
Primary Connection
Capture
Capture
Apply
Apply
Primary connection
Read-only use allowed
Updating applications
DSNA DSNB
In an Outage of the Primary
Q Apply can be allowed to process all changes still in queue at target
Updating applications switch to secondary system
Q Capture starts replication of changes to target
Primary Connection
Capture
Apply
Primary connection
< system unavailable >
DSNB
On Return of Primary
Secondarys changes are applied to the source
Switchback to primary can be planned for updating application
Secondary is usually considered winner if conflict detected
Primary Connection
Capture
Capture
Apply
Apply
Read-only use allowed
DSNB DSNA
Online Trading A case for very high speed
replication
In many online environments OLTP data is kept
separately from query/history data for better
performance of both update and query applications
This user has just made an online trade he will keep
hitting enter until he sees that the trade is complete,
in this case meaning it has been replicated to the
trade history database
Trade
Processing
Data
Q Capture
Trading
Applications
Trading
Applications
Q Apply
Trade History
Applications
Trade History
Applications
Trade
History
Data
Order Processing Exploiting II Event
Publishing
As new orders are entered into the order entry
system, the pertinent data is captured and published
into a queue
The Websphere Business Integrator Event Broker
processes the queued data
A billing transaction is created and queued in one
system and a shipping transaction is created and
queued in another system
Order Entry
Data
Q Capture
Create New
Order
Create New
Order
Create Shipping Request
WBI Event
Broker
Create Billing
Request
Create Billing
Request
Useful Tools
Replication Center
Definitional user interface based on the DB2 Control Center
asnclp
Command line processor for replication definitions
Q Replication Dashboard
asntdiff and asntrep
diff utility for comparing and repairing source and target tables
Replication Administration
Replication Center GUI
Launchpads, Wizards, Online Help
Definitions, Operations, Monitoring
Command Line Interface
Scripts or interactive mode
Example:
Java APIs
Typically used when replication is embedded
C:\asnclp
REPL > CREATE QSUB USING REPLQMAP ...
REPL > CREATE SUBSCRIPTION SET SETNAME ...
REPL > CREATE MEMBER IN SETNAME ...
Q Create Subscription Wizard
Create large numbers of
subscriptions at a time!!
Q Replication dashboard
A new graphical tool.
Gives a dashboard view of your replication setup.
Other Important Sources of Information/Education
WebSphere Information Integrator sites on the web:
http://www-306.ibm.com/software/data/integration/
http://www-306.ibm.com/software/data/db2imstools/
http://db2ii2.dfw.ibm.com/wps/portal/!ut/p/!ut/p/!ut/p/.scr/Login
Developer Works:
http://www-106.ibm.com/developerworks/db2/zones/db2ii/
Tutorials, whitepapers, samples available now
IBM Education for Q Replication:
DW240: 3 day course without MQ basics
DW241: 4 day course with MQ basics included
Redbook recently published for Q Replication, EP later this year
Consider IBM Services as part of your implementation plan
V9 Replication Product Names: z/OS
IBM WebSphere IBM WebSphere
Data Event Publisher Data Event Publisher
for z/OS for z/OS
Event publishing only HAAW910,JAAW913
WebSphere MQ prerequisite
Available for DB2 z/OS V7, V8, and V9
Includes Q Replication and SQL Replication
Different FMIDs
Available for DB2 z/OS V7, V8, and V9
Q Replication prerequisite:
WebSphere MQ
IBM WebSphere IBM WebSphere
Replication Server for Replication Server for
z/OS z/OS
Note : IBMrecently announced WebSphere Replication
Server for z/OS will be re-branded to InfoSphere Replication
Server for z/OS
Questions
THANK YOU
Lawrence Kwek
Technical Consultant
IBM Software Group | ASEAN Software Lab Services
Session I04: Introduction to Rational Developer for System z
(
(
(
RDz
RDz
RDz
)
)
)
Page 1
History of Rational Developer for System z
WebSphere Developer for zSeries
WebSphere Studio Enterprise Developer
WebSphere Developer for System z
Rational Developer for System z
2007
2006
2005
2004
Page 2
Host Workstation Overview
'Magic'
PDS 'A'
PGMA
PGMB
PDS 'B'
PGMX
PGMY
z/OS
Project
Folder 'A'
PGMA
PGMB
Folder 'B'
PGMX
PGMY
Files on the host look as though they are workstation files
Page 3
submit compile job swap to SDSF
select job
find error msg
find code line
(remember error)
swap to edit
session
exit JCL edit source find code line
change code
exit source
edit JCL
ISPF based Development
Multiple screens/sessions and multiple disparate tools
20 x 80 characters of content
Page 4
RDz IDE - Eclipse-based development
Error list in Problems view
Statement
in error
indicated in
source
Outline view presents
COBOL structure
Double-Click on
the Error
Syntax Check
Submit jobs,
access job
output, or
open source
members
with a single
click
Open and edit
multiple
source and
JCL members
simultaneously
Edit Source
Common development environment for COBOL, PL/I, C/C++, and Java
Simplified development with more information at your fingertips
Page 5
Interactive Access to z/OS
LPEX
Editor
MVS
datasets
Page 6
Content Assist for COBOL / PL/I / C or C++
Find all
statements
and
data names
Benefit: Developers complete
code more accurately and
efficiently.
Page 7
Useful Eclipse features available to zOS assets..
Like Compare
Able to merge the differences
using the icons
Benefit: help maintenance for
COBOL, PL/1, C, C++ or JCL
Page 8
Useful Eclipse features available to zOS assets..
Like Replace with Local History
Keep as many local versions as
you want and compare with the
z/OS current version..
Benefit: help maintenance for
COBOL, PL/1, C, C++ or JCL
Page 9
Use local/remote compiler to do syntax checking
Local Syntax
checking..
Benefit: Uses local CPU
Just double-
click to
find the
error
Page 10
Benefit: Developers focused on business logic and not on writing JCL, JCL
smart editor, Outline
JCL Generation and Submission to z/OS execution
JCL
generated
from
COBOL
Code
Page 11
Remote/Local debug
Change
contents,. etc..
Breakpoints,
watchpoints,
Jump to, Run to
etc..
Change
contents, etc..
Benefit: Same Debug Perspective used for COBOL, PL/I, C, C++, Java, JSP,
HTML, etc..
Page 12
Initiated based on:
Transaction
Program
User ID
Termid
NetName
etc
Can use TCP (IP)
Workstation tcpip address
Port to broadcast on
Remote CICS Debug
Setup to use RDz
Page 13
Monitoring Job Output
Benefit: Developers do not have to continually switch between systems to
use SDSF. No TSO and SDSF needs.
Page 14
TSO Commands (as well USS)
Content Assist
Page 15
Allocate, Create PDS/PDSE, Member, etc....
PDS allocation models, example PDS for COBOL source, Listing, etc.
Compress, Compress with Backup, Migrate
Copy files between different systems (local or remote)
Etc.
z/OS files and Dataset Management
Benefit: Developers can easily allocate datasets and create members
on z/OS. No need for ISPF utilities.
Page 16
Menu Manager
Allows creation of custom
menu items
Menu items can be
associated with a TSO
command or a URL
menus within Project view
and RDz code Editor
Substitution Variables based
on the current selected
object
results to appear in pop-up
window or console view
1
2
3
Page 17
Remote connection to AIX
Can use RDz to connect to AIX
Access to files on AIX
Edit, copy, compile, link
Debug support to AIX through IBM debug tool (AIX)
Pre-reqs
IBM COBOL compiler
IBM Debug tool (AIX)
Page 18
Working Offline
Allows users to edit and Syntax Checking on
files while disconnected from the host.
Detects Conflicts and changes
1
2
3
4
5
Page 19
CICS BMS Map Support
BMS Project
Wizard for creating new
BMS map set files
Drag & Drop BMS editor
Design, Source and
Preview views
Create new or
import/edit existing BMS
maps
Works with local and
remote scenarios
Page 20
Generate JCL for maps
Generate JCL directly for map files
JCL for Assemble
JCL for Assemble and Link
A properties sheet define the Properties for
the .bms member to be used for the compile
Page 21
We need interfaces to talk XML .
Web services client CICS application
<XML>
<DATA>123</DATA>
</XML>
010203
XML Data
COBOL Data
Call Get Customer Info
X
Web services client CICS application
<XML>
<DATA>123</DATA>
</XML>
010203
XML Data
COBOL Data
XSE generated
converter
Without XSE
With XSE
Page 22
Example: z/OS CICS Version 2.2 or 2.3 and adapters created
with RDz
COBOL
Data declarations
(or complete
program)
Input
converter
Driver program
XML schema
definition (.xsd)
Rational
Developer for
System z XML
Enablement
Output converter
Enables COBOL-based applications to consume and produce
XML messages Original COBOL program unchanged
WSDL
Page 23
WSDL
Test Using
Rational
Developer
for System z
Web Services
Explorer
Test the Web Services created calling z/OS CICS
Using RDz to test the CICS Web Services..
Page 24
Example: Testing using RDz
Page 25
Debug Tool Integration
Supports COBOL, PL/I, C/C++
Supports CICS, IMS, DB2,
batch (and even WebSphere
Application Server
applications!)
From the workstation:
View executing source code
Step through host code line-by-line
Set breakpoints
Alter working storage values
Alter register values
Etc.
Debug z/OS and distributed
code in the same interface,
even stepping between
runtimes and platforms!
Requires IBM Debug Tool
V7.1
Page 26
New in 7.1 - File Manager Integration
Allows for a formatted
edit session of many
dataset types. Among
the options are:
VSAM - KSDS, ESDS,
RRDS, VRRDS
QSAM PDS, SDS
Multiple views of the
data within the
formatted edit session:
Table
Single
Character
Requires
IBM File Manager V7.1
You can edit a
particular record
that is selected
from the table or
file.
Page 27
New File System Mappings
Page 28
Display Data in Character format
Page 29
Display Data in Table format
Page 30
Display Data in HEX format
Page 31
New in 7.1 - Fault Analyzer integration
Fault Analyzer gathers
information about an
application and the
surrounding environment
at the time of the abend
Integration allows RDz
user to access and view
Fault Analyzer history
files
Requires Fault Analyzer
V7.1
Page 32
Fault Analyzer for z/OS overview
User application
MVS IEAVTABX
LE CEEEXTAN
CICS XPCABND
LE CEECXTAN
SYSMDUMP
Fault history
SYSLOG
Fault Analyzer
Fault analysis
report
Abend
Invocation
exists
RDz client
Page 33
FA perspective
Outline view
Detailed view
Artefacts
view
Lookup view
Browsers
Page 34
FA report browser
Report
contents
Source
display
Abend code
lookup
Tabs
Page 35
Dump Browser
Hex dump storage display
Command
support
User notes
Dictionary
data
Page 36
Copyright IBM Corporation 2007. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any
kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor
shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use
of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in these materials may change at any time at IBMs sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product
or feature availability in any way. IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business
Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
Learn more at:
IBM Rational software
IBM Rational Software Delivery
Platform
Process and portfolio management
Change and release management
Quality management
Architecture management
Rational trial downloads
developerWorks Rational
IBM Rational TV
IBM Rational Business Partners
I06 - A Quick SOA Victory Using a Wizard's HATS:
A Case-Study of Meralcos Entry into the SOA World
Slides by: Elmer L. Senson (sensonel@ph.ibm.com)
Labs by: Anton Orpilla (orpilla@ph.ibm.com) Proctor: Mike Martinez (martinma@ph.ibm.com)
Page 1
Agenda
The SOA Entry Points
The Connectivity & Reuse Entry Points
SOA and the Mainframe Software Environment
What is Host Access Transformation Services?
The Meralco SOA Case-Study
NSPP Overview
Challenge #1: Single Sign-On (SSO) to the Host
Challenge #2: Complex Host (3270) Screen Flows
The First-of-a-Kind Success
Page 3
SOA Connects People, Process, and Information
Wired and wireless devices
Global 24x7 access and
real-time collaboration needs
Unconsolidated and untailored
information
Heterogeneous
data types and
sources
Untransformed
and inconsistent
data
Operational and
Analytical Data
Development and
integration of
application assets
Application silos
(legacy and packaged
applications)
Heterogeneous
internal and external
systems
Information
Process
People
Page 4
The People Entry Point
Efficient, collaborative, real time decision making and execution for
business agility and responsiveness
a starting point for SOA - enabling
people to interact with application and
information services supporting
business processes.
Why Its Important
People are the drivers of the business they interact with reusable business
services using the right information at the right time!
Page 5
The Process Entry Point
Integrated and synchronized business systems and/or choreographed
business & system activities into reusable process components
the process of supplementing a
manual process with an automatically
controlled alternative through the
orchestration and integration of
technology and human assets to form
streamlined processes.
Why Its Important
Automated processes reduce administrative time and adaptable and reusable
processes to enable faster reaction to business indicators.
Page 6
The Information Entry Point
Actionable information to people and processes
connects, enhances and delivers in-
context information across diverse
operating systems, applications and
legacy systems through reusable
services.
Why Its Important
Information Services enables consistent views and maintenance of data and
content, providing a single view of the truth to people and processes.
Page 7
People, Process, and Information Primary Products
Information
Process
People
WebSphere Portal
Workplace Forms
Workplace Dashboard Framework
Lotus Expeditor
Lotus Sametime
IBM Information Server
IBM Master Data
Management
DB2 9 pureXML
WebSphere Adapters
Page 8
Connectivity establishes links
between applications and services
using an Enterprise Service Bus
Reuse creates new services from
legacy assets
Connectivity
Reuse
SOA Entry Points Reuse and Connectivity
IT focused entry points to enable a Flexible IT
Page 9
Achieve business process innovation
through treating tasks as modular services
Greater productivity and flexibility through
targeted user interactions for improved
operations and collaboration
Service-enable existing assets and fill
portfolio gaps with new reusable services
Connect systems, users, and business
channels based on open standards
Provide trusted information in business
context by treating it as a service
Process
People
Reuse
Connectivity
Information
Page 10
SOA
Governance
SOA
Management
SOA Entry Points
Business
Process
Management
Interaction and
Collaboration
Services
Information
as a Service
Where Meralco
is Now
SOA
Design
Service
Creation
Service
Connectivity
SOA
Security
Page 12
Business value through SOA Connectivity
Deliver a secure, consistent user experience
Strengthen trading partner relationships
Potential savings of 2X-4X
Minimize complexity in your organization
Increased business flexibility and responsiveness
Eliminate redundancy
Decreased development and maintenance costs
*Software Strategies
Enterprise Integration Challenge 2005
Page 13
Business value of Reuse for SOA
Existing business logic is among the most
valuable assets a company owns
It is 5X less expensive to reuse existing
applications than to write new applications from
scratch*
Reusing proven, time-tested applications results
in significantly lower risks and faster time to
market
Maintenance overhead shrinks with greater use of
proven and tested code for common functions
* Software Productivity Research (SPR)
Page 14
Reuse: Service-enable existing IT assets
Eliminate expense, confusion, and
risk associated with redundancy
Automatically identify candidate
assets for service-enablement
Make systems that previously
required specialized skills
accessible to a broader pool of
users
Combine functions that come from
several sources and treat them as
a single logical service
Obtain the services you need to support your
business process by service-enabling existing assets
Page 15
Service-enable existing assets
Reuse high-value assets in more flexible
ways
Three sources of services
Consume external services
support commodity tasks with externally
provide services
Create new services from scratch
Fill in gaps in portfolio by creating new
services
Page 16
The Evolution Of Services to Business Services
Composite Business Service
Packaged, interrelated,
reconfigurable Business
Service groups building a
business function
Dynamic; Tailored to
an industry's users,
policies and methods
Business Services
Performs a
generic task
Service
Page 17
Reference: http://www.ibm.com/developerworks/webservices/library/ws-soa-progmodel11/
Just What is HATS?
(Host Access Transformation Services)
Page 19
Host Access Transformation Services
A rules-based Web-to-host transformation engine
Provides customized access to one or more host applications
Dynamically creates a new Web HTML interface,
Significantly improves the navigation and productivity of legacy
applications.
Page 20
What is HATS?
No requirement for programming skills
Default rules allow host applications to be on the Web within hours of loading the
HATS software
No need to customize every screen
No need to access or modify source code
applications.
Page 21
What is HATS?
Merge data from various sources into a single Web interface
Supports transformation and data objects from 3270 and 5250 hosts
Supports data objects from VT hosts
Add further customization at your own pace
applications.
Page 22
What is HATS?
Zero-footprint, zero-download
Only software needed on the client is a Web browser
Real-time, on-the-fly transformation means that HATS does not "break down"
when changes are made to the host application
applications
Page 23
HATS Primary Purpose is of Presentation
It provides a web-based GUI for
Green Screens Applications (3270
or 5250)
In Meralco: not really using HATS
for its primary purpose
HATS Macros
Screen Scraping
HATS Integration Objects (IO)
Pull Host data from programs (Java)
Chaining IOs
Web Express Logon (WEL)
Network Security Mappers
Credential Mappers
Page 24
HATS Macros
Simulates the 3270 Screen Flow
Screen Scrape Data from the 3270
Screen
Page 25
HATS Integration Objects
The Integration Object is the Java
representation of a HATS macro
Data extracted by the macro becomes a
Java Object
Integration Objects can be chained
Obtain data for a Java program from
multiple 3270 screens
Introduce more Complex Logic than
available in the Macro
Page 26
HATS Web Express Logon (WEL)
Provides Single Sign On (SSO) capability for HATS
Page 28
Meralco selected improving its Distribution
Management Systems for its SOA Pilot
Key Considerations:
The systems have the necessary
functions but are not integrated
There is no single application that
can meet the needs without
significant compromise
Also, a rip and replace would lead
to serious people, process and
technology risks
Rather than replace, we expose
each individual function as a web
service and combine to a composite
application
Page 29
The Network Service Provisioning Portal (NSPP) is
a composite application of the existing systems
Maintenance Work
Order Management
System
Work Order
Management System
(for New Facilities)
GIS
Outage Management
System
Page 30
The Work was Divided into 5 Streams
Stream 1: Construction and Maintenance Work Orders (WOMS & MWMS)
Single view of construction and maintenance projects and work orders
Integrated creation of work order and crew assignment
Locate affected facility (maintenance) and nearest facility (construction)
Stream 2: Management of Trouble Records (OTMS)
Integrated facility views and trouble records (locate facility with pending complaints)
Integrated trouble record management and crew assignment (list of crew/ trouble record group)
View materials requisitioned for trouble records
Stream 3: Manage Construction and Maintenance Details
Managing construction and maintenance work order details (adding incidents, budgets etc)
Stream 4: Updating Construction and Maintenance Work Order Status
Updating and closing work orders
Printing construction work order material accounting
Stream 5: Additional Features for Management of Materials
Single view of construction and maintenance material transactions
Requisition, transfer and re-use of materials
Page 31
Breakdown of Use-Case/Services Streams
Page 32
Work Streams were aligned to Use Case groups
Stream 1
Stream 2
Page 33
UC-4.2 Assign Construction Work Order to Contractor / Crew
Display
Work Order
Detail
Display Map and
Work Load Profile for
a selected Crew or
Contractor
Display List
of possible
Crews and
Contractors
Display List
of
Contractors
Contracts
Assign a
Crew /
Contractor
to WO
UC-2.5 List of Trouble Records
Input Filter
Criteria
Display List of Trouble
Records (filtered by
Responsible Centres
?)
Display Filtered List of
Trouble Records
UC-2.4 List of Crew / Trouble Record Group
Input Filter
Criteria
Display List of
Complaints Pending
Resolution (filtered by
Responsible Centres
?)
Display List of Crew /
Trouble Record
Groups
UC-1 Sign On
Input User id
and
Password
Customised
NSPP
Welcome
Screen
UC-1.5 Change Password NSPP
Change
NSPP
Password
Password
Change
Result
UC-1.4 Change Password MapGuide
Change
MapGuide
Password
Password
Change
Result
UC-1.3 Change Password OTMS
Change
OTMS
Password
Password
Change
Result
UC-1.2 Change Password WOMS
Change
WOMS
Password
Password
Change
Result
UC-1.1 Change Password MWMS
Change
MWMS
Password
Password
Change
Result
UC-2.1 List Construction and Maintenance Work Orders
Default WO
Filter Criteria
Display
Consolidated
Construction
and
Maintenance
WO List
Custom WO
Filter Criteria
Display
Consolidated
Construction
and
Maintenance
WO List
UC-2.2 List of Facilities with Complaint Pending Resolution
Input
Responsible
Centre
Display Default
Responsible Centre
and List of Facilities
with Complaint
Pending Resolution
Display Responsible
Centre and List of
Facilities with
Complaint Pending
Resolution
UC-2.0 List Construction and Maintenance Projects
Default
Project Filter
Criteria
Display
Consolidated
Construction
and
Maintenance
Projects List
Custom
Project Filter
Criteria
Display
Consolidated
Construction
and
Maintenance
Projects List
UC-2.3 Complaints Inquiry per Facility
Display List
of
Complaints
for Facility
Display
Complaint
Details
UC-3.0 Create Maintenance Work Order
Display
Division of
Work
Budget
Details
Display Project Detail
and Work Order
Creation Form
UC-3.1 Create Construction Work Order
Display List of WO
Construction Point
and WO Breakdown
by Construction Point
Display
Construction
WODetail
Display
Project
Design
Specification
Display List of
Phases for a
Construction
Point
Display List of WO
Construction Point
and Details for a
selected Phase
Display
Created Work
Orders
UC-3.2 Acknowledge Receipt of Design Sketch
Display
Acknowledge
Receipt
Screen
UC-3.3 Create Trouble Record
Display Screen to
Create Trouble
Record for selected
Equipment or
Complaint
UC-3.4 Locate Facility with Pending Complaint
Display Map of
selected TLN and
other transformers
with pending
Complaints
UC-4.0 Assign Maintenance Work Order (to Contractor / Crew)
Display List
of Possible
Crews or
Contractors
Display Map and
Work Load Profile for
a selected Crew or
Contractor
Assign a
Crew /
Contractor
to WO
UC-4.1 Locate Maintenance Work Orders Affected Facility
Display
Location
Map for
Affected
Facility
UC-4.3 Create FMS Project for Construction Work Order
Display List
of Work
Orders for
FMS
Creation
Display
Result of
FMS
Creation
UC-4.4 Display Nearest Facility to Construction WO
Display Location Map
showing nearest
Facility to WO
UC-5.1 Materials Requisition for Maintenance Work Order
Display
Materials for
Requisition
for Work
Order
Display Details of
created transaction for
Materials Requisition
UC-5.6 Re-Use of Materials for Maintenance Work Order
Display List of
Materials for Return
for Re-Use for
selected Work Order
Display Updated
Quantities
UC-5.8 Material Requisition for Construction Work Order
Display List of
Materials to be
Requisitioned for
selected Work Order
Display Created
Requisition
Transaction Details
UC-5.13 Re-Use Material for Construction Work Order
Display List of
Materials for Return
for selected Work
Order
Display Re-Use
Materials Request
(and updated
quantities)
UC-5.14 View Trouble Records Materials Requisition Transactions
Display Materials
Transactions for
selected Trouble
Record
Display
Materials
Requisition
Details
UC-5.15 Material Requisition for Trouble Record
Display List of
Required Materials for
Selected Crew
Display List
of Possible
Materials
Display List
of Stores for
selected
Material
Display
Created
Material
Requisition
UC-6.12 Modify Trouble Record
Display Details for
selected Trouble
Record
UC-4.5 Assign Trouble Record to Contractor / Crew
Display
selected
Trouble
Record
Details
Display Work Load
Profile for a selected
Crew or Contractor
Display List
of possible
Crews and
Contractors
Assign a
Crew /
Contractor
to Trouble
Record
UC-7.9 Update Trouble Record Progress
Display
selected
Trouble
Record
Details
Key UC-n.n Use Case Name
Logical
Screen
1
Logical
Screen
2
UC-n.n Use Case Name
Logical
Screen
1
Logical
Screen
2
Use Case Logical Screens
Duplicated Use Case
The Use-Case Driven Streams were translated into
front end (screens) and back-end (services) builds
Page 34
Introduced the concept of an Enterprise Security
Management Framework (eSMF)
Pilot implementation of an
enterprise Security Management
Framework (eSMF) for 100 users
using TAMeB & TDS
We deconstructed the use of
TAMeB because instead of
deploying just the traditional
pattern (i.e. access based on
WebSEAL Junctions), we also
used it for providing credentials at
the service level
Page 35
Challenge#1: Single Sign-On (SSO) to the Host-
Based Back-Ends
Contributing Factors:
NSPP used a 3-tiered Architecture
Meralcos RACF couldnt be used as an LDAP user repository
HATS Web Express Logon (WEL) couldnt solve our problems
HATS 3270 Connections couldnt be pooled across an Clustered Application
Server (HA)
A RACF user can only logon once (One User One Session)
Other Considerations:
HATS Connection Pooling improves performance
Creating a 3270 Connection to the mainframe is expensive
Creating a 3270 Connection vs. Logging On
NSPP
Web Application
Enterprise Service Bus
(Process Server)
NSPP Security
(TAMeB/WebSEAL)
Host Integration
(HATS)
Meralco Host
(WOMS/OTMS)
1 2
3
7
6
5
4
Page 36
Example: Logging On to the Mainframe (1)
then press <Enter>
Page 37
then press <Enter>
Page 38
Page 39
Challenge#1: The Meralco SSO solution
We really did use Web Express Logon
However, we did modify a lot
TAMeB
Global Sign On (GSO) LockBox
(credential vault)
Page 40
Challenge#2: Complex Host (3270) Screen Flows
Contributing Factors:
Meralcos Host Applications is a Do Action on Selected Item
Some Host Applications do not present data in tables
HATS 3270 Connections couldnt be pooled across an Clustered Application
Server (HA)
A RACF user can only logon once (One User One Session)
Other Considerations:
HATS has a settling time
Amount of data being scraped from the green screen
Our services needed to be stateless and independent
Page 41
Example: Action on Selected Item (1)
press <PF16> WOMS
Page 42
then press <Enter>
Page 43
then press <Enter>
Page 44
press <PF6> PgDn
multiple times
Page 45
then press <Enter>
Page 46
Page 47
Example: Non-Tabular Data (1)
press <PF14> OTMS
Page 48
then press <Enter>
Page 49
then press <Enter>
Page 50
then press <Enter>
Page 51
Page 52
Example: Data in Multiple Screens (1)
then press <PF15> Toggle Data
Page 53
then press <PF15> Toggle Data
Page 54
press <PF6> PgDn
multiple times
Page 55
Thats 5 screens across and 36 screens down
resulting in 180 screens of data!
Page 56
Challenge#2: The Meralco Solution to the Complex Flows
Relegated the HATS macros to data retrieval and navigation
The Logic given to Java programs (SCA components in Process Server)
The Chaining of Integration Objects
Handled Action on Selected Item
1. Peeked at data @ current line (via IO/macro)
2. If not selected item, move to next line and repeat #1
3. If selected item, do action (retrieve data, navigate to next screen)
4. End chain
Handled Data Manipulation of complex data
Non-Tabular data
Data from multiple screens
Handled complex 3270 Screen Navigation
Page 57
then Login
Page 58
Select Work Order Search
Page 59
then Search
Page 60
then Execute
Page 61
Page 62
I
t
s

t
h
e

s
a
m
e

W
o
r
k

O
r
d
e
r
!
!
!
Page 63
Page 64
The OTMS Side
Select List of Facilities
Page 65
The OTMS Side
Page 66
I
t
s

t
h
e

s
a
m
e

D
a
t
a
!
!
!
Page 67
The OTMS Side
Page 68
The OTMS Side
Page 69
The MWMS Side
Page 70
The MWMS Side
Page 71
After Clicking Logout
Page 72
Summary
What is Host Access Transformation Services?
NSPP Overview
Challenge #1: Single Sign-On (SSO) to the Host
Challenge #2: Complex Host (3270) Screen Flows
I06 - A Quick SOA Victory Using a Wizard's HATS:
A Case-Study of Meralcos Entry into the SOA World

Binder 1

Enviado por

Dados do documento

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Binder 1

Enviado por

Direitos autorais:

Formatos disponíveis

IBMs Web 2.

1a. cleartool mkview -snapshot -tag $BF_TAG -vws

hidden object issue

hidden object issue

Java heap size

Java heap size

Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer

Thread Monitor Dump Analyzer

Test Plan Elements

Common issues (cont.)

Common issues (cont.)

Manage The Complexity!

Adding Value to the WebSphere Portfolio

WMQ with HA Clusters

s Identity Flow Solution

SSO with OOB Acct Linking

SSO with OOB Acct Linking

2008 WebSphere Technical Conference

J 2EE URL Authorization

Identity Provider Flow

Service Provider Flow

Dynamic Group Membership

Placing Attributes into LTPA

Application Development Perspective

Phishing defenses aren

Biometrics are broken

RFID means I can see you but you can

Web 2.0 = Hack 2.0

Open source means open season

Você também pode gostar