Mastering Sap Netweaver Pi Throughput With Message Packaging

Mastering SAP NetWeaver PI Throughput with Message Packaging
SAP DEVELOPER NETWORK | sdn.sap.com BUSINESS PROCESS EXPERT COMMUNITY |

bpx.sap.com2007 SAP AG 1
Mastering SAP NetWeaver PI Throughput with Message
Packaging
Applies to:
SAP NetWeaver Process Integration IT Scenarios in Versions 7.0 (and higher)
Summary
This document shows how to implement message packaging, starting from the motivation and scope,
later showing the implementation details and then how to configure and monitor.
There is a final section showing some analysis charts and the variables that affect packaging at runtime.
Author: Daniel Horacio Bianchin
Company: SAP
Created on: October 2007

Author Bio

Daniel has been working with SAP technology since 1996, and in 1997 he joined SAP.
Over the last 10 years he has worked on more than 35 projects plus consulting, training
and presales activities.At SAP, he started as an ABAP and RFC (C++) consultant later
began to work with ALE interfaces, doing quality assurance, development optimization, in-
house software development and design, and then CRM Internet Sales implementation
and interface architecture design. After that, he has been leading development and
integration teams using SAP BC and XI. In 2005 he joined the SAP NetWeaver Regional Implementation
Group.
Table of Contents
Introduction and Motivation........................................................................................................................... 3
SAP NetWeaver Process Integration Specific Message Size Impact on Performance............................ 3
Typical integration patterns and message sizes ....................................................................................... 3
Queries ..................................................................................................................................................................4
Single Business Object replication.........................................................................................................................5
Files .......................................................................................................................................................................5
Optimization through Message Size Enhancement.................................................................................. 5
Packaging implementation description......................................................................................................... 7
Configuration............................................................................................................................................... 10
Activation................................................................................................................................................. 10
Configuration Types ................................................................................................................................ 10
Effect of Configuration Types in packaging creation procedure ............................................................. 13
Business Component configuration for Proxies ...................................................................................... 14
Packaging monitoring configuration........................................................................................................ 14
Considerations for IDoc Packaging functionality......................................................................................... 15
Monitoring.................................................................................................................................................... 15
Variables affecting packaging behavior...................................................................................................... 17
Message Size.......................................................................................................................................... 17
System Load............................................................................................................................................ 17
Final Remarks............................................................................................................................................. 22
Summary..................................................................................................................................................... 22
Related Content.......................................................................................................................................... 22
Copyright..................................................................................................................................................... 23

Introduction and Motivation
SAP NetWeaver Process Integration Specific Message Size Impact on Performance
If we build a chart showing the impact of message size in PI performance, we can conclude that there is
an area, where typical message sizes behave especially well.

So we recognize three different regions:
Best performing message sizes: messages ranging from rather small messages size that has a
drawback of about 50% compared to the top performer size, but still very usual, to a maximum limit after
which it is not elementary to maintain the best performance without further considerations, for example,
considering parallel processing, if you are processing big messages in parallel, you have to calculate that
the memory required at that moment (that usually ranges from 3 to 7 times the message size) has to be
multiplied by the number of messages running. So, with not many parallel messages, you will be
consuming lots of memory, what is not the case if you keep the maximum message size to a smaller limit.
Small Message Sizes: As messages get smaller, the performance drawback keeps on increasing, that is
basically due to 2 factors:
Context Switching: It represents the resources spent in creating the environment to process the
message inside its own logical unit of work, also considering the database operations. Since this
time is nearly constant, if message processing time is smaller, context switching signifies an
important overhead.
Message Header Processing: Each message has a header that requires some processing
resources too. The smaller the message, the bigger the percentage of time spent in header
processing.
Large Message Sizes: As explained before, as you leave the recommended message size region, using
larger message sizes, some other factors may become more important, perhaps disturbing normal
system operation. When the messages continue growing overstepping certain limits, throughput starts to
decline due to java virtual machine garbage collection times.
Typical integration patterns and message sizes
Typically, the message sizes are related to some integration patterns, this is a hint to help figure out if
packaging makes sense in your scenarios or not for example:

Scenarios that replicate business objects, many times using IDocs, correspond to small or best
performing message sizes.
Queries, except those that retrieve long listings of objects instead of single objects, are usually inside the
recommended region.
There are also short queries for objects attributes that tend to be on the left side, they basically retrieve
descriptions, translate codes or similar tasks.
And finally, not from a request-type approach but from an adapter-type, we find the operating system
files. A common scenario regarding files, are billing runs that generate files that has to be translated into
accounting documents. Those files usually exceed the recommended limits and have a tendency to make
PI work not as a message broker that is its natural architecture.
If we go into detail for each of these objects, we can make interesting findings regarding the optimizations
options.
Queries
Queries are synchronous messages, used to perform conversions, retrieve descriptions or details about a
particular object or perhaps a listing to narrow down a search.

Many times are located below the recommended message size.
What is very interesting is that since normally the retrieval function on the back-end system involves no
logic but an optimized database access, the percentage of time spent in the middleware is pretty
important, that is, optimizing or fine tuning PI implementation for queries, brings an overall advantage to
the end-to-end processing time of the query.
Single Business Object replication

When you replicate business objects, you normally send asynchronously a set of records to be validated
and hopefully created or changed in the back-end system database.
Most of the computational logic, supposing you have a rather intelligent back-end system, is performed
in that system, since it has to check many business rules to later post the information.
In these scenarios, assuming there are not very complex mapping rules, the percentage of time spent in
the middleware, is likely to be smaller compared to queries, so even when it is important to optimize PI
implementation, the overall benefit wont probably be so important.
Files

Files commonly represent a processing unit rather than an object instance. Thats the reason for its
tendency to exceed normal message size.
The act of generating a file can be assumed as a restrain in the business object replication flow, since the
information is not available until the file is finally closed. So, when it is released for replication all that
information must be properly managed in groups so as to improve the flow and also prevent the
middleware and receiver back-end systems to clog.
This normal splitting and packaging steps usually represent a considerable amount of time both in the
middleware and the back-end system, additionally the back-end system will have to comply with the same
requirements than single business object replication -all in all, consuming the same resources that the
middleware did-. From that point of view, PI optimization could represent either a medium or an important
benefit in the total processing time.
Optimization through Message Size Enhancement
Supposing we have to optimize the throughput in an scenario where message size is rather on the left
small messages area, packaging could be an interesting solution, since it will be pushing message size to
the right into the best performing message size area.

If, on the other hand, we are on the right side, very likely working with files, we can apply the file splitting
functionality that has been available in the file adapter, to push message size to the left.
Packaging implementation description
To better understand how message packaging works, it is required to know in advance how messages
are processed nowadays.
A message arrives to PI; it is
persisted in a message queue.
Then the message starts the first
pipeline process,

And is stored in the message queue
again.

The process is repeated for each
pipeline step until the message is
finally delivered to the proper
receiver.

The main purpose of this picture is to show that every single message is treated in isolation. It means that
every time the message is persisted in the message queue it has to be also committed at database level,
additionally every time the message resumes execution in a pipeline step, the required programs has to
be loaded into ABAP memory (rolled-in) and discarded later. That is, repeating these steps again and
again for every single message results in a very important overhead at the end of the day.
Using packaging, message processing is improved:
Several messages are received in
the message queue (same as
before).

When the scheduler starts to
process a message, it first checks if
there are any additional messages
that should be processed as well. If
so they are grouped and processed
together in the same package
loading the ABAP programs just
once.
After the pipeline process is
executed for all of them, they are
updated and committed to the
database as a whole.

Same thing happens for the
additional pipeline steps, always
checking what is available in the
queue before starting to process any
single message.
Compared to the previous message processing technique, this one is far more effective, since both the
context switch operations and database accesses are drastically reduced by grouping messages into
packages.
Message Packaging has several good features, one call is able to process several messages; this is
basically the principle of overhead reduction and leads to:
1 A reduction in context switch operations, that is, every time the system setups the execution
environment, it is reused instead of being discarded after message processing.
2 Commit executing is delayed until the last message is processes; finally all the changes are
committed at the same time in one single call.
From the application perspective there is no change at all; single the messages keep on being as atomic
as before at every level.
3 It means there is no impact in any monitoring transaction.
4 Furthermore, if a message inside the package generates an application error, the package is broken
down into single messages and is executed in isolation.
For additional details about the message packaging functionality, check SAP note 1.037.176.
Packaging is based on the integration engine message queues, that is, steps that read and process
messages from the queues, are able to packaged messages.
Bearing that in mind, we conclude that packaging supports asynchronous messages with qualities of
service exactly once and exactly once in order in the following steps:
5 Sender and receiver proxies,
6 Integration Engine internal steps (like receivers determination or mapping),
7 Receiver IDocs,
8 And receiver adapter-engine-based adapters.
Regarding the mapping pipeline step, consider that the mapping calls inside the step wont be
grouped further than before.
Configuration
Activation
To activate packaging, you have to start to integration engine administration transaction SXMB_ADM

Then execute Integration Engine Configuration, afterwards select the category RUNTIME and execute
Configuration. Finally, add a row for the parameter PACKAGING and set the current value to 1.Save to
activate changes.
Configuration Types
Configuration Types define the way packaging will be working for either general or specific service
interfaces. Start transaction SXMS_BCM to configure this Configuration Types. There are two built-in
configurations:
1 One is called DEFAULT and is the configuration delivered in the standard product to execute the
different steps. You can create as many configurations as you want and configure their use for each
filter as defined in SXMIF transaction.
2 The other built-in configuration is called NO, and should be used to avoid packaging. It could be
used for particular scenarios where packaging makes no sense, even when packaging is globally
switched on.

Each Configuration Type has 3 parameters:
3 The Wait Time parameter specifies how long a running process has to wait if it didnt reach the
maximum number of messages configured for this type.
4 The Number parameter specifies the maximum number of messages that should be collected prior
to start the message selection and size analysis. It corresponds to the number of messages for each
package.
5 The Maximum package Size in KB parameter describes the size if the package measured in KB.
Also take a look at SAP Note 1.058.366 to see a description of packaging in general.

Configuration Types has to be assigned to either global package configurations or specific package
configurations. Run transaction SXMS_BCONF to do this configuration:
1 For the global configuration, select Global Package Configuration in the tree on the left.
2 Specify the desired Configuration Type for the applications:
IS_IN for inbound processing
IS_OUT for outbound processing
3 If you dont perform any configuration, the DEFAULT configuration type will be automatically used.
You will also use the other applications for acknowledgement handling, ccBPM packaging and sender
proxy on application system level.
If you want to create a new Configuration Type either for a general or interface specific reason (as
simulated here), access transaction SXMS_BCM:
1 Hit New Entries

2 Create a meaningful name for your purpose. For example here, we are creating a Configuration
Type for Online Store Sales Order Processing and we name it SLS_ORD.
3 We assume (also in this hypothetical scenario) we need no aggregation, since each time a message
is processed there are a lot of messages already waiting in the queue, so we leave the Wait Time to
0 seconds.
4 Also, we think the number of average messages required to create the desired message size, will
contain about 1000 sales orders in it, consequently the Number parameter is set to 1.000.
5 Finally, we keep the default package size in 1MB.
By saving it, you will have created the new Configuration Type.
Following with our Sales Orders example,
1 Execute transaction SXMSIF to create a sender or receiver id definition (as you normally do, for
example, for IDoc packaging or for interface specific integration engine configurations):
Execute New Entries.
Name your Sender/Receiver ID
Add a Description
Specify, for example, the service and interface that should be filtered.

Save to activate the definition.
The last step in the interface specific Configuration Type creation is configuring the package creation.
1 Start again transaction SXMS_BCONF

2 Now, mark the Inbound Processing from Request Packages in IS row.
3 Double-click on Specific Package Configuration
4 Create a new entry (not shown here)
5 And finally specify that:
For the Inbound Processing from Request Packages in IS application,
And the Portal_SalesOrder Sender ID.
The SLS_ORD configuration type has to be used.
Repeat the configuration for the IS_OUT application to package also the outbound processing. And
Save to activate you interface specific configuration type.
From now on, every time a sales order appears in the integration engine, it will be packaged in
accordance with the previously defined rules.
Effect of Configuration Types in packaging creation procedure
Lets see now, how packages are built and what the relationship with the configuration types parameters
is.
Assume that the package creation has two major blocks or procedures:

1 The first procedure is used to read the queue:
When a message from the queue starts to process the first function module, first of all, it
checks if there are more messages in the queue, if so, they are also read considering the
Maximum Number for the Configuration Type.
Should the number of messages not reach the Maximum Number for the Configuration
Type in use, the process is set to sleep the number of seconds specified in the Maximum
queue Time parameter.
Afterwards this package will appear in the monitoring transaction as Timeout since this
process had to be set to sleep a certain amount of time, trying to enlarge the number of
requested messages by checking the queue for additional messages as shown in the flow.
2 The second block analyzes the set of messages previously retrieved:
It checks the message attributes for every message,
If this message should be assigned to the package, it is assigned,
Afterwards the process makes sure not to exceed the Maximum package size configured, if
not it keeps on reading the following message from the set.
After reaching the limit or retrieving all the messages, the package is set to execution.
Also, special conditions apply if the parameters are configured to 0:
3 If there is no Maximum Queue Time, in case of package sizes smaller than the desired (from the
number of messages point of view), there will be no further aggregation.
4 If Maximum Number or Maximum Package Size is set to 0, the corresponding limit is not checked
and everything is processed.
Business Component configuration for Proxies
When a message goes to a business system via a proxy, the system will automatically control if
packaging is supported by the target system by taking a look at the Software Component Versions for the
Business System described in the SLD (the System Landscape Directory), since as of SP12 packaging
will be supported.
In contrast, if the Communication Component is a Business Component (what we used to call Business
Service in XI and PI before), there is no information at SLD level. For that reason, table
SXMS_LCR_VERS was created. Using transaction SM30 (the typical customizing table maintenance
transaction) you can create an entry for the Business Component and specify the current version.

Packaging monitoring configuration
To configure packaging monitoring, you have transaction XMSPKSTATCNF, which corresponds to report
SXMS_BLK_STAT_CNF, and allows you to configure:
1 How statistical information will be collected
2 The level of aggregation desired
3 And the reorganization information.
Considerations for IDoc Packaging functionality
Regarding the IDoc packaging functionality, we can briefly say that it has become obsolete, since
message packaging generates the same benefits and has a broader scope.
The recommendation now, is to deactivate the IDoc packaging when you switch on message packaging.
You wont be loosing any functionality and you can also monitor the messages packages using the
normal IDX5 transaction.

Monitoring
To monitor packages, you have transaction XMSPKSTATMON, which corresponds to report
SXMS_BLK_STAT.
It allows you to filter the start and end times, all the queues, a pattern search or any specific queue
and the aggregation interval.
Then Hit Execute to see the report.

Also, by selecting a row and executing the Events function, it is possible to see further details, plus
additional statistical information.

Variables affecting packaging behavior
Message Size
Packaging importance get more relevant as messages are smaller
If you want to build 1Mb packages and you have 100Kb messages, you will be packing 10 messages in
each package, a 10 to 1 relationship.

But if you have half Mb messages, you will get a 2 to 1 relationship.
Packaging benefits disappear when the message size corresponds or exceeds the configured package
size.
System Load
System load is another important variable in packaging analysis.
It is necessary to clarify that system load here is not seen as the percentage of global system resources
in use (e.g. CPU), but the number of present messages that could be packed together and are waiting in
the queue. Because the system could be pretty busy in performing other non packaging relevant tasks at
that moment without any impact in packaging analysis.
So, if every time the messages are read from the queue, they never fulfill the package size or just get a
small aggregation, there will be nearly no benefits from using packaging. This could lead to the wrong
assumption that packaging generates no benefits.
When the scheduler retrieves the pending messages, the number is smaller than the required to complete
the optimum package, and then the packaging benefits are partially lost.
Example:
1 100Kb message size
2 1Mb configured package size

If we consider how runtime message size changed, in this low load situation, this should at least provide
some improvement, but not so important.
Result: not much
improvement in
package sizing,
average: 200Kb
package size - 2
messages per
package

Still analyzing the low load situation, if we are interested in increasing the benefits of packaging sacrificing
the message processing speed time, we could add some Wait Time
This will generate better packaging rates and also make a better usage of global system resources.
Elapsed time to process a single message could be slightly higher, but the system load will be lower, thus
increasing the potential throughput.
Example:
2 600Kb configured package size
3 Wait time configured accordingly

If we check how message size changed, we can see how a more appropriate configuration generates
better benefits even in low load situations.
Result: better improvement
in package sizing, average:
400Kb package size - 4
messages per package

In High load situations, that is, periods of time where many messages are waiting in the queue every time
it is read, the packaging benefits are more obvious.
System behavior will be more stable, with less messages waiting in the queue and more parallel
packages in execution.
In this example, since we have a 10 to 1 relationship, it will mean that at peak times where you have, lets
say, 500 messages more than in normal operation, you just neutralize the peak, generating just 50
additional runtime messages, very easy to handle.
The system will work more smoothly and taking fully benefits of packaging.
Example:
2 1Mb configured package size

Checking once again the result of our test, we see how the message size was shifted to a more optimal
position, some place close to 90% of maximum throughput for the installation.
It is basically done by reducing the context switches, which is reusing the loaded programs, and wrapping
database accesses into massive operations.
Results: Less messages
waiting in queues, 88%
reutilization of loaded ABAP
classes and SQL commit
operations reduction (number of
calls).
Average: 800Kb package size -
8 messages per package

Finally, we conclude that in low load system situations (considering the system load definition
introduced before) the benefits of packaging can be pretty poor, but as soon as system load starts to
grow, the throughput improvement quickly brings out.
Considering the impact in message processing speed improvement, we are going to have an impact at
high system load (considering the same hardware) due to a better usage of resources previously
explained.
Under low system load, and depending on the packaging configurations, you could expect a slightly
negative impact in the end-to-end message processing speed but no so important either.

Using the monitoring transaction previously explained, you can get important information to fine tune
packaging once in production and with real productive messages.

You can aggregate information and see:
1 An overall statistic on how packaging is working
2 The number of packages generated
3 The size of each package group
4 The number of errors generated
5 How many packages could not reach the desired package size and had to set the process to
sleep and query again.
You can clearly realize here that all the packages are very small, none of them reached the required size
and the aggregation rate is below 2 messages per package.
Final Remarks
To finalize the packaging analysis with some remarks, we can say that:
1 It is a method of grouping small messages at runtime to optimize resources usage.
2 It supports principal propagation.
3 The mapping pipeline step is also supporting packaging, but each mapping program is called
in the same way than before, already collecting mapping calls for same message and
technology.
4 The configuration of many parallel queues reduces single message runtime, but leads in
general to smaller packages.

Summary
To summarize, packaging increases throughput for small asynchronous messages on the Integration
Server, it is switched on globally (but can be configured scenario specific too), there are packaging
monitors available and finally the customer sees no changes in monitoring, error handling, etc.

Related Content
1 SAP Note 884865
2 SAP Note 1058366
3 SDN Webinars on message packaging / message bulking
Copyright
Copyright 2007 SAP AG. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP
AG. The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software
vendors.
Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries,
xSeries, zSeries, System i, System i5, System p, System p5, System x, System z, System z9, z/OS, AFP, Intelligent Miner,
WebSphere, Netfinity, Tivoli, Informix, i5/OS, POWER, POWER5, POWER5+, OpenPower and PowerPC are trademarks or
registered trademarks of IBM Corporation.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems
Incorporated in the United States and/or other countries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of
Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts
Institute of Technology.
J ava is a registered trademark of Sun Microsystems, Inc.
J avaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by
Netscape.
MaxDB is a trademark of MySQL AB, Sweden.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as
their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the
world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this
document serves informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP
Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors
or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in
the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as
constituting an additional warranty.
These materials are provided as is without a warranty of any kind, either express or implied, including but not limited to, the implied
warranties of merchantability, fitness for a particular purpose, or non-infringement.
SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that
may result from the use of these materials.
SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these
materials. SAP has no control over the information that you may access through the use of hot links contained in these materials
and does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages.
Any software coding and/or code lines/strings (Code) included in this documentation are only examples and are not intended to be
used in a productive system environment. The Code is only intended better explain and visualize the syntax and phrasing rules of
certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for
errors or damages caused by the usage of the Code, except if such damages were caused by SAP intentionally or grossly
negligent.

Mastering Sap Netweaver Pi Throughput With Message Packaging

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Mastering Sap Netweaver Pi Throughput With Message Packaging

Enviado por

Direitos autorais:

Formatos disponíveis

Mastering SAP NetWeaver PI Throughput with Message Packaging

SAP DEVELOPER NETWORK | sdn.sap.com BUSINESS PROCESS EXPERT COMMUNITY |

Você também pode gostar