ETags in depth

A review of ETags use within the SAP Mobile Platform

R IG T e c h : P a u l T o d d & M a d h u s u d h a n a n P e ru m a ls w a m y

TABLE OF CONTENTS

INTRODUCTION ............................................................................................................................................... 3
MANAGING CONCURRENCY ON THE INTERNET ........................................................................................ 4
Concurrency Control in HTTP ........................................................................................................................ 4
Detailed example ............................................................................................................................................. 4
ETAGS IN HTTP ............................................................................................................................................... 5
Headers ............................................................................................................................................................ 5
Syntax ............................................................................................................................................................... 5
Important notes: .............................................................................................................................................. 5
Examples: ........................................................................................................................................................ 5
HTTP operations .............................................................................................................................................. 5
ETAGS IN ODATA ............................................................................................................................................ 6
ETAGS IN SAP NETWEAVER GATEWAY ...................................................................................................... 7
Updated ETag support .................................................................................................................................... 7
Extension points .............................................................................................................................................. 8
ETAG SUPPORT IN SAP MOBILITY PLATFORM 3.0 .................................................................................... 8
Offline OData .................................................................................................................................................... 8
SUMMARY ........................................................................................................................................................ 9
REFERENCES ................................................................................................................................................ 10

2
ETAGS IN DEPTH

INTRODUCTION

This paper will explore how what ETags are and how ETags are handled within the SAP Mobile Platform 3.0
(SMP 3.0) from the device all the way through to the backend data warehouse/SAP R3 system.

The understanding of ETags is fundamental to an overall understanding of how transactions are expected to
work within applications, OData, SAP products and of course within business processes. The transactions
that ETags emulate have a critical significance in business and within SAP systems. Traditionally
transactions are modelled in the database layer since this is the basis for the rules of relational databases
and so are well understood and reliable. SAP applications however model transactions in the application
layer rather than the database layer to afford higher scalability and flexibility so it is vital that they be
understood and used correctly.

Within this document will be covered in detail why, how and where ETags are used and then an examination
of how ETags are implemented across a range of SAP products starting with SAP Netweaver Gateway
(NWGW) and moving out to the devices and platforms supported by the SAP Mobility Platform (SMP). This
will give a through and detailed view of transaction life cycles across multiple workflow steps and platforms.

3
ETAGS IN DEPTH

MANAGING CONCURRENCY ON THE INTERNET

Concurrent editing of resources leads to a lost update problem. Two users editing same resource at same
time will lose the changes for one of the editors. This scenario can be managed in one of two ways:

Pessimistic concurrency control
In this scenario the application blocks until all the operations within the transaction have can be successfully
completed. This will normally involve a performance reduction, as the application(s) using the resource will
all have to queue until the transaction can ensure the operations can successfully be applied. This scenario
also has a much higher risk of deadlocks as the transaction may be waiting for a resource that is locked by
another operation. It is pessimistic since it assumes that concurrent writes will occur and actively prevents it
by aggressively acquiring the resources.

Optimistic concurrency control
Here the operations in the transaction are all attempted and when one of them fails the entire set of
operations are rolled back. This is normally the preferred solution due to its higher transaction throughput
and less possibility of deadlocks. It is optimistic since it assumes the likelihood of a concurrent write is rare
(though it does happen) and so all the operations in the transaction are attempted and rolled back if one
fails.

Semi-optimistic concurrency control
This is a blend of the pessimistic and optimistic concurrency controls so that it is possible to get higher
throughput by managing the resource locks individually. This is normally a highly curated solution and for
very specific use cases when scenarios are identified that are environmentally specific.

Concurrency Control in HTTP

Based on the above explanation it should be evident that HTTP can only use optimistic concurrency control
since each HTTP transaction is stateless.

For example: Lost update problem with concurrency control is illustrated below:

1. User_1 accesses a document using HTTP GET and starts editing it.
2. User_2 also accesses the same document again using HTTP GET and starts editing it.
3. User_1 saves their edits using HTTP PUT.
4. User_2 saves their edits but as user didn't see User_1's edits,
5. User_1 then loses their edits as User_2 overwrites them.

Detailed example

User_1 and User_2 both issue GET request to the server for the document X with both receiving same entity
tag in the response field.
User_1 edits and does not save changes (i.e.) does not send a PUT request to the server.
User_2 edits and sends PUT request with If-Match field value as Etag value. Since the Etag value remains
same as the server, status code 204 is received, then update of the content is performed and new entity tag
is received through response. So old etag is deleted and new one is stored by User_2 for future requests.
At the same time User_1 sends PUT request with If-Match field value as Etag value. Since the Etag value is
changed, status code 428 with preconditions failed is received as response.
To get the updated version of the document X, User_1 send PUT request with If-None-Match field value as
Etag value. So now the existing version of User_1's document is overridden and new etag which User_2 has
will be received in response with the resource.
Now User_1 can change the document and send PUT request with If-Match field value as Etag value. Now
the changes will be reflected in the server.

Therefore by using entity tag along with If-Match and If-None-Match fields, concurrency is managed in
HTTP transaction.

4
ETAGS IN DEPTH

ETAGS IN HTTP

Entity tags or ETags are part of HTTP. They are used to provide HTTP for web cache validation. It is an
HTTP response header returned by an HTTP/1.1 compliant web server used to determine change in content
of a resource at a given URL. The ETag response-header field provides the current value of the entity tag for
the requested variant.

This helps resolve the Lost update problem since using ETags along with the if-match precondition stops
updates from multiple callers overwriting a backend entity. Optimized concurrency is also achieved using If-
Match, If-None-Match with ETags as explained below.

Headers
The headers used with entity tags are described with
ETag
If-Match,
If-None-Match


Syntax
The syntax of the ETag header is defined as follows:

ETag = "ETag" ":" entity-tag CRLF;
entity-tag = [weak] opaque-tag ;
weak = "W/" ;

opaque-tag = <ASCII encoded value of entityProperty rules >

Important notes:
It is an important distinction to note that ETags come in two flavours, weak and strong tags. Weak ETags
are semantically identical but not binary identical. Strong ETags on the other hand are binary identical and
thus semantically identical by definition. For example a weak ETag could be considered the last updated
time or the version of a document. A strong ETag is considered if the entire representation of the entity is
considered binary identical so all fields are compared and typically a hash of the entity is used as the strong
ETag. Weak ETags are prefixed with the W/ to indicate they are weak. Strong ETags do not have the
leading prefix weak ETags do.

The actual contents of the ETags, be it a strong or a weak ETag are expected to be opaque and have
meaning only on the server the ETag was retrieved from. The caller should not modify the value or expect it
to have meaning.

Examples:

ETag: "xyzzy"
ETag: W/"xyzzy"
ETag: ""


HTTP operations
In an HTTP transaction we are only interested in the retrieval operation to obtain the ETag and the three
modifying operations to see their effect on concurrency. ETags are supplied to the caller if the backend
webserver support ETags. It should never be assumed that all servers support ETags though most new
implementations will do to improve performance and scaling.

In HTTP, ETags are implemented in the HTTP headers. To obtain an ETag value, the entity is read from the
backend as part of the GET request and returned in the headers in the response. The caller does not need
to do anything additional to obtain the ETag; if the server supplies it then the server supports HTTP ETags.
In the PUT, MERGE and DELETE requests the If-Match header is used and in the GET requests the If-

5
ETAGS IN DEPTH

None-Match header is used to make the request conditional. The If-Match ETag header is used to decide
how the update will be applied to the backend and are used along with ETag value to define the concurrency
model. If-None-Match is used to short circuit sending back a full set of data to the client if the client already
has the data cached.

Modification requests in HTTP consist of PUT operations, MERGE operations and of course DELETE
operations. For these requests, If-Match field values are used to confirm whether the request should be
applied on the backend or not. When making the request the client includes the If-Match header together
with the ETag value obtained from a prior GET request. If the client-supplied entity matches the entity in the
server then status code is 204 with an empty HTTP response is returned and the entity is updated on the
backend. This implies that the modification operation request can be performed, as there have been no
changes to the data since the data was retrieved. If the ETag for the entity does not match the ETag
supplied, then status code with 428 (i.e. precondition required) is sent in the HTTP response and the caller
needs to handle this as an error.

The general syntax is:

If-Match: entity-tag

An asterisk (*) matches any entity, and the transaction continues only if the entity exists. Following are
possible examples:

If-Match: "xyzzy"
If-Match: *

For GET requests, If-None-Match field values are used to confirm whether the request should be performed
or not. On the server the client will look at the ETag and compare it with the ETag it has for the entity and if
the two match then the server will respond with the status code of 304 (Resource Not Modified) informing the
caller to use the cached entity on the caller rather than having the caller retrieve a new copy of the
document. If the client supplied ETag does not match the ETag in server then status code is 200 with data in
http response.

The general syntax is:

If-None-Match: entity-tag

An asterisk (*) matches any entity, and the transaction continues only if the entity does not exist. Following
are the possible examples:

If-None-Match: "xyzzy"
If-None-Match: *

It should be clear that If-Match calls are applied during updates and if-none-match is applied during reads.
The If-Match compares what was read with what is stored on the server and the If-None-Match compares
what is cached on the client with what is on the server. The two operations apply to two different directions
though both are incredibly useful for concurrency and optimization management.

ETAGS IN ODATA

The handling of ETags within OData is discussed within the context of the second version of the framework
in section 3.1. It is important to note that support for ETags is optional. The server can, at their discretion
decide whether to implement or not implement tag support. If it is not supported the service consumer will
have the responsibility to setup their down conflict resolution strategy by for example checking a version field
or checking a last modified date to ensure that the consumer does not overwrite a record that has been
updated since the record was last read.

6
ETAGS IN DEPTH

OData uses ETags as a form of optimistic concurrency control that means the resource existing on a URI
and the resource to be updated have their tag checked and if they differ then an error is raised. The actual
contents of the ETag should be an indeterminate blob. Comparisons between ETags is thus not allowed
since there is no formal relationship between one ETag and another and OData providers are free to use
their own representation for each ETag in an entity or collection implementation.

It is important to note that there is a slight semantic difference in OData when retrieving entities vs retrieving
an entity. In the case of an entity the ETag will be returned via the HTTP header whilst when requesting a
collection which will have 1 or more entities, each entities' ETag will be embedded in the entity metadata.

Finally when a client requests a modification, the client must include the ETag retrieved from a prior GET
operation and set the If-Match header. This is to ensure that the version being worked on in client is the
same as the representation on the backend server. If the versions differ then this is a logical mismatch and
the client needs to decide how to handle the fact that someone on the server has updated the record whilst
the client was updating the record. If the client fails to provide an ETag in a modification request the server
will respond with HTTP error code 412. If the client application is not interested in transactional management
then the "Match-If" header can set to the "*" value to indicate any ETag is acceptable but the entity must
exist on the backend. This is effectively a last person wins scenario that may or may not be acceptable,
depending on the use case.

ETAGS IN SAP NETWEAVER GATEWAY

Fundamental to business processes are the concept of the transaction. Without which many business
workflows would grind to a halt. For example the typical debit/credit transaction in a ledger where both the
credit and the debit need to succeed in order for the whole transaction to be completed otherwise neither
occurs.

For this section the ETag support with SAP Netweaver Gateway will be explored and in particular how this is
handled in OData. ETag support has been marked as fundamental to OData since without it is hard to
envisage updates that do not need some kind of conflict detection without the addition of a proprietary API.
Since OData is based on HTTP and HTTP already has support for conflict resolution strategies through the
ETag specification in section 14.19 of the HTTP RFC reference document.

SAP Netweaver gateway has support the use of ETags from the start however it was inefficient and
introduced additional complexity through its generic nature. This was because the decision as to whether the
object being updated was at the same version as the one currently stored on the backend was done at the
Netweaver Gateway hub and relied on a read to the backend ABAP server then a further update operation.

Updated ETag support
From SAP Netweaver Gateway 2.0 SP9 additional functionality was added to the SAP Netweaver Gateway
to support ETag handling by providing an API which moved the decision about the concurrency down to the
backend application. This means the backend application can replace the generic (Hub) ETag handling with
its own implementation allowing for improved performance and proper record locking semantics since the
application now has a context in which to lock.

This was implemented in ABAP in the Netweaver Gateway classes as a hub API to conditionally override the
default legacy ETag handling and also in a Backend enablement extension API that developers need to
explicitly implement in order to perform custom ETag handling on a per entity basis. The effect of extending
the backend classes is to provide a custom code solution so that the way ETags are handled is independent
of the implementation. This means for example that a property can be chosen from the entity such as the
modified date and that is used we the key for ETag handling which is generally the easiest solution however
there is nothing stopping partners implementing their own ETag management underneath in ABAP to use
additional state tables or point to different underlying databases. Whilst the actual implementation details for
extending the SAP Netweaver Gateway ABAP classes to support custom ETag handling are outside the
scope of this paper it is worth looking at the following classes and methods

7
ETAGS IN DEPTH

Extension points
IWFND component (Gateway HUB)
o Class /IWFND/CL_SODATA_ROOT_HANDLER method HANDLE_CONDITIONS
o Class /IWFND/CL_SODATA_PROCESSOR Method INIT_REQUEST
o Interface /IWFND//IF_MGW_CORE_TYPES
IWBEP component (Gateway BEP)
o Class /IWBEP/CL_MGW_ABS_DATA
Method /IWBEP/IF_MGW_CORE_SRV_RUNTIME~UPDATE_ENTITY
Method /IWBEP/IF_MGW_CORE_SRV_RUNTIME~DELETE_ENTITY
o Interface /IWBEP/IF_MGW_APPL_TYPES
o Interface /IWBEP/IF_MGW_APPL_SRV_RUNTIME
Method GET_IS_CONDITIONAL_IMPLEMENTED
o Interface /IWBEP/IF_MGW_REQ_ENTITY_U
Method GET_CONDITIONAL_INFO
o Interface /IWBEP/IF_MGW_REQ_ENTITY_D
Method GET_CONDITIONAL_INFO

ETAG SUPPORT IN SAP MOBILITY PLATFORM 3.0

Many partners using SMP before 3.0 will be familiar with the Mobile Business Object architecture or MBO
that was used to build a transactional scalable queued architecture for mobile platforms. Each MBO had a
conflict resolution strategy that could be set to detect and handle conflicts. This feature allowed for the
reliable automation of business workflows from the device to the backend server.

Since OData and SMP 3.0 superseded the proprietary MBO interface a new mechanism to decide conflict
resolution was required. Since there was already a strategy defined by the HTTP Standard (RFC 2616) in
ETags this was chosen as the default strategy.

If there is a conflict the following rules will be applied:

MBO OData
Client wins Match-if rule with ETag set to *
Server wins Match-if and ETag is the same
NWGW server implements custom ETag handling and/or client implements custom handling for c
User chooses
will be able to report ETag constraint violations, allowing the client code to gracefully handle the f

There is not a lot to say about ETags and the generic SMP platform since the SMP platform typically does
not cache state in SMP 3.0. Of course this was not the case in the 2.x product that had the CacheDB in the
SMP server to handle distribution of the objects and reconcile conflicts in the backend with the client.

However in SMP 3.0 the support for ETags is transparent. With no state held in the SMP server, ETags are
passed through the server transparently to the backend.

Offline OData

SAP Mobility Platform 3.0 Service Pack 5 introduced the Offline Storage API to enable developers to create
mobile applications that can handle the offline storage of enterprise data. In this type of scenario where the
application state has moved from always online to frequently connected, the chance of a conflict by having
multiple instances of the application write to the same OData service entity increases substantially. It is thus
important a good understanding of the way that the SMP offline data store works is well understood. This is
particularly visible around queued updates and deletes that can happen whilst the device goes on and out of
connectivity with the server.

The normal scenario that should be coded for is to assume the server supports ETag resolution. If does not
matter if this is a pre of post NWGW 2.0 SP9 server since the intrinsic handling of ETags is common to both.

8
ETAGS IN DEPTH

The application starts by reading an entity from the offline store and this will have an ETag associated with it.
The offline store may have to go to the server to retrieve the entity or it may use a cached copy.

This entity is then modified by another application on the server so the copy in the offline store does not
match the copy on the server. The ETag for this entity on the server now differs from the ETag in the entity
on the device in the offline store.

The application then issues a refresh call to the offline store that causes the entity to be retrieved to the
remote server and update causing the ETag in the offline store to be updated at the same time.

The application now modifies the entity and called scheduleRequest to update the entity in the offline store.
This will now fail since the entity's ETags differ and the request will not be added to the offline store and will
not be replicated. It will be up to the application to get a new copy of the entity from the offline store, apply
the changes or inform the user and then call scheduleRequest again to attempt to update the entity. Since
the entity is already updated this requires no round trips to the server so is fast.

If however the refresh call is omitted then the offline line store has no way of knowing that the copy on the
server is incorrect so the modified entity is queued and then during a flush call later on, the conflict would be
detected and the callback method offlineStoreRequestFailed will be invoked. This is a less optimal solution
as there may be many objects queued all of whom may have been sitting in the queue for a while and each
will require mediation by the application and possible user intervention to resolve the conflict. Also there is
the possibility that there are other dependent entities that will fail to update or will update incorrectly so care
needs to be taken if this route is chosen.

Of course if the NWGW gateway and hub are under the control of the development group then the whole
end to end offline scenario could be optimized to for example never have to deal with a conflict through good
design or though managing a custom conflict resolution strategy.

SUMMARY
The whole area around transaction management in OData implementations is very poorly documented. The
architecture, whilst industry standard that was chosen for concurrency management is not very well
articulated and since this is fundamental to business processes and workflows in general it is important that
the behaviour be clearly documented and well explained especially where there are complex scenarios such
as non-heterogeneous environments or where scalability is or potentially is an issue. In these cases it is vital
that the entire end-to-end story for transactions in business processes is well understood.

This paper has covered the flow of data from an offline application using the SMP 3.0 platform through the
SMP server to the backend Netweaver Gateway Hub to the backend Netweaver Gateway server and on to
the SAP data warehouse.

9
ETAGS IN DEPTH

REFERENCES
http://en.wikipedia.org/wiki/Concurrency_control
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
http://www.odata.org/documentation/odata-version-2-0/operations/
http://help.sap.com/saphelp_gateway20sp09/helpdata/en/76/278e53f9bd3e27e10000000a44538d/content.ht
m
http://scn.sap.com/community/developer-center/mobility-platform/blog/2014/11/19/12-smp3-odata-sdk-for-ios
http://scn.sap.com/docs/DOC-58063
http://help.sap.com/saphelp_smp306sdk/helpdata/en/e5/033aa25c67486ab06505f565bf6f9f/content.htm
https://msdn.microsoft.com/en-us/library/dd541486.aspx
http://en.wikipedia.org/wiki/HTTP_ETag
http://www.w3.org/1999/04/Editing/

10

www.sap.com

2015 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form
or for any purpose without the express permission of SAP SE or an SAP
affiliate company.
SAP and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP SE (or an
SAP affiliate company) in Germany and other countries. Please see
http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for
additional trademark information and notices. Some software products
marketed by SAP SE and its distributors contain proprietary software
components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for
informational purposes only, without representation or warranty of any kind,
and SAP SE or its affiliated companies shall not be liable for errors or
omissions with respect to the materials. The only warranties for SAP SE or
SAP affiliate company products and services are those that are set forth in
the express warranty statements accompanying such products and services,
if any. Nothing herein should be construed as constituting an additional
warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue
any course of business outlined in this document or any related presentation,
or to develop or release any functionality mentioned therein. This document,
or any related presentation, and SAP SEs or its affiliated companies

strategy and possible future developments, products, and/or platform
directions and functionality are all subject to change and may be changed by


SAP SE or its affiliated companies at any time for any reason without notice.
The information in this document is not a commitment, promise, or legal
obligation to deliver any material, code, or functionality. All forward-looking
statements are subject to various risks and uncertainties that could cause

actual results to differ materially from expectations. Readers are cautioned
not to place undue reliance on these forward-looking statements, which
speak only as of their dates, and they should not be relied upon in making
purchasing decisions.