Longer Version

Reliable Messaging for

Grids and Web Services

Geoffrey Fox, Shrideep Pallickara, Damodar Yemme, Hasan Bulut and Sima Patel

(gcf, spallick, dyemme, hbulut and skpatel)@indiana.edu Community Grids Lab

Message-Based Reliability

n

Web Services exchange messages and interact

with resources that produce and absorb

messages

n

Action and state (if exists) of a service defined by

messages

n

Our approach to Reliability is based on a building

a messaging infrastructure that is intrinsically

reliable

q WS-RM and WS-Reliability for web services

Database

S S

S

S SS SS SS SS SS SS SS SS

F S F S F S F S F S F S F S F

S SF

F S F S F S F S F S F S F S F S F S F S F

S Portal

F S O S O S O S O S O S O S O S O S O S O S O S O S MD MD MD MD MD MD MD MD MD

MetaData Filter Service Sensor Service Other Service Anothe Grid

Raw Data  Data  Information  Knowledge  Wisdom

Decisions S S S S Anothe Service Anothe Service S S Anothe

Grid S S

Applications of our Technology

n

Point-to-point generic linkage of services using

WSRM with messages saved in databases as

required in specification

n

Scalable Management Architecture to support

dynamic collections of entities

q Applied first to the brokers used in distributed

messaging of NaradaBrokering

n

Management of the streams of data from

sensors and web-cams

q Allow time replay and annotation based on

WSRM

n WSRM describes a protocol that facilitates the reliable

delivery of messages between two web service

endpoints in the presence of component, system or network failures.

n WSRM facilitates the reliable delivery of messages

from the source (or originator) of messages to the sink (or destination) of messages.

n The delivery (and ordering) guarantees are valid over

a group of messages, which is referred to as a

Creation of Sequences

n In WSRM prior to ensuring reliable delivery of

messages between the endpoints, the source initiates an exchange with the sink pertaining to the creation of a Sequence.

n This Sequence is intended to facilitate the grouping of

a set of related messages.

n This Sequence is identified by an identifier, typically a

UUID. Other information associated with the Sequence include information regarding ─

q The source and the sink

q Policy information related to protocol constants such as

acknowledgement and retransmission intervals.

WSRM Sequences

n In WSRM all messages issued by a source exist within

the context of a Sequence that was established prior to communications.

n Once a source has determined that all messages

within a Sequence have been received at the sink, the source initiates an exchange to terminate this

sequence.

n The specification allows for a maximum of 264 -1

messages within a Sequence.

n The specification places no limits on the number of

Sequences between a specific source and sink.

q However, it is expected that at any given time there is NO

Publishing Messages in WSRM

n

Every message from the source contains two

pieces of information ─

q The Sequence that this message is a part of and

q A monotonically increasing Message Number within

this Sequence.

n

These Message Numbers enable the tracking

of problems, if any, in the intended message

delivery at a sink.

q Message Numbers enable the determination of out

Issuing Acknowledgments

n In WSRM a sink is expected to issue

acknowledgements back to the source upon receipt of messages.

n This acknowledgement contains information about

q The Sequence and

q The Message Numbers within this Sequence.

n An acknowledgement must be issued only after a

certain time ─ the acknowledgement interval ─ has elapsed since the receipt of the first unacknowledged message.

n This acknowledgement may cover a single message

Processing Acknowledgments

n

Upon receipt of this acknowledgement a

source can determine which messages might

have been lost in transit and proceed to

retransmit the

missed

messages.

n

Thus if a sink has acknowledged the receipt of

messages 1 ─ 10 and 13 ─ 18.

q The source can conclude that messages with

Retransmissions and Error Corrections

retransmission of a message for which that an

acknowledgement has not been received within a

specified time ─ the retransmission interval ─ after which it was issued.

n In WSRM error corrections can also be initiated at the

sink; this is done through the use of negative acknowledgements.

q Negative acknowledgments identify the message

numbers that have not been received at a sink.

n Message Numbers increase monotonically. If Message

Numbers 1,2,3,4 and 8 within a specific Sequence have been received at a sink.

Notification of Errors

n

WSRM provides for notification of errors in

processing between the endpoints involved in

reliable delivery.

q These are routed back as SOAP Faults.

n

The range of errors can vary from an inability to

decipher a message’s content to complex

errors pertaining to violations in implied

agreements between the interacting source

and sink.

n

All errors are reported as faults with the

appropriate

attribute, and

Comments on WSRM Implementation

Infrastructure Institute

n We built WS-Eventing that is available in OMII 2.3.3

http://www.omii.ac.uk/news/newsdetail.jsp?id=25 in FINS Project

n WS-RM is currently being tested in OMII container

(FIRMS Project) and is expected to be finished in a month (but not released by OMII until they do further tests)

n WS-RM and WS-Eventing use SOAP handlers that

61024 4573 645 74. 730. 1220.

Parse an EnvelopeDocument to retrieve Wsa Message Info Headers

13880 2123 344 35. 350. 538.

Create an Envelope targeted to a specific WSA EPR with most WSA message information headers

7184 1276 267 21. 200. 397.

Create an Envelope targeted to a specific WSA EPR

2648 705 112 9.9 97. 151.

Create a WS-Addressing EPR

(Contains WSA ReferenceProperties)

2072 465 71 6.0 58. 87.6

Create a WS-Addressing EPR (Contains just a URL address)

34424 2802 325 60. 590. 828.

Convert SOAPMessage to EnvelopeDocument 60816 5350 1722 94. 910. 2630.

Convert an EnvelopeDocument to a SOAPMessage 1824 1183 34 19. 188. 117

Create an Axis based SOAPMessage

2192 424 108 5.0 49. 127

Create an XMLBeans based Envelope Document Memory Use (Bytes) Max Val Min Val Stand Error Stand Dev Mean Operation

18096 1619 347 30. 294.69 9 520. CreateWsrmFault 2072 380 19 3.6 36.203 24.7 CreateTerminateSequence 20624 1514 335 25. 250. 516.

Create a WSRM

SequenceAcknowledgement based on a set of message numbers

464 14 12 0.05 0.49 12.7

Add a WsrmSequenceDocument to an existing envelope. (Contains sequence identifier and message number)

2424 75 42 0.48 4.7 45. CreateWsrmSequenceDocument 18160 1174 224 23. 226. 335. CreateWsrmSequenceResponse 16392 1568 229 26. 261 352. CreateWsrmSequenceRequest Memory Util (Bytes) Max Val Min Val Stand Error Stand Dev Mean Operation

NaradaBrokering

NaradaBrokering 2003-2006

n Messaging infrastructure for collaboration, peer-to-peer and Grids

Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop)

n Order-preserving message transport with QoS and security profiles n Support for different underlying transport such as TCP, UDP,

Multicast, RTP

n SOAP message support and WS-Eventing, WS-RM and WS-Reliability.

• WS-Notification when specification agreed

n Active replay support: Pause and Replay live streams.

n Stream Linkage: can link permanently multiple streams – using in

annotation of real-time video streams

n Replicated storage support for fault tolerance and resiliency to storage

failures.

n Management: HPSearch Scripting Interface to streams and brokers

(uses WS-Management)

n Broker Topics and Message Discovery: Locate appropriate n Integration with Axis2 Web Service Container (?)

Management of services

n

We prefer to build Grids (collections of web

services) that use distributed publish-subscribe

message-oriented middleware to transport all

messages.

q This is NaradaBrokering and one can bind SOAP to

NB transport (very different from

WS-Notification/Eventing) building a handler for this

n

NB will guarantee message delivery and its

distributed nature has implicit reliability

q However we need to maximize reliability of this

Management Architecture

Multiple Distribute Manager Instances

Features of the Managee Service

n

The distributed managers use

Features of the Manager Service

n

WS-Management used for communicating

between Managers and Managees

n

Managers implement policy and user instructions

e - Annotation Playe

r

Archived stream playe

r

Annotatio

nplaye / WB r

Archieved stream

list

Real time stream

list

e -Annotation Whiteboar

d

Real time stream playe

r

Archived Real Time Real Tim

Stream List Stream List Player

e-Annotation Archived Stream Annotated e-Annotation

Generic Recording and Replay

Framework

n A generic framework for recording and replay of any

type of streaming event or data.

n Active replay of streams: Real-time (live) streams

can be replayed, paused and rewound while streams are being recorded.

q Fast forward is available for the duration of the recorded

stream.

n Note streams are collections of events and events

are essentially messages

n Rewind is same as undo (as in Office)

n Replay is same as redo i.e. re-apply sequence of

Generic Recording and Replay Framework

construct a session.

q A collaboration session can be recorded and replayed within

this framework. Examples;

n Anabas – Uses JMS events to transport data such as

whiteboard, shared display, audio, etc.

n GlobalMMCS – Uses NaradaBrokering RTP Events to

transport audio and video data.

n Streams can be added/removed to/from session

dynamically while the session is being recorded.

n Maintains metadata information for recorded sessions

and their streams.

q Dynamic metadata stored in high performance light weight

Uniform Event Type For Generic Framework

n Received events are wrapped inside NaradaBrokering

native events (NBEvent) with some event specific information.

q Received event is placed to the payload of the NBEvent as is

to preserve original data and related information.

q NBEvent also contains timestamp information to timespace

original events during replay and event type to initiate appropriate player for that event type.

n Events and related metadata are stored in database

Session Recorders

n Session recorder includes topic recorders that

subscribe to each topic defined in that session.

n Topic recorders are like subscribing clients receiving

the streaming events. Topic recorders are specialized for event types. i.e. JMS events need JMS topic

recorder to receive those type of events.

n Event types for those streams are already known from

Session Recorders II

n Figure on the left

depicts a scenario for recording of JMS event or NaradaBrokering

Session Recorders II

n Session recorders construct NBEvents with necessary

information, such as timestamps, in NBEvent’s header.

n Along with each constructed NBEvent, a companion

event is also constructed for reliable delivery mechanism.

n Session recorders support initiating/stopping a record,

Session Players

n The primary purpose of session player is to simulate

clients in the original session.

n To achieve this;

q Each recorded topic (or stream) is mapped to a new topic

and events of the same original topic are released to the mapped topic.

q While releasing the events, timespacing between events are

preserved.

Session Players II

n Figure on the left

depicts a scenario for playing a session with multiple streams (NB RTPEvent or JMS

Session Players II

n Recording of live streams are available for replay as

soon as they are stored to the reliable storage.

n Session players support replay, pause, rewind and

Time Differential Service (TDS)

n Replay of events rely on one critical service: Time differential

service.

n Each replay session has one dedicated TDS to achieve replay,

pause, rewinding and fast forwarding of the streams in the session in one operation.

n Achieves synchronization of multiple streams in the same

session by maintaining a shared buffer for those streams.

n Maintains the timespace between the replay events equal to the

timespace between the original received events .

n Resolution of this timespacing is one millisecond; events can be

Time Differential Service II

n TDS can be maintained on robust node

(NaradaBrokering node that provides stable storage) or on client side.

n Maintaining TDS on robust node increases load of

robust node because of its buffer and threads responsible for releasing the events.

q It is required to support current streaming clients such as JMS

clients, RTPEvent clients or other streaming clients that does not maintain an internal buffer

n Maintaining TDS on client side allows more session

Scalability of Generic Record/Replay

Framework

n Framework is scalable in the sense that it can support

many clients for the same replay session. For example,

q When a collaboration session (i.e. Anabas or GlobalMMCS

session) is replayed, clients only need to subscribe to the replay topics. The number of clients depends on the number of clients supported by the broker, not by the robust node.

n This framework can easily be extended to many other