Data and Metadata Architectures in a Robust Semantic Grid

Data and Metadata

Architectures in a Robust

Semantic Grid

Chinese Academy of Sciences July 28 2006

Geoffrey Fox

Computer Science, Informatics, Physics Pervasive Technology Laboratories Indiana University Bloomington IN 47401

http://grids.ucs.indiana.edu/ptliupages/presentations/

Status of Grids and Standards I

n

It is interesting to examine Grid architectures both to

see how to build great new systems but also to look at

linking Grids together and making them (or parts of

them) interoperable

n

There is agreement that one should use Web Services

with WSDL and SOAP and not so much agreement

after that

• But use non SOAP transport like GridFTP

n

Can divide Service areas into

• General Infrastructure

• Compute Grids

• Data and Information Grids

• Other …..

Status of Grids and Standards II

n General Infrastructure covers area where Industry, OASIS and

W3C are building the pervasive Web service environment

n There are important areas of debate and vigorous technical

evolution but these are within confined areas

• Relatively clear how to adapt between different choices

n Examples of areas of some contoversy

• Security critical but commercial, academic institution and Grid project

solutions still evolving

• Workflow has many choices and BPEL not clearly consensus standard;

differences between control and data flow

• Architecture of Service discovery understood but skepticism that UDDI

appropriate; it keeps getting improved

• In Management, Notification and Reliable Messaging, there are multiple

standards but rather trivial to map between them

• WSRF symbolizes disagreements in state (which is roughly meta-data

area) but roughly this is question as to whether metadata in message or context service or hidden in application

• Data transport model unclear: GridFTP v. BitTorrent v. Fast XML

The Ten areas covered by the 60 core WS-* Specifications

WSRP (Remote Portlets)

10: Portals and User Interfaces

WS-Policy, WS-Agreement

9: Policy and Agreements

WSDM, WS-Management, WS-Transfer

8: Management

WSRF, WS-MetadataExchange, WS-Context

7: System Metadata and State

UDDI, WS-Discovery

6: Service Discovery

WS-Security, WS-Trust, WS-Federation, SAML, WS-SecureConversation

5: Security

BPEL, WS-Choreography, WS-Coordination

4: Workflow and Transactions

WS-Notification, WS-Eventing (Publish-Subscribe)

3: Notification

WS-Addressing, WS-MessageDelivery; Reliable Messaging WSRM; Efficient Messaging MOTM

2: Service Internet

XML, WSDL, SOAP

1: Core Service Model

Examples WS-* Specification Area

Activities in Open Grid Forum Working Groups

Authorization, P2P and Firewall Issues, Trusted Computing

7: Security

Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model

6: Management

Network measurements, Role of IPv6 and high performance networking, Data transport

5: Infrastructure

Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management

4: Data

Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling

3: Compute

Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval,

2: Applications

High Level Resource/Service Naming (level 2 of slide 6), Integrated Grid Architecture

1: Architecture

GS-* and OGSA Standards Activities GGF Area

The NCES/WS-*/GS-* Features/Service Areas I

Portlets JSR168, NCES Capability Interfaces NCES7

WS10

FS11: Portals and Users

ECS WS9 FS10: Policy CIM NCES1 GS6 WS8 FS9: Management Globus MDS

Semantic Grid, WS-Context WS7

FS8: System Metadata & State

UDDI and extensions NCES4

WS6

FS7: Discovery

Grid-Shib, Permis Liberty Alliance ... NCES2 GS7 WS5 FS6: Security Grid Programming NCES5 WS4 FS5: Workflow

JMS, MQSeries, WS-Eventing, Notification NCES3

WS3

FS4: Notification

Core Infrastructure including reliability, publish-subscribe messaging cf. FS13C

NCES3 WS2

FS3: Service Internet, Messaging

B: Core Services (Mainly Service Infrastructure and W3C/OASIS focus)

Strategy for legacy subsystems: modular architecture

FS2: Grid of Grids

Core Service Architecture, Build Grids on Web Services. Industry best practice

WS1

FS1: Use SOA: Service Oriented Arch. A: Broad Principles

Comments NCES

(DoD)

GS-*

WS-* Service or Feature

Grids of Grids of Simple Services

• Grids are managed collections of one or more services • A simple service is the smallest Grid

• Services and Grids are linked by messages

• Internally to service, functionalities are linked by methods

• Link serices via methods  messages  streams

• We are familiar with method-linked hierarch

Lines of Code  Methods  Objects  Programs  Packages

Overlay and Compose

Grids of Grids

Methods Services Component Grids

CPUs Clusters Compute Resource Grids MPPs

Databases _DatabasesFederated

Sensor Sensor Nets

Data

Mediation and Transformation in a

Grid of Grids and Simple Service

Po rt Po rt Port Port Intern a Interf aces

Grid or Service

Po rt Po rt Port Port Intern a Interf aces

Grid or Service

Po rt Po rt Port Port Intern a Interf aces

Grid or Service

Mediation and Transformation Services Distributed Brokers between distributed ports External facing Interfaces Mediation and Transformation Services Listen, Queue Transform, Send Mediation and Transformation Services

1-10 ms Overhead Use “OGSA” to Federate?

The NCES/WS-*/GS-* Features/Service Areas II

NCES3,8

WS 2,3 GS4 GridFTP or WS Interface to non SOAP transport FS13C: Data Transport

B: Core Services (Mainly Higher level and OGF focus)

VOSpace for IVOA, JBI for DoD, WFS for OGC Federation at this layer major research area

NCOW Data Strategy

NCES8

GS4 FS14B: Information, Knowledge,

Wisdom part of D(ata)IKW

OGC SensorML FS13B: Data as Sensors and

Instruments

Current work only addresses scheduling “batch jobs”. Need networks and services

GS3

FS18: Scheduling and matching of Services and Resources

XGSP, Shared Web Service ports NCES6

GS7

FS17: Collaboration and Virtual Organizations

Ad-hoc networks; Network Monitoring GS5

FS16: Resources and Infrastructure

Standalone Services Proxies for jobs NCES9

GS2

FS15: Applications and User Services

Major Grid effort for job status etc. GS4

FS14A: Information as Monitoring

Distributed Files, OGSA-DA Managed Data is FS14B

NCES8

GS4 FS13A: Data as Repositories: Files

and Databases

Job Management major Grid focus GS3 FS12: Computing Comments NCES GS-* WS-*

Service or Feature

Interoperability etc. for FS11-14

n The higher level services are harder as the systems are more

complicated and less agreement on where standards should be defined

n OGF has JSDL, BES (Basic Execution Services) but might be

better to set standards at a different level

• i.e. users might prefer to view Condor or GT4 as collections of services as

the interface

n Idea is that maybe we should consider high level capabilities as

Grids (an EGEE or “Condor” compute Grid for example whose internals are black boxes for users) and then you need two types of interfaces

• Internal interfaces like JSDL defining how the Condor Grid interacts

internally with a computer

• External Interfaces defining how one sets up a complex problem (maybe

with lots of individual jobs as in SETI@Home) for a Compute Grid

Enabling Grids for E-sciencE

INFSO-RI-508833

gLite Grid Middleware Services

API

Access

Workload Mgmt Services

Computin Element

Workloa Manageme

nt Metadat

Catalog

Data Management

Storag

Element MovementDat File & Replic

Catalog Authorizatio

n

Security Services

Authenticati on

Information Monitoring

Information Monitoring Services

Application Monitoring

Connectivity Accounting Auditing

Jo

Provenance ManagerPackag CLI

DIRAC Architecture

UI User CLI

JobMonitorSvc JobAccountingS vc AccountingD B Job monitor ConfigurationS vc FileCatalogSvc BookkeepingSv c BK query webpage FileCatal og browser DIRAC services DIRAC Storage DiskFile gridftp LCG Resource Broker

CE 1 CE 2

Old AliEn Framework

100% perl5

SOAP

Local Site

elements

Central services

User

Database

S S

S

S SS SS SS SS SS SS SS SS

F S F S F S F S F S F S F S F

S SF

F S F S F S F S F S F S F S F S F S F S F

S Portal

F S O S O S O S O S O S O S O S O S O S O S O S O S MD MD MD MD MD MD MD MD MD

MetaData Filter Service Sensor Service Other Service Anothe Grid

Raw Data  Data  Information  Knowledge  Wisdom

Decisions S S S S Anothe Service Anothe Service S S Anothe

Grid S S

Anothe Grid S S S S S S S S S S S S S S S S F S SO_AP Messag es Por_tal O S O S F S O S O S MD MD MD F S

Data-Information-Knowledge-Wisdom Pipeline

n DIKW_i represent different forms of DIKW with different terminology in

different fields.

n Each DIKW_i has a resource view describing its physical instantiation

(different distributed media with file, database, memory, stream etc.) and

• An access view describing its query model (dir or ls, SQL, XPATH,

Custom etc.).

n The different forms DIKW_i are linked by filtering steps F. This could be a

simple format translation; a complex calculation as in the running of an LHC event processing code; a proprietary analysis as in a Search engines

processing of harvested web pages; an addition of a metadata catalog to a collection of files.

DIKW1

Access View

F DIKW2

Access View

F DIKW3

Access View

F Resource

View ResourceView ResourceView

DIKW Pipeline II

n

Each DIKW can be a complete

data grid

n

The

resource view

is typified by standards like ODBC,

JDBC, OGSA-DAI and is

internal

to DIKW Grid

• A name-value resource view is exemplified by Javaspaces

(tuple model) and WS-Context

n

The

access (user) view

is

external

view of a data grid

and does not have such a clear model but rather

• Systems like SRB (Storage Resource Broker) that virtualize

file collections

• WebDAV supports the distributed file access view

• VOSpace from astronomy community is viewed by some as

an abstraction of SRB

• WFS Web Feature Service from Open Geospatial

Consortium is an important example

WMS uses WFS that uses data sources

<gml:featureMember>

<fault>

<name> Northridge2 </name> <segment> Northridge2

</segment>

<author> Wald D. J.</author>

<gml:lineStringProperty>

<gml:LineString

srsName="null">

<gml:coordinates>

118.72,34.243 -118.591,34.176

</gml:coordinates>

</gml:LineString>

</gml:lineStringProperty>

</fault>

</gml:featureMember>

Managed Data

n Most grids have a managed data component (which we call a

“Managed Data Grid”)

n Managed data can consist of the data and one or more metadata

catalogs

• Metadata catalogs can contain semantic information enabling

more precise access to the “data”

• Replica catalogs (managing multiple file copies) are another

metadata catalog

n SRB and Digital libraries have this architecture with

mechanisms to keep multiple metadata copies coherent

n RDF has clear relevance

n However there is no clear consensus as to how to build a

Managed Data Grid

Resource and User Views

n Federation implies we integrate (virtualize) N data systems

which could be heterogeneous

n Sometimes you can choose where to federate but sometimes you

can only federate at user view

n In Astronomy Grids there are several (~20) different data

sources (collections) corresponding to different telescopes. These are built on traditional bases but expose astronomy query

interface (VOQL etc.) and one cannot federate at database level

n Geographical Information Systems GIS are built on possibly

spatially enhanced databases but expose WFS or WMS OGC interfaces

• To make a map of Indiana you need to combine the GIS of 92 separate

counties; this cannot be done at database level

n More generally when we linking black box data repositories to

the Grid, we can only federate at the interfaces exposed by the black box Resource View DIKW1 Access View Resource View DIKW2 Access View Resource View DIKW3 Access View Resource View DIKW4 Access View

User Level Federation

Resource Level Federation

Metadata Systems I: Applications

n

Semantic description of data (for each application)

n

Replica Catalog

n

UDDI or other service registry

n

VOMS or equivalent (PERMIS) authorization catalog

n

Compute Grid static resource metadata

n

Compute Grid dynamic events

n

And implicitly metadata defining workflow, state etc.

which can be stored in messages and/or catalogs

(databases)

n

Why not unify the resource view of these?

Metadata Systems II: Implementations

n There are also many WS-* specifications addressing meta-data

defined broadly

• WS-MetadataExchange

• WS-RF

• UDDI

• WS-ManagementCatalog

n And many different implementations from (extended) UDDI

through MCAT of the Storage Research Broker

n And of course representations including RDF and OWL

n Further there is system metadata (such as UDDI for core

services) and metadata catalogs for each application domain

n They have different scope and different QoS trade-offs

• e.g. Distributed Hash Tables (Chord) to achieve scalability in

large scale networks

• WS-Context

• ASAP

• WBEM

• WS-GAF

Different Trade-offs

n It has never been clear to me how a poor lonely service is meant

to know where to look up meta-data and if it is meant to be

thought up as a database (UDDI, WS-Context) or as the contents of a message (WS-RF, WS-MetadataExchange)

n We identified two very distinct QoS tradeoffs

n 1) Large scale relatively static metadata as in (UDDI) catalog of

all the world’s services

n 2) Small scale highly dynamic metadata as in dynamic workflows

for sensor integration and collaboration

• Fault-tolerance and ability to support dynamic changes with few millisecond delay

• But only a modest number of involved services (up to 1000’s

in a session)

• Need Session NOT Service/Resource meta-data so don’t use

WS-RF

XML Databases of Importance

n We choose a message based interface to a backend database n We built two pieces of technology with different trade-offs but

each could store any meta-data but with different QoS

• WS-Context designed for controlling a dynamic workflow • (Extended) UDDI exemplified by semantic service discovery

n WFS provides general application specific XML data/meta-data

repository built on top of a hybrid system supported by UDDI and WS-Context

n These have different performance, scalability and data unit size

requirement

n In our implementation, each is currently “just an

Oracle/MySQL” database (with Javaspaces cache in

WS-Context) front ended by filters that convert between XML (GML for WFS) and object-relational Schema

• Example of Semantics (XML) versus representation (SQL)

n OGSA-DAI offers Grid interface to databases – we could use this

internally but don’t as we only need to expose external interfaces

WFS and not MySQL to Grid

n

Extended UDDI

XML Metadata Service (alternative

to OGC Web Registry Services) supports

WFS

GIS

Metadata Catalog (functional metadata),

user-defined metadata ((name, value) pairs), up-to-date

service information (leasing), dynamically updated

registry entries.

n

Our approach enables advanced query capabilities

•

geo-spatial

and

temporal

queries ,

•

metadata oriented queries,

•

domain independent queries

such as XPATH,

XQuery on metadata catalog.

n

http://www.opengrids.org/extendeduddi/index.html

WFS: Geographical Information System

compatible XML Metadata Services

Context as Service Metadata

n

We define all metadata (static, semi-static, dynamic)

relevant to a service as “

Context

”.

n

Context

can be associated to a single service, a

session (service activity) or both.

n

Context can be independent of any interaction

n

slowly varying, quasi-static

context

n

Ex: type or endpoint of a service, less likely to

change

n

Context

can be generated as result of service

interactions

n

dynamic, highly updated

context

n

information associated to an activity or session

n

Ex: session-id, URI of the coordinator of a

workflow session

Hybrid XML Metadata Services –>

WS-Context + extendedUDDI

n

We

combine

functionalities of these two services:

WS-Context AND extendedUDDI in one hybrid service to

manage

Context

(service metadata).

•

WS-Context

controlling a workflow

•

(Extended) UDDI

supporting semantic service

discovery

n

This approach enables

uniform query capabilities

on

service metadata catalog.

n

http://www.opengrids.org/wscontext/index.html

Information Service WSDL IS Client WSDL WSDL HTTP(S) WSD L IS Client DB JDBC Extended WS-Context Service dynamic metadata IS Client WSDL DB JDBC Extended UDDI Registry Service

UDDI Version 3.0 WSDL

Service Interface Descriptions

uddi_api_v3_portType.wsdl

WSDL WSDL WSDL

Generalizing a GIS

n

Geographical Information Systems

GIS have been

hugely successful in all fields that study the earth and

related worlds

• They define Geography Syntax (GML) and ways to store,

access, query, manipulate and display geographical features

• In SOA, GIS corresponds to a domain specific XML language

and a suite of services for different functions above

n

However such a universal information model has

not

been developed in other areas

even though there are

many fields in which it appears possible

• BIS Biological Information System

• MIS Military Information System

• IRIS Information Retrieval Information System

• PAIS Physics Analysis Information System

• SIIS Service Infrastructure Information System

ASIS Application Specific Information System I

n a) Discovery capabilities that are best done using WS-*

standards

n b) Domain specific metadata and data including

search/store/access interface. (cf WFS). Lets call generalization

ASFS (Application Specific Feature Service)

• Language to express domain specific features (cf GML). Lets call

this ASL (Application Specific language)

• Tools to manipulate information expressed in language and key

data of application (cf coordinate transformations). Lets call this

ASTT (Application specific Tools and Transformations)

• ASL must support Data sources such as sensors (cf OGC metadata

and data sensor standards) and repositories. Sensors need

(common across applications) support of streams of data

• Queries need to support archived (find all relevant data in past)

and streaming (find all data in future with given properties)

• Note all AS Services behave like Sensors and all sensors are

wrapped as services

• Any domain will have “raw data” (binary) and that which has been

filtered to ASL. Lets call ASBD (Application Specific Binary Data)

ASIS Application Specific Information System II

n Lets call this ASVS (Application Specific Visualization Services)

generalizing WMS for GIS

n The ASVS should both visualize information and provide a way of

navigating (cf GetFeatureInfo) database (the ASFS)

n The ASVS can itself be federated and presents an ASFS output

interface

n d) There should be application service interface for ASIS from which all

ASIS service inherit

n e) There will be other user services interfacing to ASIS

n All user and system services will input and output data in ASL using

filters to cope with ASBD

AS Tool (generic

) A

“Sensor ” A Repository

AS Service (user defined)

ASVS Displa

y AS Tool

(generic )

Messages using ASL

Filter, Transformation, Reasoning, Data-mining, Analysis

n

Handheld Flexible Representation

(HHFR) is an open

source software for fast communication in mobile Web

Services. HHFR supports:

• streaming messages, separation of message contents and

usage of context store.

• http://www.opengrids.org/hhfr/index.html

n

We use WS-Context service as

context-store for

redundant message parts

of the SOAP messages.

• redundant data is static XML fragments encoded in every

SOAP message

• Redundant metadata is stored as context associated to service

conversion in place

n

The empirical results show that we gain

83%

in

message size

and on avg.

41%

on

transit time

by using

WS-Context service.

Application – Context Store usage in

communication of mobile Web Services

Optimizing Grid/Web Service Messaging

Performance

Performance

with and without Context-store

0.538 2.81±0.098

0.867 5.20±0.158

Large: 2.61KB (sec)

0.217 1.75±0.040

0.187 2.76±0.034

Medium: 513byte (sec)

Stddev Ave.±error

Stddev Ave.±error

With Context-store Without Context-store

Message Size

l

Experiments ran over HHFR

l Optimized message exchanged over HHFR after saving

redundant/unchanging parts to the Context-store

l

Save on average

83% of message size, 41% of transit time

Summary of the Round Trip Time (TRTT)

System Parameters

l T_access: time to access to a Context-store (i.e. save a context or

retrieve a context to/from the Context-store) from a mobile client

l T_RTT: Round Trip Time to exchange message through a HHFR

channel

l N: number of simultaneous streams supported by stream summed

over ALL mobile clients

l T_wsctx: time to process setContext operation

l T_axis: time consumed for Axis process

l T_trans: transmission time through network l T_stream: stream length

Context-store:

System Parameters

Summary of

T

axis

and T

wsctx

measurements

T

= T

+ T

+

T

Data binding overhead at Web Service Container is the dominant factor to message processing

l C_hhfr = nt_hhfr + O_a + O_b l C_soap = nt_soap

l

Breakeven point:

n

n

Oa(WS) is roughly 20 milliseconds

Performance Model and Measurements

0.825 5.133±0.036

Negotiation (Ob)

0.516 4.127±0.042

Context-store Access (Oa)

Stddev (sec) Average±error (sec)

Oa : overhead for accessing the Context-store Service

Ob : overhead for negotiation

Core Features of Management Architecture

n Remote Management

• Allow management irrespective of the location of the resource

(as long as that resource is reachable via some means)

n Traverse firewalls and NATs

• Firewalls complicate management by disabling access to some

transports and access to internal resources

• Utilize tunneling capabilities and multi-protocol support of

messaging infrastructure

n Extensible

• Management capabilities evolve with time. We use a service

oriented architecture to provide extensibility and interoperability

n Scalable

• Management architecture should be scale as number of

managees increases

n Fault-tolerant

• Management itself must be fault-tolerant. Failure of transports

OR management components should not cause management architecture to fail.

Management System built in terms of

n

Bootstrap System

– Robust itself by Replication

n

Registry for metadata

(distributed database) – Robust

by standard database techniques and our system itself

for Service Interfaces

n

NaradaBrokering

for robust tunneled messages – NB

itself robust using our system

n

Managers

– Easy to make robust using our system; these

are essentially agents

n

Managees

– what you are managing – Our system makes

robust – There is NO assumption that Managed system

uses NB

Basic Management Architecture I

• Stores system state.

• Fault-tolerant through replication

• Could be a global registry OR separate

registries for each domain (later slide)

• Current implementation uses a simple

in-memory system

• Will use our WS - Context service as our

registr

(Service/Message Interface to in-memory JavaSpaces cache and MySQL)

• Note metadata transported by messages but we use distributed database to implement

n Messaging Nodes

• NaradaBrokering nodes that form a scalable

messaging substrate

• Main purpose is to serve as a message delivery

mechanism between Managers and Service Adapters (Managees) in presence of varying network conditions

Registry

Read / Write from / to Registry via

pre-determined

TOPIC NB

Basic Management Architecture II

n Resources to Manage (Managee)

• If the resources DO NOT have a Web

Service interface, we create a Service

Adapter (a proxy that provides the Web

Service interface as a wrapper over the basic management functionality of the resource).

• The Service Adapters connect to existing

messaging nodes. This mainly leverages multi-protocol transport support in the messaging substrate. Thus, alternate protocols may be used when network policies cause connection failures

n Managers

• Active entities that manage the resources.

• May be multi-threaded to improve

scalability (currently under further investigation)

Managees

Manage r

Registry

…

Read / Write from / to Registry via

pre-determined

TOPIC

Service

Adapter Resource NB

Architectur

Use of Messaging Nodes

n Service adapters and Managers communicate through

messaging nodes

n Direct connection possible, however

• This assumes that the service adapters are appropriately

accessible from the machines where managers would run

n May require special configuration in routers / firewalls

• Typically managers and messaging nodes and registries are

always in the same domain OR a higher level network domain with respect to service adapters

n Messaging Nodes (NaradaBrokering Brokers) provides

• A scalable messaging substrate • Robust delivery of messages • Secure end-to-end delivery

Architectur

Bootstrapping Process

n The architecture is arranged hierarchically.

• Resources in different domains can be

managed with separate policies for each domain

n A Bootstrapping service is run in every

domain where the management architecture exists.

• Serves to ensure that the child domain

bootstrap process are always up and running.

n Periodic heartbeats convey status of

bootstrap service

• Bootstrap service periodically spawns a

health-check manager that checks health

of the system (ensures that the registry and messaging nodes are up and running and that there are enough managers for managees)

n Currently 1 manager per managee

/ROOT

/ROOT/FSU

/ROOT/CGL

Regist

ry Regist_ry

Hierarchica Bootstrap

Nodes

Architecture:

User Component

n Application-specific specification of the characteristics that the

resources/services being managed, should maintain.

• Impacts Managee interface, registry and Manager

n Generic and Application specific policies are written to the

registry where it will be picked up by a manager process.

n Updates to the characteristics (WS-Policy in future) are

determined by the user.

n Events generated by the Managees are handled by the manager.

• Event processing is determined by policy (future work),

n E.g. Wait for user’s decision on handling specific

conditions

n The event can be processed locally, so execute default

policy, etc…

n Note Managers will set up services if registry indicates that is

appropriate; so writing information to registry can be used to start up a set of services

Architectur

Structure of Managers

n Manager process starts appropriate

manager thread for the manageable resource in question

• Heartbeat thread periodically registers

the Manager in registry

• SAM (Service Adapter Manager)

Module Thread starts a

Service/Resource Specific “Resource Manager” that handles the actual management task

• Management system can be extended

by writing ResourceManagers for each type of Managee

Manager

Heartbeat Generator Thread

SAM Module

Resour ce Manag

er

Prototype

n We illustrate the architecture by managing the distributed messaging

middleware, NaradaBrokering

• This example motivated by the presence of large number of dynamic peers

(brokers) that need configuration and deployment in specific topologies

n Use WS – Management (June 2005) parts (WS – Transfer [Sep 2004], WS – Enumeration [Sep 2004] and WS – Eventing) (could use WS-DM)

• WS – Enumeration implemented but we do not foresee any immediate use

in managing the brokering system

• WS – Transfer provides verbs (GET / PUT / CREATE / DELETE) which

allow us to model setting and querying broker configuration, instantiating brokers and creating links between them and finally deleting brokers

(tear down broker network) and re-deploy with possibly a different configuration and topology

• WS – Eventing (will be leveraged from the WS – Eventing capability

implemented in OMII)

n WS – Addressing [Aug 2004] and SOAP v 1.2 used (needed for

WS-Management)

• Used XmlBeans 2.0.0 for manipulating XML in custom container. n WS-Context will replace current registry

Prototype Components

• Note NB illustrates an electronic entity that didn’t start off with an

administrative Service interface

• So add wrapper over the basic NB BrokerNode object that provides WS – Management front-end

• Also provides a buffering service to buffer undeliverable responses

n These will be retrieved later by a separate Request – Response message

exchange

n Broker Network Manager

• WS – Management client component that is used to configure a broker object through the Broker Service Adapter

• Contains a Request-Response as well as Asynchronous messaging style

capabilities

• Contains a topology generator component that determines the wiring

between brokers (links that form a specific topology)

n For the purpose of prototype we simply create a CHAIN topology

where each ith _{broker is connected to (i-1)}st _broker

Prototype

Resources/Properties Modeled (very specific to NaradaBrokering)

• Assigns a NODE / GATEWAY address to the current node if one is not already assigned • Create

NODE ADDRESS, GATEWAY ADDRESS

• Retrieves the current

configuration / a single property • Saves the specified

configuration / single property • Get

• Put

CONFIGURATION,

CONFIGURATION PROPERTY

• Creates a link between two brokers

• Deletes the link between two brokers

• Create

• Delete

LINK (Note we manage brokers and streams)

•Instantiates the broker with current configuration

• Deletes the broker node • Create

• Delete

BROKER

Description Operations

Resource URI

Response Tim

Handling Events (WS – Eventing)

n Test Resource which

does not do any work other than responding to events

n This base model shows

that up to200 resources can be

managed per manager process, beyond which response time

increases rapidly

n This number is

resource dependent and this result is illustrative.

n Equally dividing

management between 2 processes, increases response time,

although slowly.

Amount of Management

Infrastructure Required

n

N = Number of resources to manage

n

N

= Number of Manager processes

• If a manager process can manage 200 resources

simultaneously, then NMP = N/200

n

N

= Number of Messaging Nodes

• If a messaging node can support 800 simultaneous

connections then

• N_MN = (N + N/200 + 1) /800

n 1 connection is for registry

Amount of Management

Infrastructure Required

n Management Infrastructure Required

= N/200 + (N + N/200 + 1)/800 = N/160 (approximately)

• Thus, for N > 160, management is doable by adding

(N/160) * 100 ---(N + N/160)

• i.e. about, 0.625% more processes

n Thus Management architecture is scalable, and the approach

is feasible

Prototyp

Recovery costs (Individual Resources – Brokers)

Time (msec) (average values)

Initialized

(Later modifications)

110 109 94 734 1110

Un-Initialized (First time)

187.50 Delete Broker

35.25 Delete Link

43 Create Link

132.75 Create Broker

46.75 Set Configuration

Operation

Time for Create Broker depends on the number & type of transports opened by the broker

E.g. SSL transport requires negotiation of keys and would require more time than simply opening a TCP port

Recovery times I

n

Use

5 msec

read time per object from registry

and consider two different topologies

n

Ring Topology

•

N nodes, N links (1 outgoing link per Node)

•

Each Resource Management thread loads 2 objects

(read from) and write their corresponding state (2

objects) (write to) REGISTRY.

•

Time to load (theoretical) per broker

≈ 10 + 1110 + 734 + 94

= 1.9 sec

•

Time to load (observed) = 2.4 to 2.7 secs (total)

Recovery times II

n Cluster Topology

• N nodes, Links per broker vary from 0 – 3 (depending on what

level is the broker present)

n At Cluster level, we maintain a chain (hence num links is 1, and

0 for the last node)

n At Super cluster level, we again maintain a chain, so all nodes

except the last node will have an additional link

n At Super-Super-Cluster level, we maintain a chain of

super-cluster level nodes, so an additional link per node except for the last one in chain

• Each Resource Management thread loads from 1 – 4 objects (read

from) and write their corresponding state (1 - 4 objects) (write to) REGISTRY.

n 1 object for Broker Node, others for links

• Thus, Time to load (theoretical) per broker

≈ {5 – 20} + 1110 + 734 + {0 – (94*1 + 43*2)}

= 1.8 – 2.0 sec similar to ring topology