Longer version

ANABAS

Use of Grids in DoD

Applications

Geoffrey Fox, Alex H

The Ten areas covered by the 60 core WS-* Specifications

WSRP (Remote Portlets)

10: Portals and User Interfaces

WS-Policy, WS-Agreement

9: Policy and Agreements

WSDM, WS-Management, WS-Transfer

8: Management

WSRF, WS-MetadataExchange, WS-Context

7: System Metadata and State

UDDI, WS-Discovery

6: Service Discovery

WS-Security, WS-Trust, WS-Federation, SAML, WS-SecureConversation

5: Security

BPEL, WS-Choreography, WS-Coordination

4: Workflow and Transactions

WS-Notification, WS-Eventing (Publish-Subscribe)

3: Notification

WS-Addressing, WS-MessageDelivery; Reliable Messaging WSRM; Efficient Messaging MOTM

2: Service Internet

XML, WSDL, SOAP

1: Core Service Model

Activities in Global Grid Forum Working Groups

Authorization, P2P and Firewall Issues, Trusted Computing

7: Security

Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model

6: Management

Network measurements, Role of IPv6 and high performance networking, Data transport

5: Infrastructure

Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management

4: Data

Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling

3: Compute

Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval,

2: Applications

High Level Resource/Service Naming (level 2 of fig. 1), Integrated Grid Architecture

1: Architecture

The Global Information Grid Core Enterprise Services

Retention, organization and disposition of all forms of data

CES8: Storage

Includes automated and manual methods of optimizing the user GiG experience (user agent)

CES7: User Assistance

Provision and control of sharing with emphasis on synchronous real-time services

CES6: Collaboration

Includes translation, aggregation, integration, correlation, fusion, brokering publication, and other transformations for services and data. Possibly agents

CES5: Mediation

Searching data and services

CES4: Discovery

Synchronous or asynchronous cases

CES3: Messaging

Supports confidentiality, integrity and availability. Implies reliability and autonomic features

CES2: Information

Assurance (IA)/Security

including life-cycle management

CES1: Enterprise Services Management (ESM)

Major Conclusions I

n One can map 7.5 out of 9 NCOW and GiG core

capabilities into Web Service (WS-*) and Grid (GS-*) architecture and core services

• Analysis of Grids in NCOW document inaccurate

(confuse Grids and Globus and only consider early activities)

n Some “mismatches” on both NCOW and Grid sides

n GS-*/WS-* do not have collaboration and miss some

messaging

n NCOW does not have at core level system metadata

and resource/service scheduling and matching

n Higher level services of importance include GIS

(Geographical Information Systems), Sensors and

Major Conclusions II

n Criticisms of Web services in a recent paper by

Birman seem to be addressed by Grids or reflect immaturity of initial technology implementations

n NCOW does not seem to have any analysis of how to

build their systems on WS-*/GS-* technologies in a layered fashion; they do have a layered service

architecture so this can be done

• They agree with service oriented architecture

• They seem to have no process for agreeing to WS-*

GS-* or setting other standards for CES

n Grid of Grids allows modular architectures and

DoD Core Services and WS-* plus GS-* I

XGSP, Shared Web Service ports, Anabas Asynchronous Virtual

Organizations Shared Web Resources

CES 6: Collaboration

Treatment of Legacy systems. Data Transformations

WS-4 Workflow

CES 5: Mediation

Extended UDDI WS-6 UDDI

CES 4: Discovery

NaradaBrokering, Streaming/Sensor Technologies WS-2, WS-3 Service Internet Notification

CES 3: Messaging

Grid-Shib, Permis Liberty Alliance etc.

GS-7 Securit (Authorization)

WS-5

WS-Security

CES 2: Information Assurance(IA)/Security

CIM GS-6: Management

WS-8 Management

CES 1: Enterprise Services Management

B: NCOW Core Services (to be continued)

Legacy subsystems and modular architecture

Grid of Grids Composition

Industry Best Practice (IBM, Microsoft …)

Build Grids on Web Services

WS-1: Core Service Model

Use Service Oriented Architecture

A: General Principles

Others GGF

DoD Core Services and WS-* and GS-* II

OGC GIS standards

Geographical Information

OGC Sensor standards Work starting

Sensors (real-time data)

Extend computer scheduling to networks and data flow

Distributed Scheduling and SLA’s (GS-3)

Resource/Service Matching/Scheduling C2IEDM, XBML, DDMS, WFS Semantic Grid Globus MDS WS-7 System Meta-data

C: Key NCOW Capabilities not directly in CES

WS-9 Policy

Environmental Control Services ECS

Best Practice in building Grid/Web services (proxy or direct)

GS-2; invoke GS-3

CES 9: Application

NCOW Data Strategy GS-4 Data

CES 8: Storage (not real-time streams)

B: NCOW Core Services Continued

Others GGF

Information Exploitation | Information Fusion & Understanding | Information Management| Cyber Operations | Command & Control | Connectivity | Advanced Computing Architectures

Technologies:

•Business Process Execution Language •Global Multimedia Collaboration System •Narada Brokering

•Web Service Standards

•Open Grid Computing Environment

Problem: Unable to satisfy interoperability, scalability, and security information management requirements for Net-Centric Operations without an advanced grid-based scalable service oriented framework

Grid of Grids Service Architecture for Net-Centric Operations and Warfare

Objective:Integrate Global Grid Forum (GGF) information management research to provide a Grid of Grids Middleware for Net-Centric Operations consistent with the Global Information Grid (GIG).

Approach:

•Analyze Net-Centric Operations and Warfare (NCOW) service specifications and relate core enterprise services in Net-Centric Enterprise

Services (NCES) to core GGF and Web Services (WS-*) standards

•Develop the Grid of Grids architecture and information management middleware to address federation of legacy and new DoD enterprise systems with service oriented mediation between component Grids

•Develop prototype for NCES capabilities (Collaboration, Messaging, Management, Security/Information Assurance, Discovery, Mediation, User Assistance, Storage, Applications) with advanced Grid and Web Service standards support

•Demonstrate for Geographical Information System

Challenges:

•Combining scalability and interoperability of Grid and Web Services with DoD real time and security requirements

•Supporting legacy systems and standards while maximizing value of new commercial service oriented architectures

Impact:

•A scalable managed interoperable secure architecture and middleware for NCOW •Rapid implementation strategy leveraging commercially pervasive technology

Results:

•Phase I middleware addresses 6 of 9 NCOW Core Enterprise Services •Demonstrated GIS application in collaboration grid

General Message I

n Our proof of concept demonstrates many of the NCOW

core enterprise services (CES) implemented using Grid

services built on top of the WS-* Web service industry

specifications.

n We will illustrate the use of the Grid of Grids

architecture to integrate heterogeneous systems. The papers describe how all CES can be implemented using Grid technology and this is proposed in phase II SBIR.

n Note the adherence to standards with a common line

protocol SOAP implies that all service implementations

are interoperable and one takes services from multiple

General Message II: Why Grids

n Web services gives us interoperability but Grids are

essential as we aim at Information Management

n Grids are the key idea to manage complexity but

applying uniform policies and building managed systems

n Grids of Grids allows one to build out the management

in a modular fashion

n Uniform Grid messaging handles complex networks

with managed QoS such as real-time constraints

n Managed Services and Messaging gives scalability and

Scenario

n Laptop 8 Windows Client (small Sony)

n Laptop 2 Old Windows Client and PPT machine n Laptop 3 Windows Server

n Laptop 4 Linux

n Assume no Internet access

n List of CES and GS-* WS-* mapping on Laptop 2

followed by complete stream of PPT screen shots if failure

Script I: Data Mining and GIS Grid

n This will show a set of Open Geospatial Consortium (OGC)

compatible services implementing a GIS (Geographical

Information System) grid supporting streaming of feature and map data.

n Intrinsic features of a region are supplemented here by

features coming from a data-mining code that is filtering data to predict likely earthquake positions.

n This uses discovery, metadata, database, workflow,

messaging, data transformation, simulation (data-mining) services.

n Note the OGC compatible WFS (Web Feature Service) plays

role as a domain specific service interface to a database

n This used by Los Alamos for DHS simulations replacing data

Script I: Google Map Grid Service

n This first demo also illustrates how the Google map

system can be wrapped as a Grid itself front-ended by a OGC Web Map Service.

n This is used in a Grid of Grids fashion with Google

linked with traditional (NASA) Web Map services.

n Illustrates how linking NCOW to commodity Grid

technology allows access to major IT resources

• Google’s 100,000 computers

Script II: Collaborative Grid Service

n This demonstrates how streams can be formed from

messages and managed in a uniform way whether maps or video. Collaboration is achieved by multicasting of the input or output streams to Grid services.

n Our messaging infrastructure handles all multicasting

(using software) transparently to services

n First we demonstrate collaborative maps using “shared

Script III: Collaboration Grid

n Collaborative Audio-video conferencing on Laptop 3 and 8

with services running on 3

n Collaboration uses basic Grid services – metadata, discovery,

workflow, security plus the XGSP stream management services.

n Complex collaboration scenarios correspond to additional

services for particular shared applications and to gateways in Grid of Grids fashion to H323, SIP and other protocols. Annotation, record, replay, whiteboards, codec conversion, audio and video mixing become services.

n We demonstrate MPEG4 transcoding and video mixing

services

n Only Grid Web service based collaboration environment

Performance

n Reduction of message delay jitter to a millisecond.

n Dynamic meta-data access latency reduced from seconds to

milliseconds using web service context service.

n The messaging is distributed with each low end Linux node

capable of supporting 500 users at a total bandwidth of 140 Mbits/sec with over 20,000 messages per second.

n Systematic use of redundant fault tolerance services supports

strict user QoS requirements and fault tolerant Grid enterprise bus supports collaboration and information

sharing at a cost that scales logarithmically with number of simultaneous users and resources.

n Supporting N users at the 0.5 Mbits/sec level each would

ANABAS

I: Data Mining and

GIS Grid

I: Data Mining and GIS Grid

WMS handling Client requests

WMS Client UDD

I

WFS 2 Databases with

NASA, USGS features SERVOGrid Faults

WFS

1 NASAWMS

HTTP SOAP

WFS 3

Data Mining Grid

Get Feature Info allows users to get map

information. This can also be used to read feature info off the map when creating input data for applications

PI demo

combines WFS, WMS, and

HPSearch for service

orchestration.

Tool bar items allows you to adjust maps

Hot spots

calculations--areas of

increased earthquake probability in the forecast

Electric Power and Natural Gas data from LANL Interdependent Critical Infrastructure Simulations

Zoom-in Zoom-out

FeatureInfo mode Measure distance mode Clear Distance

ANABAS

I(contd): Google Grid

Service

Google Maps

as Service

accessed from

ANABAS

II: Sensor Gri

(available on movie)

Typical use of Grid Messaging in NASA

Datamining

Sensor Grid

Grid Eventing GIS Grid

Typical use of Grid Messaging

Web Feature Service

Real Time GPS

and Google Maps

Subscribe to live GPS station. Position data from SOPAC is

combined with Google map clients.

Google maps can be

integrated with Web Feature Service

Archives to filter and

browse seismic records.

ANABAS

III: Collaborative

Google Grid Service

ANABAS

IV: Collaboration Grid

GlobalMMCS Web Service Architecture

SIP H323 Access_Grid Native_XGSP

Admire

Gateways convert to uniform XGSP Messaging

High Performance (RTP and XML/SOAP and ..

Media Servers Filters Session Server XGSP-based Control NaradaBrokerin g All Messaging

Use Multiple Media servers to scale to many codecs and many versions of audio/video mixing

NB Scales a distributed

We Services

Average Video Delays for one broker –

Performance scales proportional to number of brokers

Latency ms

# Receivers One session Multipl

sessions

Chat TV

Webcam Video

ANABAS

V: Annotation Service

e - Annotation Playe

r

Archived stream playe

r

Annotatio

nplaye / WB r

e -Annotation Whiteboar

d

ANABAS

Messaging

NaradaBrokering 2003-2006

n Messaging infrastructure for collaboration, peer-to-peer and Grids

Implements JMS and native high-performance protocols (message

transit time of 1 to 2 ms per hop)

n Order-preserving message transport with QoS and security profiles

n Support for different underlying transport such as TCP, UDP,

Multicast, RTP

n SOAP message support and WS-Eventing, WS-RM and WS-Reliability.

• WS-Notification when specification agreed

n Active replay support: Pause and Replay live streams.

n Stream Linkage: can link permanently multiple streams – using in

annotation of real-time video streams

n Replicated storage support for fault tolerance and resiliency to storage

failures.

n Management: HPSearch Scripting Interface to streams and brokers

(uses WS-Management)

n Broker Topics and Message Discovery: Locate appropriate

n Integration with Axis2 Web Service Container (?)