HPSearch: Design & Development via Scripting

(1)

HPSearch

Design & Development via Scripting

Harshawardhan Gadgil

(2)

Why Script Web Services

n

Scripting Advantages

n

Rapid prototyping

n

Scripting Web Services

n

Manipulate data streams, files and web-services using

scripting language. Equivalent to shell and Perl in UNIX.

n

Allows us to control the invocation and execution of

disparate services using scripting. Equivalent to

programming a workflow

n

Setting up data-flow among components by dynamically

joining disparate data streams

n

Rapidly deploy applications (under development)

n

We have developed a tool for binding URIs to a scripting

(3)

HPSearch

n

Binds URI to a scripting language

n

We use Mozilla Rhino (A Javascript implementation, Refer:

h

ttp://www.mozilla.org/rhino),

but the principles may be

applied to any scripting language such as Perl, Python etc…

n

Every Resource may be identified by a URI and HPSearch

allows us to manipulate the resource using the URI.

n

For e.g. Read from a web address and write it to a local file

x = “http:

//trex.ucs.indiana.edu/data.txt”;

y =

“file:/

//u/hgadgil/data.txt”;

Resou

rce r = new Resource(“Copier”);

r.port[0].subscribeFrom(x);

// read from

r.port[0].publishTo(y);

// write to

f = new Flow();

(4)

HPSearch (contd.)

n

Currently provide bindings for the following

n

file://

n

socket://ip:port

n

http://, ftp://

n

topic://

n

jdbc:

n

Host-objects to do specific tasks

n

WSDL

– invoke web-services using SOAP

n

PerfMetrics

– Bind NaradaBrokering performance metrics. Store

published metrics and allow querying

n

Resource

– Every data source / filter / sink is a resource.

n

Flow

– To create a data flow between resources. Useful for streaming

applications.

n

For more information, visit

(5)

NaradaBrokering

n

Event- brokering system designed to run on a large

network of co-operating brokers.

n

Guarantees delivery in presence of failures

n

Implements high-performance protocols (message

transit time < 1 ms per broker)

n

Order-preserving optimized message transport

n

Interface with reliable storage for persistent events

n

Fault tolerant data transport

n

Support for different underlying transport

(6)

n

HPSearch uses NaradaBrokering to route data

streams

n

Consider the script as follows

Resource a = new Resource(“ResourceA”);

a.port[0].subscribeFrom(“file:///u/hgadgil/data.txt”); a.port[0].publishTo(“topic:///x”); // Optional

Resource b = new Resource(“ResourceB”);

b.port[0].publishTo(“file:///tmp/rawData.dat”); b.port[0].subscribeFrom(a.port[0].publish); f = new Flow();

f.addComponents(b); f.addStartActivities(a); f.start(“1”);

HPSearch + NaradaBrokering

Resource

A

Resource

B

HPSearch Shell

WFE

(7)

Broker

Network

HPSearch - WSProxy

WFE

WSProxyClient

WSProxy

Service

WSProxyClient can control the WSProxy using simple

Web-service calls

WSProxy is a simple

web-service that listens for

requests from the client

(

WSProxyClient

) and

steers the service

Request Handler

WFE

HPSearch Shell

The WSProxy connects to a broker and handles all the data communication on behalf of the

service

Pipe

DB Connector

(8)

Flood Modeling Example using Web

Service scripting: Components

n

Rainfall Data

n

Inches of rainfall gathered by the sensor network at specific intervals

(Simulated to be in intervals of 30 min in our example)

n

WebMap Server

n

OpenGIS based terrain data repository

n

Generates Watershed model

n

Compute Service – Runoff Model

n

Computes the runoff model using the Hydrostatic watershed model and

Hydrodynamic rainfall data

n

Visualization Service

n

Shows the visual representation of the runoff model for the terrain in question

n

Each of these components run as a separate Web service (Either direct or

wrapped as a web service)

n

Scripting to manage the chain of these disparate services into one complete

application

n

Each of these components is wrapped using WSProxy which is then

(9)

Crisis Grid - Flood Modeling

n

Compute Service is the basic control

n

On start up, MapService sends the

spatial information for the terrain

under consideration to the Compute

service.

n

Compute Service requests the

Rainfall publisher to send the next

rainfall reading (pulls the next

reading).

n

The Compute service then operates

the run off model for flood analysis.

n

The computed results are then sent to

a Visualization service which displays

the computed predicted model results

in form of an animation

Map Service

Rainfall

Publisher

Compute Service

(10)

maps = "/CGL/FLOODMODEL/MAPS"; rain = "/CGL/FLOODMODEL/RAINFALL"; visu = "/CGL/FLOODMODEL/VISUAL";

mapServer = "org.hpsearch.demo.floodModel.MapService";

mapServerLoc = "http://156.56.104.170:9090/axis/services/WSSConnector?wsdl"; mapSource = new WebServiceHandler(mapServer);

mapSource.setEndPointURI(mapServerLoc);

mapSource.setParameter("MAPSERVERTOPIC", maps);

rainFallPublisher = "org.hpsearch.demo.floodModel.RainfallPublisher";

rainFallPublisherLoc = "http://156.56.104.170:6060/axis/services/WSSConnector?wsdl"; rainFall = new WebServiceHandler(rainFallPublisher);

rainFall.setEndPointURI(rainFallPublisherLoc); rainFall.setParameter("RAINFALLTOPIC", rain); rainFall.setParameter("VISUALIZATIONTOPIC", visu);

computeService = "org.hpsearch.demo.floodModel.ComputeService";

computeServiceLoc = "http://156.56.104.170:7070/axis/services/WSSConnector?wsdl"; compute = new WebServiceHandler(computeService);

compute.setEndPointURI(computeServiceLoc); compute.setParameter("MAPSERVERTOPIC", maps); compute.setParameter("RAINFALLTOPIC", rain); compute.setParameter("VISUALIZATIONTOPIC", visu); crisisGridFlow = new Flow();

crisisGridFlow.addComponents(compute);

crisisGridFlow.addStartActivities(mapSource, rainFall); crisisGridFlow.start("1");

HPSearch Java Script

Code for creating and initializing the

WMS Component

Code for creating and initializing the

Rainfall data Component

Code for creating and initializing the

Compute service

(11)

How to initialize…

n

First Define the service to be invoked

mapServer

= "org.hpsearch.demo.floodModel.MapService";

n

The location of the service. This should be normally done via a registry lookup. For

For E.g.

NBDiscover.discover(“MapService”, specs);

Its hard-coded as of now

mapServerLoc =

"http://156.56.104.170:9090/axis/services/WSSConnector?wsdl";

n

Initialize a WebService Handler. This object invokes a

WSProxyClient

that interacts

with the

WSProxy

which is a wrapper on the actual service

mapSource

= new WebServiceHandler(mapServer);

n

Set the End-point URI

mapSource.setEndPointURI(mapServerLoc);

n

Set the service initialization properties. This is optional, but allows for more

customization. This could include QoS properties, security tokens and any other

service specific parameters

(12)

How to Initialize (contd.)

n

Finally we create a new

Flow

Object. This object helps to serialize the requests for

setting up a new Flow. The

FlowHandler

component of the WorkflowEngine then

distributes these requests to other engines and the flow is finally started.

crisisGridFlow = new Flow();

n

We add the components of the flow. All components which generate the data to be

processed are

startActivities

and are started after the other components are

started. This enables the components to start processing the data as soon as the

startActivities

publish the data

crisisGridFlow.addComponents(compute);

crisisGridFlow.addStartActivities(mapSource, rainFall);

n

Finally we start the flow. The

”1”

is for debugging purposes. Ignore it for now

(13)

In Short…(w.r.t. Flood model)

n

Handle file-based data.

n

Read from static files (archived data),

databases, For E.g. The terrain data is static

n

Support streaming data sources. For E.g.

Rainfall data read in real-time in a stream.

n

In an actual system we may expect the

frequency of observation (and hence data

generation) to be about 5 to 20 minutes.

n

In our system, we assume just one source for

(14)

Wrapping a Simple Service

n

Two implementations of

ProxyWebService

n

RunnableProxyWebService

n

Provides operations START, STOP, SUSPEND,

RESUME.

n

Service is provided the required Input and Output

streams from where it will read and write.

n

Good for applications which work on a chunk of data

at a time

n

Easiest way to create your first service

n

WrapperProxyWebService

n

Provides just the 4 operations as above.

n

The application handles its own communication

(15)

package mypackage;

public class StringProcessingService extends RunnableProxyWebService {

BufferedReader input;

BufferedWriter output;

public void initService(InputStream[] in, OutputStream[] out) {

input = new BufferedReader(new InputStreamReader(in[0]));

output = new BufferedWriter(new OutputStreamWriter(out[0]));

}

public String[] serviceExceptions() {

String[] exceptions = { "java.io.EOFException",

"java.io.IOException“};

return exceptions;

}

Coding a simple string reversal service

MUST

initialize the

service in this fashion

(16)

Coding a simple string reversal service

public void process() throws Exception {

String s, newS;

try {

s = input.readLine();

if (s == null) {

serviceFinished();

return;

}

} catch (IOException e) {

throw e;

}

newS = reverseOf(s);

output.write(newS);

output.newLine();

output.flush();

}

public String reverseOf(String s) {

…

}

(17)

Deploy using AXIS

n

Create a deployment descriptor for AXIS

n

Or use one which already exists

n

Deploy the service

(18)

Invoking our simple service

service = “mypackage.StringProcessingService”;

serviceLoc = “http://...path to WSDL”;

stringproc = new WebServiceHandler(service);

stringproc.setEndpointURI(serviceLoc);

stringproc.setInput(“input URI”);

stringproc.setOutput(“output URI”);

f = new Flow();

(19)

Useful links for additional information

n

WSProxy

n

h

ttp://www.hpsearch.org/notes/wsproxy

n

Te

chnical Report on the flood model

n

htt

p://www.hpsearch.org/documents/crisisGridFlow.pdf

n

Timi

ng Analysis etc… (incomplete as of now)

(20)

Credits…

n

Dr. Sunghoon Ko (CGL) and Dr. Jin-Yong

Choi (Purdue Univ.) for the Flood Demo