The Open Grid Computing Environments Collaboratory

(1)

The Open Grid Computing

Environments Collaboratory

Marlon Pierce

Community Grids Lab

Indiana University

(2)

NSF NMI Project for Reusable

Portal Components: Who We Are

n

University of Chicago/ANL

•

Gregor von Laszewski

n

Indiana University

•

Marlon Pierce, Dennis Gannon, Geoffrey Fox, and

Beth Plale

n

University of Michigan

•

Charles Severance, Joseph Hardin

n

NCSA/UIUC

•

Jay Alameda, Joe Futrelle

n

Texas Advanced Computing Center

•

Jay Boisseau

n

San Diego State University

(3)

Presentation Outline

n

OGCE Overview

n

Portlet Standards for OGCE

•

New JSR 168 compatible Grid portlets

•

Demonstrate compatibility with uPortal and

GridSphere containers

n

Grid Portlet Programming

(4)

What a Grid Portal Is/Is Not

n

It is

•

A tool for aggregating and managing web content

•

A user customizable view of these Web content pieces.

n

You see what you want/can see.

n

But you must log in.

•

Implemented on top of standard services

n

Like login, authorization, customization.

n

May include collaboration, etc, that depend on login.

•

A way to accomplish Grid tasks through browsers:

n

Launch, monitor jobs

n

Move files

n

Run science applications based on these services.

•

Compatible with emerging standards and best practices (such as

portlets, JSR 168 and WSRP).

n

It is not (just)

•

A web page

•

A collection of links

(5)

Port

al

User

Int

erf

ace

Grid Resource Broker Service Grid and Web Protocol s Information and Data Services Database Service Database HPC or Compute Cluster Grid Information Services, SRB Portal Client Stub Portal Client Stub Portal Client Stub JDBC, Local, or Remote Connectio n

Three-Tiered Architecture

(6)

Building Portals from Reusable

Components

n

The component architecture of choice for the Portal

community is the one based on

portlets

•

(Java) components that generate content, make local and

remote connections to services.

•

Portlet containers

manage portlet lifecycles

•

Standardized now by

JSR 168

n

A portlet is a piece of Java code similar to a servlet that

does two things:

•

Creates a fragment of HTML display that becomes part of a web

page

•

Handles any link clicks and HTML <Form> actions.

n

May involve accessing local and remote services

n

There are now many, many portlet components.

•

So don’t start from scratch.

(7)

OGCE’s Release 1

n

The OGCE Portal release is based on CHEF/Jetspeed

1.4

n

Available for download and installation from

h

ttp://www.collab-ogce.org.

n

I

t comes with many pre-configured capabilities if you

want a grid portal “out of the box”.

•

Except for the mysql jar.

•

You must still set up Grid services (MyProxy servers, Globus,

etc).

•

Globus version compatibility through the Java CoG.

n

Apache Ant-based installation procedure:

(8)

Access context services for managing metadata GridContext Portlets

Access to Anabas shared display applets Anabas

View, interact with HPC status, job, etc information. GPIR Portlets

Run simple executables on remote hosts GRAM Job Submission

Live chat services and interfaces Chat

Persistent topic-based discussion for groups Discussion

Interactive individual and group calendars Schedule

WEBDav based document system for group file sharing

Document managers

Post topics to newsgroup, manage group references and citations with access controls Newsgroups and citation portlets

Upload, download, crossload remote files. GridFTP

Basic Globus MDS browsing and navigating MDS/LDAP Browsers

Get MyProxy certs after logging in Grid Proxy Certificate Manager

Description

Portal Capabilities

(9)

Set up and run task graphs using the Java CoG CoG Workflow demonstration portlet

The backend for the Grid Context portlet XDirectory Services

Combine GridFTP and GRAM into application wizard forms.

Application Management

Interact with Condor through browser. Condor Portlets

Download and install server side of the OGCE newsgroup system.

Newsgroup Services

Manage complicated grid tasks through an extensible, Apache Ant like task list.

OGRE Job Management Services

Schedule sequences of jobs on several hosts using Community Scheduling Framework.

Job Scheduling and Sequencing

Description

(10)

What’s New for the OGCE2?

n

JSR 168

Compatible Grid portlet suite

•

Basic capabilities: MyProxy, GridFTP, GRAM, GPIR.

•

Working in uPortal, GridSphere containers

n

Container independent

services for sharing data between portlet

applications.

•

GSSCredential objects, global session data.

•

Limitation in JSR 168

n

Support for

Velocity

development

•

Velocity is the Apache web application development tool of choice for

Jetspeed1.

•

Provide backward compatibility with OGCE1

n

Maven-based

build and deploy system

•

Choose either uPortal or GridSphere

•

One-command install, but you must still install a Grid toolkit.

n NMI GRID Center: www.grids-center.org

n

Modular

portlet extensions

(11)

(12)

Grid Portlet

Programming

(13)

Grid Programming Interfaces for

Portlets

n

Portlet form actions may result in remote calls to

Grid or Web Services

•

A Portlet is just java code

n

We use two programming APIs you can choose

from

•

CoG4

API provides abstraction of Grid tasks to hide

Grid toolkit version differences.

•

GridPort

provides integrated information, CSF-based

(14)

Office of Science

U.S. Department of Energy

Java CoG Kit and Portals

• The Java CoG Kit is a bridge between Grids and

Portals as it provides an abstraction layer that is

supportive for portals developers.

• The Goals of the Java CoG Kit include reuse of a

variety of commodity tools, protocols, approaches,

methodologies, while integrating Grid software to

enable

–

Easier development of advanced Grid services

–

Easier and more rapid application development

–

Easier deployment of Grid services

–

Code reuse and use of component repositories

–

Use of Web services as part of the Grids

–

Widespread use of the Grid

–

Use of commodity technology is not limited to the client.

(15)

Java CoG Kit Abstractions

• We provide a number of abstractions that build

the foundation why we make Grid

programming easier:

–

File transfer, job submission, authentication

• We provide a workflow abstraction that makes

the specification of Grid workflows possible

helping those with complex job management

scenarios.

• We provide a software methodology that

makes adoption to new standards easier

–

We defined so called providers that allow reusing

various Grid and commodity services including

(16)

GT2 OGSA/GT3

WS-RF SSH Condor

Unicor e

CoG Abstraction Layer

(job submission, file transfer,

authentication)

CoG CoG CoG CoG CoG

Applications

Data and Task Management Layer

(workflow)

Gridfaces Layer

(portals, Swing, SWF)

Others Avaki CoG Others SETI CoG

Nano

materials

Informatics

Bio-Disaster

Managemen

t

Portals

G

ridI

DE

Development

Support

(17)

Java CoG Kit information

• More information on the Java CoG Kit can

be found at

http://www.cogkit.org

or visit

the posters at SC2004

§

P09: Karajan: A Grid Orchestration Framework:

Mihael Hategan, Gregor von Laszewski, Kaizar

Amin

§

P11: The Next Generation of the Java CoG Kit

(version 4)

Gregor von Laszewski, Kaizar Amin, Matt Bone,

Mike Hategan, Pankaj Sahasrabudhe, Mike

Sosonkin, Robert Winch

(18)

What is GridPort?

• High-level middleware that aids grid portal

developers by easing use of low level grid tools

• GridPort’s Role in Grid Computing

–

Aggregate services from grid software packages

• Globus Toolkit

• Community Scheduling Framework (CSF)

• Storage Resource Broker (SRB)

–

Simple, consistent API

–

Custom Web services

• GPIR

• Job Sequencer

• Advanced File Transfer

(19)

Web Services Portlets J2EE App Server Contains: Spring, Hibernate Web

Client HTTP /_SSL

PostgreS QL J2EE Web Server GPIR Job Sequen cer File Transfe r Job Submis sion Data Access G S I S e c u ri ty

G

ridP

ort

Service

AP

I

J D B C

GridFT

P

SRB

OGSI

GRAM

WS-GRAM

CSF

Grid Services Job Sequen cer Adv. File Transfe r GPIR Adv. File Transfe r Grid Data H T T P / S O A P JSP/Servle t GPIR Admin Client

GridPort’s 3-tier Architecture

(20)

GPIR

PostgreSQL

Information

Providers

Resources

DB

Clients

Web Scraping Java Clie nt Ingester

WS QueryWS

Portals Other Middlew are SOAP-XML HTTP Perl Clie nt

Portle

ts

edu.tacc.

gridport.gpir

JDBC

GPIR Web Services

MDS

OGSA

(Future)

(21)

GridPort Job Sequencer

• Web service for simple job workflow

composition

• Portlet interface available for specifying

sequences

• Specify job submissions and file transfers

• More information available at

(22)

CSF Integration

• Community Scheduling Framework (CSF)

• Meta-scheduler for submitting jobs to a grid

–

No need to specify a resource

–

CSF will schedule jobs to appropriate resource for

you

• OGSI grid service

• CSF is an open source project developed by

Platform computing

–

Now available for download from Sourceforge.net

(23)

More GridPort Information

GridPort Demo in TACC Booth 120

2:00-2:20 PM

(24)

TeraGrid User Portal

Eric Roberts

(25)

Motivation

n

Make joining the TeraGrid easier for users

n

Single place for users to find user information

and get user support

n

Certain information can be displayed better in a

web page than in a command shell

n

Allow novice users to start using grid resources

securely through a Web interface

n

Increase productivity of TeraGrid researchers –

(26)

TeraGrid User Porta

Service Aggregation

(27)

Current TeraGrid User Portal

Capabilities

n

User Services

•

User Information (User Guides, Support,etc.)

n

Information Services

•

System

•

Grid

•

Network

n

Interactive

•

GSI Authentication

•

Remote Command Execution

•

Job Submission

(28)

Future Directions

n

Central gateway for TeraGrid services

n

TeraGrid allocations and account

creation/management through portal

•

Streamline the process

n

Application portals

•

Science gateways that expose scientific

applications through interfaces

n

We will be contacting users for

(29)

LEAD Application Portal

(30)

L

inked

E

nvironments

for

A

tmospheric

(31)

L

inked

E

nvironments

for

A

tmospheric

D

iscovery

The LEAD Goal

Provide the IT necessary to allow

People

(scientists, students, operational

practitioners)

and

Technologies

(models, sensors, data

mining)

(32)

L

inked

E

nvironments

for

A

tmospheric

D

iscovery

Analysis/Assimilatio n Quality Control Retrieval of Unobserved Quantities Creation of Gridded

Fields

Prediction/Detection PCs to Teraflop

Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students

Traditional Methodology

STATIC OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting

Satellite Wind Profilers GPS Satellites

(33)

L

inked

E

nvironments

for

A

tmospheric

D

iscovery

(34)

L

inked

E

nvironments

for

A

tmospheric

D

iscovery

Analysis/Assimilatio n Quality Control Retrieval of Unobserved Quantities Creation of Gridded

Fields

Prediction/Detection PCs to Teraflop

Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students STATIC OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting