Application enablement & Applications on Grid

Full text

(1)

Application enablement &

Applications on Grid

Divya MG

(divyam@cdac.in)

(2)

Presentation Overview

• Nature of applications

(3)

3

Objectives

– Enable applications of national importance requiring

aggregation of geographically distributed resources

Deliverables

– Grid enablement of illustrative applications

• Disaster Management • Bioinformatics

• Geomatics

• Earthquake Research

• Weather and Climate Modeling • Evolutionary Computing

(4)

Grid Enabling an application?

• To enable applications is to take

advantage of the potential of grid

computing

• This can be done by understanding

the nature of applications

Class of Grid available

(5)

5

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand the Grid

Understand nature of application What does the appln need

Where do I execute my appln How do I submit the job

Possible states of my job

How is my job doing

(6)

6

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand nature of application What does the appln need

Where do I execute my appln How do I submit the job

Possible states of my job How is my job doing How can I control my job

(7)

Understand Grid

• Compute grid – multiple computers to solve one

application problem.

Eg: SETI@Home/Folding@Home, TeraGrid, Garuda Grid, AusGrid, etc

• Data grid – multiple storage systems to host one

very large data set.

Eg: LHC Grid

(8)

8 Pune Submit node (gridfs machine) Cluster Head Node Compute Nodes Hyderabad Linux Bangalore Grid Portal Cluster Head Node Cluster Head Node Bangalore Linux Chennai Cluster Head Node Cluster Head Node IIT, Guwahati Compute Nodes

(9)

9

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand the Grid

What does the appln need

Where do I execute my appln How do I submit the job

Possible states of my job

How is my job doing

How can I control my job

(10)
(11)

Parallel Application Flow

1. Data on each node

independent of data on other nodes

2. Same task executed on each dataset

3. Several jobs can all be executed in parallel

4. Very limited or no data exchange among the jobs. 5. Less communication

6. More computation

(12)

12

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application

Where do I execute my appln How do I submit the job

Possible states of my job

How is my job doing

How can I control my job

(13)

13

Applications and Requirements

High Computation Power

Large Memory

Large Database

High Speed Networks

High collaboration Applications

• Climate Modeling • Applied Life Science • Quantum Chemistry • Bio Informatics

• Astrophysics

– e.g., Simulations of a Supernova Explosion or – Black Hole Collision

(14)

14

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application What does the appln need

How do I submit the job

Possible states of my job

How is my job doing

How can I control my job

(15)

Resource Selection

• Based on where the code has been compiled, select the

resources

(16)

16

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application What does the appln need

Where do I execute my appln

Possible states of my job

How is my job doing

(17)

Job Submission

• Prepare a job template for Gridway Job

submission

Sample Gridway Job template .jt file:

(18)

18

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application What does the appln need

Where do I execute my appln

How do I submit the job

How is my job doing How can I control my job

(19)

Job Life Cycle

• Pending (pend): The job is waiting for a resource to run on.

• Prolog (prol): The job is preparing the remote system, by creating the execution directory in the remote host and transferring the input files

• Wrapper (wrap): The job is executing the Wrapper, which in turns executes the actual application.

• Done (done): The job is done and the user can check the exit status.

• Failed (fail): The job failed.This state is seen when the job

(20)

20

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application What does the appln need

Where do I execute my appln How do I submit the job

Possible states of my job

(21)
(22)

22

What you need for enabling

Application on the Grid

• Grid Architecture

• Application Requirement • Resource Preparation • Resource Selection • Job Submission • Job Life Cycle • Job Monitoring • Job Control

Understand Grid

Understand nature of application

What does the appln need Where do I execute my appln How do I submit the job

Possible states of my job How is my job doing

(23)

Job Control

• The jobs can be controlled to synchronize by

giving “gwwait” command. It will hold the

jobs submitted

(24)

Flow for Grid-Enabling

Analysis of requirements Does it fit on The Grid Grid-enabling Procedure Successful ? Benchmarking OK ? Production Runs

DOCUMENTATION AND REPORTS

Review the analysis

Review the procedure

(25)
(26)

• DMSAR is a RADAR* based Airborne earth imaging

system through which the real time situations of any

natural and man induced disasters can be mapped even in rough weather conditions

• Collaboration aspects. Coordination between domain

experts at geographically distributed locations

• Near real time processing

• 9 GB DMSAR data was processed in less than an hour.

Earlier it took nearly 7 hours on a 32-way SMP machine

• Redundant resources for fault tolerance

• Remote geospatial visualization between locations

DMSAR

(27)

Taxogrid

Benchmarking on GARUDA Grid

Resources • gg-hyd.cdac.org.in • gg-che.local • gg-blr.tfg • xn00.npsf.cdac.ernet.in • yuva.pnq-cdac-hq-100.garuda Software Versions Globus – 4.0.7 GridWay – 5.4 DRMAA - 1 Dataset Database: UniProtKB Mycobacterium tuberculosis H37Rv No. of sequences: 3988

Size of input dataset: 1.65 MB Size of output file: 10.4

Method : Maximum Likelihood

TaxoGrid Benchmark (Time Scale)

0 200 400 600 800 1000 1200 1400 5 10 15 20 Number of cores T im e ( in m in u te s )

TaxoGrid Benchmark (Speed-up)

1.00 1.76 2.72 3.31 0 0.5 1 1.5 2 2.5 3 3.5 0 5 10 15 20 25

No. of pipelines (160 cores)

Sp e e d -u p

Time for speedup at value ‘1’ : 1144

minutes

(28)

Reduces aerodynamic drag by altering the flow field near tip

Can convert some of the otherwise wasted energy in wing tip vortex to an apparent thrust

Improved take-off characteristics (but design optimized for it may pay penalty for cruise conditions & may be vice versa)

Can improve aircraft handling characteristics (increased roll rate & roll authority)

Reduction in noise levels is also noticed

Objective of the Study:

“To design a blended winglet for RTA wing configuration with minimum drag at cruise Cl of 0.55”

(29)

OSDD-GARUDA

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN

Internet / NKN

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN Internet / NKN OSDD User Community Grid Enabled Bioinformatics tools useful in drug discovery pipeline

Internet / NKN

Internet / NKN

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN

Internet / NKN

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN Internet / NKN OSDD User Community OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN

Internet / NKN

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN Internet / NKN OSDD User Community OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN

Internet / NKN

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB OSDD HeadNode Garuda Middleware Stack, login service, Gridway Metascheduler OSDD Customized Galaxy DB Ext DB GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

GGHYD Cluster Yuva Cluster JNU Cluster Other OSDD Cluster GARUDA Grid

Garuda Middleware Stack LRM- Torque

OSDD Tools – weka, cdk,…

NKN Internet / NKN OSDD User Community Grid Enabled Bioinformatics tools useful in drug discovery pipeline

Internet / NKN

Internet / NKN

• JNU, IGIB, UoH, IISc have

NKN connection to GARUDA; for remote

locations like MCC, GARUDA made accessible through the

Internet.

• HPC clusters to run drug discovery problems

• Galaxy Workflow for genomics proteomics applications

• Parallelizing specific applications

(30)

Benefits of Grid Enabling

PSP

PSP

Reduction in Computational Complexity

• Exploit concurrency,

speedup

&

fault tolerance

1TUP molecule (219AAs) Population

size

# Generations # Function evals

Time taken for sequential approach in minutes (219AA)

Time taken using DGAG approach in minutes (9 parts:30AA/ part)

Speedup

100 600 50000 600 (10hrs) 19 31.6

300 1800 450000 4560 (76hrs) 180 (3hrs) 25.3

•Time using single CPU – 10 hrs (100 population size) and 76hrs (200 population size) •Time taken using Grid – 19mts (100 population size) and 3hrs (200 population size)

Computational complexity for full molecule Speedup =

(31)

Conclusions

• Grid is more suitable for applications where the data is complex or multiple runs of the same task to be performed.

• A loosely coupled application spawned across the Grid could exploit the Grid resources to the

maximum

• Speedup > 25 times was obtained using Grid

(32)

References

• Gabrielle Allen et al.,”Classifying and

enabling grid applications” ,Concurrency—

practice and experience 2000; 00:1–7

(33)

Figure

Updating...

References

Updating...

Related subjects :