Using Globus Toolkit

(1)

G. Poghosyan & D. Nilsen

GridKa School 11-15 September 2006

Using Globus Toolkit

(2)

• Security Services

– GSI (Grid Security Infrastructure)

• Data Services

– GridFTP

– RFT (Reliable File Transfer)

– RLS (Replica Location Service)

• Execution Management Services

– GRAM (Grid Resource Allocation and Management)

• Information Services

– MDS (Monitoring and Discovery System)

(3)

G.Poghosyan&D.Nilsen GridKa School, 11-15 September 2006, Karlsruhe, Germany 3

• Based on

– Public key cryptography

– X.509 certificate (grid-cert-request)

– Very sensitive to system clock skew – use NTP

• Provides

– Mutual authentication

– Single sign-on

Grid Security Infrastructure (GSI)

(4)

Grid Security Infrastructure (GSI)

Certificate/Proxy - Single Sign-on

Finding information on a certificate

grid-cert-info –subject

-startdate -issuer

-enddate -file certfile

Finding information on a proxy

grid-proxy-info [-exists]

-debug

Creating a proxy

grid-proxy-init –verify

-q -valid h:m -debug

Default value is –valid 12:00

Destroying a proxy

grid-proxy-destroy

-dryrun [file1…]

-

Athorization

(5)

Certificate/Proxy - Single Sign-on

• Gives information about the proxy

– $ grid-proxy-info

• Create proxy at /tmp/x509up_u<USER ID>

– $ grid-proxy-init -debug -verify

• Gives information which proxy will be destroyed

– $ grid-proxy-destroy -dryrun

• Destroy proxy certificate

(6)

Mutual Authentication - MyProxy

Storing a credential in a repository

myproxy-init [-s hostname]

[-c credlifehours]

[–t proxylifehours]

Default values are

–c 168 (7 days)

and –t 12

Display info on stored credentials

myproxy-info [-s hostname]

Retrieving a credential

myproxy-get-delegation

(myproxy-logon)

[-s hostname] [-t hours]

-

Athorization

-

Job

Removing a credential

myproxy-destroy

[-s hostname]

(7)

Mutual Authentication - MyProxy

• Send myproxy certificate to MyProxy-Server

– $ myproxy-init -s xen-14-2.fzk.de

• Getting myproxy certificate

– $ myproxy-get-delegation -s xen-14-2.fzk.de

• Destroy proxy certificate

– $ myproxy-destroy -s xen-14-2.fzk.de

• Set enviroment

(8)

Data Services (GridFTP)

Data Services

• Handle movement, access,

replication and updates of data

• Example services in Globus Toolkit

– GridFTP

– Reliable File Transfer (RFT)

(9)

Data Services - GridFTP

Basic tool provided with Globus Toolkit

globus-url-copy [-tcp-bs TCPBufSizeBytes] [-p parallelism]

sourceURL destURL

URL format =

protocol://[host]:[port]/path

Supported protocols:

gsiftp://, file://

(on a local machine only)

, ftp://, http://, https://

It is legal to specify a user name and password in the the URL as follows

gsiftp://myname:[mypassword]@myhost.mydomain.com/file.dat

(10)

Data Services - GridFTP

local Ä local

$ globus-url-copy

gsiftp://localhost/etc/hosts file:///tmp/hosts_copy

remote Ä local

$ globus-url-copy

gsiftp://iwrcggt4.fzk.de/etc/hosts file:///tmp/hosts_copy_<IhrName>

local Ä remote

$ globus-url-copy

file:///tmp/hosts_copy_<IhrName> gsiftp://iwrcggt4.fzk.de/tmp

(11)

Gevorg Poghosyan and Dimitri Nilsen

GridKa School, 11-15 September 2006, Karlsruhe, Germany 11

Data Services - Reliable File Transfer (RFT)

• Manages a set of third-party GridFTP

transfers

• Uses a database to checkpoint transfer state

• Recovers from

– Source/destination server failures

– Network failures

– Container failures

• Transfers retried with exponential backoff

• Resumes transfers where they left off

(12)

Gevorg Poghosyan and GridKa School, 11-15 September 2006, Karlsruhe, Germany 12

Data Services - Reliable File Transfer (RFT)

Simply pass it a file with options at the top, and then

source / destination URL pairs.

rft –f my.xfr

rft-delete –f filename

Other options

-h host (defaults to localhost)

-r port (defaults to 8080)

-l lifetime (default 60 minutes)

-m security (msg, conv, trans (default))

-p protection (sig, enc (default))

-z authorization (self, host (default))

-file file to write epr to

(13)

Data Services - Reliable File Transfer (RFT)

my.xfr

#true=binary mode false=ascii

true

#Block size in bytes

16000

#TCP Buffer size in bytes

16000

#No thirdPartyTransfer (Notpt)

false

#Number of parallel streams

1 #Data Channel Authentication

true

Data Services (RFT)

# Concurrency of the request

1 #Grid Subject name of the source gridftp server

null

#Grid Subject name - destination gridftp server

null

#Transfer all or none of the transfers

false

#Maximum number of retries

10 #Source/Dest URL Pairs

gsiftp://iwrcggt4.fzk.de/etc/group

(14)

Data Services - Reliable File Transfer (RFT)

• Get .xrf file

– $ cp $GLOBUS_LOCATION/share/globus_wsrf_rft_test/ \

transfer.xfr /tmp

• Edit .xfr file and create files to transfer

– $ vi /tmp/transfer.xfr

– $ touch /tmp/rftTest.tmp

• RFT-Test

(15)

Data Services - Replica Location Service (RLS)

• Co-developed by Globus and DataGrid projects

• Provides a mechanism for registering and

locating replicas

• Maintains mappings between logical names and

target names

– Logical names – unique identifiers for data content that

may have one or more physical replicas

– Target name – physical location of a replica or other logical

names representing the data

(16)

Execution Management Services

• Handles placement, provisioning and lifetime

management of jobs

• Example services in Globus Toolkit

– Grid Resource Allocation and Management (GRAM)

• Provides a single standard interface for remote job

submission and resource management

• Requires users and application developers to learn only one

method to gain access to a large variety of local

management systems

(17)

Pre-WS GRAM (GT2) Command Line Tools

Interactive job submission

globus-job-run host:port/jobmanager-

pbs

command

Batch job submission

globus-job-submit command

Checking job status when done output

globus-job-status jobID

globus-job-

output

–out|err jobID

Kill a job

globus-job-cancel jobID

(18)

WS GRAM (GT4) Command Line Tools

Interactive job submission

globusrun-ws –submit –F JobFactoryURL

–Ft FactoryType –s –q –c command

Batch job submission

globusrun-ws –submit –batch –F JobFactoryURL

–Ft FactoryType –o EPRfile –c command

Checking job status

globusrun-ws –status –j EPRfile

Kill a job

globusrun-ws –kill –j EPRfile

(19)

WS-GRAM

• WS oriented job submission command(GT4)

– $ globusrun-ws -submit -c /bin/hostname

• Streaming the output

– $ globusrun-ws -submit -s -c /bin/hostname

• output go to file

– $ globusrun-ws -submit -s -so out.txt -c /bin/hostname

• Send job to remote host

– $ globusrun-ws -submit -s -F <host> /bin/hostname

• Send to the batch system (pbs)

– $ globusrun-ws -submit -s -F iwrcggt4.fzk.de -Ft PBS -c

/bin/hostname

(20)

WS-GRAM: End point reference

• End point reference

– globusrun-ws -submit -batch -c /bin/hostname

• createa batch job

– $ vi batchJob.sh

#!/bin/bash

sleep 10

$chmod +x batchJob

• create an end point ree file ferenc

– $ globusrun-ws -submit -batch -o epr1.file \

-c $HOME/batchJob.sh

• get status

(21)

Execution Management (WS-GRAM)

WS-GRAM

GRAM

services

GT4 Java Container

GRAM

services

Delegation

RFT File

Transfer

request

GridFTP

Remote

storage

element(s)

Local

scheduler

User

job

Compute element

GridFTP

sudo

GRAM

adapter

FTP

control

Local job control

Delegate

FTP data

Client

Job

fun

ctio

ns

Delegate

SEG

Job events

(22)

Information Services

• Provide access to information about

applications, resources and services in

a grid environment

• Example services in Globus Toolkit

– Monitoring and Discovery System (MDS)

• Used to monitor and discover resources and

services that exist in a grid environment

• MDS2 – Pre-Web Service version of MDS (GT2)

• MDS4 – Web Service based version of MDS (GT4)

(23)

MDS2

• Uses Lightweight Directory Access Protocol

(LDAP) to access this information

• Grid Resource Information Service (GRIS)

– Provides resource information

• Grid Index Information Service (GIIS)

– Provides caching services and search capabilities

• GRIS can register with a GIIS

• GIIS can register with other GIIS to create

hierarchy

(24)

MDS4 (WS MDS)

• Includes two WSRF-based services

– Index Service

• Collects data and provides a query/subscription interface to

the data

• Can create hierarchy of index services

– Trigger Service

• Collects data and takes actions based on the data

wsrf-query –s

https://iwrcggt4.fzk.de:8443/wsrf/services/DefaultIndexServic

e '/*‘

mds-servicegroup-add [options] config.xml

(25)

Information Services (MDS4)

Good Luck in looking-for

useful commands and services

in