EGEE, Miroslav Ruda CESNET. EGEE is a project funded by the European Union under contract IST

24 

Loading....

Loading....

Loading....

Loading....

Loading....

Full text

(1)

www.eu-egee.org

EGEE, 26.10. 2004

Gridy a souvisej´ıc´ı aktivity v ˇ

CR

Miroslav Ruda CESNET

(2)

www.eu-egee.org

Gridov ´e projekty v ˇ

CR

MetaCentrum

GridLab

CoreGrid

DataGrid

EGEE

DO, LCG

(3)

www.eu-egee.org

MetaCentrum

od roku 1996 jako projekt programu TEN-34

• zpˇr´ıstupn ˇen´ı prvn´ıch superpoˇc´ıtaˇc ˚u v ˇCR vˇsem uˇcitel ˚um a v ˇedeck´ym

pracovn´ık ˚um vysok´ych ˇskol a Akademie v ˇed

• Z ˇCU Plze ˇn, UK Praha, ˇCVUT Praha, MU Brno, VUT Brno

• SGI PowerChallenge, Digital Alpha, IBM SP2

od roku 1999 souˇc ´ast v´yzkumn ´eho z ´am ˇeru sdruˇzen´ı CESNET

• Z ˇCU Plze ˇn, UK Praha, MU Brno

• pozd ˇeji V ˇSB Ostrava

• SGI, Digital, IBM

• od roku 2000 clustery s architekturou IA32 a operaˇcn´ım syst ´emem Linux

(4)

www.eu-egee.org

MetaCentrum – hardware

IA32 clustery (Pentium 3 a Pentium 4 Xeon)

• 64+68CPU, + 64 CPU vyhrazen´ych pro EGEE, + 72 CPU se pr ´av ˇe instaluje

• 48 CPU s Myrinetem, zbytek giga-ethernet, 1-2GB RAM na 2 CPU

• NCBR v Brn ˇe (32+104 CPU), ITI v Plzni (32 CPU), FI MU (10 CPU)

2x SGI Origin, 2x Power Challenge – 120 MIPS CPU, 34GB RAM

Digital – 8x Alpha EV6

dvouprocesorov´y AMD Opteron 244, 3.5GB RAM

dvouprocesorov´y Intel Itanium 2 1000MHz, 6GB RAM

dvouprocesorov´y IBM Power4+ 1.2GHz, 8GB RAM

(5)

www.eu-egee.org

MetaCentrum – software

Kerberos, AFS, d ´avkov ´e syst ´emy LSF, NQE, PBS(Pro)

v´yvojov ´e prostˇred´ı

• PGI a Intel

• Vampir Trace a Debugger, Totalview

• PVM, MPI (MPICH, MPICH-GM, LAM)

Matlab, Maple, Mathematica (Z ˇ

CU)

Fluent, Ansys, Marc

(6)

www.eu-egee.org

MetaCentrum – v ´yvoj

bezpeˇcnost

• Kerberos a PKI

• hardwarov ´e tokeny

integrace cluster ˚u

• spr ´ava, d ´avkov ´e syst ´emy, pl ´anov ´an´ı, monitoring

port ´aly

Didas

• velk ´a (15 TB) ´uloˇziˇst ˇe dat pˇripojen ´a pˇr´ımo do Gridu

(7)

www.eu-egee.org

Gridlab

motivace

• n ´aroˇcn ´e v ˇedeck ´e v´ypoˇcty, pˇresahuj´ıc´ı moˇznosti jednoho

superpoˇc´ıtaˇce/clusteru

• skupiny ´uloh se z ´avislostmi (workflow)

Grid Application Toolkit

GridSphere – prostˇred´ı pro vytv ´aˇren´ı port ´al ˚u

nov´y middleware pro pl ´anov ´an´ı ´uloh,

sledov ´an´ı stavu gridu

,

informaˇcn´ı sluˇzby, vizualizace, podpora mobiln´ıch uˇzivatel ˚u

(8)

www.eu-egee.org

Gridlab – Architektura

U se r S p a c e C a p a b ilit y S p a c e Application Layer GAT Layer Service Layer Core Layer

GSI-enabled Web Service API

Third Party Services and Libraries GAT API GridLab Services GRMS Delphoi Replica Catalog Mercury Mobile Services Data Movement Viz Service

GridLab Architecture

C and Java APIs to Globus 2.X/3.X Pre-WS and other Core Services

...

ASTROPHYSICS BIOINFORMATICS OTHER LIFE SCIENCES

GridLab Testbed and Infrastructure Gridsphere Portal

Mobile clients

www.gridlab.org

(9)

www.eu-egee.org

Gridlab – Grid Application Toolkit

GAT API – neutr ´aln´ı, jednoduch ´e API pro gridov ´e sluˇzby

GAT Engine – modul ´arn´ı design, v´ym ˇena adapteru za b ˇehu

GAT Adaptors – pˇrenositelnost, podpora r ˚uzn´ych prostˇred´ı

• Gridlabov ´e sluˇzby

• prostˇred´ı Globus a Unicore

• lok ´aln´ı adaptory

urˇceno pro aplikaˇcn´ı v´yvoj ´aˇre

• Soubory, trubky

• Monitorov ´an´ı, ud ´alosti

• Ulohy, zdroje´

(10)

www.eu-egee.org

Gridlab – Testbed

urˇcen´y pro v´yvoj middleware a aplikac´ı, ne pro produkˇcn´ı v´ypoˇcty

stroje/clustery nejsou dedikovan ´e pro testbed

v souˇcasn ´e dob ˇe 25 stroj ˚u

• Evropa=partneˇri v Gridlabu

• LSU v USA, KISTI v Korei

heterogenn´ı hardware a syst ´emy

• Linux - IA32, AMD64, Itanium, G4

• Irix, AIX, Solaris, Hitachi

r ˚uzn ´e d ´avkov ´e syst ´emy (PBS, SGE, Condor, CCS)

+ Globus 3.2 + Gridlab software

(11)

www.eu-egee.org

Gridlab – Testbed monitoring

modul ´arn´ı architektura, r ˚uzn ´e prog. jazyky, v´ysledky v datab ´azi

• poˇc´ıtaˇce - dostupnost, Globus, MPI testy, . . .

• sluˇzby - jednoduch ´e testy i kompletn´ı unit testy

• instalovan´y software - dostupnost, funkcionalita

• gridlab software - noˇcn´ı kompilace a unit testy naˇseho software

upozorn ˇen´ı spr ´avc ˚um stroj ˚u/software pˇri probl ´emech

v´ysledky na webu i pˇres programov ´e API pro dalˇs´ı sluˇzby

uˇzivatel m ˚uˇze spustit test pod svoj´ı identitou

worm – aplikace migruje po testbedu, prov ´ad´ı testy lok ´aln ˇe

(12)

www.eu-egee.org

(13)

www.eu-egee.org

CoreGrid

ˇcist ˇe v´yzkumn´y projekt – S´ıˇt excelence (NoE) v r ´amci 6RP

ˇsest v´yzkumn´ych t ´emat

• knowledge & data management

• programming models

• system architecture

information and monitoring services

• model zahrnuj´ıc´ı monitorovac´ı i informaˇcn´ı sluˇzby

• ˇsk ´alovatelnost pro gridy s miliony uzl ˚u, robustnost, P2P inspirace

• ´udaje o zdroj´ıch, sluˇzb ´ach, ud ´alostech; filtrov ´an´ı; vyhled ´av ´an´ı

• resource management and scheduling

(14)

www.eu-egee.org

DataGrid

Middleware

Work Scheduling • Data Management • Monitoring services • Fabric Management • Storage Management

Integration Testbed & Support

Network

Applications

• High Energy Physics

(15)

www.eu-egee.org

Datagrid – Work Scheduling

pl ´anov ´an´ı velk ´eho mnoˇzstv´ı mal´ych ´uloh, zpracov ´avaj´ıc´ıch velk ´e

objemy dat, data jsou rozprostˇren ´a po Gridu

r ˚uzn ´e typy ´uloh

• d ´avkov ´e vs. interaktivn´ı

• sekvenˇcn´ı vs. paraleln´ı (MPI), partitionable

• uˇzivatelsky checkpointovateln ´e (a restartovateln ´e)

• z ´avislosti mezi ´ulohami (workflow) popsan ´e acyklick´ymi grafy

vyuˇz´ıv ´a syst ´emy Globus a CondorG

JDL – classads form ´at pro popis ´ulohy, z ´avislost´ı

mezi ´ulohami i vlastnost´ı v´ypoˇcetn´ıch uzl ˚u

(16)

www.eu-egee.org

Logging & Bookkeeping Service

souˇc ´ast pl ´anovaˇce ´uloh i obecn ´eho monitoringu gridu

odpov´ıd ´a na dotazy o stavu ´ulohy

syst ´em zaloˇzen´y na ud ´alostech, kter ´e pos´ılaj´ı r ˚uzn ´e

middlewarov ´e komponenty

jedin ´e persistentn´ı ´uloˇziˇst ˇe dat o b ˇeˇz´ıc´ı ´uloze, jej´ım stavu

online pˇrepoˇc´ıt ´av ´a stav ´ulohy – ˇcek ´a (proˇc, kde),

b ˇeˇz´ı (kde), skonˇcen ´a (kde, n ´avratov´y k ´od) . . .

uˇzivatelsk ´e tagy – pˇri zad ´an´ı ´ulohy nebo i za b ˇehu

bezpeˇcn ´a, neblokuj´ıc´ı sluˇzba, velk ´e vyhled ´avac´ı moˇznosti

(17)

www.eu-egee.org

CA Cesnet

kaˇzd´y st ´at zapojen´y v projektech DataGrid a CrossGrid m ´a CA

vz ´ajemn ´a kontrola politik mezi pˇredstaviteli evropsk´ych CA

CESNET CA

• moˇznost z´ıskat certifik ´at uzn ´avan´y ve v ˇetˇsin ˇe evropsk´ych projekt ˚u

• certifik ´at m ˚uˇze z´ıskat kaˇzd´y student/zam ˇestnanec vysok ´e ˇskoly a AV ˇCR

• moˇznost zˇr´ıdit Registraˇcn´ı Autoritu (ve v´yvoji)

(18)

www.eu-egee.org

EGEE

organizaˇcn´ı struktura

• NA1: Project Management

• NA2: Dissemination and Outreach

NA3: User Training and Induction

NA4: Application Identification and Support

• NA5: International Cooperation

SA1: European Grid Support, Operation Management

• SA2: Network Resource Provision

JRA1: Middleware Reengineering + Integration

• JRA2: Quality Assurance

• JRA3: Security

(19)

www.eu-egee.org

EGEE – JRA1

JRA1: Information and Monitoring

• R-GMA

JRA1: Data Management

• Storage Element • Catalog Services • Data Scheduling

Workload Management

• Workload Manager • Grid Accounting

• Logging and Bookkeeping

(20)

www.eu-egee.org

EGEE – Logging & Bookkeeping

rozˇsiˇrov ´ani Bookkeeping sluˇzby

• stabilita, ˇsk ´alovatelnost

• notifikace

• web-service interface

Job Provenance

• trval ´e ´uloˇziˇst ˇe dat pro pozd ˇejˇs´ı dotazy, data mining

• informace dostateˇcn ´e pro zreprodukov ´an´ı v´ypoˇctu

intergace s R-GMA

• jednotn ´a monitorovac´ı infrastruktura

(21)

www.eu-egee.org

EGEE – SA1 (I.)

EGEE testbed – certifikace software, instalace, provoz, podpora

uˇzivatel ˚u, virtu ´aln´ıch organizac´ı

• produkˇcn´ı testbed – nyn´ı postaven´y na software LCG2 (Datagrid), stabiln´ı

podporovan ´a verze, siln ´a (aˇz 24/7) uˇzivatelsk ´a podpora

• pre-produkˇcn´ı testbed – testov ´an´ı nov ´e verze software (gLite)

• demonstraˇcn´ı testbed – pro potˇreby ˇskolen´ı, demonstrac´ı

• certifikaˇcn´ı testbed

(22)

www.eu-egee.org

EGEE – SA1 (II.)

organizaˇcn´ı struktura

• Operations Management Centre (CERN)

• Core Infrastructure Centres (CERN, France, Italy, UK + Russia, Taipei, US)

• Regional Operations Centres (CERN, France, Italy, UK & Ireland, Germany

& Switzerland, Northern Europe, South West Europe, South East Europe,

Central Europe, Russia)

uˇzivatelsk ´a podpora – port ´al, RT syst ´em

virtu ´aln´ı organizace pro centr ´aln´ı Evropu (VOCE)

(23)

www.eu-egee.org

EGEE – NA3/NA4

NA3 User Training and Induction

• ˇskolen´ı, pˇredn ´aˇsky, ˇs´ıˇren´ı informac´ı

NA4 Application Identification and Support

(24)

www.eu-egee.org

FZU AV ˇ

CR a Gridy

D0

LCG

Datagrid

EGEE

Figure

Updating...

References

  1. www.eu-egee.org
Related subjects :