Grid PARISTIC - Bordeaux

(1)

Franck

Cappello

, Michel

Daydé

,

Frédéric Desprez

, Emmanuel

Jeannot

,

Yvon Jégou

,

Stéphane Lantéri

,

Nouredine Melab

,

Raymond

Namyst

,

Brigitte Plateau,

Pascale Primet

, Thierry

Priol

,

Olivier Richard,

Dany Vandrome

www.grid5000.

fr

Email

fci@lri

.

fr

Grid

’

_’

5000

₅₀₀₀

a large scale and highly reconfigurable

Grid experimental testbed

(2)

(3)

$ "%" &'( )* + , -.

!

-

"

$ -" &'( /)0,-* +, .

!

$ 1 &-( )*- 1. $ , &,( 0 )" -2. $ 3 # &4# ( )54-6.

!

7

$ 28 & # ( - 9 )0,2".

!

, : ;

$ 0- 2 & ( " )0,-* . $ *1 & ( "8 /) - +, .

!

4

$ 0"-, &3( ) , .

!

$ " 350 & ( )*"<. $ 20 & ( ) 2 .

!

1

$ 1-&-( '8 ) ,- #* . $ #1*-0 &2( )0,-00 =1. $ 2 & ( "8 /) - . $ 3, 0 13 &>( 2 )4 *3 . $ 2*& ( = )*'3.

!

$ 3520=>& ( 2 )*1=0. # $ & ( * ) - . # ? $ &@# ( " )*3 . # 3 $ = &'( ) -. # $ # &*( )*3 . # * $ ""-& ( ? ) ? ". # ? $ = &2( A ) , . # 0 $ 2 &@( 2 ) ,- #* . # 2

!

1

9

$ # 20,1&*( 9 )5 '. $ 2 &=( ) , . $ *3" &4( ? ) ,%"B. $ &'( ) , .

!

-$ 0 & ( - )*3 . $ % &@#*( " / ) - +,- . $ *&>( ) 2 .

(4)

'"<

&% %

#

% <.

"

9 -$

)

$ 0

9 $

$

#

: ; #

0 ,1

! " # $ # % & '&

())

*

+

,(-

* .+ %/

DG IST - F3

DG IST - F2

(5)

C

&

.

5 )

2" )

D

:

C

:

;)

C

#, : ;

)3-)

:

)

9 )

D

& .

9 : ;

E= :

E

D

C

-

E

(6)

F"%"

C

*

)

;

:

F"%"

C

-

)"

)'

)-

9 )

*

?

)

)2

)

)"

)

(

= :

E

! '

!

-! , : ;

!

(

F"%"

9

(7)

0

1 '

'

1

9 -

):

C

G. 1

&

)

( "

.

2+

3 '

4 #

5 '

%. -

9 :

(8)

0

1

0

1 !

-

%

!

:

!

3 !

-

)

!

-!

&:

.) 0-&@

.

!

2 #

)2

#9 ;

)2

#

!

1 ? ;

!

-!

-

9 :

43 !

2 )

!

0 )

9 )4

/

), : ; &2 --.

1

G. :

;

%.

!"

(9)

&

F

.

&

.

!

#

6

1 '

!

7 D , * 9

0

9 !

8-)))

-%

#

"

* 9

,

1

9 :

C

1 ;

-1

(10)

-G. ?

:

F"%"

& ;

.

!

I

9 !

&

% < "5

GA "5 .

!

9 0, 10

&'

(

0 ( , (.

!

0, 10

9 : ;

!

+

:

%. 5

!

+

:

C

!

"

)-

9 )'

1 )-!

, : ;

!

=

)6

!

"

!

"%"

)

;

1

1 "

_"

(11)

2

(12)

!

"

# #"

$ "%" &'( )* + , -.

!

-

"

$ -" &'( /)0,-* +, .

!

$ 1 &-( )*- 1. $ , &,( 0 )" -2. $ 3 # &4# ( )54-6.

!

7

$ 28 & # ( - 9 )0,2".

!

, : ;

$ 0- 2 & ( " )0,-* . $ *1 & ( "8 /) - +, .

!

4

$ 0"-, &3( ) , .

!

$ " 350 & ( )*"<. $ 20 & ( ) 2 .

!

1

$ 1-&-( '8 ) ,- #* . $ #1*-0 &2( )0,-00 =1.

!

$ 2 & ( "8 /) - . $ 3, 0 13 &>( 2 )4 *3 . $ 2*& ( = )*'3.

!

$ 3520=>& ( 2 )*1=0. # $ & ( * ) - . # ? $ &@# ( " )*3 . # 3 $ = &'( ) -. # $ # &*( )*3 . # * $ ""-& ( ? ) ? ". # ? $ = &2( A ) , . # 0 $ 2 &@( 2 ) ,- #* . # 2

!

1

9

$ # 20,1&*( 9 )5 '. $ 2 &=( ) , . $ *3" &4( ? ) ,%"B. $ &'( ) , .

!

-$ 0 & ( - )*3 . $ % &@#*( " / ) - +,- . Priol

(13)

!

, : ;

$ 0

&

:

.

$ =

&

1 ".

$ =

-

, : ; 0

!

2 :

+3-$ -

+

9 $ '

$

-- 3-

+

3 $

;

+

"%"

!

"

$

&@

)

9 .

$

# "

$

#2"

$

!

$ 2

#

&

+

'

.

$ *

9 &0

)

#

)

/

) ' )

$ 2

)

9 B

C

+

:::(

( +

(

+

C

0

(14)

C

6 C

! "

$ 0

)

! -

9 $

& "5)

)

: ;.

$

$ ,

9 ! '

#

$ 1

7 &

.)

&

.

$ '

:

;(

(15)

0

0 "%"

:

;

"%"

:

;

"%" 2

:

3 -"

0 , : ;

9

(16)

0 : ;

:

0 : ;

:

*

+

9 &

.

1 J

?

( E

G

9 &

(.

(

0

9

0 ( 3AE

(17)

2

(18)

1

1 $%%%

&%%

&%% '

&%%

0, 10

$

"

( $%

"

(19)

=

:

-

B

,

B

@

L

3

2 L

@ +

@

L

-

L

3 L

&- L.

,

L

-

+

:

'

-

"

'

0 Switch

Proto to

Grid’5000

(20)

"

!

)%%*

)%%&

"

9

0 )%%+

9 )%%,

G%

"5

B

"

1 )%%-"

F,

%

#."

(21)

1 C

!

0 !

:

!

+

:

(22)

Switch Grid

9 . 6 : 1 6; < $91 /

Router RENATER

MPLS (level 2)

Routing.

8 VLANs per site

; & <

Switch/router labo

: ; . / Router RENATER Router RENATER Routeur RENATER

Grid5000 site

RENATER

-

C

-

C

* 9

% 9

&G

.

J

(

'

* 9(

* ?

5 " /

G5k site

Grid5000 site

DiffServ

(IP Premium)

(23)

* ?+

' :

Cluster

Users/

(ssh loggin + password)

' : +

Cluster

rsync (directory)

G5k site

A) A single account for every user

B) Every site manages the account of its users.

C) Every site runs a LDAP server containing

the same tree

D) the local administrator has read-write access

to its user accounts.

E) other account are periodically synchronized

from remote servers and are read-only.

G5k site

/tmp/user

/home/site1/user

/site2/user

/site…

/home/site1/user

/site2/user

/site…

rsync (directory)

_LDAP

admin/

(ssh loggin + password) + + +

LDAP

G5k site

(24)

* 9

, : ;

* ?+

' :

Cluster

OAR

Site 3

Users

OAR

Cluster

OAR

G5k site

0 -0

-J -

C3

J -

C3

G 9

G5k site

A) In every cluster, OAR controls

the local resource for

B) OAR provides priority scheduling by queues, advance reservations, backfilling and resource

match making. C) In Grid mode, broker submits a

reservation to every local OAR, in a round robin sequence. If one reservation is refused, all previously accepted reservations are canceled.

OAR-Grid

Broker

Users

D) A specific queue allows user reconfiguring the nodes. Rights are granted on all related nodes

accordingly. After the experiment the rights are revoked

(25)

* 9

, : ;

* ?+

' :

Cluster

Kdeploy2

Site 3

Users/

admin

(ssh loggin + password)

Kadeploy2

Cluster

Controler:

(Boot server + dhcp)

Kadeploy2

G 9 & .

G5k site

4) System kernels and distributions are downloaded on the local disk from a boot server in a pipeline mode 1) Prepare the software images

to distribute and replicate them on

related sites 2) Register the software images In the Kadeploy database

3) All related nodes are rebooted in a deployment OS, preparing the local disc

6) All related nodes are rebooted from the local disc, using a network loaded bootloader

5) Postinstallation scripts set node specific information

(Keys for authentication, etc.)

(26)

Switch Grid

& ,-) * ") ,'-) + ) 9 ) = ") ? .

Router RENATER

MPLS

7 VLANs per site

' : +

Switch/router labo

* ' # & 9 . Router RENATER Router RENATER Routeur RENATER

Grid5000 site

RENATER

2

* 9

% 9

&G

.

'

* 9(

* ?

5 " /

G5k site

Grid5000 site

1 .!1 /

!

=$19

(27)

9 *

1 -3

?

(28)

9

(29)

(30)

(31)

(32)

(

(33)

(34)

2

(35)

(36)

"

!

7 "

$2

:

;

$

C

9 )

)

;

)

(

!

%

−

, : ;

9

(37)

PARISTIC - Bordeaux - 2005 37 30/12/2005

$ '

:

"

'

$

C

9 )

)

!

%: ? .

%

:

?

7 /

−

0 C

9

9 −

"

9 )

:

)

99 )

!

8-))) >

$ 4

9

01 $

01

9 ;

$ 0

:

1*-0

$ 4

/

:

4 / 01

!

0 <<

'< +?%

0 <<&&&

'<

<

(38)

N: ;

&

3

9

9 G

G

#

GL%

.

!

-!

"

1 &,#6

' :

.(

1

9 :

9 "

)

"

)

9 "

(

1 ,#6

&L

. :

:

9 ,#7

9 ), 9

9 9 )

1 ' :

&B

.

G<

"5

C

G%

9 J

9 &0 00)

0 - ),

. J B

"5

(

(39)

(40)

* 9

, : ;

* ?+

' :

Cluster

Kdeploy2

Site 3

Objective:

Boot to Boot time < 10 minutes

Kadeploy2

Cluster

Controler:

(Boot server + dhcp)

Kadeploy2

G5k site

2

G 9 & .

G5k site

' : +

(41)

0

50

100

150

200

250

300

0

20

40

60

80

100

120

140

160

180 time (sec)

# nodes

first check

preinstall

transfert

last check

( A 9 " ( 5 0 5 A 9

(42)

0

50

100

150

200

250

300

0

200

400

600

800 1000

1200

# nodes

time (sec)

deploying

deployed

deployed_site1

deploying_site2

T=0 s: all nodes run an OS

T=30 s: deployement sequence is launched T=50 s: all nodes are rebooting

T=160 s: all nodes rebooted and prepare the disc T=200 s: Second boot order

T=320 s: Site2 machines rebooted with user OS T=450 s: Site1 machines rebooted with user OS

(43)

2

(44)

DAS2: 400 CPUs exp. Grid

-%&% %. C

?

! =

O

!

:

$

9 B(%

;

$ "?-J2

! "

$ 2" &2"

=# 2)2"

=# %.)"42)"

$ "

! "

$

) JJ)@

)'

KK+

I +

I

VU

UvA

Leiden Delft

Utrecht

#nodes

72

32

32 Memory

(GB)

1

1.5

1

1 Local disks

(GB)

20

80

60

20

20 File server

(GB)

6 * 36 6 * 36 6 * 36

2 * 18 2 * 18

(45)

4 #

"

:

#

4 "

: ;

#

4

0 #

4 #

4

9 "

#

4 "

* 9

(46)

*

8

8 F

F

8

87

8

7 " 8

9

8

3

8 &="

) " "-)

)

(.

3 D

;

&

) " "-.

3

8 &

N: ;)

.

" 8

9 '

&

8 .

8

9

0

8 &

.

8 @

&

,

) , -#@

-1)- ;

)

(.

"

5- &87

8 ,

),-'+,

.

(47)

!

9

9 !

:

%

C

$ I

9 I

'

)

$

9 %

"5 )

$

9 %) 1?

)

$

9 G

1?

)

$

9 M

9 + &

.

9 :

$

9 P G 1

+

$

9 Q

+

3-+

2 :

+

R

!

'

@

%

!

: 9

% <

!

&,

)@

.

(48)

650-13,-E

(49)

(50)

* 9

, : ;

* ?+

' :

Cluster

C

9

9 1%

1%

9 %

G5k site

1) Design the virtual Grid:

a) Select one node to run the GT2 services. b) Select another node to run the cluster front

end (batch scheduler, user accounts, etc.)

' : +

Cluster

G5k site

39 C1

;

1%

GT2 Gate Keeper

C

Certificate

C

3

Grid Front node

Globus MDS

F

(51)

* 9

, : ;

* ?+

' :

Cluster

9 %

G5k site

2) Construct the software images for:

a) The Grid front end (with GT2 services) b) The cluster front end (with batch scheduler) c) The cluster nodes