CASA project ( )

(1)

2A

‐

SI

6 – Ouverture

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

Stéphane Vialle

Stephane.Vialle@supelec.fr http://www.metz.supelec.fr/~vialle

SI de grande taille Stephane Vialle

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Premise of Grid Computing…

CASA

project

(1990

‐

1995)

Two research issues:

• Impact of high speed WAN on HPC applications?

• What algorithms and implementations to achieve speedupacross a high speed WAN?

Objective: distribution of large computations on several supercomputers across a (high speed) Gigabit WAN

CASA

project

(1990

‐

1995)

t

Cray YMP Intel DELTA

Network

Calcul 1 Calcul 2

Vector computation Parallel scalar computation

Hiding network time:

•3 stage pipeline

•Overlaped computations

and communications

Chemical computation:

Results:

•Cray Y‐MP + Intel DELTA vsCray Y‐MP: speedup 3.4

•Pipeline: overlapping at 98%

Synthetic

Force

project

(1990

‐

1998)

Objective: Distributed Interactive Simulation of War.

Distributed Multi‐Agent system.

Research issue: Size Up(× 100) n ×100000 entities n ×10000 vehicles Military training: • Fine simulation of military operations. • Many simultaneous users • Interactive simulation

Synthetic

Force

project

(1990

‐

1998)

Multi‐Agent simulation:

Application with many different computational parts Interactive application with many simultaneous users Interconnection of standard and specific hardware March 1998: simulation of 100000entities

(2)

Results

of

the

experiments

Need of a specific (and new) middleware

(grid middleware)

Main difficulties:complex to develop and to deploy Distributed computing across a WAN: Possible: achieves speedup Allows to use specific (adapted) resources (not locally available) Interactive and distributed computing on heterogeneous resources: Possible: achieves size up Allows more realistic simulations, with more users.

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Grid Computing Emergence

Ian

Foster’s

view

• An infrastructure to provide computing/storage/communication capabilities to users, when they need, where they need, and in a transparent way.

• To build « virtual organizations ». Ian Foster (Globus – 1st_Grid_middleware)

Proof

of

concept

(1995‐2000)

• 125 sites, 23 countries

• A testbed for the first Grid middleware

GUSTO: Globus Ubiquitous Supercomputing Testbed Organization

Different

kinds

of

Grids

Grid of (reliable) supercomputers:

•for size up

•to access the right supercomputer (with the right architecture)

Different

kinds

of

Grids

Grid of available (and volatile) desktop PCs everywhere on Internet: •Ex: Seti@Home, LHC@Home, …

•Highly dynamic Grids (resources appear and disappear …) •A generic middleware will be developed

(3)

Different

kinds

of

Grids

Grids of DATA: •To share huge data.

•Repository can not be centralized… (too long access times)

•file migration vsreplication?

•distributed vscentralized directory?

•directory and repository coherence?

Management of

CERN experiment

results

Data storage + Computing rsrc

Different

kinds

of

Grids

Collaborative Grid: distributed virtual and enhanced reality • Real time issues, requires QoS …

• Concert across Internet in 2003, with expensive devices

Collaborative work on virtual objects in 2009, medium cost devices

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Inside the Grid

Key

issues

of

a

Grid

middleware

(1)

• Local functionalities on the resources with standard interfaces

Ex : authentication on the local resource • Global (collective) services

Ex : authentication all over the Grid

Adaptability is required to maintain

functionalities when the Grid is changing

Heterogeneous rsrc and many potential failures

low level modules high level services Distributed resources Adaptive applications • Minimal set of available (and standard) functionalities on each resource The « hourglass » model (of Globus‐1):

Inside the Grid

Key

issues

of

a

Grid

middleware

(2)

Each site has its own authentication mechanisms… can be long and complex to log on each site! ‐A user authenticate only once, using a high level authentication service (with a std interface) ‐Then he is automatically authenticated on any resources he is using, with the help of low level modules Low level modules High level services Distributed resources Adaptive applications From high level services to low level modules

Inside the Grid

Key

issues

of

a

Grid

middleware

(3)

Based on certificat exchange

GridFTP (extension of « ftp ») + Grid directory LDAP based (hierarchy of LDAPs) Brooker and scheduler + DataBase of resource

availability « Withoutwould be security no Grid issues, middleware there research ». G. Khan, 2004.

(4)

Inside the Grid

Becoming

closer

of

Web

services

Network OGSA Enabled Storage OGSA Enabled Servers OGSA Enabled Messaging OGSA Enabled Directory OGSA Enabled File Systems OGSA Enabled Database OGSA Enabled Workflow OGSA Enabled Security OGSA Enabled Web Services OGSI – Open Grid Services Infrastructure

OGSA Architected Services Applications Network OGSA Enabled Storage OGSA Enabled Servers OGSA Enabled Messaging OGSA Enabled Directory OGSA Enabled File Systems OGSA Enabled Database OGSA Enabled Workflow OGSA Enabled Security OGSA Enabled Web Services OGSI – Open Grid Services Infrastructure

Web Services OGSA Architected Services

Applications

WSRF

Open Grid Service Architecture GLOBUS III

GLOBUS IV using web services GLOBUS IV evolution

Inside the Grid

Grid

middleware

for

independent

tasks

XtremWeb server Internet PC cluster (workers) Desktop PCs (clientsor workers) Client running: embarrassingly parallel applications, or sequential tasks Key issues: •to deal with security of desktop workers •support high volatility of desktop workers Many similar industrial products and

Inside the Grid

Grid middleware

for

independent

tasks

Rq : hostRegister Contact the last known XtremWeb server, or

contact the “root” XtremWeb server Rq : workRequest Send its features to the XtremWeb server and wait for a task to process

Rq : workAlive Signal its activity to its XtremWeb server

Rq : workResult

Send its result to a specified server or to its XtremWeb

server Becoming unused Waiting for a task Processing a task Task finished Register the new worker

Send back a task:

‐binary code, ‐data,

‐server @

(to reply) Re‐schedule the task if no “I am alive”

message received Communication protocol Worker ‐‐XtremWeb server:

•communication started by the worker

•data sent back to the worker by the server

User/Worker Task server

Compatible with

statelessfirewalls

Inside the Grid

Network

Enabled

Server

(NES):

an

API

and

some

Grid

middlewares for

RPC

//GridRPC … grpc_call… … Brooker (« agent ») allocating

resources _Computing_servers (running « solvers ») User laptop User application API: GridRPC Specific Grid middleware (NetSolve, Ninf, Diet) Distributed resources + + Allows to implement more complex parallel applications: •can call a parallel solver (run on one site) •can overlap solver runs and/or communications (async. RPC)

Inside the Grid

Network

Enabled

Server

(NES):

an

API

and

some

Grid

middlewares for

RPC

Components of the brooker(or « agent »)

On demand analyse, allocation and scheduling Continuous performance measurement Statistical DB building Client Server Server Server Server Server Server Server Server Server Scheduler Monitor Monitor Monitor Data‐ base 1 2 3 4 5

Generic mechanism for

resource allocation:

Inside the Grid

API

and

Grid

middleware

for

Active

Objects

JAVA based

Support a large variety of parallel algorithms:

•Independent computations

•GRID RMI (object version of GRID RPC)

•Master‐Workers

(5)

Inside the Grid

API

and

Grid

middleware

for

a

virtual

shared

memory

JavaSpace _{A virtual shared memory}

spaceis located on one machine.

A space is R/W by any task

from any machine.

Industrial products with distributed spaces (scalable spaces) exist :

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Evolution

of

Grids

from

1998

to

2006

e‐Science Grids

Enterprise Grids Experimental Grids HPC – meta‐computing Web services

3 families

of

Grids

e

‐

Science

Grids

Middleware: Globus‐II Globus‐II ++ gLITE

2008: 150/200 Virtual Organizations 7500/14000 users 150000 jobs/day 2008: • 259 sites • 52 countries EGEE: European production Grid for scientific research »

Each participant brings resources,

and can access all resources.

Grid engineers make compatible

security policies of all sites

Experimental

Grids

Objective 1: 5000 processors

objective: 5000 cores

achieved, and surpassed

Grid’5000: French experimental Grid

Objective 2: a real multi‐site Grid

15 clusters on 9 sites, +++

Private RENATER network,

10Gbit/s

Objective 3: a Grid for computer

science research

NO production of applicative

results

On demand OS deployment !!

« first cloud in the world » 

Enterprise

Grids

Definition: One enterprise uses its PCs to run its independent & compute‐intensive tasks. The enterprise Grid is installed inside its secure network. PCs of the Grid are not volatile: •desktop PCs: PC of absent people, •server farm: reliable and monitored. Examples:

•September 2006: BNB‐Paribas used a 3000 server‐PC Grid (with QoS)

(6)

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Example of structured Grid

New

e

‐

Science

Grid for

LHC

LHC@home LCG: LHC Computing Grid

Voluntary computing + unused PCs in the laboratories

A World Grid both for data and computations:

•15 PetaOctets/year •Huge computations

CERN

‐

LHC

Example of structured Grid

New

e

‐

Science

Grid

for

LHC

LCG: the LHC Computing Grid

Node T‐0 : data production site, at CERN Node T‐1 : Complete storage sites

in the world

Node T2 : Partial storage site in the world (database caches)

Node T3 : Computing nodes in the world (ex: clusters)

A hierarchyof tier nodes, to store and cache huge data, and to run huge computations.

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Cloud Computing Emergence

An

industrial

initiative

(?)

Business model: •A company provides computing resources •Price is function of the consumed CPU time (parallelism is free) •Data security inside the cloud •Secure connection to join the cloud •QoS is provided •Virtualization on each node allows to support various applications from various customers Personal point of view: the next step after Enterprise Grids.

An

industrial

initiative

(?)

Large data centers:

•have global cooling and control of the machines

•allows to reduce management cost Mutualisation:

•a customer requires machines only when he needs (on demand)

•allows to share resources between users (customers)

Ex : in March 2009, Salesforce.commanaged data of 54 000 entreprises

with only 1000 servers. Virtualization:

•each application is run inside a virtual machine

•provides the environment required by the customer application

•allows to always use the most efficient available machines

•allows to use at 100% each machine (if machine are shared)

(7)

A

basic

classification

Cloud of software (Software as a Service)

Ready to run some identified applications

Ex: Supelec experimented a private cloud of Matlab Cloud of plateform (Platform as a Service)

Ready to provide a development environment on an available hardware resource. Cloud of hardware (Infrastructure as

a Service)

Ready to provide hardware resources . The user/customer has to deploy an

environment before to use these machines.

Google search engine ? Google mail ?

Grid’5000 ?

Amazone ?

First

cloud

assessment

« CLOUD » buzzword is a success! Several major actors make money with « cloud technology » Cloud was initially less ambitious than Grid, but aimed to be easier to deploy, to manage and to use From a customer point of view : •using extra resources on demand is a nice idea •how can I be sure to get these extra resources when I need ?

limit of the « mutualisation » and QoS •how can I be sure my data are in a secure system ?

limit of virtualization, remote security and transfer security Private cloud can be an interesting compromise

From

Grid

to

Cloud

?

e‐Science Grids Enterprise Grids Experimental Grids HPC Web services

3 kinds

of

Clouds

IaaS PaaS SaaS Virtualisation technics Security

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

New Cloud Computing Approach

New

approach

in

cloud

computing

But … looks like a Grid ?

Grid middleware technology is now introduced has the « Grid infrastructure for Cloud » (HPCS 2011 conference) New cloud researches:

•Multi‐site Clouds

•Inter‐Clouds (interconnecting clouds of different institutions)

 fusion of different cloud providers

 a cloud provider can rent resources of another one

50% news about « cloud » claim this new approach is promising 50% news about « cloud » say « WARNING »!

•it will (technically) fail!

•there is no business model!

•there is no need for these new clouds!

New

classification

The NIST definition of cloud computing (September 2011):

•Axis 1: IaaS, PaaS, SaaS •Axis 2: who can use the cloud ?

Private cloud: exclusive use by a single organization comprising multiple consumers (business units)

Community cloud: exclusive use by a specific community of

consumers from different organizations

Public cloud: open use by the general public

Hybrid cloud: composition of 2 or more cloud infrastructures (private, public, or community)

(8)

New

classification

Possible complete classification:

IaaS PaaS SaaS

Private clouds Community clouds Public clouds Hybrid clouds Users Service Owner & manager (?)

Institution of the only user One limited institution

Some limited institutions Any institutions Where are stored the data ? Ex: “EU data have to stored inside EU”

Future

of

the

Cloud

Computing

2008 – 2010: Europe has started some FP7 projects about « cloud »

•development of new Cloud middleware

•some of these projects are now calling for experiments

currently no real evaluation

September 2011: Europe opened a new call about « cloud » •to not develop new middleware •to develop « cloud applications » : with a business model at European scale 2012 : Governments want to be sure institutional/national data are stored inside the country ! Pb with inter‐clouds/cloud roaming

Future

of

the

Cloud

Computing

Questions :

Missing real large cloud applications? A European cloud infrastructure has a sense? Multi‐sites, multi‐institution clouds have a sense? HPC‐Cloud ? Aneo & Supelec are experimenting the new MS Azur‐HPC cloud…

Ex.

de

SI

de

grande taille :

Grids,

Clouds

et

Réseaux de

Capteurs

1. Premise of Grid computing 2. Grid computing emergence 3. Inside the Grid (Key issues and examples) 4. Classification of Grids 5. Example of structured Grid 6. Cloud computing emergence 7. New Cloud computing approach 8. Sensor networks

Réseaux de capteurs

Des SI de grande taille avec de nouvelles contraintes

Capteurs fixes Capteurs mobiles

Alimentation électrique pérenne Ex : trackingen environnement urbain ou industriel Ex : capteurs embarqués

dans des voitures

Batteries ou

alimentation non

pérenne

Ex : mesure de température

dans la nature

Ex : capteurs portés par

des personnes Motivation : un moyen de « réaliser des mesures au plus

près d’un phénomène » Aujourd’hui : de k×100 à k×1000 capteurs.

demain : k×100 000.

Différentes configurations avec différentes contraintes

Des

applications

très

variées

Exemples d’applications :

•Mesure du graphe de mobilité des personnes dans un hôpital

pour détecter des contaminations et propagation de maladies

nosocomiales.

•Collecte de mesure environnementales (températures, pluie) et

comportementale (allumage phares, déclenchement d’ABS) par

des capteurs dans les voitures, pour améliorer la gestion du

trafic.

•Supervision de la radioprotection en temps réel dans les

centrales nucléaires (EDF) par des réseaux ad hoc de capteurs

(9)

Besoin

spécifiques

aux

réseaux

de

capteurs

Besoin de nouveaux protocoles réseaux pour :

•minimiser les dépenses d’énergie

•tolérer de nombreuses pannes de nœuds du réseau

•supporter l’apparition et la disparition fréquente de nœuds mobiles

•…

Besoin de traitement de signal complexe pour : •analyser de gros flux d’information •détecter des données erronées •…

Besoin de nouvelles applications

distribuées pour :

•exploiter ces masses de

données en temps réel

•réagir rapidement en cas

de situation anormale

•…

Exemple : projet “Live‐E!”

En février 2009 : 106 stations météo dans13 pays