Extreme Computing: The Bull Way

(1)

Architect of

an Open World™

Dr.-Ing. Joachim Redmer, Director HPC

(

[email protected]

)

(2)

Bull is an Information Technology

company, focusing on open and

secure systems

Our mission is to help corporations

and public sector organizations

optimize the architecture,

operations and financial return of

their Information Systems,

supporting their core business

processes

Bull is the only European IT

company that is positioned to

deliver all the key elements of the

IT value chain

2009 Bull activities

w/o consolidation of Amesys activities

(3)

3 ©Bull, 2011 Bull Extreme Computing

Reached 100 M€ products revenue in 2009

-

On track to exceed 10% market share in Europe

Anticipated 150 M€ products revenue in 2010

Introduced bullx range in 2009

-

Delivering new levels of performance and innovation for

Extreme Computing

-

Xeon-based as well as hybrid blades (NVIDIA)

-

bullx nominated best HPC server product by HPCWire

Signed landmark deals for bullx servers in 2010

-

GENCI /TGCC (France)

-

AWE - United Kingdom

-

RWTH Aachen - Germany

-

Dassault Aviation (France)

-

Société Générale (France)

-

Ineris (France)

-

Reims University (France), …

first petascale system in Europe at CEA

500 specialists dedicated to HPC in Europe

(4)

SOLUTIONS &

INTEGRATION

2008: SIRUS - Vertical ISV &

SI in France

2008: CSB Consulting - IT

consulting in Belux

2007: Siconet - SI in Spain

2006: Address vision - Postal

automation ISV & SI in USA

2006: AMG - Telco SI in

Poland

EXTREME COMPUTING

-

2008: Science +

Computing - Extreme

Computing Solutions &

Services in Germany

-

2007: Serviware - Extreme

Computing SI in France

SECURITY &

OUTSOURCING

-

2010: Amesys

-

2006: Agarik - Internet SI

& hoster in France

-

2005: Enatel

enatel

(5)

Structural Mechanics

Implicit

Structural Mechanics

Explicit

Computational Fluid

Dynamics

Electro-Magnetics

Computational Chemistry

Quantum Mechanics

Reservoir Simulation

Rendering / Ray Tracing

Climate / Weather

Ocean Simulation

Data Analytics

Computational Chemistry

Molecular Dynamics

Computational Biology

Seismic Processing

eXtreme Computing applications

A dedicated team of experts in

application performance

(6)

Architect of

an Open World™

(7)

Architect of

an Open World™

2010: TERA 100

1.25

Peak PFlops

1.05

Linpack PFlops

4 300

bullx S nodes

140 000

Intel Nehalem-EX cores

300

TB of memory

20

PB of disk storage

QDR InfiniBand interconnect

500

GB/s bandwidth to the global file system

(8)

Bullx Supercomputer: Best Top10 Linpack efficiency

0,3

0,4

0,5

0,6

0,7

0,8

0,9

(9)

"With technical support from CEA, through a

competitive tendering process, we were

able to assess the excellence of Bull's

offering. This means we will soon have at

our disposal a machine that will offer

French and European scientists the

resources they need to carry out their

research work at the highest possible

level in a highly competitive global

environment,"

Catherine Rivière, CEO of GENCI

1

st

phase Implemented in October

2010

CURIE in figures



1.6

PetaFlops

(90,000+ Xeon cores)



10

PB of storage



250

GB/s data throughput



200

m² footprint

(10)

Architect of

an Open World™

2011: RWTH Aachen

292

Peak TFlops

1350

bullx blades

362

bullx S nodes

16 200

Intel Westmere cores

11 500

Intel Nehalem-EX cores

1500

TB HPC storage

1500

TB Home storage

QDR InfiniBand interconnect

(11)

Architect of

an Open World™

(12)

bullx blade system

bullx rack-mounted systems

bullx SMP system

NVIDIA Tesla Systems

Bull Storage

Cool cabinet door

bullx cluster suite

(13)

(14)

bullx blade system – overall concept

General purpose, versatile

-

Xeon Westmere processor

-

96 GB RAM per blade

-

Local HDD/SSD or Diskless

-

IB / GBE

-

RH, Suse, Win HPC2008, CentOs, …

-

Compilers: GNU, Intel, …

Uncompromised performances

-

Support of high frequency Westmere

-

Memory bandwidth: 12x mem slots

-

Fully non blocking IB QDR interconnect

-

2.64TFLOPS per chassis (Intel Xeon

X5675 3.06GHz)

-

Up to 15.8TFLOPS per rack (with CPUs)

High density

-

7U chassis

-

18x blades with 2 proc, 12x DIMMs,

HDD/SSD slot/IB connection

-

1x IB switch (36 ports)

-

1x GBE switch (24p)

-

10 GigE Uplink (optional)

-

Ultracapacitor

Leading edge technologies

-

Intel Westmere

-

InfiniBand QDR

-

Diskless

-

Ready for GPU blades

Optimized Power consumption

-

Typical 6.5 kW / chassis

-

High efficiency (90%) PSU

-

Smart fan control in each chassis

-

Smart fan control in water-cooled rack

(15)

bullx blade system – Block Diagram

18x compute blades

-

2x Westmere-EP sockets

-

12x memory DDR3 DIMMs (12x 8GB= 96GB)

-

1x SATA HDD/SSD slot (optional – diskless

an option)

-

1x IB ConnectX/QDR chip

1x InfiniBand Switch Module (ISM) for

cluster interconnect

-

36 ports QDR IB switch

-

18x internal connections

-

18x external connections

1x Chassis Management Module (CMM)

-

OPMA board

-

24 ports GbE switch

-

18x internal ports to Blades

-

3x external ports

1x optional Ethernet Switch Module

(ESM)

-

24ports GbE switch

-

18x internal ports to Blades

-

3x external ports

1x optional Ethernet Switch Module

(TSM)

-

GigE Switch with 10GigE Uplinks

1x optional Ultra Capacitor Module

(UCM)

(16)

SATA

SSD

diskless

I/O Controller

(Tylersburg)

QPI

Westmere EP

Nehalem EP

Westmere EP

31.2GB/s

12.8GB/s

Each direction

31.2GB/s

QPI

PCIe 8x

4GB/s

QPI

Westmere EP

Nehalem EP

Westmere EP

31.2GB/s

12.8GB/s

Each direction

31.2GB/s

In

finiB

a

n

d

PCIe 16x

8GB/s

A

c

e

ler

a

to

r

PCIe 8x

4GB/s

In

finiB

a

n

d

PCIe 16x

8GB/s

A

c

e

le

ra

to

r

bullx B500 compute blade

bullx B505 accelerator blade

PCIe 8x

4GB/s

Inf

iniB

an

d

QPI

I/O

Controller

(Tylersburg)

I/O

Controller

(Tylersburg)

bullx blade system – blade block diagrams

SATA

SSD

diskless

(17)

Ultracapacitor Module (UCM)

NESSCAP Capacitors (2x6)

Leds

Board

Embedded protection against short

power outages

Protect one chassis with all its

equipment under load

Up to 250ms

Avoid on site UPS



save on infrastructure costs



save up to 15% on electrical costs

Improve overall availability

(18)

bullx chassis packaging

LCD

unit

18x blades

7

U c

h

a

s

is

PSU x4

ESM/TSM

CMM

(19)

bullx B505 accelerator blade

• 2 x Intel Xeon 5600

• 2 x NVIDIA T20

• 2 x IB QDR

7U

18.5 TFLOPS in 7 U

(20)

bullx blade system

bullx rack-mounted systems

bullx SMP system

Bull Storage

Cool cabinet door

bullx cluster suite

(21)

R424 E2

Supports latest graphics

& accelerator cards



4U or tower



2-Socket



Xeon 5600



18 DIMMs



2 PCI-Express x16 Gen2



8x SATA2 or 8x SAS HDD



Powerful power supply



Hot-swap Fans

Enhanced connectivity

and storage



2U



Xeon 5600



2-Socket



18 DIMMs



2 PCI-Express x16 Gen2



8x SATA2 or 8x SAS

HDD



Redundant power supply



Hot-swap fans

4 nodes per 2U

for unprecedented density



Xeon 5600



2x 2-Socket



2x 12 DIMMs



QPI up to 6.4 GT/s



2x 1 PCI-Express x16 Gen2



InfiniBand QDR embedded

(optional)



3x SATA2 hot-swap HDD



92% PSU efficiency

CO

M

PUTE

NO

DE

R423 E2

SER

VICE

NO

DE

R425 E2

VISUA

LI

ZA

TI

O

N

(22)

bullx blade system

bullx SMP system

Bull Storage

Cool cabinet door

bullx cluster suite

(23)

Mesca’s fundamentals

SMP of up to 16 sockets based on Bull Coherent Switch

-

Intel Xeon Nehalem EX processors

-

Coherent memory of up to 1 TB

Several types of packaging

-

High-density compute node

-

High I/O connectivity node

RAS features

-

Self-healing of the QPI and XQPI

-

Hot swap disk, fans, power supplies

Green features

(24)

Module level diagrams

Full width QPI link

IOH

For 4 socket-only systems

NHM

XCSI link

Repeated n times

for >4 socket systems

NHM

IOH

BCS

(25)

Bullx S60x0 – CC-NUMA server

Node maximum configuration :

4 modules

16 sockets

128 cores (Nehalem-EX)

128 memory slots (2TB)

BCS

NHM

EX

NHM

EX

NHM

EX

NHM

EX

IOH

BCS

SMP (CC-NUMA)

Large nodes :

Large shared memory (pre/post-processing)

Many more cores (SMP)

Fewer nodes

Simpler system administration

(26)

Taking advantage of SMP nodes with MPI

Optimized intra-node

throughput

-

Enable direct copy from sender

to receiver

-

Rely on SSE instruction set

-

Achieve a transfer rate of half

the memory bandwidth

Optimized intra-node latency

-

Lock-free shared memory device

-

Take advantage of socket

architecture

-

Shared cache latency: 200 nsec

0

0,2 0,4 0,6 0,8 1 1,2 1,4 1,6 1,8 4 cores, Mvapich2 8 cores, Mvapich2 16 cores, Mvapich2 4 cores, MPIBull2 8 cores, MPIBull2 16 cores, MPIBull2 To socket 1, core 2 To socket 1, core 3 and 4 To socket 2

(27)

bullx blade system

bullx rack-mounted systems

bullx SMP system

Bull Storage

Cool cabinet door

bullx cluster suite

(28)

Bull Storage for HPC clusters

*: with Lustre

A complete line of

storage systems

• Performance

• Modularity

• High Availability*

A rich

management suite

• Monitoring

• Grid & standalone

system deployment

• Performance

(29)

Bull Storage Systems for HPC

StoreWay

Optima 1500

DataDirect Networks

SFA 10k

(consult us)

SAS/SATA

3 to 144 HDDs

Up to 12 host ports

2U drawers

SAS/SATA

Up to 1200

HDDs

8 host ports

4+ 2/3/4 U

drawers

FC/SATA

Up to 480 HDDs

Up to 16 host ports

3U drawers

StoreWay

EMC CX4

cluster

suite

(30)

Bull Storage Systems for HPC - details

Optima1500

CX4-120

CX4-480

SFA 10k

couplet

#disk

144

120

480

1200

Disk type

SAS 146/300/450 GB

SATA 1TB FC 146/300/400/450 GB SATA 1TB FC 10Krpm 400 GB 15Krpm 146/300/450 GB SATA 1TB SAS 10Krpm 400 GB SAS 15Krpm 300/450/600 GB SATA 1000/2000 GB

RAID

R1, 3, 3DP, 5, 6, 10, 50 and

TM

R0, R1, R10, R3, R5, R6

8+2 (RAID 6)

Host ports

2/12 FC 4

4/12 FC4

8/16 FC4

16 FC8 / 8 QDR

Back end ports

2 SAS 4X

2

8

20 SAS 4X

Cache size

(max)

4 GB

6GB

16GB

5 GB

RAID-protected

Controller size

2 U base with disks

3 U

4 U

Disk drawer

2 U

12 slots

3 U

15 slots

3 U

15 slots

4 U

60 slots

Performance

(

MB/s; Raid5)

R: Read; W:Write

R: up to 900 MB/s W: up to 440 MB/s R: up to 720 MB/s W: up to 410 MB/s R: up to 1.25 GB/s W: up to 800 MB/s R&W: up to 20 GB/s

(31)

bullx blade system

bullx rack-mounted systems

bullx SMP system

Bull Storage

Cool cabinet door

bullx cluster suite

(32)

Bull Cool Cabinet door

No impact on server behaviour

air flow through doors is adjusted to match

the drawer air flows

No impact on computer room

Select outlet Air temperature

2 way or 3 way valve controls heat

exchanger water flow.

No more hot spots

better MTBF

Cools up to 40kW per rack

ready for Bull Extreme Computing systems

Inlet water 7°C

Server

ΔT 15°C

Server

ΔT 15°C

Server

ΔT 15°C

Server

ΔT 15°C

Server

ΔT 15°C _Room pressure

Water cooled door

M Δ p Rack pressure 35°C 20°C E x ch a n g e r 20°C

(33)

(34)

Cool cabinet door: Characteristics

Width

600mm (19

”

)

Height

2020mm (42U)

Depth

200mm (8

”

)

Weight

150 kg

Cooling capacity

Up to 40 kW

Power supply

Redundant

Power consumption

700 W

Input water temperature

7-12

°

C

Output water temperature

12-17

°

C

Water flow

2 liter/second (7 m3/hour)

Ventilation

14 managed multi-speed fans

Recommended

cabinet air inlet

20

°

C +- 2

°

C

Cabinet air outlet

20

°

C +- 2

°

C

Management

Integrated management board for local regulation

(35)

bullx blade system

bullx SMP system

NVIDIA Tesla Systems

Bull Storage

Cool cabinet door

bullx cluster suite

(36)

bullx supercomputer suite

Standard

Edition

Advanced Edition

eXtreme Pack

thousands of nodes

(37)

(38)

Advanced Edition / Extreme Pack: components

• Super-Fast image based provisioning

• Web-based Multi-level supervision

• Power management

• Automated health management

• Maintenance management

bullx MC

• Highly available cells based architecture

• Increased throughput and scalability

bullx PFS

• Advanced placement policies

• Topology aware resource allocation

bullx BM

• Multi-path network failover

• Abnormal patterns detection

• Topology aware operations

bullx MPI

• Complete best of breed set of tools (from compiling,

debugging to profiling and optimizing activities)

bullx DE

• HPC Enabled (OS jitter reduction, Optimized operations for

increased application performance)

• Enhanced OFED

bullx Linux

Management Center

Parallel File System

Batch Management

(39)

bullx blade system

bullx SMP system

Bull Storage

Cool cabinet door

bullx cluster suite

Windows HPC Server 2008

(40)

Bull and Windows HPC Server 2008

Clusters of bullx R422 E2 servers

-

Intel® 5600 processors

-

Compact rack design: 2 servers in 1U

-

Fast & reliable InfiniBand interconnect

supporting

Microsoft

®

-

Simplified cluster deployment and management

-

Broad application support

-

Enterprise-class performance and scalability

Common collaboration with leading ISVs to provide

complete solutions

The right technologies to handle

industrial applications efficiently

(41)

Microsoft® Windows

Server® 2008 HPC

Edition

Microsoft® HPC

Pack 2008

+

=

Microsoft®

Windows® HPC

Server 2008

•

Support for high

performance

hardware (x64 bit

architecture)

•

Winsock Direct

support for

RDMA for high

performance

interconnects

(Gigabit Ethernet,

InfiniBand, Myrinet,

and others)

•

Support for Industry

Standards MPI2

•

Integrated Job

Scheduler

•

Cluster Resource

Management Tools

•

Integrated “out of the

box” solution

•

Leverages past

investments in

Windows skills and

tools

•

Makes cluster

operation just as

simple and secure as

operating a single

system

Combining the power of the Windows Server platform

with rich, out-of-the-box functionality to help improve the productivity

and reduce the complexity of your HPC environment

(42)

A complete turn-key solution

Bull delivers a complete ready-to-run solution

-

Sizing

-

Factory pre-installed and pre-configured

(R@ck’n Roll)

-

Installation, integration in the existing infrastructure

-

1st and 2nd level support

-

Monitoring, audit

-

Training

(43)

bullx cluster 400-W

bullx cluster 400-W4

-

4 compute nodes to relieve the

strain on your work stations

-

8 compute nodes to give

independent compute resources to

a small team of users, enabling

them to submit large jobs or

several jobs simultaneously

-

16 compute nodes to equip a

workgroup with independent high

performance computing resources

that can handle their global

compute workload

A solution that combines:

The performance of bullx rack

servers equipped with Intel

®

Xeon

®

processors

The advantages of Windows HPC

Server 2008

-

Simplified cluster deployment and

management

-

Easy integration with IT

infrastructure

-

Broad application support

-

Familiar development environment

And expert support from Bull

’

s

Microsoft Competence Center

Enter the world of High Performance Computing with

bullx cluster 400-W running Windows HPC Server 2008

(44)

(45)