Potsdam Scientists to Tackle New Type of Weather Simulations with IBM idataplex

(1)

Potsdam Scientists to Tackle New Type of

Weather Simulations with IBM iDataPlex

Jan 21, 2009 - The Potsdam Institute for Climate Impact

Research (PIK) is rolling out a new IBM supercomputer that will

increase its computing capacity more than 30-fold.

Potsdam researchers plan to employ IBM’s high-performance

iDataPlex servers to more precisely predict weather events that

have so far proven to be incalculable – extreme, short-term

phenomena such as torrential rain or drought.

[Joint press release by Potsdam-Institute for Climate-Impact

Research (PIK) and IBM Germany Ltd.]

(2)

Outline

➢

Acknowledgments

➢

Requirements

➢

Procurement and Installation

➢

Hardware, Software Overview

➢

System Architecture

➢

Compute Subsystem IBM iDataPlex ™

➢

Voltaire Infiniband Interconnect

➢

Benchmarks

➢

Electrical Power Consumption & Cooling

➢

I/O Subsystem

(3)

Acknowledgments

➢ Dr. Werner von Bloh ➢ Roger Grzondziel ➢ Achim Glauer ➢ Karsten Kramer ➢ Dr. Ciaron Linstead ➢ Kerstin Heuer, ➢ Frauke Haneberg ➢ Ingo Deutsch ➢ Carsten Goldhan ➢ Torsten Kurz ➢ Klaus Hassels ➢ Martin Hiegl ➢ Christoph Pospiech ➢ Klaus Gottschalk ➢ Michael Lauffs ➢ Torsten Bloth ➢ Michael Julien ➢ Steffen Schwab ➢ Mike Kruse-Heidler

➢ Maik Bornhardt, Axel Jahn and

PIK

Gneise 66

(4)

Requirements

➢ Reliable, high performance general purpose compute facility

➢ Application benchmarks: LPJ (C+MPI2), CLIMBER (F77), MOM4 (F90+MPI), S (C++/

MPI)

➢ Power uptake limited to 100 kW (compute) /150 kW (system) maximum

➢ Parallel file system, Backup/Restore and Hierarchical Storage Management

➢ File system with 4 GB/s, extension of existing tape library and TSM (LAN free) storage

infrastructure

➢ Integration into building infrastructure (UPS, Cooling)

➢ Replacement of old Cluster, installation of new UPS with 30 Minutes/250 kVA ➢ Control engineering, limited airflow of about 17.000 m3/h (max. 100 kW by air)

➢ Service and technical support

➢ 4 yrs. Hardware maintenance and Software Support (OS, Management, Compilers) +

Microcode support

➢ Financing

(5)

Procurement and Installation

➢

Tender and negotiation process (EU)

➢ June 11th - October 30th 2008.

➢

IBM awarded contract for secondary offer (Nebenangebot) based on

iDataPlex ™ systems

➢ October 31st 2008.

➢

Preparation of raised floor – electricity and cooling

➢ November 3rd - 14th 2008.

➢

Hardware delivery and on-site installation

➢ November 17th – December 3rd 2008.

➢

Software Installation (OS, LAN, IB, SAN, Management)

➢ December 3rd - 17th 2008.

(6)

Installation (cont.)

➢

System accepted with conditions

➢ December 31st 2008.

➢

Annual maintenance facilities – extended by UPS installation and

control engineering

➢ January 2nd - 9th 2009.

➢

I/O tuning

➢ January 12nd - 23rd 2009.

➢

TSM/HSM Installation and Integration

➢ February 2nd - 6th 2009.

➢

Additional Software (Compilers, Debugger, Batch Queuing, etc.)

➢ February 9th - on-going!

➢

Scheduled production:

(7)

Hardware

➢

2560 Cores in 320 Nodes with

two Intel Xeon E5472

3GHz/1600 MHz QC CPU each.

➢

10 TByte RAM 800 MHz

FB-DIMM.

➢

4x DDR Voltaire Infiniband

Interconnect (two fabrics).

➢

200 TByte GPFS / 15krpm FC.

➢

1 Pbyte tape cartridges and 8 x

(8)

Software

Control ➢ Novell SLES10 SP2 x86-64 ➢ IBM AIX 6.1 ➢ OFED 1.3 ➢ Xcat 2.1 ➢ IBM GPFS 3.2.1 (RDMA) ➢ IBM LoadLeveler 3.5.3

➢ Tivoli Storage Manager 5.5.3,

incl. HSM

➢ Cisco IOS 12.2(18)SXF7

➢ Voltaire Fab. Manager 5.2.0

➢ Brocade Fab. Manager 5.3.1

Applications

➢ Intel Cluster Toolkit Compiler

Edition 3.2 (C++ , FTN, MPI)

➢ Intel Vtune Performance

Analyzer 9.1

➢ GCC 4.1.2/ Open MPI

➢ Total View Debugger 8.6.2

➢ Matlab

➢ Mapping

➢ Optimization ➢ Signal Processing ➢ Statistics

(9)

System Architecture

(10)

Compute Subsystem

IBM iDataPlex ™

Model IBM dx360 ➢Diskless ➢16 x memory slots ➢eth0 + bmc/ipmi 2.0 ➢Mellanox ConnectX Dual-Port 4X DDR IB PCI-E 2.0 x8 5.0GT/s ➢Emcore Connects

Optical Cables Two 3-phase PDU+

➢12 x C13 outlets ➢Webserver

Two Cisco 3750G-48TS 4 x 1000Base-TX uplinks Reardoor Heat Exchanger ~ 25 kW

(11)

MEMORY BANDWIDTH

Intel Harpertown/Seaburg

E5472/5400B

➢

Dual QC Xeon 3 GHz

➢

32 GB RAM - 16 x 2GB

DDR2-800 FBDIMM AMB+

➢

4 memory channels x 800 MHz

➢

UMA

➢

64 MByte Snoop-Filter

➢

25,6 GB/s max. Bandwidth

(4 x 800 Mhz x 8 Byte)

STREAM

MB/s COPY SCALE ADD TRIAD

dx360/8 11242.36 11273.16 9721.32 9742.68

js22/4 13606,58 13589,12 15416,37 15456,58

(12)

FIRST MEMORY TEST

(13)

Voltaire Interconnect

➢ Two Voltaire 2012 x 192 Ports (8 of 12

linecards; 2 x 5 ports free).

➢ Mellanox ConnectX Dual-Port 4X DDR

IB PCI-E 2.0 x8

➢ Optical cables highly recommended!

➢ Netpipe MPI Latency (Intra/Interswitch)

1,48s / 2.25s.

➢ Netpipe MPI Bandwidth

(Intra/Interswitch) 14.9 Gbps / 14.6 Gbps. root@nsds01[0]:~# ibstatus Infiniband device 'mlx4_0' port 1 status: state: 4: ACTIVE phys state: 5: LinkUp rate: 20 Gb/sec (4X DDR) Infiniband device 'mlx4_0' port 2 status:

(14)

Voltaire Interconnect

(15)

-Application Benchmarks

Runtime/s HS20 2 DC Woodcrest 3 GHz DX360 2 QC Harpertown 3 GHz JS22 2 DC Power6 4 GHz CLIMBER (1) 1120.00 785.68 1407.56 * MOM-4 (45) 15259.13 9038.66 4378.00 LPJ spinup (32) 2984.00 2256.00 2408.00 LPJ output (32) 698.5 498.00 637.00

(16)

Electrical Power Consuption

PDU+ x 8 ➢ Node ➢ 2 x 80 W Chip TDP ➢ 32 GB FBDIMM ➢ 190 W (idle) ➢ 312 W (busy) ➢ PDU+ – 40 x DX360 = ½ Rack ➢ 7.5 kW (idle) ➢ 12.5 kW (busy) ➢ System ➢ 60 kW (idle) ➢ 100 kW (busy) +/- 40 kW

(17)

Reardoor Heat Exchangers

Secondary cold water circuit

➢16° C ➢14 m3/h

Water pumps adjust every 30 Minutes!

(18)

… by the way …

iDataplex has no redundant power!

“Autonomiezeit” 30 minutes

with 200 kW load.

…

How would you cool 200 kW without electrical power?

(19)

GPFS I/O Subsystem

[Klaus Gottschalk, Torsten Bloth, IBM]

NSDS:

2 x PCIe 8x

 Dual Port HBA

 Dual Port HCA

2 xPCIe 4x

Maximum transfer rates for one 200 Tbyte Filesystem using 16 clients

(20)

GPFS I/O Subsystem

NSDS:

2 x PCIe 8x

 Dual Port HBA

 Dual Port HCA

2 xPCIe 4x

 Single Port HBA

and one 1 TB file:

➢ 5.3 Gbps write

(21)

GPFS I/O Subsystem

NSDS:

2 x PCIe 8x

 Dual Port HBA

 Dual Port HCA

2 xPCIe 4x

(22)

I/O Tuning Considerations

➢ Storage Servers (DS)

➢ Enclosure Cabling, LUN Layout + LUN Mapping

➢ Block + Cache Settings (i.e. dynamic prefetch disabled)

➢ Network Shares Disk Servers (NSDS)

➢ Drivers (!) MPP vs. RDAC (Downgrade to SLES10/SP1)

➢ Max. Sectors (disk) + Max. Depth (hba)

➢ Filesystem

➢ Disk Layout, Blocksize (256KB x 8 = 2 MB), number of threads, pinned

memory, prefetch, etc.

➢ Storage Area Network

➢ Zoning (!), WWPN vs. Port Zoning, Switch Firmware, Port to ASIC

➢ Interconnect

(23)

LUNs (DataAndMetadata); CTRL A A = dam1 B = dam2 C = dam3 D = dam4 E = dam5 F = dam6 G = dam7 CTRL B h = dam8 i = dam9 j = dam10 k = dam11 l = dam12 m = dam13 A A A B B C C D D E E E F F G G Enc 11 slot 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 A A B B B C C D D E E F F F G G Enc 22 A A B B C C C D D E E F F G G G Enc 33 A A B B C C D D D E E F F G G HS Enc 44 h h i i j j k k k l l m m n n HS Enc 66 h h i i j j j k k l l m m n n n Enc 77 h h i i i j j k k l l m m m n n Enc 88

New LUN Disk Layout

(24)

1 2 3 4 Server 1 1 2 3 4 Server 2 1 2 3 4 Server 3 1 2 3 4 Server 4 A B 1 2 3 4 1 2 3 4 A Ctrls san03 B Ctrls san04 A B 1 2 3 4 1 2 3 4 A B 1 2 3 4 1 2 3 4 A B 1 2 3 4 1 2 3 4 2/2 3/1 2/1 4/1 ds12 ds13 ds14 ds15 2/2 3/1 2/1 4/1 2/2 3/1 2/1 4/1 2/2 3/1 2/1 4/1 B1

SAN Zoning

(25)

TSM/HSM Subsystem

[Klaus Gottschalk, IBM]

Restore of one file system from new tapes/tape drives into preproduction GPFS. One IBM E06 drive used! Restore processing finished Total number of objects restored: 2 661 756 Total number of objects failed: 62 Total number of bytes transferred: 12.34 TB Data transfer time: 77 943.60 sec Network data transfer rate: 170 093.47 KB/sec Aggregate data transfer rate: 124 869.86 KB/sec Elapsed processing time: 29:29:32

(26)

(27)

Summary

➢

IBM iDataplex promised best price/performance and best electrical

efficiency for PIK application benchmarks.

➢

Detailed technical planning upfront installation require, authorization

recommended.

➢

UPS is mandatory – but mind the cooling after power is interrupted!

➢

Though it (still) looks like an easy set-up the devil is in the detail:

➢ Electrical power uptake varies significantly with application.

➢ Tight control of secondary chilled water circuit required (a good

facility manager, that is),

➢ Would have liked a Cisco NAM2 or eq. installed in the central ADMIN

LAN switch for debugging.