Implementation and experience with a 20 4 TFLOPS IBM BladeCenter cluster

(1)

Implementation and experience with a 20.4

TFLOPS IBM BladeCenter cluster

Craig A. Stewart

Matthew Link, D. Scott McCaulay, Greg Rodgers,

George Turner, David Hancock, Richard Repasky,

Faisal Saied, Marlon Pierce, Ross Aiken,

Matthias Mueller, Matthias Jurenz, Matthias Lieber

(2)

Outline

• Background about Indiana University

• Brief history of implementation

• System architecture

• Performance analysis

(3)

Introduction - IU in a nutshell

• ~$2B Annual Budget

• One university with 8 campuses; 90.000 students, 3.900 faculty

• 878 degree programs, including nation’s 2nd largest school of medicine

• President Elect: Michael A. McRobbie

• IT organization: >$100M/year IT budget, 7 Divisions

• Research Technologies Division - responsible for HPC, grid, storage, advanced viz

• Pervasive Technology Labs (Gannon, Fox, Lumsdaine)

(4)

Big Red - Basics and history

• Spring 2006: assembled in 17 days at IBM

facility, disassembled, shipped to IU,

reassembled in 10 days.

• 20,4 TFLOPS peak theortical, 15,04 achieved

on Linpack. 23rd on June 2006 Top500 List.

• In production for local users on 22 August 2006,

for TeraGrid users 1 October 2006

• Best Top500 rankingin IU history

• Upgraded to 30,72 TFLOPS Spring 2008, ???

on June 2007 Top500 List

(5)

Motivations and goals

• Initial goals for 20,4 TFLOPS system:

Local demand for cycles exceeded supply

TeraGrid Resource Partner commitments to meet

Support life science research (Indiana Metabolomics

and Cytomics Initiative - MetaCYT)

Support applications at 100s to 1000s of processors

• 2nd phase upgrade to 30,7 TFLOPS

(6)

TeraGrid

Motivation for being part of TeraGrid:

• Support national research agendas

• Improve ability of IU

researchers to use national cyberinfrastructure

• Testbed for IU computer science research

(7)

Why a PowerPC-based cluster?

• Processing power per node

• Density, good power efficiency relative to

available processors

• Possibility of performance gains through use of

Altivec unit & VMX instructions

• Blade architecture provides flexibility for future

• Results of RFP

6,0 167

PowerPC 970 MP (dual core)

(8)

Networks Disk storage

Computational hardware

96 GB/sec - Myrinet 2000 5 GB/sec - Gigabit Ethernet Bisection bandwidth

40 Gbit/sec Total outbound network bandwidth

30.72 TeraFLOPS Theoretical performance

25 TB Home directory space

535 TB Lustre

266 TB GPFS scratch space

2.25 TB total Local hard disk per blade

6 TB Total system memory

1,536 processors; 3,072 processor cores No. of processors; cores

768 No. of JS21 blades

Two 2.5 GHz PowerPC 970MP processors, 8 GB RAM, 73 GB SAS Drive, 40 GFLOPS

JS21 components

(9)

(10)

(11)

(12)

(13)

(14)

IBM e1350 vs Cray XT3

(15)

(16)

66,3

13,53

20,40

510 nodes

HPCC

Top500

Benchmark

set

70,9

21,79

30.72 768 nodes

73,4

15,04

20,48

512 nodes

%

Achieved

Peak

Nodes

(17)

(18)

(19)

(20)

(21)

Comparative performance-NAMD

DataStar (SDSC, 768 P, 14.3 TFLOPS) Mercury (NCSA, 1262 P, 10.23 TFLOPS) Big Red (IU, 512 P, 20.4 TFLOPS) Cobalt (NCSA, 512 P, 6.55 TFLOPS) 0.024 0.022 0.012 0.005 256 0.039 0.024 0.019 0.019 128 0.075 0.041 0.034 0.032 64

(22)

• Simulation of TonB-dependent transporter (TBDT)

• Used systems at NCSA, IU, PSC

• Modeled mechanisms for

allowing transport of molecules through cell membrane

• Work by Emad Tajkhorshid and James Gumbart, of University of Illinois Urbana-Champaign • Mechanics of Force

Propagation in

(23)

ChemBioGrid

• Analyzed 555,007

abstracts in PubMed in ~ 8,000 CPU hours

• Used OSCAR3 to find SMILES strings -> SDF format -> 3D structure

(GAMESS) -> into Varuna database and then other applications

(24)

WxChallenge

• Over 1,000 undergraduate students, 64

teams, 56 institutions

• Usage on Big Red:

~16,000 CPU hours on Big Red (most of any

TeraGrid resource)

63% of processing done on Big Red

(25)

(26)

Overall usage to date

???

WRF

NAMD

(27)

Overall user reactions

• NAMD, WRF users very pleased

• Some community codes essentially

excluded

• Porting from Intel instruction set a

significant perceived challenge in a

cycle-rich environment

(28)

Overall evaluation & conclusions

• The manageability of the system is excellent

• For a select group of applications, Big Red

provides excellent performance and reasonable

scalability

• We are likely to expand the 10GigE from Big

Red to the rest of the IU cyberinfrastructure

• We are installing a 7 TFLOPS Intel cluster;

model in future to be Intel-compatible processors

as “default entry point,” more specialized

(29)

Pace of change

• The most powerful system attached to the TeraGrid has changed 3 times since June 2006

• Absolute rate of change feels very fast

3,1 322

Clovertown (quad core, 2.3 GHz)

5,0 200

Woodcrest (dual core, 3 GHz)

6,0 167

PowerPC 970 MP (dual core)

Mwatts/ PetaFLOPS TFLOPS/

(30)

Conclusions

• A 20.4 TFLOPS system with “not the usual” processors was successfully implemented serving local Indiana University researchers, and the national research audience via the

TeraGrid (IU is 5th in providing cycles to TeraGrid at present) • We had excellent success in some regards with the system;

excellent response in some niches

• In the future Science Gateways may be more and more important in improving usability:

It’s impossible to expect most scientists to chase after the fastest available system when the fastest system is changing 3 times a year

Programmability of increasingly unusual architectures not likely to become easier

For applications with broad potential user bases, or extreme

(31)

Acknowledgements - funding

agencies

• IU’s involvement as a TeraGrid Resource Partner is supported in part by the National Science Foundation under Grants No. ACI-0338618l, OCI-0451237, OCI-0535258, and OCI-0504075 • The IU Data Capacitor is supported in part by the National Science Foundation under Grant

No. CNS-0521433.

• This research was supported in part by the Indiana METACyt Initiative. The Indiana METACyt Initiative of Indiana University is supported in part by Lilly Endowment, Inc.

• This work was supported in part by Shared University Research grants from IBM, Inc. to Indiana University.

• The LEAD portal is developed under the leadership of IU Professors Dr. Dennis Gannon and Dr. Beth Plale, and supported by grants ###___

• The ChemBioGrid Portal is developed under the leadership of IU Professor Dr. Geoffrey C. Fox and Dr. Marlon Pierce and funded via the Pervasive Technology Labs (supported by the Lilly Endowment, Inc.) and the National Institutes of Health ###)))

• Many of the ideas presented in this talk were developed under a Fulbright Senior Scholar’s award to Stewart, funded by the

(32)

Acknowledgements - People

• Malinda Lingwall deserves thanks for most of the .ppt layout work • Maria Morris contributed to the graphics used in this talk

• Marcus Christie and Surresh Marru of the Extreme! Computing Lab contributed the LEAD graphics

• John Morris (www.editide.us) and Cairril Mills (cairril.com Design & Marketing) contributed graphics

• This work would not have been possible without the dedicated and expert efforts of the staff of the Research Technologies Division of University Information Technology Services, the

faculty and staff of the Pervasive Technology Labs, and the staf of UITS generally.

(33)

Author affiliations

Craig A. Stewart; [email protected]; Office of the Vice President and CIO, Indiana University, 601 E. Kirkwood, Bloomington, IN

Matthew Link; [email protected]; University Information Technology Services, Indiana University, 2711 E. 10th_{St., Bloomington, IN 47408}

D. Scott McCaulay, [email protected], University Information Technology Services, Indiana University, 2711 E. 10th_{St., Bloomington, IN 47408}

Greg Rodgers; [email protected]; IBM Corporation, 2455 South Road, Poughkeepsie, New York 12601

George Turner; [email protected]; University Information Technology Services, Indiana University, 2711 E. 10th_{St., Bloomington, IN 47408}

David Hancock; dyhancoc@iupui,edu; University Information Technology Services, Indiana University — Purdue University Indianapolis, 535 W. Michigan Street, Indianapolis, IN 46202

Richard Repasky; [email protected], University Information Technology Services, Indiana University, 2711 E. 10th_{St., Bloomington, IN 47408}

Peng Wang; [email protected]; University Information Technology Services, Indiana University — Purdue University Indianapolis, 535 W. Michigan Street, Indianapolis, IN 46202

Faisal Saied; [email protected]; Rosen Center for Advanced Computing, Purdue University, 302 W. Wood Street, West Lafayette, Indiana 47907 Marlon Pierce; Community Grids Lab, Pervasive Technology Labs at Indiana University, 501 N. Morton Street, Bloomington, IN 47404

Ross Aiken; [email protected]; IBM Corporation, 9229 Delegates Row, Precedent Office Park Bldg 81, Indianapolis, IN 46240;

Matthias Mueller; [email protected]; Center forInformation Services and High Performance Computing (ZIH) Dresden University of Technology D-01062 Dresden, Germany

Matthias Jurenz; [email protected]; Center forInformation Services and High Performance Computing (ZIH) Dresden University of Technology D-01062 Dresden, Germany

(34)