Implementation,
performance, and science
results from a 30.7 TFLOPS
IBM BladeCenter cluster
Craig A. Stewart,
1,∗Matthew Link,
2D. Scott
McCaulay,
2Greg Rodgers,
3George Turner,
2David
Hancock,
2Peng Wang,
4Faisal Saied,
5Marlon
Pierce,
6Ross Aiken,
7Matthias S. Mueller,
8Matthias Jurenz,
8Matthias Lieber
8and Jenett
Tillotson
2Beth A. Plale
91 Pervasive Technology Institute, Indiana University, 601 E. Kirkwood, Bloomington, IN 47408
2 University Information Technology Services, Indiana University, 2711 E. 10thSt.,
Bloomington, IN 47408
3 IBM Corporation, 2455 South Road, Poughkeepsie, NY 12601
4 University Information Technology Services, Indiana University–Purdue University
Indianapolis, 535 W. Michigan Street, Indianapolis, IN 46202
5 Rosen Center for Advanced Computing, Purdue University, 302 W. Wood Street, West
Lafayette, IN 47907
6 Community Grids Lab, Pervasive Technology Institute, 501 N. Morton Street, Bloomington,
IN 47404
7 IBM Corporation, 9229 Delegates Row, Precedent Office Park Bldg 81, Indianapolis, IN
46240
8 GermanyTechnische Universit¨at Dresden, 01062 Dresden, Germany
9 Data to Insight Center, Pervasive Technology Institute, Indiana University, 215 Lindley
Hall, Bloomington, IN 47405-7104
SUMMARY
This paper describes Indiana University’s implementation and performance testing of a large high performance computing system locally and on the US TeraGrid. IU’s Big Red, a 20.4 TFLOPS IBM e1350 BladeCenter cluster, was ordered on 13 April 2006; on 28 June 2006, it appeared in the 27th
Top500 list as the 23rd
fastest supercomputer in the world. In spring 2007, this computer was upgraded to 30.72 TFLOPS. The e1350
∗Correspondence to: Craig A Stewart, Office of the Vice President and CIO, Indiana University, 601 E.
Kirkwood, Bloomington, IN 47408
†E-mail: stewart@iu.edu, mrlink@indiana.edu, smccaula@indiana.edu, rodgersg@us.ibm.com,
turn-erg@indiana.edu, dyhancoc@indiana.edu, fsaied@purdue.edu, mpierce@cs.indiana.edu, rmaiken@us.ibm.com, Matthias.Mueller@tu-dresden.de, Matthias.Jurenz@tu-dresden.de, Matthias.Lieber@tu-dresden.de, jtillots@indiana.edu, plale@cs.indiana.edu
architecture, including two internal networks accessible to users and user applications and two networks used exclusively for system management, has enabled the system to provide good scalability on many important applications while being well manageable. Implementing a system based on the JS21 Blade and PowerPC 970MP processor within the US TeraGrid presented certain challenges, given that Intel-compatible processors dominate the TeraGrid. However, the particular characteristics of the PowerPC have enabled it to be highly popular among certain application communities, particularly users of molecular dynamics codes and weather forecasting. A critical aspect of Big Red’s implementation has been a focus on Science Gateways, which provide graphical interfaces to systems supporting end-to-end scientific workflows. Several Science Gateways have been implemented that access Big Red as a computational resource–some via the TeraGrid, some not affiliated with the TeraGrid. In summary, Big Red has been successfully integrated with the TeraGrid, is used by many researchers via grids and Science Gateways, and has been a success in terms of enabling scientific discoveries at IU and, via the TeraGrid, across the US.
key words: HPC; blade cluster; TeraGrid; Science Gateway; benchmarking
1. INTRODUCTION
Assembling a very large high-performance computing system is a significant challenge. Making such a system straightforward to manage is a relatively greater challenge. Creating a very large-scale high-performance computing (HPC) system and making it straightforward to program and use is even more difficult. The purpose of this paper is to describe the architecture, management facilities, performance characteristics, and interfaces of a large HPC cluster—an IBM e1350 BladeCenter cluster called Big Red. We also discuss some of the practical challenges encountered in implementing Big Red as part of the TeraGrid and describe highlights of our experiences with this system.
At an initial capability of 20.48 TFLOPS, Big Red was ranked 23rd
on the June 2006 Top500 list of fastest (unclassified) HPC systems in the world [1]. At that time, it was the fastest academic supercomputer in the Western Hemisphere. After an upgrade to 30.72 TFLOPS, Big
Red ranked 30th
on the June 2007 Top500 list [1]. Big Red is the largest HPC system in absolute terms, and has had the highest placement on the Top500 list, in Indiana University’s history. Indiana University (IU) had several purposes for implementing such a large supercomputer system. Acceleration of life science research was a key goal, as IU has set the life sciences and information technology as two of its top strategic priorities [2, 3]. In addition, the National Science Foundation funds IU as a Resource Partner within the TeraGrid [4], and Big Red was planned as part of IU’s contribution to support the national research community via the TeraGrid.
Big Red’s users are diverse in skill and background. They range from some of the most sophisticated computational scientists in the US to undergraduate students who use the system as part of a weather prediction contest with little knowledge of the details of HPC systems. Our experiences implementing this large system should be of interest to the high performance
computing community in general, and in particular to institutions planning to take new steps in significantly increasing their local HPC facilities.
2. ARCHITECTURE OF BIG RED
Big Red is based on the open blade cluster architecture [5], employing the IBM PowerPC 970MP processor. Systems based on these processors are unusual within both the Top500 list and the TeraGrid. There were several reasons for selecting a system based on the PowerPC architecture. Having decided in early 2006 to make a major investment in an HPC system, IU issued a call for proposals for a large supercomputer cluster, in compliance with IU policies and state laws. This call was issued on 3 March 2006. The system, purchased on 13 April 2006, was selected based on analysis of the proposals submitted by supercomputer vendors. The primary factors in the final selection were functionality in support of IU’s science agenda, price/performance, and quality of vendor support for implementation. Technical reasons for choosing the Power architecture, and the JS21 blade and PowerPC 970MP processor in particular, were as follows:
• The JS21 blade, with two dual-core PowerPC 970MP processors per blade, provides 40
GFLOPS per node, one of the fastest individual nodes in any cluster at the time of purchase.
• The PowerPC 970MP processor provides an energy efficient dual-core processor
(considering the processor alone, 200 TFLOPS per megawatt, or 5 megawatts per PFLOPS). This was one of the most power-efficient processors available at the time of purchase.
• The PowerPC 970MP offers the possibility of optimizing applications by exploiting the
vector capabilities of the AltiVec vector processing unit.
There are also benefits to the blade architecture in general, including the high density of processing power per square yard of machine room space, ease of management, and the possibility of upgrading the system in the future simply by replacing the blades.
After the initial 20.4 TFLOPS implementation, Big Red was upgraded to 30.72 TFLOPS in spring 2007. This upgrade was part of an economic development initiative in the state of Indiana (US), led jointly by Indiana University, Purdue University, the Indiana Economic Development Corporation (IEDC), and IBM, Inc. [6].
The hardware specifications of Big Red are summarized in Table I, showing its initial 20.48 TFLOPS configuration as well as the upgraded 30.72 TFLOPS configuration. The basic architecture of Big Red is very much like MareNostrum [7]. Full technical details on the e1350 BladeCenter, the IBM JS21 blade, and the PowerPC 970MP are available at [8] and [9].
Figure 1 shows a schematic of Big Red, secondary storage systems that support it, and its network connections to the TeraGrid. The basic subunits of Big Red are the JS21 blade and the IBM BladeCenter H Chassis. Physically, fourteen JS21 blades are installed in one chassis. Four chassis are installed per rack, for a total of 56 blades per rack.
The performance and ease of management of this system are rooted in the software management and in the network architecture. The Big Red network architecture includes
14 x 2 Gb Myrinet Network Myrinet CLOS 256+256 Myrinet CLOS 256+256 Myrinet CLOS 256+256 Myrinet Spine1024 Agg 266TB Data Capacitor Agg 535TB x8 Big Red x12 Lustre Disk Agg 1TB x2 Lustre Disk NAS Agg 25TB NFS N5500 A20 N5500 A20 Dell 2950 Dell 2950 Dell 2950 DDN S2A9500 Force10 E1200 Lustre
OSS 10Gb 10Gb Lustre Metadata 1Gb
Multiple 10Gb to campus, I-Light, Abilene To TeraGrid 2 x 10Gb IBM p505 IBM p505 DDN S2A9550 GPFS Servers Force10 E600 Force10 E600 4 x 10Gb 4 x 10Gb 4 x 1Gb x 38 4 x 1Gb x 18 Global Gigabit Network
GPFS Disk BladeCenter ® H Chassis 2 x 1Gb 2 x 1Gb 256 x 2Gb 256 x 2Gb 256 x 2Gb 4 x 1Gb 4 x 1Gb DDN S2A9550 ... ... ... x 56 JS21
Figure 1. Schematic diagram of Big Red (in its 30.72 TFLOPS configuration) and the IU Data Capacitor, showing connections to the IU local cyberinfrastructure and TeraGrid. The IBM JS21 blades each have an individual connection to one of three Myrinet CLOS switches. Each IBM BladeCenter H chassis has 4 x 1 Gbps connections to one of two Force10 E600 switches. Multiples of
components are indicated by xN. (e.g., there are 56 BladeCenter H chassis within Big Red.)
four interconnects. Two are accessible to users and user applications (shown in Figure 1); two are accessible only to systems administrators and system management applications (not shown in Figure 1).
A management and monitoring network and the system software network support the operation of Big Red. The management and monitoring network is used by the system components that monitor system status and control blade functions. This represents a routine approach to operation of a large cluster. The system software network is part of an innovative approach to systems management. This network delivers software images to blades at boot time using IBM’s Distributed Image Management (DIM) system [10]. The system software network connects a master node, software image servers, and the blades within Big Red. The master node holds the authoritative copy of the system software used in each of the JS21 blades (the system includes a hot failover spare master node). The master node connects via Force10 E600 switches to one IBM p505 image server installed in each rack. This image server connects to a Nortel Ethernet switch located in each BladeCenter H Chassis. These Nortel switches then connect with each JS21 Blade within the chassis. There is, then, a dedicated network connecting the master node to each image server, and then connecting each image
server to the JS21 blades within each of the racks within Big Red. A cold boot, starting from all servers in a powered-off state (including GPFS servers), takes less than 15 minutes. The transfer of images from the p505 image servers to nodes is faster than reading the images off of the local hard disks internal to the JS21.
In addition to speeding the boot process, the DIM system dramatically speeds and simplifies the process of updating and changing system images. It takes up to ten minutes to propagate a changed file from the master node to all of the image servers. The speed of the propagation varies depending on file size and number of changes. A significant change in the system software can be installed on the master node and be in use on the entire system in 25 minutes or less. The two interconnects within Big Red available to users and user applications are a low-latency internal Myrinet2000 interconnect and gigabit Ethernet. Each JS21 blade has a connection to the Myrinet2000 switch fabric. We recommend this to users and software developers as the preferred interconnect for MPI communications. Each JS21 blade has a connection to an Ethernet network in the backplane of the BCH chassis within which it resides. From each BCH chassis there are four or six outbound Gigabit Ethernet connections to a Force10 E600 switch. The gigabit Ethernet connections to each JS21 blade are thus oversubscribed—14 blades to 4 or 6 gigabit Ethernet connections. In the current 30.72 TFLOPS configuration, 512 JS21 blades are connected to one Force10 E600 switch and 256 JS21s are connected to a second E600. The two E600s are interconnected by four 10 Gigabit Ethernet connections.
I/O to disk storage traverses the Gigabit Ethernet network. Storage in support of user applications is provided by four systems—two internal to Big Red and two accessed by the 4 x 10 GigE connections from Big Red. These four 10 GigE connections provide communication with a Force10 E1200 switch that serves as a machine room backplane for the IU Bloomington machine room. The two disk systems within Big Red are as follows:
• Scratch space.A 266 TB General Parallel File System (GPFS) pool [11] provides scratch
space. There are a total of 16 IBM p505 nodes acting as GPFS servers, each with two bonded 1 Gbit Ethernet connection to the Force10 E600 switches. Each p505 GPFS server connects to 2 DataDirect S2A9550 controllers, and each controller connects to 240 DataDirect SATA 500 GB 7200 RPM disks. These disks are configured for RAID 6 using eight data drives and two parity drives within a logical unit (LUN). This allows for written data to be recovered in case of up to two disk failures within a LUN and also to provide automatic data correction on reads if parity errors are detected during those reads [12]. The I/O rate for an individual GPFS server is the network bound, which limits file I/O to 250 MB/sec per GPFS server. The GPFS pool is thus capable (theoretically) of an aggregate 4 GB/s read and write rate.
• Scratch space local to each node. One 73 GB SAS drive internal to each JS21 node.
Application developers and users are discouraged from using these drives and encouraged to use the much more robust and fault-tolerant GPFS pool instead.
Big Red communicates through the Force10 E1200 switch with other components of the IU cyberinfrastructure and national and international networks. Within the IU cyberinfrastructure are two additional disk systems accessible to Big Red:
• Home directory space.An IBM N5500 Network Attached Storage (NAS) system with 25 TB of RAID5 storage provides home directory storage. Two N5500 model A2 NAS heads are configured for automatic failover, providing redundant access to home directories. The system uses 112 IBM 320 GB 7200 RPM SATA disks in fiber channel attached enclosures.
• High speed, short-term storage.A Lustre file system named the Data Capacitor provides
high speed, short-term storage. (The technical details of the Data Capacitor and information on its performance characteristics are described in [13, 14, 15].)
Connections to national networks include 10 Gbps link to the Abilene network, the I-Light network within Indiana, and two 10 GigE network links to the TeraGrid backplane. (The TeraGrid network is described in detail in [4].) In order to provide good support for grid applications, each node within Big Red is accessible by its own routable IP address.
3. SYSTEM IMPLEMENTATION, MANAGEMENT, AND INTEGRATION WITH THE TERAGRID
Big Red was implemented very rapidly. The system was assembled and benchmarked at an IBM facility over the course of 17 days, moved to IU’s Bloomington campus, and reassembled in 10 days. Big Red was operated in “friendly user” modes for a very few weeks and was placed in production mode serving local users on 22 August 2006. Integration with the TeraGrid took somewhat longer. All systems attached to the TeraGrid run the Coordinated TeraGrid Software and Services (CTSS) [16]. A very large cluster running Linux on the Power architecture was a new challenge for the TeraGrid’s CTSS development team. It took 8 weeks from initial reqsts to the CTSS team until we had a functioning implementation of CTSS on PowerPC and Linux. The elapsed time was primarily the result of the delays inherent in the feedback process of the CTSS team making changes in the code, implementing the changes on Big Red (which generally required waiting for a scheduled reboot on a Sunday morning), testing, and identifying problems. With a need to make local production use of Big Red at the same time we were implementing and testing CTSS patches, 8 weeks was a highly satisfactory time to solution. Big Red was operational and supporting allocations within the TeraGrid on 1 October 2006.
4. PERFORMANCE
Table II shows High Performance Computing Challenge (HPCC) benchmark suite results [17] for tests using 510 Blades (1020 PowerPC 970MP processors; 2040 cores each supporting one MPI process). These benchmarks were run using the MPICH 1.2.7 message passing interface [19] and GotoBLAS libraries [18].
Figure 2 shows Kiviat diagrams comparing the performance of Big Red against other large systems implemented shortly before Big Red—a 24.96 TFLOPS Cray XT3 at Oak Ridge National Labs and a 2.09 TFLOPS HP cluster owned by HP, Inc.
0.2 0.4 0.6 0.8 1.0 PP-HPL y c n e t a L g n i R m o d n a R h t d i w d n a B g n i R m o d n a R M M E G D - N S SN-STREAM Triad E T F F - P P s s e c c A m o d n a R - P P S N A R T P - P P
Model Processors Interconnect Date
IBM e1350 1020 x 2.5 GHz PPC 970MP Myrinet2000 Jun 07 Cray XT3 5200 x 2.4 GHz Opteron Cray XT3 MPP Aug 05
PP-HPL M M E G D - N S SN-STREAM Triad E T F F - P P s s e c c A m o d n a R - P P S N A R T P - P P 0.2 0.4 0.6 0.8 1.0 y c n e t a L g n i R m o d n a R h t d i w d n a B g n i R m o d n a R
Model Processors Interconnect Date
IBM e1350 1020 x 2.5 GHz PPC 970MP Myrinet2000 Jun 07 HP XC4000 256 x 2.6 GHz Opteron Infiniband 4xSDR Dec 06 Figure 2. Kiviat diagrams comparing HPCC results of Big Red and other HPC systems. On left is shown a comparison of the e1350 (Big Red) rated at 20.4 TFLOPS Rpeak vs. the Cray XT3 (Jaguar) rated at 24.96 TFLOPS Rpeak at Oak Ridge National Labs. The Cray XT3 includes 5200 single core 2.4 GHz AMD Opteron processors. On right is shown a comparison with a 2.09 TFLOPS HP XC4000
owned by HP, Inc., including 256 dual-core AMD Opteron processors.
Figure 2 shows some results just as one would expect. HPL (High Performance Linpack) and DGEMM (double-precision General Matrix Multiply)—the benchmarks most dependent upon single processor or single node performance, and lightly dependent upon network characteristics—are faster on Big Red than on the Cray XT3 or the HP with Opteron processors. The difference in comparison with the Cray on a per processor basis shows much faster performance in large part because the PowerPC 970MP processor in Big Red is a dual-core processor, while the Opterons in the Cray system are single-dual-core processors. However, the comparison still favors the PowerPC even when done on a per-core basis. The per-core numerical performance advantage for Big Red is a result of the PowerPC 970MP’s ability to perform four floating point operations per core per clock cycle. Big Red is also better than either the Cray or HP in the Random Access benchmarks, and better or nearly as good in Random Ring Latency—a result of lower latency in the fairly flat Myrinet2000 interconnect of Big Red. The smaller size of the HP system contributes to the lower latency figures as compared to Big Red. The Cray XT3 with the MPP interconnect and HP XC4000 with an Infiniband interconnect are both better than Big Red in performance on bandwidth sensitive benchmark applications (Random Ring Bandwidth, PP-PTRANS, and PP-FFTE).
Table III shows a variety of HPL runs at different scales and configurations. There is a noticeable difference in HPL performance optimized for Top500 submission as opposed to the unoptimized base runs for HPCC. The unoptimized basic HPCC runs resulted in just over 66% efficiency for Big Red in its 20.4 TFLOPS configuration. Configurations optimized to maximize HPL results for the Top500 list resulted in achievement of 73.4% efficiency on that benchmark using Big Red’s original 20.4 TFLOPS configuration. We have achieved a Top500 run with HPL of 22.33 TFLOPS, or 72.7% efficiency, for Big Red in its upgraded 30.72 TFLOPS configuration. (The most important configuration change is simple—allowing pages up to 1 MB rather than the default 4 KB.)
We have run benchmark tests for several applications, including the application benchmarks in the suite used by the US National Science Foundation for some of its recent solicitations (PARATEC, MILC, WRF, and HOMME) [20] as well as other applications of particular interest to our users and collaborators (e.g., NAMD, a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems [21]). PARATEC (PARAllel Total Energy Code) is an ab initio quantum mechanics code [22]. MILC (MIMD Lattice Computation) is a particle physics lattice quantum chromodynamics code [23]. WRF (Weather Research & Forecasting Model) is a mesoscale weather prediction application [24]. HOMME (High Order Method Modeling Environment) is a general atmospheric circulation model [25].
A complete report of benchmark results is available for all of the applications mentioned and others [26]. The scaling behavior of Big Red is shown for two example applications in Figure 3. The scaling efficiency for several of the key NSF benchmark applications at different numbers of processors is shown in Table IV.
In Table V, we present WRF performance data on Big Red as compared to two systems at the Center for Information Services and High Performance Computing at the Technische Universit¨at Dresden (Germany)—Deimos and Mars. Deimos is a 13.4 TFLOPS Linux Networx Evolocity II system based on AMD x86 64 Opteron dual-core 2.6 GHz processors with an Infiniband interconnect [27]. Mars is a 13.1 TFLOPS SGI Altix 4700 system based on Intel IA-64 Itanium2 1.6 GHz processors with an SGI NUMAlink 4 interconnect [28]. The faster runtime of WRF on Mars and Deimos in comparison to Big Red are mainly due to the faster interconnects which achieve higher bandwidths. However, the scalability of WRF on Big Red is only slightly worse than on Deimos and even as good as on Mars. We used Vampir [29] as a tool to better understand the performance characteristics of the applications on the systems at Technische Universit¨at Dresden and at IU.
Data for comparing NAMD performance are available from [21]. Big Red has been since its installation among the fastest systems within the TeraGrid—on a per processor basis—for running NAMD.
Several comments and observations may be drawn from the performance analyses. Big Red provides reasonable to excellent performance and scalability on several applications, particularly NAMD. NAMD and some other molecular modeling applications have the joint characteristics of being heavily dependent upon floating point performance and using relatively small messages. Such applications are ideally suited to Big Red’s performance characteristics. Big Red, as an example of the IBM e1350 architecture, generally delivers high performance with good scalability on a broad range of applications, including computational physics
Processors i T ) c e s ( e m
Achieved vs Linear Scaling for WRF Standard and Large
1 100 1000 10,000 100,000 4 16 64 256 1 8 32 128 Achieved Linear Dataset Large Standard i T ) c e s ( e m
Achieved vs Linear Scaling for NAMD
1 10 100 1000 2 8 32 128 512 1 1 4 16 64 256 Achieved Linear NAMD a b Processors
Figure 3. Scalability of two application benchmarks, (a) WRF (Weather Research and Forecasting) and (b) the NAMD Molecular Dynamics Software. WRF scaling uses the NSF’s “standard” and “large” benchmark data sets. Results for NAMD show time to complete the 500 timesteps on the ApoA1 benchmark. Results for WRF use input from the NSF data benchmarking
(PARATEC), computational chemistry (GAMESS), computational biology (NAMD), and weather and climate modeling (WRF, HOMME).
There are other systems that provide better performance on some applications, such as WRF. We believe the difference in performance is the result of several factors, including the higher-cost interconnects and the internal communication characteristics of the processors and nodes in these other systems. We also note that it is difficult to find published performance data in peer-reviewed publications to form a basis for good comparisons on such performance data. For example, far fewer organizations have submitted data to the HPC Challenge Web site than to the Top500 list. There are particularly few peer-reviewed sources of data about HPC system performance on the NSF benchmark suite.
While we initially believed that we could get potentially interesting speedups with MILC using the AltiVec vector processor and the VMX instruction set, several months of effort failed to result in performance with MILC on Big Red that was comparable to the Cray systems available within the TeraGrid. This is a result of the importance of communications relative to computation overall in that application.
5. USABILITY—PORTALS AND SCIENCE GATEWAYS
One of our key challenges with Big Red was making it as easily accessible as possible to a group of researchers diverse in scholarly disciplines and in levels of experience and expertise with supercomputers. Traditional supercomputer users with many years experience often have the expertise and the willingness to go to great lengths to use powerful new supercomputers. However, as noted by Ahalt [30], there is a much larger group of scientists who are not using high performance computing systems because the barriers to adoption either are or seem to be very high. This is the case for many scientists (thousands to perhaps tens of thousands) even though use of HPC systems might profoundly accelerate their research. Portals and Science Gateways are two means for dramatically improving the accessibility of advanced cyberinfrastructure systems to mainstream research scientists. Portals are Web interfaces that provide one or more specific functions, without necessarily being organized into a specific task or logical series of tasks; one excellent example is the TeraGrid User Portal [31]. Science Gateways provide end-to-end support for a particular scientific workflow system [32, 33, 34]. One excellent example is LEAD (Linked Environments for Atmospheric Discovery), which provides a Science Gateway in support of dynamic prediction of severe weather [35].
We have developed a user portal that includes a set of tools making it easier for users to use Big Red. The Big Red user portal tools manage authentication, provide important user job submission information, manage movement of files to and from archival storage, and provide a portal for submission of jobs for important applications. This portal is described in detail in [36]. Considerable information of value to users of Big Red is also accessible via the TeraGrid User Portal [31].
While Science Gateways are a general concept, the TeraGrid has been at the forefront of promoting and implementing the gateway [37]. So far, Big Red has been integrated as a component of five TeraGrid Science Gateways. Of them, LEAD and nanoHUB (a Science Gateway supporting nanotechnology research and education [38]) have recorded large
amounts of usage on Big Red—more than 15,000 hours of CPU utilization each in 2008. IU has implemented an additional three Science Gateways that access Big Red in support of other federally- or locally-funded projects. One example is the Chemical Informatics and Cyberinfrastructure Collaboratory (CICC) [39] (an IU project funded by the US National Institutes of Health).
Using LEAD, weather researchers can select weather data, including real-time Doppler radar data, predict severe weather using WRF, and have the results of predictions displayed graphically on a laptop computer [40]. LEAD provides a graphical workflow composer in which a researcher assembles application tasks in a scientifically sensible order (e.g., data preparation must come before weather simulation). LEAD workflows bind to real time observational data at point of execution, so analysis is carried out on data that are minutes old. Gateway services enable weather researchers and students in fields as distant as crop prediction to run sophisticated weather simulations on a supercomputer without first becoming expert computational scientists.
An example of the power of the Science Gateway approach was provided by the 2007 WxChallenge collegiate weather forecasting challenge [41]. WxChallenge is a “playoff” competition structured like the United States college basketball championship. Sixty-four teams of college students predict the weather using LEAD. The spring 2007 competition used several TeraGrid resources, with Big Red providing more than 60% of the total computational resources. The students who participated in the WxChallenge used Big Red without ever having to interact with it at the level of the Unix shell prompt or having to learn the intricacies of the job management system; all of that was handled by the LEAD gateway. Students merely connected components running on Big Red into a graphical workflow created within the LEAD workflow management system. Big Red provided the majority of the computational resources for two primary reasons: the ability to use Big Red in concert with the IU Data Capacitor to manage and analyze large data sets was extremely helpful for the purposes of doing weather forecasting; and IU and TeraGrid staff expended considerable effort ensuring that Big Red ran WRF jobs from the LEAD portal properly.
The use of Science Gateways has contributed to growth of the number of users of the TeraGrid. Table 6 shows data from 2006 through 2008 on the total number of distinct accounts running jobs on the TeraGrid. During this time, the total number of distinct accounts per year that ran jobs on the TeraGrid grew from 1,731 to 2,476. This count underestimates the total number of distinct individuals running jobs, since every job from any particular Science Gateway runs under a single account. The number of distinct individuals running jobs via the LEAD portal rose from 11 in 2006 to 181 in 2008. Many (but not all) of the WRF jobs initiated via LEAD ran on Big Red. The LEAD portal contributed significantly to the number of different individuals who used the TeraGrid. Science Gateways such as LEAD and such as the WxChallenge, many new users of the TeraGrid execute their first jobs on the TeraGrid via a Science Gateway. Table 6 also shows the total number of jobs on HPC systems executed per year and the total number of workflows executed by users of LEAD. The total number of jobs on the TeraGrid is in the millions per year; total number of workflows executed via LEAD is in the thousands. While each LEAD workflow involves several individual HPC jobs, it is clear from these data that LEAD is contributing more strongly to the number of individuals using the TeraGrid than the number of HPC jobs executed on the TeraGrid.
The three Science Gateways developed at IU but not related to the TeraGrid all focus on some area of bioinformatics or cheminformatics. These gateways have been created in collaboration with software developers to make their software more easily accessible within and beyond Indiana University. One such exampled is the NIH-funded Chemical Informatics and Cyberinfrastructure Collaboratory (CICC) [39]. CICC is a prototype cyberinfrastructure for drug discovery research. Big Red provides the computing power needed to produce chemical structure data products for public use from NIH PubMed abstracts—although these are computed in advance, not in real time. PubMed abstracts are analyzed on Big Red by OSCAR3, a text mining application developed by Cambridge University [42] to find chemical structures. These compounds are then converted into SMILES strings, a two-dimensional representation of the structure [43]. The SMILES strings are then converted into Structure Data Format (SDF) files [44] that can be used as input to molecular mechanics applications that calculate the classical molecular structure, such as Jaguar [45]. In addition, the classical structures can also be used as inputs to protein docking applications. The results are then stored in a database wrapped in a Web service. We initially analyzed 555,007 abstracts in PubMed for the years 2005–2006 in an elapsed wall clock time of about 100 minutes. These calculations are a large scale but pleasingly parallel application. An interesting design choice in creating this Web service is the use of Big Red to precompute characteristics of a large library of compounds, which are then available for retrieval when needed via Web services. This enables separation of the availability of calculation results from availability of time on Big Red to perform calculations. In terms of allocation of computer use, this approach decreases perceived barriers to use of the information provided by the CICC. Once the structures are calculated, the computer time to search and retrieve structures is minimal. Data may be retrieved via Web services through a local variant of the NIH PubChem database (with additional data added) or through a locally developed database of chemical structures. We believe this model is better for this particular purpose than a “compute on demand” model because once the results are calculated, they are straightforward to organize and reusable and useful indefinitely. (This cheminformatics workflow system is described in detail in [46].)
There is another, practical value to use of Science Gateways that has to do with the rate of change of systems within a national infrastructure such as the TeraGrid. Between June 2006, when Big Red first appeared on the Top500 list, and the release of the November 2008 Top500 list, the fastest system integrated within the TeraGrid changed three times. While some computational experts have the skills and staff to constantly change from one system to another in order to use the fastest supercomputer available, this is not practical for many domain scientists. Science Gateway implementers can create systems that either use the fastest systems available automatically, or include application composition tools that enable researchers to easily select the fastest available computational system on which that researcher has an allocation.
We see Science Gateways as particularly important in achieving the goals set out in the
NSF’s “Cyberinfrastructure Vision for 21st
Century Discovery” [47]. The use of Science Gateways is a critical—and we think the most practicable—means to make capability and leadership-class cyberinfrastructure accessible to domain scientists who are not themselves computational experts. Complaints that high-performance computing systems are too hard to program and use effectively go back decades. Ease of programming of capability and
leadership-class supercomputers can be expected to be less rather than more straightforward as supercomputers become even more complex in the future. The use of many-core processors and specialized processors such as FPGAs is a current and clear trend in high performance computing hardware, and a trend not likely to increase ease of programming.
6. USER EXPERIENCES AND ADOPTION
There was a great deal of publicity surrounding the initial commissioning of the 20.4 TFLOPS Big Red system. The impact on the overall processing power of IU’s cyberinfrastructure was dramatic—from a total theoretical peak processing capability of roughly 3 TFLOPS to roughly seven times that. The perception of the great capability of Big Red dramatically increased demand for high-performance computing resources within IU. For example, three IU researchers proposed work that, together, would have saturated Big Red for the expected lifespan of the system. The aggregate aspirations of hundreds of IU researchers were far beyond the capabilities of Big Red. Likewise, the TeraGrid user community has made good use of Big Red. During 2007, Big Red was the fourth most heavily used supercomputer within the TeraGrid. User reaction to Big Red has thus been tremendously positive. Implementation of this large system has garnered significant user interest and enabled new scientific research that would not have been possible without it.
IU has also been successful in engaging private concerns located within the state of Indiana as users of Big Red. Industrial users, given access to Big Red through the Indiana Economic Development Initiative mentioned earlier, are now using annually tens of thousands of hours of CPU time on Big Red. These industrial users have indicated that use of Big Red is aiding their design and engineering activities, which should contribute directly to better profitability for these companies.
In both the local IU community and the national community of TeraGrid users, we have discovered benefits and costs to the implementation of a system based on the PowerPC processor. In terms of benefits, the PowerPC 970MP processor is the primary reason that Big Red has proved to be very popular with users of NAMD, Jaguar, and other molecular dynamics and modeling codes. For example, Big Red was one of three TeraGrid-connected supercomputers used by a research group modeling transport of large molecules across bacterial cell membranes with NAMD [48]. This pathbreaking simulation has significantly enhanced understanding of an important and highly complex biological process, and would not have been possible without access to such extensive computational resources. (During 2007, use of NAMD on Big Red totaled 2,079,655 CPU hours.) Because IU invested the effort required to make Big Red available via the LEAD portal, it has also proved popular with users of the weather forecasting code WRF. However, given the relative abundance of large Intel-compatible supercomputer clusters within the TeraGrid, the fact that Big Red is based on the PowerPC processor has been perceived as an obstacle to its use by some researchers. For others, it has been a real obstacle. For example, some community codes used by the physics community must be certified on any architecture they are used; the process of certification is so involved that it creates a nearly insurmountable obstacle to use of anything other than Intel-compatible architectures. Some researchers who use the TeraGrid find that their applications
run sufficiently well on the many Intel-based systems within the TeraGrid, and pursuit of optimizations available using the Power architecture seems not to be worth the effort.
The vector processing capabilities of the PowerPC architecture, which we initially viewed as one of its key advantages, bear comment. Our initial targets for VMX optimizations were MILC and BLAST. None of our optimization efforts resulted in performance as good as that available on other systems in the TeraGrid. As regards BLAST, the VMX optimized version of BLAST provides significant performance advantages over the non-optimized version (the VMX version of BLAST runs in roughly 72% of the time required for the non-VMX version) [26]. Still, we found it not practical to implement VMX versions of BLAST. The performance of BLAST on Big Red was, relative to many other systems, excellent without the VMX enhancements. The difficulty of implementing VMX optimizations and then verifying results for the several versions of BLAST used by the researchers we serve was simply more work than seemed merited.
7. CONCLUSION
We implemented a 20.4 TFLOPS IBM e1350 BladeCenter cluster, Big Red, in support of the Indiana University research community and the US national research community via the TeraGrid, and subsequently upgraded that system to 30.72 TFLOPS. The basic performance characteristics of Big Red are extremely good. This is a result of the processing power of the JS21 blade on which this system is based, the nonblocking Myrinet network interconnect, and the accessibility of the Data Capacitor Lustre file system for high-performance I/O. The use of IBM’s Distributed Image Management software and the interconnect design—with two networks accessible to users, two networks reserved for system management activities—has made the system very easily manageable.
The integration of Big Red with the TeraGrid is a good demonstration of the extensibility of the TeraGrid. Prior to Big Red, the basic software stack (the Coordinated TeraGrid Software and Services) had never been compiled and run on a Linux system based on PowerPC processors. Use of the PowerPC 970MP as the processor at the core of Big Red has had significant benefits in some regards and created, in other regards, limitations of use. Applications such as NAMD and Jaguar run particularly well on Big Red, since they take good advantage of the very high per-node performance. Other applications, particularly ones heavily dependent upon interconnect topology and latency, run better on other systems. Furthermore, the perceived cost/benefit ratio of porting applications from Intel-compatible environments to the Power environment has limited the breadth of applications used heavily on Big Red by the national TeraGrid user community. However, by focusing at the national level on a few key applications that run particularly well on Big Red, and integrating Big Red with nationally important Science Gateways such as LEAD, we have attracted a significant group of researchers to use Big Red and helped enable pathbreaking science as a result. Users local to IU have made tremendous use of Big Red; for many such researchers Big Red’s status as the largest locally available resource has led to a willing conversion of code from Intel-compatible systems to the Power architecture, with highly positive results.
One of the most significant accomplishments in deployment of Big Red has been its accessibility to multiple scientific communities via Science Gateways. Our experiences support
our view that Science Gateways will become an extremely important means by which domain science experts are able to perform computational science on a large scale. Indeed, the largest number of users of Big Red who were also new users of TeraGrid facilities used Big Red via Science Gateways. Big Red has enabled IU to achieve its local goals of accelerating research by IU scientists, particularly in the life sciences. Science Gateways have played a strong role in locally developed applications as well, as exemplified by the Chemical Informatics and Cyberinfrastructure Collaboratory.
The use of Science Gateways separates ease of programming from ease of use. Science Gateways eliminate the need for domain scientists to continually learn the details of the suites of compilers, libraries, and job-management systems implemented on the constantly changing suite of systems accessible within a given cyberinfrastructure. Computational scientists and gateway architects can build gateways that utilize the most advanced high-performance computers, and their skill and expertise allow such gateways to be built in ways that take advantage of the most sophisticated and powerful supercomputers available. Sophisticated domain scientists can then use these gateways to perform science at scale. While many Science Gateways that exist today serve as interfaces primarily to high-throughput applications, we believe that the most important long-term value of Science Gateways is to enable many more scientists to do computational science and data analyses on a large scale.
The implementation of the Big Red e1350 BladeCenter cluster has been a very successful example of grid-based use of a high-performance computing system. Its integration within the TeraGrid has proved a strong verification of the TeraGrid’s approach to the Common TeraGrid Software Stack. Our ability to attract users new to the TeraGrid, and users from within IU who are new to high performance generally, has been a strong demonstration of the value of Science Gateways as a means to enable and empower scientists from diverse disciplines and backgrounds to use high-performance computing systems.
ACKNOWLEDGEMENTS
Indiana University’s Big Red supercomputer was funded in part by the Indiana METACyt Initiative and is supported in part by a Shared University Research grant from IBM Inc. to Indiana University. The Indiana METACyt Initiative is supported by a grant from the Lilly Endowment, Inc. to Indiana University. Big Red is connected to the NSF-funded TeraGrid. IU’s participation in the TeraGrid is funded by National Science Foundation grant numbers 0338618, 0504075, and 0451237. The IU Data Capacitor was funded by grant number CNS-0521433. IU’s work with LEAD has been supported by NSF grants ATM 0331480 and EIA-0202048. CICC was supported by P20HG003894-01. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF) or the National Institutes of Health (NIH). The authors wish to thank the people involved in building CTSS software on Big Red: Eric Blau, Charles Bacon, JP Navarro (University of Chicago), and Jason Brechin (Nationl Center for Supercomputer Applications). The MyCluster component of CTSS was built by Evan Turner (Texas Advanced Computing Center) and the SRB client was built by George Kremenek (San Diego Supercomputer Center). Thanks to David Hart and Nancy Wilkens-Diehr (SDSC) and Suresh Marru (IU) for providing information and statistics related to the TeraGrid, LEAD, and Science Gateways generally. We would like to thank the staffs of the Research Technologies Division of UITS, the Pervasive Technology Institute at IU, and ZIH at Technische Universitaet Dresden, for their contributions and effort that made the implementation of Big Red and the Data Capacitor, their
integration within the TeraGrid, and the performance analysis possible. The authors would particularly like to thank Malinda Lingwall for editing, and John Morris and Cairril Mills for graphics.
REFERENCES
1. Top500 supercomputing sites.http://top500.org/ [10 March 2009].
2. Indiana University Information Technology Strategic Plan.http://www.indiana.edu/~ovpit/strategic/
[10 March 2009].
3. Indiana University Life Sciences Strategic Plan. http://lifesciences.iu.edu/background/plan.shtml
[10 March 2009].
4. TeraGrid home page.http://www.teragrid.org/[10 March 2009].
5. Blade open specification.http://www-03.ibm.com/systems/bladecenter/signin.html [10 March 2009]. 6. Indiana Initiative for Economic Development.http://rtinfo.indiana.edu/IIED/[11 March 2009]. 7. Labarta, J. Navigating the MareNostrum.Proceedings of International Supercomputer Conference 2005.
Heidelberg, Germany.
8. IBM BladeCenter JS21: The power of blade innovation.http://www.redbooks.ibm.com/redbooks/pdfs/ sg247273.pdf[10 March 2009].
9. IBM BladeCenter JS21.http://www.ibm.com/systems/bladecenter/js21/index.html[10 March 2009]. 10. Morjan P, Rodgers G, Aiken R, Turner G, Hancock D, Feinswog L. Distributed image management (DIM)
for cluster administration. 2006.http://hdl.handle.net/2022/517[10 March 2009]. 11. GPFS.http://www.ibm.com/systems/clusters/software/gpfs.html[10 March 2009].
12. DataDirect Networks Silicon Storage Appliance. http://www.datadirectnet.com/pdfs/ddn_s2a_9500_ datasheet.pdf[10 March 2009].
13. Data Capacitor.http://datacapacitor.iu.edu/[19 July 2007].
14. Simms SC, Pike GG, Teige S, Hammond B, Ma Y, Simms LL, Westneat C, Balog DA. Empowering distributed workflow with the Data Capacitor: Maximizing Lustre performance across the wide area network. Proceedings of the 2007 Workshop on Service-Oriented Computing Performance: Aspects, Issues, and Approaches. Monterey, CA, June 25, 2007. ACM Press: New York. 2007; 53–58. DOI: 10.1145/1272457.1272465
15. Simms S, Pike G, Balog D. Wide area filesystem performance using Lustre on the TeraGrid.Proceedings of the TeraGrid 2007 Conference, June 2007.
16. TeraGrid software and services (CTSS). Chicago: Author.http://www.teragrid.org/userinfo/software/ ctss.php[10 March 2009].
17. Dongarra, J., Luszczek, P. Introduction to the HPCChallenge benchmark suite,ICL Technical Report, ICL-UT-05-01,2005.http://icl.cs.utk.edu/news_pub/submissions/hpcc-challenge-intro.pdf [10 March 2009]
18. GotoBLAS.http://www.tacc.utexas.edu/resources/software/#blas[11 March 2009]
19. MPICH-A Portable Implementation of MPI.http://www-unix.mcs.anl.gov/mpi/mpich1[11 March 2009] 20. National Science Foundation. Benchmarking information referenced in the NSF 05-625 “High performance computing system acquisition: Towards a petascale computing environment for science and engineering”. 2005.http://www.nsf.gov/pubs/2006/nsf0605/nsf0605.jsp[10 March 2009].
21. NAMD performance.http://www.ks.uiuc.edu/Research/namd/[10 March 2009]. 22. Parallel total energy code.http://www.nersc.gov/projects/paratec/ [10 March 2009].
23. MIMD lattice computation (MILC) collaboration.http://www.physics.indiana.edu/~sg/milc.html[10 March 2009].
24. Michalakes J, Dudhia J, Gill D, Henderson,T, Klemp J, Skamarock W, Wang W. The weather reseach and forecast model: Software architecture and performance. In11th ECMWF Workshop on the Use of High Performance Computing In Meteorology,Mozdzynski G (ed.). European Centre for Medium-Range Weather Forecasts: Reading, U.K., 2004.http://www.wrf-model.org[10 March 2009].
25. HOMME: High-order methods modeling environment. National Center for Atmospheric Research (NCAR),
http://www.homme.ucar.edu/[10 March 2009].
26. Big Red performance.http://rtinfo.uits.iu.edu/pubs_and_presentations/bigredperformance.shtml
[10 March 2009].
27. Deimos.http://www.top500.org/system/8512/[10 March 2009].
29. Kn¨upfer, Andreas and Brunst, Holger and Doleschal, Jens and Jurenz, Matthias and Lieber, Matthias and Mickler, Holger and M¨uller, Matthias S. and Nagel, Wolfgang E. The Vampir Performance Analysis Tool Set.Proceedings of the 2nd I nternational Workshop on Parallel Tools for High Performance Computing.
July 2008, HLRS, Stuttgart. DOI= 10.1007/978-3-540-68564-7.http://www.springerlink.com/content/ kv841756110lv004/fulltext.pdf[13 March 2009].
30. Ahalt SC, Kelley KL. Blue collar computing: HPC for the rest of us.ClusterWorld,2004;2(11):10–18. 31. TeraGrid user portal.https://portal.teragrid.org/gridsphere/gridsphere[10 March 2009].
32. Alameda J, Christie M, Fox GC, Futrelle J, Gannon D, Hategan M, Kandaswamy G, v. Laszewski G, Nacar MA, Pierce ME, Roberts E, Severance C, Thomas M. The open grid computing environments collaboration: Portlets and services for science gateways. Concurrency and Computation: Practice and Experience 2007;19:921-942. DOI: 10.1002/cpe.1078
33. . Pierce ME, Fox GC, Youn C-H, Mock S, Mueller K, Balsoy O. Interoperable Web services for computational portals. ACM/IEEE Supercomputing 2002 Conference (SC 2002) 2002; 39. DOI: 10.1109/SC.2002.10030
34. Perera S, Gannon D. Enabling Web service extensions for scientific workflows.HPDC2006 Workshop on Workflows in Support of Large-Scale Science (WORKS06), Paris, France. June 2006.
35. Droegemeier KK , Chandrasekar V, Clark R, Gannon D, Graves S, Joseph E, Ramamurthy M, Wilhelmson R, Brewster K, Domenico B, et al. Linked environments for atmospheric discovery (LEAD): A cyberinfrastructure for mesoscale meteorology research and education.20th Conference on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology,2004.https://portal. leadproject.org/gridsphere/gridsphere[10 March 2009].
36. Nacar M, Choi J, Pierce M, Fox GC. Building a grid portal for TeraGrid’s Big Red.Proceedings of the TeraGrid 2007 Conference,June 2007.
37. TeraGrid science gateways.http://www.teragrid.org/gateways/ [10 March 2009]. 38. nanoHUB.http://nanohub.org/[10 March 2009].
39. The chemical informatics and cyberinfrastructure collaboratory (CICC) at Indiana University. http: //www.chembiogrid.org/index.html[10 March 2009].
40. Plale B, Alameda J, Wilhelmson R, Gannon D, Hampton S, Rossi A, Droegemeier KK. User-oriented active management of scientific data with my LEAD.IEEE Internet Computing 2004;9:27–34.
41. The weather challenge.http://wxchallenge.com/[10 March 2009].
42. . Open source chemistry analysis routine. http://sourceforge.net/projects/oscar3-chem [10 March 2009].
43. SMILES Simplified molecular input line entry system. http://www.daylight.com/smiles/ [10 March 2009].
44. Dalby A, Nourse JG, Hounshell WD, Gushurst AKI, Grier DL, Leland BA, Laufer J. Description of several chemical structure file formats used by computer programs developed at Molecular Design Limited.Journal of chemical information and computer sciences,1992;32:244–255.
45. Jaguar.http://www.schrodinger.com/ProductDescription.php?mID=6&sID=9 [10 March 2009].
46. Dong X, Gilbert K, Guha R, Kim J, Pierce M, Fox GC, Wild DJ. Web service infrastructure for chemoinformatics.Journal of Chemical Information and Modeling2007;47(4):103–1307.
47. Cyberinfrastructure vision for 21stcentury discovery.http://www.nsf.gov/pubs/2007/nsf0728/index.jsp
[10 March 2009].
48. Gumbart J, Wiener MC, Tajkhorshid E. Mechanics of force propagation in TonB-dependent outer membrane transport.Biophysical Journal,July 2007;93:496-504. http://www.pubmedcentral.nih.gov/ articlerender.fcgi?artid=1896255[10 March 2009].
Table I. System hardware characteristics of Big Red.
Feature 20.48 TFLOPS 30.72 TFLOPS
Computational hardware, RAM
JS21 components Two 2.5 GHz PowerPC 970MP processors, 8 GB RAM, 73 GB SAS 10K RPM drives, 40 GFLOPS
No. of JS21 blades 512 768
No. of chassis 38 56
No. of processors 1,024 1,596
No. of cores 2,048 3,192
Total system memory 4 TB 6 TB
Disk storage
GPFS scratch space 266 TB 266 TB
Lustre file system 535 TB 535 TB
Home directory space 25 TB 25 TB
Networks
Bisection bandwidth 64 GB/sec 96 GB/sec
Worst-case ping pong latency
Myrinet 2000 5.3 ms 6.1 ms
Gigabit Ethernet 20.3 ms 23.0 ms
Table II. HPC Challenge benchmark results for Big Red, in its original 20.4 TFLOPS configuration using 2,040 of its 2,048 processor cores.
Processors (cores) Count G-HPL TFlop/s G-PTRANS GB/s G-FFTE GFlop/s G-Random Access Gup/s EP-STREAM Triad (system) GB/s EP-STREAM Triad GB/s EP-DGEMM GFlop/s Random Ring Band-width GB/s Random Ring Latency µsec HPL % of peak % 2,040 13.53 40.76 67.33 0.2497 2468 1.21 8.27 0.0212 17.74 66.3 ∗
Table III. High Performance Linpack (HPL) results for several configurations of Big Red. Processors cores Count G-HPL TFlop/s HPL % of peak % MPI Library Optimized HPL runs? 2,040 13.53 66.3 MPICH No 2,048 15.04 73.4 MPICH Yes 3,072 22.33 72.7 OpenMPI Yes
Table IV. Number of processors and scaling efficiency for several key benchmarks using NSF data sets. (Note: the number of MPI processes and the number of cores are both
twice the number of processors.)
Scaling efficiency (100% corresponds to smallest job run) Number of
cores
MILC GAMESS WRF PARATEC
Large Extra Large Medium Large Medium Large Medium Large 4 8 100.0% 100.0% 100.0% 16 97.3% 91.6% 95.8% 100.0% 32 94.0% 100.0% 78.4% 89.3% 100.0% 89.8% 100.0% 64 89.5% 97.9% 58.7% 100.0% 83.6% 93.5% 63.9% 78.6% 96 42.5% 99.3% 93.2% 128 86.2% 95.4% 32.0% 96.9% 71.1% 84.6% 45.9% 192 91.0% 58.3% 256 85.4% 48.0% 74.7% 512 63.2%
Table V. Comparison of WRF execution on the NSF medium-sized data set on Big Red and two HPC clusters at the Technische Universit¨at Dresden (Deimos, an Opteron PC
cluster, and Mars, an SGI Altix 4700 system.) Number of
cores
Big Red Deimos Mars
Time (s) Scaling efficiency Time (s) Scaling efficiency Time (s) Scaling efficiency
64 1,253 100% 949 100% 721 100%
128 737 85% 546 87% 460 78%
256 546 57% 362 66% 317 57%
Table VI. Usage of TeraGrid overall and LEAD portal. Year Number of distinct accounts running jobs on TeraGrid Number of distinct individuals executing jobs on TeraGrid via
LEAD portal Total number of jobs on HPC systems within TeraGrid Total number of workflows executed by users of LEAD 2006 1,731 11 1,046,051 194 2007 2,216 179 1,606,192 2,603 2008 2,476 181 2,227,057 2,505
