ACES and Computational Science

(1)

Computational Science

ACES an

Computational Science

Maui Meetin July 30 2001 Geoffrey Fox

IPCRES Laboratory for Community Grids

Computer Science, Informatics, Physics

(2)

Abstract of ACES an

Computational Science Presentation

• We describe HPCC and Grid trends and how they could be folded into a ACES computational environment

• A Peer to Peer Grid of Services supporting Earthquake science

• We describe what works (MPI), what sort of works

(Objects), what is known (parallel algorithms), what is active (datamining, visualization), what failed (good parallel environments), what is inevitable (petaflops), what is simple but important (XML), what is getting more complicated (applications) and the future (Web and Grids)

2

03/02/202

(3)

Trends of Importance

• Resources

of increasing performance

–

Computers, storage, sensors, networks

• Applications

of increasing sophistication

–

Size, multi-scales, multi-disciplines

• New

algorithms

and mathematical techniques

• Computer science

–

Compilers, Parallelism, Objects, Components

• Grid

and

Internet

Concepts and Technologies

–

Enabling new applications

National Projects -- New sensors PDA’s

Multiple Scales

FEM, Fast Multipole, Datamining Visualization

Object based Programming XML

Portals

Collaboration

3

03/02/202

(4)

Projected Top 500 Until Year 2009

• First, Tenth, 100th, 500th, SUM of all 500 Projected in Time

Earth Simulator from Japan

http://geofem.tokyo.rist.or.jp/

4

03/02/202

(5)

Top 500 June 2001

5

03/02/202

(6)

Top 500 by Vendor systems

June 2001

6

03/02/202

(7)

Top 500 by Vendor Total Power

June 2001

7

03/02/202

(8)

PACI 13.6 TF Linux TeraGrid

32 32 5 32 32 5

Cisco 6509 Catalyst Switch/Router 32 quad-processor McKinley

Servers

(128p @ 4GF, 8GB memory/server)

Fibre Channel Switch HPS S HPS S ESnet HSCC MREN/Abilene Starlight 10 GbE 16 quad-processor McKinley

Servers

(64p @ 4GF, 8GB memory/server)

NCSA

500 Nodes 8 TF, 4 TB Memory

240 TB disk

SDSC

256 Nodes 4.1 TF, 2 TB Memory

225 TB disk

Caltech

32 Nodes 0.5 TF 0.4 TB Memory 86 TB disk

Argonne

64 Nodes 1 TF 0.25 TB Memory 25 TB disk

IA-32 nodes 4 Juniper M160 OC-12 OC-48 OC-12 574p IA-32 Chiba City 128p Origin HR Display & VR Facilities

= 32x 1GbE

= 64x Myrinet

= 32x FibreChannel

MyrinetClos

Spine Spine MyrinetClos

Chicago & LA DTF Core Switch/Routers Cisco 65xx Catalyst Switch (256 Gb/s Crossbar)

= 8x FibreChannel

OC-12 OC-12 OC-3 vBNS Abilene MREN Juniper M40

1176p IBM SP Blue Horizon OC-48 NTON 32 24 8 32 24 8 4 4 Sun E10K 4 1500p Origin UniTree 1024p IA-32 320p IA-64 2 14 8 Juniper M40 vBNS Abilene Calren ESnet OC-12 OC-12 OC-12 OC-3 8 Sun Starcat 16 GbE

= 32x Myrinet

HPS S 256p HP X-Class 128p HP V2500 92p IA-32 24 Extreme Black Diamond

32 quad-processor McKinley Servers (128p @ 4GF, 12GB

memory/server) OC-12 ATM Calren 2 2 8 03/02/202

(9)

Caltech

Hypercube

JPL Mark II 1985 Chuck Seitz 1983

Hypercube as a cube

9

03/02/202

(10)

From the New York Times 1984

• One of today's fastest computers is the Cray 1, which can do 20 million to 80 million operations a second. But at $5 million, they are expensive and few scientists have the resources to tie one up for days or weeks to solve a problem.

• ``Poor old Cray and Cyber (another super computer) don't have much of a chance of getting any significant increase in speed,'' Fox said. ``Our ultimate machines are expected to be at least 1,000 times faster than the current fastest computers.'' (80 gigaflops predicted. Livermore just installed 12000 gflops) • But not everyone in the field is as impressed with Caltech's

Cosmic Cube as its inventors are. The machine is nothing more nor less than 64 standard, off-the-shelf microprocessors wired

together, not much different than the innards of 64 IBM personal computers working as a unit.

• The Caltech Hypercube was “just a cluster of PC’s”!

10

03/02/202

(11)

From the New York Times 1984

• ``We are using the same technology used in PCs (personal computers) and Pacmans,'' Seitz said. The technology is an 8086 microprocessor capable of doing 1/20th of a million operations a second with 1/8th of a megabyte of primary storage. Sixty-four of them together will do 3

million operations a second with 8 megabytes of storage.

• Computer scientists have known how to make such a computer for years but have thought it too pedestrian to bother with.

• ``It could have been done many years ago,'' said Jack B. Dennis, a

computer scientist at the Massachusetts Institute of Technology who is working on a more radical and ambitious approach to parallel

processing than Seitz and Fox.

• ``There's nothing particularly difficult about putting together 64 of these processors,'' he said. ``But many people don't see that sort of machine as on the path to a profitable result.'‘

• So clusters are a trivial architecture (1984) ……

• So architecture is unchanged ; unfortunately after 20 years research,

programming model is also the same (message passing)

11

03/02/202

(12)

Technology Trends and Principles

• All performance and capability measures of infrastructure continue to improve

• Gilder’s law says that network bandwidth increases 3 times faster than CPU Performance (Moore’s Law)

• The Telecosm eclipses the Microcosm ….

George Gilder

Telecosm : How

Infinite Bandwidth Will Revolutionize Our

World (September 2000, Free Press; ISBN: 0684809303, #146(3883) in Amazon Sales Jan 15 2001(July 29 2001))

12

03/02/202

(13)

Small Devices Increasing in Importance

• There is growing interest in wireless

portable displays in the

confluence of cell phone and personal digital assistant

markets

• By 2005, 60 million

internet ready cell

phones sold each year

• 65% of all Broadband Internet accesses via non desktop appliances

CM5

13

03/02/202

(14)

The HPCC Track

• The 1990 HPCC 10 year initiative was largely aimed at enabling large scale simulations for a broad range of computational science and engineering problems

• It was in many ways a success and we have methods and machines that can (begin to) tackle most 3D simulations

– ASCI simulations particularly impressive

– DoE still putting substantial resources into basic software and algorithms from adaptive meshes to PDE solver libraries

• Machines are still increasing in performance

exponentially and should achieve petaflops in next 7-10 years

• Earthquake community needs to harness these capabilities

– Japan’s Earth Simulator activity (GEOFEM) major effort

14

03/02/202

(15)

Some HPCC Difficulties

• An Intellectual failure: we never produced a better programming model than message passing

– HPCC code is hard work

– “High point” of ASCI software is “Grid FTP”

• An institutional problem: we do not have a way to produce complex sustainable software for a niche (1%) market like HPCC.

– POOMA support just disappeared one day (foundation of first proposal GEM wrote)

– One must adopt commodity standards and produce “small” sustainable modules.

– Note distributed memory becoming dominant again with bizarre clustered SMP architecture – not clear that “wise” to exploit advantages of shared memory architectures

15

03/02/202

(16)

My HPCC Advice to ACES/GEM

• KISS:

K

eep

i

t

Simple

and

Sustainable

• Use

MPI

and

openMP

if needed for performance

on shared memory nodes

• Adaptive Meshes

• Load Balancing

• PDE Solvers including

fast multipoles

• Particle dynamics

• Other areas such as datamining, visualization

and data assimilation quite advanced but still

significant research

}

Are well understoo

to get high performanc parallel simulation

Use broad communit expertise

16

03/02/202

(17)

Use of Object Technologies

• The claimed commercial success in using Object and

component technology has not been a clear success in

HPCC

– Object technologies do not naturally support either

high performance or parallelism

– C++ can be high performance but CORBA and Java

are not

– There is no agreed HPCC component architecture to produce more modern libraries (DoE has very large

CCA – Common Component Architecture – effort)

• Fortran will continue to decline in importance and

interest – the community should prefer not to use it

– It’s use will not attract the best students

17

03/02/202

(18)

Application Structure

• ACES applications are typically scale and

multi-disciplinary

– i.e. a given simulation is made of multiple components with either different time/length scales and/or multiple authors from possibly multiple fields

• I am not aware of a systematic “Computational

renormalization group” – a methodology that links different scales together

• However composition of modules is an area where technology of growing sophistication is becoming available

– Needed commercially to integrate corporate functions

– CCA controversial “small grain size”; Gateway example of clearly successful large grain size integration

Integration of data and simulatio is one example of compositio

which is “understood”

18

03/02/202

(19)

Object Size & Distributed/Parallel Simulations

• All interesting systems consist of linked entities

– Particles, grid points, people or groups thereof

• Linkage translates into message passing

– Cars on a freeway – Phone calls

– Forces between particles

• Amount of communication tends to be proportional to

surface area of entity whereas simulation time proportional to volume

• So communication/computation is surface/volume and

decreases in importance as entity size increases

• In parallel computing, communication synchronized; in distributed computing “self contained objects” (whole programs) which can be scheduled asynchronously

19

03/02/202

(20)

Complex System simulations

This lack of global time

synchronization in “complex systems” stops natural

parallelism in classic HPCC approaches

• Networks of particles and (partial differential equation) grid points interact “instantaneously” and simulations reduce to iterating calculate/communicate phases

“calculate at given time or iteration number next

positions/values” (massively parallel) and then update

– Scaling parallelism guaranteed

• Complex (phenomenological) systems are made of

agents evolving with irregular time steps – event driven simulations do not parallelize

20

03/02/202

(21)

Los Alamos Delphi Initiative

National traffic systems Epidemics

Forest Fires

Cellular and other communication networks e.g. the Internet

Electrical, Gas, Water .. Grids Business processes

Battles

• http://www.lanl.gov/delphi/index.shtml

• Aims at large complex systems simulation of global and national scope in their size and significance

• Demonstrates success of new methods (SDS – Sequential

dynamical Systems) that parallelize well and outperform previous approaches

– General applicability (e.g. to earthquakes) not clear

• Could be relevant to cellular automata like models of earthquakes This work part of “D Division” at Los Alamo

– Decision Support Applications – could be relevant to SCEC interes

in supporting planning and decision making

21

03/02/202

(22)

Some Problem Classes

• Hardest:

smallish objects with irregular time

synchronization (Delphi)

• Classic HPCC:

synchronized objects with regular

time structure (communication overhead

decreases as problem size increases)

• Internet Technology and Commercial

Application Integration:

Large objects with

modest communications and without difficult

time synchronization

– Compose as independent (pipelined) services

– Includes some approaches to multi-disciplinary simulation linkage

22

03/02/202

(23)

What is a Grid or Web Service?

• There are generic Grid system services: security, collaboration, persistent storage, universal access

• An Application Service is a capability used either by another service or by a user

– It has input and output ports – data is from sensors or other services

• Consider NASA Space Operations (CSOC) as a Grid Service – Spacecraft management (with a web front end)

– Each tracking station is a service

– Image Processing is a pipeline of filters – which can be grouped into different services

– Data storage is an important system service

– Big services built hierarchically from “basic” services

• Portals are the user (web browser) interfaces to Grid

services

23

03/02/202

(24)

Computational Science Data base Matrix Solver MPP MPP Parallel D Proxy Senso Contro Origin 200 Proxy NetSol v Linear Alg Server

Integration of Grid Services

IBM S Proxy Grid Gateway Supportin Seamles Interface Agent-base Choice o Compute Engine Multidisciplinar Control

Object Grid Programming Environment

Classic HPCC Resources

Image Processin Server Dat Minin Server 24 03/02/202

(25)

Overall Grid/Web Architecture

• General Vision? NCSA Vision

Science Portals & Workbenches

Twenty-First Century Applications

Computational Services P e r f o r m a n c e

Networking, Devices and Systems

Grid Services (resource independent)

Grid Fabric (resource dependent)

Access Services & Technology

Access

Grid ComputationalGrid

Community Portals

Next Generation Web

Education Services

Business Services

Commerce

Grid EducationGrid

v e n i e n c e 25 03/02/202

(26)

The Application Service Model

• As bandwidth of communication (between) services increases one can support smaller services

• A service “is a component” and is a replacement for a library in case where performance allows

• Services are a sustainable model of software

development – each service has documented capability with standards compliant interfaces

– XML defines interfaces at several levels

– WSDL at Grid level and XSIL or equivalent for scientific data format

• A service can be written in Perl, Python, Java Servlet,

Enterprise Javabean, CORBA (C++ or Fortran) Object …

• Communication protocol can be RMI (Java), IIOP

(CORBA) or SOAP (HTTP, XML) ……

26

03/02/202

(27)

Services support Communities

• Grid Communities (ACES, JPL, Earth Science, High

School Classes) are groups of communicating

individuals sharing resources implemented as Grid Services

• Access Grid from Argonne/NCSA is best Audio/Video

conferencing technology

• Peer to Peer networking describes a set of technologies

supporting community building with an emphasis on less structured groups than classic “users of a

supercomputer”

• Peer to peer Grids combine the technologies and support

“small worlds” – optimized networks with short links between each community member

• My presentation on collaboration will discuss in more detail

27

03/02/202

(28)

Classic Grid Architecture

Database Database

Netsolv e

Neo s

Securit y Porta

l

Compositio n

Porta l

Resources

Client

s Users and Devices

Middle Tie Brokers Service Providers

Typically separate Clients Servers Resources

28

03/02/202

(29)

Peer to Peer Network

User

Resource Service

Routing

User

Resource Service

Routing

User

Resource Service

Routing User

Resource Service

Routing

User

Resource Service

Routing

User

Resource Service

Routing

Peers

Peers are Jacks of all Trades linked to “all” peers in community_{Typically Integrated Clients Servers and Resources}

29

03/02/202

(30)

Services GMS Routing

Peer to Peer Grid

User Resource Service Routing User Resource Service Routing User Resource Service Routing User Resource Service Routing User Resource Service Routing User Resource Service Routing Dynami Message or Even Routing fro Peers o Servers 30 03/02/202

(31)

ACES HPCC and Grid Strategy I

• Decide what services are well enough understood and useful enough to be encapsulated as application services

– Parallel FEM Solvers – Visualization

– Parallel Particle Dynamics – Access to Sensor Data

• Make as small as possible – smaller is simpler and more sustainable but with higher communication needs

• Establish teams to design and build services

• Use a framework offering needed Grid System services

• Build ACES electronic community with collaboration tools, resources and ACES wide networking (cf.

TRANSPAC project from Indiana)

31

03/02/202

(32)

ACES HPCC and Grid Strategy II

• Some capabilities – such as fast multipole package – should be built as classic libraries or templates

• Other services – such as datamining or support of multi-scale simulations – need research using a toolkit

approach if one can design a general structure

• Need “hosts” for major services – access and storage of sensor data

• Need funds to build and sustain “infrastructure” and research services

• Use electronic community tools to enhance ACES Collaboration

32

03/02/202

(33)

Sensor Grid Service

Distributed Sensor Service

in

ports

out por universal sensor acces people/computers

33

03/02/202

(34)

ACES Peer to Peer Grid Community

APAN Network linkin

Access Grids 34

03/02/202

(35)

Researcher

Share fro

deskto

or PDA

35

03/02/202