Scientific Computing Programming with Parallel Objects

(1)

Esteban Meneses, PhD

School of Computing, Costa Rica Institute of Technology

Scientific Computing Programming 

with Parallel Objects

(2)

Scientific Computing Programming with Parallel Objects

Parallel Architectures Galore

2

Personal Computing

Embedded Computing

Mobile Computing

Supercomputing

Moore’s Law

(3)

My Parallel Laptop

3

Processor

(multicore)

Accelerator

(manycore)

Intel Core i7

NVIDIA GeForce GT

750M

4 cores

384 cores

2.5 GHz

967 MHz

(4)

It’s movie time!

4

(5)

Speedup

5

Heat Transfer Problem

Sp

eedup

1

10

100 Sequential

Multicore

Manycore

1

3.7

68.78 Time (seconds)

0

10

20

30

40 Sequential Multicore Manycore

0.475

8.83

(6)

Supercomputer

6

(7)

Top500

7

Source:

http://www.top500.org

(June 2015)

(8)

Exascale

Challenges:

• Heterogeneity

• Low resilience

• Thermal variation

• Irregular computation

• Programability

8

Big Data

Big Network

Big Compute

(Internet of Things)

(Exascale)

Source: http://www.top500.org (June 2015)

Big Intelligence

(9)

Single Program Multiple Data (SPMD)

9

Sequential

Parallel

CPU

send

data decomposition + communication

receive

MPI

Poor functional decomposition

Synchronized communication

CPU

(10)

Parallel Objects

10

NAMD

Charm++

Entities and interactions

Asynchronous communication

Flexible Distribution

Non-blocking communication operations

Source: http://charm.cs.illinois.edu

(11)

Parallel Objects Model

• An application is

decomposed into wudus

(work and data units).

• Objects are reactive

entities: interface of

remote methods.

• All message-passing

operations are

non-blocking: asynchronous

method invocation.

• A message-driven

execution similar to

Active Messages.

• Objects know how to

serialize/deserialize,

also called the

pack-unpack (PUP)

framework. 

 

Goals:

❖

Latency hiding

❖

Load balancing

❖

Adaptivity 

 

11

!

"

#

$

%

&

'

(

(12)

Introspective Runtime System

• A thin layer between

the application and

the machine.

• Based on object-based

overdecomposition:

many more objects

than processing

entities.

• Components:

• Message scheduler.

• Routing tables.

• Load and communication

monitoring.

12

!

"

#

$

%

&

'

(

Node A

Node B

Node C

Node D

!

"

#

$

%

&

'

(

(13)

Migration

• The underlying system consists of a collection of

processing entities (processors, or nodes).

• Objects are distributed among the processing entities.

That assignment may change dynamically if load

imbalance arises.

• An introspective runtime  

system detects  

performance bottlenecks 

and balances load by 

moving objects around.

13

Node A Node B Node C Node D

!

"

#

$

%

&

'

(

#

(14)

Dynamic Load Balancing

• NP-complete problem.

• Runtime system collects

load information and

communication graph.

• Greedy strategies, graph

partitioning.

• Runtime system shuﬄes

objects around to avoid

overloading.

• Principle of persistence.

• Based on PUP

framework.

(15)

Charm++

• Actively developed since mid 90’s.

• Features language extensions,

network layers, load balancers,

tools, and several applications.

• Objects are called chares.

• Chare arrays are the  

main collection of objects.

15

(16)

Charm++ (cont.)

16

(17)

Charm++ (cont.)

17

(18)

Charm++ Runtime System

18

(19)

MPI vs Charm++

19

MPI

Charm++

Over-decomposition

No*

Yes

Load Balancing

No*

Yes

Fault Tolerance

No*

Yes

Non-blocking Collectives

Yes**

Yes

Dynamic Adaptivity

No

Yes

Introspection

No

Yes

Wide Adoption

Yes

No

* Some third-party libraries may implement this feature. 

** As of MPI-3 standard.

(20)

Example: Heat Transfer Problem

20

(21)

Example: Heat Transfer Problem

21

(22)

Computational Fluid Dynamics

22

**#"Grids" #"Par*cles" #"Species"**

Required"

Memory"

GBs"

GFLOP"per"

**iteraon" #"Iteraons"**

Serial""""

**Run>*me""**

(1"GFLOP/s)"

10

6$

_6$x$10

6$

_9$

_1.69$

_29.5$

_60,000$

_20.5$days$

10

6$

_6$x$10

6$

_19$

_2.48$

_90.7$

_60,000$

_63$days$

5$x$10

6$

_50$x$10

6$

_19$

_24.0$

_544.7$

_220,000$

_3.8$years$

(23)

IPLMCFD

• IPLMCFD:

❖

Irregularly Portioned Lagrangian Monte Carlo Finite Diﬀerence.

❖

A massively parallel solver for turbulent reactive flows.

• LES via filtered density function (FDF).

(24)

Load Imbalance

• IPLMCFD uses a graph partitioning library

(METIS) to redistribute work.

• Requires to split execution between calls to

repartition cells.

(25)

IPLMCFD

• Goals:

• Load balance processors through

weighted graph partitioning.

• To minimize the edge-cut.

• Irregularly shaped

decompositions:

• Disadvantages:

• Nontrivial communication patterns

• Increased communication cost.

• Advantage (major):

• Evenly distributed load among

partitions.

25

P. H. Pisciuneri et al., SIAM J. Sci.

Comput., vol. 35, no. 4, pp.

(26)

Simulation of a Premixed Flame

(27)

Performance of IPLMCFD

27

(28)

Cost of Repartitioning

28

O(10

2

_)-O(10

3

₎

(29)

HPC Languages

29

Fortran

C/C++

Python

Chapel

UPC

HPF

CAF

(30)

Parallel Objects in Python

30

class

Patch:

particles = ...

def

send():

computes[i,j].

recv

(particles)

def

update(part_info):

...

class

Compute:

def

recv(particles):

...

patches[i].

update

(part_info)

patches[j].

update

(part_info)

Node X

Node Y

Node Z

(31)

Acknowledgments

• University of Illinois:

❖

Prof. Laxmikant V. Kalé (Computer Science)

• University of Pittsburgh:

❖

Dr. Patrick Pisciuneri (Center for Simulation and Modeling)

❖

Prof. Peyman Givi (Mechanical Engineering)

• Images extracted from Wikipedia and

www.defenceindustrydaily.com 

www.maclife.com 

www.theregister.co.uk 

www.geforce.com 

(32)

Conclusions

• Big potential of parallel objects in  

scientific computing:

❖

Simplified programming model

❖

Improved performance due to overdecomposition

❖

Dynamic load balancing

• Research opportunity:

❖

Parallel-objects abstractions in Python

32

Thank you!

 

[email protected] 

www.emeneses.org