Cosmological simulations on High Performance Computers

Cosmological simulations

on High Performance Computers

Cosmic Web Morphology and Topology

Cosmological workshop meeting Warsaw, 12-17 July 2011

Maciej Cytowski

Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw

Topics

• Introduce POWIEW and PRACE projects

• Describe the potential collaboration models

• Show examples of joint work

POWIEW project

• HPC Infrastructure for Grand Challenges of Science and Engineering

• Consortium of 3 biggest Polish HPC centers (Warsaw, Poznań, Cracow)

• Competence centers and working groups for Grand Challenges:

– Climate modelling – Materials sciences – Cosmology – Visual Analysis – Neurobiology – …

POWIEW HPC Infrastructure

Selection criteria:

• large scale and complex scientific computations • support for long computations (stable systems)

• mission critical computations (weather forecasting) • application driven specialized architectures

Fat-node systems Massively Parallel Processing (MPP) systems Versatile Symmetric Multiprocessing systems Hybrid systems (GPUs and FPGA)

IBM Blue Gene/P system (notos)

• IBM Blue Gene/P – 1 rack

• ~14 Tflop/s

• High performance dedicated interconnect

• 4 TB RAM

• 178 TB storage

• Installation date: December 2010

Example: REBEL model on Blue Gene/P

Main objectives of joint work:

• measure the performance of GADGET-3 code on Blue Gene/P

architecture

• test different BG/P execution modes and their performance (pure MPI, MPI + pthreads)

• test scalability of computations on Blue Gene/P architecture

• prepare configuration for the 10243 „Warsaw Universe” simulation

Example: REBEL model on Blue Gene/P

• Computations does not scale in VN mode (1024+ nodes)

IBM POWER7 system (boreasz)

• 1TFlop nodes (4 processors)

• Very high memory bandwidth

Examples: POWER7 and cosmology

• Blue Waters project – NCSA (University of Illinois)

– ENZO code will be ported and optimized on Blue Waters infrastructure by the team lead by Brain O’Shea

More: http://www.ncsa.illinois.edu/News/Stories/dark_days/

• Performance measurements and optimization (3x improvement) of the Gadget code on POWER7 architecture

More examples: Dark matter and bright minds

• Collaboration with Computer Science students:

– Summer Schools – Thesis

– Part-time jobs • Tomasz Kłos

– Gadget-2 optimization on RoadRunner architecture

– Thesis: „Cosmological simulations on hybrid architectures”

– Currently: hired by European Space Agency as software developer

• Jakub Katarzyński

How to access POWIEW resources and services

• Computational Grants reviewed by the ICM’s scientific board

• Additional info and help: [email protected]

• Special assistance for cosmological type projects:

– Maciej Cytowski ([email protected]) – Wojciech A. Hellwing ([email protected])

PRACE

– Partnership for Advanced Computing in Europe

• Vision: Enable and support European global leadership in public and private research and development. High Performance

Computing resources and services for public research.

• European HPC pyramid (ESFRI) – Tier-0: 3-6 European Centers for Petaflop – Tier-0: ? European Centers for Exaflop – Tier-1: National Centers

– Tier-2: Regional/University Centers PRACE

PRACE Resources

• 1st PRACE System (JUGENE)

– BG/P by Gauss Center for Supercomputing at Juelich • 294912 CPU cores, 144 TB memory

• 1 PFlop/s peak performance • 825.5 TFlop/s Linpack

• 2nd PRACE System (2010/2011) – GENCI (CURIE) – Intel based system by Bull

• ~90,000 cores

• 1.6 PFlop/s peak performance

• 3rd PRACE System 2011 HLRS (HERMIT)

• 4th PRACE System 2012 LRZ (SUPER-MUC)

How to access PRACE resources and services

• Project Access:

– intended for individual researchers and research groups including multi-national research groups

– has a one year duration

– calls for Proposals for Project Access are issued twice yearly (May and November) – 3 Regular Calls so far

Example: „Large Scale simulations of Ly-alpha and Ly-break galaxies in the high-z universe: Probing the epoch of reionization.”

Project leader: Gustavo Yepes, Universidad Autonoma de Madrid, Spain

• Preparatory Access:

– intended for resource use required to prepare proposals for Project Access – applications for Preparatory Access are accepted at any time

Example: ENZO model on JUGENE

• Internal PRACE Call (similar to Preparatory Access) • ICM joined project team as PRACE development staff

• ENZO model was build and optimized on JUGENE supercomputer

• INITS code used for creating initial conditions for ENZO runs (part of ENZO distribution) was fully parallelized by ICM

• Large simulations: 4096^3 particles

• Follow-up: proposal for 3rd PRACE Regular Call

• Whitepaper:

„Large Scale Simulations of the Non-Thermal Universe”,

Claudio Gheller (CINECA), Graziella Ferini (CINECA), Maciej Cytowski (ICM), Franco Vazza (Jacobs University) (available soon athttp://www.prace-ri.eu)

Summary

• We invite you to collaborate with us:

– Parallelization and optimization

– New algorithm design and implementation – Workflow design and implementation

– Assistance in getting access to POWIEW and PRACE resources

Maciej Cytowski ([email protected]) Wojciech A. Hellwing ([email protected])

Links

• POWIEW project: http://www.wielkiewyzwania.pl • PRACE project: http://www.prace-ri.eu

• Blue Waters project: http://www.ncsa.illinois.edu/BlueWaters/ • ICM site: http://www.icm.edu.pl