Modern methods of analysis of complex systems

Modern methods of analysis of complex systems

Lecture, Practical training 1

HybriLIT: heterogeneous computation platform

Dubna University

14 March, 2020

https://indico-hybrilit.jinr.ru/event/192/

O.I. Streltsova

*

, D.V. Podgainy, M.I. Zuev, М.А.Matveev

*

,

M.V.Bashahin

*

, Yu. Butenko

Heterogeneous Computation Team HybriLIT

Laboratory of Information Technologies, JINR, Dubna, Russia

2

Rank Site System Cores Rmax (TFlop/s) Rpeak (TFlop/s) Power (kW) 1 DOE/SC/Oak Ridge National Laboratory United States

Summit – IBM Power System AC922, IBM

POWER9 22C 3.07GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband

IBM

2,414,592 148,600.0 200,794.9 10,096

2 DOE/NNSA/LLNL United States

Sierra – IBM Power System S922LC, IBM

POWER9 22C 3.1GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband

IBM / NVIDIA / Mellanox

1,572,480 94,640.0 125,712.0 7,438 3 National Supercomputing Center in Wuxi China

Sunway TaihuLight – Sunway MPP, Sunway

SW26010 260C 1.45GHz, Sunway NRCPC 10,649,600 93,014.6 125,435.9 15,371 . . . 93 Moscow State University - Research Computing Center Russia

Lomonosov 2 – T-Platform A-Class Cluster,

Xeon E5-2697v3 14C 2.6GHz,Intel Xeon Gold 6126, Infiniband FDR, Nvidia K40m/P-100 T-Platforms

64,384 2,478.0 4,946.8

TOP500 List – June 2019 (53

rd

_edition)

TOP500 List – June 2019 (53

rd

_{edition). Accelerators statistic}

Accelerator/Co-Processor

Count Rmax (GFlops) Rpeak (GFlops)

NVIDIA Tesla V100 Family

109

509,125,000

794,381,281

NVIDIA Tesla K20x, K40, K80 13

45,462,660

76,048,249

Intel Xeon Phi Family

6

8,639,450

12,573,411

Heterogeneous platform HybriLIT

RSC Tornado nodes based on Intel® _{Xeon Phi}™_:

• Intel® _{Xeon Phi}™ _{7190 processors (72 cores)}

• Intel® _{Server Board S7200AP}

• Intel® _{SSD DC S3520 (SATA, M.2)}

• 96GB DDR4 2400 GHz RAM

• Intel® _{Omni-Path 100 Gb/s adapter}

RSC Tornado nodes based on Intel® _Xeon® _Scalable:

• Intel® _Xeon® _{Gold 6154 processors (18 cores)}

• Intel® _{Server Board S2600BP}

• Intel® _{SSD DC S3520 (SATA, M.2), 2 x Intel}®

SSD DC P4511 (NVMe, M.2) 1TB • 192GB DDR4 2666 GHz

• Intel® _{Omni-Path 100 Gb/s adapter}

21

40

Hyper-converged system allows to use all Storage nodes as computing ones in parallel with store/retrieve data. This will add 230 TFlops to “Govorun” system, almost doubling the CPU part performance.

Modules of fast scalable parallel file system (Lustre,

EOS etc.)

5

Education and testing polygon "HybriLIT"

2880

CUDA GPU cores

Memory

12 GB

GDDR5

Peak performance

4.29

TFlops single-precision

1.43

TFlops double-precision

NVIDIA Tesla K40 “Atlas”

2x

Kepler GK210

4992

CUDA GPU cores

Memory

24 GB

(

12

GB

per GPU)

Peak performance

(with GPU Boost)

8.74

TFlops single-precision

2.91

TFlops double-precision

NVIDIA Tesla K80

HybriLIT: GPUs

5120

CUDA cores

Memory

16 GB

Peak performance

15.7

TFLOPS single-precision

7.8

TFLOPS double-precision

NVIDIA Tesla V100

8

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

The GPU-component consists of 5 NVIDIA DGX-1 servers. Each server has 8 GPU NVIDIA Tesla V100 based on the latest architecture NVIDIA Volta. Moreover, one server NVIDIA DGX-1 has 40960 cores CUDA, which are equivalent to 800 high-performance central processors. A whole number of novel technologies are used in DGX-1, including the NVLink 2.0 wire with the bandwidth up to 300 Gb/s.

The GPU-component gives a users of the supercomputer a possibility to allow as massively parallel computation for general-purpose tasks using such technologies as CUDA and OpenCL, as well as use applications already adapted for this architecture. Also, GPU-component allow to use machine learning and deep learning algorithms for solving applied problems by neural network approach.

The GPU-component of the supercomputer “Govorun”

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

The unified software and information environment of the HybriLIT platform allows users to use the education and testing polygon is aimed at exploring the possibilities of novel computing architectures, IT-solutions, to develop and debug their applications, furthermore, carry out calculations on the supercomputer, which allows them to effectively use the supercomputer resources.

Software and information environment of the HybriLIT

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Software and information environment of the HybriLIT

11

The unified software and information environment including the unified system level (the operation system, the job scheduler, file systems and software) as well as a set of services allowing users to quickly get the answers to their questions, jointly develop parallel applications, receive information about conferences, seminars and meetings dedicated to parallel programming technologies.

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

To provide all the possibilities both for developing mathematical models and algorithms and carrying out resource-intensive calculations including graphics accelerators, which significantly reduce the calculation time, an ecosystem for tasks of machine learning and deep learning (ML/DL) and data analysis has been created and is actively developing for HybriLIT platform users.

The created ecosystem has two components:

• the computing component is aimed at carrying out resource-intensive, massive parallel tasks of neural network training using NVIDIA graphics accelerators;

• the component for the development of models and algorithms on the JupyterHub basis, i.e. a multi-user platform for working with Jupyter Notebook (known as IPython with the

possibility to work in a web browser), including several libraries and frameworks.

Implementation of the neural network approach, methods

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

_{The HybriLIT platform is used as a polygon for training students, post-graduate students and young scientists}

in the field of HPC. Training is conducted both by young staff members of the Institute and its Member States and by students of Dubna University. The main tool in the educational process is an education and testing polygon which is the basic platform for basic courses on “Computer System Architecture”, “High Performance Computing Technologies” and “Mathematical Models in Physics”; the number of students is about 200 people per academic year.

Educational program on the HybriLIT platform

Ecosystem for HPC and ML/DL tasks

Servers with NVIDIA Volta & Intel

Xeon Gold

https://jhub2.jinr.ru

Computation component

Dell Volta specs:

GPU: 4x Nvidia Volta V100-SXM2 NVLink 32Gb HBM2

CPU: 2x Intel(R) Xeon(R)

Gold 6148 CPU @ 2.40GHz 20 Cores/40 Threads RAM: 512 GB DDR4 2666MHz SSD: 2*240 GB

HPClab component

VM with JupyterHub

https://jhub.jinr.ru

VM: CPU: 24 Cores RAM: 32 GB

Development component

VM with

JupyterHub and ssh

https://jlabhpc.jinr.ru/

VM: CPU: 24 Cores RAM: 64 GB

Education and testing polygon "HybriLIT"

It is not easy to follow modern trends.

Modification of the existing codes or developments of new ones ?

In the last decade novel computational

facilities and technologies has become available:

MPI-OpenMP-CUDA-OpenCL...

How to control hybrid hardware:

MPI – OpenMP – CUDA - OpenCL ...

#node 1 _{#node 2}

Parallel technologies: levels of parallelism

Классификация архитектур вычислительных систем

Классификация по способу

организации памяти:

Distributed memory

• each processor has its own memory address

space

• examples: clusters, Blue Gene/L

CPU0 CPU1 CPU2

Mem0 Mem1 Mem2

Shared memory

• single address space for all processors

• examples: IBM p-series, multi-core PC

CPU0 CPU1 CPU2

Memory

Uniform Memory Access

Memory CPU CPU Memory CPU CPU Memory CPU CPU Memory CPU CPU

Non-Uniform Memory Access (ccNUMA) Shared-Memory Parallel Computers

Вычислительные системы с общей памятью

Вычислительные системы

с распределённой памятью

Bus Interconnect

Communication environment

Cache Cache Cache

18

Классификация архитектур вычислительных систем

Вычислительные системы

с общей памятью

Вычислительные системы

с распределённой памятью

CPU0 CPU1 CPU2

Mem0 Mem1 Mem2

Communication environment

CPU0 CPU1 CPU2

Memory

Cache Cache

Cache

Классификация архитектур вычислительных систем

Вычислительные системы

с общей памятью

Вычислительные системы

с распределённой памятью

CPU0 CPU1 CPU2

Mem0 Mem1 Mem2

Communication environment

CPU0 CPU1 CPU2

Memory Cache Cache Cache

Технологии параллельного программирования

Open

Specifications for Multi-processing (

OpenMP

)

MPI is a library standard for

programming distributed memory

system

(

MPI

)

Гибридные технологии: MPI+OpenMP

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

22

For Windows users:

In order to connect from the Windows OS, it is necessary to use a special program – SSH-client, e.g. PuTTY. 1. To install PuTTY, it is necessary to copy putty.exe file to your computer and launch it.

2. Enter hydra.jinr.ru in the field Host Name (or IP address)

Enter the preferable name for the current connection in the field Saved Sessions

3. To support remote graphical interface, choose Connection > SSH > X11 and mark it in the field Enable X11 forwarding 4. Check if the field Local ports accept connection from other hosts is marked in Connection > SSH > Tunnels

5. After the chosen settings go back to the Sessions and click Save in order to save the current settings. 6. Click Open to connect HybriLIT and enter login and password.

Remote access to the cluster

Remote access to the cluster for Linux users:

Launch the terminal, type in the command line:

$ ssh [email protected]

where username is your login

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

23

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

24

Module commands Description

module avail list available modules module add <name of the module> load chosen module

module list lists currently loaded modules

module rm <name of the module> unloads module foo and removes all changes

that it made in the environment

module clear unloads all modules

SLURM commands Description

sinfo reports the state of partitions and nodes

managed by SLURM

squeue reports the state of jobs

sbatch <name of the script> submit a job script for later execution (the script

typically contains one or more srun commands to launch parallel tasks)

scancel cancel a pending or running job

The list of main SLURM commands:

The list of main commands in the work with the modules:

5 basic steps to carry out computations on the platform

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

25

OpenMP

(

Open

specifications for

M

ulti-

P

rocessing)

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP (Open specifications for Multi-Processing)

OpenMP (Open specifications for Multi-Processing) is an API that supports

multi-platform shared memory multiprocessing programming in

Fortran, C, C++.

• Compiler

directives

• Environment

variables

• Library

routines

OpenMP is managed by consortium

OpenMP Architecture Review Board (or OpenMP ARB) from 1997

OpenMP website: http://openmp.org/

OpenMP for Fortran 1.0, in 1997 С/C++ in 1998 OpenMP for Fortran 2.0, in 2000 С/C++ in 2002 Version 3.0 was released in May 2008. Version 4.0 July 2013

26

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Master-thread

thread-0

thread-1

thread-2

#pragma omp parallel

Master-thread

OpenMP (Open specifications for Multi-Processing)

OpenMP (Open specifications for Multi-Processing) is an API that supports

multi-platform shared memory multiprocessing programming in

Fortran, C, C++.

• Compiler

directives

• Environment

variables

• Library

routines

export

OMP_NUM_THREADS=

3

http://openmp.org/wp/

27

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Технология

OpenMP

OpenMP (Open specifications for Multi-Processing) – Одно из наиболее популярных средств программирования компьютеров с общей памятью (в том числе многоядерных ПК). • OpenMP базируется на традиционных языках программирования и использовании специальных псевдокомментариев. Все основные конструкции для этих языков похожи. • _{Реализован привлекательный, с точки зрения пользователя, подход к} распараллеливанию: проблема возлагается на компилятор, а пользователь только выдает компилятору ценные указания: «вот это можно исполнить параллельно, а вот это – нельзя!» Ценные указания оформляются как псевдокомментарии в тексте программы. • _{За основу берется последовательная программа, а для создания ее параллельной} версии пользователю предоставляется набор директив, процедур и переменных окружения. • OpenMP работает в системах с общей памятью и основан на понятии «легковесного процесса» или нити (thread).

28

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP compilers

(Full list at http://openmp.org/wp/openmp-compilers/)

Compiler Flag Information

GNU gcc

_-fopenmp

gcc 4.2 – OpenMP 2.5 gcc 4.4 – OpenMP 3.0 gcc 4.7 – OpenMP 3.1 gcc 4.9 – OpenMP 4.0 Intel C/C++ and

Fortran

-qopenmp

on Linux or Mac OSX -Qopenmp on Windows

• OpenMP 3.1 API Specification

• Support for most of the new features in the OpenMP* 4.0 API Specification

Portland Group Compilers and Tools

-mp

Full support for

OpenMP 3.1

gcc -fopenmp start_openmp.c -o test1

pgcc -mp start_openmp.c -o test1 icc -qopenmp start_openmp.c -o test1

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP Programming Model

OpenMP is an explicit (not automatic) programming model, offering the programmer full control over parallelization.

Master-thread

thread-0

thread-1

thread-2

#pragma omp parallel

Master-thread

export OMP_NUM_THREADS=3

Compiler

directives Library routines Environment variables OpenMP (_{Multi-Processing) is an API that} Open specifications for

supports multi-platform shared memory multiprocessing programming in Fortran, C, C++.

OpenMP parallel programming technology

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

31

1. #include <stdio.h> 2. #include<math.h> 3. #include <omp.h> 4. #define Nvec 8000000 5. double f(int j); 6. double g(int j);

7. int main (int argc, char *argv[]) { 8. double* A; double* B; double* C;

9. A = (double*) malloc(Nvec*sizeof(double)); 10. B = (double*) malloc(Nvec*sizeof(double)); 11. C = (double*) malloc(Nvec*sizeof(double)); 12. // Step 1: initialization of A and B

13. #pragma omp parallel

14. {

15. #pragma omp for

16. for (i = 0; i < N; i++) {

17. A[i] = f(i);

18. B[i] = g(i);

19. }

20. //Step 2: C= A + B

21. #pragma omp for

22. for (i = 0; i < N; i++) {

23. C[i] = A[i]+B[i]

24. } 25. }

26. free(A); free(B); free(C); 27. return 0; 28. } Compiler directive Compiler directive Compiler directive Master thread (Thread 0) Thread 0 Thread 1 Master thread (Thread 0) Thread 2 task1-openmp.c

Thread 0 Thread 1 Thread 2

A:

𝑎

_𝒊

= 𝑓 𝑖 = sin 81 ∙ 𝑖 + 6 − 10 cos 15 ∙ 𝑖 , 𝑖 = 0 … 𝑁;

B:

𝑏

_𝒊

= 𝑔 𝑖 = sin 12 ∙ 𝑖 + 3 − 10 cos 13 ∙ 𝑖 , 𝑖 = 0 … 𝑁;

Step 1.

C= A+B:

𝑐

_𝒊

= 𝑎

_𝒊

+

𝑏

_𝒊

, 𝑖 = 0 … 𝑁;

Task: find the sum of two arrays C = A + B, the elements of which are determined by some functions.

OpenMP parallel programming technology: example

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Compilation and launch of OpenMP-applications

1. #!/bin/sh

2. # Example with 5 cores for OpenMP

3. #SBATCH

-p tut

4. # Number of cores per task

5. #SBATCH

-c 5

6. #Set

OMP_NUM_THREADS

to the same# value as –c

7. if [ -n "$SLURM_CPUS_PER_TASK" ]; then

omp_threads=$SLURM_CPUS_PER_TASK

8. else omp_threads=1

9. fi

10. export

OMP_NUM_THREADS

=$omp_threads

11. export OMP_DISPLAY_ENV=TRUE

12. ./test1

13. # End of submit file

$ module add intel/v2019.0.018

$ icc -qopenmp solver_Openmp.c -mkl -o test1

$ sbatch script_omp.sh

script_omp.sh

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP:

Описание параллельных областей (1)

Для выполнения кода в фигурными скобках, порождается

OMP_NUM_THREADS

- нитей.



Процесс, выполнивший директиву

parallel

(нить-мастер), всегда получает

номер 0.



OMP_NUM_THREADS

– переменная окружения, определяющая количество

потоков (нитей) в параллельной программе.



Все нити исполняют код параллельной секции, после чего автоматически

происходит неявная синхронизация всех нитей.



Как только все нити доходят до этой точки, нить-мастер продолжает

выполнение последующей части программы, а остальные нити уничтожаются.

Выделение параллельного блока программы:

#pragma omp parallel [опции] {

структурный блок программы }

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP: Construct parallel

#include <omp.h>

main () {

// Serial code ………

//Fork a team of threads:

#pragma omp parallel {

...

structured block ...

}

//Resume serial code ……..

}

PROGRAM Start !Serial code !...

! //Fork a team of threads:

!$OMP PARALLEL

...

structured block ...

!$OMP END PARALLEL

! Resume serial code

END

Fortran , General Code Structure: C / C++ , General Code Structure :

• _{The parallelism has to be expressed explicitly.}

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP: Construct parallel

#pragma omp parallel {

structured block

}

Meaning:

• The entire code block following the

parallel-directive is executed by all threads concurrently

• _{This includes:}

- creation of team of ”worker” threads - thread executes a copy of the code

within the structured block

- barrier synchronization (implicit barrier) - termination of worker threads.

#pragma omp parallel [clause ...] newline if (scalar_expression) private (list)

shared (list)

default (shared | none) firstprivate (list)

reduction (operator: list) copyin (list) num_threads (integer-expression) { structured block }

35

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP: общие и локальные переменные

В параллельной области все переменные программы разделяются

на два класса: общие (

shared

) и локальные (

private

).

• _{Общая переменная всегда существует лишь в одном экземпляре для всей программы и} доступна всем нитям под одним и тем же именем. • Объявление локальной переменной вызывает порождение своего экземпляра данной переменной для каждой нити. Изменение нитью значения своей локальной переменной, естественно, никак не влияет на изменение значения этой же локальной переменной в других нитях. • _{Можно указать, что по умолчанию (default) тип будет private} • Firstprivate, присутствуют в последовательной части кода, предшествующей параллельной секции. В параллельной секции это значение присваивается локальным переменным с тем же именем в каждой нити. • Lastprivate: после завершения блока переменная сохраняет значение, полученное в завершившемся самым последним параллельном потоке.

36

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Private variables and Shared variables

………

int a; // shared automatic

int j;

int k=3;

#pragma omp parallel private (j,k) {

int b; //private automatic b=j ; //b is not defined

foo (j,b,k); }

Shared: the data within a parallel region is shared, which means visible and accessible by all threads simultaneously.

Private: the data within a parallel region is private to each thread, which means each thread will have a local copy and use it as a temporary variable.

What if we need to initialize a private variable?

firstprivate: private variables with initial values copied from the master thread’s copy

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Specification of number of threads

Setting OpenMP environment

variables is done the same way you set any other environment variables, and depends upon which shell you use.

Via runtime functions:

sh/tcsh setenv OMP_NUM_THREADS 4

sh/bash export OMP_NUM_THREADS=4

omp_set_num_threads(4);

Other useful function to get information about threads: Runtime function omp_get_num_threads()

• Returns number of threads in parallel region

• Returns 1 if called outside parallel region

Runtime function omp_get_thread_num()

•Returns id of thread in team (Value between [0,Nthreads-1] )

•_{Master thread always has id 0}

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

#include <stdio.h>

#include <omp.h>

int main(int argc, char *argv[])

{

omp_set_num_threads(2);

#pragma omp parallel

num_threads(3)

{

printf("Параллельная область 1\n");

}

#pragma omp parallel

{

printf("Параллельная область 2\n");

}

•Перед 1ой параллельной

областью вызов

omp_set_num_threads(2)

выставляет кол-во нитей,

равное 2.

•НО! к ней применяется опция

num_threads(3)

, поэтому

сообщение "Параллельная

область 1" напечатают три

нити.

•К 2ой параллельной области

опция num_threads не

применяется, поэтому

действует значение,

установленное

omp_set_num_threads(2).

•Поэтому сообщение

"Параллельная область 2"

будет выведено двумя

нитями.

Пример: функция

omp_set_num_threads

();

опция

num_threads

39

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Подсчет общего количества порождённых нитей.

#include <stdio.h>

int main(int argc, char *argv[])

{

int count = 0;

#pragma omp parallel

reduction

(+:

count

)

{

count

++;

printf("Текущее значение

count: %d\n",

count

);

}

printf("Число нитей: %d\n",

count

);

}



Каждая нить инициализирует

локальную копию переменной

count = 0.



Далее, каждая нить

увеличивает значение своей

копии переменной count на 1

и выводит полученное число.



На выходе из параллельной

области происходит

суммирование значений

переменных count по всем

нитям.



Полученная величина

становится новым значением

переменной count в

последовательной области.

Пример: применение опции

reduction

40

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Если в параллельной области встретился оператор цикла, каждая нить выполнит

все итерации данного цикла.

Для распределения итераций цикла между различными нитями нужно

использовать директивы !$OMP DO (фортран) и

#pragma omp for

(C\C++),

которые относятся к идущему следом оператору цикла.

Пример:

#pragma omp parallel shared (a,b) private (j)

{

#pragma omp for

for (j=0; j<N; J++)

а[j] = a[j]+b[j]

}

По умолчанию в конце цикла реализуется барьерная синхронизация, которую

можно отменить с помощью опции nowait

OpenMP:

распараллеливание операторов цикла

41

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Work Sharing Constructs in OpenMP

The for directive specifies that the iterations of the loop immediately following it must be executed in parallel by the team. This assumes a parallel region has already been initiated, otherwise it executes in serial on a single processor.

#pragma omp for [clause ...] newline

shedule ( type [ chunk])

ordered

private (list)

firstprivate (list)

lastprivate (list)

shared (list)

reduction (operator: list)

collapse ( n ) nowait

for_loop

int i;

#pragma omp parallel

private(i) {

#pragma omp for

for (i=0; i<N; i++) { array[i] =foo(i); }

}

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Work Sharing Constructs in OpenMP

int i;

#pragma omp parallel private(i) {

#pragma omp for

for (i=0; i<N; i++) { array[i] =foo(i); }

}

Directive for:

• _{distributes loop iterations among (already running) threads} • does not start / stop any threads

• _{performs a synchronization barrier}

int i;

#pragma omp parallel for

private(i)

for (i=0; i<N; i++) { array[i] =foo(i); }

• _{Implicit barrier at the end of the for loop, can be disabled with the} nowait clause

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Reductions

The reduction clause performs a reduction on the variables that appear in its list.

reduction (operator : list)

• _{the variable has a local copy in each thread, but the values of the local copies will be} summarized (reduced) into a global shared variable

operator init() values

+ (addition) 0

- (subtraction) 0

*(multiplication) 1

&(bitwise and) ~0

|(bitwise or) 0

^(bitwise exclusive or) 0 && (conditional and) 1

|| (conditional or) 0

C and C++

reduction applies to the

following directives: • _for

• parallel • _sections

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

Example 3: Reduction

𝑎, 𝑏 = 𝑎_𝒊 ∙ 𝑏_𝒊 𝑁−1 𝒊=0 Scalar product:

Exercise #2: Create a program that computes the

𝜋

. Use OpenMP directives to make it run in parallel.

𝜋 =

4 1+𝑥2

𝑑𝑥

1 0 𝐹 𝑥_𝑖 ∙ ℎ ≈ 𝜋, 𝐹 𝑥 = _1+𝑥1 ₂ 𝑁−1 𝑖=0 ℎ = 1 𝑁, 𝑥𝑖 = (𝑖 + 0.5) ∙ ℎ

45

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

OpenMP Time Measurement

Function signature:

#include <omp.h>

double omp_get_wtime( void );

Description:

Elapsed wall clock time in seconds.

//…………

double start, end, Time; start = omp_get_wtime();

//beginning of computation ...

//end of computation

end = omp_get_wtime();

//Measuring the elapsed time (in seconds)

Time = end-start;

omp_get_wtick () – Get timer precision

Description:

Gets the timer precision, i.e., the number of seconds between two successive clock ticks.

C/C++ prototype: double omp_get_wtick(void);

Heter

ogeneou

s platf

orm “Hybr

iLI

T”

47

Quake 1

(1996)

GPGPU

– General-purpose graphics

processing units

Tesla K80

(2014)

Tesla V100

(2017)

https://developer.nvidia.com/cuda-zone

NVIDIA

GeForce 8

(G80 chips)

CUDA

NVIDIA unveiled

in 2006

48

2880

CUDA GPU cores

Memory

12 GB

GDDR5

Peak performance

4.29

TFlops single-precision

1.43

TFlops double-precision

NVIDIA Tesla K40 “Atlas”

2x

Kepler GK210

4992

CUDA GPU cores

Memory

24 GB

(

12

GB

per GPU)

Peak performance

(with GPU Boost)

8.74

TFlops single-precision

2.91

TFlops double-precision

NVIDIA Tesla K80

HybriLIT: GPUs

5120

CUDA cores

Memory

16 GB

Peak performance

15.7

TFLOPS single-precision

7.8

TFLOPS double-precision

NVIDIA Tesla V100

49

CPU / GPU Architecture

https://svi.nl/HuygensGPU

Intel Xeon Gold 6154

18 cores

NVIDIA V100

5120 cores

CUDA architecture (Kepler TM GK110/210)

51

http://international.download.nvidia.com/pdf/

kepler/NVIDIA-Kepler-GK110-GK210-Architecture-Whitepaper.pdf

CUDA architecture (Kepler TM GK110/210)

Streaming Multiprocessor (SMX) Architecture:

•

₁₉₂

single-precision CUDA cores

•

64

double-precision units

•

32

special function units (SFU)

•

32

load/store units (LD/ST)

52

1. Add module:

module add cuda/v10.1-1 2. Compile a program:

CUDA architecture GV100

Volta GV100 Streaming Multiprocessor

The GV100 SM is partitioned into

four processing blocks, each with

•

16

FP32 cores;

•

8

FP64 cores;

•

16

INT32 cores;

•

two

mixed-precision

Tensor cores;

54

CUDA programming model

55

Device Memory Hierarchy

Registers are fast, off-chip local memory has high latency

Tens of kb per block, on-chip, very fast

Size up to 16 GB, high latency

Random access very expensive! Coalesced access much more efficient

Hybrid model of code

Kernel_2 <<< nBlock, nThread >>> (args);

…

Serial code

…

Serial code

Kernel_1 <<< nBlock, nThread >>> (args);

__global__ void Kernel_1 ( args ){

}

int main{ ...

Kernel_1 <<< nBlock, nThread, nShMem, nStream >>> ( parameters ); ... } Language extensions: Kernel execution directive

57

CUDA programming model. CUDA C

Definitions: device = GPU; host = CPU; kernel = function that runs on the device

Parallel parts of an application are executed on the device as kernels

• One kernel is executed at a time

• Several threads execute each kernel

kernel

<<< dim3 dG, dim3 dB, nSgMem, nStream >>> ( args )

dim3 is an integer vector type that can be used in CUDA code.

dim3 dG(512); // 512 x 1 x 1

dim3 dB(1024, 1024); // 1024 x 1024 x 1

• dim3

dG

–

number of blocks dimension (grid)

• dim3

dB – dimension of blocks

• nShMem

– the amount of shared memory to be allocated for each block

• nStream

– the stream number

__global__

__host__

CPU

GPU

__global__

__device__

CUDA. Function Type Qualifiers

Declaration specifier

Callable from

Executed on

__global__

host

device

__host__

host

host

__device__

device

device

Scheme program on CUDA C/C++ and C/C++

CUDA

C / C++

1. Memory allocation

cuda

Malloc (void** devPtr, size_t size);

void* malloc (size_t size);

2. Copy variables

cuda

Memcpy (void* dst, const void* src,

size_t count,

enum cudaMemcpyKind kind

);

void* memcpy (void* destination,

const void* source, size_t num);

copy: host > device, host < device,

host <> host, device <> device

---

3. Function call

kernel

<<<

gridDim, blockDim

>>>

(args);

function (args);

4. Copy results to host

cuda

Memcpy (void* dst, const void* src,

size_t count, host < device);

---

Threads and blocks

Thread 0 Thread 1

Thread 0 Thread 1

Thread 0 Thread 1

Thread 0 Thread 1

Block 0

Block 1

Block 2

Block 3

tid – index of threads

int tid = threadIdx.x + blockIdx.x * blockDim.x

Task 1: GPU vector sums

void sum(int N, int *a, int *b, int *c){ int tid = 0;

while(tid < N){

c[tid] = a[tid] + b[tid]; tid += 1;

} }

Serial code

𝒄

_𝒊

= 𝒂

_𝒊

+ 𝒃

_𝒊

, 𝒊 = 𝟎, . . . , 𝑵 − 𝟏

__global__ void kernel(int N, int *a,

int *b, int *c){ int tid = threadIdx.x + blockIdx.x * blockDim.x; while(tid < N){

c[tid] = a[tid] + b[tid];

tid += blockDim.x * gridDim.x; }

}

CUDA code

$ module add cuda/v10.1-1

$ nvcc prog1.cu -o test1

Compilation

Error Handling

//--- Getting the last error ---//

cudaError_t

error

=

cudaGetLastError

();

if(

error

!= cudaSuccess){

printf("CUDA error, CUFFT: %s\n", СudaGetErrorString(error));

exit(-1);

}

Run-Time Errors:

every CUDA call (except kernel launches) return an error code of type

cudaError_t

Getting the last error:

cudaError_t

cudaGetLastError (void)

that works with any CUDA call that returns an error code.

Example:

#define CUDA_CALL(x) do{ cudaError_t err= x; if(( err ) != cudaSuccess ) { \

printf ("Error \"%s\" at %s :%d \n" , cudaGetErrorString(err), \

__FILE__ , __LINE__ ) ; return -1;\

}} while (0);

Error Handling: uses macro substitution

GPU kernel calls return before completing, cannot immediately use

cudaGetLastError

to get errors:

 First use cudaDeviceSynchronize()

as a synchronization point;

 Then call cudaGetLastError.

CUDA_CALL( cudaMalloc((void**)dA, n*n*sizeof(float)) );

Timing a CUDA application using events

//--- Event Management ---//

cudaEvent_t

start, stop

;

cudaEventCreate

(&

start

);

cudaEventCreate(&

stop

);

cudaEventRecord

(start);

//--- Some operation on CPU and GPU ---//

cudaEventRecord(

stop

);

float

time0

= 0.0;

cudaEventSynchronize

(

stop

);

cudaEventElapsedTime

(&

time0, start, stop

);

printf(“GPU compute time_load data(

msec

): %.5f\n”,

time0

);

//--- Destroy events ---//

cudaEventDestroy

(

start

);

Unified Memory: from CUDA 6

CPU

GPU

Unified Memory

cudaMallocManaged()

More info in CUDA Toolkit Documentation

cudaMemcpy()

cuda

Malloc()

CPU

GPU

GPU

memory

System

memory

66

Unified Memory: from CUDA 6

__host__ cudaError_t

cudaMallocManaged

( void** devPtr, size_t size, unsigned int flags = cudaMemAttachGlobal )

More info in CUDA Toolkit Documentation

CPU

GPU

Unified Memory

devPtr

Pointer to allocated device memory

size

Requested allocation size in bytes

flags

Must be either

•

cudaMemAttachGlobal

or

•

cudaMemAttachHost

(defaults to

cudaMemAttachGlobal

)

Allocates memory that will be automatically managed by the Unified Memory system

NVIDIA cuBLAS NVIDIA cuRAND NVIDIA cuSPARSE NVIDIA NPP

Vector Signal

Image Processing GPU Accelerated

Linear Algebra NVIDIA cuFFT

C++ STL Features for CUDA IMSL Library Matrix Algebra on GPU and Multicore ArrayFire Matrix Computations Sparse Linear Algebra https://developer.nvidia.com/gpu-accelerated-libraries

Use numerical libraries for GPUs

Task 2: Element-wise vector multiplication by a scalar

float

double

CuComplex

cuDoubleComplex

𝐴

_𝑘

= alpha ∙ 𝐴

_𝑘

float

double

CuComplex

cuDoubleComplex

$ nvcc testBlas_CUDA.cu -lcublas -o exec_cuda

cuBLAS library

Compilation

cuBLAS library: cublas<t>scal()

The cuBLAS library is an implementation of BLAS (Basic Linear Algebra

Subprograms) on top of the NVIDIA CUDA runtime.

• cublasStatus_t cublas

S

scal(cublasHandle_t handle,

int

n,

const

float

*alpha,

float

*x,

int

incx)

• cublasStatus_t cublas

D

scal(cublasHandle_t handle,

int

n,

const

double

*alpha,

double

*x,

int

incx)

• cublasStatus_t cublas

C

scal(cublasHandle_t handle,

int

n,

const

cuComplex *alpha, cuComplex *x,

int

incx)

• cublasStatus_t cublas

Cs

scal(cublasHandle_t handle,

int

n,

const

float

*alpha, cuComplex *x,

int

incx)

• cublasStatus_t cublas

Z

scal(cublasHandle_t handle,

int

n,

const

cuDoubleComplex *alpha,

cuDoubleComplex *x,

int

incx)

• cublasStatus_t cublas

Zd

scal(cublasHandle_t handle,

int

n,

const

double

*alpha, cuDoubleComplex

*x,

int

incx)

Shared memory

cuBLAS: Level-1 Basil Linear Algebra Subprograms (BLAS1)

𝑥

_𝑛

, 𝑛 = 0,1, … , 𝑁 − 1

cublas<t>asum():

𝑹𝒆 𝒙

_𝒏

+ 𝑰𝒎 𝒙

_𝒏 𝑵−𝟏 𝒏=𝟎

cublas<t>nrm2():

𝒙

_𝒏

∙ 𝒙

_𝒏 𝑵−𝟏 𝒏=𝟎

𝑺𝒖𝒎 = 𝒙

_𝒏

∙ 𝒙

_𝒏 𝑵−𝟏 𝒏=𝟎

=

𝑹𝒆 𝒙

_𝒏 𝟐

+ 𝑰𝒎 𝒙

_𝒏 𝟐 𝑵−𝟏 𝒏=𝟎

₇₁

Task 3: Shared memory (Parallel realization of complex

numbers’ module sums’ computations)

block 0

block 1

block 2

N

BLOCKS ×TH_BLOCKS The number of

blocks in the grid

The number of

threads per block

Shared memory available per block in bytes

49152 bytes

int i= threadIdx.x+blockDim.x*blockIdx.x;

tid=threadIdx.x tid= 0 tid= 1 tid= 2 tid= 0 tid= 1 tid= 2 tid= 0 tid= 1 tid= 2 i =2+3*1=5

𝑺𝒖𝒎 = 𝒙

_𝒏

∙ 𝒙

_𝒏 𝑵−𝟏 𝒏=𝟎

=

𝑹𝒆 𝒙

_𝒏 𝟐

+ 𝑰𝒎 𝒙

_𝒏 𝟐 𝑵−𝟏 𝒏=𝟎

72

Task 3: Shared memory

block 0

block 1

block 2

N

BLOCKS = 64 TH_BLOCKS = 128 __shared__ double cache[TH_BLOCKS]; tid=threadIdx.x tid= 0 tid= 1 tid= 2 tid= 0 tid= 1 tid= 2 tid= 0 tid= 1 tid= 2 cache[TH_BLOCKS] cache[TH_BLOCKS] cache[TH_BLOCKS]

73

kernel_sum_shared ()

__global__ void kernel_sum_shared( int nx, cuDoubleComplex* dev_vrC, double* c ){

volatile __shared__ double cache[TH_BLOCKS];

int tid= threadIdx.x;

int i= threadIdx.x+blockDim.x*blockIdx.x;

double a= 0.0; while( i<nx ){

a+= cuCabs(dev_vrC[i]);

i+= blockDim.x * gridDim.x; } cache[tid]= a; __syncthreads(); int s= blockDim.x/2; while( s!=0 ){ if( tid<s )

cache[tid]+= cache[tid+s]; //reduction

__syncthreads(); s /= 2;

}

if( tid==0) c[blockIdx.x]= cache[0];

__syncthreads(); }

copy to shared memory and

compute

abs(dev_vrC[i])

in shared memory

The Discrete Fourier Transform (DFT)

The Discrete Fourier transform (DFT) maps a complex-valued vector

𝑿

_𝒌

= 𝒙

_𝒏

𝒆

−𝟐𝝅𝒊𝑵 𝒌𝒏 𝑵−𝟏 𝒏=𝟎

, 𝒌 = 𝟎, 𝟏, . . . , 𝑵 − 𝟏

𝑥

_𝑘

(time domain)

𝑋

_𝑘

(frequency domain representation)

Escape: no normalized output 𝐼𝐹𝐹𝑇 (𝐹𝐹𝑇 𝑨 = 𝒍𝒆𝒏 𝑨 𝑨

𝑋

_𝑘

𝑥

_𝑘

Forward transform Inverse transform

𝒙

_𝒏

=

𝟏

𝑵

𝑿

𝒌

𝒆

+𝟐𝝅𝒊_𝑵 𝒌𝒏 𝑵−𝟏 𝒌=𝟎

, 𝒏 = 𝟎, 𝟏, . . . , 𝑵 − 𝟏

75

CUFFT library

http://docs.nvidia.com/cuda/cufft/



_{Key Features}



1D

,

2D

,

3D

transforms of complex and real data types;



1D transform sizes up to

128

million elements;



Flexible data layouts by allowing arbitrary strides between individual

elements and array dimensions:



_{FFT algorithms based on Cooley-Tukey and Bluestein}



Familiar API similar to FFTW Advanced Interface



Streamed asynchronous execution



Single and double precision transforms;



Batch

execution for doing multiple transforms;



In-place and out-of-place transforms;



Flexible input & output data layouts, similar to FFTW “Advanced Interface”;



Thread-safe & callable from multiple host threads

cuFFT uses algorithms based on the well-known Cooley-Tukey and Bluestein algorithms

Algorithms highly optimized for input sizes that can be written in the form

2

𝑎

∙ 3

𝑏

∙ 5

𝑐

∙ 7

𝑑

cuFFT library allocates space for

8 ∙

𝑏𝑎𝑡𝑐ℎ

∙ 𝑛 0 ∙. . .∙ 𝑛

𝑟𝑎𝑛𝑘

− 1

cufftComplex or cufftDoubleComplex elements

where

-

batch

denotes the number of transforms that will be executed in parallel

-

rank

is the number of dimensions of the input data

77

Task 4: 1D transforms of complex data type for vector length NX

Task 5: NY×1D transforms of complex data type for vector length NX

NX NX … NX NX

NY

CUFFT - Transform types:

‣R2C: real to complex ‣ C2R: Complex to real ‣ C2C: complex to complex

‣ D2Z: double to double complex ‣ Z2D: double complex to double

‣ Z2Z: double complex to double complex

Typical scheme :

• allocate memory on the

device

• Create and configure the

transformation and

cufftPlan

• Perform a transform

• Free memory and transformation _{‣ cufftPlan1d()} ‣ cufftPlan2d() ‣ cufftPlan3d() ‣ cufftPlanMany A_input[𝑖_𝑥 + 𝑁𝑋 ∙ 𝑖_𝑦], 𝑖_𝑥= 0,1,…, 𝑁𝑋 − 1, 𝑖_𝑦= 0,1,…, 𝑁𝑌 − 1

78

CUFFT library. cufftPlanMany

cufftResult

cufftPlanMany

( cufftHandle *

plan

,

Plan

int

rank

,

1

int *

n

,

num[0]= NX

int *

inembed

,

inembed[0]=

0

int

istride

,

1

int

idist

,

NX

int *

onembed

,

onembed[0]=

0

int

ostride

,

1

int

odist

,

NX

cufftType

type

,

CUFFT_Z2Z

int

batch

);

NY

$ nvcc testBlas_CUDA.cu –lcublas -lcufft -o exec_cuda

cuFFT library

Compilation

CUDA materials



_{NVIDIA website:}

_{http://www.nvidia.com/}



_{NVIDIA CUDA Zone:}

_{https://developer.nvidia.com/cuda-zone}



_{Jason Sanders, Edward Kandrot. CUDA by Example: An}

Introduction to General-Purpose GPU Programming.



_{List of Books:}

https://developer.nvidia.com/cuda-books-archive