HOW TO VISUALIZE YOUR GPU-ACCELERATED SIMULATION RESULTS. Peter Messmer, NVIDIA

HOW TO VISUALIZE YOUR

GPU-ACCELERATED SIMULATION RESULTS

RANGE OF ANALYSIS AND VIZ TASKS

 Analysis: Focus quantitative

 Visualization: Focus qualitative

TRADITIONAL HPC WORKFLOW

TRADITIONAL WORKFLOW: CHALLENGES

I/O becomes main simulation bottleneck

Viz resources need

to scale with simulation High-end viz

neglected due to workflow complexity

OUTLINE

High-level overview. Some parts platform dependent. Check with your sysadmin.

NON-REPRESENTATIVE VIZ TOOLS SURVEY

OF 25 HPC SITES

Surveyed sites:

NON-REPRESENTATIVE VIZ TOOLS SURVEY

OF 25 HPC SITES

Surveyed sites:

VISIT

 Scalar, vector and tensor field data features

— Plots: contour, curve, mesh, pseudo-color, volume,.. — Operators: slice, iso-surface, threshold, binning,..

 Quantitative and qualitative analysis/vis

— Derived fields, dimension reduction, line-outs — Pick & query

 Scalable architecture

 Open source



Cross-platform

— Linux/Unix, OSX, Windows



Wide range of data formats

— .vtk, .netcdf, .hdf5,..



Extensible



Embeddable



Python scriptable

VISIT

VISIT’S SCALABLE ARCHITECTURE

 Client-server architecture

 Server MPI parallel

 Distributed filtering

 (multi-)GPU accelerated, parallel rendering*

PARAVIEW

 Scalar, vector and tensor field data features

— Plots: contour, curve, mesh, pseudocolor, volume,.. — Operators: slice, iso-surface, threshold, binning,..

 Quantitative and qualitative analysis/vis

— Derived fields, dimension reduction, line-outs — Pick & query

 Scalable architecture

 Developed by Kitware, open source

PARAVIEW’S SCALABLE ARCHITECTURE

 Client-server-server architecture

 Server MPI parallel

 Distributed filtering

 GPU accelerated, parallel rendering*

SOME OTHER TOOLS

 Wide range of visualization tools

 Often emerged from specialized application domain — Tecplot, EnSight: structural analysis, CFD

— IndeX: seismic data processing & visualization — IDL: image processing

 Early adopters of visual programming

FURTHER READING

 Paraview Tutorial:

http://www.paraview.org/Wiki/The_ParaView_Tutorial

 VisIt Manuals/Tutorials:

VISUALIZATION TOOLKIT

 Focus on visualization, not (only) rendering

 Provides more complex operations on data (“filtering”)

 Introduces visualization pipeline

 At the core of many high-level viz tools

VTK PIPELINE

Source

Filter

Mapper

Renderer

Raw data, shapes

Transform raw data

Map data to geometry

OpenGL

OPENGL

OPENGL: API FOR GPU ACCELERATED

RENDERING

• Primitives: points, lines, polygons

• Properties: colors, lighting, textures, ..

• View: camera position and perspective

• Shaders: Rendering to screen/framebuffer

• C-style functions, enums

See e.g. “What Every CUDA Programmer Should Know About OpenGL”

A SIMPLE OPENGL EXAMPLE

glColor3f(1.0f,0,0);

glBegin(GL_QUADS);

glVertex3f(-1.0f, -1.0f, 0.0f); // The bottom left corner

glVertex3f(-1.0f, 1.0f, 0.0f); // The top left corner

glVertex3f(1.0f, 1.0f, 0.0f); // The top right corner

glVertex3f(1.0f, -1.0f, 0.0f); // The bottom right corner

glEnd(); glFlush(); State-based API (sticky attributes) Drawing Render to screen

glColor3f(1.0f,0,0); glBegin(GL_QUADS); glVertex3f(-1.0f, -1.0f, 0.0f); glVertex3f(-1.0f, 1.0f, 0.0f); glVertex3f(1.0f, 1.0f, 0.0f); glVertex3f(1.0f, -1.0f, 0.0f); glEnd(); glFlush(); float* vert={-1.0f, -1.0f, ..}; float* d_vert; cudaMalloc(&d_vert, n); cudaMemcpy(d_vert, vert, n, cudaMemcpyHostToDevice); renderQuad<<<N/128, N>>>(d_vert); flushToScreen<<<..>>>();

?

CUDA-OPENGL INTEROP: MAPPING MEMORY

• OpenGL: Opaque data buffer object

• Virtex Buffer Object (VBO)

• User has very limited control

• CUDA: C-style memory management

• User has full control

• CUDA-OpenGL Interop:

RENDERING FROM A VBO

init

()

display()

RENDERING FROM A VBO WITH CUDA

INTEROP

Register VOB with CUDA

Map VOB to CUDA

Unmap VBO from CUDA

init

()

display()

MAIN OPENGL-CUDA INTEROP ROUTINES

 Register VBO for CUDA

cudaGraphicsGLRegisterBuffer(cuda_vbo, *vbo, flags);

 Mapping of VBO to CUDA

cudaGraphicsMapResources(1, cuda_vbo, 0);

cudaGraphicsResourceGetMappedPointer(&d_ptr, size, *cuda_vbo);

 Unmapping VBO from CUDA

CAN ALL GPUS SUPPORT OPENGL?

 GeForce : standard feature set including OpenGL 4.3/4.4

 Quadro : + certain highly accelerated features (e.g. CAD)

 Tesla: If GPU is in “All on” operation mode*

nvidia-smi –-query-gpu=gom.current nvidia-smi –q

*requires “m” or “X” class devices

Graphics capabilities disabled

OPENGL SUMMARY

+ Relatively simple for basic viz

+ Existing/fixed/simple rendering pipeline

- Low level

- Triangles, points, rather than isosurfaces, height-fields

- (currently) depends on Xserver for context creation - What about remote, parallel etc?

MORE OPENGL INFORMATION AT GTC

S4455 - Multi-GPU Rendering, 03/24, 14:30, 210A

S4825 - Tegra K1 Developer Tools for Android: Unleashing the Power of the Kepler GPU with NVIDIA's latest Developer Tools Suite, 03/24, 14:30, 210E

S4379 - OpenGL 4.4 Scene Rendering Techniques, 3/25, 13:00, 210C

S4810 - NVIDIA Path Rendering: Accelerating Vector Graphics for the Mobile Web, 3/25, 13:30, LL21C

S4610 - OpenGL: 2014 and Beyond, 3/25, 14:00, 210C

OPENGL CONTEXT

 State of an OpenGL instance

— Incl. viewable surface

— Interface to windowing system

 Context creation: platform specific

— Not part of OpenGL

— Handled by Xserver in Linux/Unix-like systems

LOCAL RENDERING

GPU, monitor attached 2D/3D X Server

X-FORWARDING: THE SIMPLEST FORM OF

“REMOTE” RENDERING

GPU, monitor attached 2D/3D X Server

On remote system:

export DISPLAY=59.151.136.110:0.0

X-FORWARDING

+ Simple! + ssh –X

+ No need to run X on remote machine

- Lots of data crosses the network

- All rendering performed on the local Xserver

- Not useful for visualization of remote CUDA Interop apps (X-server doesn’t “see” remote GPU)

SERVER-SIDE RENDERING + REMOTE VIZ

APPLICATION

GPU, monitor attached

Network

SERVER-SIDE RENDERING + SCRAPING

GPU, monitor attached

Network Sc rape r Images Client

SERVER-SIDE RENDERING + SCRAPING

+ Full GPU acceleration + No X server on client

Question: when to scrape?

— Xserver not informed about direct rendering — Intercept glxSwapBuffers()

- Not multi-user

GLX FORKING: OUT-OF-PROCESS

GPU, monitor attached Client App Network X11 Events _Cmds X11 Proxy X Server OpenGL/ GLX GLX

GLX FORKING: OUT-OF-PROCESS

+ Full GPU acceleration + No X server on client + Multi-User

- All traffic through Proxy X server

GLX FORKING WITH INTERPOSER LIBRARY

Driver GPU

3D X Server Application

libGL VirtualGL Xlib

VirtualGL client

GLX

OpenGL Images Images

X11 Events

X11 Commands VGL transport

(compressed)

Driver

GPU, monitor attached 2D X Server

GLX FORKING WITH INTERPOSER LIBRARY

Driver GPU

3D X Server Application

libGL VirtualGL Xlib

GLX OpenGL Images

Driver

GPU, monitor attached

Network X11 Events X11 Cmds Proxy X Server Images Client App

VIRTUAL GL + TURBOVNC

+ Compressed image transport + Transparent to the application + Fully GPU accelerated OpenGL

+ Client with or without Xserver

- Requires Xserver to access GPU

HOW TO SET UP VIRTUAL GL + TURBOVNC

 Requires Xserver running on server

— Root privileges for Xserver

 Requires installation of VirtualGL, TurboVNC on server

 Start VirtualGL-accelerated VNC server

vncserver :3

 TurboVNC viewer on client

CONNECTING TO REMOTE VNC SERVER VIA

A GATEWAY

 Establish tunnel on client

ssh –L 3333:node01.cluster.net:5903 login.cluster.net

 Connect client to localhost:3333

Gateway

node01.cluster.net login.cluster.net client

LAUNCHING REMOTE CUDA/OPENGL

APPLICATIONS VIA VGLRUN

vglrun :3 glxgears export DISPLAY=:3 vglrun simpleGL

REMOTE VIZ IN PARAVIEW/VISIT

 Approach 1: VirtualGL and VNC

export DISPLAY=:3 vglrun paraview

 Approach 2: Local Client

— On remote server:

pvserver

On local workstation:

paraview

Paraview Client & Server under VirtualGL

REMOTE VISUALIZATION SUMMARY

 Multiple approaches to remote rendering

— X forwarding, remote viz app, scraping, interposer process/library

 Currently requires X server to generate context

Parallel Viz

PARALLEL VISUALIZATION

PARALLEL VISUALIZATION

 Parallelism at multiple levels

— Filtering — Rendering

- Both supported by VisIt & Paraview

- Heavy lifting already done!

- Challenge: Setup in parallel environment

- Both tools provide support for most common cases

BASIC STEPS TO PARALLEL VISUALIZATION

 Launch visualization server processes

— Most likely through queuing system

 Connect client to head node

 Tunnel from workstation

DOMAIN DECOMPOSITION OF DATA

- Visualization algorithms can work on decomposed data

- May require ghost cells

- May lead to load imbalance No ghost cells required

PARALLEL COMPOSITING

- Distributed geometry - Render with depth

information

- Composition using IceT

PARALLEL VISUALIZATION

 Particularly important for large datasets

 Visualization time often determined by filtering

 Rendering important if

— Highly complex visualization — Complex visualization effects — Low-power CPU

BENEFIT OF IN-SITU VISUALIZATION

 Pipeline simulation cycle

— Visualization/analysis integral part of simulation

 Immediate feedback

 Reduce pressure on file system

BUILD VISUALIZATION PIPELINE FOR

RUNNING APPLICATION

BASIC INTERACTION/STEERING WITH

RUNNING APPLICATION

INTERACTIVE VIZ APPLICATION WITH

LIBSIM

 User implements callbacks for VisIt

— meta-data, mesh, variables and domains

 GetData: VisIt requests data for visualization

— Work directly on simulation data — Transfer data from GPU

 Command server: Interaction VisIt front-end <-> simulation

— “Steering”

http://www.visitusers.org/index.php?title=VisIt-tutorial-in-situ

The Future

THE FUTURE

VISUALIZATION DATA FLOW

CPU

GPU

VISUALIZATION WITH GPU ACCELERATED

FILTER

CPU

GPU

VISUALIZATION WITH GPU ACCELERATED

FILTER AND HARDWARE RENDERING

CPU

GPU

FULLY GPU ACCELERATED VISUALIZATION

CPU

GPU

SDAV – SCIDAC-3 INSTITUTE ON VIZ/ANALYSIS AT

SCALE (2011-2016)

 Management, Analysis, Visualization

— In-situ analysis, indexing/compression — I/O-, Viz frameworks

— Supporting application teams

 SDAV tools deployment: Paraview, VisIt, IceT, ..

SDAV – SCIDAC-3 INSTITUTE ON VIZ/ANALYSIS AT

SCALE (2011-2016)

 PISTON (LANL): Data Parallel Visualization Operators

— Isosurface, cut, threshold — Built on top of Thrust

— Support of most Paraview operators — Incorporation into VTK

 DAX (Sandia), DIY (Argonne), EVAL (ORNL)

S4553 - Productive Programming with Descriptive Data: Efficient Mesh-Based Algorithm Development in EAVL

4620 - DAX: A Massively Threaded Visualization and Analysis Toolkit for Extreme Scale

VIZ TALKS AT GTC2014

 S4571 - Applications of GPU Computing to Mission Design and Satellite Operations at NASA's Goddard Space Flight Center

 Abel Brown ( Principal Systems Engineer, A.I. Solutions

 S4632 - Exploring the Earth in 3D: Multiple GPUs for Accelerating Inverse Imaging

 S4553 - Productive Programming with Descriptive Data: Efficient Mesh-Based Algorithm Development in EAVL

 S4620 - Dax: A Massively Threaded Visualization and Analysis Toolkit for Extreme Scale

 S4410 - Visualization and Analysis of Petascale Molecular Simulations with VMD

 S4745 - Now You See It: Unmasking Nuclear and Radiological Threats Around the World

 S4203 - Gesture-Based Interactive Visualization of Large-Scale Data using GPU and Latest Web Technologies

 S4516 - Scientific Data Visualization on GPU-Enabled, Hybrid HPC Systems

 S4599 - An Adventure in Porting: Adding GPU-Acceleration to Open-Source 3D Elastic Wave Modeling

 S4778 - Interactive Processing and Visualization of Geospatial Imagery

 S4140 - Live, Interactive, In-Situ, In-GPU Visualization of Plasma Simulations Running on GPU Supercomputers

 S4400 - Petascale Molecular Ray Tracing: Accelerating VMD/Tachyon with OptiX

SUMMARY

 Different methods of visualization at different levels

— OpenGL, VTK, VisIt & Paraview

 Remote visualization concepts

— X forwarding, Viz app, scraping, GLX forking

 Parallel rendering and compositing

— Handled transparently in key tools

 In-situ visualization concepts

— Expose simulation variables to visualization tool, interactive viz

 Future directions

— GPU accelerated filtering and rendering

Thanks to Gilles Fourestey (CSCS), Nina Suvanphim (Cray), Jean Favre (CSCS), Adam DeConinck (NVIDIA), Robert Crovella (NVIDIA), Dale Southard (NVIDIA), Hank Childs (U Oregon), Jeremy Meredith (ORNL), Ian Williams (NVIDIA), Steve Parker (NVIDIA), Kitware, Paraview, VisIt and VirtualGL developers for their support!

ENJOY THE CONFERENCE AND SEE YOU AT

GTC 2015!

ABSTRACT (FOR REFERENCE ONLY)

Learn how to take advantage of GPUs to visualize results of your GPU-accelerated simulation! This session will cover a broad range of visualization and analysis

techniques allowing you to investigate your data on the fly. Starting with some basic CUDA/OpenGL interoperability, we will introduce more sophisticated data models allowing you to take advantage of widely used tools like ParaView and VisIt to visualize your GPU resident data. Questions like parallel compositing, remote

visualization and application steering will be addressed in order to allow you to take full advantage of the GPUs installed in your supercomputing system.