Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains

Multiresolution Techniques Multiresolution Techniques for Real

for Real - - Time Visualization of Time Visualization of Urban Environments and Terrains Urban Environments and Terrains

Marco Di Benedetto

PhD Thesis

MIMOS 2012 Award Finalist

Supervisors

About Me

„

Researcher at the Visual Computing Lab – ISTI – CNR, Pisa

… Directed by Roberto Scopigno

„

Received a Ph.D. in Computer Science from Università di Pisa in 2011

… Paolo Cignoni and Fabio Ganovelli as Supervisors

„

My work is mostly focused on

… Realtime Rendering

… Huge Models Processing & Visualization

… Parallel Algorithms & GPU-Assisted Computations

… Computer Graphics on Mobile Devices

„

Interested both in Academic Research and Industrial Technology Transfer

… Æ a CNR Spin-Off can be smelled in the air...

Motivations - Scenario

„

Graphics Processing Unit (GPU)

… Low-cost 3D hardware accelerators

… Many cores

… Programmable

„

Terrain & Urban Data

… Shape and appearance

… Increasing availability

… Increasing detail captured

„

Remote Exploration & The WWW

… Widespread, fast Internet connections

… New web technologies

Huge Datasets,

Hard-to-Handle Information Move complex tasks

from CPU to GPU

Planet-scale content delivery,

HW acceleration in Web Pages

Motivations - Problems

„

Real-World Datasets (more memory than we can handle)

… Huge memory footprint Æ out-of-core preprocessing & rendering

… Large data movements Æ cache levels, compression

„

Interactive Visualization Systems (smooth exploration, responsiveness)

… Rendering must be fast… Æ GPU-efficient data structures & algorithms

… …and interruptible Æ represent the original data at different levels of detail

(multiresolution) to cope with lack of computational resources, cache misses, ...

„

Urban Models (hard to represent at different levels of detail)

… Many small, simple shapes (buildings) with lot of associated image data

Æ state-of-the-art simplification algorithms fail, new data representation needed

„

Computer Graphics and the Web (lack of efficient CG tools)

… Close the gap between CG and Web developers Æ need for web-friendly data

Contribution

Compressed Batched Dynamic Adaptive Meshes

State-of-the-art, out-of-core, wavelet-based compression scheme for terrain data, real-time decompression & rendering on the GPU

BlockMaps

State-of-the-art, multiresolution data representation for urban models, GPU-based construction and real-time rendering

Data Compression

SpiderGL

Computer Graphics data structures & algorithms for the Web, multiresolution framework, performance-oriented software

GPU Representation

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

Multiresolution: Intuitive Idea (2D)

Multiresolution: Intuitive Idea (3D)

Multiresolution: Intuitive Idea (3D Perspective)

Multiresolution Rendering

„

Why Multiresolution

… Memory & Computiational restrictions

… No need to have more information than one can show in few pixels:

better performances, less aliasing

… At any time some representation of the entire dataset should be presented to the user

„

Data is represented at several levels of detail

… DAG, k-ary Tree hierarchies

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

Our Goal

- To combine the generality and adaptivity of chunked multiresolution structures with the compression rates of nested regular grid techniques

Our Goal

- To combine the generality and adaptivity of chunked multiresolution structures with the compression rates of nested regular grid techniques

Data Compression and Rendering for Terrain Models

„

Two main approaches

1) Adaptive coarse refinement from out-of-core multiresolution data structures:

… PRO: very efficient in approximating planar and spherical domains, fast GPU updates

… CON: large multiresolution structure require out-of-core data management

2) In-core rendering from compressed pyramidal structures

… PRO: exploits structure regularity to reach very high compression rates, often in-core

… CON: limited to planar domains, ignores local adaptivity

C-BDAM

„

Compressed Batched Dynamic Adaptive Meshes

„

Based on a right-triangles hierarchy

… Each “triangle” is actually a triangle-shaped batch of thousands of small triangles

„

Right-triangles are arranged in diamonds to form a hierarchy

… Leaf nodes are subregions of the input

… Internal nodes are simplified versions of their children

„

Multiresolution is achieved by selecting a cut in the DAG that meets a

tradeoff between projected on-screen error and computational cost

C-BDAM: Compression

„

Wavelet-based compression

„

Can manage several surface attributes (normals, colors, ...)

„

Compression is an out-of-core process

… Each diamond is processed independently

… Synchronization at each level completion

„

Wavelet compression produces residual matrices from high-resolution diamonds

… Resulting matrices are sparse

… Integer quantization

… Matrices are intepreted as quadtrees and compressed

C-BDAM: Rendering

„

A cut is selected based on error and available resources

„

Regular connectivity of triangle batches is shared among all diamonds

„

Finer diamonds geometry is generated from coarser ones and residual matrices

„

Fast Decompression

… Executed in parallel on the GPU

„

Smaller CPUÆGPU Transfers

… Transferred data is compressed

Real-Time Rendering with C-BDAM

„

State-of-the art in terrain rendering

… Up to 57:1 compression rates, real-time (60-300 Hz on commodity hardware)

… Combine the generality and adaptivity of batched dynamic adaptive meshes with the compression rates of nested regular grid techniques

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

What We Want

- Output-sensitive visualization framework

- Capture salient shape and appearance information

What We Want

- Output-sensitive visualization framework

- Capture salient shape and appearance information

A Multiresolution Representation for Urban Models

„

Data availability

… Cadastral maps, ad-hoc 3D scanning

… Aerial, street-level photographs

„

Hundreds to Thousands of Buildings

… Simple shapes (prism-like, few polygons)

… Hi-res façade images

„

Simplification methods

… Rely on dense meshes (not appliable)

… Buildings risk to be joined or collapsed

… Impostors fail to reproduce parallax motion

The BlockMap Data Structure

„

Key Idea: to compactly encode in a GPU-friendly way both shape and appearance of a generic urban model sub-block

„

Requirements:

… Fast rendering (real-time interaction)

… Hierarchical representation (multiresolution)

Textured Geometry BlockMap Texture GPU-Generated Image

Intuitive Idea: Build a BlockMap

Input Textured Geometry BlockMap

BlockMap Components

HeighMap

OffsetMap

Roofs Color

Roofs Normals

Walls Normals

Walls Color

Walls AO

Walls Strips

Intuitive Idea: Draw a BlockMap

1. Draw block bounding box

2. Lay down heightmap on bottom face 3. Cast rays from observer to box

Ray intersection cases:

a. Nothing hit: don’t write pixel

Observer’s Perspective

Ambient Occlusion

„

Surface attributes: we can extract them …

… … but also compute others

„

Ambient Occlusion

… How many sunlight reaches a point on surface

… Enhances shape perception

… Approximates Global Illumination

… Most perceived effect in urban contexts

„

Computed on the GPU

… Math formulation: integral of light coming from all direction in the exposed surface hemisphere

… Approximation

„ During virtual scan, keep track of how many times a surface point is visible

Dataset Processing

„

Build a Hierarchy: embed source dataset in a quadtree

… No assumption on source representation (textured polygons, implicit surfaces, …) as long as we can draw it

„

Data Acquisition: Virtual Scan

… Acquire from several viewpoints

… Leaf nodes built directly from input

… Internal nodes generated from children BlockMaps Æ multiresolution as a side effect

...

The BlockMap Pipeline

Preprocessing Preprocessing

BlockMaps DB BlockMaps

DB

Rendering Rendering Hierarchical

Subdivision Hierarchical Subdivision

BlockMaps Generation

BlockMaps Generation

BlockMaps BlockMaps DB

BlockMaps Streaming BlockMaps

Streaming

Real-Time Visualization

Real-Time

Visualization

Huge Datasets: Out-of-Core

„

Datasets size exceeds memory capacity

„

On-demand data loading

… Multithreaded network client and cache system (disk, system RAM, video RAM)

„

Cache policy

… Ensure some data exists from any node to the root

… Priority as a function of timestamp, projection error and distance from observer

„

Can handle several GB of data

BlockMaps DB BlockMaps

DB

Network ServerNetwork Server

NetNet

Network ClientNetwork Client

Disk CacheDisk Cache

Sys RAM CacheSys RAM Cache

Gfx RAM CacheGfx RAM Cache

Rendering the BlockMap QuadTree

Not Visible Below Error Above Error

Candidate Cache Miss Backup Ideal Cut

Render Cut

Sort by distance (nearest first)

(binary tree shown for clarity)

Results in Urban Models Visualization

„

State-of-the art in multiresolution urban models rendering

… Novel compact and efficient representation, handles ground, roofs & vertical walls

… Effective simplification results where other methods fail

… Runs in real-time (30 – 200 Hz) on commodity hardware

… Image-based storage, web-ready

Technology behind BlockMap Rendering

„

Each BlockMap can be represented as a standard image (e.g. RGBA PNG)

„

The whole dataset needs few numbers per tree node

… Node bounding box, number of strips in the Walls Map

… Æ A text file is enhough

„

Shaders do not use special instructions

… Executable by graphics hardware since 2006

„

Emerging 3D technologies for the WWW have enhough expressiveness

… Can we try to explore cities directly within web browsers using BlockMaps?

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

Computer Graphics for the Web Platform

„

Many attempts have been done so far

… Proprietary solutions, browser extension, ...

„

Standardization needed

… WebGL allows the use of the graphics hardware inside web pages (low-level)

„

Most CG software libraries tend to be too high-level

… Developers are forced to comply to some specific paradigm

… Software users want to reuse as much as possible of their former knowledge

… CG experts often find limitations due to high-level abstraction

Objective

- Close the gap between CG and Web developers

-

Ease the use of WebGL without introducing unnecessary abstraction - Provide the most common CG data structures & algorithms

Objective

- Close the gap between CG and Web developers

-

Ease the use of WebGL without introducing unnecessary abstraction - Provide the most common CG data structures & algorithms

SpiderGL: 3D Graphics for the WWW

„

Lightweight, open source library ( http://spidergl.org )

„

Relies on WebGL for real-time rendering

„

Goals:

… Efficiency (asymptotic bounds are not the only concern)

… Simplicity & short learning time

… Flexibility (help with common tasks without preventing low-level access)

„

Leading design considerations

… Do not impose any design choice a priori

… Avoid over abstraction

… Allow seamless cooperation with native API

SpiderGL Architecture

„

GL: rendering

„

Mesh: editable or renderable 3D model

„

Space: geometric s. & a.

„

Async: asinchronous data requests managing

„

UI: user and html interface

Geometry, Assets, Asynch Fetches, UI

„

Support for CG-related linear algebra (vectors, matrices, quaterions, ...)

„

Geometric entities (planes, spheres, rays, boxes, ...)

„

Geometric transformation chain

„

Mesh editing

„

Asynchronous assets loading from several formats

„

Dynamic priority queues for multiresolution data fetches

„

User interface interactors

Rendering Large Meshes

„

The provided mesh module

allows rendering of large models, transparently overcoming WebGL limitations

„

Automatic chunk partitioning

„

Efficient rendering paths with run-time code generation

„

More than 200 Hz for 1Mtri meshes

„

Hundreds of millions of triangles

at 60 Hz with Nexus technology

Large Datasets on the Web

„

Reduce the CPU load by executing complex tasks on the GPU

… Reduce the impact on performances of interpreted languages

„

Use asynchronous communication model typical of web applications

… And exploit built-in browser cache

„

Terrain datasets composed of millions of triangles are explorable in real-time on common laptops

„

Multiresolution for Gigapixels image browsing

Bring Real-Time CG to the Web

„

SpiderGL is being used by major web browsers developers as a test harness platform and by several open source projects

„

Efficient library for standard CG algorithms and multiresolution techniques

„

Does not introduce measurable execution overhead, short learning times

Outline

„

Multiresolution Idea

„

Data Compression and Rendering for Terrain Models

„

A Multiresolution Representation for Urban Models

„

Computer Graphics for the Web Platform

„

Conclusions

Conclusions

„

Data Compression and Rendering for Terrain Models

… C-BDAM: high compression rations, real-time rendering

… Combine the generality and adaptivity of batched dynamic adaptive meshes with the compression rates of nested regular grid techniques

„

Multiresolution for Urban Models

… BlockMap: new data representation for city blocks

… Captures salient shape and appearance information

… Enhanced environment perception with ambient occlusion

… Image-based data structure, real-time rendering

„

Computer Graphics for the Web Platform

… SpiderGL: a fast and flexible library to close the gap between CG and Web developers

… Multiresolution framework

…

Publications

„ C-BDAM - Compressed Batched Dynamic Adaptive Meshes for Terrain Rendering Enrico Gobbetti, Fabio Marton, Paolo Cignoni, Marco Di Benedetto, Fabio Ganovelli Computer Graphics Forum, Volume 25, Number 3 - 2006

„ Ray-Casted BlockMaps for Large Urban Models Streaming and Visualization

Paolo Cignoni, Marco Di Benedetto, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton, Roberto Scopigno Computer Graphics Forum, Volume 26, Number 3 - 2007

„ Interactive Remote Exploration of Massive Cityscapes

Marco Di Benedetto, Paolo Cignoni, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton, Roberto Scopigno The 10th International Symposium on Virtual Reality, Archaeology and Cultural Heritage - 2009

„ SpiderGL: A JavaScript 3D Graphics Library for Next-Generation WWW Marco Di Benedetto, Federico Ponchio, Fabio Ganovelli, Roberto Scopigno Web3D 2010, 15th Conference on 3D Web Technology - ACM - 2010

„ Visualization Methods for Molecular Studies on the Web Platform

Marco Callieri, Raluca Mihaela Andrei, Marco Di Benedetto, Monica Zoppé, Roberto Scopigno Web3D 2010, 15th Conference on 3D Web Technology - ACM - 2010

Thank You!

Questions?

Building a BlockMap: Ray-Casting Structures

„

HeightMap

… Approximate terrain and buildings shape

… Acquired by sampling depth from above

„

OffsetMap

… Detect discontinuities on the heightmap

… Parametrizes façades

… Used to index walls colors/normals columns in the WallsMap

Strips

offset

Building a BlockMap: Shading Structures

„

Rooftops Attributes

… Color [R, G, B]

… Ambient Occlusion

… Normal [Nx, Ny]

„

Walls Strips Attributes

… Color [R, G, B]

… Ambient Occlusion

… Normal [Nx, Ny, Nz]

Building a BlockMap: Attributes Acquisition

Color Target Parametrization Target

1. Draw Geometric, Material and Parametrization data to texture targets

. Superimpose the two targets and draw one point primitive per pixel

3. Scatter: redirect points to final destination & output (combine) fetched data

Stacked Walls: better W:H ratio Render

Targets

SpiderGL Performances

„

JavaScript just-in-time compilers allow complex models to be visualized in real time

… For 3D-heavy applications, the bottleneck is the graphics hardware

„

SpiderGL rendering process guarantees the minimum number of graphics API calls

… 3D drivers are optimized for that

„

Performances

… Models with tens of thousands of triangles can be rendered at more than 1K Hz (strict realtime is 30 Hz!)

… More than 100 Hz for 1 millions of triangles

… Multiresolution terrain and BlockMaps visualization at 200 Hz

Future Work

„

Investigating different compression strategies in C-BDAM

„

Enhancing visual feedback in BlockMaps with precomputed radiance

„

BlockMaps contruction completely delegated to GPU

„

Extending SpiderGL with more complex structures for scene description

„

Working on an automatic pipeline that allows to create multiresolution,

web-friendly datasets for generic 3D models to be visualized on standard

browsers