Objectives. Visualization. Radiological Viewing Station. Image Visualization. 2D 3D: Surface plots

(1)

TM Deserno, Master Program BME Visualization 1

MEDIZINISCHE

INFORMATIK

Prof. Dr. Thomas M. Deserno, né Lehmann Department of Medical Informatics RWTH Aachen University, Aachen, Germany

Visualization

MEDIZINISCHE

INFORMATIK

Objectives

Visualization of Medical Images



Visualize all types of medical image data



Classify fundamental methods

for image data visualization



Differ surface- from volume-based approaches



Explain the importance of illumination



Appraise computational complexity of methods

for real time visualizations

MEDIZINISCHE

INFORMATIK

Radiological Viewing Station



High-end display devices

2048 x 2560 = 5 Megapixel

(full HD: 1920 x 1080 = 2 Megapixel) 13771 (4096 simultaniously) gray scales

13,5 / 12 Bit 0.165 mm pixel spacing 800:1 contrast 

25.000 € total costs

2 x 10.000 € display Graphics adapter Computer 17.11.2009

MEDIZINISCHE INFORMATIK

Image Visualization

 2D to 3D Surface Plots  3D to 2D Slice representation Windowing  3D to 3D Surface-based Volume-based Model-based Hybrid approaches  3D to 4D Animation Virtual Reality

MEDIZINISCHE

2D



3D: Surface plots



Idea

Interpret gray scale as height

MEDIZINISCHE

2D



3D: Surface plots



Idea

Interpret gray scale as height



Advantages

Local contrast enhancement Isolines interpretation

(2)

MEDIZINISCHE

INFORMATIK

Surface Plots



Example

MEDIZINISCHE

INFORMATIK

Surface Plots



ImageJ

MEDIZINISCHE

INFORMATIK

3D



2D: Slice Representation



Orthogonal

Primary & secondary Coupled windows (crosshair)

axial (primary) sagittal (secondary) coronar (second.)

Example

volumetric reconstruction sagittal axial 

CT Visualization

coronar Wikipedia.org

MEDIZINISCHE

3D



2D: Slice Representation



Others

Interpolation

required

MEDIZINISCHE

3D



2D: Slice Representation

(3)

MEDIZINISCHE

INFORMATIK

3D



2D: Windowing



Linear histogram transform

Selection of gray value range for display

Example: bone window

0 0 50 100 150 200 255 4096 2048

MEDIZINISCHE

INFORMATIK

3D



3D: Surface-based



Surface generation

Cuberille approach

Segmentation & triangulation

Marching cubes



Surface visualization

Illumination

Transparency

Shading

MEDIZINISCHE

INFORMATIK

Cuberille Approach



Idea

Based on labeled voxels (requires segmentation) Binary labels Direct visualization 

Problems

Cubic artifacts L. Herman (1979)

MEDIZINISCHE

INFORMATIK

Triangulation



Idea

Based on contours in slices (requires segmentation) Connect contours with triangles 

Problems

Point correspondence Branching Tiling (triangulation) contour in slicek contour in slicek+1 Meyers et al. 1992

MEDIZINISCHE

Marching Cube Algorithm



History

1985 patented 1987 published



Idea

Based on labeled voxels (requires threshold only) Representation via basic patterns 

2D case

Marching squares Isoline, iso = 0.5 D. Bartz

MEDIZINISCHE

Marching Cube Algorithm



Ambiguities

(4)

MEDIZINISCHE

INFORMATIK

Marching Cube Algorithm



3D case

14 base pattern 

Algorithm

Position cube Determine case Obtain triangles from LUT Move to next position D. Bartz

MEDIZINISCHE

INFORMATIK

Marching Cube Algorithm



Problems

Change of topology ambiguities (e.g. holes) High number of triangles thinning at low curvatures 

Extensions

Heuristics for ambiguities Triangle reduction

Smooting

H. Handels

MEDIZINISCHE

INFORMATIK

Post-Processing



Smoothing

B. Preim

MEDIZINISCHE

INFORMATIK

Smoothing



Visual impression vs. exact measurements

B. Preim

MEDIZINISCHE

Surface Visualization



Illumination model

Phong

Whitten & Kay 

Ray tracing

Forward Backward 

Schading of triangles

Flat Gourand Phong

MEDIZINISCHE

Phong (1973)



Ideal reflection



Model

fatt W k L N R O

)

(

cos

)

(

)

cos(

att att



n L L d a a

I

f

k

I

f

I

W

k

I











- attenuation factor - weight, often W()=ks - object-specific coefficient - direction of light source - surface normal - direction of ideal reflection - direction to observer out in





 L N O R in out www.wikepedia.org

(5)

TM Deserno, Master Program BME Visualization 25 MEDIZINISCHE INFORMATIK 

Ideal transmission

Snell’s law Transparency 

Model

Whitted & Kay (1980)

 L N O R _ k L N R O T - refractive index - coefficient

- direction of light source - surface normal - direction of ideal reflection - direction to observer - direction of transmission trans reflect local k I k I I I  rg  tg T in out t t in in t

)

sin(

)

sin(







in



t



MEDIZINISCHE

INFORMATIK

Ray Tracing



Communication

Sender: point source (L) Receiver: observer (O) 

Direction

Forward: L O E 

Direction

Backward: O L O O L L

MEDIZINISCHE

INFORMATIK

Recursion Depth



Model



Data structure

Ray tree 

Termination

Maximum depth No object is hit O L R4 N4 L2 L1 L3 L4 L1 L3 L4 L2 N1 R1 T1 T1 R1 N2 R2 termination R2 N3 T3 termination R3 T3 R3

MEDIZINISCHE

INFORMATIK

Recursion Depth



Example

depth = 1 depth = 2

depth = 3 depth = 4

I. Scholl

MEDIZINISCHE

Transparency



Representation of nested surfaces

(relative position of inner structures)

Distinct coloring of surfaces Modeling of transmission

C. Teich T. Gerster

B. Preim

MEDIZINISCHE

Shading



Aim

Depicting depth by varying levels of darkness.



Problem

Triangulation 

Methods

Flat Gouraud Phong

(6)

MEDIZINISCHE

INFORMATIK

Flat Shading



Method

Intensity-computation for each normal vector Allocation to whole surface segment 

Result

No specular highlights Surface elements

often still observable (depending on size)

H. Handels

Gouraud Shading



Method

Computation of intensities at corner points Intensities inside of surface segments by linear interpolation 

Result

Smooth intensity transitions De facto no speckles

H. Handels

MEDIZINISCHE

INFORMATIK

Phong Shading



Method

Computation of surface normal vectors Interpolation

of normal vector at corner points

Computation

of illumination for each point 

Result

Smooth surfaces

Speckle reflections

H. Handels

3D



3D: Volume-based



Direction of processing

Back-to-front image-based every layer is seen Front-to-back

object-based more efficient 

Scales of reflections

Ray tracing (only direct reflections) Ray casting (secondary reflections)

MEDIZINISCHE

Basic Equation



No shading or scatter



Numerical approximation



Notation

ds e s c r x I L tdt s



   0 ) ( 0 ) ( ) , (    λ Iλ r x L s cλ  cλ(si) (si)

))

(

1 (

*

)

(

)

,

(

1 0 / 0 j i j i s L i

s

c

r

x

I









_



 





residual visibility wave-length intensity of light direction of beam position in image length of beam current position reflected/emitted light density of particles local color at si transparency atsi

MEDIZINISCHE

Algorithm



Color and transparency

Iterative composition of semi-transparent voxels Interpolation 

Initialization



Iteration

Color Opacy 

Termination

Small c c₀ c c_N I0=c0; 0=1 c_i:= c_i-1* (1-_i) + _ic_i i:= i-1 * (1-i) + i i

(7)

TM Deserno, Master Program BME Visualization 37 MEDIZINISCHE INFORMATIK

Example



MRT

0 200 400 600 800 1000 1200 0 100 200 300 400 500 600 700 800 900 skin tumor brain liquor

MEDIZINISCHE

INFORMATIK

Projection Methods



Integral projection

Mean ray-voxels

Simulation of X-ray (CT data) 

Maximum intensity projection (MIP)

Brightest voxel

Noisy data (tumor location) 

Depth shading

First voxel above threshold Vessels (resection planning)

MEDIZINISCHE

INFORMATIK

Direct Volume Rendering



ImageJ

MEDIZINISCHE

INFORMATIK

3D



3D: Texture Mapping



Algorithm

Equidistant slices parallel to view port plane Textured polygons

Semi-transparent superimposition (back-to-front) Distance-based sorting of texture blocks in memory 

Hardware support in PC graphics boards

2D texture memory

Storage in 3 directions, selection of best fit 3D texture memory

ATI Radeon since 10/2000 NVidia, GForce 3 since 2/2001 Graphics Processing Unit (GPU)-based

MEDIZINISCHE

Texture Mapping



2D texture memory

Axis parallel



3D texture memory

View port parallel

Silicon Graphics Silicon Graphics

MEDIZINISCHE

Texture Mapping



Hierarchical slicing

Axis parallel View port parallel

(8)

MEDIZINISCHE

INFORMATIK

Examples: Texture Mapping



3D Texture mapping



Brick artifacts

slices texture result

P. Hastreiter

MEDIZINISCHE

INFORMATIK

3D



3D: Model-based



Idea

Reconstruction of anatomical structures using geometric base models



Methods

Subdivision and convolution surfaces Fast reconstruction by cylinder or frustum



Problems

Geometrical continuity on surface Structures inside the objects Anatomical details

Model-based Visualization

 Example Vessel analysis  Graph-based model Shape Thickness Branching  Processing Segmentation Thinning Branch detection Radius composition Visualization Selle, 2000 Ehricke, 1994 Oeltze, 2004

MEDIZINISCHE

INFORMATIK

Model-based Visualization



Example: Vena portae

1990 2005

Steffen Oeltze, Magdeburg

2000

MEDIZINISCHE

Model-based Visualization



Example:

Pig

liver

Steffen Oeltze Magdeburg

MEDIZINISCHE

Model-based Visualization



Example: Diffusion MRI

Microscopic diffusion characteristics of brain Movement of water molecules

are hindered by

cell membrane

fibered tissue other macro-molecules

dMRI signal Two-tensor axes Spherical harmonic signal James Malcom, Boston

(9)

MEDIZINISCHE

INFORMATIK

Model-based Visualization



Example: dMRI clustered Tractography

Left hemisphere

from outside

James Malcom, Boston

Right hemisphere

from inside

MEDIZINISCHE

INFORMATIK

Model-based Visualization

James Malcom Boston

3D



3D: Hybrid Approaches



Combination of

Surface-based Volume-based Model-based 

Examples

Visible Human Voxel Man

Pommert & Höhne et al., 2001 liver

liver y a

MEDIZINISCHE

INFORMATIK

Visible Human



1989 National Library of Medicine: project start



1994 Visible Human Male (4 mm)



1995 Visible Human Female (0.33 mm)



Data public domain

1.871 slices, 15 GB data

CT 512 x 512, 12 bit

MRI 256 x 256, 12 bit

Photo 2048 x 1216, 24 bit color

5.189 slices, 40 GB data

MEDIZINISCHE

Visible Human



Data acquisition (male)

Photo

CT

Karl Heinz Höhne, Hamburg

MEDIZINISCHE

Voxel Man (1995)



Anatomical atlas

Manual labeling 

Computation

Enhancement Segmentation Registration Visualization 

Application

Teaching Research Therapy planning

(10)

TM Deserno, Master Program BME Visualization 55 MEDIZINISCHE INFORMATIK

3D



4D: Virtual Endoscopy



Example: gastroscopy (1996)



Acquisition

Data: CT, MR Segmentation Visualization 

Application

Diagnostics

Karl Heinz Höhne, Hamburg



Avoids invasive

examinations

MEDIZINISCHE

INFORMATIK

3D



4D: Animation

W. Jainek, 2008

MEDIZINISCHE

INFORMATIK

Summary



Types of data and display

2D 3D (surface plot) 3D 2D (slices)

3D 3D (direct volume rendering) 3D 4D (animation) 4D 4D (cardiology) 

3D



3D

Model-based Texture-based Surface-based Volume-based object data

MEDIZINISCHE

INFORMATIK

Summary



Marching cube algorithm



Illumination models

Phong

Whitted & Kay



Shading

Flat

Gouraud

Phong



Example: Visible Human

MEDIZINISCHE