Tone mapping

(1)

Tone mapping

Digital Visual Effects g Yung-Yu Chuang

with slides by Fredo Durand, Lin-Yu Tseng, and Alexei Efros

(2)

Tone mapping

• How should we map scene luminances (up to 1:100 000) to display luminances (only around 1:100,000) to display luminances (only around 1:100) to produce a satisfactory image?

Li li ? th h ldi ? Linear scaling?, thresholding?

10

^-6

dynamic range 10

⁶

Real world radiance

y g

10

^-6

10

⁶

Display

intensity

Pixel value 0 to 255

CRT has 300:1 dynamic range

(3)

The ultimate goal is a visual match

visual adaption

We do not need to reproduce the true radiance as long

as it gives us a visual match.

(4)

Eye is not a photometer!

• Dynamic range along the visual pathway is only

• Dynamic range along the visual pathway is only around 32:1.

The key is adaptation

• The key is adaptation

(5)

Eye is not a photometer!

Are the headlights different in two images? Physically,

they are the same, but perceptually different.

(6)

We are more sensitive to contrast

• Weber’s law

Just-noticeable Difference (JND)

% 1

b

~

 I I

b

b k d

background intensity

flash

(7)

How humans deal with dynamic range

• We're more sensitive to contrast (multiplicative)

A ratio of 1:2 is perceived as the same contrast as a – A ratio of 1:2 is perceived as the same contrast as a

ratio of 100 to 200

– Makes sense because illumination has a multiplicative effect

– Use the log domain as much as possible

• Dynamic adaptation (very local in retina)

– Pupil (not so important) – Neural

– Chemical

• Different sensitivity to spatial frequencies

(8)

Preliminaries

• For color images



 _w

d L L R





 







w d

d d

L L G

L G

R



 









 



 w

d w d

L L B B L

• Log domain is usually preferred.



 L^w

(9)

HDR Display

• Once we have HDR images (either captured or synthesized) how can we display them on

synthesized), how can we display them on normal displays?

LCD LCD 300:1

diffuser

Theoretically, 240,000:1.

DLP

800:1

Due to imperfect

optical depth optical depth,

54,000:1 measured

HDR display system, Sunnybrook Technology, SIGGRAPH2004

(10)

Sunnybrook HDR display

Slide from the 2005 Siggraph course on HDR

(11)

How it works

Slide from the 2005 Siggraph course on HDR

(12)

Brightside HDR display

37”

200000:1

Acquired cqu ed

by Dolby

(13)

Tone mapping operators

• Spatial (global/local)

F d i

• Frequency domain

• Gradient domain

• 3 papers from SIGGRAPH 2002

 Photographic Tone Reproduction for Digital Images

 Fast Bilateral Filtering for the Display of High-

 Fast Bilateral Filtering for the Display of High- Dynamic-Range Images

 Gradient Domain High Dynamic Range Compression Gradient Domain High Dynamic Range Compression

(14)

Photographic Tone Reproduction Photographic Tone Reproduction for Digital Images g g

Erik Reinhard Mike Stark Peter Shirley Jim Ferwerda

SIGGRAPH 2002

(15)

Global v.s. local

(16)

Photographic tone reproduction

• Proposed by Reinhard et. al. in SIGGRAPH 2002 M i d b di i l i

• Motivated by traditional practice, zone system by Ansel Adams and dodging and burning

• It contains both global and local operators

(17)

Zone system

(18)

The Zone system

• Formalism to talk about exposure, density

• Zone = intensity range in powers of two

• Zone = intensity range, in powers of two

• In the scene, on the negative, on the print

Source: Ansel Adams

(19)

The Zones

(20)

The Zone system

• You decide to put part of the system in a given zone

zone

• Decision: exposure, development, print

(21)

Dodging and burning

• During the print

Hid f h i d i

• Hide part of the print during exposure

– Makes it brighter

From The Master Printing Course, Rudman

(22)

Dodging and burning

From Photography by London et al.

dodging burning

(23)

Dodging and burning

• Must be done for every single print!

Straight print After dodging and burning

(24)

Global operator

 

 

 

^L ^x ^y

L

1 log( ( ))

exp 

Approximation of scene’s key (how light or dark it is).

 

  

 

y x

w

w L x y

L N

,

)) ,

( log(

exp  ^{y (} ^g ⁾

Map to 18% of display range for average-key scene

a

User-specified; high key or low key

) ( x y ) L

, ( )

,

( L x y

L y a

x

L

_w

w

m



) , ( 1

) , ) (

,

( L x y

y x y L

x L

m m

d

 

f f i

transfer function to compress high luminances

(25)

Global operator

It seldom reaches 1 since the input image does not have infinitely large luminance values

) , 1 (

)

( L x y

y x

L

^m

 

 



not have infinitely large luminance values.

) (

1 ) , 1 (

) , ( )

, (

2

y x L

y x y L

x L

y x

L

^white

m

d



 

  

 1  L

_m

( x , y )

L _hit is the smallest luminance L_white is the smallest luminance to be mapped to 1

(26)

low key (0.18) high key (0.5)

(27)

Dodging and burning (local operators)

• Area receiving a different exposure is often bounded by sharp contrast

bounded by sharp contrast

• Find largest surrounding area without any sharp t t

contrast

) (

)

( x y L x y G x y

L

^blur

( x , y )  L ( x , y )  G ( x , y )

L

_s _m



_s

blur

x y L x y

L ( )  ( )

blur s s s

s

a s L

y x L

y x y L

x

V 

 

₂ ^¹

2 ) , ( )

, ) (

,

(

_



 y) (x,

max

: V

s

s ( , y)

max smax

(28)

Dodging and burning (local operators)

) , ) (

,

( L x y

y x

L

_d



^m_bl

A darker pixel (smaller than the blurred )

, ( ) 1

, (

max

x y

y L x

L

_blur

s

d



• A darker pixel (smaller than the blurred

average of its surrounding area) is divided by a larger number and become darker (dodging)

larger number and become darker (dodging)

• A brighter pixel (larger than the blurred

f i di ) i di id d b average of its surrounding area) is divided by a smaller number and become brighter (burning)

• Both increase the contrast

(29)

Dodging and burning

(30)

Frequency domain

• First proposed by Oppenheim in 1968!

U d i lifi d i

• Under simplified assumptions,

i ill i * fl t

image = illuminance * reflectance

low-frequency

attenuate more high-frequency tt t l

attenuate more attenuate less

(31)

Oppenheim

• Taking the logarithm to form density image P f FFT h d i i

• Perform FFT on the density image

• Apply frequency-dependent attenuation filter

c kf c

f

s ( )  ( 1  ) 

P f i FFT

f kf

 ) 1

( )

(

• Perform inverse FFT

• Take exponential to form the final image

(32)

Fast Bilateral Filtering for the

Display of High-Dynamic-Range Images

Images

Frédo Durand & Julie Dorsey J y SIGGRAPH 2002

SIGGRAPH 2002

(33)

A typical photo

• Sun is overexposed

F d i d d

• Foreground is underexposed

(34)

Gamma compression

• X  X 

l h d

• Colors are washed-out

Input Gamma

(35)

Gamma compression on intensity

• Colors are OK, but details (intensity high- frequency) are blurred

frequency) are blurred

Gamma on intensity

Intensity y y

Color

(36)

Chiu et al. 1993

• Reduce contrast of low-frequencies

K hi h f i

• Keep high frequencies

Reduce low frequency

Low-freq. q q y

High-freq.

Color

(37)

The halo nightmare

• For strong edges

B h i hi h f

• Because they contain high frequency

Reduce low frequency

Low-freq. q q y

High-freq.

Color

(38)

Durand and Dorsey

• Do not blur across edges N li fil i

• Non-linear filtering

Output Large-scale g p

Detail

Color

(39)

Edge-preserving filtering

• Blur, but not across edges

Edge-preserving Gaussian blur

Input

• Anisotropic diffusion [Perona & Malik 90] p [ ]

– Blurring as heat flow – LCIS [Tumblin & Turk]

• Bilateral filtering [Tomasi & Manduci, 98]

(40)

Start with Gaussian filtering

• Here, input is a step function + noise



J f ^ I

output input

(41)

Start with Gaussian filtering

• Spatial Gaussian f



J f ^ ^I

output input

(42)

Start with Gaussian filtering

• Output is blurred

J  ^f ^ ^I

output input

(43)

Gaussian filter as weighted average

• Weight of  depends on distance to x

) , ( x 

f I (  )

 ) (x

J 



x x x



output input

(44)

The problem of edges

• Here, “pollutes” our estimate J(x) I i diff

) (  I

• It is too different

) , ( x 

f I (  )

 ) (x

J 



) (x I



x

) (

output input

(45)

Principle of Bilateral filtering

• [Tomasi and Manduchi 1998]

• Penalty g on the intensity difference

 ) (x

J  f ( x ,  ) g ( I (  )  I ( x )) I (  ) )



( 1

x

k ( )

_

) (x

x I ⁽ ⁾

) ( 

I (  )

output input

(46)

Bilateral filtering

• Spatial Gaussian f

 ) (x

J  f ( x ,  ) ^g ⁽ Î ⁽ ^ ⁾ ^ Î ⁽ ^x ⁾⁾ Î ⁽ ^ ⁾

)



( 1

x

k ( )

_

x

output input

(47)

Bilateral filtering

• Spatial Gaussian f

• Gaussian g on the intensity difference

 ) (x

J  f ( x ,  ) g ( I (  )  I ( x )) I (  ) )



( 1

x

k ( )

_

x

output input

(48)

Normalization factor

• k(x)=  ^f ⁽ ^ ⁾ ⁽ ^I ⁽ ^ ⁾ ^I ⁽ ⁾⁾

• k(x)= 



) , ( x 

f g ( I (  )  I ( x ))

 ) (x

J  I (  )

)



( 1

x

k f ( x ,  ) g ( I (  )  I ( x )) )



(

x

output input

(49)

Bilateral filtering is non-linear

• The weights are different for each output pixel

 ) (x

J  f ( x ,  ) g ( I (  )  I ( x )) I (  ) )



( 1

x

k ( )

_

x x

x

output input

(50)

Contrast reduction

Input HDR image

Contrast

too high!

(51)

Contrast reduction

Input HDR image

Intensity

Color

(52)

Contrast reduction

Input HDR image

Intensity Large scale

Fast Fast

Bilateral Filter

Color

(53)

Contrast reduction

Input HDR image

Intensity

Large scale

Detail

Fast Detail

Fast

Bilateral Filter

Color

(54)

Contrast reduction

Input HDR image

Scale in log domain Scale in log domain

Intensity

Large scale

Reduce Large scale

Detail Fast

contrast Detail

Fast

Bilateral Filter

Color

(55)

Contrast reduction

Input HDR image

Intensity

Large scale

Reduce Large scale

Detail Fast

contrast

Detail Detail

Fast

Bilateral Filter

Detail Preserve!

Color

(56)

Contrast reduction

Input HDR image

Output

Intensity

Large scale

Reduce Large scale

Detail Fast

contrast

Detail Detail

Fast

Bilateral Filter

Detail Preserve!

Color Color

(57)

Bilateral filter is slow!

• Compared to Gaussian filtering, it is much slower because the kernel is not fixed

slower because the kernel is not fixed.

• Durand and Dorsey proposed an approximate h t d

approach to speed up

• Paris and Durand proposed an even-faster

approach in ECCV 2006. We will cover this one

when talking about computational photogrphy.

(58)

Oppenheim bilateral

(59)

Gradient Domain High Dynamic Gradient Domain High Dynamic Range Compression g p

Raanan Fattal Dani Lischinski Michael Werman SIGGRAPH 2002

SIGGRAPH 2002

(60)

Log domain

• Logorithm is a crude approximation to the perceived brightness

perceived brightness

• Gradients in log domain correspond to ratios (l l t t) i th l i d i

(local contrast) in the luminance domain

(61)

The method in 1D

log

log derivativederivative

attenattennuatenuate

integrate integrate exp

exp integrateintegrate exp

exp

(62)

The method in 2D

• Given: a log-luminance image H(x,y)

• Compute an attenuation map ^  ^ ^H 

• Compute an attenuated gradient field G :

 

 ^H 

y x

H y

x

G ( , )   ( , )   

• Problem: G may not be integrable!

(63)

Solution

• Look for image I with gradient closest to G in th l t

the least squares sense.

• I minimizes the integral: ^g  ^F  ^ ^I ^, ^G  ^dxdy

2 2



 



 

 ^, ^y

 ^, 

² ²

_

²



 



 



 

 

 



 



 









_x

G

_y

y G I

x G I

I G

I

F   x    y 

G G I

I   



² ²

_Poisson

y G x

G y

I x

I

_x _y



 



 



 



2 2

Poisson

equation

(64)

Solve ^ ^I ^ ^ ^I ^ ^ ^G

^x

^ ^ ^G

^y

2 2

Solve

y x

y

x  



 



² ²

) 1 ,

( )

, ( )

, 1 (

) ,

( x y  G x  y  G x y  G x y  G

_x_x

( , y )

_x_x

( , y )

_y_y

( , y )

_y_y

( , y )

) , ( 4 )

1 ,

( )

1 ,

( )

, 1 (

) , 1

( x y I x y I x y I x y I x y

I        

 

 

 

 

 

 

 



 

 

 



 



 



 



.. 1 … 1 -4 1 … 1 ..

I

 

 



 



 



 



 



     



(65)

Solving Poisson equation

• No analytical solution M l i id h d

• Multigrid method

• Conjugate gradient method

(66)

Attenuation

• Any dramatic change in luminance results in large luminance gradient at some scale

large luminance gradient at some scale

• Edges exist in multiple scales. Thus, we have to d t t d tt t th t lti l l

detect and attenuate them at multiple scales

• Construct a Gaussian pyramid H

_i

(67)

Attenuation

)

1

, ) (

, (



 

  







_k

x y ^H

^k

^x ^y  ~ 0 . 8

Attenuation ⁽ ^, ⁾ _

 

 



_k

x y

 H

 1 0 .



gradient magnitude gradient magnitude log(Luminance)

log(Luminance) attenuation map attenuation map

(68)

Multiscale gradient attenuation

interpolate interpolate

X

X ==

interpolate interpolate

X

X ==

(69)

Final gradient attenuation map

(70)

Performance

• Measured on 1.8 GHz Pentium 4:

16

–

512 x 384: 1.1 sec

– 1024 x 768: 4.5 sec 1024 x 768: 4.5 sec

¹²

8

4

• Can be accelerated using processor-optimized

0

0 1000000 2000000 3000000

• Can be accelerated using processor optimized

libraries.

(71)

Informal comparison

Bilateral Photographic Gradient domain Bilateral

[Durand et al.]

Photographic [Reinhard et al.]

Gradient domain

[Fattal et al.]

(72)

Informal comparison

Bilateral Photographic Gradient domain Bilateral

[Durand et al.]

Photographic [Reinhard et al.]

Gradient domain

[Fattal et al.]

(73)

Informal comparison

Bilateral Photographic Gradient domain Bilateral

[Durand et al.]

Photographic [Reinhard et al.]

Gradient domain

[Fattal et al.]

(74)

Evaluation of Tone Mapping

Operators using a High Dynamic Range Displa

Range Display

Patrick Ledda Alan Chalmers Tom Troscinko Helge Seetzen

SIGGRAPH 2005

(75)

Six operators

• H: histogram adjustment B bil l fil

• B: bilateral filter

• P: photographic reproduction

• I: iCAM

• L: logarithm mapping

• A: local eye adaption

(76)

23 scenes

(77)

Experiment setting

HDR display tonemapping

result tonemapping

result

(78)

Preference matrix

• Ranking is easier than rating.

15 i f h A l f

• 15 pairs for each person to compare. A total of 345 pairs per subject.

preference matrix (tmo2->tmo4 tom2 is better than tmo4)

preference matrix (tmo2 >tmo4, tom2 is better than tmo4)

(79)

Statistical measurements

• Statistical measurements are used to evaluate:

A t h th t th ki

– Agreement: whether most agree on the ranking between two tone mapping operators.

Consistency: no cycle in ranking If all are confused – Consistency: no cycle in ranking. If all are confused

in ranking some pairs, it means they are hard to

compare. If someone is inconsistent alone, his

ranking could be droped.

(80)

Overall similarity

• Scene 8

(81)

Summary

(82)

Not settled yet!

• Some other experiment said bilateral are better than others

than others.

• For your reference, photographic reproduction f ll i b th t

performs well in both reports.

• There are parameters to tune and the space

could be huge.

(83)

References

• Raanan Fattal, Dani Lischinski, Michael Werman, Gradient Domain High Dynamic Range Compression, SIGGRAPH 2002.

• Fredo Durand, Julie Dorsey, Fast Bilateral Filtering for the Display of High Dynamic Range Images SIGGRAPH the Display of High Dynamic Range Images, SIGGRAPH 2002.