Inverse Volume Rendering

(1)

Inverse Volume Rendering

Absorption + Emission, Automatic Differentiation

Sebastian Weiß

Lehrstuhl f¨ ur Computergrafik Technische Universit¨ at M¨ unchen

November 2020

1 / 13

(2)

Overview

1. Models and Assumptions

1.1. Absorption 1.2. Emission 1.3. Transfer Function

2. Automatic Differentiation

2.1. Forward Mode 2.2. Backward Mode

(1) (2)

(1)https://www.youtube.com/watch?v=HwJyg5o3HGg

(2)https://commons.wikimedia.org/wiki/File:Convolutional_Neural_Network_with_Color_Image_Filter.gif

2 / 13

(3)

Overview

1. Models and Assumptions

1.1. Absorption 1.2. Emission 1.3. Transfer Function

2. Automatic Differentiation

2.1. Forward Mode 2.2. Backward Mode

(1) (2)

(1)https://www.youtube.com/watch?v=HwJyg5o3HGg

(2)https://commons.wikimedia.org/wiki/File:Convolutional_Neural_Network_with_Color_Image_Filter.gif 2 / 13

(4)

1. Models and Assumptions

1. Scalar Field d : R ³ → R ⁺ ₀ of densities 2. Ray r (t) = x ₀ + tω , x ₀ ∈ R ³ , ω ∈ S ² 3. Received Illumination I out (x 0 , ω) ∈ R ⁺ ₀

d ∶ ℝ

³

→ ℝ

₀⁺

r(𝑡) = 𝑥

₀

+ 𝑡𝜔

𝐼

_𝑖𝑛

: ℝ

₀⁺

𝐼

_𝑜𝑢𝑡

: ℝ

₀⁺

3 / 13

(5)

1.1 Only Absorption

Assume a medium that only absorbs incoming light, no emission or scattering.

I _out (x ₀ , ω) = Z ∞

0 g (t) exp

− Z t

0 τ (v (r (t ⁰ )))dt ⁰

dt with g (s) = c(s)τ (s) +

Z

Ω

p(ω, ω ⁰ )I _out (r (t), ω ⁰ )d ω ⁰

Set g (∞) = I _in and zero everywhere else (only absorption). Set τ (d ) = d

⇒ Transparency T (s) = exp

− Z s

0 d (r (t))dt

⇒ Received Illumination I _out = T (t max )I in

Nelson Max. Optical Models for Direct Volume Rendering. TVCG 1995 Kajiya, von Herzen. Ray Tracing Volume Densities. Computer Graphics 1984

4 / 13

(6)

1.1 Only Absorption

Assume a medium that only absorbs incoming light, no emission or scattering.

I _out (x ₀ , ω) = Z ∞

0 g (t) exp

− Z t

0 τ (v (r (t ⁰ )))dt ⁰

dt with g (s) = c(s)τ (s) +

Z

Ω

p(ω, ω ⁰ )I _out (r (t), ω ⁰ )d ω ⁰ Set g (∞) = I _in and zero everywhere else (only absorption).

Set τ (d ) = d

⇒ Transparency T (s) = exp

− Z s

0 d (r (t))dt

⇒ Received Illumination I _out = T (t max )I in

Nelson Max. Optical Models for Direct Volume Rendering. TVCG 1995 Kajiya, von Herzen. Ray Tracing Volume Densities. Computer Graphics 1984

4 / 13

(7)

1.1 Only Absorption

Assume a medium that only absorbs incoming light, no emission or scattering.

I _out (x ₀ , ω) = Z ∞

0 g (t) exp

− Z t

0 τ (v (r (t ⁰ )))dt ⁰

dt with g (s) = c(s)τ (s) +

Z

Ω

p(ω, ω ⁰ )I _out (r (t), ω ⁰ )d ω ⁰ Set g (∞) = I _in and zero everywhere else (only absorption).

Set τ (d ) = d

⇒ Transparency T (s) = exp

− Z _s

0 d (r (t))dt

⇒ Received Illumination I _out = T (t max )I in

Nelson Max. Optical Models for Direct Volume Rendering. TVCG 1995 Kajiya, von Herzen. Ray Tracing Volume Densities. Computer Graphics 1984

4 / 13

(8)

1.1 Only Absorption

5 / 13

(9)

1.1 Only Absorption: Solutions

Densities on a grid v i

(1)

Trilinear Interpolation N i (x)

(2)

⇒ log I _out = X

i ∈voxels 8

X

j =1

v _ij Z t

₁^{(i )}

t

^{(i )}₀

N _j (r (t))dt

Assemble into matrices for all rays:

⇒ log I _out = Nv

(1)https://en.wikipedia.org/wiki/Regular_grid

(2)https://en.wikipedia.org/wiki/Trilinear_interpolation

6 / 13

(10)

1.1 Only Absorption: Solutions

Densities on a grid v i

(1)

Trilinear Interpolation N i (x)

(2)

⇒ log I _out = X

i ∈voxels 8

X

j =1

v ij

Z t

₁^{(i )}

t

₀^{(i )}

N j (r (t))dt

Assemble into matrices for all rays:

⇒ log I _out = Nv

(1)https://en.wikipedia.org/wiki/Regular_grid

(2)https://en.wikipedia.org/wiki/Trilinear_interpolation

6 / 13

(11)

1.1 Only Absorption: Differentiation

log I out = N v

output images matrix of

voxel-pixel contributions

voxel densities

⇒ Linear in v , simply solve it

7 / 13

(12)

1.1 Only Absorption: Differentiation

log I out = N v

output images matrix of

voxel-pixel contributions

voxel densities

⇒ Linear in v , simply solve it

7 / 13

(13)

1.2 With Emission

Now every point in space also emits light g (d )

Usually dependent on τ (d ): g (d ) := c(d )τ (d ) for some color c(d ).

I _out = T (t _max )I _in + Z t

max

0 g (t)T (t)dt

with T (s) = exp

− Z s

0 τ (v (r (t)))dt

This is a nested integral and cannot be solved analytically.

8 / 13

(14)

1.2 With Emission

Now every point in space also emits light g (d )

Usually dependent on τ (d ): g (d ) := c(d )τ (d ) for some color c(d ).

I _out = T (t _max )I _in + Z t

max

0 g (t)T (t)dt

with T (s) = exp

− Z s

0 τ (v (r (t)))dt

This is a nested integral and cannot be solved analytically.

8 / 13

(15)

1.2 With Emission

τ (d ) = d , c(d ) = d

τ (d ), c(d ) arbitrary

9 / 13

(16)

1.2 With Emission

τ (d ) = d , c(d ) = d τ (d ), c(d ) arbitrary

9 / 13

(17)

1.2 With Emission

τ (d ) = d , c(d ) = d τ (d ), c(d ) arbitrary

9 / 13

(18)

2. Automatic Differentiation

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f _k (x _k−1 , w _k ) with

k ∈ N number of operations x _i ∈ R ⁿ state variables

x _k ∈ R ^m size of the output / last state w _i ∈ R ^p parameters

⇒ dx _k dw ₁

10 / 13

(19)

2. Automatic Differentiation

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f _k (x _k−1 , w _k ) with

k ∈ N number of operations x _i ∈ R ⁿ state variables

x _k ∈ R ^m size of the output / last state w _i ∈ R ^p parameters

⇒ dx _k dw ₁

10 / 13

(20)

2.1 AutoDiff: Forward Mode

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Replace each x i , w i by

˜ x _i =

x _i , dx

_i

dw

1

, ˜ w _i =

w _i , dw

_i

dw

1

Note that ˜ x ₀ = hx ₀ , 0i, ˜ w ₁ = hw ₁ , 1i

Replace a function c = f (a, b) by ˜ c = ˜ f ( ˜ a, ˜ b) with

c, dc

dc 1

=

f (a, b),

J _{f |a} (a, b) da i

dw ₁ + J _{f |b} (a, b) db i

dw ₁

⇒ O(np) memory

⇒ O(kn ² + knp) computations

11 / 13

(21)

2.1 AutoDiff: Forward Mode

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Replace each x i , w i by

˜ x _i =

x _i , dx

_i

dw

1

, ˜ w _i =

w _i , dw

_i

dw

1

Note that ˜ x ₀ = hx ₀ , 0i, ˜ w ₁ = hw ₁ , 1i

Replace a function c = f (a, b) by ˜ c = ˜ f ( ˜ a, ˜ b) with

c, dc

dc 1

=

f (a, b),

J _{f |a} (a, b) da i

dw ₁ + J _{f |b} (a, b) db i

dw ₁

⇒ O(np) memory

⇒ O(kn ² + knp) computations

11 / 13

(22)

2.1 AutoDiff: Forward Mode

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Replace each x i , w i by

˜ x _i =

x _i , dx

_i

dw

1

, ˜ w _i =

w _i , dw

_i

dw

1

Note that ˜ x ₀ = hx ₀ , 0i, ˜ w ₁ = hw ₁ , 1i

Replace a function c = f (a, b) by ˜ c = ˜ f ( ˜ a, ˜ b) with

c, dc

dc 1

=

f (a, b),

J _{f |a} (a, b) da _i

dw ₁ + J _{f |b} (a, b) db _i dw ₁

⇒ O(np) memory

⇒ O(kn ² + knp) computations

11 / 13

(23)

2.1 AutoDiff: Forward Mode

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Replace each x i , w i by

˜ x _i =

x _i , dx

_i

dw

1

, ˜ w _i =

w _i , dw

_i

dw

1

Note that ˜ x ₀ = hx ₀ , 0i, ˜ w ₁ = hw ₁ , 1i

Replace a function c = f (a, b) by ˜ c = ˜ f ( ˜ a, ˜ b) with

c, dc

dc 1

=

f (a, b),

J _{f |a} (a, b) da _i

dw ₁ + J _{f |b} (a, b) db _i dw ₁

⇒ O(np) memory

⇒ O(kn ² + knp) computations

11 / 13

(24)

2.2 AutoDiff: Backward Mode / Adjoint Method

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Assume x k ∈ R is a scalar (m = 1), define x ˆ _i = dx

_k

dx

_i

∈ R ⁿ , ˆ w _i = dx

_k

dw

_i

∈ R ^p , Note: ˆ x _k = 1.

1. Forward Pass, compute x k normally, but save intermediate results 2. Backward Pass: For i = k, ..., 1

ˆ

x _{i −1} = J _f

_i

_|x

_{i −1}

(x _{i −1} , w _i )

^T

x ˆ _i ˆ

w _i = J _f

_i

_|w

_i

(x _{i −1} , w _i )

^T

x ˆ _i

⇒ O(kn) memory

⇒ O(kn ² + np) computations

12 / 13

(25)

2.2 AutoDiff: Backward Mode / Adjoint Method

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Assume x k ∈ R is a scalar (m = 1), define x ˆ _i = dx

_k

dx

_i

∈ R ⁿ , ˆ w _i = dx

_k

dw

_i

∈ R ^p , Note: ˆ x _k = 1.

1. Forward Pass, compute x k normally, but save intermediate results 2. Backward Pass: For i = k, ..., 1

ˆ

x _{i −1} = J _f

_i

_|x

_{i −1}

(x _{i −1} , w _i )

^T

x ˆ _i ˆ

w _i = J _f

_i

_|w

_i

(x _{i −1} , w _i )

^T

x ˆ _i

⇒ O(kn) memory

⇒ O(kn ² + np) computations

12 / 13

(26)

2.2 AutoDiff: Backward Mode / Adjoint Method

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Assume x k ∈ R is a scalar (m = 1), define x ˆ _i = dx

_k

dx

_i

∈ R ⁿ , ˆ w _i = dx

_k

dw

_i

∈ R ^p , Note: ˆ x _k = 1.

1. Forward Pass, compute x k normally, but save intermediate results

2. Backward Pass: For i = k, ..., 1 ˆ

x _{i −1} = J _f

_i

_|x

_{i −1}

(x _{i −1} , w _i )

^T

x ˆ _i ˆ

w _i = J _f

_i

_|w

_i

(x _{i −1} , w _i )

^T

x ˆ _i

⇒ O(kn) memory

⇒ O(kn ² + np) computations

12 / 13

(27)

2.2 AutoDiff: Backward Mode / Adjoint Method

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Assume x k ∈ R is a scalar (m = 1), define x ˆ _i = dx

_k

dx

_i

∈ R ⁿ , ˆ w _i = dx

_k

dw

_i

∈ R ^p , Note: ˆ x _k = 1.

1. Forward Pass, compute x k normally, but save intermediate results 2. Backward Pass: For i = k, ..., 1

ˆ

x _{i −1} = J _f

_i

_|x

_{i −1}

(x _{i −1} , w _i )

^T

x ˆ _i ˆ

w _i = J _f

_i

_|w

_i

(x _{i −1} , w _i )

^T

x ˆ _i

⇒ O(kn) memory

⇒ O(kn ² + np) computations

12 / 13

(28)

2.2 AutoDiff: Backward Mode / Adjoint Method

x ₀ = const x ₁ = f ₁ (x ₀ , w ₁ ) x ₂ = f ₂ (x ₁ , w ₂ )

. . .

x _k = f n (x k−1 , w k )

Assume x k ∈ R is a scalar (m = 1), define x ˆ _i = dx

_k

dx

_i

∈ R ⁿ , ˆ w _i = dx

_k

dw

_i

∈ R ^p , Note: ˆ x _k = 1.

1. Forward Pass, compute x k normally, but save intermediate results 2. Backward Pass: For i = k, ..., 1

ˆ

x _{i −1} = J _f

_i

_|x

_{i −1}

(x _{i −1} , w _i )

^T

x ˆ _i ˆ

w _i = J _f

_i

_|w

_i

(x _{i −1} , w _i )

^T

x ˆ _i

⇒ O(kn) memory

⇒ O(kn ² + np) computations

12 / 13

(29)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops

I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(30)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(31)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading

I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(32)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(33)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(34)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

(35)

2.3 AutoDiff: Comparison

Forward Mode I ⇒ O(np) memory

⇒ O(kn ² + knp) ops I Arbitrary number of outputs

I Compile-time: implement with operator overloading I Expensive for large p

⇒ use if k large, p small

⇒ Sensitivity Analysis in simulations

Backward Mode I ⇒ O(kn) memory

⇒ O(kn ² + np) ops

I Only one output! Or repeat backward pass

I Dynamic computation graph Dynamic memory

I Lots of memory

⇒ use if k small, p large

⇒ Neural networks

For DVR we need some hybrid method...

13 / 13

Inverse Volume Rendering