Institute of
Computer Graphics Institute of
Computer Graphics
Adaptive Coded Aperture
Photography
Oliver Bimber, Haroon Qureshi, Daniel Danch
Institute of Computer Graphics
Johannes Kepler University, Linz
Anselm Grundhoefer
Institute of
Computer Graphics
Institute of
Computer Graphics
Institute of
Computer Graphics
Institute of
Computer Graphics
Institute of
Computer Graphics
Motivation
Institute of
Computer Graphics
•
Narrow apertures
•
large depth of field (= high frequencies in out-of-focus regions)
•
in low light throughput (= low signal-to-noise ratio)
•
JPEG compression
•
attenuates high frequencies (in focused and out-of-focus
regions)
•
Can we optimize apertures with respect to JPEG compression?
•
frequencies attenuated by JPEG compression do not have to
be supported optically by the aperture
•
results in higher light throughput (= higher signal-to-noise ratio
Institute of Computer Graphics Veeraraghava n, et al, ,‘07 Levin, et al, ,‘07 Bando, et al, ,’08 Zhou, et al, ,’09 Liang, et al, ,’08 Nagahara, et al, ,’10 Grosse, et al, ,’08 Grosse,
et al, ,’10 this paper
binary intensity color static dynamic adapted zero crossings Fourier magnitudes noise models
Applications: post-exposure refocusing, defocus deblurring, depth
reconstruction, matting, light field acquisition, projector-defocus
compensation.
Institute of Computer Graphics Veeraraghava n, et al, ,‘07 Levin, et al, ,‘07 Bando, et al, ,’08 Zhou, et al, ,’09 Liang, et al, ,’08 Nagahara, et al, ,’10 Grosse, et al, ,’08 Grosse,
et al, ,’10 this paper
binary intensity color static dynamic adapted zero crossings Fourier magnitudes
Related Work
Institute of Computer Graphics FT FT
F*
pre-computed
threshold
frequencies apply pseudo-inverse
input image binarize
(optional) dynamic aperture pattern projected image
(Adaptive) Coded Aperture Projection, Grosse, Wetzstein, Grundhoefer,
and Bimber, ACM Transaction on Graphics, 2010
Institute of
Computer Graphics
Institute of
Computer Graphics
set aperture and capture image
Institute of
Computer Graphics
set aperture and capture image
JPEG compression and
frequency filtering
Institute of
Computer Graphics
set aperture and capture image
JPEG compression and
frequency filtering
compute and set coded aperture
Institute of
Computer Graphics
set aperture and capture image
JPEG compression and
frequency filtering
re-capture
image
compute and set code aperture
Institute of
Computer Graphics
set aperture and capture image
JPEG compression and
frequency filtering
re-capture
image
compute and set code aperture
transform bokeh and
depth of field
Institute of
Computer Graphics
set aperture and capture image
JPEG compression and
frequency filtering
re-capture
image
compute and set code aperture
Institute of
Computer Graphics
Institute of Computer Graphics
or
iginal
(a
ll
freque
nc
ies)
mask
ed
spec
trum
nc
ies
Frequency Filtering
Institute of
Computer Graphics
Construct a binary frequency mask (m) and compute intensity aperture
pattern (a) by minimizing the variance of its Fourier transform for all
important frequencies:
Institute of
Computer Graphics
M is the diagonal matrix containing the binary frequency mask values
of m, F is the discrete Fourier transform matrix (i.e., the set of
orthogonal Fourier basis functions in its columns), a is the unknown
vector of the coded aperture pattern, and e is the vector of all ones -
this can be solved quickly with the pseudo-inverse:
Construct a binary frequency mask (m) and compute intensity aperture
pattern (a) by minimizing the variance of its Fourier transform for all
important frequencies:
Institute of
Computer Graphics
q=70
masked spectrum
masked spectrum
MTF of coded intensity mask
MTF of coded intensity mask
MTF of binarized mask
MTF of binarized mask
magnitude
low
high
q=50
binarize
Institute of
Computer Graphics
Institute of
Computer Graphics
bokeh transformation
Institute of Computer Graphics regular c oded (bef ore bok eh trans fo rm a tion) rm a tio n)
Bokeh Transformation
Institute of
Computer Graphics
Capturing an image through an aperture with given PSF can be
considered as convolution (multiplication in frequency domain):
Institute of
Computer Graphics
Capturing an image through an aperture with given PSF can be
considered as convolution (multiplication in frequency domain):
Bokeh transformation can be carried out as follows:
Institute of
Computer Graphics
Capturing an image through an aperture with given PSF can be
considered as convolution (multiplication in frequency domain):
Bokeh transformation can be carried out as follows:
However, the scales (s´ and s´´) are entirely unknown since the scene
depth is unknown!
Institute of Computer Graphics
13
x
13
1
1x
1
1
9x
9
7x
7
5x
5
3x
3
1x
1
co
nvolut
ion
sc
ale
(s
‘‘)
simulated
measured
Bokeh Transformation
con
volu
tio
Institute of
Computer Graphics
Institute of
Computer Graphics
color and brightness matching
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
Institute of
Computer Graphics
S
S
S
S
S
S
Institute of
Computer Graphics
color and brightness matching
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
Institute of
Computer Graphics
S
S
S
S
S
S
Institute of Computer Graphics
con
vol
utio
n
(s‘
‘)
de
con
vol
utio
n
(s‘)
scal
e
di
ffere
nce
Bokeh Transformation
Institute of
Computer Graphics
3.2s/6.4s • + 1.6s/6.4s • + 0.8s/6.4s • + 0.4s/6.4s • + (1/5)s/6.4s • + (1/10)s/6.4s • + (1/20)s/6.4s • + (1/40)s/6.4s • =
Institute of
Computer Graphics
3.2s/6.4s • + 1.6s/6.4s • + 0.8s/6.4s • + 0.4s/6.4s • + (1/5)s/6.4s • + (1/10)s/6.4s • + (1/20)s/6.4s • + (1/40)s/6.4s • =
Institute of
Computer Graphics
3.2s/6.4s • + 1.6s/6.4s • + 0.8s/6.4s • + 0.4s/6.4s • + (1/5)s/6.4s • + (1/10)s/6.4s • + (1/20)s/6.4s • + (1/40)s/6.4s • =
3.2s 1.6s 0.8s 0.4s 1/5s 1/10s 1/20s 1/40s
Institute of Computer Graphics re g u la r coded
uncompressed (original) uncompressed (increased and matched brightness) close-up
(regular)
close-up (coded)
Institute of Computer Graphics re g u la r coded
Results
Institute of Computer Graphics 90 70 50 30 reg ula r coded compr ess ion 10% 27% regular coded
regular aperture opening (2%, 10%, 27%)
reg u lar ap er tu re o p en in g q=90 q=70 q=50 q=30
Institute of
Computer Graphics
regular
coded
front center back focus (front, center, back)
fo cu s regular n d iti o n s
Results
Institute of
Computer Graphics
Note that if we ignored other noise sources, such as dark noise and
read noise, and considered shot noise only, then the gain in SNR
would be proportional to the square root of the light throughput gain.
Institute of
Computer Graphics
•
LCA has low light transmittance (only 30% when completely
transparent), low contrast (7:1), and is small (limited DOF
difference)
•
use larger reflective DMAs or LCoS panels!
•
Coded aperture pattern is scaled manually to roughly match the
depth of field while remaining depth-of-field differences are
removed by the bokeh transformation
•
automize this scale estimation
•
Explore alternatives
•
downsampling instead of / together with compression (i.e.,
trade resolution / compression for light through put or shutter
Institute of
Computer Graphics
