Information Extraction from Non-Line-of-Sight Objects. Using Plenoptic Data from Scattered Light A DISSERTATION SUBMITTED TO THE FACULTY

(1)

Information Extraction from Non-Line-of-Sight Objects

Using Plenoptic Data from Scattered Light

A DISSERTATION SUBMITTED TO THE FACULTY

OF THE UNIVERSITY OF MINNESOTA

BY

Takahiro Sasaki

IN PARTHIAL FULFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF DOCTOR OF PHIOSOPHY

James R. Leger, advisor

(2)

(3)

i

Acknowledgements

I wish to express my sincere appreciation and gratitude to my advisor, Prof. James R. Leger, for giving me the wonderful opportunity to take part in this research project. His help was not only with advanced insights in optics from his diverse experiences but also by encouraging me to cope with challenging problems for higher achievement. Moreover, he showed me a role model of leaders to organize researchers and facilitate their cooperation to overcome difficulties in the project, and entrusted me with a crucial experimental demonstration of our researches exhibited to the project’s sponsors. These unforgettable experiences have been motivating me to pursue further growth as an independent scientist. Everything had started since he kindly accepted me as a group member when I had hardship and needed changing the research group for an unexpected circumstance in my master’s degree. I would also like to thank Prof. Jarvis Haupt and Prof. Joseph Talghader for the meaningful technical discussions and Prof. James Fienup from the University of Rochester for detailed feedback to our work. Regarding my Ph.D. final defense, I would like to appreciate Prof. Jarvis Haupt, Prof. Gary Meyer and Prof. Maria-Carme Calderer for serving as the committee members. I would want to thank Dr. Di Lin for the meaningful critical discussions to improve my researches and Connor Hashemi for his help with the laboratory experiments. I am also deeply indebted to Dr. William ‘Mint’ Kunkel for encouraging me when I had hardship, and it is unforgettable that we often enjoyed chatting and joking in our office!

This research was conducted as a part of Revolutionary Enhancement of Visibility by Exploiting Active Light-fields (REVEAL) program by the Defense Advanced Research Projects Agency (DARPA) under grant number HR0011-16-C-0024 P000005.

(4)

ii

Dedication

This dissertation is dedicated to my parents, Hiroshi and Masako Sasaki. Since I decided to change my career in Japan and started pursuing the doctorate degree at the University of Minnesota, they have always been on my side and believing in my success. Without their support and encouragement, I would not have been able to have the unforgettable experiences and to achieve the success.

(5)

iii

Abstract

We investigate the utility of plenoptic data for extracting information from a scene where the light from unknown objects in the scene is viewable only after scattering from a diffuse surface. A primary goal of this research is to estimate the objects’ locations in the hidden scene, and to extract additional information, such as the object’s shape and brightness.

In the first part, we derive a rigorous relationship between the object and the scattered light fields, which is cast in terms of a system of Fredholm integral equations of the first kind with the BRDF of the scattering surface, and the object information is reconstructed by solving these equations. Based on the Fourier transformation, we propose a simple BRDF model and analyze the reconstruction errors by introducing newly defined parameters reflecting the BRDF’s characteristic, the degree of specularity and the effective SNR. We then obtain optimal regularized solutions to the equations under a variety of conditions. Moreover, we provide a fundamental limit of retrievable information content from the scattered light. A comparison with experimental results is reported.

In the second part, we investigate the use of plenoptic data for locating objects from a scattered light field by using the results obtained in the first part. The resolution limits of the transverse and longitudinal location estimates are derived from fundamental considerations on the scattering physics and measurement noise. Based on the refocusing algorithm developed in the computer vision field, we derive a simple alternative formulation of the projection slice theorem in a form directly connecting the light field and spatial frequency spectrum. Using this alternative formulation, we propose a spatial frequency filtering method that is defined on a newly introduced mixed space-frequency plane and achieves the theoretically-limited depth resolution. Moreover, we propose an improved refocusing algorithm to more accurately estimate the object’s brightness information. An experimental verification is provided.

(6)

iv

Figure 2-10: Experimentally measured BRDF samples: (a) Brushed metal, (b) coated white-paper with highlights #1, (c) coated white-white-paper with highlights #2, and (d) white paint with a pearl finish applied to glass. ... 33 Figure 2-11: Comparison between the theoretical and experimental relative noise errors from four different scattering materials. ... 34 Figure 2-12: Measured signal𝐿𝑚𝑒𝑎𝑠 and reconstructed 𝐿𝑜𝑏𝑗 using the two regularization

methods. ... 36 Figure 2-13: Details pertaining to the truncation regularization method. (a) Mean-squared error of the reconstruction as a function of truncation term. (b) The first derivative of MSE calculated from eq. (2-44). ... 37 Figure 2-14: Details pertaining to the Wiener filter. (a) The left-hand-side matrix from eq. (2-46) showing logarithm of each q-s element. (b) Logarithm value of the regularizer term calculated from eq. (2-47). (c) Five trials of the Wiener filter estimates. ... 38

Chapter 3.

Figure 3-1: Simplified optical system to illustrate depth estimation methods. The systems is shown unfolded and without scattering plane tilt for simplicity. ... 46 Figure 3-2: Point source light field as a basis object. (a) Light field distribution at the plane that includes the point source, and (b) light field on a plane that is placed at a finite distance from the point source. ... 48 Figure 3-3: The focal-stack obtained by two different refocusing algorithms assuming two point-source objects emitting the same radiance (simulation). (a) and (b) are obtained by the conventional refocusing algorithm from eq. (3-2). (c) and (d) are computed by the new radiance preserving refocusing algorithm (RP-refocusing) from eq. (3-3). ... 49 Figure 3-4: The system operator as a spatial convolution integral. ... 51 Figure 3-5: Relation between the angular and the spatial bases; (a) relation expressed in y-z spatial plane, (b) relation in 𝑦𝑜-𝜃𝑜 light field plane. ... 54

Figure 3-6: (a) The system after the deconvolution process. (b) Light field distribution on the mirror plane reflecting the two blurred sources. ... 55

(14)

xii

Figure 3-7: Measured light field 𝐿𝑚𝑒𝑎𝑠 on the measurement plane (i.e. the scattering plane)

and the solvable spatial domain. (a): Scattered light field from two point sources. (b) and (c): Angular distributions at two different scattering points at 𝑦𝑚= 𝑦1 and 𝑦𝑚= 𝑦2. ... 57

Figure 3-8: Reconstructed light field 𝐿0𝑜𝑏𝑗 on the measurement plane and various definitions

for the new formulation. ... 60 Figure 3-9: The refocusing as a basis mapping. (a) 𝐿𝑝_𝑦.𝑘 distribution on the 𝑝𝑦- 𝑘 plane

obtained from 𝐿_𝑜𝑏𝑗0 (𝑦𝑚, 𝜃𝑚) by Fourier transforming in the 𝑦𝑚 direction and by discrete

sampling in the 𝜃𝑚 direction. (b) 𝑝𝑦-𝑞𝑧 full spatial frequency plane. ... 63

Figure 3-10: (a) Reconstructed 𝐿0𝑜𝑏𝑗(𝑦𝑚, 𝜃𝑚) on the measurement plane and (b) spatial

frequency spectrum and the ODF defined on the 𝑝𝑦-𝑞𝑧 plane (upper half) assuming a single

point-source located at 𝑧 = −𝑑1. ... 65

Figure 3-11: (a) 𝑝𝑦-𝑞𝑧 spectral distribution (upper half). (b) The mixed space-frequency

representation corresponding to (a). (c) A comparison of z-profiles obtained by three different methods. (Note: These were experimentally obtained with a single small object.) ... 67 Figure 3-12: Weakness and improvements for the ODF method. Figures are experimentally obtained using a single small object for (a) and two small objects for (b)~(d). (a) z-profiles obtained at multiple 𝑝𝑦 locations from the 𝑝𝑦-𝑧 plane of fig. 3-11 (b). (b) y-z distribution

obtained by the RP-refocusing method in an example case of two objects having similar z-locations. (c) The 𝑝𝑦-𝑧 distribution with interference, and (d) The 𝑝𝑦 distribution after

applying the interference removal procedure. ... 69 Figure 3-13: An experimental example of three object case. (a): y-z distribution obtained by the RP-refocusing. (b): The space-frequency mixed distribution corresponding to the bottom half section of (a). (c) z-profiles of the three objects by the ODF method (note: seven profiles were multiplied for each result). ... 71 Figure 3-14: (a) Experimental setup (top view). (b) Example of the measured light field 𝐿𝑚𝑒𝑎𝑠

and the rectangular measurable and solvable domain. (c) Reconstructed 𝐿0_𝑜𝑏𝑗 corresponding to the rectangular domain in (b). ... 72 Figure 3-15: (a) and (b) Examples of the y-z distribution and the 𝐿𝑜𝑏𝑗0 of an object located at

(𝑦𝑜, 𝑧) = (0, −224.5) [mm] where 𝑝̂𝑚𝑎𝑥= 10. (c) Comparison of the actual and estimated

(15)

xiii

Figure 3-16: Depth estimation performance of the ODF with multiplication method. (a): Depth estimation results corresponding to the objects placed on the z-axis in fig. 3-15 (c). (b) Depth estimation results for the objects having different y-locations showed in fig. 3-15 (c). ... 74 Figure 3-17: z-dependence of theoretical and experimental blur widths. ... 75 Figure 3-18: y-dependence of theoretical and experimental blur widths. (a) y-blur as a function of y-location; (b) z-blur as a function of y-location. ... 75 Figure 3-19: Statistics on the object location estimation comparing the RP-refocusing and the ODF with multiplication. ... 77 Figure 3-20: Interaction among multiple objects; A, B and C. (a) y-z profile obtained by the RP-refocusing method and the reconstructed 𝐿0_𝑜𝑏𝑗 in a higher 𝑝̂𝑚𝑎𝑥 case; (b) the y-z profile

for a lower 𝑝̂𝑚𝑎𝑥 case; (c) A comparison of z-profiles obtained by the RP-refocusing and the

ODF method with multiplication. ... 78

Appendix A.

Figure A: (a) Two different basis functions, and (b) their relation 𝛽𝑞−𝑝. ... 88

Appendix B.

Figure B1: The rectangular solvable and measurable domain and the reconstructed 𝐿0𝑜𝑏𝑗

including line spread functions representing light from small objects. One red line spread function having the slope of 𝑧𝑚𝑖𝑛 intersects two diagonals of the domain, and two blue line

spread functions having the slope of 𝑑1 intersect one of the diagonals of the domain. ... 98

Figure B2: Light field of two small objects. (a) 𝐿𝑜𝑏𝑗0 (𝑦𝑚, 𝜃𝑚). (b) and (c) Light fields obtained

by propagating 𝐿0𝑜𝑏𝑗 to each object’s z-location. ... 100

Figure B3: Numerical algorithm for the RP-refocusing. (1) 𝐿_𝑜𝑏𝑗0 expressed as a collection of 1-D arrays in 𝜃𝑚-direction. (b) Rays converging to the particular location (𝑦𝑗, 𝑧𝑖). ... 103

Figure B4: Comparisons between the RP-refocusing from eq. (3-3) and the conventional refocusing from eq. (3-2). (a): y-dependency of the peak value (the colored area corresponds to 𝑊𝑦). (b): z-dependency of the peak value. (c) and (d): Comparison of y-z spatial distribution

in the case of three small objects.

(16)

１

Chapter 1. INTRODUCTION

1.1 Research objective

A full description of light propagation and scattering under the geometrical optics approximation is provided by the plenoptic function, where both the location and angle of each ray are known [11]. In this research, we fully utilize light-field data consisting of a radiance function of angular and transverse spatial variables to extract information from a scene where the light from unknown objects in the scene is viewable only after scattering from a rough surface such as a wall. This imaging scenario is illustrated in fig. 1-1 which consists of the unknown objects, a scattering surface and a camera for measurements.

Fig.1-1 Assumed imaging scenario

The object information conveyed by the scattered light is reduced by the scattering event; however, in many cases significant information can be retrieved by solving the inverse scattering problem, assuming prior knowledge of the scattering surface, such as its reflectivity properties and its geometrical location and shape. Although the extractable information of the objects can have a variety of forms, we develop methods and analyses primarily aiming at estimating the hidden objects’ locations from the scattered light. Other related information, such as the

(17)

２

objects’ count, shape and brightness, is considered as well. We expect these methods to find many applications in the areas of security, safety and rescue.

1.2 Research organization

This research consists of two major components: the unknown object’s light field reconstruction from the scattered light (chapter 2), and the object’s location estimation from the reconstructed light field (chapter 3). First, some fundamentals, such as definition of the light field and basic light field operations, are reviewed in this chapter as a preparation for the following chapters. Second, chapter 2 describes the followings: 1) a derivation of the rigorous system equation connecting the unknown objects and the scattered light fields, 2) a procedure to reconstruct the object light field by solving the system equation, 3) detailed analyses on the reconstruction error by introducing a simple shift-variant BRDF model, and 4) fundamental regularization methods to realize optimal reconstruction. Third, based on the results obtained in chapter 2, chapter 3 explains the followings: 1) an object depth estimation procedure as a combination of the light field reconstruction process in chapter 2 and the refocusing algorithm adopted from the computer vision field, 2) theoretical resolution limits of the transversal and longitudinal location estimation of the objects, 3) a derivation of alternative projection slice theorem directly connecting the light field and the full spatial frequency spectrum, and 4) a spatial frequency filtering method to obtain the object’s depth profile achieving the theoretical limit. In both chapters, experimental verifications are provided, assuming a variety of scattering conditions to prove applicability of our methods in real scenes.

1.3 Definitions and basic operations for light fields

The most detailed description of light information assuming geometrical optics approximation is generally expressed as the plenoptic function [11], which is a function of radiance and may include

(18)

３

dependencies on location and angle of each ray, time, wavelength and polarization. In this work, we choose to express the information as a so-called light field, which is a reduced form of the plenoptic function describing the angular and transverse spatial variation of the radiance in a given plane. (Note: a variety of different expressions were proposed in the computer vision and the computer graphics fields [9,10].) It is originally a four dimensional (4-D) function, two spatial and two angular variables; however, we focus on the most important two of the four variables for simplicity, one in space and one in angle as expressed as 𝐿(𝑦, 𝜃). Extension to the 4-D is straight forward.

Fig. 1-2 shows an unfolded optical system equivalent to the imaging scenario in fig. 1-1 for definitions. The object and measurement planes are virtual to express the object’s light field 𝐿_𝑜𝑏𝑗 and the measured light field 𝐿_{𝑚𝑒𝑎𝑠} respectively. The scattering surface is modelled by a plane that maps the incoming light field 𝐿𝑖𝑛 into an outgoing light field 𝐿𝑜𝑢𝑡.

Fig. 1-2. Unfolded optical system equivalent to fig. 1-1.

Our system illustrated in figures 1-1 and 1-2 can be fully described by two simple linear light field operations: propagation and scattering. The propagation operator is denoted as 𝑃_𝑑 , and is expressed as a change-of-variables in eq. (1-1) with a propagation distance 𝑑, given by the ABCD matrix of geometrical optics. (𝑦′_{, 𝜃}′_{) and (𝑦, 𝜃) represent the spatial location and propagation angle}

(19)

４

𝑃_𝑑[𝐿(𝑦′, 𝜃′)](𝑦, 𝜃) = 𝐿(𝑦 − 𝑑 tan 𝜃 , 𝜃)

Fig. 1-3 illustrates the propagation operation, where fig. 1-3 (a) is expressed in the spatial 𝑦-𝑧 plane, and (b) is in the 𝑦-𝜃 light field plane. In fig. 1-3 (b), the small circle on the left and that on the right represent corresponding rays before and after the propagation respectively. We can see from the figures that the propagation is interpreted as a shear in the spatial y-direction in the light field plane. As is commonly known, the radiance theorem [9,10] states that the radiance conveyed along a particular ray is preserved before and after the propagation, which can be seen in fig. 1-3 (b) from the fact that the area of the small square on the left and that of the small parallelogram on the right are the same because the ABCD matrix’s determinant has a value of one.

Fig. 1-3. Visually illustrated propagation operation.

Eq. (1-2) defines the scattering operator 𝐵 at a particular point on the scattering plane 𝑦𝑠 as an integral

that maps incoming rays 𝐿_𝑖𝑛(𝑦_𝑠, 𝜃_𝑖𝑛) to outging rays 𝐿_𝑜𝑢𝑡(𝑦_𝑠, 𝜃_𝑜𝑢𝑡) from the scattering plane through the bidirectional reflectance distribution function (BRDF) 𝑓𝐵𝑅𝐷𝐹 defined as

𝐿𝑜𝑢𝑡(𝑦𝑠, 𝜃𝑜𝑢𝑡) = 𝐵[𝐿𝑖𝑛(𝑦𝑠, 𝜃𝑖𝑛)] ≡ ∫ 𝑓𝐵𝑅𝐷𝐹(𝑦𝑠, 𝜃𝑖𝑛, 𝜃𝑜𝑢𝑡)𝐿𝑖𝑛(𝑦𝑠, 𝜃𝑖𝑛)𝑑𝜃𝑖𝑛 𝑏

𝑎

, (1-2) (1-1)

(20)

５ where 𝑓𝐵𝑅𝐷𝐹(𝑦𝑠, 𝜃𝑖𝑛, 𝜃𝑜𝑢𝑡) ≡

𝐿_𝑜𝑢𝑡(𝑦_𝑠, 𝜃_𝑜𝑢𝑡) 𝑑𝐿𝑖𝑛(𝑦𝑠, 𝜃𝑖𝑛) cos 𝜃𝑖𝑛𝑑𝜃𝑖𝑛

Figure 1-4 illustrates the scattering operation, where fig. 1-4 (a) is expressed in the spatial 𝑦-𝑧 plane, and fig. 1-4 (b) is in 𝑦-𝜃 light field plane. In fig. 1-4 (b), the small circle on the left and that on the right represent corresponding rays, and the scattering can be visually interpreted as a spread in 𝜃 -direction in the light field plane.

Fig. 1-4. Visually illustrated scattering operation.

Finally, another light field operation, projection, is introduced as eq. (1-3), which is denoted 𝑆_𝜃 and is defined as light-field integration in terms of the angular variable 𝜃 across the angular domain [𝑎, 𝑏]. As seen, this operator removes the 𝜃-dependency from light fields, and will play a major role in extracting spatial information as will be explained in chapter 3.

𝑆_𝜃[𝐿(𝑦, 𝜃)](𝑦) = ∫ 𝐿(𝑦, 𝜃)𝑑𝜃

𝑏

(21)

6

Chapter 2 OBJECT LIGHT FIELD RECONSTRUCTION

FROM SCATTERED LIGHT USING PLENOPTIC DATA

2.1 Introduction of this chapter

In this chapter, we develop methods and analyses for the hidden-objects’ light field reconstruction from the scatterd light. First, in section 2.3, we derive a system equation showing a rigorous relationship between the object light field and the scattered light field, which is cast in terms of a system of Fredholm integral equations of the first kind, where the BRDF of the scattering surface is incorporated into the kernel of the integral equations. And then, the object’s full light field (including both spatial and angular dimensions) is retrieved by solving the system equation assuming prior knowledge of the BRDF. Second, in section 2.4, the BRDF is modeled as a combination of a quasi shift-invariant term and a separable term in accordance with the behavior of most common scattering materials, which leads to an analytical solution in the frequency domain to the equations. Based on the analytical solution, we analyze the error in the reconstructions by introducing the newly defined BRDF-related parameters; the degree of specularity and the effective SNR. Third, in section 2.5, we derive two fundamental regularization methods in the form capable of employing the above-mentioned BRDF model; the Fourier trunction method and the Wiener optimal filter. Finally, in section 2.6, data from scattering experiments quantitatively verify the error analysis and the regularization methods under a variety of conditions, and the fundamental limit of retrievable information content is discussed.

2.2 Related previous work

(22)

7

forward problems have been actively researched because they play significant roles in computer-rendering which is one of the most fundamental techniques in their fields. In this section, relevant previous works are reviewed, and their applicability to our current purpose is discussed.

- Early inverse scattering work in the computer vision field

In early works on the inverse scattering problem, Marschner [6] proposed a framework involving the three fundamental elements of the computer graphics: a model (i.e. our scattering surface), lighting from an illumination (i.e. our unknown object) and a camera. In the framework, he developed a method to estimate lighting directions in a photograph of the model, assuming prior knowledge of a set of many basis photographs of the model that were obtained using basis illuminations from many different angles. However, this work is not directly applicable to our current purpose because the method is limited to directional (or angular) information of light fields, and detailed reconstruction mechanisms is not explained.

- Previous frequency-domain methods and analyses for the scattering problems

In our work, frequency-domain analyses and methods are developed to consider the scattering physics. Similarly to our work, forward scattering problems [16,39-42] and inverse scattering problems [1-5] were investigated in the frequency domain in the computer vision and the computer graphics fields. By assuming a particular optical system consisting of infinitely far illumination (i.e. our unknown objects) and an object having a curved homogeneous scattering surface (i.e. our scattering surface), Cabral [39] introduced a shift-invariance between the incident angle of the lighting and the surface-normal angle of the scattering surface, and derived a convolution integral including the BRDF as a convolution kernel. (Note: The above-mentioned optical system is commonly assumed for computer-graphics and computer-vision related purposes.) Moreover, based on the convolution equation derived

(23)

8

by Cabral, Ramamoorthi [1] established a frequency-domain signal-processing framework for inverse rendering problems. Although these frequency-domain methods are advantageous for their purposes because they can handle a variety of BRDF types, they only consider lighting directional information (i.e. the angular component of light field) and cannot handle flat or inhomogeneous scattering surfaces. Thus, these authors’ works are not adequate for our current purpose.

-Previous full light field reconstruction research

Our methods and analyses fully consider both spatial and angular dimensions of light field. Similarly to our work, Aoto [7,8] proposed a method called irradiance decomposition to estimate the four-dimensional full light field of an illumination pattern. He formulated a linear equation connecting the illumination’s light field (i.e. our unknown object’s light field) and an irradiance distribution on a scattering surface (i.e. our scattering plane) measured by a camera. For making the problem solvable, these authors formulated a set of equations by changing the scatter's location, while the illumination and the camera remained fixed. This work is similar to our work in terms of the full light field recovery and using a virtual plane to express light fields; however, there are the following differences; 1) it uses an irradiance distribution to represent the scattered light which is reduced from the light field and only contains spatial information, and 2) it requires moving the scattering surface and seems only applicable to highly diffusive Lambertian-like scatters. For these reasons, this method is not appropriate for our current purpose.

2.3 Linear composite operator connecting input and output light fields and its

discrete expression in the frequency domain

(24)

9

In this sectoin, we derive the system operator connecting 𝐿𝑜𝑏𝑗 and 𝐿𝑚𝑒𝑎𝑠 by taking the four steps

illustrated in fig. 2-1. In the derivation, the light field operators defined in chapter 1, propagation and scattering, will be employed. For the imaging senario and the definitions, see figures 1-1 and 1-2.

Fig. 2-1. Four steps for deriving the system operator. (a) Object discrete sampling in 𝜽𝒐. (b) Propagation to the scattering plane. (c) Scattering expressed by the BRDF.

(d) Propagation to the measurement plane.

The overall system operator is linear as it is composed of a concatenation of linear propagation and linear scattering operations. First, the sifting theorem of delta functions is used to express the object light field as 𝐿𝑜𝑏𝑗(𝑦𝑜, 𝜃𝑜) =

∫ 𝐿𝑜𝑏𝑗(𝑦𝑜, 𝜗)𝛿(𝜃𝑜− 𝜗 )𝑑𝜗 𝑏

𝑎 . This is shown in fig. 2-1 (a) at two discrete angles 𝜗1 and 𝜗2. Second, the

propagation from the object plane to the scattering plane is applied, where 𝐿𝑖𝑛 can be obtained as eq. (2-1) from

eq. (1-1). (Note: [𝑎, 𝑏] represents the angular domain.) This operation does not change the y-profile of the light field but rather shears it in the y-direction as a function of 𝜃 as shown in fig. 2-1 (b).

𝐿𝑖𝑛(𝑦𝑠, 𝜃𝑖𝑛) = ∫ 𝐿𝑜𝑏𝑗(𝑦𝑠− 𝑑1tan 𝜗 , 𝜗)𝛿(𝜃𝑖𝑛− 𝜗)𝑑𝜗 𝑏

𝑎

(25)

10

Third, by integrating the product of 𝐿𝑖𝑛 and the BRDF with respect to 𝜃𝑖𝑛, 𝐿𝑜𝑢𝑡 is obtained as eq. (2-2) from

eq. (1-2). (Note: the BRDF is assumed to be homogeneous for the moment, but this restriction will be removed in the next section.) This operation can be viewed as a spreading in 𝜃𝑜𝑢𝑡, as shown in fig. 2-1 (c).

𝐿𝑜𝑢𝑡(𝑦𝑠, 𝜃𝑜𝑢𝑡) = ∫ 𝐿𝑖𝑛(𝑦𝑠, 𝜃𝑖𝑛)𝑓𝐵𝑅𝐷𝐹(𝜃𝑖𝑛, 𝜃𝑜𝑢𝑡) cos 𝜃𝑖𝑛𝑑𝜃𝑖𝑛 𝑏 𝑎 = ∫ 𝐿𝑜𝑏𝑗(𝑦𝑠− 𝑑1𝑡𝑎𝑛 𝜗 , 𝜗) 𝑏 𝑎 {∫ 𝛿(𝜃𝑖𝑛− 𝜗)𝑓𝐵𝑅𝐷𝐹(𝜃𝑖𝑛, 𝜃𝑜𝑢𝑡) cos 𝜃𝑖𝑛𝑑𝜃𝑖𝑛 𝑏 𝑎 } 𝑑𝜗 = ∫ 𝐿𝑜𝑏𝑗(𝑦𝑠− 𝑑1tan 𝜗 , 𝜗)𝑓𝐵𝑅𝐷𝐹(𝜗, 𝜃𝑜𝑢𝑡) cos 𝜗 𝑑𝜗 𝑏 𝑎

Finally, by applying the propagation from the scattering plane to the measurement plane and replacing 𝜗 with 𝜃𝑜, 𝐿𝑚𝑒𝑎𝑠 can be obtained as shown below:

𝐿𝑚𝑒𝑎𝑠(𝑦𝑚, 𝜃𝑚) = ∫ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝐿𝑜𝑏𝑗(𝑦𝑚− 𝑑2tan 𝜃𝑚− 𝑑1tan 𝜃𝑜, 𝜃𝑜)𝑑𝜃𝑜 𝑏

𝑎

B. Two necessary conditions to establish a Fredholm integral equation of the first kind

The Fredholm integral equation of the first kind, expressed in the form 𝑓(𝑥) = ∫ 𝐾(𝑥, 𝑦)𝑔(𝑦)𝑑𝑦_𝑎𝑏 , is commonly observed in linear inverse problems [20-22]. However, eq. (2-3) differs from a Fredholm equation in two aspects. First, the input 𝐿𝑜𝑏𝑗 has a dependency on the output variables 𝑦𝑚 and 𝜃𝑚. To convert eq. (2-3)

to a conventional Fredhom equation, the dependent part of 𝐿𝑜𝑏𝑗 must be fixed to a constant 𝑐𝑟 as expressed in

eq. (2-4). Second, the spatial variable 𝑐𝑟− 𝑑1tan 𝜃𝑜 in 𝐿𝑜𝑏𝑗 must be expressed in term of 𝑦𝑜, as in eq. (2-5).

These two conditions are necessary to make eq. (2-3) solvable. As is apparent from eq. (1-1), eq. (2-4) represents the light field diverging from a point on the scattering plane at 𝑦𝑠 = 𝑐𝑟. Similarly, eq. (2-5) represents the light

field converging to the same point. (Note: These conditions are illustrated as two cones in fig. 2-2).

𝑦𝑚− 𝑑2tan 𝜃𝑚= 𝑐𝑟

(2-2)

(2-3)

(26)

11

𝑐𝑟− 𝑑1tan 𝜃𝑜= 𝑦𝑜

Fig. 2-2. Entire (unfolded) system model including inhomogeneous and arbitrary shaped scattering surface.

Because of the two conditions in eq. (2-4) and eq. (2-5), we need to solve a set of independent Fredholm equations at each point 𝑦𝑠 = 𝑐𝑟 for 𝑟 = 0, 1, 2 …on the scattering plane in order to cover the entire two

light fields 𝐿𝑜𝑏𝑗 and 𝐿𝑚𝑒𝑎𝑠. Therefore, the complete relationship between object plane light field and

the measurement plane light field can be expressed as a system of independent equations. Moreover, the BRDF can be inhomogeneous and the scattering surface can have an arbitrary shape because each equation includes only one point on the scattering surface. By introducing 1) a spatial variable 𝑐𝑟 into

the BRDF, 2) the new notation of distances 𝑑1,𝑟 and 𝑑2,𝑟 and 3) the surface normal angle 𝛩𝑟 as

defined in fig. 4, the entire system can be expressed as: 𝐿𝑚𝑒𝑎𝑠(𝑦𝑚, 𝜃𝑚) = ∑ 𝐿_{𝑚𝑒𝑎𝑠𝑟}(𝑐𝑟+ 𝑑2,𝑟tan 𝜃𝑚, 𝜃𝑚), 𝑟 where 𝐿_{𝑚𝑒𝑎𝑠𝑟}(𝑐𝑟+ 𝑑2,𝑟tan 𝜃𝑚, 𝜃𝑚) = ∫ 𝑓𝐵𝑅𝐷𝐹(𝑐𝑟, 𝜃𝑜− 𝛩𝑟, 𝜃𝑚− 𝛩𝑟) cos(𝜃𝑜− 𝛩𝑟) 𝑏 𝑎 𝐿𝑜𝑏𝑗(𝑐𝑟− 𝑑1,𝑟tan 𝜃𝑜, 𝜃𝑜)𝑑𝜃𝑜 (2-5) (2-6)

(27)

12

C. Formulation of a system of linear equations in the frequency domain

Without further assumptions, the solution to eq. (2-6) generally needs to be performed numerically using techniques such as singular value decomposition [22]. In this work, however, we introduce an efficient numerical computation by considering the physical properties of real scattering materials [28,31]. Many BRDFs can be approximated with two components: 1) a specular component originating from surface reflection off the scattering surface, and 2) a diffuse component formed from subsurface and/or internal reflections in the scattering material [12,13]. The specular component is often quasi-shift-invariant between the incoming and outgoing angles 𝜃𝑜 and 𝜃𝑚 [14,15], although a slow shift variance often remains in real BRDFs. For this reason, eq. (2-6) can be

expressed as a convolution-like integration and thus complex exponentials can serve as an approximate eigenbasis function of eq. (2-6) even though the BRDF is not perfectly shift-invariant.

The Hermitian L2_{inner product is defined in eq. (2-7) and the norm is defined as ‖𝑓(𝜃)‖}2_≡

〈𝑓(𝜃), 𝑓(𝜃)〉𝜃 in the normal manner. Similarly, the Fourier basis is defined within the angular domain [a, b] in

eq. (2-8), where 𝑎, 𝑏 ∈ (− 𝜋 2⁄ , 𝜋 2⁄ ) [28,29]. The asterisk denotes the complex conjugate. The basis set is orthonormal and the inner product of the two basis functions is expressed as the Kronecker delta function 〈𝜑𝑝(𝜃), 𝜑𝑞(𝜃)〉𝜃= 𝛿𝑝,𝑞. All of the functions in this chapter are assumed to be L2 functions and

(b-a)-periodically extended. 〈𝑓(𝜃), 𝑔(𝜃)〉𝜃≡ 1 𝑏 − 𝑎∫ 𝑓(𝜃)𝑔 ∗_{(𝜃)𝑑𝜃} 𝑏 𝑎 𝜑𝑝(𝜃) ≡ exp (𝑖 𝑝 2𝜋 𝑏 − 𝑎(𝜃 − 𝑎 + 𝑏 2 ))

In the following, one of independent integral equations at a single scattering point 𝑦𝑠= 𝑐𝑟 in eq.

(2-6) is chosen as an example. Since we are only considering scattering from this single point, 𝑐𝑟 is a constant and

(2-8) (2-7)

(28)

13

the spatial variable in the BRDF can be omitted. In addition, the surface normal angle at this scattering point, 𝛩𝑟,

is assumed to be already included in 𝜃𝑜 and 𝜃𝑚. Therefore, the equation reduces to eq. (2-9):

𝐿𝑚𝑒𝑎𝑠(𝜃𝑚) = ∫ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝐿𝑜𝑏𝑗(𝜃𝑜)𝑑𝜃𝑜 𝑏

𝑎

𝐿𝑜𝑏𝑗 and 𝐿𝑚𝑒𝑎𝑠 can be expressed as linear combinations of the basis functions. Below, 𝑂𝑞 denotes the

Fourier coefficient of 𝐿𝑜𝑏𝑗 and 𝑀𝑝 is that of 𝐿𝑚𝑒𝑎𝑠 (note: all Fourier coefficients are expressed as upper case

symbols.): { 𝑂𝑞= 〈𝐿𝑜𝑏𝑗(𝜃𝑜), 𝜑𝑞(𝜃𝑜)〉𝜃𝑜 𝐿𝑜𝑏𝑗(𝜃𝑜) = ∑ 𝑂𝑞𝜑𝑞(𝜃𝑜) ∞ 𝑞=−∞ { 𝑀𝑝= 〈𝐿𝑚𝑒𝑎𝑠(𝜃𝑚), 𝜑𝑝(𝜃𝑚)〉𝜃_𝑚 𝐿𝑚𝑒𝑎𝑠(𝜃𝑚) = ∑ 𝑀𝑝𝜑𝑝(𝜃𝑚) ∞ 𝑝=−∞

By substituting eq. (2-9) and (2-10) into eq. (2-11), we arrive at:

𝑀𝑝= 〈𝐿𝑚𝑒𝑎𝑠(𝜃𝑚), 𝜑𝑝(𝜃𝑚)〉𝜃𝑚 = 〈∫ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝐿𝑜𝑏𝑗(𝜃𝑜)𝑑𝜃𝑜 𝑏 𝑎 , 𝜑𝑝(𝜃𝑚)〉𝜃𝑚 = 〈∫ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜{ ∑ 𝑂𝑞𝜑𝑞(𝜃𝑜) ∞ 𝑞=−∞ } 𝑑𝜃𝑜 𝑏 𝑎 , 𝜑𝑝(𝜃𝑚)〉𝜃𝑚 = ∑ 𝑂𝑞 ∞ 𝑞=−∞ 1 𝑏 − 𝑎∬ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑜𝑑𝜃𝑚 𝑏 𝑎

By denoting the 2-D Fourier coefficient of 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) × cos 𝜃𝑜 as 𝐵𝑝,𝑞, the frequency-domain expression

of eq. (2-9) can be obtained below:

(2-10)

(2-12) (2-11) (2-9)

(29)

14 { 𝑀𝑝= ∑ 𝐵𝑝,𝑞𝑂𝑞 ∞ 𝑞=−∞ for any 𝑝 𝐵𝑝,𝑞 = 1 𝑏 − 𝑎∬ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑜𝑑𝜃𝑚 𝑏 𝑎

In the following section, the system matrix [𝐵] contains the elements 𝐵𝑝,𝑞, and the vectors 𝑂 and 𝑀

contain the element 𝑂𝑞 and 𝑀𝑝 respectively.

D. Procedure to solve the entire system equation in the frequency domain

In this section, we show a simulation of the solution to the entire system equation given in eq. (2-6) using the frequency domain formulation in eq. (2-13). The assumed object light field 𝐿𝑜𝑏𝑗 is shown in fig. 2-3

(a) as consisting of a central region (𝑦𝑜≅ 0 ) that contains only large angle rays and edges that contain a complete

cone of rays. This somewhat unusual light field is chosen here primarily for demonstration purposes. The assumed BRDF is illustrated in fig. 2-3 (b) showing scattering distributions from three different object angles. 𝐿𝑚𝑒𝑎𝑠 in

fig. 2-3 (c) is computed as a forward problem by numerically integrating eq. (2-6).

Fig. 2-3. Conditions for the simulation. (a) Assumed 𝑳𝒐𝒃𝒋 . (b) Assumed

homogeneous BRDF. (c) Numerically computed 𝑳𝒎𝒆𝒂𝒔

Fig. 2-4 shows the three-step procedure to solve the entire system equation in the frequency domain and recover the original object light field from its measured components. First step: Sample the measured light field 𝐿𝑚𝑒𝑎𝑠 along a curve defined in eq. (2-4). Second step: Calculate the Fourier transform of the sampled 1-D

data to produce vector 𝑀, and solve eq. (2-13) to obtain vector 𝑂. Third step: Calculate the inverse Fourier

(30)

15

transform of the vector 𝑂 and align the results on a curve defined in eq. (2-5). Note that a single scattering point 𝑦𝑠 = 𝑐𝑟 gives rise to a single curve in the light field planes. By repeating the three steps described above at many

different points 𝑦𝑠 = 𝑐𝑟 for 𝑟 = 0, 1, 2 … on the scattering surface, the entire two-dimensional light field 𝐿𝑜𝑏𝑗

can be reconstructed.

Fig. 2-4. Reconstruction procedure

Fig. 2-5. Reconstructed 𝑳𝒐𝒃𝒋 in the presence of measurement noise in 𝑳𝒎𝒆𝒂𝒔 with

three differently truncated matrices [𝑩]: (a) truncated at 12th_{term, (b) truncated at}

(31)

16

The system matrix [𝐵] in eq. (2-13) is ill-conditioned and 𝐿𝑜𝑏𝑗 is generally distorted due to

reconstruction error from noise in 𝐿𝑚𝑒𝑎𝑠. Fig. 2-5 shows simulated reconstructions of the object light field by

truncating the matrix [𝐵] at various Fourier terms assuming a given amount of measurement noise. We develop analytical expressions to describe their behavior and investigate two fundamental regularization methods in the second half of this chapter.

2.4 Analytical closed-form solution and analysis of reconstruction error

In this section, we derive an analytical solution of eq. (2-9) by introducing a simple BRDF model. Reconstruction mechanisms and related errors due to measurement noise are considered. We simplify the mathematics by considering only one point on the scattering surface and omitting the scattering surface spatial variables.

A. A shift-variant BRDF model and analytical solution

Previous authors in the computer graphics and computer vision fields have made a variety of simplifying assumptions to BRDFs [12,13]. For example, the BRDF has been approximated as shift-invariant to compute the forward scattering problem in the frequency domain [16] and the diffuse component of the BRDF has been approximated as a constant (commonly known as the Lambertian term) to obtain solutions to the inverse scattering problem [4]. In addition, the BRDF has been modeled as separable, such as for data compression purposes [17,18]. These assumptions, although convenient for computer graphic related purposes, are not suitable for analysis of the reconstruction.

Fig. 2-6 illustrates two types of BRDFs expected in real scenes. Fig. 2-6 (a) shows a large and slowly varying diffuse component superimposed on a small quasi-shift-invariant specular component. Fig. 2-6 (b) shows a BRDF that contains only a quasi-shift-invariant specular component, but is restricted to a limited angular range. Since the specular component is considerably wider than the measurement range, the boxed area can be considered

(32)

17

as a diffuse component. In the following, we introduce a BRDF model in eq. (2-14) that covers these common shift-variant BRDFs, enabling an analytical solution to eq. (2-9). Such analytical solutions are required to fully understand the errors in reconstruction and the effect of regularization methods. Note that our BRDF model combines Durand’s model [16] for the forward scattering analysis and Shirley's model [19] for matte/specular materials such as polished woods and tiles. Our model is thus expected to have wide applicability to real-world scattering surfaces.

Fig. 2-6. Analytically modeling two candidate BRDF functions. (a) Large diffuse component and small specular component. (b) Wide specular component with limited measurable range.

Fig. 2-6 (a) is modeled mathematically by including a shift-invariant specular term 𝑓𝑠𝑝𝑒𝑐 that is

multiplied by a slowly varying window function 𝑓𝑤𝑖𝑛𝑑 to allow for commonly observed changes in specular

peak-values as a function of object angle 𝜃𝑜 [16]. A symmetric diffuse component 𝑓𝑑𝑖𝑓𝑓(𝜃𝑜) × 𝑓𝑑𝑖𝑓𝑓(𝜃𝑚) is

then added that is separable and reciprocal in terms of the input and output angles [19]:

𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) = 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜)𝑓𝑠𝑝𝑒𝑐(𝜃𝑚− 𝜃𝑜) + 𝑓𝑑𝑖𝑓𝑓(𝜃𝑜)𝑓𝑑𝑖𝑓𝑓(𝜃𝑚)

In the computer graphics and computer vision fields, the diffuse component is usually expressed as a constant Lambertian term. In our treatment, we allow the diffuse component to take on any functional form, as long as the function is separable with respect to the input and output angles. This generalization allows for a far more complex diffuse term to be expressed. However, the separability constraint implies that no object information is

(33)

18

conveyed to the measured signal as explained in the subsequent sections. Moreover, if 𝑓𝑠𝑝𝑒𝑐 is sufficiently

narrow compared to 𝑓𝑤𝑖𝑛𝑑 (i.e. 𝑓𝑠𝑝𝑒𝑐 is considered to be delta-function like), then 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜) × 𝑓𝑠𝑝𝑒𝑐(𝜃𝑚−

𝜃𝑜) ≅ 𝑓𝑤𝑖𝑛𝑑(𝜃𝑚) × 𝑓𝑠𝑝𝑒𝑐(𝜃𝑜− 𝜃𝑚) meaning the model has Helmholtz reciprocity considering the

symmetricity of the specular and diffuse component.

We first introduce a weighted inner product [29] (denoted by a double bracket in eq. (2-15)) and weighted basis (denoted by a tilde in eq. (2-16)) to simplify the frequency-domain expression of eq. (2-9). This basis is orthonormal with respect to the weighted inner product, so 𝐿𝑜𝑏𝑗 can be expressed as a linear combination

of 𝜑̃𝑞 as shown in eq. (2-17), where 𝑂̃𝑞 is a coefficient of 𝐿𝑜𝑏𝑗 with respect to 𝜑̃𝑞. (The relation between 𝑂𝑞

and 𝑂̃𝑞 is explained in appendix A1.)

〈〈𝑓(𝜃), 𝑔(𝜃)〉〉𝜃≡ 1 𝑏 − 𝑎∫ 𝑓(𝜃)𝑔 ∗_{(𝜃) cos}2_{(𝜃) 𝑑𝜃} 𝑏 𝑎 𝜑̃𝑞(𝜃) ≡ 𝜑𝑞(𝜃) 𝑐𝑜𝑠 𝜃 { 𝑂̃𝑞= 〈〈𝐿𝑜𝑏𝑗(𝜃𝑜) , 𝜑̃𝑞(𝜃𝑜)〉〉𝜃_𝑜 𝐿𝑜𝑏𝑗(𝜃𝑜) = ∑ 𝑂̃𝑞𝜑̃𝑞(𝜃𝑜) ∞ 𝑞=−∞

By substituting eq. (2-9) and (2-17) into (2-11) and referring to eq. (2-12), a new frequency domain expression can be obtained in eq. (2-18). Note that the cosine obliquity factor has been cancelled and 𝐵̃𝑝,𝑞 is simply the 2-D

Fourier coefficient of the BRDF. In the following, the system matrix [𝐵̃] contains the elements 𝐵̃𝑝,𝑞.

{ 𝑀𝑝= ∑ 𝐵̃𝑝,𝑞𝑂̃𝑞 ∞ 𝑞=−∞ for any 𝑝 𝐵̃𝑝,𝑞= 1 𝑏 − 𝑎∬ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚)𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑜𝑑𝜃𝑚 𝑏 𝑎

By substituting the BRDF model of eq. (2-14) into eq. (2-18), the following expression for 𝐵̃𝑝,𝑞 is obtained:

(2-15)

(2-16)

(2-17)

(34)

19 𝐵̃𝑝,𝑞 = 1 𝑏 − 𝑎∬ 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜)𝑓𝑠𝑝𝑒𝑐(𝜃𝑚− 𝜃𝑜)𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑜𝑑𝜃𝑚 𝑏 𝑎 + 1 𝑏 − 𝑎∬ 𝑓𝑑𝑖𝑓𝑓(𝜃𝑜)𝑓𝑑𝑖𝑓𝑓(𝜃𝑚)𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑜𝑑𝜃𝑚 𝑏 𝑎

By applying a change of variable 𝜃𝑚− 𝜃𝑜= 𝜁 to the first term of eq. (2-19) and considering the (b-a)-pe

riodicity of the functions, eq. (2-20) is obtained, where 𝑆𝑝,𝑝 is the Fourier coefficient of 𝑓𝑠𝑝𝑒𝑐 defined in eq.

(2-21), and 𝑊𝑝−𝑞 is the Fourier coefficient of 𝑓𝑤𝑖𝑛𝑑 defined in eq. (2-22). (Note: 𝜑𝑞(𝜃𝑜)𝜑𝑝∗(𝜃𝑜) = 𝜑𝑝−𝑞∗ (𝜃𝑜).)

First term = ∫ 𝑓𝑠𝑝𝑒𝑐(𝜁) exp (−𝑖 𝑝

2𝜋 𝑏 − 𝑎𝜁) 𝑑𝜁 𝑏 𝑎 1 𝑏 − 𝑎∫ 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜)𝜑𝑞(𝜃𝑜)𝜑𝑝 ∗_(𝜃 𝑜)𝑑𝜃𝑜 𝑏 𝑎 = 𝑆𝑝,𝑝𝑊𝑝−𝑞 𝑆𝑝,𝑝≡ ∫ 𝑓𝑠𝑝𝑒𝑐(𝜁) exp (−𝑖 𝑝 2𝜋 𝑏 − 𝑎𝜁) 𝑑𝜁 𝑏 𝑎 𝑊𝑝−𝑞 ≡ 1 𝑏 − 𝑎∫ 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜)𝜑𝑝−𝑞 ∗ _(𝜃 𝑜)𝑑𝜃𝑜 𝑏 𝑎

Because we have assumed that the diffuse component is separable, the second term of eq. (2-19) can be written as eq. (2-23), where 𝐷𝑝 is the Fourier coefficient of 𝑓𝑑𝑖𝑓𝑓 defined in eq. (2-24).

Second term = 1 √𝑏 − 𝑎∫ 𝑓𝑑𝑖𝑓𝑓(𝜃𝑜)𝜑𝑞(𝜃𝑜)𝑑𝜃𝑜 𝑏 𝑎 × 1 √𝑏 − 𝑎∫ 𝑓𝑑𝑖𝑓𝑓(𝜃𝑚)𝜑𝑝 ∗_(𝜃 𝑚)𝑑𝜃𝑚 𝑏 𝑎 = 𝐷𝑝𝐷𝑞∗ 𝐷𝑝≡ 1 √𝑏 − 𝑎∫ 𝑓𝑑𝑖𝑓𝑓(𝜃)𝜑𝑝 ∗_{(𝜃)𝑑𝜃} 𝑏 𝑎

Finally, by substituting eq. (2-20) and (2-23) into (2-19), 𝐵̃𝑝,𝑞 is expressed in the following simple form:

𝐵̃𝑝,𝑞= 𝑆𝑝,𝑝𝑊𝑝−𝑞+ 𝐷𝑝𝐷𝑞∗

Recall that we have assumed that 𝑓𝑤𝑖𝑛𝑑 is a slowly varying function. To make further progress in the analytical

solution of eq. (2-18), we approximate it as 𝑓𝑤𝑖𝑛𝑑(𝜃𝑜) ≅ 1. We then have

(2-20) (2-21) (2-22) (2-23) (2-24) (2-25) (2-19)

(35)

20 𝑊𝑝−𝑞≅ 1 𝑏 − 𝑎∫ 𝜑𝑝−𝑞 ∗ _(𝜃 𝑜)𝑑𝜃𝑜 𝑏 𝑎 = 〈𝜑𝑞(𝜃𝑜), 𝜑𝑝(𝜃𝑜)〉𝜃𝑜= 𝛿𝑝𝑞

By replacing 𝑊𝑝−𝑞 with 𝛿𝑝𝑞 in eq. (2-25), the following approximate expression is obtained (Note: We will

return to the expression 𝑊𝑝−𝑞 in the next section.):

𝐵̃𝑝,𝑞= 𝑆𝑝,𝑝𝛿𝑝𝑞+ 𝐷𝑝𝐷𝑞∗

By substituting eq. (2-27) into (2-18), and dividing both sides by 𝑆𝑝,𝑝, we have

𝑂̃𝑝= 𝑀𝑝 𝑆𝑝,𝑝 − ∑ 𝐷𝑝 𝑆𝑝,𝑝 𝐷𝑞∗𝑂̃𝑞 ∞ 𝑞=−∞

Eq. (2-28) is a discrete form of the Fredholm integral equation of the second kind [30], and it has a closed-form solution because of the separable kernel in the discrete sum. By expressing eq. (2-28) with a constant 𝛼 defined in eq. (2-29) and multiplying both sides by 𝐷𝑝∗, eq. (2-30) is obtained.

𝛼 ≡ ∑ 𝐷𝑞∗𝑂̃𝑞 ∞ 𝑞=−∞ 𝐷𝑝∗𝑂̃𝑝= 𝐷𝑝∗𝑀𝑝 𝑆𝑝,𝑝 −|𝐷𝑝| 2 𝑆𝑝,𝑝 𝛼

By calculating the discrete sum of p for both sides of eq. (2-30), the following is obtained:

∑ 𝐷𝑝∗𝑂̃𝑝 ∞ 𝑝=−∞ = ∑ 𝐷𝑝 ∗_𝑀 𝑝 𝑆𝑝,𝑝 ∞ 𝑝=−∞ − 𝛼 ∑ |𝐷𝑝| 2 𝑆𝑝,𝑝 ∞ 𝑝=−∞

The left-hand-side of eq. (2-31) is equal to 𝛼, and therefore this equation can be solved in terms of 𝛼 as expressed in eq. (2-32). The expression for 𝛼 does not include unknown 𝑂̃𝑝, and therefore, an analytical closed-form

solution can be obtained in eq. (2-33) from eq. (2-28), (2-29) and (2-32).

(2-27) (2-28) (2-29) (2-30) (2-31) (2-26)

(36)

21 𝛼 = ∑ 𝐷𝑝 ∗ 𝑆𝑝,𝑝𝑀𝑝 ∞ 𝑝=−∞ 1 + ∑ |𝐷𝑝| 2 𝑆𝑝,𝑝 ∞ 𝑝=−∞ 𝑂̃𝑝= 1 𝑆𝑝,𝑝 (𝑀𝑝− 𝛼𝐷𝑝)

B. Understanding the structure of the system matrix [𝑩]

From the analytical expressions in the previous section, it is possible to interpret the structure and eigenvalues of the system matrix [𝐵] in eq. (2-13). From the relation between 𝑂𝑞 and 𝑂̃𝑞 in eq. (a3) in

appendix A1, 𝐵𝑝,𝑞 in eq. (2-13) and 𝐵̃𝑝,𝑞 in eq. (2-18) have the following relation: 𝐵𝑝,𝑞≅ 𝐵̃𝑝,𝑞⁄ , where 𝛽𝛽0 0

is a constant related to the cosine obliquity factor and is given by eq. (a2) in appendix A1, and therefore, the structure of [𝐵] can be directly seen from [𝐵̃].

Let us first consider the simplest case where 𝑓𝑤𝑖𝑛𝑑 is unity. Fig. 2-7 (a) illustrates 𝐵̃𝑝,𝑞 in eq. (2-27)

where it can be seen that the diagonal elements of [𝐵̃] are 𝑆𝑝,𝑝, or the Fourier coefficients of 𝑓𝑠𝑝𝑒𝑐 in eq. (2-21),

and the off-diagonal elements are 𝐷𝑝𝐷𝑞∗ or the outer product of the Fourier coefficients of 𝑓𝑑𝑖𝑓𝑓 in eq. (2-24).

Moreover, it can be understood from eq. (2-33) that 𝑆𝑝,𝑝 can be considered as the approximate eigenvalues of

[𝐵̃] because the right-hand-side of eq. (2-33) includes division by 𝑆𝑝,𝑝. As is commonly known, many actual

BRFDs contain specular components whose shape can be approximated by a Gaussian function and whose Fourier coefficients monotonically decrease as a function of frequency [12,13], and therefore, the diagonal elements tend to have near zero values at higher frequencies. The extent of the non-zero portion of these diagonal elements of the matrix indicates the range of angular frequencies that can be recovered without being adversely affected by measurement noise. The length of this recoverable region is Fourier-transform related to the sharpness of the specular component (see fig. 2-7 (b)).

(2-33) (2-32)

(37)

22

Second, let us consider a more general BRDF where 𝑓𝑤𝑖𝑛𝑑 is not unity but rather a slowly varying

function. From eq. (2-25), the diagonal elements 𝑆𝑝,𝑝 are seen to be convolved with 𝑊𝑝−𝑞, leading to a blur in

the diagonal elements. Fig. 2-7 (b) shows an experimentally measured system matrix [𝐵] from a coated white paper. The width of the blur in this figure is a measure of the shift-variance of the specular component. In many common scattering materials where 𝑓𝑤𝑖𝑛𝑑 is slowly varying and the width of this blur is small, the 𝑆𝑝,𝑝 terms

are still good approximations to the actual eigenvalues.

Fig. 2-7. Understanding the structure of the system matrix [𝑩] . (a) Schematic structure of [𝑩̃]. (b) Experimentally obtained [𝑩].

C. Influence of measurement noise on reconstruction error

In many inverse problem solutions, the relative reconstruction error can be expressed as the product of the system matrix condition number and the inverse of the measurement system signal-to-noise ratio (SNR) [25,32]. In this section, we will derive a similar expression by making use of the analytical solution of the previous section together with a newly defined BRDF parameter which we call the degree of specularity.

We assume the linear noise model shown in eq. (2-34) as a description of the signal 𝐿𝑚𝑒𝑎𝑠 detected

by the camera. The measurement noise 𝑋𝜃𝑚 is assumed to be an additive zero-mean wide-sense stationary (WSS)

(38)

23

𝐿𝑚𝑒𝑎𝑠(𝜃𝑚) = ∫ 𝑓𝐵𝑅𝐷𝐹(𝜃𝑜, 𝜃𝑚) cos 𝜃𝑜𝐿𝑜𝑏𝑗(𝜃𝑜)𝑑𝜃𝑜 𝑏

𝑎

+ 𝑋𝜃𝑚

Since the process is WSS, the noise can be characterized by random variables 𝐴𝑝 that are the Fourier coefficients

(or more generally the Karhunen-Loéve coefficients [33,35]) of 𝑋𝜃𝑚 with respect to eq. (2-7) and (2-8). When

taking the measurement noise into consideration, eq. (2-11) becomes

𝐿𝑚𝑒𝑎𝑠(𝜃𝑚) − 𝑋𝜃_𝑚= ∑ {𝑀𝑝− 𝐴𝑝}𝜑𝑝(𝜃𝑚) ∞

𝑝=−∞

Therefore, an analytical solution considering the measurement noise, denoted 𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒, can be obtained in eq.

(2-35) simply by replacing 𝑀𝑝 with 𝑀𝑝− 𝐴𝑝 from eq. (2-33), where 𝑒̃𝑝 is an error caused by the noise.

𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒= 𝑂̃𝑝+ 𝑒̃𝑝, where 𝑒̃𝑝 ≡ −

𝐴𝑝

𝑆𝑝,𝑝

We will now derive the relative noise error ∆̃𝑝 as defined below, where 𝑉𝑎𝑟 means variance:

∆̃𝑝≡

√𝑉𝑎𝑟[𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒]

|𝑂̃0|

By using eq. (2-35) and the linearity of the expectation operator 𝐸, the variance of 𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒 can be obtained as

shown below, where the pth_{coefficient of the power spectral density of the measurement noise is denoted as}_𝑃𝑆𝐷 𝑝

(note: for definition, see eq. (a15) in appendix A7. In addition, 𝐸[𝐴𝑝] = 0 and 𝐸 [|𝐴𝑝| 2 ] = 𝑃𝑆𝐷𝑝 from [33,35]). 𝑉𝑎𝑟[𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒] = 𝐸 [|𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒| 2 ] − |𝐸[𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒]| 2 = 𝐸[(𝑂̃𝑝+ 𝑒̃𝑝)(𝑂̃𝑝+ 𝑒̃𝑝) ∗ ] − (𝐸[𝑂̃𝑝+ 𝑒̃𝑝])(𝐸[𝑂̃𝑝+ 𝑒̃𝑝]) ∗ = 𝐸 [|𝑒̃𝑝| 2 ] = 𝑃𝑆𝐷𝑝 |𝑆𝑝,𝑝| 2 (2-35) (2-36) (2-34)

(39)

24

where the constant 𝛾 is defined in eq. (2-38) as the degree of specularity:

𝛾 ≡ 1 + ∑ 𝐷𝑝 ∗ 𝑆𝑝,𝑝(𝐷𝑝− 𝐷0 𝑀𝑝 𝑀0) ∞ 𝑝=−∞ 1 + ∑ |𝐷𝑝| 2 𝑆𝑝,𝑝 ∞ 𝑝=−∞ ≅ 1 1 + ∑ |𝐷𝑝| 2 𝑆𝑝,𝑝 ∞ 𝑝=−∞

The approximation in eq. (2-38) comes from the fact that value of the second term in the numerator can often be approximated as near zero (for further details, see appendix A2). The approximated 𝛾 is totally determined by the BRDF, and has the following properties; it becomes 1 for perfectly specular BRDF i.e. 𝐷𝑝 = 0 for all 𝑝, and

becomes 0 for perfectly diffusive BRDF i.e. 𝑆𝑝,𝑝= 0 for all 𝑝. In section 2.6, it will be experimentally shown

that eq. (2-38) can accurately represent the characteristic of BRDFs from a variety of scattering surfaces.

If the noise 𝑋𝜃𝑚 is white and Poissonian and the mean of the measured signal is sufficiently large, its

statistics can be approximated by a white-Gaussian noise with a standard deviation which is a function (generally, the square root) of the signal mean. By defining the standard deviation of 𝑋𝜃𝑚 as 𝜎(𝑀0), where 𝑀0 (average

value of 𝐿𝑚𝑒𝑎𝑠 from eq. (2-11)) represents the signal mean, we can make the following approximation:

√𝑃𝑆𝐷𝑝≅ 𝜎(𝑀0). (Note: For satisfying Parseval’s law, 𝜎(𝑀0) needs to be divided by √𝑁 when an 𝑁-point

DFT is taken for computation.)

D. Intuitive understanding of the reconstruction error

An inspection of eq. (2-37) reveals the following;

(2-37)

(40)

25

1) The first term |𝑆0,0⁄𝑆𝑝,𝑝| in eq. (2-37) can be interpreted as the condition number [25,32] of a matrix

obtained by totally removing the diffuse component from the system matrix [𝐵], assuming 𝑆𝑝,𝑝 is a

monotonically decreasing function of 𝑝 and truncating the matrix to include all terms less than or equal to 𝑝. This matches our intuition that BRDFs having sharper and/or narrower specular peaks will retain higher frequency components in the object reconstruction. By contrast, BRDFs having more rounded and/or wider specular peaks will lose higher object frequency information. A similar effect was noted in fig. 2-7. In addition, the above-mentioned specular width is measured relative to the domain range [𝑎, 𝑏]. For example, as the range becomes wider, the relative specular width can be seen to be narrower.

2) Instead of the more common definition of the signal-to-noise ratio (or SNR) in eq. (2-39), the second term in eq. (2-37) containing 𝛾 acts as the effective signal-to-noise ratio. We define this in eq. (2-40) as 𝑆𝑁𝑅𝑒𝑓𝑓.

𝑆𝑁𝑅 = 𝑀0 √𝑃𝑆𝐷𝑝

𝑆𝑁𝑅𝑒𝑓𝑓≡

𝑀0𝛾

√𝑃𝑆𝐷𝑝

The numerator 𝑀0𝛾 represents the part of 𝐿𝑚𝑒𝑎𝑠 originating from the specular component of the BRDF.

This indicates that BRDFs having larger diffuse components result in smaller values of 𝛾 and 𝑆𝑁𝑅𝑒𝑓𝑓,

making the object reconstruction more difficult. In the case of a perfectly diffusive BRDF (𝛾 = 0), the 𝑆𝑁𝑅𝑒𝑓𝑓 becomes zero, meaning no object information can be retrieved. (Note: We defined the diffuse

component as a separable function, and in general, any non-separable component of the BRDFs can convey object information.) In addition, the 𝛾 coefficient depends on the domain range [𝑎, 𝑏]. For example, if the measurable range in fig. 2-6 (b) becomes wider, 𝛾 becomes larger.

(2-39)

(41)

26

For categorizing different types of BRDFs from the viewpoint of solving the inverse problem, the two parameters, |𝑆0,0⁄𝑆𝑝,𝑝| and 𝑆𝑁𝑅𝑒𝑓𝑓, are informative because they directly reflect the reconstruction error which determines

the quality of the reconstructed light field 𝐿𝑜𝑏𝑗.

E. Examples of noise analysis on wavelength and polarization

Observing the scene with longer wavelength light, such as infrared and terahertz wavelengths, can sometimes lead to a BRDF with a narrower specular peak [37]. This reduces |𝑆0,0⁄𝑆𝑝,𝑝| in eq. (2-37) meaning

the error is suppressed and higher frequency object information can be preserved.

Moreover, the specular and diffuse reflections each have polarization properties governed by the characteristics of the scattering surface. The 𝑆𝑁𝑅𝑒𝑓𝑓 can be taken to determine suitable usage of

polarizers for better light field reconstruction. If we assume the simplest situation, such as P-polarized specular reflection and un-polarized Lambertian diffuse, it can be shown that the highest 𝑆𝑁𝑅𝑒𝑓𝑓 can

be achieved simply by using a P-polarizer during measurement. Some imaging systems attempt to remove the diffuse component by subtracting a second measurement with S-polarization from that with P-polarization [38]. However, this procedure does not improve the 𝑆𝑁𝑅𝑒𝑓𝑓 because the subtraction adds

more noise. See appendix A3 for details.

2.5 Two fundamental regularization methods derived from the analytical

solution

A. Fourier truncation regularization

As explained in section 2.4.B, the Fourier coefficients 𝑆𝑝,𝑝 of the specular component serve as

approximate eigenvalues of the system matrix [𝐵] in eq. (2-13). Thus, removing small values of 𝑆𝑝,𝑝 by

(42)

27

this is similar to the commonly known truncation procedure for Fourier series when performing deconvolution (or inverse filtering) and truncation of the SVD when performing general inversions [22,26]. We previously derived an expression for the relative noise error in eq. (2-37). However, this expression is not appropriate for the present purpose since it lacks the error inherent in a truncated Fourier series [29]. Instead, we will derive a general error expression for the mean-squared error (MSE) to include both errors, and then use the first derivative of this MSE with respect to Fourier terms p as a criterion for selecting the best truncation point for regularization.

- Total MSE expressed as a sum of the truncation error and the noise error

𝐿𝑜𝑏𝑗 is the original object and 𝐿𝑒𝑟𝑟𝑜𝑏𝑗 is the reconstructed object defined in eq. (2-42) to include both

the measurement noise error and the error from truncating the Fourier series at ± 𝑝𝑚𝑎𝑥 (note: 𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒 is from

eq. (2-35), and 𝛽0 is a constant given by eq. (a2) in appendix A1).

𝑀𝑆𝐸 ≡ 𝐸 [‖𝐿𝑜𝑏𝑗(𝜃𝑜) − 𝐿𝑒𝑟𝑟𝑜𝑏𝑗(𝜃𝑜)‖ 2 ] 𝐿𝑒𝑟𝑟𝑜𝑏𝑗(𝜃𝑜) ≡ ∑ 𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒𝜑̃𝑝(𝜃𝑜) 𝑝_𝑚𝑎𝑥 𝑝=−𝑝𝑚𝑎𝑥 ≅ ∑ 𝛽0𝑂̃𝑝𝑛𝑜𝑖𝑠𝑒𝜑𝑝(𝜃𝑜) 𝑝_𝑚𝑎𝑥 𝑝=−𝑝𝑚𝑎𝑥

By expanding eq. (2-41) (see appendix A4 for details), the total MSE can be obtained in eq. (2-43) as a function of the truncation term 𝑝𝑚𝑎𝑥. The first and second terms represent the truncation error, and the third term represents

the measurement noise error.

𝑀𝑆𝐸 ≅ ‖𝐿𝑜𝑏𝑗(𝜃𝑜)‖ 2 − 𝛽02 ∑ |𝑂̃𝑝| 2 𝑝𝑚𝑎𝑥 𝑝=−𝑝𝑚𝑎𝑥 + 𝛽02 ∑ 𝐸 [|𝑒̃𝑝| 2 ] 𝑝𝑚𝑎𝑥 𝑝=−𝑝𝑚𝑎𝑥

- The first derivative of MSE as a criterion to determine the best truncation term for regularization (2-41)

(2-42)

Information Extraction from Non-Line-of-Sight Objects. Using Plenoptic Data from Scattered Light A DISSERTATION SUBMITTED TO THE FACULTY