In order to increase the resolution of multi-camera, multi-projector structured light systems, we directly exploit the information contained in the overlaps of different pattern projections on the surface. Therefore, we have to consider a novel, modified encoding to generate super-resolution codewords (see Section3.3.1) and also the respective decoding of these observed super-resolution codewords (see Section
3.3.2).
3.3.1
Super-Resolution Labeling
For the sets of projectors P, cameras C and Gray code patterns G, let I = {Ic,p,g|(c, p, g) ∈ C × P × G} be the captured images. Thus, Ic,p,g ∈ I is an
image (with domain Ωc ⊂ R2) captured by camera c, illuminated by projector p
3.3. MULTI-PROJECTORSUPER-RESOLUTION
By combining the |G| different pattern images for each pair of camera c and projector p, the Gray code can be decoded to labels l ∈ [0, . . . , 2|G|]. This defines a function χc,p : Ωc → Lc,p, assigning a projector-dependent label l ∈ Lc,p to
every image point (x, y) ∈ Ωc. Here, Lc,p ⊂ [0, . . . , 2|G|] is the set of labels that
were actually decoded for camera c under illumination p. By inverting the labeling function χc,p, we can identify the image regions Rc,p,l ⊂ Ωc, which have been
labeled with l.
The super-resolution labeling function is defined as
ξc: Ωc → Lc
(x, y) 7→ (χc,p1(x, y), . . . , χc,p|P|(x, y)) (3.1) assigning to every image point (x, y) a projector-independent label
L = (L1, . . . , L|P|) ∈ Lc ⊂ Lc,1× Lc,2× . . . × Lc,|P| (3.2)
with Li = χc,pi(x, y). L can be understood as the tuple concatenating the labels of the individual projectors. Analogical to the projector-dependent label, we can invert ξ and get the image regions Rc,L⊂ Ωcwhich have been labeled with L.
As shown in Figure3.1, the region Rc,Lis the intersection
T|P|
i=1Rc,pi,Li, resulting in a usually more precise localization in the image for the super-resolution labels when compared to the original labels.
Figure 3.1: Super-resolution scheme for structured light: By combining the two setsL1(red) andL2(green) of labels obtained from two independent projectors into
super-resolution labels L, much smaller regions on the surface can be identified uniquely. For example, the highlighted blue region R(4,4) = R1,4∩ R2,4 is much
smaller than both of the intersected regions.
To increase clarity and shorten the notation, we will dismiss the camera and projector indices in the remainder of this chapter, unless they are needed. Please
keep in mind that the super-resolution labels are always defined per camera and the projector dependent labels per pair of camera and projector.
3.3.2
Robust Matching of Codewords
For a robust matching, we extend the codomain of the projector-dependent labeling function χ to L0 = L ∪ {⊥}, including ⊥ to express undefined. The symbol ⊥ is used for all image points (x, y) for which the label could not be decoded, either because they were not lit by the projector due to occlusions by the geometry or because they could not be classified reliably as shown in Figure3.2.
1
0
?
>
<
≈
1 0 ᴛFigure 3.2: Illustration of the derivation of structured light encodings: If a pixel value is significantly brighter under illumination with a certain stripe pattern than the respective value under illumination by the corresponding inverse structured light pattern (marked in green), the respective bit in the encoding is set to1. In contrast, when the pixel value is darker than its corresponding value observed under illumination by the inverse pattern (marked in red), the bit in the encoding is set to0. If the values under the structured light pattern and its inverse are close to each other, no reliable decision can be made for the respective bit in the decoding (marked in blue) which we mark by using⊥ in the encoding.
As a consequence, the super-resolution labels are extended as well to L0, with the tuple L ∈ L0 containing ⊥ at entries where the label for the corresponding projector could not be decoded. Please note that this will introduce a new inverse
3.4. ITERATEDROBUSTBUNDLEADJUSTMENT
labeling function ξ0−1 : L0 → Ω
L 7→ {(x, y) ∈ Ω|∀i ∈ [1, . . . , |P|] : Li = ⊥ ∨ Li = χpi(x, y)} (3.3) for a super-resolution label L = (L1, L2, . . . , L|P|). This means that R0Lis in fact
the intersection of the regions of all projections i for which a sub-label Li could
reliably be identified, ignoring the indeterminable region of ⊥ in projections with undefined entries.
It might occur that a normal label l projected by projector pk, which was correctly
decoded in camera ci, was not reliably classified in camera cj. Therefore, all super-
resolution labels {(L1, L2, . . . , L|P|)|Lk = l} ⊂ L0ci of camera cithat contain l at the kthindex cannot be found in L0
cj. However, if the labels in all other projections were classified correctly in both cameras, we would still like to use this partial match, providing us with a reasonably precise regionT
i6=kRpi (see Figure3.3for an example). Thus, we introduce a partial matching function PM
c , that will robustly
match any super-resolution label in any camera with all super-resolution labels detected in camera c. PcM : [ c0∈C L0c0 → 2L 0 c (L1, . . . , L|P|) 7→ {(M1, . . . , M|P|) ∈ L0c| ∀i : Li = Mi∨ Li = ⊥} (3.4)
Consider a super-resolution label L ∈ L0 that was incompletely identified in camera cj and the matched set of super-resolution labels PcMi (L) in camera ci. The region in ci corresponding to the region of L in cj is then determined by
S
M ∈PM ci(L)ξ
0−1(M ). For a sketch we refer to Figure3.3. Please note that even
though the position of a region belonging to a label is the actual information needed in the multi-view-triangulation, its size also yields important information about the accuracy, which will be used in the bundle adjustment (see Section3.4) as well as the surface reconstruction (Section3.5). This fact is especially important in our case, since we perform a triangulation with super-resolution labels of very different sizes and therefore different accuracies.