Image clean-up

The encoding pattern from the previous section was projected onto the object and a camera used to capture the coloured light reflections. The Bayer filter over the CCD separates the light into red, green and blue sections. Due to the nature of these Bayer filters the light is however, not separated perfectly and each colour channel is influenced by light reflections from all the other channels. This phenomenon is known as CCD colour cross-talk.

The colour detection was further complicated by the fact that each person’s skin reflects light in a unique spectrum. Some skin types are more absorbent and have darker appearance while other skin types reflect more light and appear lighter. In addition to this, every skin type has a unique reflection pattern across the colour spectrum and will reflect colours differently. In order to correct for this, a two-fold approach was followed.

Firstly, to correct for cross-talk, Caspi et al. (1998) proposed a model that relates the camera image colours 𝑆 to the projected colours 𝑃:

[𝑅

where 𝐴 is the camera-projector colour cross-talk matrix, 𝐾 is known as the albedo matrix for a point on the object and 𝑂 represents the ambient light present at that same point. The cross-talk matrix can be determined by projecting a red, blue and green scene onto a white surface in zero ambient light conditions and capturing three camera images. The average of the RGB values is used to construct 𝐴. Matrix 𝐾 can be determined by projecting uniform white illumination onto the object. Colour-corrected camera image values can be obtained by multiplying each camera colour channel with the inverse of the cross-talk matrix, when the ambient light is assumed to be negligible:

Ś = 𝐴⁻¹𝑆 = (𝑘_𝑅𝑟 𝑘_𝐺𝑔

𝑘_𝐵𝑏) + 𝐴⁻¹𝑂 (5)

The above method requires several reference images for each object scene. In order to improve on this, several experiments with different object surfaces were done to develop a better colour correction strategy. Our newly developed method requires no reference images and provides a simplified, yet powerful colour correction that automatically takes the ambient lighting conditions into account.

Colour correction is done by taking colour samples of red, green and blue sections in the captured image of the object under illumination. Each sample is representative of the respective colour and compared to the known projected values. For example, the red sample might look like this: (R,G,B) = (0.745, 0.115, 0.077), and when compared to the projected image, which we know is (R,G,B) = (1, 0, 0), it is clear that the green and blue channels contained noise due to cross-talk, ambient lighting and object reflectance. Note that this approach only works because the object under illumination, i.e. the human skin, is assumed to have relative uniform reflectance. The three samples are used to construct a simplified

“cross-talk” matrix by taking the channel averages with the known projected values:

𝐴_{𝑠𝑖𝑚𝑝𝑙𝑖𝑓𝑖𝑒𝑑} = [

𝑅₁^𝑎𝑣𝑔 𝑅₂^𝑎𝑣𝑔 𝑅₃^𝑎𝑣𝑔 𝐺₁^𝑎𝑣𝑔 𝐺₂^𝑎𝑣𝑔 𝐺₃^𝑎𝑣𝑔 𝐵₁^𝑎𝑣𝑔 𝐵₂^𝑎𝑣𝑔 𝐵₃^𝑎𝑣𝑔

] (6)

Finally, the colour corrected camera values are determined by multiplying the camera values with the inverse of 𝐴_{𝑠𝑖𝑚𝑝𝑙𝑖𝑓𝑖𝑒𝑑}:

Ś = 𝐴_{𝑠𝑖𝑚𝑝𝑙𝑖𝑓𝑖𝑒𝑑}⁻¹𝑆 (7)

An example is shown in Figure 24 where colour correction was applied to part of the camera image. Note the improved colour intensity and saturation which made the colours easier to identify.

Figure 24: Camera image with colour correction applied to part of the image. Typical colour selection samples (three red blocks) are shown bottom right.

The samples used to construct 𝐴_{𝑠𝑖𝑚𝑝𝑙𝑖𝑓𝑖𝑒𝑑} is represented by the three small red blocks shown in the bottom of the enlarged section on the right. These samples could be selected anywhere on the image. For example, selecting darker samples will typically produce brighter colours while lighter samples tend to darken the final image. This simplified method of colour correction is far from perfect, but with some experience a user can make good sample selections that lead to excellent results in most cases. The main advantages of this simplified method are that no extra reference images are required, and the user has some added flexibility to deal with variations in ambient light and object reflectance.

Noise removal

The final step before the colour-corrected image could be used for edge detection was to apply a smoothing filter. Because resolution of the camera is much higher than that of the projector, the individual pixels of the projected image are often visible. To remove these pixel borders, as well as any other image noise, a noise filter is used. Such a filter must be carefully selected to ensure that it removes the noise without changing the locations of the colour transition edges. The edge locations are used to establish correspondence and their locations must be recorded as precisely as possible to ensure accurate 3D reconstructions.

After testing several different filters, a non-local means filter (NLMF) produced the most desirable results. A Matlab implementation developed by Kroon was used in its default configuration (Kroon, 2010). Figure 25 shows the filter results. Note that the colour bands are smooth and without noise while edge locations are not altered in any way. The bottom colours appear brighter because the colour correction and NLMF are applied together. The filter effect can therefore also be seen in Figure 24.

Figure 25: Non-local means noise filter, before (top) and after (bottom).