There are many opportunities to continue this research in interesting directions. The following suggests a few of these.
Stationary Occlusion Removal for Structured Illumination Mi-
croscopy
A few modifications to the stationary occlusion removal techniques presented in Chap- ter 4 would adapt this technique to removing the semitransparent micropattern from structured illumination microscopy images. This would be desirable if the final goal of the observation is an unmodulated view of the specimen and not just tracking informa-
tion. When observing moving specimens, the pattern layer acts as a stationary partial occluder, which can be removed using the stationary occlusion removal approach with no modifications.
In structured illumination microscopy of fixed specimens, the pattern and specimen move together. Even in this case, knowledge of the appearance of the pattern layer enables finding an appropriate transmission mask that removes the effect of the pattern. There are two approaches, depending on whether the pattern is repetitive with a small wavelength or unique over a large area. In the former case, the transmission mask can be determined from the many views of the pattern gathered over time. Registering all views so that the pattern in each image appears in the same location while the specimen being observed appears in different locations creates a situation in which stationary occlusion removal would compute the pattern layer, given a sufficient number of frames. In the latter case, the transmission mask must be computed based on knowledge of the pattern design and the defocus of the optical system.
Multiple Focus System
One of the limits of structured illumination microscopy as described here is that the signal from the pattern degrades as it moves farther from focus, and tracking becomes hopeless when the signal drops below the noise floor. A practical solution to this problem may be to use two cameras focused at different planes, one to provide tracking informa- tion and the other to provide specimen observations. Most research microscopes split the light in the optical train between the image sensor, which records digital images, and the ocular lens, in which the researcher observes the specimen and searches for regions of interest. The optical path length along these two directions is usually adjusted to set the same focal plane for both of these views. However, one could replace the ocular lens with another camera (a tracking camera), and adjust its position up or down the optical path to focus on the pattern layer. This adjustment would be made at the tracking
camera, so as not to affect the image of the specimen acquired by the specimen camera. The alignment of the microscope would not provide K¨ohler illumination for the tracking camera, but flat-field calibration would provide some improvement in that regard.
“Global Positioning” Patterns
In Chapters 5 and 6, I described tracking methods that determine position relative to an initial starting position. All of the patterns considered repeated over a relatively small wavelength, up to the size of a field of view. It may be possible to design patterns that provide absolute position information from a single view. For example, sinusoids with relatively prime wavelengths repeat only at intervals of the product of the two wavelengths. The position within that range can be obtained by comparing the phases of the two sinusoids. Extending to 2D would require a coordinate system established by several patterns with relatively prime wavelengths. At a minimum, three sinusoids would be required, arranged to provide a triangular region within which each set of phases would be unique. A pair of sinusoids along each axis would provide a Cartesian coordinate system. Using a few relatively prime wavelengths would disambiguate a larger range of motions. For example, along an axis with sinusoidal wavelengths of
L1 = 58 pixel andL2 = 75 pixel, motion would be unambiguous up to 58∗275 = 2175 pixel
in each direction, equivalent to several fields of view for many image sensors. Expanding the pattern to contain more wavelengths could extend this disambiguation to span an entire slide or a whole collection of slides. This latter case could be useful in histology where sections of a single tissue sample may be spread over a large number of slides.
Semitransparent Augmented Reality Markers
Human-scale augmented reality systems often use patterns printed on markers to pro- vide an image feature that can be recognized in images. The augmented reality system determines the geometry of the marker in the scene, and projects an arbitrary digital
model onto the marker, appropriately scaled and oriented. In some cases, the marker is still visible in the augmented reality image. Using semitransparent pattern markers would enable removing the marker completely from the scene, enabling the projection of semitransparent virtual objects that maintain visual consistency in the scene. Addi- tionally, if the shadow of the semitransparent marker could be recognized in the video, an appropriately scaled and oriented shadow of the virtual object could be projected into the scene.
Removing Atmospheric Effects
Many videos, especially of outdoor sports events and nature videos, are corrupted by atmospheric effects, such as falling rain and snow, and water droplets splashed onto camera lenses. Falling snow and rain behaves similarly to a moving, semitransparent layer in the video, and it should therefore be possible to extend some of the concepts in stationary occlusion removal to digitally enhance such videos. This is an active area of research—recent research has used spatio-temporal correlation analysis to detect streaks in videos characteristic of rain and snow [GN04, BKN07, GN07]. Water droplets on lenses constitute an image layer that refracts light, redirecting the intensities in an image from one location to another. In some cases, an inverted image of the entire scene appears within a water droplet. The occlusion removal technique has been shown to provide some enhancement of video corrupted by water droplets, but a more complete solution would involve modeling the refocusing of light through the water droplets.
Appendix A
ImageTracker: Motion Analysis Software
One component of my research has included designing image analysis software for micro- scopists. My software package, ImageTracker, contains analysis algorithms developed by myself (such as stationary occlusion removal [ET07]) and others (including the local- global optical flow algorithm of Bruhn, Weickert, and Schn¨orr [BWS05]). ImageTracker is free software distributed by Computer Integrated Systems for Microscopy and Ma- nipulation (CISMM).
This appendix includes a description of the ImageTracker software as well as instruc- tions for building the application on Windows and Linux systems.