Another active field of research is computational microscopy. The unifying premise of research in this field is that if microscopy images are to be analyzed by a computer, the optimal images that enable this analysis may differ significantly from optimal images for analysis by humans.
A flurry of papers have been published recently on using structured illumination to enhance the lateral resolution of fluorescence microscopy [GAS00, FKS00, RHM+03,
HB06, Car08, TRL+08]. These all rely on the same principles, with the clearest explana-
tion provided by Carlton. The approach leverages theconvolution theorem—convolution in the spatial domain is equivalent to multiplication in the frequency domain—and the fact that lenses perform Fourier transforms [Goo68, BW75]. A structured illumination pattern is used to excite fluorophores such that an individual frequency-domain image of the emitted light contains the sum of magnitudes from frequencies that fall within the Abbe limit, but also from frequencies that exceed the Abbe limit. Shifting the phase of the structured illumination pattern to capture multiple images enables recovering the individual magnitudes at all frequencies. The Fourier transform of the reconstructed frequencies provides an image with resolution beyond the Abbe limit.
Levoyet al. propose another computational microscopy technique that trades lateral resolution for obtaining depth information from a single image [LNA+06]. The technique records a light field by placing a lens array at the intermediate image plane which forms
multiple images of the specimen from slightly different perspectives. Combining the information in these subimages enables constructing multiple perspective views of the specimen or constructing a sequence of images focused at different depths through the specimen, all from a single image.
The structured illumination microscopy technique I present in Chapters 5 and 6 contributes to the field of computational microscopy. The images acquired have nonuni- form illumination, making them non-optimal for human observers. However, the extra information in the images enables obtaining 3D tracking information through computer analysis. The occlusion removal technique presented in Chapter 4 provides a method to remove the structured pattern layer from the images.
Chapter 4
Removal of Stationary Semitransparent
Image Layers
A significant component of the work in this chapter was presented at the 12th Interna- tional Conference on Computer Analysis of Images and Patterns (CAIP) in 2007 [ET07]. Video microscopy figures prominently in many fields of research, such as biology, pathology, materials science, and physics. In some situations, an experiment imposes constraints that introduce undesirable artifacts into the captured images. One such scenario arises when using long working distance lenses in microscopy of moving speci- mens. A long working distance lens, which has a smaller numerical aperture (NA) than a standard lens at a particular magnification, enables focusing further into a specimen, but also increases the depth of field. Consequently, a thicker slab of the specimen is visible in the final image.
Figure 4.1 shows several frames from a video of beating cilia on epithelial lung cells. A microbead in the bottom third of the image frame is attached to one clump of beating cilia. The inverted microscope used to create these images focuses through the substrate and cell bodies onto the cilia layer, and these components modulate the image of the moving cilia. This appears as a blurry constant background behind the beating cilia.
Other artifacts, such as debris on the image sensor (also seen in Figure 4.1), slide, or cover slip, also contribute to the final image. Microscope cleaning and alignment are
(a) (b) (c) (d) (e) Figure 4.1: Ciliated epithelial lung cells move a bead in bright-field microscopy observed with a long working distance lens. The spot circled in the first frame is from dirt on the image sensor. a-d) Frames from the original captured video. e) The mean of 61 frames from the video. Original images from David Hill.
crucial steps for obtaining the clearest images. The reality, however, is that a great deal of microscopy data is collected with suboptimal alignment and cleaning. The data collected is interesting, but corrupted by optical impurities.
With these considerations, a typical microscopy video consists of one or more moving “foreground” specimen objects and static “background” impurities. (The use of fore- ground and background does not imply which object is physically in front of another; I am choosing the convention that assigns the object of interest, the specimen, to the foreground.) The background impurities will obscure the foreground specimen as the specimen moves across the image plane. An occluding object and specimen become darker where they overlap, but rarely does an object absorb all light.
In this chapter, I present two methods for repairing video microscopy data that suf- fers from stationary partial occlusions. These approaches use the simplified geometric model of image formation in the bright-field microscope from Chapter 2, and rely on motion to enable separating the video into moving and stationary components. In ad- dition to improving the visual clarity of microscopy videos, these techniques are useful as a preprocessing step for other image analysis tasks, such as optical flow computation. Compared to background subtraction, these approaches have the important character- istic of preserving the relative intensity relationships between image sensor locations, which is necessary for photometry. The techniques presented here each have strengths
and weaknesses; I will compare the effectiveness of these methods, and describe which method suits different scenarios.
Section 4.1 discusses prior research related to occlusion removal; Section 4.2 discusses the principles behind the general occlusion removal technique; Section 4.3 discusses algorithm implementation details; Section 4.4 quantitatively and qualitatively evaluates the repair methods; Section 4.5 summarizes the findings presented here.