Microscopy Data

Figure 4.4 shows the results of applying stationary occlusion removal to the video of beating cilia seen in Figure 4.1. Background subtraction does a reasonable job of re- moving stationary objects, but suffers from ghosting of the moving bead in the bottom center of the image. The LTM method succeeds in removing most of the dust occlusion in the center of the frame; a faint diffraction halo remains visible around the dust spot because the light model only handles occlusions that absorb light. The GLT meth- ods suffer less from this artifact. Diffraction effects are also visible around the moving bead, but these foreground components have no effect on any of the stationary occlusion removal methods.

The LTM method does the worst job of removing the large stationary occlusions of the cell culture. In this example, the local neighborhood is constrained to a 30 pixel2 area, but the cell culture occlusions are large and irregular enough that a larger neigh- borhood would not likely improve the result. The GLT methods do not suffer from these limitations because they form a complete model of all non-moving components of the video. Removing the stationary cell culture components accentuates the motion of the cilia.

The GLT-median is the only method that does not suffer from ghosting of the moving

(a) (b) (c)

(d) (e)

Figure 4.4: Stationary occlusion removal applied to a microscopy video of beating cilia, 248×250 pixels, 61 frames: a) one frame from the original sequence with a circle surrounding a dust spot on the image sensor, b) the same frame recovered with mean background subtraction, c) LTM (r = 15), d) GLT-mean, and e) GLT- median.

bead in the bottom center of the images. This example highlights the strength of the GLT-median approach—when the edge of a foreground object covers an image location for fewer than half the frames in a video, its effect does not perturb the background of the repaired video.

Figure 4.5: GLT-median stationary occlusion removal applied to cilia-driven mu- cus flow video, 648×484 pixel, 60 frames. Top: cilia on human airway epithelium cells drive mucus flow towards the upper left corner of the frame. Middle: station- ary occlusion removal applied to this video makes it easier to discern moving cilia, mucus, and particles trapped in the mucus. Bottom: the stationary component recovered from this video displays the cell layer. Original images from David Hill.

Having demonstrated that GLT-median most effectively removes the visible artifacts from stationary occlusions in cilia videos, Figure 4.5 shows the method applied to another data set. This example displays a much larger portion of a ciliated cell culture. The selection of a long working distance objective lens required to focus through the specimen

means that the cell culture, cilia, and mucus layers are visible. Although motion— especially of particles in the mucus layer—is visible in the video, the cell culture is the predominant visible feature. After stationary occlusion removal, however, the moving mucus layers are featured prominently, and it is easy to discern beating cilia structures in some regions. The stationary component of the video, shown on the bottom of Figure 4.5, contains a clear view of the epithelial cell culture.

The occlusion removal example from Figure 4.5 used 60 frames from a 600 frame video to construct a model of the stationary cell layer. Using the same transmission mask to repair the entire video reveals that the stationary cell layer moves slowly over the entire video, leading to ghosting artifacts in later frames. This suggests that a sliding window of frames should be used to compute a transmission map for long videos with slowly moving background components. Hillet al. used such a sliding window approach to enhance motion in DIC videos of vesicle transport [HPBH04]. An interesting avenue of future research is to extend the single layer transmission estimate to multiple layers. In this example, a stationary transmission map would be computed for the entire video, and a time-varying transmission map would account for the slow motion of the cell layer. A third transmission map may account for mucus flow over the cilia.

Figure 4.6 shows one step along the path towards estimating motion and composition of multiple deforming, semitransparent image layers. The image in the background of Figure 4.6a is a single frame from a video of cilia-driven mucus flow. In the original video, the flow is visible as a subtle layer over the top of the cells, with the predominant flow downwards in a “channel” in the center of the field. The arrows in Figure 4.6a represent the net optical flow computed from the original video. Flow computation is performed with an implementation of the combined local and global gradient-based technique of Bruhn et al. [BWS05], as discussed in Section 3.1. The magnitude of an arrow indicates the total flow starting at a single pixel from the beginning of the video. In this computation, the high-contrast, stationary boundaries of the cells corrupt the

(a) (b)

Figure 4.6: Flow computation for cilia-driven mucus flow. The gray scale image is a single frame from a bright-field microscopy video of cilia-driven mucus flow. Ar- rows show net computed optical flow from the beginning of the video (150 frames). a) Flow computed without stationary occlusion removal. b) Flow computed after applying GLT-median stationary occlusion removal. Original images from Jeremy Cribb.

flow estimation—the dominant computed flow is at the left edge of the image frame instead of the middle. Additionally, this computed flow does not preserve the volume of fluid moving through the channel—there is more flow in the middle of the frame than towards the bottom.

Figure 4.6b shows the result of optical flow computation at the same frame for the video preprocessed with stationary occlusion removal using the GLT-median metric. Here, the dominant motion in the video is correctly attributed to the channel in the middle of the image frame. The volume of mucus flow through the middle appears to be conserved. The cluster of arrows pointed upwards in the top right of the field of view indicate motion from a beating cluster of cilia.