Matching particles

The next phase of my program is to scan every frame in the video sequence, looking for matches to the original template(s). An initial scan of a frame locates candidate particle locations using information obtained during the template learning phase. These candidate locations are then re-assessed in more detail and awarded a numeric score out of 1000. Candidate locations whose scores exceed a user-defined threshold value are recorded as

successful matches and their (x, y) coordinates are stored in a three-dimensional rawdata

array, with each page of the array containing all of the matches from the corresponding frame in the sequence.

At this point the main program can run in two modes. In thesimple tracker mode, the program uses the same user-defined templates throughout the entire image sequence. This is ideal for colloidal experiments using spherical particles which retain the same appearance. This assumes that lighting conditions and the depth of the particles with respect to the focal plane of the microscope objective do not change appreciably over the course of the experiment.

The NI pattern-matching routine can be run in a rotationally invariant mode, so that particles of a non-spherical shape can be tracked if they rotate in the focal plane. Unfortu- nately, if they rotate out-of-plane, this can be tantamount to a change in size or shape as far as the computer is concerned, which makes continuous tracking impossible in the simple tracker mode.

It may be desirable to be able to track colloidal particles that change size or shape. Biological cells, which often have flexible membranes and a complex appearance that may change upon rotation, are a good example. If the rate of change of the appearance of the particle is slow compared to the frame rate of the image sequence, continuous tracking re- mains possible. To do this, the program can be run in anadvanced tracker mode. This mode is similar to the simple tracker mode, except that it updates the particle image template after a user-defined number of video frames. As mentioned, the feature-matching routine attaches a score out of 1000 to each match. When it is time to update the template for a given species, the program simply uses the particle that had the highest score in that partic- ular frame as the basis for a new template. Since there is a user-defined margin for error in acceptable matches, this allows for small changes in particle appearance between template updates. As a consequence, particles that change shape slowly relative to the frame rate of the video sequence can be continuously tracked, while still maintaining distinguishability between different species. Obviously, if for some reason, two or more species’ appearances evolve so that at some point they are very similar, the program will be unable to distinguish them and will return an error.

3.4. StAT: The St Andrews Tracker 72

Spheres suspended in liquid can move in three dimensions, even when held in an optical trap, due to Brownian motion. As a result they can move in and out of focus, which can change their appearance substantially. Additionally, spheres moving in a 2D plane, such as spheres lying on the bottom of the sample cell, can still move out of focus due to mechanical drift of the sample stage. When running StAT in simple tracker mode, this can present serious problems, as the program will lose the particles when they move out of focus. If the out-of-plane drift is sufficiently slow compared to the video frame rate however, the advanced tracker can cope with the change and the particles can be continuously tracked.

Unfortunately, running the tracker in advanced mode presents some problems. Firstly, it is much slower, because the program has to learn new templates every time they are updated. This means that real-time applications at video rates using this mode are probably unrealistic. The real problem of running the tracker in advanced mode is however a problem which I call template drift. Due to the necessary pixelisation of the images, an error is introduced every time the image template is re-sampled. Over a long sequence of images, these errors can combine, causing the computed templates to drift away from the actual image of the subject particles.

Since the pattern matching algorithm locates the objects with sub-pixel precision, the advanced tracker can use this to resample the template to try and optimise the real match, in a manner similar to the smart templating discussed in section 3.4.3. Re-sampling the template helps, but it does not completely eliminate the drift. It also adds computation time to the pattern matching process. By choosing the rate of the template updating so that it is faster than the rate at which particles change appearance appreciably, but slow enough so that template drift is not significant over the video sequence, it can be possible to find a happy medium that allows the user to effectively track a complex sequence, such as one might find with biological cells. When tracking the movement of particles such as cells, it is often the case that sub-pixel resolution is not required, and so a small degree of template drift can be tolerated.

During the matching phase all of the measurement data is stored in computer memory. Avoiding writing to hard disk maximises the speed of the program. Once tracking is com- plete, the three-dimensional array is written to hard disk as a sequence of tab-delimited text

Figure 3.5: (a) An example of the output displayed during the tracking process. Here 2.3µmspheres have been effectively tracked, independently of the 5.0µm spheres, without any image processing required. (b) Therawdata text file showing thex, ymatch positions associated with this particular frame. (x= 0, y = 0) corresponds to the top left corner of the image being examined.

files. A typicalrawdata text file showing the results from one frame is shown in figure 3.5. Once all the rawdata files have been written, it is then necessary to sort the data so that individual particle trajectories can be determined.