112
11.8
User Study
Figure 11.11: 52 cell tracks created by the au- thor for a 101 timepoint time-series dataset of aPlatynereis embryo. The tracks were cre- ated in about 40 minutes. See the supple- mentary video for the creation of a single track, and a debug visualisation showing in- tersections with the nucleus.
In order to evaluate the performance and usability of the Bionic Tracking method, we have conducted a user study with seven experts in either manual or algorithmic cell tracking, or both (median age 36, s.d. 7.23, 1 female, 6 male). In the study, the users were given the task to track cells in thePlatynereis dataset also featured in Figure11.11. One of the participants was already familiar with the dataset. The user study was conducted on a Dell Precision Tower 7910 workstation (Intel Xeon E5-2630v3 CPU, 8 cores, 64 GB RAM, GeForce GTX 1080Ti GPU) running Windows 10, build 1909, with a HTC Vive VR headset equipped with eye trackers by Pupil Labs.
The users who participated in the study had no or very limited experience with using VR interfaces up to this point (5-point scale, 0 means no experience, and 4 daily use: mean 0.43, s.d. 0.53), only one of them had previously used an eye-tracking-based user interface. (same 5-point scale: mean 0.14, s.d. 0.37).
11.8.1 Procedure
Before starting the experiment, the users were informed of goals and potential risks of the study (e.g. simulator sickness). In a questionnaire that was split into a pre- experiment and a post-experiment part, the users were asked about the presence of any motor or visual impairments, previous VR experience, and their current wellbeing. The full questionnaire is available in AppendixC.
b i o n i c t r a c k i n g : u s i n g e y e t r a c k i n g f o r c e l l t r a c k i n g
113
ments in general, if necessary. After the fit of the headset was ensured, the eye trackers were calibrated. The users were then asked to create as many tracks as they liked and are comfortable with. If any of the created tracks did not satisfy them, the offending track could be deleted.
After the experiment was done, the post-experiment part was filled out, in this part users had to judge the usability and suitability of the software, were asked again for their wellbeing, and in addition had to rate their experience with both the NASA TLX questionnaire [Hart and Staveland,1988] and the Simulator Sickness Questionnaire (SSQ, [Kennedy et al.,1993]). The questions about the usability and suitability of the software were based on both the System Usability Score [Brooke,1996] and the User Experience Questionnaire [Laugwitz and Held,2008].
As final element of the study, a free-form interview was conducted in which the users could comment about the software, and suggest improvements.
11.8.2 Results
In the experiment, users created up to 32 cell tracks in 10 to 29 minutes.
The average SSQ score was25.6 ± 29.8s.d. (median14.9), approximately on par with other VR applications that have been evaluated using SSQ [Singla et al.,2017]. For the NASA TLX score, we used all categories (mental demand, physical demand, temporal demand, success, effort, insecurity) on a 7-point scale where 0=Very Low and 6=Very High for thedemand metrics, and 0=Perfect, 6=Failure for the performance
metrics. Users reported medium scores for mental demand (2.71 ± 1.70) and for
effort (2.86 ± 1.68), while reporting low scores for physical demand (1.86 ± 1.95), temporal demand (1.57 ± 0.98), and insecurity (1.14 ± 1.68). Most importantly, the participants did judge themselves to have been rather successful with the cell tracking tasks (1.71 ± 0.75).
The users explicly expressed interest in using Bionic Tracking for their own track- ing tasks (3.43 ± 0.53; 5-point scale here and for the following questions: 0=No agreement, 4=Full agreement). The tracks created were judged to look reasonable (2.57 ± 0.98), and Bionic Tracking was deemed to provide an improvement over their current manual tracking methods (3.14 ± 0.90). Furthermore, the users stated that they could create new cell tracks not only with reasonable confidence (2.86 ± 0.69), but much faster (3.29 ± 0.76). Users also found the software to be relatively intuitive (2.43 ± 0.98) and did not need long to learn how to use it (0.59 ± 0.79). Especially the ergonomics of the method were remarked about in the follow-up interviews:
”It was so relaxing, actually, looking at this [cell] and just looking.” (P2, the user remarked further after the interview that the technique might prevent carpal tunnel issues often encountered when tracking using mouse and keyboard.)
”I figured this could be like a super quick way to generate the [cell] tracks.” (P7)
114
Fu lly di sa gre e Fu lly ag re e Av g M ed ia n St dd evThe software felt responsive to my inputs. 3.14 3.00 0.38 Being in an isolated VR environment
irritated me. 0.00 0.00 0.00
I had trouble orienting myself. 0.71 1.00 0.76
I would have liked a different input/control
method. 0.57 0.00 0.79
The usage felt very natural and intuitive. 2.43 2.00 0.98 I had to keep track of too many things at
once. 0.86 1.00 0.69
I was put off by the prototype character of the
software. 0.29 0.00 0.76
I needed a long time to learn how to use the
software. 0.57 0.00 0.79
The interaction felt very precise. 2.43 2.00 0.98
Having my eyes tracked irratated me. 0.29 0.00 0.49
The cell tracks created looked reasonable
to me. 2.57 3.00 0.98
I could complete the tracking tasks with
confidence. 2.86 3.00 0.69
I could imagine adopting the presented
technique for tracking of my datasets. 3.43 3.00 0.53 The presented technique provides an
improvement over current techniques. 3.14 3.00 0.90
The presented technique would allow me to
perform tracking tasks faster. 3.29 3.00 0.76
The presented technique would allow me to
perform tracking tasks more precisely. 2.29 2.00 0.76
Frequency 0 1 2 3 4 5 6
Figure 11.12: Results of usability and accep- tance question from the user study. Note that the questions are formulated both posi- tively and negatively.
The results from all questions related to software usability and acceptance are summarized in Figure11.12.
We made two more interesting observations in the user study:
First, we saw that users adjust playback speed more often than image size in VR. Af- ter exploring different settings – users could choose speeds from 1-20 timepoints/sec- ond – all users independently settled on a playback speed of 4-5 timepoints/second for tracking, corresponding to 200-250 ms of viewing time per timepoint, which coincides with the onset delay of smooth-pursuit eye movements (see Section2.2.1,
Eye movements, and [Duchowski,2017]). The chosen visual size of the dataset was
also usually chosen to be approximately human-scale (which was also the default setting, but experimented with by the users).
Second, despite having no or limited previous VR or eye tracking experience, the users did not at all feel irritated by the environment (0.00 ± 0.00), nor by the use of eye tracking (0.29 ± 0.49).
Our preliminary results and user study show that cell tracks can be reliably re- constructed by “just looking at them”, using eye, head and body movements that are used in everyday life. Importantly, the users estimated that the Bionic Tracking
b i o n i c t r a c k i n g : u s i n g e y e t r a c k i n g f o r c e l l t r a c k i n g
115
method would yield a speedup of a factor 2 to 10 (3.33 ± 6.25) compared to tracking cells with a 2D interface.
Scan this QR code to go to a video show- ing tracking of a cell viaBionic Tracking in earlyPlatynereis development. For a list of supplementary videos seehttps://ul- rik.is/writing/a-thesis.
11.9
Discussion and Future Work
In this chapter we have introduced theBionic Tracking strategy for tracking cells in
3D microscopy images in an effort to speed up manual tracking and proofreading and developed a proof of concept. Preliminary results show that we might be able to achieve approximately an order of magnitude speedup compared to manually tracking cells. Before we can bring this strategy into actual use for biologists, we need to do two more things:
• First, implement interactions that allow to track or proofread lineage trees. Such an interaction could for example include the user pressing a certain button when- ever a cell division occurs, and then track until the next cell division, and • Second, Bionic Tracking has to benchmarked against other automatic solutions,
e.g. on cell tracking challenge datasets (see e.g.CellTrackingChallenge, [Ulman et al.,2017]).
We foresee the limitation that for tracking large lineages entirely, Bionic Tracking will not work, simply for combinatorial reasons. It can however be used to track early- stage embryos where cells may have less-defined shapes, or it may provide constraints to training data to machine learning algorithms. Furthermore, Bionic Tracking could be used in a divide-and-conquer manner in conjunction with an automatic tracking algorithm that provides uncertainty scores, and only be applied in regions where the algorithm cannot cross a given uncertainty threshold. We could further increase the usefulness of Bionic Tracking by not just searching for local maxima along rays, but actually extract the centroids of cells.
Ultimately, we would like to integrate Bionic Tracking into existing tracking soft- ware, such that it can be helpful for a more general audience. Current developments in eye tracking hardware indicate falling prices in the near future, such that those devices might become way more common soon. Alternatively, one could imagine just having one or two eye tracking-enabled HMDs, and make them available to users in a bookable item-facility-like manner.