Ground Truthing

Once the data was collected for all participants, a custom software tool was developed for data visualization and the identification of ground truth steps. The tool developed to accomplish this task is shown in Figure 2.18. The tool displays the accelerometer signals and video simultaneously and allows the reviewer to mark steps, including information about which foot took the step. There are nine accelerometer signals shown. The top signal is the x-axis from the Shimmer3 device worn

on the non-dominant wrist. The second and third rows are the y and z axes from the wrist device. This pattern is repeated for the device worn on the hip and the device worn on the ankle, providing the 9 total signals shown in the figure. The green rectangle is the indicator of the current index being examined. Since the Shimmer3 devices were set to record at 15 Hz and the video was recorded at 30fps, signals could be examined and steps identified with a 67 millisecond level of precision. Reviewers can add or remove steps, and the steps are displayed as vertical lines, with solid red lines for steps taken with the right foot and dashed blue lines for steps taken with the left foot. Identifying the correct foot is of particular importance since only one wrist and one ankle are instrumented, so the accelerometer signals may look significantly different depending on which foot is taking the step. In addition to the graphical representation of the acceleration for the x, y, and z axes for each sensor shown in the plot, the tool also displays the numerical values for the raw accelerometer data (in gravities) at the very top of the image. In addition, the time in the recording and current index being examined are displayed. Reviewers are able to move forward and backward and play and pause. In addition, reviewers can jump to an index through use of a dialog box. Through menu selections, reviewers can select to display only one sensor at a time. Finally, the reviewer can zoom in or out on the data in order to observe finer detail or examine how accelerometer patterns change over wider swaths of time.

In the segment of data shown in Figure 2.18, the participant is walking during the regular gait activity. Of the three wrist axes, the greatest motion is seen on the X and Y axes. This is because the wrist worn sensor captures the rhythmic oscillations present as the arm swings back and forth (the up and down motion of the body, and the forward and backward motion of the arm as it swings in the direction of walking provide the largest portions of motion). Because there is relatively little motion in the wrist on the axis perpendicular to the direction of motion (this would be seen if the arm moved inward, towards the hip or outwards away from the hip), the Z axis for the wrist motion in the figure is relatively inactive. Similarly, the hip sees vertical motion as the body moves up and down with each step as well as motion in the direction of walking. There is little motion perpendicular to the direction of walking. Unlike the wrist, in which both the X and Y axes have significant accelerations for each step, the hip’s motion is more concentrated on a single axis, the Y axis, because the greatest accelerations are seen in the upward and downward motion of the body (the forward acceleration is relatively constant, compared to the wrist, because the additional accelerations of moving the arm forward and backward are not present). The ankle

Wrist X Wrist Y Wrist Z Hip X Hip Y Hip Z Ankle X Ankle Y Ankle Z

Figure 2.18: The tool developed for visualizing the accelerometer data and identifying ground truth steps. The 9 axes of accelerometer signal (x, y, and z axis for wrist, hip, and ankle) are displayed on the left, and the video recording is displayed in the top right. Steps taken with the right foot are marked with a solid red line and steps taken with the left foot are marked with a dashed blue line.

exhibits a unique pattern, compared to the wrist and hip. The ankle follows generally displays two moments of acceleration followed by a relatively long period with no acceleration whatsoever. Since only the left ankle is being examined, there is essentially zero acceleration in the time period in which the left foot is planted on the ground as the right foot takes a step. There is then substantial acceleration as the left foot is picked up and transitions to moving forward. A second period of acceleration occurs when the foot transitions from moving forward to stopping and planting on the ground.

The data was reviewed by this author as well as Sriram Madhivanan. Twenty participants were annotated by this author, and ten participants were annotated by Sriram, with additional review and revision by this author to ensure consistency across participants. The manual annotation process took approximately 4 hours per participant for a total of 120 hours of annotation. A total of 60,853 steps were identified and marked.