Sensor-Fusion Results

After approximate alignment of the IMU and camera frames with the Qualisys system, the inertial- and visual-tracking results are fused using the sensor fusion scheme of Chapter 4.

The IMU information is captured at a sampling rate of 100 Hz; whereas, the camera frames are captured at approximately 15 Hz. The accelerometer information is upsampled to 105 Hz in order to have the sensor sampling rates be an integer-multiple of each other. Qualitative results of applying the basic sensor fusion and asynchronous fusion schemes are shown in Fig. 5.11 and 5.12, respectively. The RMSE (m) between the fused position estimates and the Qualisys (i.e. ground-truth) estimates are reported in Table 5.2. In the control group scenario, the cameras have constant line-of-sight of target and accordingly the vision-only tracking results have a high degree of precision. The synchronous and asynchronous sensor-fusion schemes are out-performed by the vision-only estimate which highlights the difficulty in effectively applying inertial measurements, since small errors in attitude estimation or

Experiment

Scenario World-Domain Trajectory Error (RMSE in m) Synchronous

(see Fig. 5.10) 0.04045 0.04361 0.02582

Occlusion

(see Fig. 5.10) 0.05324 0.05408 0.05416

Table 5.2: RMSE performance of the proposed fusion schemes on the real-world image sequence.

bias-removal (i.e. mitigating drift error) will impede fusion from improving the vision-based estimate. As mentioned in Chapter 4, visual-inertial sensor fusion is most needed in scenarios where the target is not in direct line-of-sight of the cameras (i.e. occlusions).

An occlusion scenario is synthetically created by cropping a portion of the camera frames, which corresponds to the target being occluded from one or both of the camera views. The second row of Table 5.2 shows the results of applying sensor-fusion to this more challenging (i.e. occlusion) image sequence. Both the synchronous and asynchronous sensor-fusion are shown to achieve a better RMSE than the vision-only system. Fig. 5.14 and 5.13 show the world-domain trajectory estimates from the synchronous and asynchronous sensor-fusion schemes, respectively.

5.4 Conclusion

Overall, the real-world experiment demonstrated a lot of the unexplored pitfalls when transferring from a simulation to a laboratory environment. Notably, the camera calibration procedure is much more involved. Albeit, the tracking results are encouraging for the camera calibration and visual-tracking algorithms proposed. The sensor-fusion results indicate the difficulty in improving the positional precision of the visual-tracking results with inertial measurements. The role of inertial-trackings appears to be better suited to bolstering the visual-tracking results in the presence of clutter and other noise which would greatly impede the performance of vision-only systems. In future work, a tightly-coupled sensor-fusion scheme could be tested such that the visual measurements would be able to play a role in the attitude estimation of the IMU device.

0 100 200 300 400 500 600 700

Figure 5.11: Plots of world-domain position (m) using proposed synchronous sensor-fusion scheme. The IMU is resampled to 105Hz in order to be an integer multiple of the 15Hz camera frame rate.

Figure 5.12: Plots of world-domain position (m) using proposed asynchronous sensor-fusion scheme. The IMU is resampled to 105Hz in order to be an integer multiple of the 15Hz camera frame rate. This fusion scheme will adaptively adjust the delay between the inertial and visual information.

0 100 200 300 400 500 600 700

Figure 5.13: Plots of world-domain position (m) tracking on the occlusion experiment, using proposed synchronous sensor-fusion scheme. The IMU is resampled to 105Hz in order to be an integer multiple of the 15Hz camera frame rate.

0 100 200 300 400 500 600 700

Figure 5.14: Plots of world-domain position (m) tracking on the occlusion experiment, using proposed asynchronous sensor-fusion scheme. The IMU is resampled to 105Hz in order to be an integer multiple of the 15Hz camera frame rate. This fusion scheme will adaptively adjust the delay between the inertial and visual information.

Chapter 6 Conclusion

6.1 Contributions

Many industries stand to benefit from a system capable of delivering high-precision tracking results, on-par with contemporary vision-based systems, as well as providing the portability and robustness of an inertial-tracking setup. Of particular interest is the impact of such a system to the field of motion capture, such as in the film and animation indus-try. The idealized hybrid system would greatly reduce the overhead of current vision-only systems and expand the scope of environments accessible to motion capture. The approach taken in this thesis towards such a system has been a bottom-up: addressing the founda-tions of a strong vision-based system with the discussion of camera calibration and lens distortion correction, all the way up to investigating sensor-fusion schemes capable of han-dling asynchronous sensor measurements across the different sensing modalities. The thesis provided three main contributions along the exploration of visual-inertial sensor fusion: a novel lens distortion correction method using an under-utilized transform model, which provided a robust approach; a simulation environment for quick debugging and testing of visual-inertial sensor fusion techniques, validated by a proposed visual-tracker and sensor-fusion algorithm; and an asynchronous sensor-sensor-fusion scheme for dealing with measurement misalignments between the various distributed sensing modalities arising in a real-world set-ting. The following sections summarize the contributions in each of these areas and provide suggestions for future work.

6.2 Robust Domain-Filling Lens Distortion Correction

Camera-calibration is a crucial part of any system sporting a vision-based component.

High-quality calibration can not be over-stated as an integral step in any system attempting to achieve high-precision results. Lens distortion correction is an often under-valued part in the complete calibration of the system. Wide-angle lens play a key role in surveillance and tracking applications as the accompanying larger FOV minimizes the number of cameras required to monitor a given tracking volume. However, the wide-angle lens comes at the cost of lens distortion. The novel lens distoriton correction scheme presented in Chapter 2 emphasized the utility of the forward-transform model for lens distortion. We presented an explanation of how the model could be used to provide both robust correction techniques in the presence of image noise as well as a small sample size of training images as well as simultaneously providing an algorithm better suited for hardware acceleration then its contemporaries. Experimental results were presented to validate the claims of robustness and demonstrate the efficacy of the novel lens distortion scheme. The camera calibration techniques were further tested as a single part of a real-world sensor-fusion experiment, shown in Chapter 5. The proposed lens distortion correction scheme presented in thesis can be expanded later on to apply the solve for the forward-transform model using different cost functions, notably [2] present a cost function based on the slope differences of adjacent line points which would be an alternative to the regression line residuals used.

6.3 Simulation Environment for Visual-Inertial Sensor Fusion

Development of novel visual-inertial fusion schemes can be encumbered by the barrier to entry for testing algorithms spanning multiple different sensing modalities. The simu-lation environment presented in Chapter 4 provides a straightforward means of generating both multi-view camera data as well as accompanying inertial information (i.e. accelerome-ter, gyroscope, magnetometer) in order to guide algorithmic development. The simulations are easy to setup and provide comprehensive noise models for both the visual and inertial components. In preparation of testing on real-world data, the simulation environment also provides means of testing synchronization issues between the various sensors involved. The visual and inertial sub-systems of the simulation environment have been independently vali-dated by well-known tracking algorithms, including specifically proposed visual-trackers for

the application at hand- hybrid motion capture systems. A proposed loosely-coupled sensor fusion algorithm has verified the collective use of the simulation environment through an array of different test scenarios. Going forward the simulation environment of [137] will provide more detailed models of the IMU’s in the camera frames; moreover, the proposed visual-tracking algorithms will be expanded to handle multiple data-associations in order to validate the currently implemented multiple IMU capabilities of the environment.