Hand tracking provides possibilities (and affordances) for natural mapping of hand movement and fine motor skills. On the down side, fine motoric tasks also require more development capacity to respond to the different gestures. For example: opening a soda can is a delicate movement, and having full finger tracking will prime the user to use their fingers to open the can, as they would also do in the real world (depicted in Fig.30). For VR, the reaction to this gesture needs to be specifically programmed each object individually, as picking up a heavy object such as a fire extinguisher with the same gesture will not align with the user’s expectation and will interfere with the experience.
Fig.30: Grabbing a heavy fire extinguisher with this motion will feel unnatural.
Chapter 3 - Concept Generation
This also points out the second obstacle on the side of immersion when using full hand tracking: haptic feedback. Haptic feedback is an umbrella term for two types of feedback: tactile and kinaesthetic feedback. Tactile feedback refers to the type of sensation that can be felt on the surface of the skin, for example texture or vibration. Kinaesthetic feedback refers to the sensation generated from the proprioceptive sensors in the muscles, joints, etc. These sensors give information about the approximate size, weight and position relative to the body of an object [7]. Tactile feedback, in the form of vibration, is a well-known solution used in a variety of digital devices to give feedback to the user, however, kinaesthetic feedback is much harder to mimic. One option is to omit kinaesthetic feedback entirely, resulting in the user only having to perform the hand gesture of picking up/using an object (and possibly providing a rumble to give some tactile feedback). This way the user grabs in space, making a fist or other gesture on the location of the virtual object, thereby picking it up. A setup for this approach was created with Manus VR gloves. The advantage of this method that the gesture is quite natural, reducing the learning curve. The drawback is that the balance is shifted to fidelity without
53
providing a similar increase in immersion, whereby the user’s presence is compromised. There are however solutions to compensate for the lack of immersion. Through the use of additional VR hardware like HaptX gloves (shown before in Fig.14) the kinaesthetic feedback of touching a virtual object can be simulated to a certain extent. This allows for less compromise when feeling the shape of certain objects, but still limits the resistance that is felt when e.g. trying to push through a solid wall. This level of kinaesthetic feedback might then set a certain level of expectations, which will then be broken because they are incomplete. Besides the expectation barrier, the extra setup time and having to wear all the extra gear might actually diminish the experience. A different proposed solution could be to have an external robot arm move around the user’s play area in anticipation of virtual objects being touched. This robot arm would then places physical surfaces at the locations where the user touches the virtual object, and provides resistance to the user’s movement to provide kinaesthetic feedback. However, the sheer complexity and additional hardware of this solution are, at least at this point in time, considered to be too excessive for the implementation of the fire-safety training.
Fig.31: Manus VR test setup. Top: real life hand pose; Bottom: virtual hand pose
In order to circumvent the issues caused by the option above, and to provide more natural and accurate immersion, tracked props can be used to provide physicality to the virtual objects. The advantage of this solution is that it allows for real haptic feedback as accurate as the props are matched to the virtual props. The downside of this solution in general is that the virtual environment becomes constrained to the same physical limitations of the real world, and the available virtual area becomes directly proportionate to the physical space. Another issue with this solution is that once some intractable props become physical, others must also be physical in order to match the achieved expectations for this level of immersion.
In case of the fire-safety training, this solution is undesirable, since it would require having a complete physical mock-up of a cabin, as well as a virtual skin to overlay over the physical mock-up. If such a solution is considered to recreate a part of the real world, it may be worthwhile to look at AR as a solution instead of overlaying an entire virtual world over a similar physical environment.
This does not mean that this method would be undesirable in all cases. Section 6.2 briefly discusses a different training where a physical
54
interface could significantly benefit the experience without having to manufacture an entire physical environment.