TacSnap : Push Button Behavior on Touch Surfaces

With the TacSnap project, we10 present our twofold approach to transfer the multimodal characteristics of push buttons onto interactive surfaces: First, we analyzed the force-path- characteristics of physical buttons and deduced a descriptive model which substitutes input force with input dwell time. Second, we implemented the concept using a remote tactile feedback in- terface. Subjective opinions on the concept and the perception of the stimuli were collected in a preliminary evaluation. This work was also published in [Richter and Schmidmaier, 2012].

6.2 Proactive, Reactive and Detached Tactile Feedback 123

Mechanical Push Buttons for Interactive Surfaces

The activation of a mechanical push button is a multimodal experience which has five phases: 1. Taking aim and reaching towards the button with the hand in the air.

2. Touching the element collecting haptic cues which immediately inform us about the but- ton’s function and current status.

3. Pressing of the button; varying forces and displacements give feedback during this re- versible process towards activation.

4. Activation; a confirming ’snap’ can be felt or heard. Subsequently, the user’s finger reduces the pressure and the button moves back to the starting position.

5. Finally, the touching finger leaves the button’s surface.

This 5-step process can be performed rapidly, depending on the mechanical characteristics of the button. Multimodal feedback is given throughout all stages.

These rich sensory cues continuously convey the button’s location, form and the state of our inter- action with it. Therefore, the manipulation of mechanical buttons demands very little visual and cognitive attention; their feedback allows for typing on a computer keyboard or turning the in-car ventilation knob fast and precisely. According to Abigail Sellen’s classification of sensory feed- back [Sellen et al., 1992], this information can be characterized by the time it happens. Reactive feedback helps to acknowledge an action whereas proactive feedback can help to determine the current modebefore taking action. For interactions with digital information, Sellen states that "by providing sensory feedback, a common class of error (mode errors) can be significantly re- duced for both novices and experts" and that combined visual/tactile feedback can "significantly improve performance" [Sellen et al., 1992]. However, this form of sensory information is very limited on today’s touchscreens, as the (programmed) non-visual feedback is reduced to a short ’buzz’ or ’click’aftertouching the on-screen keyboard. Sellen describes tactile feedback as be- ing ’sustained, demanding, and actively maintained’. This salience of tactile feedback increases its efficiency in preventing mode errors. Our TacSnap protyotype can provide remote tactile feedback coupled with a multi-step push button input on a touch surface.

Physical push buttons directly transfer the forces coming from the user’s finger into mechani- cal movement. On touchscreens, the user’s pressure has to be measured and processed in or- der to utilize it as a form of input mechanism. Force sensing on touchscreens can be done directly by implementing force sensors into the touch surface or into individual segments of the screen. A method of indirectly estimating the amount of applied pressure is to analyze the size of the contact area between fingertip and screen using capacitive or optical sensing [Benko et al., 2006]. The size of the contact area grows with the amount of applied force. How- ever, this value differs for every single finger of the hand and for multiple users. In 2003, Nashel et al. [Nashel and Razzaque, 2003] suggested the use of linger or dwell time during touch in- put for the estimation of pressure on touch devices. The longer the finger stays on an element, the harder a button is pushed. However, they did not formalize or generalize this notion. This interesting idea is backed by findings from Kaaresoja et al. [Kaaresoja et al., 2011] who found that users perceive buttons with longer feedback delays as heavier and harder to press. In order to prevent from the need to implement force sensing technology into interactive surfaces, we

124 6 Inherent Characteristics of Remote Tactile Feedback

Figure 6.12: Examples for mechanical push buttons.

developed a substitution model which formalizes the connection of input force and input dwell time.

Substitution Model

The basic principle of our substitution model can be described as follows: The substitution of input force with input dwell time can be deduced from the movement speed of the button during the phases of interaction: The more force is needed, the slower the movement of the button. If we disregard the actual displacement during input (which can not be recreated on solid touchscreens), we may substitute the force of input with the speed of input (which also affects the speed of output/feedback).

In an initial analysis, we measured the ratio of input force and displacement for different physical push buttons (see figure 6.12). We applied two measurement techniques: The first one was a mechanism consisting of a stepper motor and a Force Sensing Resistor11. The button was pushed automatically, the applied force was measured for each step. This method resembled the (more elaborate) test-rig developed by Nagurka et al. [Nagurka and Marklin, 2005]. As our collected force and displacement data lacked resolution, we implemented a second method to measure the buttons’ characteristics. With an FSR sensor mounted to the fingertip, each button was pressed several times. We averaged the resulting ratios for each button and translated the values in Newton using the sensor’s resistance-force curve.

Fo rce S en sin g Re sistan ce [ kΩ ] Displacement 2.0 3.0 4.0 5.0 6.0

s1

s2 s3 s4

s5

s6

Figure 6.13: Overlay of measured force-path-behaviors of a basic push button and separation into six linear sections. The segments are described in the text [Richter and Schmidmaier, 2012].

The resulting force diagrams of the measured buttons have several key components forming the tactile characteristics of the button (see figure 6.13). These characteristic sections can be defined

6.2 Proactive, Reactive and Detached Tactile Feedback 125

by strong changes in the curve’s gradient. We averaged the FSR vales for the start and the end of each section and determined fixed ratios between force and displacement.

Figure 6.14 depicts the resulting schematic for a force profile with six sections a to f: For sec- tion a, a higher amount of input force∆f is needed to achieve the corresponding displacement ∆d. Following our substitution model, this higher amount of needed force would be emulated

with an increased amount of input dwell time, resulting in slow feedback. The opposite holds true for section c: A low amount of input force is needed to pass through the section, result- ing in a reduced time that is needed to pass through the section caused by short dwell times and fast feedback. For most push buttons, the activation happens in this section. In general, our approach allows for a user-defined accuracy of approximation: The more individual section are defined or known from measurement, the more detailed the force profile can be recreated [Richter and Schmidmaier, 2012]. The formula in figure 6.15 describes our substitution model for each distinct section more formally.

∆f

∆d

a

b

c d e f

force

displacement

Figure 6.14: Schematic example for the segmentation of force-path behaviors. The dotted line represents the button’s mechanical behavior during the return phase [Richter and Schmidmaier, 2012].

This way, we accentuate the strong and defining aspects of a button’s mechanic behavior. A button which is harder to press takes longer to activate on a touchscreen. A button with a defining ’snap’ during activation can be replicated as this defining characteristic is replicated by the rapid change of feedback on the touchscreen. The action is reversible before the actual activation of the button. Following this model, we can recreate visual, auditory and tactile cues of push buttons on touch surfaces without pressure sensing. In the following, I present our implementation of this principle with a remote tactile interface.

dwellTimesection= (Δforce + forceStart) * delayFactor

dwellTimesection duration of section [msec]

Δforce amount of force for this section [N]

forceStart force needed to start the button’s movement [N] delayFactor describes the relation between force and dwellTime

Figure 6.15: The formula describes the substitution of input force with input dwell time for a single section in the force-path-behavior of a push button [Richter and Schmidmaier, 2012].

126 6 Inherent Characteristics of Remote Tactile Feedback

Figure 6.16: The TacSnap prototype is applied to the ball of the thumb during use [Richter and Schmidmaier, 2012].

Prototype

In order to implement the principle for tactile feedback on touch surfaces, we developed the simple feedback device depicted in figure 6.16. The device consist of a high-torque servo motor12 and a linkage system which moves a pin up and down, similar to the FEELEX’s piston crank system [Iwata et al., 2001]. The mechanism could be applied to different locations of the body, such as the back (when implemented in the chair) or the wrist (when using a small-scale wearable implementation). The user is resting the non-dominant hand on the device. When the dominant fingertip touches a virtual push-button on the screen, the pin pushes against the ball of the user’s other thumb.

The functional principle of the system is depicted in figure 6.17. The virtual button’s tactile characteristics during its activation are transferred to the non-dominant hand in upward direction. Thus, we recreate the deformation of the skin which is happening when our fingertip presses a mechanical button: The harder we press a physical button, the stronger the skin is deformed. This deformation is reproduced on the non-dominant hand. For each section of the virtual button’s force-path-behavior, we apply our substitution model, i.e. harder sections take longer to pass through. Consequently, the movement speed of the tactile pin varies for each section13. After complete activation of a button, the contact pin retreats into the encasing.

Preliminary Evaluation

In a preliminary evaluation with nine participants, we wanted to optimize our model and collect first user feedback. Our first goal was to optimize the force-to-speed substitution by identifying a general value for the force-to-speed-substitution ratio (i.e.delayFactor). Therefore, we depicted three virtual representations of three physical buttons on the touchscreen. When touched, the

12_{Modelcraft MC-630 MG}

13_{The servomotor moves the pin in small discrete steps of 0.25 mm. By varying the delay between each of these} steps, we could define the movement speed.

6.2 Proactive, Reactive and Detached Tactile Feedback 127

Figure 6.17: Functional principle of theTacSnapprototype: a: A physical button is pushed down which results in a deformation of the skin on the fingertip. b: This deformation of skin is recreated remotely by the upward movement of the interface’s pin when a virtual push button is pressed [Schmidmaier, 2011].

visual representation (e.g. height, color, brightness) changed according to the model, the tactile information was given using the prototype. Participants were allowed to freely try the physical and virtual version. Meanwhile, the participants were asked to adjust the replay speed of the virtual buttons to make the feedback similar to the stimuli coming from the mechanical buttons. We assumed a similar speed ratio across all participants. However, the value was highly variant and differed greatly from participant to participant. This topic needs further attention in future evaluations.

In guided interviews, we asked the participants about their opinion on several topics such as: the relocation of the stimulus, other possible areas for application, potentials of the system and the difference between virtual buttons with and without remote tactile feedback. All but one participant (who was ’irritated by the relocation’) stated that the TacSnap feedback felt ’good’ or ’interesting’ and the relocation was ’forgotten’ after a ’short time’. The nine participants perceived great variances in the quality of the stimuli (’too slow’, ’missing sounds’), but could easily discriminate between the virtual buttons due to the feedback. When presented with the virtual buttons with visual-only feedback, 7 out of nine participants stated that they preferred the tactile virtual buttons, because they are more ’distinguishable’, ’pleasant’ and one feels more ’connected to them’.

Discussion and Conclusions

TheTacSnapconcept is an exploration of the potential of recreating the rich proactive and reactive multimodal characteristics of mechanical push buttons. We presented a model which substitutes input force with input speed, thus avoiding the necessity to implement force sensing on touch surfaces. Using a remote tactile interface, we exemplify the principle of feedback with dynamic speed changes. In a preliminary user study, we collected positive user responses on the feeling of the created stimulus and the feasibility of the relocation. Furthermore, the augmented buttons were perceived as highly discriminable and involving.

The concept presented here is clearly a work-in-progress, future implementations and evaluations are necessary to strengthen our first positive results: Future implementations should consider

128 6 Inherent Characteristics of Remote Tactile Feedback

other, more practical locations of application. The maximum displacement and size of the pin should be adapted accordingly. Also, the force feedback is a trade-off between rich stimuli and interaction speed: The harder a button is to press, the longer the activation is taking which might be undesired or unnecessary in situations with sufficient visual feedback. This concept is very basic research, I can not recommend a clear usage scenario yet. However, participants in the study could imagine to have buttons on touchscreens which are ’harder to press’ because they activate important functions. Others embraced the ’finer control’ and the distinguishability of the buttons.

In summary, theTacSnapprinciple demonstrates that remote tactile stimuli can provide elaborate proactive and reactive feedback. This work concentrated on the substitution of input force with input speed, resulting in dynamic feedback speed. This concept can be applied to visual, auditory and tactile feedback to create more lifelike and distinguishable digital control elements. Further- more, special widgets with no physical representation can be created by applying the substitution.