M. Kurosu (Ed.): Human-Computer Interaction, Part IV, HCII 2013, LNCS 8007, pp. 601–609, 2013.
© Springer-Verlag Berlin Heidelberg 2013
Magnetic Field Based Near Surface Haptic and Pointing Interface
Kasun Karunanayaka, Sanath Siriwardana, Chamari Edirisinghe, Ryohei Nakatsu, and Ponnampalam Gopalakrishnakone KEIO NUS Cute Center, Interactive and Digital Media Institute,
National University of Singapore, Singapore
{g0800474,sanath,chamari,elenr,antgopal}@nus.edu.sg
Abstract. Magnetic field based Near Surface Haptic and Pointing Interface is a new type of pointing interface which provides mouse interactions, haptic feedback and other enhanced features. It could also be configured as a haptic display, where users can feel the basic geometrical shapes in the GUI by moving the finger on top of the device surface. These functionalities are attained by tracking 3D position of a neodymium magnet, using Hall Effect sensors grid and generating like polarity haptic feedback using an electromagnet array.
Keywords: Pointing interface, haptic mouse, near surface haptic feedback, tactile display.
1 Introduction
The pointing devices are widely used as input interfaces to control and provide data to the graphical user interfaces (GUI) using physical gestures [1]. Movements and com- mands send by those pointing devices are echoed on the screen by movements of the mouse pointer (or cursor) and other visual changes. Mouse is the most common poin- ter device used today and there are also other devices such as track pad, track ball, stylus, and joystick. Recently, there were some attempts to add haptic feedback sensa- tions to the pointing input interfaces. Most of those implementations used technolo- gies such as piezoelectric actuators, pneumatic actuators and vibration motors. It can be understood that the addition of haptic sensations could enhance the attachment between the user and the computer [10].
Haptic Mouse is a new type of pointing interface which provides mouse interac- tions, haptic feedback and other enhanced features. The key advantage of this system over the other haptic pointer interfaces is that users do not required to touch the surface of the device. Instead the users could move the neodymium magnet worn on the fingertip near to the device surface and controls the cursor movement (Fig.1). As a result, it enables the haptic sensations in 3D space which will be a novel experience.
Cursor movements are handled once the north pole of the neodymium magnet face downwards and mouse commands can be execute when south pole of the neodymium magnet downwards.
Different haptic sensations provided by this system can be felt like attraction, re- pulsion and various patterns of vibrations. Furthermore, this interface can be confi- gured as a Haptic display. It is possible for a user to move his/her finger on top of the surface and sense the basic shapes of the objects on the screen. Simple geometrical shapes which are bigger than 200 pixels can be sensed and identified.
Fig. 1. Magnetic field based Near Surface Haptic and Pointing Interface system: User can move the neodymium magnet worn on the finger tip above the device surface and interact with the computer& sense haptic feedback for their inputs
2 Related Works
This section will discuss prior research with which the authors are arguing for the novelty value of the Magnetic field based Near Surface Haptic and Pointing Interface.
Liquid Interface [2] was a previous work of authors, which utilizes ferrofluid as an output display and input buttons embodied with musical notes. Using a matrix of Hall Effect sensors, magnetic fields generated by neodymium magnets worn on the finger- tips are measured and then converted into signals that provide input capability. This input actuates an array of electromagnets and generates ferrofluid bubbles. By match- ing like polarities between the electromagnets and the neodymium magnet, haptic force feedback was achieved.
FingerFlux [4] is an output technique which generates near-surface haptic feedback in interactive tabletops. It combines electromagnetic actuation with a permanent magnet attached to the user’s hand. FingerFlux provides enhanced features like, feel the interface before touching, attraction and repulsion, development of applications such as reducing drifting, adding physical constraints to virtual controls, and guiding the user without visual output.
Texture Display Mouse[13] is a haptic offers the capability of displaying properties such as patterns, gratings, and roughness. The array can represent micro-scale shapes
with various surfaces, such as gratings, grooves, patterns, shapes of icons, and Braille, and provides the user with cutaneous stimuli. Tactile Explorer [5] is a device which provides access to computer information for the visually handicapped people using tactile sensations. The tactile mouse resembles a regular computer mouse, but differs in having two tactile pads on top that have pins that move up and down.
Microsoft explorer mouse [6] is a commercially available mouse implementation which combines a light haptic sensation. Haptic-feedback, in the form of vibration through the touch-sensitive strip, indicates which one of the three scrolling speeds has been selected. Both Tactile Explorer and Microsoft tactile mouse are mouse imple- mentations combined with Haptic. It supports enhanced haptic interactions. However, operations and sensations are limited to the device surface. Furthermore, the haptic actuation is limited to a small area of the device surface.
3 System Overview
Magnetic field based Near Surface Haptic and Pointing Interface contains three modules. They are Sensing System, Software Interface Driver and Actuation System.
These three parts are described in the following sections.
3.1 Sensing System
Sensing mechanism basically concentrates the tracking of the neodymium magnet using an array of Hall Effect sensors. Hall Effect sensors array is a 2D implementa- tion, as can be seen in the figure 1. The distance between two Hall Effect sensors of the array is 2cm along the X and Y directions. Once the neodymium magnet is moving on top of the sensors grid, sensors output DC voltage values which can be converted in to a digital using the ADC converters. We have implemented a scale to measure the strength of the magnetic field detected by the sensors. The scale contains values from 0 to 1024, where first half of the range (0-512) is used when a sensor sensed a magnetic field produced by the north pole of the neodymium magnet and second half of the range is used (513-1024) when a sensor sensed a magnetic field produced by the south pole of the neodymium magnet.
We have recorded the outputs of a single Hall Effect sensor while changing the position of the neodymium magnet along X,Y, and Z axis. There, we have discovered that the sensor reading is at the highest reading for the Z axis in between the range from 0mm to 20mm. Therefore we had to lift the device surface 2cm above the sensors grid to obtain dynamically changing readings with the distance. We were able to track 3D position of the neodymium magnet which attached to the fingertip, up to 3 cm above the device surface. The mathematical model used to track the position is mentioned in the section 3.2.
3.2 3D Localization Algorithm
To move the mouse pointer in the screen we have used the Neodymium magnet attached in the finger tip. By moving the particular finger user was able to change the position of the mouse pointer. It is crucial to detect the correct position of The neo- dymium magnet over the time because failing to localize the magnet interrupt the
continuous movement of the mouse pointer. In our previous work [14] we have pre- sented a 2D localization algorithm to determine the position of the magnet. However, its accuracy was limited to surface of the device. Once the neodymium magnet it some millimeters above the surface at some points the system has failed to detect the magnet. To overcome the shortages of the previous algorithm and looking for the possibilities to apply 3D gestures we have developed a new 3D localization algorithm.
We have done a preliminary experiment is to investigate the variation in the magnetic field strength vs. the distance of all three axes and determine the strength of the magnetic field. This experiment was conducted by positioning the neodymium magnet on top of the Hall Effect sensor and measure output readings at various distances in all three axis and results are shown in the Table 1.
Table 1. Halleffect sensor readings for the X,Y and Z axis
Distance X Y Z
0 963 960 955
2.5 933 933 931
5 895 893 898
7.5 840 839 835
10 765 760 757
12.5 672 675 665
15 635 635 636
17.5 597 588 590
20 563 565 564
22.5 543 542 543
25 528 527 523
27.5 519 518 520
30 513 514 516
Table 2. Exprissions to determine the distance between the hall effect sensor and neodymium magnet based on the hall effect readings
Distance(mm) M C Expression
0-2.5 -12 963 Y=-12X+963 2.5-5 -15.2 971 Y=-15X+971 5-7.5 -22 1005 Y=-22X+1005 7.5-10 -30 1065 Y=-30+1065 10-12.5 -37.2 1137 Y=-37.2X+1137 12.5-15 -14.8 857 Y=-14X+857 15-17.5 -15.2 863 Y=-15.2X+863 17.5-20 -13.6 835 Y=-13.6+835 20-22.5 -8 723 Y=-8X+723 22.5-25 -6 678 Y=-6X+678 25-27.5 -3.6 618 Y=-3.6X+618 27.5-30 -2.4 585 Y=-2.4X+585
According to the results shown in table 1, it is clear that sensor reading values are following nonlinear curves but along the X,Y and Z axis the readings are approx- imately the same.
As shown in figure 2, the distances to the neodymium magnet from a sensor can be illustrated as circles or a spheres [12]. Data from a single sensor helps to narrow the possibility of the neodymium magnet’s position down to a large area of sphere around the particular sensor. Adding data from a second sensor narrows position down to the region where two spheres overlap. Adding data from a third sensor provides two poss- ible points where magnet can be exists. However, in this setup we placed all the sensors such as Z=0 and coordinates of the two possible points becomes (x,y,z) and (x,y,-z). Since the magnet is placed on top of the surface we have the freedom to select (x,y,z) as the correct position.
Fig. 2. Distance from the sensors to the neodymium magnet can be calculated using the output voltage of the sensors
Further, to simplify the calculations, the equations are formulated as the location of the sensor which forms a right angle triangle (Sensor 1) is at the origin, and one other is on the x-axis (Sensor 2). By using the general equation for spheres,
(1)
We could write the expressions for the S1, S2, and S3 as follows
(2)
(3)
(4) By subtracting the third equation from the second equation we can obtain a solution for x,
(5) We assume that S1 and S2 spheres intersect in more than one point. In this case subs- tituting the equation for x back into the equation for the S1 produces the equation for a circle, the solution to the intersection of the first two spheres,
(6)
By rearrange the formula for the first sphere to find the z-coordinate,
(7) After finding the solution relative to the point which causes a right angle triangle (sensor 1), we have transformed the position of the neodymium magnet to the original three dimensional Cartesian coordinate system using the coordinates of S1.
3.3 Software Interface Driver
This driver accepts the row sensor values converted to digital from the microcontrol- ler (Arduino Mega 2560) of the Hall Effects sensors grid as the input. These sensor values are sorted in the descending order and if the magnet is North Pole downwards, software searches for the positions of the sensors in the grid where it received the maximum readings. Sensors which are nearest to the neodymium magnet, output the maximum values. Based on those intensity values relative distance to the neodymium magnet from the nearest three sensors are calculated using the localization algorithm in the Section 3.3. By finding the position of the neodymium magnet and comparing it with the next position, relative X,Y displacement can be calculated. Then these rela- tive displacements are mapped to the last coordinates of the mouse curser position and moves the cursor to a new X,Y location.
In the case of identified mouse commands, firstly, driver identifies the neodymium magnet which is placed South Pole downwards by reading the digitally converted values. If the magnet is South Pole downwards, software driver searches for the three minimum sensor reading values and determines the coordinates of those sensors.
Then, the distance to the neodymium magnet from each sensor is calculated and its
position is determined using the 2D Trilateration based technique presented in our earlier paper [14]. The movement path of the neodymium magnet is tracked and if the path follows the gestures defined for the mouse commands, the driver activates the appropriate commands. As the final step, it updates Electromagnet controller circuit about the necessary vibration pattern which would eventually provide the user with the vibration feeling.
3.4 Actuation System
Haptic Mouse provides attraction and repulsion sensations by changing the polarity of the electromagnets. Polarity is changed by swapping the positive and negative voltage supply to electromagnets using a controller circuit. When the neodymium magnet worn on the finger tips and the electromagnet array positioned in the opposite polarity (N – S or S - N) users feel an attraction towards the device surface. Users feel the repulsion sensation when those magnets are in like polarity (S - S or N-N) positions.
Vibration sensations are provided by setting up neodymium magnet and magnetic array in a like polarity position and then rapidly switching on and off the electromag- netic array in certain frequencies. This rapid switching on and off dynamically changes the magnetic field it produces and affects the static magnetic flux developed by the neodymium magnet worn on the finger tips. While electromagnet is switched off neodymium magnet comes down but when the electromagnet is switched on it rises and this is felt by the user as a vibration. In the case of sensing the shapes, driver software keeps a selected vibration pattern until the user move the mouse cursor on top of the interested object in the screen. Once the cursor is moved away from the object boundary, driver sends commands to the microcontroller of the electromagnet controller circuit to change the output frequency.
This part of the system is made with six electromagnets, Magnet controller circuit and Arduino based microcontroller. As the total power required by the electromagnets array is high at 6V and 13A [3], it becomes necessary to control the power supplied to the electromagnets via a relay circuit. To address this, the relay circuit acts as a mechanism that is able to switch on a much larger power to drive the electromagnets.
For this power up electromagnets, six N-Type MOSFET [8] were used, one for each electromagnet.
4 Results
We have evaluated the accuracy of the 3D localization algorithm discussed in the section 3.2. Hall Effect sensor grid used in this device is a 4*3 array (4 sensors along the X axis and 3 sensors along the Y axis). The space between two Hall Effect sensors was allocated as 100 pixels. Therefore, all the sensor values recorded are represented as X,Y coordinates (0-300 in X axis and 0 to 200 in Y axis). This experiment was conducted by moving the neodymium magnet on top of the device surface along four straight lines which are randomly picked. Y=80,Y=(3/5) X, Y= -(2/3)X +200 , X=170 are those lines. We have used two rulers and a digital Vernier caliper to place the Neodymium magnet in the correct position. The result of the experiment is illustrated in the “Fig.3”.
Fig. 3. Accuracy test of the X, Y and Z axes
According to Figure 3, the sensors were capable to detect the motion of the neodymium magnet in near liner fashion on the surface. Further, sensors managed to detect the position more than 90% of points with less than 5% of error. This line does not reflect the movement of the mouse curser. Mouse curser position is calculated by adding the difference of the X,Y displacement between two neodymium magnet position readings. Therefore, the accuracy of the movement of the mouse cursor was further improved by cancel out differences which passes certain threshold value.
Figure 3 also shows the position detection readings of the neodymium magnet along the Z axis from the device surface to the 4cm above. Sensors were able to track the position near accurately; however, increasing the height from the surface level along the Z axis sensing module loses the accuracy. This may be due to two reasons, the limitations of the Hall Effect sensors and inaccuracies of the 3D localization algorithm.
5 Conclusion and Future Work
To conclude, in this paper we have presented a new type of computer interface which provides basic pointing interface functionalities with near surface haptic feedback.
Haptic feedback can be felt up to 6 cm of height and pointing functionality is worked up to 3cm above the device surface. Implementing variable friction for haptic interface using this technology will be an interesting research topic in future since variable friction has not been implented for touch sensitive haptic feedback systems.TeslaTouch [11] is the closests execusion of such haptic display . This device can be improved as an interface for visually handicapped who rely mostly on touch sensation. In orderto improve to this level of proficiency, this system is required to minimize the size of the electromagnets and increase the density of electromagnets packed in the electromagnets array which will provide a better resolution. This device could also be improved as an easy learning tool for children, which can be used to draw some basic shapes or characters that will enhance the interactive enjoyment.
Acknowledgment. This research is carried out under CUTE Project No. WBS R- 7050000-100-279 partially funded by a grant from the National Research Foundation (NRF) administered by the Media Development Authority (MDA) of Singapore.
References
1. Virvou, M., Kabassi, K.: Reasoning about users’actions in a graphical user interface.
Hum.-Comput. Interact. 17(4), 369–398 (2002)
2. Karunanayaka, K., Koh, J.T.K.V., Naik, E.B., Cheok, A.D.: Hall effect sensing input and like polarity haptic feedback in the liquid interface system. In: Keyson, D.V., Maher, M.L., Streitz, N., Cheok, A., Augusto, J.C., Wichert, R., Englebienne, G., Aghajan, H., Kröse, B.J.A. (eds.) AmI 2011. LNCS, vol. 7040, pp. 141–145. Springer, Heidelberg (2011) 3. Arduino, http://www.arduino.cc/en/
4. Weiss, M., Wacharamanotham, C., Voelker, S., Borchers, J.: FingerFlux: near-surface haptic feedback on tabletops. In: Proceedings of the 24th Annual ACM Symposium on User interface Software and Technology, UIST 2011, Santa Barbara, California, USA, October 16-19, pp. 615–620. ACM, New York (2011)
5. Tactile Explorer, http://www.tactile-world.com/
6. El Saddik, A., Orozco, M., Eid, M., Cha, J.: Haptics Technologies Bringing Touch to Multimedia. Springer Series on Touch and Haptic Systems (2011)
7. Magnetech Corporation,
http://www.magnetechcorp.com/Opposite_Pole.htm
8. MOSFETs, http://www.falstad.com/circuit/e-nmosfet.html 9. Engelbart, D.C., et al.: A Research Center for Augmenting Human Intellect
(demonstration) Stanford Research Institute, Menlo Park (1968), http://sloan.stanford.edu/MouseSite/1968Demo.html
10. El Saddik, A.: The potential of haptics technologies. IEEE Instrumentation & Measure- ment Magn. 10, 10–17 (2007)
11. Bau, O., Poupyrev, I., Israr, A., Harrison, C.: TeslaTouch: Electrovibration for Touch Surfaces. In: Proceedings of UIST 2010, pp. 283–292. ACM (2010)
12. Manolakis, D.E.: Efficient Solution and Performance Analysis of 3-D Position Estimation by Trilateration. IEEE Trans. on Aerospace and Electronic Systems 32(4), 1239–1248 (1996)
13. Kyung, K.-U., Kim, S.-C., Kwon, D.-S., Srinivasan, M.A.: Texture display mouse kat:
Vibrotactile pattern and roughness display. In: Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 478–483 (2006)
14. Karunanayaka, K., Siriwardana, S., Edirisinghe, C., Nakatsu, R., Gopalkrishnakone, P.:
Haptic Mouse - Enabling Near Surface Haptics in Pointing Interfaces. In: The Sixth Inter- national Conference on Advances in Computer-Human Interactions, ACHI 2013, Nice, France, pp. 336–341 (March 2013)
