HandSmart – Enhancing Mobile Engineering User Interfaces with MARISIL
Peter Antoniac
University of Oulu, Department of Information Processing Science, PoBox 3000, 90014 Oulu, Finland, [email protected]
Abstract
In the following paper we introduce some ideas to support mobile engineering. We understand by mobile engineering the activities that take place in normal engineering tasks but are available from anywhere at anytime. In particular, we investigate the devices that can be used and the user interfaces to support the concurrent engineering in this type of environment. The research has been focused on Augmented Reality, a technique that allows the computer to overlap the user environment with artificial graphics. By using this technique we predict that the user will have a better access and interaction to various information from non-desktop devices (i.e. mobile devices). The research can be extended in the future to support other techniques in order to improve the collaboration and communication.
Keywords: Augmented reality, mobile engineering, concurrent engineering, user interfaces, ubiquity, sign language, wearable computer and mobile computers.
1 Introduction
In a mobile environment the users are able to interact and work from anywhere, anytime. This flexibility of time and space allows a better environment and promises to bring more services to users and enable a better management of leisure time over working time.
The research of mobility and devices to support it is based on the research in User Interfaces.
Mobile devices design process has to consider several restrictions concerning the physical size of the device, the ability to handle interaction in an ergonomic manner and how to display the information to user using a small screen. To overcome this restriction a solution arises from using Augmented Reality [Butz et al., '99].
An environment to support mobile engineering should integrate some of the new concepts, like:
Augmented Reality. A technique to overlay computer generated images on the real world [Azuma, '97]. This technique allows us to integrate virtual objects into the real world. The use of this technique in combination with Image Processing [Sasaki et al., '00] gesture recognition has lead to the specification of Mobile Augmented Reality Interface Sign Interpretation Language (MARISIL) [Pulli and Antoniac, '00].
Deviceless Interface. An interface between user and an information appliance that has no mechanical interference with the user [Project, '01]. These types of interfaces will enable a better understanding of the user and will include several functionalities that are done usually separated (i.e. filtering, interface customisation, personalization, etc.). The users will have a personalized interface with the devices surrounding them and they will not need to change the interface when changing the device. This is seen as a step further towards device separation.
Mobile Collaboration. The term mobile is usually associated with mobile phones. For many people mobile means communication - therefore it is perceived as social experience. Mobile collaboration is the extension of the communication with support for collaborative activities.
Mobile Augmented Reality Systems (MARS) can enhance the user’s experience in such an environment [Hollerer et al., '01] providing them with tools that they cannot otherwise access in real life.
Mobile Engineering. When a person performs an engineering task while being unrestrictedly mobile it is associated with the term Mobile Engineering. The field has been explored mainly for workers with engineering tasks in outdoor environment (power cables, GIS planners, Constructions etc.). Based on the definition given above, two characteristics can be extracted that define the field: the capacity to perform an engineering task, and the capability of being autonomously mobile. For an engineer that has to work with computers (mobile computer engineer), to achieve both characteristics, some special designed devices will have to replace the normal desktop computer.
In previous work the authors have surveyed the area and searched for solutions to achieve this devices. In the past years some emphasis was placed on trust when using this type of devices [Antoniac and Pulli, '00] and a definition of a framework for User Interface in Virtual Enterprise have been introduced in ICE 2001 [Antoniac and Pulli, '01]. The current paper will present the current status of research and how to apply it to the field of Mobile Engineering as seen by many applications using Augmented Reality technique.
2 Proposed Theories and Techniques
2.1 Mobile Augmented Reality
Augmented Reality technique have been used in several application, most of them having the attribute of being mobile. The technique itself consists in overlaying computer information on real world. The user is able to see information about the surroundings including some of the computer generated ones. Various applications have been proposed from different fields:
medical, military, engineering, maintenance, entertainment (see Figure 1), navigation etc.
Figure 1 Example of Augmented Reality use for outdoor interaction in this picture, an action game (Courtesy Monica project, University of Oulu and J-P Metsävainio Design Oy) When interacting in an Augmented Reality based environment the user communicates with the computer in a more intuitive way. Furthermore, by using such AR based interfaces we can achieve a better movement autonomy from the user point of view. After all, the interface can be kept in the hands (see Figure 1).
Based on the characteristics accomplished while using such techniques, engineers would benefit from it when using it as a interaction tool. When using this technique, the engineer is able to get more information overlapped on the real object. A better technique is to enable the engineer to interact with others in this environment [Butz et al., '99] but also to have access to objects from any place (mobility) [Antoniac and Pulli, '01].
2.2 Mixed Reality
A short definition of the field can be given as mixing real world with virtual world. The term was introduced as between Augmented Reality and Augmented Virtuality. The Augmented Virtuality
meaning the technique of enhancing a virtual environment with objects and data extracted from real world. Several applications are described in [Tamura et al., '01]. The technique itself is very useful in collaboration scenarios when users have to access from real world virtual objects and vice-versa [Hickey et al., '00].
Figure 2 Extending the desktop using mixed reality
Applications of mixed reality can be applied to extend the desktop from the office like in the Figure 2. The user will need to use special Augmented Reality glasses in order to be able to see the virtual desktop. You can see how the video window is looking as if it is real screen. When using this technique a high level of integration between real objects from the desk and virtual ones is achieved, allowing a person to perform the work in a more ergonomic way than in the classic approach.
2.3 Virtual Prototyping
This technique enables the simulation of physical prototypes in a virtual environment. Doing this allows the analysis of a product without actually making the physical prototype available in the real world. The applications vary from e-commerce, modelling [Yin et al., '99] to virtual enterprise [Pallot and Sandoval, '98].
When using it in combination with other techniques, like Augmented Reality, an increased level of visualisation in the real environment can be experienced. The engineer has access to the prototype and can play with it in the real environment (see Figure 3). This interaction makes it easy to focus on the creativity and not on how to use the computer, in other words, better ergonomic and freedom of movement.
Figure 3 The engineer can play with the virtual prototype on his palm. A set of well-defined gestures enables the user to freely interact in the real world with the virtual prototype
2.4 Deviceless Interface
These types of interfaces are defined as “the interface between user and an information appliance that has no physical interference with the user.” [Antoniac and Pulli, '01]. The main technique
used to achieve that is Augmented Reality. This type of device (see Figure 4) allows a better separation between the device and its interface [Project, '01]. By physically separating the device from the interface a better level of personalisation and customisation is achieved. The person that is using them should be able to change or upgrade them like a pencil, without having to learn again how to write. This is seen as a step towards User Interface Appliances [Antoniac and Pulli, '00].
Figure 4 A Deviceless Interface appliance system. The appliance consist in a video camera, SeeThrough glasses and a powerful enough computer (PIII 600)
Because such devices are using Virtual Reality and Augmented Reality objects to display and interact with the user, there is a high chance to implement different applications to fit individual mobile engineering platform.
2.5 User Interface Appliances
The HandSmart interface [Antoniac et al., '01a] is seen as one example of a User Interface Appliance. The term describes a device that has the role of providing the interface between the user and multiple devices. A hybrid of this type could be seen in mouse or keyboard. Figure 5 presents a more advanced User Interface appliance, where the user is not interacting physically with the interface (deviceless) and the interface can be universal [Antoniac et al., '01b] meaning that it could be used with various devices.
This type of separation has been done before only on the application level (i.e. Java AWT, X- Windows etc.) but never on the device level.
2.6 Portable Computers
Portable computers [Newman and Clark, '99] are small laptop devices that are able to run standard software for desktop computers without changes. They are expected to use a standard display and input devices as a desktop computer, but it is almost impossible to use these devices while walking. For a desktop machine, this might be acceptable but for a machine that has to be mobile it is unrealistic.
2.7 Mobile Computers
They are small devices that are capable of accessing the information by using wireless network (whether by that we mean AdHoc network, GSM, or Wireless LAN). They should provide a lower power consumption and less processing power. Their ability to access to resources via wireless network allows them to map the applications seen on a normal Portable Computer.
2.8 Mobile Engineering
Mobile Engineering is a field in which a person performs an engineering task while being unrestrictedly mobile. Based on this definition, we have two characteristics that emerge: the capacity to perform an engineering task, and the capability of being autonomously mobile. To achieve them we will need devices that will enable the mobile engineers to perform their task as
if they were on a desktop computer. For a mobile engineer to achieve the goals that can be met while using the classic desktop computer, but from a mobile platform, will require a new interface design approach. The interface has to consider that the user might not be able to look all the time in the screen or access the input device on a fixed table. This constrains are suggesting a different approach in order to achieve a better mobility. The designer should consider implementing an interface that will be able to interact with the user at anyplace, anywhere and without needing a physical presence. From previous research on mobile devices and multimedia phones design a solution is possible when using a hybrid Mobile Augmented Reality Systems [Hollerer et al., '01].
3 Research Approach
This section will discuss a possible approach to implement a mobile engineering solution for a computer engineer. Special concern is for the equipment and the environment in which mobile engineering is taking place. A device to support the interaction for this kind of environments is presented in details and a prototype system is introduced.
3.1 Desktop paradigm for mobile engineers
A typical example of engineering work on a desktop computer is that of a person sitting in front of a desk that has desktop computer that displays the information by using a monitor screen. The user interacts with the computer by using various input devices like mouse, keyboard, touch-pad, etc. These devices are positioned for the user in various places depending on the user and on the input they provide. Some people are using the keyboard in front of the eyes since they want to look on the keys while typing and others like it in a more ergonomic position. On the other hand other devices (like mouse) do not require such eyesight contact since the user can see the output on the screen.
3.2 Mobile Augmented Reality System implementing mobile computing
The solution is an implementation of Mobile Augmented Reality System [Hollerer et al., '01, Julier et al., '00] that the user interacts with by using own hands [Pulli and Antoniac, '00]. The implementation proved that the interface increases the way of interacting with a computer while on the move. Also, the interface can morph into anything. It can be a phone, a browser or a video player. When using such interfaces, once the user assimilates the way to use them, they can be used to operate other devices, not only phones or computers. This feature allows the engineers to access various information while on the move (like location, device ID’s etc. [Azuma, '97]).
The designer of a mobile computer platform that implements an augmented reality user interface as described in [Antoniac et al., '01a] has to solve some of the problems from this field, like registration and calibration. Some experiments and prototype systems have resulted in finding possible solutions and it is just a matter of time until a working prototype will implement them all and become commercially available.
In Figure 5 an example of a mobile engineering task performed by an architect using a mobile computer system. It can be seen the way the user interacts with the virtual object and perform a design task by using the new interface paradigm like HandSmart interface.
Figure 5 A mobile engineering prototype platform. The user is editing the model by using HandSmart interface; the UI is specially designed to support high mobility applications (table
design courtesy of Meri Ruuskanen)
3.3 HandSmart Interface
The User Interface Appliances that uses the users hand as a panel for showing the interface are called HandSmart interfaces. They are called “Smart” because the hand functions are extended and because these types of interfaces support intelligent agent customisation and interaction [Antoniac et al., '01b]. An example of using the HandSmart interface for architectural design is showed in the Figure 5. The user is able to carry out a task of modifying a previous designed object while using the prototype system. During the testing phase the prototype used was not a mobile computer but a classic desktop. This was due to the size of modern computers. The author believes that a working prototype can be made available with small if at none effort (see Figure 4 for a hardware version already available on the test beds).
The interface is using a set of pre-defined gestures that the user can do in order to interact with the virtual prototype. The set of gestures is based on common life interactions but it is open to modifications (in Japan people like to keep the fingers up while browsing [Sasaki et al., '00]).
The implementation of such interfaces requires a lot of the microprocessor time to be spent on tasks concerning the UI (i.e. image processing, registration, calibration etc.). These limitations have lead to the separation of interface from the device that it serves: a concept called “user interface appliance” [Antoniac et al., '01a].
A big question for further research is how to create the separation of the user interface from the applications without changing the applications, and how to achieve that at a hardware level. The interfaces are regarded as part of the operating system or, at least, part of the applications. A way of separation was achieved on the look-and-feel, but nobody has addressed the problem at the functional level. An ideal implementation of user interface appliance would embed the functions used within normal user interfaces without any special changes in the applications. This appliance could interact with the information devices transparently and the user should not need any special set-ups in order to achieve that communication.
The basic problem currently limiting Augmented Reality interfaces design is the registration problem. The objects (the user hands for our instance; see Figure 2, Figure 3 and Figure 5) have to be aligned with respect to the interface synthetic image. Also, it has to consider the characteristics of the head mounted display and the position of the eyes of the user [Genc et al., '00], i.e. the projection model. The registration and the calibration process have to be done dynamically and with few user interventions (semi-automatic).
There are several techniques that can be used. One method consists in an interactive process to collect data in one step that does not require the user to keep his head still during the calibration.
The problem is that it is done when the user starts to use the system, but the action will have to repeat next time after the head mounted display is removed. Another way is to handle the calibration based on the scanning of the eyes and gathering the data from there, but this method
has not been explored (there are some medical issues that has not been addressed properly in the field). In our prototype system, the user calibrates the system manually, when mounting the see- through glasses. Future implementations will deal with the problem in more detail.
So far, the problems from implementing such interface are arising from Augmented Reality field, but there are other implementations problems encountered when designing the sign recognition program. When using single camera, the system has difficulties in recognizing the pointing figure. Our approach was to use a fiducial marker in order to facilitate the pointing finger as it is showed in Figure 6.
Another way, which requires more calculus and therefore more processing to be done, is by using a second camera (stereo). With this technique, if done properly, the user’s pointing finger can be detected without problems.
Figure 6 Finger marker tracking example.
The system architecture described above it is believed to fulfil the requirements for a mobile engineering platform. By using present infrastructure of wireless coverage and future 3G implementations a user will have, in the future, the capability of boasting a fully autonomous mobile computer that will replace our common way of doing work with a computer.
4 Conclusion
The working prototype described here has demonstrated that such system can be implemented.
Unfortunately, due to hardware integration problems, a full mobile computing platform was not available. In the future, after implementing a better prototype, several user tests are planned with specific applications. The main tests will deal with the user ergonomics and usability of this type of interfaces with various engineering tasks.
A good outcome is that with current technology such interface designs can be elaborated. A mobile engineer could have such interface device in order to ease up the interaction when on the move. Another good application is for future media phones. They will need a revolutionary interface to attract customers and this type of interface will overcome some of the requirements in place enabling other markets than common mobile phone one.
The area of Mobile Augmented Reality is still uncharted. Powerful concepts are arising and good applications and implementation became available. Applying the new mobile user interfaces design to this area will open new opportunities to integrate services and applications and therefore enhancing the Mobile Engineering. Augmented Reality and mobility, used together, offer a new range of services that can intensify the person experience and communication capabilities within its professional community. A better mobility will bring better communication skills and better communication - meaning faster product development, faster design and faster time to market.
Acknowledgement
The study has been carried out within Paula project. The project is funded by Academy of Finland Telectronics Programme. Some parts have been carried out within Cyphone project, which is funded by TEKES, Nokia Mobile Phones, Sonera and Polar Electro. We would like to acknowledge key persons at University of Oulu and VTT Electronics who have contributed to the research presented in this paper: Petri Pulli, Kari Kuutti, Tino Pyssysalo, Juha Röning, Tapio Repo, Marko Salmela, Petri Mähönen, Seamus Hickey , Dan Bendas, Tony Manninen and Isabela Ion. Research cooperation has been established with NAIST, Nara Institute for Science and Technology, Information Processing Laboratory. Special thanks for the hand menu system development by: Hiroshi Sasaki, Dr.
Tomohiro Kuroda and Prof. Kunihiro Chihara. Kind appreciation and thanks for the support and suggestions provided while writing the paper to Meri Ruuskanen. Last but not least special thanks to Jukka-Pekka Metsävainio for designing and visualising some of the concept models.
References
Antoniac, P., Kuroda, T. and Pulli, P. (2001a): 2nd WWRF Workshop MeetingHelsinki.
Antoniac, P. and Pulli, P. (2000): 6th International Conference on Concurrent Enterprising(Eds, Weber, F. and Pawar, K.) Center for Concurrent Enterprising, Toulouse, pp. 287-294.
Antoniac, P. and Pulli, P. (2001): 7th International Conference on Concurrent Enterprising (ICE 2001)Bremen, pp.
171-180.
Antoniac, P., Pulli, P., Kuroda, T., Bendas, D., Hickey, S. and Sasaki, H. (2001b): World Multi-Conference on Systemics, Cybernetics and InformaticsOrlando.
Azuma, R. T. (1997): Presence: Teleoperators and Virtual Environments, pp. 355 - 385.
Butz, A., Hollerer, T., Feiner, S., MacIntyre, B. and Beshers, C. (1999): 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR '99), pp. 35 -44.
Genc, Y., Sauer, F., Wenzel, F., Tuceryan, M. and Navab, N. (2000): International Symposium on Augmented Reality (ISAR 2000), pp. 165 -174.
Hickey, S., Manninen, T. and Pulli, P. (2000): Worl Multiconference on Systemics, Cybernetics & Informatics, Vol. 3 Florida, pp. 65-70.
Hollerer, T., Feiner, S., Hallaway, D., Bell, B., Lanzagorta, M., Brown, D. and Julier, S. (2001): Computers-&- Graphics, Vol. 25, pp. 799-810.
Julier, S., Lanzagorta, M., Baillot, Y., Rosenblum, L., Feiner, S., Hollerer, T. and Sestito, S. (2000): IEEE and ACM International Symposium on Augmented RealityIEEE and ACM,, pp. 3-11.
Newman, N. J. and Clark, A. F. (1999): International Symposium on Handheld and Ubiquitous Computing (HUC 99)Karlsruhe, Germany.
Pallot, M. and Sandoval, V. (1998):,Kluwer Academic Publishers,, pp. 161-168.
Project, I. (2001): Book of Visions 2001, Vol. 1 (Ed, Wireless Strategic Initiative, T. f. E. I. p.-. ) IST,, pp. 120- 121.
Pulli, P. and Antoniac, P. (2000): 2nd International Symposium on Mobile Multimedia Systems & Applications (MMSA 2000)Delft, pp. 1-5.
Sasaki, H., Kuroda, T., Manabe, Y. and Chihara, K. (2000): Virtual World, Vol. 1834 (Ed, Heudin, J.-C.) Springer- Verlag,, pp. 294-302.
Tamura, H., Yamamoto, H. and Katayama, A. (2001): IEEE Computer Graphics and Applications, 21, 64-70.
Yin, Z., Ding, H., Tso, S. K. and Xiong, Y. (1999): IEEE International Conference on Systems, Man, and Cybernetics (IEEE SMC '99), Vol. 4, pp. 463 -468.
