Video feedback has been used for telerobotics since the 1950s, but streaming video on the Internet only started to become popular at the end of the 1990s [2]. Initially, the main technical limitations of streaming video were its requirement for high CPU load and Internet bandwidth to support the required data transmission rates, especially if the TUI (Telerobotics User Interfaces) was using more than one video stream [6]. With current devices available, the CPU load is not an issue anymore. On the other hand, while high bandwidth is becoming more readily available, it still has practical limitations. Other limitations include the restriction of viewpoints in camera positions, difficulty in conveying non-visual information and the lack of machine-readable information to support automation.
This chapter reports the experiment that was conducted to assess the effectiveness of the MR concept on two gaming environments (Second Life and Simmersion‘s Mycosm) as telerobotics user interfaces. Besides, the experiment also tried to assess the effect of the level of user attention on the interface while performing the task. Before further description about the experimental setting, the result and the discussion, the next section describes telerobotics implementation which was used for the experiment.
3.2 Prototype Implementation
Based on the initial communication implementation on Chapter 2, and due to limited access to real mining robots to test the proposed telerobotic interfaces, I built a smaller replica robot arm as a telerobot. This robot has been built to replicate the Rockbreaker robot function (manipulation robot arm).
I built the robot by modifying a haptic device. Despite this robot arm having a different shape compared with the real Rockbreaker with three rather than four links, I added an engraver as a hammer tip to this haptic device to transform this robot into a miniature Rockbreaker robot. Figure 3.1 shows the modified shape of the telerobot model.
Chapter 3
Evaluation of Gaming Environments with MR Concept
69
Figure 3.1: Robot arm, workspace and two remote cameras
For the interfaces, a 3D model of the modified robot arm including the workspace was built inside the gaming environments. In order to apply the MR concept, the streaming video from a remote camera was embedded in the virtual environment to show the elements of the scene which were not represented in the model (e.g. people or any undefined modelled objects) and to allow operators to monitor their performance. The streaming video worked by replacing the surface texture of a virtual object inside the virtual world, so it acts as a video window. In addition, the overlaying virtual dot pointer was attached to the video stream to show the position of the robot tip, which was determined by calculating inverse kinematics from the measured joint angles and projecting the three dimensional tip positions onto the plane of the video.
In order to test the possibility of using a gaming environment as a basic platform for telerobotic user interfaces, I applied the MR concept on the two different engines, Second Life (SL) and Simmersion-Mycosm (Sm), as described below.
3.2.1 Utilisation of Second Life as Mixed Reality Telerobotics Interface
The first interface to which the MR concept was applied was Second Life (SL) from Linden Lab. Figure 3.2 shows that the 3D model of the robot arm was built together with the workspace. Based on the experiment scenario, which will be explained in the following section of this chapter, an arena marked with various coloured stars was used as a workspace and also added as
70
3.2 Prototype Implementation
Chapter 3
a virtual model. Two live videos were included in the virtual environment and they work by presenting the virtual object inside the virtual world. The position of the robot model was represented by the virtual object in the form of two dots that were embedded in the streaming video surface.
As discussed in Section 2.1.2 regarding the SL gaming environment, the limitation of SL is that interaction between the human operator and the model is only available through an avatar. The operator controls the avatar to move the pointer by clicking a desired position or by dragging the model pointer (e.g. the users clicks and holds the tip model and also moves it around). To send a command to the telerobot, a clicking function was also provided on the streaming video. The operator can move the robot arm by clicking on the streaming video surface. The robot will then move in a plane perpendicular to the video camera lens axis. The two cameras are perpendicular to each other to allow movements that are commanded from any direction.
Figure 3.2: MR concept built in Second Life (SL)
Each defined position was sent as a single command to move the telerobot. The SL triggered an event on the script to send a single command each time the users clicked a position on the workspace/screen, or each time they released their finger after dragging the tip model causing the 3D model to move to that position.
Chapter 3
Evaluation of Gaming Environments with MR Concept
71
3.2.2 Utilisation of Simmersion as a Mixed Reality Telerobotics Interface
Figure 3.3 shows the Simmersion-Mycosm (Sm), another gaming engine that used to apply MR concept for telerobotic interface. The interface was built using the Mycosm library from Simmersion Holdings Pty Limited.
A similar MR concept was applied in this gaming environment. A robot arm model and a workspace were built inside the virtual world including streaming video. In contrast to the streaming videos positions in SL, I placed the videos at various places behind the robot arm model. The front view of the streaming video was set to always facing the users, so the users did not lose any information from the streaming video.
This Sm environment allows the users to give a command to a 3D model without using the avatar (the avatar model could be disabled), which I believe does not greatly affect user performance. This environment also allows the 3D robot model to move together with the user commands, (this feature was limited in the SL environment). Similar to the SL environment, the virtual dot tip pointer was also added to the streaming video surface. The clicking and dragging model input was also applied to this Sm environment, by using a mouse as an input device.
Figure 3.3: MR concept built in Simmersion (Sm)
The pointer movement was in accordance with the tip position on both interfaces. The difference between the Sm and SL environments was that the pointer moves as the model moves