Introduction 130 

The ultimate goal of semi-autonomous robots is to achieve synergy between teleoperators and robots. For example, robots are often fitted with on- board sensors that allow accurate measurement of physical quantities often needed to make important decisions. Although a robot can provide this information, it generally lacks the capability to make an informed decision based on the information it has gathered, particularly in implicit tasks. Thus, decision making is best made by a human operator. Such a situation shows the importance of human-in-the-loop control in dynamic and unstructured situations. This is particularly true in hazardous environments, as robots can provide important information for decision making while maintaining the safety of human operators.

One advantage when deploying semi-autonomous robots is that they do not require continuous supervision as they have the capability to complete simple tasks autonomously upon human direction. Thus, human operators can be freed

from pure teleoperation, allowing them to control multiple robots if required and fulfilling a team leader style role. As discussed in previous chapters, the teleoperation of multiple robots introduces a new set of challenges, such as the introduction of task switching [284-286] and maintaining situation awareness [287, 288], especially with teams that are heterogeneous in nature [244].

In any teleoperation application, the UI or, in this case, the VRUI plays an important role by providing the operator with vital information regarding the remote environment and motion control functionality to teleoperate assigned robots effectively. If the teleoperator is assigned multiple robots, then the teleoperation VRUI is required to change and adapt for different robots within the team or even when switching between different teleoperation controls. A recent comprehensive review into the human factors that affect teleoperation control of multiple robots was conducted in [289], where gaming experience was listed as a significant factor. Several articles showed that gamers have a range of desirable attributes for the teleoperation of multiple robots, this includes multiple object tracking, visual spatial memory, and multi-tasking performance. This finding indicates that particular aspects of gaming interfaces could lend itself to teleoperation VRUI design.

The robot interactive display environment (RIDE) developed as part of research into the teleoperation of multiple mobile robots provides an example of using game interfaces for multiple robot teleoperation. RIDE provides teleoperators with three UI modes inspired by the gaming industry [290]. The three modes listed in this work are considered the supervisory mode, this provides a god-like view inspired by real-time strategy games, and the common

third-person and first-person views; all these controls have a long history in gaming. RIDE allows teleoperators to switch easily between the three different teleoperation modes depending on a given situation. This research shows that many of the challenges faced in teleoperating robots align with those faced when controlling game characters; therefore, RIDE leverages gaming interfaces because of their continual refinement over several decades.

As highlighted throughout this thesis, the recent breakthrough in low- cost and effective VR systems, such as the HTC Vive, Oculus Rift, and Gear VR, has ignited research into the possible benefits of deploying VR for teleoperation systems. Advantages such as depth perception [291], head tracking [264], gesture control [206, 292], and full body tracking [293] show a range of potential benefits for the teleoperation of robots. The research conducted in this thesis aims to take advantage of existing game engine development environments that support VR hardware to develop a dynamic teleoperation VRUI framework for controlling multiple robots deployed within the ROS environment.

The dynamic teleoperation VRUI presented in this thesis aims to provide teleoperators with an intuitive and immersive virtual environment, which dynamically assigns a teleoperation VRUI based on the current teleoperation requirements of a selected individual robot. As introduced in the previous chapter, the dynamically assigned VRUI is dependent on the currently selected robot category, the number and type of on-board sensors, and the motion control strategies available. Employing this approach means that different robot models that are part of the same robot category with the same or similar characteristics

share the same VRUI. For example, two UAVs with 360° cameras and direct flight controls can share the same VRUI even if they are completely different models because they share the same characteristics. The addition of using VR as an immersive medium improves the operator’s virtual environment by providing larger virtual real estate over conventional laptop or desktop screens. Another advantage of using VR is the deployment of more immersive controls, such as head and hand tracking systems, that provide more intuitive HCI controls than the traditional keyboard, mouse, and joystick controls often used in teleoperation systems.

This chapter presents Layer 4 of the dynamic teleoperation framework, which finalises the framework by providing the dynamic assignment of VRUI configurations during the teleoperation of a heterogeneous robot team. The layer is developed within the Unity game engine as the selected VR development environment using the Oculus Rift HMD and Touch controller VR hardware. Then, the dynamic teleoperation system is tested against conditions similar to those presented in the RoboCup Robot Virtual Rescue League (RVRL) teleoperation competition using the same four different robots across two robot categories in the 2016 challenge to test the system’s capability to dynamically assign a suitable VRUI configuration.