Sensor and Motion Control Strategies 140 

A robot generally consists of one or more on-board sensors, such as LIDAR systems, lasers, cameras, or sonar sensors, that are used to obtain information about the robot’s surrounding environment. These sensors play an integral role in teleoperation by providing the operator with essential information about the remote environment. Without these sensors, the operator would be effectively blind. Therefore, the sensory information obtained from on-board sensors of a robot should be presented to the teleoperator in the best possible format. VR hardware such as HMD can provide a valuable way to present this sensory information, particularly when it contains 3D or other suitable information.

For example, the use of 360 cameras are increasing and images are usually captured using several connected cameras to obtain an Equirectangular video. This video format is best viewed when projected onto spherical geometry, and a VR HMD provides an intuitive way to interact with 360 video as if the viewer was within the scene being recorded. In fact, matching the tracking system of a standard camera to that of a gimbal control is also possible so that a user’s head rotations affect the camera’s pan and tilt angles, providing a similar

result to a 360 video. Another characteristic that VR devices provide is the ability for users to view stereoscopic camera vision, meaning the view has a perception of depth that is otherwise removed when viewing content via traditional 2D displays. VR devices can also be used to view 3D scanned information obtained from RGB-D cameras, as presented in Chapter 3, or 3D LIDAR systems, both of which can be more difficult to understand when viewing on 2D displays.

ROS consists of several standardised messages that provide relevant information about a robot’s on-board sensors through the publishing and subscribing of ROS topics. The ROS “sensor_msgs” package provides several message formats for describing data captured using common on-board sensors, such as camera information, images, and 3D point clouds. When using the Rosbridge suite, sharing information about these sensors through ROS topics using the JSON interface over a WebSocket connection becomes possible, as described in Chapter 3 using Layer 1. Layer 4 presented in this chapter assigns ROS topics that utilise the sensors messages available in the “sensor_msg” package and automatically assigns the appropriate presentation method in Unity to present the information to the teleoperator (Table 8). This VRUI is updated when the teleoperator opts to teleoperate a different robot within the team or switches between different sensors on an individual robot.

Similar to how ROS contains standardised messages for sharing robot sensor information, Robot Motion Control strategies can be achieved by publishing teleoperator commands to particular teleoperation topics. For

example, “/geometry_msgs/Twist” is a ROS message type used to provide pure teleoperation to different robots that typically require publishing to the

“/cmd_vel” topic. This twist message allows the communication of two 3D

vectors with one for translations and the other for rotations, allowing the direct movement of robots such as UAVs or UGVs. The “joint_state_publisher” is a ROS package that provides direct control to individual robot joints by publishing

“sensor_msgs/JointState” messages to the “joint_states” topic. MoveIt!, a

mobile manipulation package, uses the “joint_state” topics to perform motion planning for movements such as pick and place, which are commonly performed by manipulators, humanoids, and torso robots. This chapter is assigned such topics to provide motion control strategies, as listed in Table 8, and determine the required VRUI configuration required using Layer 3, as discussed in detail in the previous chapter.

6.3. Using Unity for the Dynamic Assignment of Teleoperation VRUI

This section describes in detail the integration and practical application of the first three layers presented in the previous three chapters by using the Unity game engine. Layer 1 provides the main virtual control room and communication platform, Layer 2 categorises each robot using the robot auto- categorisation system, and Layer 3 is used to determine the number of VRUI configurations for a given heterogeneous robot team.

Given that ROS is intrinsically limited to the use of a single ROS master that typically represents a single robot, a management system for multiple robots is required. Although third-party ROS packages such as Rapyuta [274] and

Multimaster [296] are available, they are only deployable within the ROS environment. Therefore, for this chapter, a simple robot manager is built within the Unity game engine to deal with the individual connections to each of the ROS-supported robots that make up the heterogeneous robot team. As shown in Figure 55, the “RobotManager” script is attached to a game object within the Unity environment and requires the Unity developer to provide a series of robot objects, which will be described later in this section. Then, the “RobotManager” script is responsible for connecting and disconnecting the connections to individual robots when the operator switches teleoperation control between individual robots within the robot team. This is achieved using the namespacing convention previously discussed by implementing a prefix name for each robot. This could simply be swapped to managing separate Rosbridge WebSocket connections if a ROS master represents a single robot as is common practice on physical robots. The example shown in Figure 55 consists of four ROS- supported robots, Hector QuadRotor, Pioneer 3DX, Pioneer 3AT, and TurtleBot; these robots make up the heterogeneous team used for testing within this chapter.