Overview

A PC-based multithreaded, hard real-time controller for a sensor-assisted telerobotic system was developed. The implemented assistive force feedback system used simple sensors such as a laser range finder to guide the user's motion and a CCD camera for visual feedback. The user gets visual as well as haptic feedback on the remote PC that has Phantom Omni as the master. It was shown that the force feedback provided by the telerobotic controller and the sensors is consistent and in real-time, even though the computational resources used for the implementation were purposely limited to support a wide range of users. In order to coordinate the parallel execution of the telerobotic tasks to run in real-time, a multithreaded architecture was developed. This approach allowed the telerobotic control of the arm, sensory integration, and the computations of the different forms of assistance without incurring in high costs, increased complexity and scalability problems associated with multiprocessor workstation systems.

The control strategy described in this dissertation used sensory signals for regular, scaled and virtual fixtures using position based and velocity based control, autonomous mode, and force-based virtual fixture teleoperation during user interactions. The user was enabled to switch between autonomous control mode, force and motion-based virtual

fixtures, and scaled teleoperation modes. Several experiments were conducted to validate the trajectory following capabilities of the telerobotic system as well as the sensor-based assistance to guide the user's motion. A virtual environment for object manipulation was provided to the user in the form of a virtual cube, and a sphere was displayed as a visual cue of the position and orientation of the tip of the haptic device. In addition to the virtual environment, three (3) graphical views presented the sensory information to the user for enhanced visual perception of the object's location relative to the end-effector of the robot manipulator.

A testbed was created for conducting both simulated and physical experiments. The simulation was developed using a virtual reality model of the Puma 560 arm in MatLab and the Virtual Reality Toolbox. The C++ programming software was developed to interface the Phantom Omni software and the virtual reality simulations. For the physical experiments, the Phantom Omni Haptic device from SensAble Technologies is used as the master. It runs on a Pentium computer, with 1GHz single processing unit. The Phantom Omni device uses the OpenHaptics software which runs on Windows XP OS. The robot arm was equipped with a CCD camera and a Sick DT60 laser range finder. A Pentium II-666 MHz single processor computer was used to run the QNX Real-time Operating System. The Puma 560 software-based control strategy is a form of a PD plus gravitational compensation controller. The testing procedures of the supervisory control scheme included circular, polynomial, Bezier curves, and linear trajectories with force feedback along the Cartesian axes (X, Y, Z) as the user deviates from any of those trajectories. During those interactions, the virtual environment described previously as well as the camera views were displayed simultaneously on the

screen for visualization of the telerobotic environment. The control system architecture designed to satisfy the real-time constraint consists of the following main threads:

1. The determination of the target position and orientation with respect to the Puma end-effector (in joint or Cartesian space) and mapping this position and orientation to the Phantom Omni tip.

2. A trajectory generation thread which computes intermediate points of the trajectory to reach the target.

3. The computation of the joint angles of the PUMA for trajectory-following using inverse kinematics based on the resolved-rate algorithm.

4. The computation of the torques using a proportional-derivative (PD) controller with gravity compensation which was required to drive the motors in the PUMA. 5. The sensor information from the camera and the laser was fused to determine the

position and orientation of the target with respect to the PUMA‟s end-effector and this data was sent to the Phantom Omni for further processing.

6. The communication thread handles the position and orientation information of the Phantom Omni‟s end-effector. This information was used by the PUMA software controller for position-based and velocity-based teleoperation modes.

Also the processor that handles the Phantom Omni device has the following threads: 1 The graphics thread: It renders a virtual target, end-effector position and a

trajectory on the user screen that is similar to the PUMA environment at a refresh rate that conforms to the PUMA and Phantom end-effector movement.

2 The haptic thread: This thread computes the feedback forces based on the sensory information about the trajectory of the PUMA and the users‟ movement of the Phantom Omni. As the user deviates from the trajectory, the assistive forces required to bring the user back on the trajectory were calculated and delivered to the user using the OpenHaptics software and the actuators of the Phantom Omni interface.

3 The communication thread handles the packets containing the Cartesian position and the Euler‟s angles sent to the Phantom Omni application from the PUMA software controller.