2.3
Application of Visual Control to Robots
Vision has been used to control robots since at least the early 1970s [22]. A number of researchers have applied vision in control loops to ground-based robots using optic flow [5, 23, 24] for collision avoidance. Other techniques that have been commonly used in ground-based robots include stereo vision and target tracking.
During the life of this thesis project, visual control of UAVs has been a very active area of research. Consequently, there have been a number of other groups using similar approaches to mine which have published in the literature after I completed the fundamental milestones of successful landmark tracking hover [25] in 2000 and optic flow controlled hover [26] in 2001. I have included the later work in the following brief overview of progress in the field.
2.3.1
Biologically Inspired Visual Flight Control
For his PhD dissertation [27], Barrows developed a VLSI optic flow sensor for use in a micro-UAV. The sensor consisted of a single chip sensor head containing photo receptors and analog processing combined with a microcontroller that completed the processing and output servo commands. The optic flow sensor was tested on a small glider and used to avoid the floor and walls of an indoor hallway. The control algorithm employed was ‘bang-bang’ in that if a certain threshold of optic flow was exceeded, the aircraft elevator (or rudder) would be moved to a preset deflection. In later work [28], optic flow sensors have been used outdoors by Barrows to control the altitude of a small radio-controlled aircraft at altitudes ranging from 2 metres to 10 metres.
In [29] a tethered 100 gram helicopter is described which used optic flow for ter- rain following over an indoor circular track. The autopilot called OCTAVE (Optical altitude Control sysTem for Autonomous VEhicles) was used to control a single rotor mounted on a whirling arm. This arrangement constrained the rotorcraft to flight in a circular path so that only one dimensional flow in the tangential direction needed to be considered. The pitch of the rotor was adjusted by the operator to set the desired speed of the helicopter. The height of the rotorcraft was controlled by the autopilot by changing the speed, and hence thrust, of the rotor in response to the perceived height over the terrain determined by optic flow. This technique allowed the rotorcraft to maintain a relatively constant height over the terrain which included a shallow ramp. In another experiment, the helicopter was able to land by maintaining constant optic flow whilst the forward speed was reduced, a tech- nique suggested by Srinivasan et al in [4]. For these experiments, the optic flow was measured using a 1DElementary Motion Detector (EMD) [30] inspired by the physiological structure of a housefly’s visual system. The EMD circuit measured the output of 20 photoreceptors arranged in a line and was implemented firstly on a Field Programmable Gate Array (FPGA) [31] and later using a tiny microcontroller to produce a sensor weighing less than one gram. In a related project, Zufferey [32] used optic flow to control a number of robots. In Zufferey’s work, the output of a 1D camera was processed by a PIC microcontroller to determine optical flow in
10 Related Work
one direction. The resulting optic flow computation was used to control a ground vehicle and a small fixed wing aircraft. A number of behaviours were achieved:
• Steering control. Two 1D cameras, oriented at 45◦ from the forward axis, were mounted on the fuselage of the fixed wing. The aircraft was flown inside a large room and when optic flow on either camera exceeded a certain threshold, the aircraft was turned by about 90◦ to avoid hitting a wall. The control was achieved by smoothly applying rudder to full deflection and turning the aircraft over a period of one second. Using this technique, the airplane was able to fly four minutes without colliding with a wall.
• Simulated altitude control. A wheeled robot was used to maintain a con- stant distance from a wall to simulate altitude control. By keeping optic flow constant in a feedback loop, the robot maintained a relatively constant dis- tance from the wall.
Reiser and Dickinson [33] describe object avoidance using optic flow in an insect- inspired robotic control test bed comprising a 5 degree of freedom gantry. A vision sensor comprising a 30 frame per second 115o x 95o field of view camera was placed on the gantry and was free to move within a 92cm diameter cylindrical arena. A PC attached to the sensor using a frame grabber was able to calculate optic flow using an array of spatio-temporal correlation elements, using the method proposed by Hassenstein and Reichardt [34]. For the experiments, the vertical position and translational speed of the robot was kept constant and pitch and roll attitude was kept fixed. Detectors for image expansion were used to trigger insect-like saccades, such that the robot would rapidly change heading when approaching an obstacle. The results showed that simple image loom detectors were enough to prevent the robot from hitting the walls of the arena and obstacles placed inside the arena.
In [35,36] experiments were conducted on theAVATAR helicopter which suggest that slow flying UAV may be able to avoid obstacles in an urban environment using vision. For this experiment, the AVATAR was fitted with a forward looking stereo camera and two cameras providing optic flow computation on the side. In forward flight the sideways looking optic flow cameras determined range to objects in their field of view. As the helicopter was flown towards obstacles on the side, such as trees, the helicopter tended, in most instances, to turn away from those obstacles. This technique might be extended to the point where a helicopter could safely pick a path through an urban environment, such as flying down the middle of a street autonomously.
Chahl tested a scheme for terrain following on a 1.2m wingspan delta-wing UAV using optic flow [37]. For flight test, a downward looking camera with a field of view of 100 degrees was used with a control-by-telemetry scheme implemented on a 500Mhz Pentium III ground computer. For each local optic flow vector calculated, the scheme calculated the corresponding climb angle needed to clear the obstacle represented by that flow vector. The maximum climb angle from the set of all climb angles calculated was used as a reference input for the longitudinal controller. Chahl noted that the rotational effect on the optic flow calculation dominates the
§2.3 Application of Visual Control to Robots 11
raw results, so that it is critical to subtract off the effect of rotations using a rate gyroscope before calculating the range to obstacles.
2.3.2
Non-Biological Examples of Visual Flight Control
One of the earliest successful uses of vision to control a helicopter was undertaken by Amidi for his PhD dissertation [38], submitted in 1996. In Amidi’s work, carried out at Carnegie Melon University (CMU), a pair of cameras was used to track natural features on the ground using high-speed template matching. The initial features were selected by picking a window of pixels at the centre of the image. By locking on to ground objects as the helicopter moves, the vision system was able to estimate the helicopter’s position and velocity. Stereo vision was used to determine the range to the features. A series of PD controllers were used in a feedback loop to control the position of the helicopter in hover and slow forward flight. For outdoor experiments, the vision system was implemented on a 67kg Yamaha R-50 helicopter and was shown to be able to control the helicopter with a position accuracy of 3-10cm in hover.Saripalli et al, at the University of Southern California (USC), have landed their AVATAR autonomous helicopter on a helipad using vision and inertial information [39]. The AVATAR is a small radio controlled helicopter with a PC-104 based avionic architecture. In their work, a pattern comprised of polygons painted on the helipad was detected and tracked so that the helicopter could align itself with the pattern and then use its relative pose to land. The image was first segmented with a fixed intensity threshold and then the first, second and third invariant moments of the thresholded pixels were calculated and compared against a template descriptor of the moments from the known target geometry (see [40] for an explanation of this technique). The advantage of this approach is that the invariant moments are not affected by translation, rotation or scaling and that the target descriptor is stored as a relatively small vector. The helicopter was able to track the pattern even when the helipad was moving, however, the helipad motion was stopped for the actual landing.
In [41], Mejias et al worked with members from the USC group to achieve visual servoing of the AVATAR and a similar small autonomous helicopter designated the COLIBRI from the Universidad Polit´ecnica de Madrid. The visual servoing target was a window from a building. In the case of the COLIBRI, template matching based on the Lucas-Kanade tracker [42] was used to match the image of the window being tracked to a stored reference template of the window. This provided the coordinates of the corners of the window which were used to generate a set of velocity commands to an inner loop controller, to move the helicopter into a position where the target was centred in the field of view. The helicopter was flown to within about 4m of the window and then the visual servoing loop was activated. The helicopter trajectory successfully converged to a position where the target was centred in the image.
Researchers at the University of California Berekely used a Yamaha R-50 un- manned helicopter to test a vision system for control of landing [43, 44]. In this work, a target comprising a black and white pattern of small squares of different
12 Related Work
sizes was placed on the ground. A camera image was first thresholded by using a histogram technique with the intensity cuttoff set by trial and error. Once the landing target was segmented from the background, the corners of the squares were detected and each square was identified based on the centres of gravity. A non-linear optimisation technique was used to calculate the position and attitude of the UAV relative to the target. In their flight tests, the output of the vision algorithm was compared to the output of the onboard GPS/INS system. The results compared to within 5cm in translation and 5◦ in attitude.
Amongst other UAV programs, Georgia Tech operate a Yamaha RMAX UAV dubbed the GTMax. This helicopter has been fitted with a variety of sensors in- cluding differential GPS, inertial, magnetometers, an ultrasonic height sensor and a camera. In their experiment, a target on the ground comprising a dark square on a light background was tracked from a camera on the GTMax. Given the position and orientation of the target apriori, an Extended Kalman Filter (EKF) was used to fuse inertial data with data from the visual system, to provide the attitude and position of the helicopter without GPS [45]. Using an EKF output as a reference, the GTMax was able to follow position commands from the ground [46].
Proctor et al [47] constructed a glider with a camera and onboard video trans- mitter which responds to commands telemetered from a control computer on the ground. The guidance and control system used a high-speed pattern matching tech- nique [48, 49] to track the outline of an open window using only a nose mounted camera on the glider, with the aim of getting the glider to fly through the window. A discrete EKF was used to estimate the states of the glider from the observed window geometry. The results of simulation and flight test of the glider provided an indication that, under some conditions, it would be possible to guide an aircraft using vision alone.