Computer Vision Estimation

Computer vision (CV) is an interdisciplinary field that deals with how computers can be made to gain high-level understanding from digital images or videos. It helps in tasks that include methods for acquiring, processing, analysing and understanding digital images, and in general, deal with the extraction of high-dimensional data from the real world in order to produce numerical or symbolic information. The CV estimator uses a vision based system as the only sensor input and therefore it does not require any mounting of sensors on the load. The CV system uses images from a downwards looking camera to calculate a position vector of the load in the MRUAV fixed frame pointing from the camera to the load. This makes it ideal for augmenting an already autonomous MRUAV with slung load capabilities. This estimation system requires a slightly bigger MRUAV platform (Fig. 7.4) that the one presented in Section 3.5, in order to be able to carry a gimballed camera sensor.

Sensor system

The vision system is a camera mounted on the MRUAV frame looking down on the load. A two-axis gimbal was designed to keep sensor system downwards. If the camera is only fixed to the vehicle, there can be a huge amount of false detection estimations of the position of

Fig. 7.4.: Test-beds size comparison, higher: CV platform, lower: Standard platform.

the slung load when the vehicle is pitching and rolling (drastic change in attitude). The gimbal is made with 3D printed parts and contains two rc-servos to compensate the pitch and roll angles of the MRUAV attitude when travelling, it can be seen in figure 7.5. With a gimbal, the camera will always be pointing down regardless the attitude of the vehicle. The rc-servos are commanded by the flight controller using the reversed attitude angle and transformed to PWM, which is the input signal to the rc-servo. One disadvantage of this sensor system is that requires a larger MRUAV (Fig. 7.5) to be able to carry and accommodate properly the extra equipment. Having a larger vehicle, approximately 450mm from rotor to rotor centre, will limit the trajectories that can be performed inside the MAST Lab due to the size of the aircraft, but for outdoors flying is a very good estimation option.

Computer vision algorithm

To easily identify the load amongst other objects that appear in the camera view the load has a specific colour; in this case the slung load is predominantly black while the ground is a light colour. Therefore a colour tracking algorithm is needed. The location of the colour

area in the image is thus an estimate of the position of the load relative to the orientation

of MRUAV. Several colour algorithms were created to achieve the goal of finding the slung load position, more detailed information of the algorithms can be seen in Appendix A.5. Such algorithms use the OpenCV (Open Source Computer Vision) library of programming

Two-axis-gimbal / Camera System

Fig. 7.5.: Test-bed v2 Quadrotor with gimbal/camera system mounted.

functions (Bradski, 2000). This library is mainly aimed at real-time computer vision. The algorithm steps are as follows:

1. Grab an image frame from the camera video stream 2. Convert that image from BGR to HSV format

3. Check if the converted HSV array elements lie between the elements of two other HSV arrays (colour selection is a range of black colour)

4. Apply two very common morphology operators, dilation and erosion

5. Find contours in the binary output image based on Suzuki et al., 1985, with a result similar to the image in Fig. 7.6

6. If the identified area is larger than the minimum area pre-established, calculate the centroid of the area and report back the pixel position

If the area found is to small it is assumed that the algorithm has made a false detection and the measurement is discarded. This could happen if the load is outside the field of view (FOV) of the camera. Note that the camera is fixed in the MRUAV which means that both load swing and the MRUAV roll and pitch can result in the load disappearing from FOV. When the vision algorithm has detected the pixel position of the load this measurement

Fig. 7.6.: Graphical description of the findContours algorithm. θ φ SL SL Px Py

Fig. 7.7.: 3D spatial location perspective view (left) and 2D camera view (right).

must be mapped to a 2D slung load position. This is done by first transforming the pixel position to two angles θSLand ϕSLas shown in figure 7.7. The angles are calculated as:

ϕSL= KSLPx

θSL= KSLPy

(7.28)

Where Kslis the relationship between the field of view (FOV) of the camera and the number of diagonal pixels of the selected resolution (Eq. 7.29) and [Px, Py] are the reported pixel position of the centroid of the area colour target found.

KSL=

F OV Dpix

Because the camera is gimballed and in the same position of the CoG of the vehicle a rotation is needed (Eq. 7.30) and the position of the camera it is considered the same as the vehicle (Eq. 7.31).

RE C=      cos ψQ sin ψQ 0 − sin ψQ cos ψQ 0 0 0 1      (7.30) ΓE_C = ΓE_Q (7.31)

The position of the slung load can be calculated as:

ΓE_SL= ΓE_C+RE_CΓC_SL (7.32)

Where the position of the slung load in regards with the camera view is given by Eq. 7.33 and the rotation of the camera against the earth frame by Eq. 7.30.

ΓCSL=      LSLsin ϕSL LSLsin θSL LSLcos ϕSLcos θSL      (7.33)

The 2D slung load position calculated with Eq. 7.32 is used by a swing-free controller to prevent aggressive oscillations of the load. Such controller is described in section 7.4.

Implementation

The proposed algorithm must run on-board the companion computer, so that the swing-free controller can use the estimation of the position and then reduce the oscillations of the load, it is important to keep the delay in the vision system low. It is noticed that almost any kind of computer vision algorithm consumes a considerable amount of CPU time, therefore it must be designed to run on a multi-threaded structure so that the other tasks running in parallel with the CV algorithm do not suffer from performance. The DronePilot framework is therefore ideal for this implementation, the CV algorithm is then added as an extra thread that runs in parallel with the control thread (that flies the vehicle), the communications one and the rest of the add-on functions. Several colour tracking algorithm implementations were created and tested, the best results are achieved with the Color-6 1 _{algorithm, that}

is contained on the rpi-opencv open source repository. The rates can be seen in table 7.1. In figure 7.8 the result of the algorithm can be appreciated in a graphic manner with a rectangle being drawn surrounding the area found with the findContours function. On the

Device

Detection @ 640x480px [sec]

Detection @ 500x500px [sec]]

MBPR

0.005

0.003

RPI2

0.15

0.09

RPI3

0.12

0.05

Tab. 7.1.: Performance of the CV algorithm implementation using different CPUs.

real implementation this step is not needed (as no monitor is connected when flying) and it can reduce the load of CPU when removed. Even with the current (2016) most powerful on-

Fig. 7.8.: Frame of a positive colour area found, the slung load position can then be esti- mated.

board credit-card-size companion computer, the algorithm is extremely heavy for the CPU. Achieving 20Hz to get a new slung load position, which in some cases is not fast enough to compensate or attempt a control action to reduce the oscillations.