Open source computer-vision based guidance system for UAVs on-board decision making

This is the author’s version of a work that was submitted/accepted for

pub-lication in the following source:

Choi, Hyunwoong,

Geeves, Mitchell,

Alsalam, Bilal, &

Gonzalez, Luis F.

(2016)

Open source computer-vision based guidance system for UAVs on-board

decision making. In

2016 IEEE Aerospace Conference, 5-12 March 2016, Yellowstone

Confer-ence Center, Big Sky, Montana.

This file was downloaded from:

http://eprints.qut.edu.au/93430/

c

Copyright 2016 IEEE

Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Notice: Changes introduced as a result of publishing processes such as

copy-editing and formatting may not be reflected in this document. For a

definitive version of this work, please refer to the published source:

Abstract— The use of UAVs for remote sensing tasks; e.g. agriculture, search and rescue is increasing. The ability for UAVs to autonomously find a target and perform on-board decision making, such as descending to a new altitude or landing next to a target is a desired capability. Computer-vision functionality allows the Unmanned Aerial Vehicle (UAV) to follow a designated flight plan, detect an object of interest, and change its planned path. In this paper we describe a low cost and an open source system where all image processing is achieved on-board the UAV using a Raspberry Pi 2 microprocessor interfaced with a camera. The Raspberry Pi and the autopilot are physically connected through serial and communicate via MAVProxy. The Raspberry Pi continuously monitors the flight path in real time through USB camera module. The algorithm checks whether the target is captured or not. If the target is detected, the position of the object in frame is represented in Cartesian coordinates and converted into estimate GPS coordinates. In parallel, the autopilot receives the target location approximate GPS and makes a decision to guide the UAV to a new location. This system also has potential uses in the field of Precision Agriculture, plant pest detection and disease outbreaks which cause detrimental financial damage to crop yields if not detected early on. Results show the algorithm is accurate to detect 99% of object of interest and the UAV is capable of navigation and doing on-board decision making.

Keywords: Airborne Vision System, Remote Piloted Aircraft Systems.

T

C

1. INTRODUCTION ... 1

2. SYSTEM ARCHITECTURE ... 2

3. ON-BOARD SOFTWARE SYSTEM ... 3

4. TEST RESULTS ... 3

5. CONCLUSION AND FUTURE WORK ... 4

REFERENCES ... 4

BIOGRAPHY ... 5 978-1-4673-7676-1/16/$31.00 ©2016 IEEE

1. I

There is a growing interest in developing Unmanned Aerial Vehicles (UAV’s) with on-board decision making using way-point guidance and computer vision techniques [1, 2]. Computer vision techniques are regularly used for both military, civilian and agricultural applications [3]. Feedback position control small fixed-wing UAVs have also been used for tracking an unpredictable moving ground vehicle [4]. Target detection based on information gained from surroundings was discussed by Price, et al. [5]. Liu and Dai [6] also discussed computer vision in UAV autonomy. Their research was based on vision-based navigation, aerial surveillance and airborne visual Simultaneous Localization and Mapping. Raja [7] and Yang, et al. [8] discussed landing methods for UAVs using vision algorithms to detect landmarks. Simulated flight video was utilized to check the accuracy of the system [8]. Fu, et al. [9] also mention visual tracking algorithm for UAVs to land on an arbitrary field. The real time algorithm is evaluated with aerial images from auto-landing flights using manual labelled ground truth database. Their results show the algorithm is highly robust to track the helipad and accurate enough for closing the vision-based control loop.

The aim of this and software is to describe a low cost open-source system architecture and software using UAV under 2 Kg. The open-source design of this system involves integration the compact and cost effective open source computing platform (Raspberry Pi 2) with a commercially available open source autopilot (Pixhawk). The Raspberry Pi microprocessors uses a Linux based operating system to execute image processing algorithms. The Raspberry Pi continuously monitors the flight path in real time through a Raspberry Pi camera module.

The rest of this paper is organised as follows: Section II describes system architecture; section III describes the image processing system. Section IV will discuss the results. Final section presents conclusion and future works.

Open Source Computer-Vision Based Guidance System for UAVs

On-Board Decision Making

Hyunwoong Choi Queensland University of Technology Brisbane, Australia, QLD 4001 [email protected] du.au Mitchell Geeves Queensland University of Technology Brisbane, Australia, QLD 4001 mitchell.geeves@student. qut.edu.au Bilal Alsalam Queensland University of Technology Brisbane, Australia, QLD 4001 bilalhazimyounus.alsalam@h dr.qut.edu.au Felipe Gonzalez Queensland University of Technology Brisbane, Australia, QLD 4001 Tel.: +61-7-3138-1363 [email protected]

2. S

A

The system architecture is illustrated in Figure 1. All image processing is designed to be performed on-board the UAV utilising a Raspberry Pi 2 microprocessor interfaced with a camera. The Raspberry Pi and the autopilot (Pixhawk) are physically connected through serial and communicate via MAVProxy. The Raspberry pi is connected to GPS in order to send the location for captured image to the ground station.

A. UAV

The hardware system consists of six components. A 3DR IRIS, manufactured by 3D Robotics is the frame used in this system, see Figure 2. The frame can be modified for different application. The UAV is capable of a payload capacity of 425 grams [10, 11]. A 5000mAh 3S 30C Lipo battery is used to power the system and can produce approximately 15-20 minutes flight time without any payload [10]. Pixhawk autopilot system which contains in built sensors including accelerometer, gyroscope, barometer and magnetometer which are integrated with other subsystems is used [10, 12]. Four AC2830-358 850Kv motors are used to produce high thrust. Two motors run clock wise and others run in counter clock wise in order to provide pitch, roll and yaw control. Also, four 10x4.7 propellers are mounted on each motor to produce thrust to fly the UAV [10].

Figure 1 : System Architecture

Figure 2: System

B. On-board Computer

The Raspberry Pi 2 and the autopilot are connected by the telemetry cable with a 6-Postion DF13 plug and soldering individual PCB female jumper wires as shown in Figure 3. In addition, the DF13 plug is inserted into the secondary telemetry port (Telem2) of the Pixhawk. Both Mission Planner and the Raspberry Pi can receive telemetry data as a failsafe protocol. After connecting the Pixhawk, the female jumper wires are connected to the Raspberry Pi’s serial port pins and ground (Tx, Rx, and Gnd).

C. MAVProxy

MAVProxy is a Ground Control System (GCS) software package for communication between unmanned systems (land or airborne) via the MAVLink protocol. Use of MAVProxy is limited to UAVs developed with supporting hardware. For our system the PixHawk supports MAVLink communication. The Dronekit API, developed by 3D Robotics is utilised as a development platform for the system. Dronekit implemented on the Raspberry Pi 2 allows integration of image processing capability with the onboard sensors of the Pixhawk. Communication operates between the Raspberry Pi 2 and the Pixhawk via serial connection at a baud rate of 57600.

D. Computer vision

OpenCV and MAVProxy were used to develop the image processing system. Both MAVProxy and OpenCV support python language so that UAV control and computer vision algorithm was written in one single software code. The high accuracy of colour detection was achieved by implementing an algorithm using binarization and contouring techniques. Binarization isolates target pixels based on a specific colour range [9]. Contour functions in OpenCV provide location information of the binarized target pixels.

3. O

-

Figure 4 illustrates the on-board software system is divided into five stages. Firstly, the UAV takes off to the designated altitude and mission, this is initiated in stage one.

In step one the UAV starts flying towards the next way point after stage 1. A pre check function determines whether there is a target in an image frame. The pre check function was designed to improve efficiency of the image process. It uses a binarization method if a target is detected; the pre-check triggers the initiate target tracking function.

The tracking function provides location information of the target expressed in Cartesian coordinate system(X, Y) which is sent to a target estimation module function.

Currently, the UAV current location in GPS coordinate is also sent to the target location estimation function. These are shown in stage three.

The target location estimation function performs two main calculations. One that defines the magnetic heading of the UAV based on current location and past location information. A second task is to estimate the location of target in GPS coordinate system and the latitude and longitude of the target which are sent to stages.

The latitude and longitude is updated as a new way point through MAVProxy. UAV’s altitude is adjusted in the lower altitude function. The altitude can be increased or decreased depends on a user preference. In this case, the altitude is lowered for closer target inspection. The take snapshot function takes a picture at this new location or altitude. After stage four, the flight plan is updated. If there is residual flight plan, stages two to four are repeated. Otherwise, the UAV lands and completes its mission.

Figure 4: On-board processing flowchart This process is a fully autonomously system because there is no input required to operate the UAV after stage one is started.

4. T

R

Two tests were conducted in a controlled environment. The first test confirmed the functionality of the detection algorithm and the ability of the system to estimate the target location. The second test ensured the whole steps (1-10) works without issues.

1. Test Case 1

It was required to modify stage four where the low altitude and take snap shot functions were replaced by a landing function. Therefore, the UAV flies toward the target and lands next to the target. The test process is shown on Figure 5. The UAV

Figure 5: Functionality test

lands near the target which is shown in Figure 6. Test Case 1 results showed the estimated target location was close to the actual target location and image process algorithm works during the flight. Five repetitions of Test Case 1 yielded an average offset distance from the target of 87 centimetres.

2. Test Case 2

As shown in Figure 7, a way point was chosen as a final destination. When the UAV finds a target, it flies toward the target location with designated altitude which is 2 meters and then climbs back to the original flight path. The UAV lands on the final way point and the mission terminates.

The “Take Snapshot” process in stage four captures an original image and binary processed images used by the UAV for on-board decision making when the UAV has detected a red target. Figures 8A and B shows and example of how the algorithm perceives the target. Binarization of the RGB image isolates the red target from the surrounding environment.

Figure 6: UAV landed near the target location

Figure 7: Test case 2

(A)

(B)

Figure 8: A) Captured frame when red target detected. B) Binarized Image

5. C

F

W

This paper discussed an on-board decision making capability of UAVs of less than two kilograms. Our system interfaces the OpenCV Python libraries with MAVProxy. This enables the Raspberry Pi 2 to dynamically alter a flight plan if image processing algorithms detect a target. Results demonstrate that our system is capable of on-board target detection to 99% accuracy.

Further development of the system could see it being used for prototyping in the fields of precision Agriculture.

Ongoing research involves replacing the RBG camera for an infrared camera and developing appropriate Python code for such a sensor.

Our Python code is open source and can be obtained via email request to the corresponding author. A video of our flight test can be found here: https://youtu.be/CYMUltPAuA0

R

[1] A. Malaver, F. Gonzalez, A. Depari, P. Corke, and N. Motta, "Towards the development of a gas sensor system for monitoring pollutant gases in the low troposphere using small unmanned aerial vehicles," in Proceedings of Workshop on

Robotics for Environmental Monitoring, Sydney, Australia, 2012.

1 2

3

4

4

3

2

1

5

[2] R. Glassock, J. Y. Hung, L. F. Gonzalez, and R. A. Walker, "Design, modelling and measurement of a hybrid powerplant for unmanned aerial systems," Australian

Journal of Mechanical Engineering, vol. 6,

p. 69, 2008.

[3] H. H. A. Kadouf and Y. M. Mustafah, "Colour-based Object Detection and Tracking for Autonomous Quadrotor UAV," in IOP Conference Series: Materials

Science and Engineering, 2013, p. 012086.

[4] S. A. Quintero and J. P. Hespanha, "Vision-based target tracking with a small UAV: Optimization-based control strategies,"

Control Engineering Practice, vol. 32, pp.

28-42, 2014.

[5] A. Price, J. Pyke, D. Ashiri, and T. Cornall, "Real time object detection for an unmanned aerial vehicle using an FPGA based vision system," in Robotics and

Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on,

2006, pp. 2854-2859.

[6] Y.-c. Liu and Q.-h. Dai, "Vision aided unmanned aerial vehicle autonomy: an overview," in Image and Signal Processing

(CISP), 2010 3rd International Congress on, 2010, pp. 417-421.

[7] V. M. Raja, "Vision based landing for unmanned aerial vehicle," in Aerospace

Conference, 2011 IEEE, 2011, pp. 1-8.

[8] K. Yang, D. Jung, and S. Sukkarieh, "Continuous curvature path-smoothing algorithm using cubic B zier spiral curves for non-holonomic robots," Advanced Robotics, vol. 27, pp. 247-258, 2013.

[9] C. Fu, A. Carrio, M. Olivares-Mendez, and P. Campoy, "Online learning-based robust visual tracking for autonomous landing of Unmanned Aerial Vehicles," in Unmanned

Aircraft Systems (ICUAS), 2014 International Conference on, 2014, pp.

649-655.

[10] 3DRobotics. ''3DR - Drone & UAV Technology'', 2015. [Online]. Available: http://3drobotics.com/, [Accessed 20 March 2015].

[11] Developer. ''APM open source autopilot'',

2015. [Online]. Available:

http://dev.ardupilot.com/, [Accessed 25 March 2015].

[12] pixhawk.org. ''Pixhawk Autopilot'', 2015.

[Online]. Available:

https://pixhawk.org/modules/pixhawk, [Accessed 20- March-2015].

6. B

Hyunwoong Choi is currently studying his Bachelor of Engineering at the Queensland University of Technology, majoring in Aerospace Avionics. Before studying at QUT he undertook complete training to become a Licensed Aircraft Maintenance Engineer (LAME) at Aviation Australia. Whilst there, he studied aircraft mechanical and avionics maintenance engineering. He has extensive knowledge and experience working with aircraft systems from a design and maintenance perspective. His interests are avionics instrumentation, general aviation and unmanned aerial systems. Mitchell Geeves is currently in his final year of study at the Queensland University of Technology, studying a Bachelor of Engineering, majoring in Aerospace Avionics. He is the founding Vice-President of QUT Aerospace Society, formed in 2015, a society for students with a passion for aerospace and developing UAV projects. His interests include airworthiness regulation, aviation and Unmanned Aerial Systems development for civilian uses. Bilal ALSALAM is currently studying a Masters of Engineering at the Queensland University of Technology, developing UAV for agriculture application. He obtained his Bachelor’s of Electrical Engineering at the University of Mosul in 2011. His interests include airworthiness regulation, aviation and Unmanned Aerial Systems. Felipe Gonzalez is a Senior Lecturer (UAV) in the Science and Engineering Faculty, QUT and Theme leader for UAV Remote Sensing at ARCAA. He holds a BEng (Mech) and a PhD from the University of Sydney. His research explores bioinspired optimization, uncertainty based UAV path planning and UAVs for environmental monitoring.