As we have seen, MPCs are often used for path planning in systems where multiple UAVs cooperate to perform different tasks. However, there are few sources where a single UAV and a gimbal system are controlled in order to track multiple moving objects located by a computer vision module. The purpose of this thesis is to merge different technologies to develop and realize an object tracking system where a MPC is used to provide control action to an individually operating UAV equipped with a two-axis gimbal system. The MPC should generate the UAV’s path and the gimbal system’s reference angles based on the UAV’s and the objects’ positions and velocities. A pre-implemented CV module is providing information about objects to be tracked, including object velocities and positions. The main focus for the system development should be search and rescue missions, but the system should be flexible enough to also be capable of performing other missions such as geological surveillance and tracking, or infrastructure inspections. Before proceeding with the system topology, we need to define the term object tracking system.
Definition 1.3.1 (Object tracking system)
By object tracking system it is meant all components that together provides object tracking services, including the UAV and its components, together with a ground control station.
It is important that the solution provides functionality to ensure safety if a fault or an error occurs. For the solution to be considered acceptable, the object tracking system must be able to perform the following tasks:
• Navigate to, and monitor a single stationary object.
• Navigate efficiently between several stationary objects while monitoring the objects. • Efficiently track single and multiple moving objects.
• Choose the sequence of objects to track in a logical and efficient manner.
• Compensate for changes in the UAV’s attitude and use efficient, but not hasty, movements. • Enable remote control from a ground station.
• Provide the system’s operator with the ability to efficiently monitor and utilize the system’s capabilities to the fullest.
• Provide a smooth platform for both the IR and still camera, i.e. reduce vibrations, induced by the engine, and turbulence.
• The MPC has to conform to hardware limitations.
• Operate within the limitations in power supplied from the autopilot. • The payload should never be allowed to jeopardize the UAV’s safety.
4 1.3. Thesis outline
• Endure a sufficient amount of HIL testing with a performance level that is acceptable to start pre-flight and flight tests.
• Perform a sufficient amount of flight tests to ensure the system works as designed. We assume constant knowledge of the objects’ positions as well as the velocities. This information is assumed to be obtained from either an on-board computer vision module, mission control or possibly from the objects themselves. We also assume accurate and dependable real-time UAV measurements, including attitude and altitude. This means we will not discuss the possibilities regarding loss of GPS signals and IMU measurements in any detail. We also want to investigate the possibilities for using the camera’s calculated ground field of view (GFV) to more precisely determine if the objects are within the image frame.
Since the system is intended to be implemented on an UAV platform, we have based the present thesis on a given set of hardware components. The UAV used in this thesis is the Penguin B by UAV Factory, the autopilot is the Piccolo SL developed by Cloud Cap Technology and the gimbal system is the BTC-88 by Micro UAV. A simplified system topology is shown in figure 1.1. The gimbal is controlled by the running environment, running the MPC, through interactions with the Piccolo, which in turn controls the gimbal using a PWM module. The IR camera, FLIR Tau 2 with a 19mm lens provided by FLIR Systems, is installed in the gimbal and is connected to the CV module. The CV module runs on a PandaBoard installed in the UAV’s payload and provides object identification and live image streams to a ground station. In addition to the IR camera, a still camera is installed in the payload. The still camera is meant for delivering snapshots of the ground to the ground station1. The ground
station is equipped with two radio links and a human machine interface (HMI), which enables surveillance and control of the object tracking system. The system includes remote control of the UAV and gimbal, live image streams from the cameras and object identification provided by the CV module. The operator communicates with the UAV through a dedicated HMI, NEPTUS, or additional external HMIs, and the Piccolo Command Center to relay mission information to the control system, or give direct commands to the Piccolo. The MPC is intended to run on the PandaBoard installed in the UAV’s payload, however we suspect the PandaBoard to have insufficient resources. This means if the PandaBoard is not replaced with a more suitable device, the MPC would be a part of the ground station and not the UAV’s payload.
The majority of our effort went into developing a robust control system using a MPC, as well as developing and assembling all hardware components in the UAV’s payload, which are needed to realize the object tracking system. A focus throughout this thesis has been to get a working system prototype in the air for flight testing. The process leading up to flight testing is documented in detail, in the hope that some of it will prove useful to NTNU’s UAVlab in the future. This includes developing thorough procedures for HIL tests, pre-flight checks and flight tests. Some of the time has been spent trying to improve on existing hardware, such as the gimbal, and creating new hardware in the form of a passive wire damping mount for the still camera and power cut-off circuit for the payload. In addition, a simplified HMI (Human Machine Interface) has been implemented to provide remote control of the object tracking system.
1
Chapter 1. Introduction 5
Before we start with mathematical models of the UAV in chapter 3 we briefly present computer vision in chapter 2. In chapter 4 we use known theory for mapping the extent of the camera’s field of view onto the earth’s surface. In chapter 5 we introduce Model Predictive Control, present a solution for controlling both the gimbal’s pan and tilt angles, and the UAV’s path using way-points. We present the design of a control system in chapter 6 which is intended to run the MPC and provide interactions with additional systems. A simplified HMI system implementation, enabling remote control of the object tracking system, is described in chapter 7. In chapter 8 we present simulation results of the object tracking system. The hardware setup is discussed in chapter 9 before HIL and field tests are represented in chapter 10 and 11. The conclusion, in chapter 12, briefly revisit our most significant findings. The final chapter of this thesis is a list of further work, where we suggest recommended features and improvements to the object tracking system.
Chapter 2
Computer vision
Computer vision (CV) technology is essential in an object tracking system. This is because a camera sensor provides important measurements, such as object positions and velocities, which are vital for detecting objects and determining their relevance, thus ensuring real-time feedback to the system. In this chapter we briefly discuss some of the aspects of computer vision technology in conjunction with object tracking. In section 2.1 we will provide a basic introduction to the CV technology. In section 2.2 we present practical use of CV implemented on an UAV platform to detect objects of interest using a camera sensor. In the last section of this chapter we briefly describe the CV module, which is assumed to be implemented on the UAV platform used in this thesis.
2.1 Introduction to computer vision
Computer vision, or machine vision (MV), is the discipline where images and videos are processed and analyzed to extract data and information. The ultimate goal for CV is to use computer software and hardware to extract as much useful information from an image as possible. CV could be used to detect and identify specific information by processing images and videos in combination with knowledge from different fields, e.g. computer science, mathematics, physics and engineering science. The CV hierarchy can be divided in three levels (Ji, 2014):
• Low-level vision: Process images for feature extraction (edges, corners or optical flow). • Middle-level vision: Object recognition, motion analysis and 3D reconstruction using
features obtained from low-level vision.
• High-level vision: Interpretation of the evolving information provided by the middle level vision as well as directing what middle and low level vision tasks should be performed. Interpretation may include conceptual description of a scene like activity, intention and behavior.
In many control systems object detection is used to provide information from the data collected by a camera sensor. Object detection and identification is the process of searching image data for recognizable shapes and identify whether they are of interest. This process can be divided in two parts, namely a recognition algorithm and a learning algorithm. The learning algorithm is fed with image data, both positive images, which includes image data, shapes and patterns of interest, and negative images, which includes image data, shapes and patterns of no interest, to develop a classifier. This process, to develop a classifier, is often referred to as training a classifier. The recognition algorithm uses the classifier when processing image data to extract useful information. The classifier is used to identify the objects in the image found by the recognition algorithm and compares the recognized part of the image with stored reference classes and finds the best match. The set of classes could contain only one reference. This would be described as a two-class classification, resulting in a true/false answer to whether the recognized part of the image is the object one is looking for. The stored references are referred