Soft-computing based visual control for unmanned vehicles

Full text

(1)Universidad Politécnica de Madrid Escuela Técnica Superior De Ingenieros Industriales Doctorado en Automática y Robótica. Soft-Computing Based Visual Control for Unmanned Vehicles. A thesis submitted for the degree of Doctor of Philosophy in Robotics and Automation. ´ Miguel Angel Olivares Méndez BS Computer Engineer. 2013.

(2)

(3) Escuela Técnica Superior De Ingenieros Industriales Doctorado en Automática y Robótica Universidad Politécnica de Madrid Soft-Computing Based Visual Control for Unmanned Vehicles A thesis submitted for the degree of Doctor of Philosophy in Robotics and Automation Author: ´ Miguel Angel Olivares Méndez BS Computer Engineer Director: Dr. Pascual Campoy Cervera PhD Full Professor Industrial Engineer. 2013.

(4)

(5) T´ıtulo:. Soft-Computing Based Visual Control for Unmanned Vehicles Autor: ´ Miguel Angel Olivares Méndez BS Computer Engineer. Tribunal nombrado por el Mgfco. y Excmo. Sr Rector de la Universidad Politécnica de Madrid el d´ıa.........de...................de 2011. Tribunal Presidente. :. ......................................., ............. Secretario. :. ......................................., ............. Vocal. :. ......................................., ............. Vocal. :. ......................................., ............. Vocal. :. ......................................., ............. Suplente. :. ......................................., ............. Suplente. :. ......................................., ............. Realizado el acto de lectura y defensa de la tesis el d´ıa ........... de ................ de 2011.. calificación de la Tesis.......................................... El Presidente:. El Secretario:. Los Vocales:.

(6)

(7) A mi hermano y a mi prima Patri por su apoyo, A mis padres, por haberme enseñado a creer en mi, A Eduard Punset, por haberme enseñado a creer en la ciencia, A Pascual Campoy, por creer tanto en mi, A mi Lu-z, por el impulso diario, sin su energ´ıa hubiera sido imposible, A Marco.. Miguel A. Olivares Mendez. V.

(8)

(9) Acknowledgements Son muchas las personas que han hecho posible que haya conseguido llegar a escribir estas lineas en este documento. En primer lugar quisiera agradecer a la gran mayor´ıa de miembros del Computer Vision Group, entre otros, a Sergio Dominguez, por darle un poco más de alegr´ıa al d´ıa a d´ıa, Iván Mondragón, por estar siempre disponible a ayudar en todo momento, Carol Mart´ınez, por su ayuda y gran disposició, Ignacio Mellado, por su apoyo y colaboración en los momentos dificiles del proyecto de Siemens, José Lu´ıs, por estar siempre disponible a ayudar con un sonrisa, David Galindo, por su colaboración y ayuda, Paloma, por ayudarme tanto en la fase final de los papeleos de la tesis, a Ramón, por su ayuda en el poryecto Omniworks y especialmente a Changhong Fu por su inestimable ayuda y motivació durante le u´ ltimo año de grandes esfuerzos. A todos ellos gracias por hacer tan especial y agradable mi docotrado. Quisiera dar las gracias también a la gente del departamento, sufridores también conmigo del d´ıa a d´ıa en el laboratorio. Lisandro, Carlos, Ińaki, Roberto, Gonzalo, Gabriel, Alberto, Joao, y alos que se fueron de vuelta Hector, Isela y Alberto Trasloceros. También quisiera dar las gracias al personal del DISAM, en especial a Rosa, Tere y Carlos, por su ayuda y su infinita paciencia conmigo. A los profesores qye hacen posible que siga funcionando el departamento y que mi estancia aqui haya sido tan satisfactoria, Ricardo Sanz, Manuel Ferre, Claudio Rossi, Roque Saltaren, Antonio Barrientos, Enrique Pinto, Jaime, Angel, Rafael y Agust´ın. Gracias también al Ministerio de Educación y Ciencia ayuda en la subvención del proyecto DPI2007-66156 “Visión por computador para UAV: Navegación, seguimiento e inspección”. A la Universidad Politécnica de Madrid for su ayuda en la financiación de las estancias docotrales internacional y europea, asi como a la Comunidad Europea por financiar el proyecto ”IRSES FP7-IRSES IPCUAS”.

(10) Me gustar´ıa agradecer también al ”Australian Research Centre for Aerospace Automation” ARCAA - Australia y al ”Laboratory of Intelligent System” del EPFL - Lausanne por acogerme en mis estancias doctorales y al profesor Felix Schill, y especialmente a Luis Mej´ıas por su apoyo, compañ´ıa y por los buenos momentos pasados en Australia. A Andrades y Clara por tantos buenos ratos y por sus inspiradores vinitos. Al señor Victrol por su apoyo y animos. A los tantos amigos que he conocido, me han ayudado y animado durante todo este tiempo Mónica, Evelina, Cesar, Valeria, Rafa, Filippo, Laura, la pequeña Lydia, y especialmente a Raúl (Baloo) por tantas enriquecedoras conversaciones. Y finalmente a toda mi familia, a mis segundos padres Puri y Manolo, por sus animos constantes, a mis primas Julia, Ana y Alba. A mi abuelo Celestino, a Rafa, Nieves y Paola. Tambien quiero expresar mi gratitud de manera muy especial a Mirella, por su apoyo, animo incondicional y por haberme dado el regalo más grande. Y a los que se han convertido ya en parte de familia Pascual, Ana, Pablo, Alicia y Miguelito, gracias por todos los buenos momentos, especialmente a Pascual por haber creido en mi durante todo este tiempo, por darme una “visión” nueva de la investigación y por ayudarme en todos los momentos dificiles del doctorado.. Gracias a todos de corazón. Miguel A. Olivares Mendez.

(11) Resumen El objetivo principal de esta Tesis es extender la utilización del “SoftComputing” para el control de vehiculos sin piloto utilizando visión. Este trabajo va más allá de los t´ıpicos sistemas de control utilizados en entornos altamente controlados, demonstrando la fuerza y versatilidad de la logica difusa (Fuzzy Logic) para controlar vehiculos aereos y terrestres en un amplio abanico de applicaciones. Para esta Tesis se ha realizado un gran número de pruebas reales en las cuales los controladores difusos han manejado una plataforma visual “pan-and-tilt”, un helicoptero, un coche comercial y hasta dos tipos distintos de quadrirotores. El uso del método de optimización “Cross-Entropy” ha sido utilizado para el ajuste de algunos parámetros de los controladores mejorando as´ı su comportamiento para la evitación de colisiones con veh´ıculos aéreos. Todos los controladores difusos presentados en e´ sta Tesis han sido implementados utilizando la librer´ıa de C++ desarrollada por el candidato para tal efecto, llamado MOFS (Miguel Olivares’ Fuzzy Software). Diferentes algoritmos visuales han sido utilizados para adquirir la información visual del entorno, “Camshift”, descomposición de la homograf´ıa y detección de marcas de realidad aumentada, entre otros. Dicha información visual ha sido utilizada como entrada de los controladores difusos para comandar los veh´ıculos en las diferentes applicacines autonomas. El volante de un veh´ıculo comercial ha sido controlado para realizar pruebas de conducción autónoma en condiciones de tráfico similares a las de una ciudad. El sistema ha llegado a completar con e´ xito pruebas de más de 6 km sin ninguna interacción humana, mediante el seguimiento de una l´ınea pintada en el suelo. El limitado campo visual del sistema (50 × 30 cm) no ha sido impedimento para alcanzar velocidades de hasta 48 km/h y ser guiado autonomamente en curvas de radio reducido. Objetos estáticos y móviles han sido seguidos desde un helicoptero no trip´ ulado, mediante el control de una plataforma visual “pan-and-tilt”. Este mismo helicoptero ha sido controlado completamente para su aterrizaje autonomo,.

(12) X. RESUMEN. mediante el control del movimiento lateral (roll), horizontal (pitch) y en altitud. El seguimiento de objetos voladores desde una distancia de seguridad ha sido resuelto mediante el control horizontal (pitch) y de orientación (heading) de un quadrirotor. Para tareas de evitación de obstáculos se ha implementado un controlador difuso para el manejo de la orientación (heading) de un quadrirotor. En el campo de la optimización de controladores se ha aportado al estado del arte una extensión del uso del método “Cross-Entropy”. Está Tesis presenta una novedosa implementación de dicho método para la optimización de las ganancias, la posición y tamaño de los conjuntos difusos de las funciones de pertenecia y el peso de las reglas para mejorar el comportamiento de los controladores difusos. Dichos procesos de optimización se han realizado utilizando “ROS” y “Matlab Simulink” para obteniendose grandes mejoras en la evitación de colisiones con veh´ıculos aéreos no tripulados. ´ Esta Tesis demuestra que los controladores basados en lógica difusa son altamente capaces de controlador sistemas sin tener en cuenta el modelo del veh´ıculo a controlador, estando en entornos altamente perturbables con un sensor de bajo coste como es una cámara. El ruido presente en la adquisición de imagenes causado por los cambios de iluminación y la alta incertidumbre en la detección visual han sido manejados satisfactoriamente por e´ sta técnica de de “Soft-Computing” para distintas aplicaciones tanto con veh´ıculos aéreos como terrestres..

(13) Abstract The aim of this Thesis is to exploit the use of Soft Computing to control unmanned vehicles using vision. This works goes beyond the typical control systems used in highly controlled environments, by demonstrating the power of the Fuzzy Logic Controllers (FLCs) to command aerial and ground vehicles in a sort of different tasks. A huge amount of real tests are presented in which the implemented Fuzzy controllers manage a visual pan and tilt platform, a helicopter, a commercial car and two different types of quadcopters. The use of the Cross-Entropy method to optimize the behavior of these controllers is also shown. All the visual servoing controllers presented in this Thesis were implemented using the self-developed software tool called MOFS (Miguel Olivares’ Fuzzy Software). Different visual algorithms were used to acquire the information of the surrounding environment of the vehicles. The CamShift, homography decomposition, and augmented reality mark detection among others. This visual information was used as input of the Fuzzy controllers to manage the vehicle to do different autonomous tasks. The steering wheel of a commercial car was controlled to implement a driverless vehicle for inner-city tests. Long distance of more than 6 km was covered without driver in a close circuit using a vision line following algorithm. The limited field of view (50 × 30 cm) of the system was not an impediment to reach a top speed of 48km/h and guide the vehicle inside low radius curves. Static and moving objects like cars were tracking from an unmanned helicopter controlling an on board pan and tilt visual platform. A full control of altitude, lateral and forward movements was implemented for an auto-landing task of a helicopter. An implementation of pitch and heading controllers were used to command a quadrotor for object following task. The heading was also controlled for See and Avoid task with this type of UAVs. The Cross-Entropy optimization method is not wide used for control in the.

(14) XII. ABSTRACT. literature. This Thesis presents the way to optimize the gains, membership function sets’ position and size and the rules’ weight to improve the behavior of a Fuzzy controller. This optimization process was done using ROS and Matlab Simulink to obtain better results for See and Avoid tests for UAVs. This Thesis demonstrates that the Fuzzy Logic Controllers are widely capable to command free-model systems in high disturbance environments with a low cost sensor. The noisy effects of illumination changes and the high uncertain of the visual detection were manage in a gentle way by this Soft Computing technique to approach different tasks with different aerial and ground vehicles..

(15) Nomenclature The conventions used in this document are defined as follows: Matrix will be represented by capital bold letters such as H or R. Vectors will be represented by a lower case bold letters with or without a upper bar, such as x¯ or t. A vector is a column vector, unless indicated otherwise. Scalar variables are normal italic letters such as i or k. Images are normally represented by capital italic letters such as I or T..

(16)

(17) Acronyms Following is a description of most commonly acronyms referenced on this work: ATC: Air Traffic Control ATM: Air Traffic Management CAA: United Kingdom Civil Aviation Authority CASA: Australian Civil Aviation Safety Authority CPU: Central Processing Unit DOF: Degrees of freedom EASA: European Aviation Safety Agency FAA: Federal Aviation Administration FOV: Field Of View GPS: Global Positioning System GPU: Graphics Processing Unit HOT: Hight Order Terms IBVS: Image-Based Visual Servoing ICIA: Inverse Compositional Image Alignment IMU: Inertial Measurement Unit ITSE: Integral Time of Square Error ITAE: Integral Time of Absolute Error KF: Kalman Filter.

(18) XVI LKT: Lucas-Kanade Tracker MAV: Micro-Aerial Vehicles. MTOW: Maximum Take-Off Weight NAAs: National Aviation Authorities NAS: National Airspace System PBVS: Position-Based Visual Servoing PID: : Proportional-Integral-Derivative controller RANSAC: RANdom SAmple Consensus RC: Radio Controlled RMSE: Root Mean Square Error RPM: Revolutions per Minute R&D: Research and Development SIFT: Scale-Invariant Feature Transform SLAM: Simultaneous Localization And Mapping SURF: Speeded Up Robust Features UAS: Unmanned Aircraft System UAV: Unmanned Aerial Vehicle VTOL: Vertical Take-Off and Landing. ACRONYMS.

(19) Contents Acknowledgements. VII. Resumen. IX. Abstract. XI. Nomenclature. XIII. Acronyms. XV. List of Figures. XXI. List of Tables. XXXI. 1. Introduction 1.1. Motivation and Problem Statement . . . . . . . . . . . . . . . . 1.2. Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . 1.3. Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. State of Art 2.1. Introduction . . . . . . . . . . . 2.2. Unmanned Vehicles . . . . . . . 2.2.1. Perception . . . . . . . 2.2.2. Localization . . . . . . . 2.2.3. Planning and Navigation 2.3. Visual Control . . . . . . . . . . 2.4. Fuzzy Control . . . . . . . . . . XVII. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 1 1 3 4 7 7 8 8 9 11 11 13.

(20) XVIII. Contents. 3. Fuzzy Logic Control for Ground Vehicles 3.1. Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Unmanned Ground Vehicle Description . . . . . . . . . . . . . 3.3. Visual Localization Algorithms . . . . . . . . . . . . . . . . . . 3.3.1. Lateral Estimation Error . . . . . . . . . . . . . . . . . 3.3.2. Global Localization by Visual Marks . . . . . . . . . . 3.4. Autonomous Driving with Local Localization . . . . . . . . . . 3.4.1. Fuzzy Control for Lateral Estimation . . . . . . . . . . 3.4.2. Experiments . . . . . . . . . . . . . . . . . . . . . . . 3.5. Autonomous Driving with Global Localization . . . . . . . . . 3.5.1. Fuzzy Control for Lateral Estimation and Global Localization . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2. Experiments . . . . . . . . . . . . . . . . . . . . . . . 3.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 17 18 20 22 22 23 25 25 29 35. 4. Fuzzy Logic Control for Unmanned Helicopters 4.1. Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Unmanned Helicopter Description . . . . . . . . . . . . . . . . 4.3. Eye in the sky: Autonomous Visual Servoing for Object Tracking 4.3.1. Visual Tracking . . . . . . . . . . . . . . . . . . . . . . 4.3.2. Fuzzy Control of a Pan and Tilt Visual Platform and UAV’s Heading . . . . . . . . . . . . . . . . . . . . . . 4.3.3. Experiments . . . . . . . . . . . . . . . . . . . . . . . 4.3.3.1. Laboratory Experiments . . . . . . . . . . . . 4.3.3.2. Real Tests with the UAV. . . . . . . . . . . . 4.4. Autonomous Landing . . . . . . . . . . . . . . . . . . . . . . . 4.4.1. 3D Estimation Based on Homographies . . . . . . . . . 4.4.2. Fuzzy Control of UAV’s Pitch, Roll and Altitude. . . . . 4.4.3. Experiments . . . . . . . . . . . . . . . . . . . . . . . 4.4.3.1. 3D positioning experiments . . . . . . . . . . 4.4.3.2. Autonomous Descend Experiments . . . . . . 4.4.3.3. Fully Autonomous Landing . . . . . . . . . . 4.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 53 54 55 57 57. 5. Fuzzy Logic Control for Unmanned Quadrotors 5.1. Related Works . . . . . . . . . . . . . . . . . . . . . . . 5.2. Unmanned Quadrotors Description . . . . . . . . . . . . 5.2.1. Basic Quadrotor Mechanics . . . . . . . . . . . 5.2.2. Pelican Ascending Technologies Quadcopter . . 5.2.3. AR.Drone Parrot Quadrotor . . . . . . . . . . . 5.3. Color-Based Tracking . . . . . . . . . . . . . . . . . . . 5.4. Object Following . . . . . . . . . . . . . . . . . . . . . 5.4.1. Fuzzy Control of Quadrotor’s Pitch and Heading 5.4.2. Experiments . . . . . . . . . . . . . . . . . . . 5.5. Avoiding Obstacles . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. 36 36 51. 57 61 61 65 71 71 75 80 80 84 85 90 91 92 93 93 94 94 95 96 98 101 106.

(21) Contents. XIX. 5.5.1. Fuzzy Control of Quadrotor Heading . . . . . . . . . . 106 5.5.2. Experiments . . . . . . . . . . . . . . . . . . . . . . . 109 5.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6. Optimization of Fuzzy Logic Controllers using the Cross-Entropy method 113 6.1. Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 6.2. Cross-Entropy optimization method . . . . . . . . . . . . . . . 115 6.3. Fuzzy Control Optimization General Approach . . . . . . . . . 117 6.4. Gains optimization of Fuzzy controllers: Applied to UAV See and Avoid task. . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.4.1. Color-Based Tracking . . . . . . . . . . . . . . . . . . 118 6.4.2. Scaling Factor Optimized Fuzzy control of Quadrotors’ Heading for See and Avoid . . . . . . . . . . . . . . . . 119 6.4.3. Optimization process using the ROS-Gazebo simulation environment . . . . . . . . . . . . . . . . . . . . . . . 120 6.4.3.1. Software Implementation . . . . . . . . . . . 120 6.4.3.2. Simulator Experiments . . . . . . . . . . . . 122 6.4.4. Experiments . . . . . . . . . . . . . . . . . . . . . . . 125 6.5. Membership Functions’ sets and Rules’ Weight optimization of Fuzzy controllers: Applied to UAV See and Avoid task. . . . . . 131 6.5.1. Object detection and Positioning Estimation using QR codes . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.5.2. Membership Functions and Rules Optimized Fuzzy control of Quadrotors’ Heading for See and Avoid . . . . . 132 6.5.3. Optimization process using the Matlab-Simulink simulation environment . . . . . . . . . . . . . . . . . . . . 136 6.5.3.1. Software Implementation . . . . . . . . . . . 136 6.5.3.2. Simulator Experiments . . . . . . . . . . . . 140 6.5.4. Experiments . . . . . . . . . . . . . . . . . . . . . . . 152 6.6. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 7. Thesis Conclusions. 163. A. Publications derived from this thesis A.1. Journals with JCR . . . . . . . . . . . . . . . A.2. Peer-Reviewed Congress . . . . . . . . . . . A.3. Book Chapters . . . . . . . . . . . . . . . . A.4. Digital media . . . . . . . . . . . . . . . . . A.5. Participation on Technology Transfer Projects A.6. Research Positions . . . . . . . . . . . . . .. 167 167 168 171 172 172 173. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. B. MOFS: Miguel Olivares’ Fuzzy Software 175 B.0.1. How to use MOFS . . . . . . . . . . . . . . . . . . . . 177.

(22) XX C. UAV specifications C.1. Gas and Electric powered UAV helicopters . . C.1.1. Helicopter Autopilot API . . . . . . . C.2. UAV quadcopter . . . . . . . . . . . . . . . . C.2.1. AscTec Pelican . . . . . . . . . . . . C.2.1.1. Quadcopter autopilot SDK D. Research Groups on UAS and Vision Systems. Contents. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 181 181 181 185 185 186 189.

(23) List of Figures 1.1. Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . .. 5. 2.1. Position-Based Visual Control. . . . . . . . . . . . . . . . . . . . 2.2. Image-Based Visual Control. . . . . . . . . . . . . . . . . . . . .. 12 13. 3.1. 3.2. 3.3. 3.4. 3.5.. 18 19 19 20. Driverless Car developed by Sebastian Thrun et. altres . . . . . Driverless Car projects before 2000. . . . . . . . . . . . . . . . SARTRE Volvo project. . . . . . . . . . . . . . . . . . . . . . . VisLab driverless van, tested from Parma to Shangai. . . . . . . The high level of traffic nowadays in every city make the detection of the lane almost impossible. . . . . . . . . . . . . . . . . 3.6. Automated Citroën C3 Pluriel . . . . . . . . . . . . . . . . . . 3.7. Metal structure of the visual system. Limits the field of view of the camera to 30 x 50 cm. . . . . . . . . . . . . . . . . . . . . . 3.8. Representation of the circuit at INSIA installations took with Google Earth tool. . . . . . . . . . . . . . . . . . . . . . . . . . 3.9. Line and mark detection by the comnputer vision algorithm. . . 3.10. Control loop of the visual servoing system for lateral control. FuzzyPD + I schema. . . . . . . . . . . . . . . . . . . . . . . . 3.11. First input variable of the Fuzzy controller: the error between the center of the line and the center of the image, in pixels. . . . . . 3.12. Second input variable of the Fuzzy controller: the difference between the last error and the current, in pixels. . . . . . . . . . . 3.13. Output variable of the Fuzzy controller: the steering wheel angle, in degrees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XXI. 20 21 22 22 25 26 26 27 27.

(24) XXII. List of Figures. 3.14. Representation of the steering wheel response. The delay from the command sent to the steering wheel reach the commanded angle position is 0.4 seconds. . . . . . . . . . . . . . . . . . . . 3.15. Step response to a 50 pixels step at 10 km/h in straight. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 7.1666 cm. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.16. Step response to a 50 pixels step at 15 km/h in straight. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 7.0592 cm. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.17. Step response to a 50 pixels step at 10 km/h in curve. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 7.8007 cm. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.18. Step response to a 50 pixels step at 15 km/h in curve. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 7.2187 cm. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.19. Evolution of the system during 18 laps inside the circuit without mark detection. Constant speed was applied. . . . . . . . . . . . 3.20. Zoom to one lap of the circuit. . . . . . . . . . . . . . . . . . . 3.21. Zoom of 170 pixels step at the beginning of the second curve. . . 3.22. Evolution of the system during the emergency stop’s test. . . . . 3.23. Evolution of the system during 4 laps inside the circuit without mark detection. Variable speed was applied. . . . . . . . . . . . 3.24. Control loop of the visual servoing system with global localization. Fuzzy + I + O f f set schema. . . . . . . . . . . . . . . . . 3.25. Step response to a 50 pixels step at 10 km/h in straight with mark approach system. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 6.8402 cm. . . . . . . . . . 3.26. Step response to a 50 pixels step at 15 km/h in straight with mark approach system. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The RMSE value of the test is 6.719 cm. . . . . . . . . . . . . . 3.27. Step response to a 50 pixels step at 10 km/h in curve with mark approach system. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 5.1034 cm. . . . . . . . . . 3.28. Step response to a 50 pixels step at 15 km/h in curve with mark approach system. The Error measurement, and the steering wheel movements, and the Fuzzy controller output are shown. The value of the RMSE of the test is 5.5429 cm. . . . . . . . . .. 29. 30. 30. 31. 31 32 33 33 34 35 36. 37. 37. 38. 38.

(25) List of Figures. XXIII. 3.29. Evolution of the system during 9 laps inside the circuit with mark detection. Speed is set to 15 km/h. . . . . . . . . . . . . . . . . 3.30. Evolution of the system during this test 13 laps inside the circuit with mark detection. Variable speed with an average of 20 km/h. 3.31. Evolution of the system during 13 laps inside the circuit with mark detection. Speed is set to 15 km/h. . . . . . . . . . . . . . 3.32. Evolution of the system during 21 laps inside the circuit with mark detection. Speed is set to 14 km/h. An emergency stop was applied. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.33. Evolution of the system during 30 laps inside the circuit with mark detection. Speed is set to 15 km/h. . . . . . . . . . . . . . 3.34. Evolution of the system with mark detection and speed changes from 0 to 48 km/h. . . . . . . . . . . . . . . . . . . . . . . . . 3.35. Evolution of the system during 17 laps inside the circuit with mark detection. Speed changes from 15 to 25 km/h. . . . . . . . 3.36. Evolution of the system during 4 laps inside the circuit with mark detection. Speed changes from 6 to 38 km/h. . . . . . . . . . . . 3.37. Evolution of the system during 6 laps inside the circuit with mark detection. Speed changes from 14 to 42 km/h. . . . . . . . . . . 3.38. Evolution of the system during 4 laps inside the circuit with mark detection. Speed changes from 14 to 42 km/h. . . . . . . . . . . 3.39. Evolution of the system during 14 laps inside the circuit with mark detection. Speed changes from 10 to 23 km/h. . . . . . . . 3.40. Evolution of the system during 17 laps inside the circuit with mark detection. Speed changes from 14 to 24 km/h. . . . . . . . 4.1. 4.2. 4.3. 4.4. 4.5. 4.6.. 4.7. 4.8. 4.9.. 4.10.. Autonomous electric helicopter SR20. . . . . . . . . . . . . . . Autonomous gas-powered helicopter SRA1. . . . . . . . . . . . Visual platform yaw controller. . . . . . . . . . . . . . . . . . . Visual platform pitch controller. . . . . . . . . . . . . . . . . . Heading controller. . . . . . . . . . . . . . . . . . . . . . . . . Different rules-base and output definition fuzzy systems used in laboratory tests. a) Without more significant sector defined in the output. b) With a output minor range of action than the one show in Figure 4.5(c), sets in -10, -7.5, -5, -2.5, 0, 2.5, 5, 7.5 and 10. c) the selected one shown in Figure 4.5(c) . . . . . . . . . . . . A sample window of the helicopter Simulator. . . . . . . . . . . Error between the static object tracked and the center of the image, running with the UAV simulator. . . . . . . . . . . . . . . Response of the Fuzzy control for the Pitch axis of the visual platform tracking a static object with the simulator of the UAV control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Response of the Fuzzy control for the Yaw axis of the visual platform tracking a static object with the simulator of the UAV control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 40 40 41. 42 43 44 45 46 47 48 49 50 55 56 59 59 60. 62 63 63. 63. 64.

(26) XXIV. List of Figures. 4.11. Response of the Fuzzy control for the heading of the helicopter.. 64. 4.12. Heading movement. . . . . . . . . . . . . . . . . . . . . . . . .. 64. 4.13. 3D flight reconstruction from the GPS and the IMU data from the UAV. Where, the ’X’ axis represents the NORTH axis of the surface of the tangent of the earth, the ’Y’ axis represents the EAST axis of the earth, the ’Z’ is the altitude of the helicopter and the red arrows show the pitch angle of the helicopter. . . . .. 65. 4.14. Different pitch and yaw movements of the UAV. . . . . . . . . .. 66. 4.15. Error between center of the image and center of the object to track. 66 4.16. Output from the Fuzzy Controller. . . . . . . . . . . . . . . . . . .. 67. 4.17. Input and Output Variables for the Pitch and Yaw controllers. . .. 68. 4.18. 3D flight reconstruction using the GPS and the IMU data from the UAV. Where, the ’X’ axis represents the NORTH axis of the surface of the tangent of the earth, the ’Y’ axis represents the EAST axis of the earth, the ’Z’ is the altitude of the helicopter and the red arrows show the pitch angle of the helicopter. . . . . . . . . . . . . . . . . . . . . .. 69. 4.19. Velocity changes of the UAV. . . . . . . . . . . . . . . . . . . .. 69. 4.20. Axis changes of the UAV in degrees. . . . . . . . . . . . . . . .. 70. 4.21. Error between center of the image and center of the object to track. . .. 70. 4.22. Error between center of the image and center of the dynamic object (a van) to track. . . . . . . . . . . . . . . . . . . . . . .. 71. 4.23. Response of the Fuzzy control for the Yaw axis of the visual platform tracking a dynamic object (a van). . . . . . . . . . . .. 71. 4.24. Response of the Fuzzy control for the Pitch axis of the visual platform tracking a dynamic object (a van). . . . . . . . . . . .. 71. 4.25. Projection model on a moving camera and frame-to-frame homography induced by a plane. . . . . . . . . . . . . . . . . . .. 73. 4.26. Fuzzy controller for UAV’s altitude. . . . . . . . . . . . . . . .. 76. 4.27. Fuzzy controller for UAV’s roll. . . . . . . . . . . . . . . . . .. 77. 4.28. Fuzzy controller for UAV’s pitch. . . . . . . . . . . . . . . . . .. 78. 4.29. Helipad, camera and U.A.V coordinate systems . . . . . . . . .. 80. 4.30. Measures of the X axis of the UAV (Roll) based on the homography estimation. . . . . . . . . . . . . . . . . . . . . . . . . .. 81. 4.31. Measures of the Y axis of the UAV (Pitch) based on the homography estimation. . . . . . . . . . . . . . . . . . . . . . . . . .. 82. 4.32. Measures of the altitude of the UAV based on the homography estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 83. 4.33. Measures of the heading of the UAV (yaw) based on the homography estimation. . . . . . . . . . . . . . . . . . . . . . . . . .. 83.

(27) List of Figures. XXV. 4.34. 3D pose estimation based on a helipad tracking using Robust Homography estimation during a UAV landing process. Landing flight test beginning at an altitude of 5.2m. For all images, the reference image I0 is on the small rectangle on the upper left corner. Left it the current frame and Right the Optical Flow between the actual and last frame. Superimposed are the projection of the original rectangle, the translation vector and the Tait-Bryan angles. 4.35. 3D flight and heading reconstruction for the flight sequence show on figure 4.34 (flight test 1) . . . . . . . . . . . . . . . . . . . . 4.36. Comparison between the Z axis displacement for homography estimation and IMU data during the test. . . . . . . . . . . . . . 4.37. 3D flight reconstruction of first test of fully autonomous landing. 4.38. Homography estimation for pitch, roll and altitude of second test of fully autonomous landing. . . . . . . . . . . . . . . . . . . . 4.39. 3D flight reconstruction of second test of fully autonomous landing. 4.40. Homography estimation for pitch, roll and altitude of second test of fully autonomous landing. . . . . . . . . . . . . . . . . . . . 4.41. 3D flight reconstruction of third test of fully autonomous landing. 4.42. Homography estimation for pitch, roll and altitude of third test of fully autonomous landing. . . . . . . . . . . . . . . . . . . . 4.43. 3D flight reconstruction of fourth test of fully autonomous landing. 4.44. Homography estimation for pitch, roll and altitude of fourth test of fully autonomous landing. . . . . . . . . . . . . . . . . . . . 5.1. Quadrotor mechanics. Motors 1 and 3 spin clockwise and motors 2 and 4 spin counter-clockwise. Picture from www.wikipedia.org 5.2. CVG-UPM (CVG, 2012) Pelican Quadrotor UAV used for Aerial Object Following tests . . . . . . . . . . . . . . . . . . 5.3. AR.Drone Parrot . . . . . . . . . . . . . . . . . . . . . . . . . 5.4. Camshift tracking of a colored red target on a image sequence. The white circle corresponds to the boundaries of the tracked colored area . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5. Image captured with the onboard camera. . . . . . . . . . . . . 5.6. Aerial Object Following application described with the onboard and external images of a real situation. . . . . . . . . . . . . . . 5.7. Definition of the Yaw controller: First input. Estimation of the deviation of the object from the centre of the image capture from the UAV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8. Definition of the Yaw controller: Second input. Difference between the last two measurements. . . . . . . . . . . . . . . . . 5.9. Definition of the Yaw controller: Output. Velocity commands to change the heading of the UAV. . . . . . . . . . . . . . . . . . . 5.10. Definition of the Pitch controller: First input. Estimation of the deviation of the object from the centre of the image capture from the UAV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 84 85 85 86 87 87 88 88 88 89 89 94 95 95. 96 97 97. 98 98 99. 99.

(28) XXVI. List of Figures. 5.11. Definition of the Pitch controller: Second input. Difference between the last two measurements. . . . . . . . . . . . . . . . . 5.12. Definition of the Pitch controller: Output. Velocity commands to change the heading of the UAV. . . . . . . . . . . . . . . . . 5.13. Object Following Dynamic image-based look and move fuzzy servoing system architecture. . . . . . . . . . . . . . . . . . . . 5.14. Trajectory of the Pelican-UAV during the First Aerial Object Following Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.15. Measurements for the distance between the UAV and the aerial object and response of the Pitch controller of the first test. . . . . 5.16. Measurements for the orientation of the UAV and response of the Yaw controller of the first test. . . . . . . . . . . . . . . . . . . 5.17. Trajectory of the Pelican-UAV during the Second Aerial Object Following Test. . . . . . . . . . . . . . . . . . . . . . . . . . . 5.18. Measurements for the distance between the UAV and the aerial object and response of the Pitch controller of the second test. . . 5.19. Measurements for the orientation of the UAV and response of the Yaw controller of the second test. . . . . . . . . . . . . . . . . . 5.20. Trajectory of the Pelican-UAV during the Third Aerial Object Following Test. . . . . . . . . . . . . . . . . . . . . . . . . . . 5.21. Measurements for the distance between the UAV and the aerial object and response of the Pitch controller of the third test. . . . 5.22. Measurements for the orientation of the UAV and response of the Yaw controller of the third test. . . . . . . . . . . . . . . . . . . 5.23. Definition of the Yaw controller. . . . . . . . . . . . . . . . . . 5.24. Schematic diagram of the control Loop. . . . . . . . . . . . . . 5.25. 3D flight reconstruction of the flight path. The obstacle to avoid is a orange traffic cone located at the position (5,0,1.1). . . . . . 5.26. Onboard images taken during the execution of the test. Figures 5.26(a) and 5.26(b) are previous to the control activation. Figures 5.26(c) and 5.26(d) are during the control process and Figure 5.26(e) is when the obstacle has been overtaken. . . . . . . . . . 5.27. Evolution of the error during a real test. . . . . . . . . . . . . . 6.1. Image captured with the onboard camera. . . . . . . . . . . . . 6.2. PID-Fuzzy Controller: Membership function of the first input, the error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3. PID-Fuzzy Controller: Membership function of the second input, the derivative of the error. . . . . . . . . . . . . . . . . . . 6.4. PID-Fuzzy Controller: Membership function of the third input, the integral of the error. . . . . . . . . . . . . . . . . . . . . . . 6.5. PID-Fuzzy Controller: Membership function of the output, heading degrees to turn. . . . . . . . . . . . . . . . . . . . . . . 6.6. Flowchart of the optimization process. . . . . . . . . . . . . . .. 99 99 101 102 103 103 104 104 105 105 106 106 107 109 110. 111 111 119 119 120 120 120 122.

(29) List of Figures. XXVII. 6.7. Interaction between the ROS-Gazebo 3D simulator and the two other process developed for this work. . . . . . . . . . . . . . . 6.8. Control loop with the optimization of the Cross-Entropy method. 6.9. Evolution of the probability density function for the first input gain. The standard variance converge in 12 iterations to a value of 0.0028 so that the obtained mean 0.9572 can be used in the real tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10. Evolution of the probability density function for the second input gain. The standard variance converge in 12 iterations to a value of 0.0159 so that the obtained mean 0.4832 can be used in the real tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11. Evolution of the probability density function for the third input gain. The standard variance converge in 12 iterations to a value of 0.0015 so that the obtained mean 0.4512 can be used in the real tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.12. Evolution of the ITAE error during the 12 Cross-Entropy iterations. The ITAE value of each iteration correspond to the mean of the first 5 of 30 controllers of each iteration. . . . . . . . . . 6.13. Evolution of the gains of each input. The value of the gain correspond to the first 5 of 30 controllers of each iterations. . . . . 6.14. Parrot-AR.Drone, the platform used for the real tests. . . . . . . 6.15. Control loop with the optimization of the Cross-Entropy method. 6.16. Explanation of the avoiding task approach. The quadcopter starts at point 0.0 (1.Motor ignition) and flies 0.5 meters keeping the obstacle to avoid in the center of the image (2.Avoiding task. Start). Then the reference to one of the edge of the image is added to the position of the obstacle in the image plane until 3.5 meters (3.Avoiding task. Finish). The quadrotor continues. The last yaw command is send after the avoiding task is finished. The obstacle to avoid is at point (0, 4.5). . . . . . . . . . . . . . . . 6.17. Onboard images during the execution of the test. . . . . . . . . 6.18. Evolution of the error during a real test at 0.04 m/s forward speed using the non optimized fuzzy controller. A RMSE of 9.0081 has been obtained. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.19. Evolution of the error during a real test at 0.04 m/s forward speed using the fuzzy controller optimized using the Cross-Entropy method. A RMSE of 5.271 has been obtained, more than 40% reduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.20. 2D reconstruction of the trajectory defined during a real test at 0.04 m/s forward speed using the non optimized fuzzy controller. 6.21. 2D reconstruction of the trajectory defined during a real test at 0.04 m/s forward speed using the fuzzy controller optimized using the Cross-Entropy method. . . . . . . . . . . . . . . . . . .. 122 123. 123. 124. 124. 124 125 125 125. 126 127. 129. 129 129. 130.

(30) XXVIII. List of Figures. 6.22. Evolution of the error during a real test at 0.08 m/s forward speed using the non optimized fuzzy controller. A RMSE of 9.0081 has been obtained. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.23. 2D reconstruction of the trajectory defined during a real test at 0.08 m/s forward speed using the non optimized fuzzy controller. 6.24. Evolution of the error during a real test at 0.08 m/s forward speed using the fuzzy controller optimized using the Cross-Entropy method. A RMSE of 5.271 has been obtained, more than 40% reduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.25. 2D reconstruction of the trajectory defined during a real test at 0.08 m/s forward speed using the fuzzy controller optimized using the Cross-Entropy method. . . . . . . . . . . . . . . . . . . 6.26. Detection of AR marker using the AruCo software. . . . . . . . 6.27. See-and-avoid task for UAV . . . . . . . . . . . . . . . . . . . 6.28. This is the initial definition of the variables of the Fuzzy Logic Controller before any optimization. . . . . . . . . . . . . . . . . 6.29. Quadcopter’s Matlab Simulink block. . . . . . . . . . . . . . . 6.30. Matlab Simulink block of the flying object to avoid. . . . . . . . 6.31. Onboard UAV camera’s Matlab Simulink block. . . . . . . . . . 6.32. Fuzzy Logic Controller Matlab Simulink block. . . . . . . . . . 6.33. Seeing and Avoiding Task Chart module. . . . . . . . . . . . . 6.34. Real-time dynamic training process in Matlab Simulink . . . . . 6.35. Interaction between the three parts of the matlab process. The Cross-Entropy optimization method, the Simulink block structure and the main process that manage the evolution of the simulations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.36. Representation of the software implementation in a flowchart. . 6.37. 25 degrees step response of the best controller of each optimization phase using the error estimation RMSE (Table 6.10). . . . . 6.38. 25 degrees step response of the best controller of each optimization phase using the error estimation ITSE (Table 6.11). . . . . . 6.39. Control loop of the Cross-Entropy method for the membership functions optimization. . . . . . . . . . . . . . . . . . . . . . . 6.40. Evolution of the probability function of the inputs’ membership functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.41. Evolution of the probability density function of the output’s membership functions. . . . . . . . . . . . . . . . . . . . . . . 6.42. Evolution of the sigma value for the membership functions optimization phase. . . . . . . . . . . . . . . . . . . . . . . . . . . 6.43. Evolution of the values for the membership functions optimization phase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.44. ITSE evolution at the first part of the optimization process. . . . 6.45. Variable definition of the Fuzzy logic controller after the two optimization phases of the membership functions. . . . . . . . .. 130 130. 131. 131 132 133 134 136 137 137 137 137 138. 139 139 141 142 143 144 145 146 146 147 148.

(31) List of Figures. XXIX. 6.46. Control loop of the Cross-Entropy method for the rules’ weight optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.47. Evolution of the probability density function of three selected rules’ weight. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.48. Evolution of the sigma value for the selected rules and the mean of all the rules. . . . . . . . . . . . . . . . . . . . . . . . . . . 6.49. Evolution of the values of the selected rules during the optimization process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.50. ITSE evolution at the first part of the optimization process. . . . 6.51. Description of the avoiding task test. . . . . . . . . . . . . . . . 6.52. Significant processed images from the avoiding task states described in Figure 6.51. . . . . . . . . . . . . . . . . . . . . . . 6.53. Control loop for the real tests. . . . . . . . . . . . . . . . . . . 6.54. Real tests results of the three controller for 0.2m/s pitch speed. . 6.55. Real tests results of the three controller for 0.4m/s pitch speed. . 6.56. Real tests results of the three controller for 0.6m/s pitch speed. . 6.57. Real tests results of the three controller for 0.8m/s pitch speed. . 6.58. Real tests results of the three controller for 1.0m/s pitch speed. . 6.59. Full Optimized controller with 1.2m/s pitch speed . . . . . . . . 6.60. Full Optimized controller with 1.4m/s pitch speed . . . . . . . .. 149 150 151 151 151 152 153 153 156 157 158 158 159 159 159. B.1. Software definition. . . . . . . . . . . . . . . . . . . . . . . . . 176 B.2. Example of a variable with a most important sector definition in the center of it . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 C.1. UAVs rotary wing helicopters used on this thesis . . . . . . . . 182 C.2. UAVs rotary wing quadcopter used on this thesis . . . . . . . . 185.

(32)

(33) List of Tables 2.1. Classification of sensors used in unmanned vehicles. A, active; P, passive; P/A, passive/active; PC, proprioceptive; EC, exteroceptive. Table from (Siegwart and Nourbakhsh, 2004) . . . . . . 3.1. 3.2. 3.3. 3.4.. Base of rules of the Fuzzy controller. . . . . . . . . . . . . . . . Results of the 50 pixels step perturbation. . . . . . . . . . . . . Results obtained with the driveless car without mark detection. . Results of the 50 pixels step perturbation for the system with mark detection. . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5. Results obtained with the driveless car with mark detection. . . . 4.1. Meaning of the acronyms of the linguistic values of the the fuzzy variables inputs and the output. . . . . . . . . . . . . . . . . . . 4.2. Rules’ base for the Yaw and Pitch controllers. . . . . . . . . . . 4.3. Rules’ base for the Heading controller. . . . . . . . . . . . . . . 4.4. Measures of the most relevant Fuzzy system. . . . . . . . . . . 4.5. Data from big attitude changes sections of the flight. . . . . . . 4.6. Rules’ Base of the altitude Fuzzy controller. . . . . . . . . . . . 4.7. Rules’ Base of the roll Fuzzy controller. . . . . . . . . . . . . . 4.8. Rules’ Base of the pitch Fuzzy controller. . . . . . . . . . . . . 4.9. Comparison between homography estimation of X, Y, Z and IMU data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10. Comparison between homography estimation of heading and IMU data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11. Autonomous landing tests. . . . . . . . . . . . . . . . . . . . .. 9 28 32 35 39 45 60 61 61 62 67 79 79 79 82 82 89. 5.1. Rules’ base of the Heading Fuzzy logic controller. . . . . . . . . 100 5.2. Rules’ base of the Pitch Fuzzy logic controller. . . . . . . . . . 100 XXXI.

(34) XXXII. List of Tables. 5.3. RMSE of Pitch and Yaw controllers for the three presented tests. 107 5.4. Base of rules of the Fuzzy controller. . . . . . . . . . . . . . . . 108 6.1. Base of Rules with value for the third input (Integral of the Error) equal to Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.2. Base of Rules with value for the third input (Integral of the Error) equal to Negative . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.3. Base of Rules with value for the third input (Integral of the Error) equal to Positive . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.4. Comparison between the non optimized and the Cross-Entropy optimized Fuzzy controllers at different speeds. . . . . . . . . . 128 6.5. Base of rules with value for the third input (integral of the error) equal to zero, without CE optimization . . . . . . . . . . . . . 134 6.6. Base of rules with value for the third input (integral of the error) equal to negative, without CE optimization . . . . . . . . . . . 135 6.7. Base of rules with value for the third input (integral of the error) equal to big negative, without CE optimization . . . . . . . . . 135 6.8. Base of rules with value for the third input (integral of the error) equal to positive, without CE optimization . . . . . . . . . . . 135 6.9. Base of rules with value for the third input (integral of the error) equal to big positive, without CE optimization . . . . . . . . . 135 6.10. Optimization using RMSE. . . . . . . . . . . . . . . . . . . . . 141 6.11. Optimization using ITSE. . . . . . . . . . . . . . . . . . . . . . 142 6.12. Optimization results for the membership functions. Here are presented the different values of the set’s location of the different membership functions after the different optimization process. . 147 6.13. Base of rules with value for the third input (integral of the error) equal to negative, after CE optimization of the membership functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 6.14. Base of rules with value for the third input (integral of the error) equal to zero, after CE optimization of the membership functions.150 6.15. Base of rules with value for the third input (integral of the error) equal to positive, after CE optimization of the membership functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 6.16. Comparison between the non optimized (Non-Opt), and two Cross-Entropy optimized controller at different stage. One has the membership functions optimized (Half-Opt), and the other include the membership functions and weight rules optimization (Full-Opt). The three controllers were tested at different speeds. Four tests were done to check the behavior of each controller at each speed. The best test is highlighted with bold. . . . . . . . . 155 C.1. Rotomotion UAVs technical specifications . . . . . . . . . . . . 183 C.2. Pelican AscTec UAV technical specifications . . . . . . . . . . 186.

(35) computer HAL “I’m sorry, Frank, I think you missed it, Queen to bishop three, bishop takes queen, knight takes bishop, mate” cosmonaut Dr. Frank Poole “Uh. Yeah, looks like you’re right. I resign” HAL “Thank you for a very enjoyable game” the cosmonaut “Yeah, thank you”. 2001: A space Odyssey, Arthur C. Clarke, 1968..

(36) XXXIV. List of Tables.

(37) Chapter. 1. Introduction Humans have to interact with the environment in which they are. To do this we have to detect things, to measure distances, size, etc, to evaluate the situation and act. Robotics is trying to copy these actions for ages. For the first two actions we use senses and robots use sensors. To evaluate situations we use the brain and our previous learned knowledge, and for robots a set of different artificial intelligence techniques and strategies have been developed. Finally we use our members commanded by the brain with a continue feedback information to interact with the environment. Robots have to use its own robotic members to accomplish the interaction with the environment. We can use robots to accomplish certain missions by substituting a person, in situations not accessible for a person or because of live risk, or by helping him/her to do this action in a better way. In this thesis we try to cover both cases in different situations with ground and aerial vehicles. In this Thesis the sensor used to interact is vision, because of the much information provided and artificial intelligence technique is the Fuzzy control, because of its resembling to human reasoning.. 1.1.. Motivation and Problem Statement. Nowadays there is a large number of vehicles inside cities and out of them. Car accidents are one of the primary cause of death in Europe. In order to re1.

(38) 2. 1.1. Motivation and Problem Statement. duce the number of accidents robotics can substitute the driver by a complete autonomous driving system or given an extra help to the driver to increase the safety of the car occupants, other vehicles and pedestrians. There are some research projects around the world trying to automatize cars for an autonomous driving with a big number of expensive sensors. Those systems are focused to drive out of cities or in places with low traffic. Inside cities the high level of traffic, the occlusion of the street lanes, GPS accuracy problems, among other problems increase the difficult to solve this task. The principal car companies are including more and more sensors each day in their vehicles, helping the driver to park, to brake in a danger situation, etc. A specific case inside city traffic is the public services which are doing the same path every day and every hour. This problem could face as a guided vehicle in large scale. Usually the autonomous guided vehicles were used in very restricted areas like companies stores. Here we present a solution for driving help focused in public services based on autonomous guided vehicles. Aerial vehicles are widely used for more than passenger transport. A view from a certain altitude distance could give us a different aspect of a problem. The inspection of big structures, large areas or places without ground access could not be done using ground vehicles. Some of these tasks could be involve high live risks for the people who are in the aerial vehicle, like in catastrophe hazard or burning areas. In other cases could imply a high cost reduction for a company to use an unmanned aerial vehicle to access to reduced space areas or to do inspection tasks. The UAV is a recent highlight topic in robotics because of the range of possibilities in different tasks. Here we present a set of research works using different unmanned aerial vehicle to accomplish different inspection and autonomous navigation tasks. As is previously mentioned there is a set of sensors to measure the environment of the robot. Each one has its own pros cons and its different costs. In aerial vehicles we also have a weight restriction for the choice of the sensor. For both type of vehicles (ground and aerial) we use a vision sensor to get the information of the robots surroundings. This sensor has the benefit that the information obtained could be processed in different ways. There a big number of algorithms developed for different utilities like object detection, moving measure, or image segmentation. Because of it and its low price is the most used sensor in robotics. In this Thesis a set of different visual algorithm were used for accomplish different tasks When a robot is used in not controlled environment unexpected disturbances increase the difficult of the problem. The robots are non linear machines that imply a hard work to identify the model, which could be affected by little mechanics mismatches from one day to another. The non-linear controls techniques have to be used for control robots in high dynamics environments. Soft-Computing techniques are getting the front position in this robotics field, based on their free-model controllers and how these techniques manage the uncertainty of the sensors. One of most important technique of Soft-Computing is the Fuzzy logic..

(39) Chapter 1. Introduction. 3. This technique is used in the presented Thesis to control ground and aerial vehicles for different task using vision.. 1.2.. Main Contributions. This Thesis contributes with new solutions for an extended number of applications of Fuzzy control for unmanned vehicles using visual control. These are distributed in chapters depending the type of the vehicle to control (ground, helicopter and multicopter). The high capability of cameras to measure the environment for real time applications is also presented. A final chapter presents the contributions of this dissertation on Fuzzy control optimization. There is a wide range of sensors to measure the robot’s surroundings. Each one have its pros and cons, and the visual sensors are not and exception. The weak resistance against lightening changes face with the good relation between efficiency and low cost is one of the big pros of this sensor, and the big number of developed algorithms presented in the literature for different visual applications. In this dissertation four different visual algorithms are used for extracting characteristics from the environment to give enough information to the Fuzzy control to operate the unmanned vehicle for different applications. The control strategy to use is always a hard work when a process have to be automatized. When the process is a vehicle in outdoors environment the number of possible solutions is reduced drastically. The different possible perturbations presented in this type of tests, adding the high difficult to model a real vehicle, makes Soft-Computing techniques one of the best option. Fuzzy control technique is used in this research work to control 4 different model of 3 vehicles. A driver-less car, a RC helicopter and two quadcopters. All the presented Fuzzy control are implemented using the self-developed software (MOFS). The different control system are used for 5 different applications, showing in all the cases a set of real tests. The final contribution of this Thesis to the literature is the novel approach of use the Cross-entropy optimization method to tune different parameters of a Fuzzy controller. This method was used previously in the literature for the optimization of the input’s gains. In this research work is presented a complementary optimization of the inputs’ gains. A contribute of the use of this method for optimization/tuning of the membership functions sets and rules’ weight is presented and tested with quadrotor for avoiding collisions task. It have to be mentioned that all the software developed during this research work, the MOFS for design and configure Fuzzy controllers and the CrossEntropy implementation for optimize Fuzzy controllers, are available under open source policy for who want to use it..

(40) 4. 1.3. Outline. 1.3.. Outline Chapter 2 presents a review of control of unmanned ground and aerial vehicles and the solutions presented in the the literature. There are many possibilities to face this problem, and in this Thesis we focused on visual control, because of the efficiency of this sensor. In this chapter is also presented how a camera sensor can be used to close the control loop, and the selected methodology of image-based visual control. Since the principal aim of this Thesis is to contributes with novel Fuzzy control systems for different robots without knowing its model, in the last section of this chapter this technique is presented and the most relevant works for unmanned vehicle control and other industrial control systems are mentioned. Chapter 3 presents a new Fuzzy control system for an unmanned ground vehicle. A driver-less car is presented using just one camera with a limited field of view. The visual algorithm for detection the painted line on the road and the design and detection of visual marks for car location is also presented. The lateral Fuzzy control of the vehicle is presented and tested with a big amount of experiments with location information and without it. Chapter 4 presents novel Fuzzy control systems using visual information to control an autonomous helicopter. Firstly, a Fuzzy control of a visual pan and tilt platform and the vehicle’s heading is used for track moving targets, as an eye in the sky. Secondly, the control of the pitch, roll and altitude of the helicopter is presented for autonomous landing task. The different visual algorithm used in both application are also explained. In chapter 5 a contribute in visual Fuzzy control is presented for unmanned multicopters. A color based tracking is used for two different applications with two different quadrotors. First, a Fuzzy control of the pitch and heading of an AscTech Pelican quadcopter is used to follow a flying object. Second, a Fuzzy control of the heading is used to control an AR.Drone for see and avoid task. Chapter 6 present a novel approach of the Cross-Entropy optimization method for tuning different parameters of a Fuzzy controller. A color based and a QR code detection visual algorithms are used for avoid collisions. Firstly, the optimization of the inputs’ gains using the robotics operative system (ROS) is presented. Second the optimization of the membership functions sets’ size and location and the rules’ weight is shown. Chapter 7 presents the Thesis conclusions and the possible lines for future work.. A visual outline of the presented Thesis is shown in Figure 1.1..

(41) Figure 1.1: Thesis Structure. Chapter 1. Introduction 5.

(42)

(43) Chapter. 2. State of Art This chapter presents the state of art of unmanned vehicles, the visual control and the Fuzzy control. More detailed information about the related works presented in the literature for each different application explained in this Thesis is presented in each chapter.. 2.1.. Introduction. Unmanned vehicle is term that refer to a vehicle without a human pilot onboard. In the specific case of aerial robots are commonly known as a drone. For ground vehicles the are also called as driver-less or pilot-less. In both cases the vehicle has to measure its environment to know its local and/or global position to execute the desired actions. The UAV and UGV history, development, architectures and navigation control theory are out of the focus of this thesis. The aerial robotics chapter (Handbook of robotics) (Siciliano and Khatib, 2008a) gives a general review of the UAVs history, flight concepts and applications, as well as is presented in (Puri et al., 2007). Aiming to UGV some interesting information about historical facts and the evolution of mobile robots can be found in (Siegwart and Nourbakhsh, 2004), (Nehmzow, 2003) and the mobile robotics chapter in (Siciliano and Khatib, 2008a) . 7.

(44) 8. 2.2. Unmanned Vehicles. 2.2.. Unmanned Vehicles. The interaction between robots and its surrounding area is generally divided in perception, localization, and planning and navigation. Next are presented a brief description of each problem. For a more detailed information the reader could find it in (Siegwart and Nourbakhsh, 2004), (Siciliano and Khatib, 2008a).. 2.2.1.. Perception. One of the most important task of an autonomous system or vehicle of any kind is to acquire knowledge about its environment. This is done using one or more sensors for taking measurements and extracting the meaningful information. There are a wide variety of sensors used in unmanned vehicles. Some sensors are used to measure simple values like the rotational speed of the motors. Other, more sophisticated sensors can be used to acquire information about the robot environment or even to directly measure a robot’s global position. Sensors could be classify as proprioceptive/exteroceptive and passive/active. Proprioceptive sensors measure robot’s internal values, like motor speed, battery voltage. Exteroceptive sensors acquire information from the robot’s environment, like distance, light intensity. This information must be interpreted to extract the meaningful information. Passive sensors measure ambient environmental energy entering the sensor, like microphones and cameras. Active sensors emit energy into the environment and measure it reaction, like laser and sonar. Table 2.1 shows a classification of most useful sensors for unmanned vehicles applications. This classification is arranged in ascending order of complexity and descending order of technological maturity. Tactile sensors and proprioceptive sensors are critical to almost all mobile robots, and are well understood and easily implemented. On the other extreme, visual interpretation by means of one or more cameras provides a broad set of potential functionalities. The most important sensors for unmanned vehicles are extereoceptive ones. The choice between passive and active sensors depends on the task to solve and the battery limitations of the vehicle. Active sensors could be more accurate than passive sensors but imply an extra energy consumption, furthermore, the sensor’s signal may suffer interference. In mobile vehicles the battery capacity are bigger than in aerial robots, being this type of sensor more used in the first type of robots. The most promising sensor for the future of unmanned vehicles is vision and section 2.3 present an overview of visual control in robotics. This sensor provide.

(45) Chapter 2. State of Art General classification (typical use) Tactile sensors (detection of physical contact or closeness; security switches) Wheel/motor sensors (wheel/motor speed and position). Heading sensors (orientation of the robot in relation to a fixed reference frame) Ground-based beacons (localization in a fixed reference frame) Active ranging (reflectivity, time-of-flight, and geometric triangulation). Motion/speed sensors (speed relative to fixed or moving objects) Vision-based sensors (visual ranging, whole-image analysis, segmentation, object recognition). 9 Sensor Sensor System Contact switches, bumpers Optical barriers Noncontact proximity sensors Brush encoders Potentiometers Synchros, resolvers Optical encoders Magnetic encoders Inductive encoders Capacitive encoders Compass Gyroscopes Inclinometers GPS Active optical or RF beacons Active ultrasonic beacons Reflective beacons Reflectivity sensors Ultrasonic sensor Laser rangefinder Optical triangulation (1D) Structured light (2D) Doppler radar Doppler sound CCD/CMOS camera(s) Visual ranging packages Object tracking packages. PC/EC. A/P. EC EC EC PC PC PC PC PC PC PC EC PC EC EC EC EC EC EC EC EC EC EC EC EC. P A A P P A A A A A P P A/P A A A A A A A A A A A. EC. P. Table 2.1: Classification of sensors used in unmanned vehicles. A, active; P, passive; P/A, passive/active; PC, proprioceptive; EC, exteroceptive. Table from (Siegwart and Nourbakhsh, 2004) a big amount of data of the robot’s surroundings. A wide range of different kind of algorithms have been developed in the last two decades.. 2.2.2.. Localization. Navigation is one of the most challenging competences required of an unmanned robot. Success in navigation depends on success in each of the four components of navigation: perception, the robot have to extract the meaningful information acquired by its sensors; localization, the robot have to determine its position in the environment; cognition, the robot have to decide what is the next step to accomplish its mission; and motion control, the robot must modulate its motors or servos outputs to modify its trajectory to the desire one. Between all of theses four blocks, localization has received the greatest research attention in the past decade and, significant advances have been obtained on this front. If one could attach an accurate GPS sensor to a unmanned vehicle, much of the localization problem would be solved. Unfortunately, this sensor is not currently practical for this specific task. The accuracy provided by the current GPS networks is within several meters, which is unacceptable for human-scale robots and needless to say with minirobots. Beyond GPS limitations, localization im-.

(46) 10. 2.2. Unmanned Vehicles. plies more than knowing one’s absolute position in the Earth’s reference frame; it means building a map, and then identifying the robot’s position in relation to that map. Clearly, the robot’s sensors and effectors play an integral role in all the localization forms. The inaccuracy and incompleteness of these sensors, there are a set of sensors and strategies to face this problem. To have a high accuracy in localization the most extended sensor used is the laser. The problem used to be that are quite big, heavy and expensive, and give you a 2D plane information. Nowadays there scaled version of laser sensors that could be used onboard small quadcopters. There are also other strategies to localize robots using other sensors like vision, landmarks and odometry. Laser & encoders localization Encoders onboard mobile robots provide an estimation of the position of the robot. This information is not reliable enough because of the the presence of errors. The uses of other sensors to detect e objects from previous map can help, but also this information is not reliable enough. The laser sensor is the most used sensor for simultaneous localization and map building (SLAM). The most effective and popular solutions to SLAM have been based on probabilistic techniques. Some of them are Fast-SLAM, EKF-SLAM and Graph-SLAM. The main difference between them concerns basically the environment representation and the uncertainty description. A extended review of all the map building techniques could be found in (Thrun, 2002) and (de la Puente Yusty, 2012). Visual localization Visual SLAM could be done using landmarks or based on feature detection. The early works of Davison in (Davison, 1998), (Davison, 2003), presents a single camera SLAM algorithm based on a Kalman filter. The features are initialized by using a particle filter which estimates their depth. Montiel extended this framework in (Montiel et al., 2006) by proposing an inverse depth parameterization of the landmarks. Chekhlov use a SIFT-like feature descriptors and track the 3D motion of a single camera by using an unscented Kalman filter (Chekhlov, 1998). Recently Klein and Murray (Klein and Murray, 2007) developed the parallel tracking and mapping (PTAM) that introduced a new strategy based on keyframes for a robust visual SLAM. Landmark-based navigation Landmarks are generally defined as passive objects in the environment that provide a high degree of localization accuracy when they are within the robot’s field of view. The control system for a landmarkbased navigator consists of two different phases. When a landmark is detected, the robot update its position without cumulative error. But when the robot is in no landmark “zone”, then only action update occurs, and the robot accumulates position uncertainty until the next landmark is detected. The robot is thus effectively dead-reckoning from landmark zone to landmark zone. This in turn.

(47) Chapter 2. State of Art. 11. means the robot must consult its map carefully, ensuring that each motion between landmarks is sufficiently short, given its motion model, that it will be able to localize successfully upon reaching the next landmark. Some of the first works with robot navigation with landmarks are (Lazanas and claude Latombe, 1992), (Lazanas and Latombe, 1995), (Nourbakhsh et al., 1999).. 2.2.3.. Planning and Navigation. Cognition generally represents the purposeful decision-making and execution that a system use to achieve its highest-order goals. In case of unmanned vehicles, the specific concept of cognition directly linked to robust mobility is navigation competence. Navigation enclose the ability of the robot to act based on its knowledge and sensor values to reach its goal positions as efficiently and as reliably as possible. In the artificial intelligence community planning and reacting are often viewed as contrary approaches or even opposites. In fact, when applied to physical systems such as unmanned robots, planning and reacting have a strong complementarity. In most of the application with unmanned vehicles this fact is happens, but when the goal position depends of dynamic position of something in the environment it is possible to based our navigation strategy just in the sensor information. This is the case of applications like object following, sense and avoid. There are a wide range of control architectures and techniques used to manage unmanned vehicles, that are explained widely in the literature, some of these works are (Siegwart and Nourbakhsh, 2004), (Nehmzow, 2003). In this Thesis we use the Fuzzy logic to develop the control system to commands the different unmanned vehicles. In section 2.4 is presented a brief description of this technique and some of the works presented in the literature.. 2.3.. Visual Control. Vision is a useful robotic sensor since it mimics the vision’s human sense and allow for non-contact measurement of the environment. The use of vision with robots has a long history, starting with the work of Shirai and Inoue (Shirai and Inoue, 1973). They describe how a visual feedback loop can be used to correct the position of a robot to increase task accuracy. Today vision system are available from major vendors that are highly integrated in robots systems. Typically visual sensing and manipulation are combined in a open-loop fashion, “looking” and then “moving”. The accuracy of the results depends directly on the accuracy of the visual sensor, the robot end-effector and the controller. An alternative to increase overall accuracy of the system is use a visual feedback control loop. The term of visual servoing was first mentioned by Hill and Park (Hill and Park, 1979), to distinguish their approach from the earlier “blocks world” experiments where the system alternated between picture taking and moving..

(48) 12. 2.3. Visual Control. Prior to this concept was introduced, the less specific term viual feedback was generally used. Visual servoing is no more than the use of vision at the lowest level, with simple image processing to provide reactive or reflexive behavior to servo-position a robotic system. Classical visual servo control was developed for serial-link robotic manipulators with the camera typically mounted on the end-effector, also called eye-in-hand. For the purpose of this Thesis, the task in visual servoing is to use the visual information to control the unmanned vehicle’s pose with respect objects or landmarks presented on the environment. Since the first work about visual servoing system, presented by Sanderson (Sanderson and Weiss, 1983) at the decade of the 80’s, progress in visual control of robots has been fairly slow. It was with the increase of the computing power when was able the analysis of images at a sufficient rate to “servo” a robot manipulator. A huge amount of applications appeared in the last two decades. A comprehensive review of the literature in this field, as well the history and robotic applications, is given by Croke (Corke, 1994) and includes a large bibliography. Vision-based robot control using an eye-in-hand system is classified into two groups (Hamel and Mahony, 2002), (Hutchinson et al., 1996): position-based and image-based control systems. In the literature is also called by PBVS as Position-Based Visual Servoing and IBVS as Image-Based Visual Control. PBVS involves reconstruction of the target pose with respect to the robot and leads to a Cartesian motion planning problem. This kind of control is based on the tri-dimensional information from the scene (Soatto and Perona, 1994) (Ma et al., 2000). So is needed the geometric model of the object to track and a calibrated model of the camera, with it get a estimation of the position and orientation of the object (Figure 2.1). The sensitivity of PBVS design to camera calibration is particularly worrying when low quality camera are used. x d +. e. x. Figure 2.1: Position-Based Visual Control. In contrast, for IBVS the control task is defined in terms of image features. A controller is designed to maneuver the image features for a desire configuration. The original Cartesian motion planning problem is solved. The approach is inherently robust to camera calibration and target modeling errors reducing the computational cost (Figure 2.2). However, this configuration imply an extra complexity for the control design problem. The approach used in classical IBVS control (Espiau et al., 1992) can not be used for the integrated control of non-linear fully dynamic systems. Recent approaches to robustly extending IBVS to dynamic systems include; dis-.

No results found