Space debris: pose estimation using stereo vision

(1)

Pose Estimation Using Stereo Vision

by

Willem Carel de Jongh

Thesis presented in partial fulfilment of the requirements for the degree of

Master of Engineering at Stellenbosch University

Supervisors:

Dr H.W. Jordaan Dr C.E. van Daalen Department Electrical and Electronic Engineering

(2)

Plagiarism Declaration

1. I have read and understand the Stellenbosch University Policy on Plagiarism and the definitions of plagiarism and self-plagiarism contained in the Policy [Plagiarism: The use of the ideas or material of others without acknowledge-ment, or the re-use of one’s own previously evaluated or published mater-ial without acknowledgement or indication thereof (self-plagiarism or text-recycling)].

2. I also understand that direct translations are plagiarism.

3. Accordingly all quotations and contributions from any source whatsoever (in-cluding the internet) have been cited fully. I understand that the reproduction of text without quotation marks (even when the source is cited) is plagiarism. 4. I declare that the work contained in this assignment is my own work and that I have not previously (in its entirety or in part) submitted it for grading in this module/assignment or another module/assignment.

W.C.

de

Jongh

April

201

9

Student Number Initials and Surname Date

(3)

Abstract

Tracking the relative attitude and position of uncooperative in-orbit objects is vi-tal for autonomous operations in space. Vision-based solutions have low power consumption and can provide a system with valuable information to perform pose determination. Estimation algorithms are required to extract the system states from visual measurements and many similar approaches have been investigated in mobile robotics.

In this thesis, a chaser satellite is fitted with stereo cameras which are used to extract unique features on the surfaces of an uncooperative, unknown target. The scale invariant feature transform (SIFT) feature detector is used to identify and establish correspondence of the target features. A full state kinematic estimator is implemented using an extended Kalman filter (EKF) based on the simultaneous localisation and mapping (SLAM) approach. The filter makes use of the observations from the feature extractor to estimate the position and orientation of the target relative to the chaser along with the angular and linear velocities of the target. Shape and size reconstruction of the target is made possible using the sparsely tracked features.

A simulation environment providing ground truth is used to verify the performance of the estimation algorithm. The integration of the estimator with the feature ex-tractor is assessed using real world data. Experimental data is obtained from image sequences of a moving target in a laboratory set-up. Results show that the filter estimates the system states successfully and that the developed feature extractor is capable of detecting robust features with reliable correspondence. It is concluded that the use of a stereo-vision-based estimator is a viable option for autonomous operations such as space debris removal and satellite service missions.

(4)

Uittreksel

Akkurate lokalisering van onbekende ruimte voorwerpe in verhouding tot ’n volger-satelliet is noodsaaklik vir outonome ruimte operasies. Skatting van die oriëntasie en posisie van die voorwerp deur middel van visuele sensors soos kameras, is ’n gewilde oplossing in die robotika-veld. Visuele sensors het ’n lae kragverbruik en is goedkoop om te implementeer. Lokaliseringsalgoritmes word benodig om die toestande van die voorwerp uit die visuele metings te onttrek.

Hierdie tesis bespreek ’n stereo-kamera paar wat, saam met die skaal bestande ken-merk transform (SIFT) algoritme, gebruik word om unieke punte op die oppervlaktes van ’n nie-samewerkende, onbekende voorwerp te vind. Die algoritme is só ontwerp om rekord te hou van ooreenstemmende punte in opeenvolgende beelde. ’n Kinema-tiese toestands-skatter word geïmplementeer met behulp van ’n uitgebreide Kalman filter (EKF). Die skattingsalgoritme gebruik die gelyktydige lokalisering en karte-ring (SLAM) benadekarte-ring. Die filter skat die relatiewe posisie en oriëntasie van die voorwerp af met betrekking tot die kameras. Die hoek- en lineêre-snelhede van die voorwerp word ook onttrek. ’n Verteenwoordiging van die voorwerp se grootte en vorm word saamgestel vanuit die geskatte posisies van die unieke voorwerp-punte. ’n Simulasie-omgewing, wat grondwaarheid voorsien, word gebruik om die werking van die skattingsalgoritme te toets. Die integrasie van die skatter met die beeldver-werkingsalgoritme word getoets deur gebruik te maak van eksperimentele beelde. Eksperimentele beelde word vasgelê deur ’n bewegende voorwerp waar te neem in ’n laboratorium-opstelling.

Die stelsel toon bevredigende uitslae. Die EKF-SLAM benadering, in samewerking met die beeldverwerkingsalgoritme, is daartoe in staat om die voorwerp te lokaliseer relatief tot die kameras. Die studie kom tot die gevolgtrekking dat stereo-visie-gebaseerde skatters voldoende is vir outonome ruimtesendings soos die diens van satelliete en die verwydering van ruimterommel.

(5)

List of Figures

2.1 Space-based applications of pose determination. . . 4

2.2 Chart summarising all catalogued objects in orbit around the earth. . . 6

2.3 Illustration of the DEOS mission objectives. . . 7

2.4 Targets for RemoveDebris mission. . . 9

2.5 Extracting triangle lines from images of a non-cooperative satellite. . . . 12

2.6 Artistic rendition of the TerraSAR-X satellite and the reconstructed rear side of the satellite. . . 13

2.7 Cooperative sensor configuration for 3D point cloud generation. . . 14

2.8 Illustration of the influence of poor a initial guess on the performance of the ICP algorithm. . . 15

3.1 Illustration of the classic SLAM problem. . . 19

3.2 A stationary sensor or robot observes a moving target in inertial space. 19 3.3 System framework block diagram. . . 20

3.4 Recursive estimator algorithm flow chart. . . 21

3.5 Euler 1-2-3 rotation . . . 24

3.6 Quaternion rotation . . . 25

4.1 Pinhole camera model . . . 29

4.2 Epipolar geometry between two cameras. . . 31

4.3 Rectified stereo geometry . . . 33

4.4 Measurement extraction framework. . . 34

4.5 The reference feature set,FR . . . 35

4.6 Image plane feature extraction algorithm . . . 36

4.7 Result after background subtraction and binary thresholding . . . 38

4.8 Region of interest (ROI) created from largest set of contours . . . 38

4.9 Image scale space created by Gaussian convolutions. . . 39

4.10 Detected SIFT features . . . 41

4.11 Keypoint descriptor created from histogram binning of image gradients. 41 4.12 A feature in the left hand frame and possible matches in the right hand frame. . . 43

4.13 Stereo matching between features in the left and right hand images. . . 44

5.1 Target reference frame relative to the camera reference frame. . . 48

5.2 A feature described in the target reference frame. . . 50

5.3 Simulated target with randomly selected features shown in red. . . 56

5.4 Estimated quaternion attitude from simulated measurements. . . 57

(8)

5.5 Estimated angular velocities from simulated measurements. . . 58

5.6 Relative displacement between the sensor and the target. . . 58

5.7 Linear velocities between the sensor and the target. . . 59

5.8 Angular velocities estimated using knowledge of target moments of inertia. 61 5.9 Estimated quaternion attitude vector using features that are always visible. 62 5.10 Number of visible features for occlusion testing. . . 63

5.11 Estimated quaternion attitude vector using occluded features. . . 64

5.12 Quaternion error between estimated and ground truth values due to the influence of outliers. . . 66

6.1 Experimental hardware: Target fitted to the robotic gantry. . . 68

6.2 Experimental hardware: FLIR BlackFly stereo camera pair. . . 68

6.3 Layout of the experimental hardware. . . 69

6.4 Adjusting the axis of rotation using adjustment anglesα and ψ. . . 70

6.5 Calibrating the hardware reference frame . . . 70

6.6 Extracted stereo matches from experimental data sequence. . . 72

6.7 Number of measurements per timestep, extracted from experimental data. 73 6.8 Estimated quaternion attitude vector using real world data. . . 74

6.9 Vector magnitude of the estimated angular velocities. . . 75

6.10 Estimated angular velocities using real world data. . . 75

6.11 Relative displacement magnitude between the target and the sensor. . . 76

6.12 Estimated linear velocity of the target using experimental data. . . 76

6.13 Excerpts from target reconstruction sequence. . . 78

6.14 Perspective view of reconstructed target. . . 79

(9)

List of Tables

2.1 Deutsche Orbitale Servicing Mission (DEOS) sensor configuration. . . . 8 2.2 Allocated risk attributes of different active debris removal methods. . . 11 5.1 Simulation parameters . . . 56 5.2 Occurrence of outliers in simulated data. . . 65 6.1 Experiment parameters . . . 71

(10)

Nomenclature

Abbreviations and Acronyms

ADR active debris removal CAD computer aided design CRF camera reference frame DCM direction cosine matrix

DEOS Deutsche Orbitale Servicing Mission EKF extended Kalman filter

FOV field of view

ICP iterative closest point ISS International Space Station KF Kalman filter

LEO low earth orbit

Lidar light detection and ranging RMS root mean square

ROI region of interest SFM structure from motion

SLAM simultaneous localisation and mapping SIFT scale invariant feature transform SURF speeded up robust features UKF unscented Kalman filter

Syntax and Style

(11)

NOMENCLATURE xi

x The scalar valuex

x The vector x(usually lowercase)

¯

x Unit vector

X The matrix X (usually uppercase)

(12)

Chapter 1

Introduction and Problem

Description

1.1 Problem Background

Over the past decade it has become evident that the success of current and future space missions is jeopardised by the presence of space debris. More than 8500 spacecraft launches have taken place since the launch of the Soviet Union satellite, Sputnik1 in 1957 [1]. A large number of man-made debris has been created by these space activities. Decommissioned satellites, rocket bodies and other hardware are scattered around earth in various orbits. These objects have very high velocities and a collision with a satellite or spacecraft can lead to immediate mission failure. Presently, passive debris removal systems are incorporated in new space missions to reduce the number of new debris caused by the mission. This includes drag sails and inflatable balloons that deploy at end-of-life to force the satellite to de-orbit. Amer-ican aerospace company, Space-X, has also recently implemented reusable heavy-lift engines like the Falcon 9 [2], which reduces the amount of debris caused by satellite launches. Passive methods, however, will not aid in reducing the estimated 500 000 pieces of orbital debris that already exist [3]. Active debris removal (ADR) meth-ods and missions are required to decrease the amount of debris and consequently mitigate the risk of collisions.

Accurate pose information of a target relative to a chaser satellite is required in order for a chaser satellite to perform ADR to capture and remove the target from its orbit. The appearance and shapes of debris pieces are generally unknown to the chaser before the mission starts, thus no prior information can be utilised. The target in the case of this project is assumed to be uncooperative, meaning that there is no communication link between the chaser and the target. A close-range measurement system is therefore required to determine the pose of the target and its angular and linear velocities relative to the chaser. This information is critical to perform safe rendezvous operations with the target.

(13)

CHAPTER 1. INTRODUCTION AND PROBLEM DESCRIPTION 2

1.2 Project Aim and Proposed Solution

This project aims to develop a system capable of determining the relative pose between an unknown, uncooperative target and a chaser satellite. A stereo-camera sensor is used to perform visual-based estimation of a freely moving target. Meas-urements are based on the calculation of geometric positions of natural features on the target. A tracking algorithm is implemented to estimate the state of the target’s pose along with the positions of the features on the target.

1.3 Document Outline

The next chapter investigates aspects of pose determination for space based ap-plications. The current state and challenges of space debris and ADR is discussed. Chapter 3 defines the project problem in detail and describes the system model. The relevant reference frames used in this project are also introduced. Sensor devel-opment is discussed in Chapter 4. This deals with modelling the sensor, developing a measurement extraction algorithm and describing the associated image processing algorithms. Chapter 5 discusses the development of the tracking algorithm and provides results that were obtained from simulated measurements. A practical ex-periment was conducted and the results are presented in Chapter 6. Final conclu-sions, findings and future recommendations are drawn in Chapter 7.

(14)

Chapter 2

Literature Study

2.1 Introduction

This chapter investigates typical space-based pose determination applications and how they can be exploited for debris removal. Aspects of space debris are explored to gain a better understanding of why and how to mitigate debris. Two recent ADR missions are investigated to identify the current state-of-the-art technologies that are implemented. Several methods for ADR are also discussed along with their associated risks and the sensor information required to successfully execute these missions. Previously researched methods for determining the pose of in-orbit objects are reviewed and conclusions are made with regards to the effectiveness of these methods and their relevance to this project.

2.2 Space-Based Pose Determination

Pose determination is concerned with the calculation of the relative position and attitude of a target object in orbit relative to a chaser satellite. The application is specifically focused on close-proximity operations like rendezvous, docking, monit-oring and servicing [4]. Close-proximity operations require autonomy since delays from ground control communications would deem it impossible to safely perform these procedures. Pose determination is a problem with a broad range of applica-tions, as shown in Figure 2.1. Two main categories are defined, namely cooperative and uncooperative pose determination. A target is considered to be cooperative if it possesses useful information that can be exploited for pose estimation, such as on-board sensors and communication links with the chaser.

Uncooperative pose determination is sub-divided into known- and unknown-target applications. Prior data is available when interacting with known targets. This in-cludes information like geometrical shape and size or the positions of known features like antennas or solar panels. In other cases an exact model of the target is available prior to the mission start [4]. Applications for uncooperative, but known, targets include the removal or servicing of defunct and decommissioned satellites.

(15)

CHAPTER 2. LITERATURE STUDY 4 Pose Determination Cooperative Uncooperative Known Unknown On-orbit servicing Formation flight Debris removal On-orbit servicing Debris removal Asteroid exploration

Figure 2.1 – Space-based applications of pose determination.

Applications with no prior target information are classified as missions with unco-operative, unknown targets. Both natural debris such as asteroids and man-made objects like rocket bodies or satellites present the greatest challenges for close-proximity operations since the debris may be tumbling with large angular rates. Models of man-made objects may be inaccurate after years of exposure in space or if the target satellite was damaged to such an extent that it is considered to be fragmentation debris. There are high risks associated with the interaction of uncooperative, unknown targets and therefore the requirements for accuracy and robustness of the pose determination are strict.

Opromolla et al. [4] reviewed several techniques for close proximity operations, in-cluding typical sensor configurations that are appropriate for the applications shown in Figure 2.1. Relative navigation in space makes use of primarily infrared (active) and visible light (passive) sensors. Laser range finders operate in the infrared spec-trum and are capable of measuring large distances with high accuracy. Line-of-sight measurement systems like range finders are, however, not suited for pose determ-ination applications since they make use of non-steerable single lasers, resulting in coarse measurements. Light detection and ranging (Lidar) systems operate in the same frequency spectrum as range finders, but are capable of creating 3D point clouds of scanned objects. Multiple samples of a moving target using Lidar can provide enough information to perform pose estimation of the target [4].

Monocular- and stereo-cameras are deemed passive sensors operating in the vis-ible light spectrum. Cameras have a lower power consumption than active sensors and are less expensive to implement than Lidar. An additional benefit of camera sensors lies in the fact that they are easier to use for human-in-the-loop actions. This includes ground control verifications, human controlled operation and visual confirmation. Cameras do however, suffer in poor lighting conditions or when seg-mentation errors occur and objects in the background are tracked with the target. The former can be overcome by fitting the chaser with illumination devices like the

(16)

system used by the Proximity Operation Sensor (PXS). The PXS was developed by the National Space Agency of Japan (NASDA) [5]. The camera sensor was equipped with a light emitting diode (LED) array that emitted visible light in a 30◦ cone around the sensor.

Stereo cameras are capable of acquiring sparse 3D point cloud information similar to Lidar, although it requires a bit more processing. Data generated by camera sensors are more textured than Lidar scans. Lidar is more accurate over larger distances and the functionality and range of stereo cameras are often limited by the baseline distance between the left and right hand camera.

With all this considered, it is noted that the sensor configuration should be selected based on the mission specifications. Some missions choose to make use of several different sensors depending on the mission phase [6; 7]. Therefore, the optimal solution may lie in the combination of sensors working in different frequency and light domains, using data fusion to perform the best possible pose estimation given the problem.

2.3 Overview of Space Debris

Kessler and Cour-Palais [8] predicted in 1978 that the frequency of collisions between satellites would increase as more objects are launched into space. The density of objects in low earth orbit (LEO) would become so high that an ablation cascade will occur where the exponential increase in collisions will render space inaccessible for many years.

The U.S. Space Surveillance Network maintains a catalogue of space debris objects. The data shown in Figure 2.2 shows two significant increases in the amount of debris in 2007 and 2009. The first increase in 2007, was caused by an anti-satellite missile test that was conducted by China. A SC-19 missile was used to destroy an ageing Chinese weather satellite [9]. This explosion created more than 3000 pieces of trackable debris. The second was a catastrophic collision between a decommissioned Russian satellite, Kosmos-2251 and a U.S. communication satellite, Iridium 33 in 2009. The collision created over a 1000 pieces of debris [10]. This was the first time in history that an unplanned collision between two satellites occurred, providing a glimpse of the danger of space debris that Kessler and Cour-Palais [8] predicted in 1978.

(17)

CHAPTER 2. LITERATURE STUDY 6

Figure 2.2 – Chart summarising all catalogued objects in orbit around the earth. Catalogue created by the U.S. Space Surveillance Network. The pink data represents fragmentation debris, which includes satellite breakup debris, while the mission-related debris in orange includes all dispensed, separated and released objects as part of a mission procedure. Figure reprinted from Orbital Debris Quarterly News, [11].

The Inter-Agency Space Debris Coordination Committee (IDAC) requires that all new satellites that are launched into LEO are equipped with passive removal tech-nologies, which forces the satellite to de-orbit itself within 25 years after its mission ends. Although effective in mitigating the creation of new debris, Liou [12] has found that the amount of debris that already exists will spiral out of control within the next 200 years due to debris inter-collisions. Therefore, the need for ADR is clear, with several methods having been proposed in the past. A study by NASA shows that approximately five objects with high collision probabilities need to be removed per year to prevent the Kessler syndrome [12]. These are objects with masses larger than one ton and with dimensions bigger than two metres in diameter. Potential targets that fit into this category are the upper stages of rocket bodies commonly used to deploy satellites [12].

2.4 Active Debris Removal

Current ADR technologies have limitations. DEOS and RemoveDebris are two ADR missions that aim to investigate a number of debris removal methods.

(18)

2.4.1 DEOS

The Deutsche Orbitale Servicing Mission (DEOS) [6] was a German in-orbit satellite servicing concept with the purpose to evaluate technology for rendezvous, capture and de-orbiting of uncooperative satellites. The proposed mission consisted of two satellites. A client satellite serves as the non-cooperative target and is captured by the active chaser satellite. The chaser satellite is equipped with a robotic manip-ulator for berthing with the target. The objectives of the mission, illustrated in Figure 2.3, were:

• Use a robotic manipulator to de-tumble and capture a non-cooperative target. • Simulate a servicing mission such as refuelling or repairing the target.

• Remove the target from orbit by forcing it to re-enter the atmosphere.

Figure 2.3– Illustration of the DEOS mission objectives. Reprinted from Reintsema

et al.[6].

The chaser satellite is equipped with several sensors, each designed for a different phase of the mission. These include a far-range monocular camera (FMC), Lidar, Radar, a mid-range stereo camera (MSC), a close-range stereo camera (CSC) and a monocular docking camera (MDC). Rendezvous with the target consists of four phases, each with their respective sensors as shown in Table 2.1. An inspection manoeuvre is done to estimate the motion of the target. The primary sensors used for pose determination are the mid-range stereo cameras and radar.

(19)

Phase Range Sensors Description

Formation Flying and Phasing

>4 km FMC Homing in on target using absolute naviga-tion techniques. Reduce phase angles between target and chaser. Far Range Approach 85 m - 4 km FMC

Lidar

Proceed to close in on target and switch over to relative navigation techniques.

Mid Range Approach 6 m - 85 m RADAR Lidar MSC Final approach of target. If target is unknown or non-cooperative, execute a fly-around inspection of the target at 15 m. Close Proximity

Opera-tion

<6 m RADAR MDC

Follow a straight line trajectory to the target and proceed with berth-ing operations.

Table 2.1– DEOS sensor configuration.

Reintsema et al.[6] also investigated the effect of illumination changes on the per-formance of the camera sensors. It was found that additional lighting is required when difficult illumination conditions arise, such as total darkness or if the target is positioned between the sun and the chaser. The project timeline proposed the launch of the mission in 2018, but the project was cancelled soon after the definition phase.

2.4.2 RemoveDebris

The RemoveDebris mission was launched in 2017 with the aim of performing a low-cost ADR demonstration using a net and harpoon [13]. Vision based navigation (VBN) and drag sails are also implemented to test the feasibility of these technologies in a space environment. Two debris satellites, DS-1 and DS-2 shown in Figure 2.4, are loaded into the main mission platform. These are used as target debris and are launched from the platform after deployment into orbit from the International Space Station (ISS). The main platform was successfully released from the ISS on the 20th of June 2018 [14].

Three key objectives were identified for RemoveDebris over the course of the 6 month operation time [7]:

• Deploy DS-1 at a low velocity (≈ 5 cm/s) from the main platform. After successful deployment, DS-1 inflates a balloon to increase its target area. The

(20)

main platform then proceeds to eject a net to capture DS-1. Thereafter, DS-1 is allowed to de-orbit at an accelerated rate due to the increased surface area from the inflated balloon.

• Extend a 10 x 10 cm target 1.5 m away from the platform and strike it with a harpoon. The main platform is equipped with two supervision cameras to assess the outcome of the harpoon and net experiments.

• Deploy DS-2 at a velocity of≈2 cm/s from the main platform and perform a series of manoeuvres to capture image and Lidar data of DS-2’s movement.

Figure 2.4 – Targets for RemoveDebris mission. DS-1 CubeSat (left) is used for the net experiment and DS-2 CubeSat (right) used for VBN [13].

DS-2 is fitted with deployable panels to represent a satellite object with solar panels. It is also equipped with on-board sensors to determine its own attitude and a data link to the main platform to transfer this ground truth data. The captured data is processed offline using dedicated image processing algorithms. An extended Kalman filter (EKF) performs data fusion and estimates the motion of DS-2. The quality of the estimates is compared to the data that is acquired from DS-2’s on-board sensors. RemoveDebris completed the net capture procedure in September of 2018 and the VBN experiment would have commenced in October 2018 [14].

A major factor in the success of DEOS and RemoveDebris is the effectiveness and quality of their respective pose determination systems. In the case of DEOS, failure to predict the target’s pose may result in a crash with the target. Although not so critical for RemoveDebris, since a berthing operation is not done, the VBN system plays a vital role in the research of future ADR missions. DEOS makes use of a complex sensor configuration to increase the robustness of the system should one of the sensors fail. However, no consideration was made for additional light or illumination sources, even though it was a concern for the accuracy of the on-board optical sensors. The use of Lidar (DEOS and RemoveDebris) and Radar (DEOS)

(21)

overcomes this challenge, but active sensors like Lidar and Radar are expensive and consume more energy than passive optical sensors.

2.4.3 Active Methods for Debris Removal

An ADR mission consists of various phases, as mentioned in Section 2.4.1. In general, the process consists of: launch and early orbit phase (LEOP), far-range rendezvous phase, close-range rendezvous phase, capturing and removal phase [15]. Pose estima-tion plays a major role in the close-range rendezvous and capturing phases. Hakima and Emami [16] assessed five different ADR methods for LEO debris. Removal by net, laser, electrodynamic tether, ion beam shepherds and robotic arm were selected and their performances compared using a Monte Carlo simulation. The feasibility of these methods were assessed based on several criteria including missions costs, technology readiness and risks associated with each method. Table 2.2 summarises the risks associated with each of these methods.

It was noted that one of the main concerns of contact ADR methods is uncontrolled debris attitude rates and the need for a robust vision/tracking system is evident. Hakima and Emami [16] concluded that throw nets, space-based lasers and robotic arms are the most feasible for ADR given the current state of knowledge and tech-nology readiness.

(22)

Method Mission Risk Description

NET

R1.Explosion of debris energy stores and further fragmentation

R2.Damages to remover spacecraft due to uncontrolled debris attitude rates

R3.Debris capturing is unsuccessful

LSR

R1.Thrust degradation due to attenuation of the laser beam cause by the ejected plume

R2.Explosion of debris energy stores and further fragmentation due to extremely high temperature created during ablation

R3. Vision system fails to detect, acquire and track the target debris

IBS

R1. Shepherd fails to keep a safe distance from the target and collides with it

R2. Fragmentation of target due to excessive spin-up resulting from misalignment between ion beam centre of pressure and target centre of mass

R3.Secondary ions backscattered from the target surface contaminate sensitive parts

EDT

R2.Tether gets tangled or collides with other in-orbit objects and creates more debris

ARM

R2. Damages to remover spacecraft due to uncon-trolled debris attitude rates

Table 2.2 – Allocated risk attributes of different ADR methods. The ADR methods that are assessed are throw nets (NET), space based lasers (LSR), ion beam shepherds (IBS), electrodynamic tethers (EDT) and capturing via robotic arm (ARM). Adapted from Hakima and Emami [16].

2.5 Pose Estimation Methods From Previous Research

A number of different pose estimation methods, for space-based applications, have been proposed in the past.

(23)

Malan [17] used monocular vision to perform relative pose estimation between two satellites in formation flight. This was done by placing fiducials on the target satellite and tracking their positions using the camera sensor. Kalman filters were used to estimate the target pose using known dynamics of the target, including the moments of inertia. Malan was able to successfully and accurately track the target satellite using both simulated and experimental data. Experimental results using real world data with ground truth was acquired by fixing a target to a robotic arm. This method proved to be an effective solution in the case where the shape and mass of the target are known.

Song and Cao [18] also used monocular vision to perform pose estimation of a non-cooperative target. A sliding window Hough transform was used to identify the triangular structures, shown in Figure 2.5, that are commonly used to fix solar panels to satellites. Pose calculations were based on a feature recognition algorithm that compared the image features of the solar panel structures to a pre-loaded model of the target. This method does not require the placement of fiducials or markers, but still relies on the target having triangular solar panel mounts. It was found to be highly accurate, but this approach is not feasible when designing for the capture of fragmentation debris or satellites with a different attachment of solar panels.

Original image Edge Detection

Image filtering Line extraction

Figure 2.5 – Extracting triangle lines from images of a non-cooperative satellite [18].

Modelling the appearance of a satellite at close range for pose estimation with the application of on-orbit servicing was investigated by Oumer et al. [19]. A German satellite, TerraSAR-X, shown in Figure 2.6, was identified as the target for their research. An exact replica of the outer surface of the rear side of the satellite was constructed and multiple images were captured in a spherical arrangement around the target. This learning data was used offline to build a vocabulary tree of im-age features. Online estimation of the target was performed by retrieving imim-age

(24)

correspondences between sensor measurements and the appearance model. The cor-respondence between 2D image plane features and 3D model features were used with the random sample consensus (RANSAC) algorithm to compute the pose of TerraSAR-X relative to the monocular camera sensor.

Figure 2.6 – Artistic rendition of TerraSAR-X (left) and the rear side replica of the satellite (right), used for appearance learning [19].

Dong [20] proposed the dynamic capturing of a non-cooperative object by means of a robotic arm. The pose of the target was determined using a vision-based tracking system that used an EKF to perform recursive estimation of the target’s states. A monocular camera was attached to the end of the robot to capture images of the moving target. However, the target was only moving in one plane and therefore depth calculation was not necessary.

Lichter and Dubowsky [21] proposed an architecture to compute the states and shape of an object using 3D vision sensors. Their solution overcomes the drawbacks of the methods proposed by Malan [17] and Song and Cao [18] by making use of four 3D sensors arranged in a tetrahedron formation around the target as shown in Figure 2.7. A cloud of points is constructed at every timestep using the data captured by the sensors. A coarse estimate of the target’s pose is computed by means of an eigenvalue decomposition using the relative geometric moment of inertia tensor. The inertia tensor is calculated using the point cloud information. The coarse pose estimates are fed to an EKF that makes use of a dynamic motion model to provide a refined pose estimate along with a probabilistic reconstruction of the target. Although this solution provides accurate estimates of the target’s states and shape in simulation, the use of 3D sensors on multiple satellites is not practical. This would require the four separate satellites with their own navigation systems to continuously update their positions relative to each other to achieve accurate capturing of the point cloud.

(25)

Figure 2.7 – Cooperative sensor configuration as proposed by Lichter and Dubowsky [21]. Four 3D sensors are arranged around the target to capture a dense point cloud of the target.

He et al. [22] proposed a solution similar to Lichter and Dubowsky [21]. Non-cooperative pose estimation is performed using point cloud data generated by a series of Lidar images. A particle filter uses the curvature, densities and geometric characteristics of the points to estimate the pose of the unknown moving object. Due to the availability of only sparse point cloud data, He et al. opted to rather use a kinematic motion model. Although it may yield slower convergence or erratic behaviour, the use of a kinematic model makes much more sense, since inertia and mass information is normally not available when dealing with space debris. It was, however, concluded that the use of the particle filter is computationally expensive and an alternative solution or improvement in the algorithm architecture is required for real-time pose estimation applications.

The iterative closest point (ICP) algorithm is often used for pose estimation by searching for the best alignment between two point clouds. The distance between corresponding points is iteratively minimized until a best fit is found. Shahid and Okouneva [23] investigated pose determination of high value satellites in geo-synchronous orbits using Lidar scans and ICP matching. Scans of satellite surfaces were matched with the known computer aided design (CAD) models of the targets. Woods and Christian [24] also used the ICP algorithm with flash Lidar measure-ments. It was found that even though the ICP algorithm is able to converge quickly, around 25 iterations in the case of Shahid and Okouneva [23], that a good initial guess is required. A poor initial guess can lead to a significant errors in pose estim-ation as illustrated in Figure 2.8. The shortcomings with using the ICP algorithm alone was overcome by Woods and Christian [24] by implementing a novel clustered viewpoint feature histogram (CVFH) method. This solution was coupled with an EKF to perform pose estimation of the target.

(26)

Figure 2.8– Illustration of the influence of poor a initial guess on the performance of the ICP algorithm. Wrong convergence to a local minimum due to errors in the initial guess is shown in the top part of the figure. The bottom part shows correct convergence to the absolute minimum point and the achievement of successful pose matching. Figure reprinted from Woods and Christian [24].

A novel pose estimation method was proposed by Pesce et al. [25] using a stereo vision sensor and an EKF. No prior information regarding the target was assumed, but relative attitude dynamics was still exploited by performing a parametrisation of the target’s inertia matrix. Results from this research showed that inertia compon-ents can successfully be estimated, given that the target has high angular velocities. This helps the dynamic model to converge to consistent and exact values for the inertia parameters. Incorrect inertia ratios resulted in a decrease in the capability to accurately determine the angular velocities and the attitude vectors. Pesceet al. [25] also concluded that a robust and reliable stereo feature tracker is required for space-based applications. Occlusions and adverse lighting can have a significant influence on the capabilities of the vision system and the method heavily relies on accurate point cloud information of the target [25].

Biondi et al. [26] proposed a method for obtaining space debris angular rates using feature-based Kalman filtering techniques. This method provides a solution to the limitations of Segal et al. [27] and Pesce et al. [25] where the estimation methods diverge when a lack of consecutive measurements exists. When this is the case, no measurement updates can be done by the Kalman filter and the estimation is purely dependent on the prediction made by the motion model of the target. Stereo vision sensors are used to track target features and a partial pose is computed at timesteps where features are indeed available. The complete attitude signal is then retrieved using compressed sensing techniques often used for signal recovery. Non-linear programming is used based on the assumption that the signal can be represented as a linear combination of a few independent non-linear basis functions [26]. A linear Kalman filter is used to estimate the angular velocities of the target once a full attitude quaternion is available.

A stereo-vision-based filter was developed by Segalet al.[27] to estimate the state of a target relative to a chaser satellite. A coupled dynamic motion model was used to

(27)

predict the state propagation. Multiple iterated extended Kalman filters (IEKF’s) were run in parallel to select an inertia tensor that best suited the dynamic behaviour of the observed object. Analysis indicated that attitude and structure information could be extracted from spatial measurements and could be used for estimating a the full state of a target relative to the sensors.

A survey was done by Chen [28] to investigate the present state and use of Kalman filters for robotic vision and perception applications. It was found that 800 pub-lications were made between 1983 and 2010 where some form of the Kalman filter was used for vision-based implementations. More than 20 variants of the Kalman filter (KF) have been developed over the past 30 years. The main consideration in the complexity of the algorithms lies within the linearity of the systems. Several solutions are used to efficiently deal with non-linear systems, with the EKF and unscented Kalman filter (UKF) being the most common variants. The EKF linear-ises the system states around the current mean and covariances, where the UKF chooses sample points from the current state distribution, puts them through the non-linear function and creates a new distribution based on the transformed points. It was found that more than 70% of the recent publications in the field of robotic vision used the EKF for non-linear state estimation [28]. Noteworthy cases where the EKF is used to track objects for space based applications are found in the works of Malan [17]; Lichter and Dubowsky [21]; Dong and Zhu [20] and Segal et al.[27].

2.6 Conclusion

The current state of space debris and ADR methods has been investigated. Space debris is a real threat to the viability of space exploration in the future. Active debris removal is a heavily researched field and a lot of progress has been made towards it. RemoveDebris was able to successfully capture debris using its net experiment in September of 2018 [14]. This is the first step to a cleaner space, but many challenges still exist. Uncooperative objects with unknown physical properties poses many technical difficulties and not one method is suitable to deal with all types of scenarios [15].

Obtaining accurate pose information of debris and uncooperative objects still re-mains a difficult task and selecting the correct sensors and estimation algorithms is not an trivial undertaking. Pose estimation using spatial sensors is a well-known field and combining multiple sensors is a popular way to improve system reliability [6; 28]. The use of a monocular camera as primary sensor has been suggested by Malan [17], Song and Cao [18], Oumer et al. [19] and Dong and Zhu [20]. Mon-ocular vision is elegant and is the least expensive method to implement in terms of hardware and computation costs. On its own, however, it is not suitable for uncooperative, unknown pose determination since depth information cannot be cal-culated. Auxiliary sensors like Lidar or range finders are required to make this a feasible option.

The use of stereo camera sensors was explored by Pesce et al. [25], Biondi et al. [26] and Segal et al. [27]. Stereo vision cameras overcome the drawback of solely using monocular cameras, since triangulation is done using point correspondences.

(28)

This makes stereo cameras viable as a standalone sensor for pose determination of unknown targets.

It was found that passive sensors like cameras experience a decrease in performance when operating in bad lighting conditions. Maintaining correspondence is also a challenge when using visible light sensors. The option of using Lidar for pose estim-ation was also suggested in literature [21; 22; 23; 24] and it can be concluded that Lidar sensors can deliver reliable measurements over large distances, but require more processing and have higher implementation costs. Lidar sensors are usually used with an ICP algorithm to perform model matching of the target.

Kalman filters were found to be used extensively throughout this study. The EKF is the most popular filter used in the past 30 years for pose estimation in the field of autonomous robotics [28]. The performance of Kalman filters is, however, dependent on the input of initial conditions and measurements with strong assumptions about noise properties. This does not affect the convergence quality, but may influence the performance of the filter in terms of convergence speeds and accuracy [20]. It is evident from this investigation that pose estimation is a genuine problem with real-world applications in various fields. The chapters that follow will present a solution to determine the position and attitude of an uncooperative, unknown target relative to a sensor.

(29)

Chapter 3

Modelling

3.1 Introduction

This project focuses on the pose estimation of a rigid body relative to an observer. This is essentially a localisation and tracking problem and requires a realistic de-scription of the system. The aim of this chapter is to sufficiently define the problem and the proposed solution. Estimation algorithms are discussed and an estimator is chosen to solve the localisation problem. Further, attitude representations of a rigid body are introduced along with the dynamic and kinematic models used to describe the motion of a freely rotating target in inertial space. Attention is given to quaternion attitude representations along with their propagation using angular rates.

3.2 Problem Definition

The mechanics for the rendezvous phase of active debris removal (ADR) are de-scribed by different parts, namely chaser attitude dynamics and kinematics, target attitude dynamics and kinematics and the relative translational dynamics and kin-ematics between the target and the chaser. In this project, it is assumed that the chaser’s attitude and position is known and that the inertial reference frame is fixed to the chaser’s reference frame, (C). Therefore only the target’s translation and attitude mechanics are modelled and described relative to the chaser.

The problem of localising a robot in an unknown environment is often solved using simultaneous localisation and mapping (SLAM). SLAM is a method used by a robot to map an unknown environment and simultaneously locate itself in the map [29]. The probabilistic SLAM approach is used to estimate the pose of a sensor/robot relative to its environment [30]. The sensor, with a pose,x_t, receives control inputs,

ut, and measurements, zt, at a given timestep, t. Given the inputs and measure-ments, the aim is to estimate the location of the landmarks and the sensor’s location relative to the landmarks. The sensor, with reference frame C, shown in Figure 3.1, moves in an environment and uses landmarks in its vicinity to estimate its attitude, position and velocity relative to the inertially defined reference frame,I. In this case

(30)

the inertial reference frame is fixed to the map, m, which consists of the landmark locations. t t+ 1 t+ 2 Ct m I=m Ct+1 C_t₊₂

Figure 3.1– Illustration of the classic SLAM problem, where a robot moves through an unknown environment. Landmarks, indicated by the red crosses, are observed and

used to build a mapm, of the environment and is used by the robot to localise itself

in the map.

For this project, the sensor is considered stationary and the environment or target, moves relative to the sensor. The sensor reference frame is fixed to a chaser satellite and the target can be a debris object or another satellite. Instead of using landmarks like trees or buildings like in the typical SLAM problem, distinct reference points on the target are used. Figure 3.2 illustrates a target with a body-fixed reference frame (B), moving in inertial space fixed to the sensor reference frame. Features on the target are denoted by red crosses and are used to build a body-fixed map (m) of the target and simultaneously estimate the pose of the target relative to the sensor. There are no control inputs, ut, in this problem, since the sensor is stationary and the target is uncooperative.

I=C t t+ 1 t+ 2 B_t Bt+1 B_t₊₂ m m m

Figure 3.2– A stationary sensor or robot observes a moving target in inertial space. The features on the target are indicated by red crosses and are used to construct a map of features on the target over time. The movement of these features are used to estimate the motion of the target relative to the sensor.

(31)

CHAPTER 3. MODELLING 20

The estimation solution is composed of two distinct parts that are designed sep-arately. The first is a stereo camera sensor that provides 3D measurements of the target. Secondly, the observations are used in an estimator algorithm that extracts the states using kinematic models of the system. A block diagram of the system framework is shown in Figure 3.3. This solution does not make use of any prior knowledge of the target. This implies that there is no knowledge regarding the mass properties of the target; therefore, no prior moments of inertia or centre of mass information is available. There is also no appearance information or known fiducials on the target. Robust features are identified that can be tracked reliably while they are in the field of view. This makes the proposed solution powerful in the sense that it can be used to track the pose of unknown objects. The solution is not limited to space technology and can be used in other robotic problems such as autonomous pick-and-placing [31; 32]. Sensor Feature Extraction image right Triangulation Motion Update Measurement Update image left Feature Extraction 3D Points Estimator

Image Data Pose Estimate

Figure 3.3– System framework block diagram.

The solution makes use of well known mechanics introduced in Section 3.4 to model the motion of the target. It is assumed that the sensor and target are in the same orbit and that the sensor is in a rendezvous phase with the target, as discussed in Section 2.4. It is further assumed that the target is rigid and there are no external forces or torques present on the target. The total energy of the target will thus remain constant.

3.3 Recursive Estimation

This section describes the elements of recursive estimators along with a few commonly-used estimators in the fields of localisation and tracking. A recursive estimator uses the previous distribution over a set of system states and current sensor data to es-timate the current state distribution. Figure 3.4 illustrates the basic workings of a discrete recursive filter.

(32)

Figure 3.4– Recursive estimator algorithm flow chart [33].

The state vector is represented byx_tfor the estimation problem in the discrete time domain. The process, or state transition function, is expressed as

x_t=f(x_t−1,ut−1,wt−1), (3.3.1)

where f is either a linear or non-linear transition function and w_t represents the process noise. New observation data, zt, is available at discrete timesteps and can be related to xt by the measurement function,

zt=h(xt,vt). (3.3.2)

The measurement uncertainty is represented by vt and h is the observation model, which can also either be linear or non-linear. The goal is to obtain the posterior dis-tribution,p(x_t|z_1:_t), over the state vector x_t. This is done by recursively performing the process and measurement updates.

At a timet, the posterior distribution overxt−1 at timet−1 is known and the prior distribution at tis calculated as [34],

p(xt|z1:t−1) =

Z

p(xt|xt−1)p(xt−1|z1:t−1)dxt−1. (3.3.3)

The measurement update is used to calculate the new posterior, at timet, given the prior state distribution [34] according to Bayes’ rule,

p(xt|z1:t) =

p(z_t|x_t)p(x_t|z_1:_t−1)

p(zt|z1:t−1)

(33)

The Kalman filter is a popular estimator used in pose estimation problems. It is a special case of the Bayes filter, where Gaussian noise distributions are assumed [34]. The assumption is also made that the initial distribution of the system can be represented by a Gaussian distribution. A control and measurement update is executed at each sampling instant to update the distribution over the states. If the previous state distribution is Gaussian, then the updated current distribution will also be Gaussian and therefore the best estimate is chosen as the mean of the distribution.

Different variants of the Kalman filter exist, of which the extended and unscented Kalman filters are the most popular, each with their own unique characteristics.

• The extended Kalman filter (EKF) overcomes the restrictions of the linear filter by approximating non-linear functions to be linear using a first-order Taylor expansion. The mean position of the state vector is used as the linear-isation point around which the tangent of the non-linear function is calculated, allowing the use of standard Kalman filter equations. It is typically more effi-cient than other non-linear filters which sometimes comes at a cost of reduced accuracy.

• The unscented Kalman filter (UKF) uses stochastic linearisation to deal with non-linear systems. Given a distribution with known mean and covariance, a set of weighted points, known as sigma points, are chosen and transformed using the non-linear function. A new distribution is determined from the transformed sigma points. The process and observation functions do not need to be differentiable and the output is based on values in a larger region, rather than a local approximation.

Kalman filters are well suited for localisation problems since the nature of these systems are normally non-linear. Both the linear and non-linear variants of the Kalman filter are concerned with estimating states using motion and measurement models fused with sensory data. Kalman filters use a system model to perform this data fusion. The following equations are used to model the system [34]:

x_t=Atxt−1+Btut−1+ z_t=Ctxt+ζ , (3.3.5) where =N(0;R) ζ =N(0;Q). (3.3.6)

The matrices R and Q are the known covariance matrices of the process and ob-servation noise, respectively and the matrices A, B and C form part of the linear functions. In Algorithm 1, Thrun et al.[30] describe the calculation of the updated distribution over xtas,

(34)

p(x_t|u_1:_t,z_1:_t) =N(µ_t; Σ_t). (3.3.7)

Lines 1 to 3 describe the process update. The measurement update is performed in lines 4 to 6 to calculate the posterior distribution, after which the updated mean and covariance is returned in line 7.

Algorithm 1 Kalman Filter

1: Given knownµ_t−1 and Σt−1:

2: µ¯_t=Aµ_t−1+But 3: Σ¯_t=AtΣt−1ATt +Rt 4: Kt= ¯ΣtCtT(CtΣ¯tCtT +Qt)−1 5: µ_t= ¯µ_t+Kt(zt−Ctµ¯t) 6: Σ_t= (I−KtCt) ¯Σt 7: return µ_t,Σt

The state estimation problem in this project makes use of non-linear system models, and a high-dimensional state space. The EKF is well suited for this problem, since it accommodates non-linear process and observation models and is capable of dealing with high-dimensional state spaces. The EKF is often used for the SLAM problem and is well known in the field of robotics and localisation. The EKF is chosen for this project and is discussed in greater depth in Section 5.4.

3.4 Rigid Body Mechanics

3.4.1 Kinematics

The pose of a rigid body in a reference frame consists of the position and attitude of the body. The attitude, or orientation of a body-fixed reference frame with respect to a known reference frame, is usually represented by a rotation matrix, often referred to as a direction cosine matrix (DCM) [35]. A rotation about a single coordinate axis is referred to as a coordinate rotation. A coordinate rotation about the x-, y -and z-axes with angles φ,θ and ψ, of a body can respectively be described as,

Rx(φ) =    1 0 0 0 cos(φ) sin(φ) 0 −sin(φ) cos(φ)    Ry(θ) =    cos(θ) 0 −sin(θ) 0 1 0 sin(θ) 0 cos(θ)    Rz(ψ) =    cos(ψ) sin(ψ) 0 −sin(ψ) cos(ψ) 0 0 0 1   . (3.4.1)

(35)

Any rotation in 3D space can be described by three coordinate rotations. The DCM describing the attitude of the target in the camera reference frame (CRF), AB_C , can be represented by three Euler angles. Each of the angles corresponds to one coordinate rotation. The order of the Euler 1-2-3 rotation, shown in Figure 3.5, is expressed as, AB_C =Rx(φ)Ry(θ)Rz(ψ) (3.4.2) =    a1,1 a1,2 a1,3 a2,1 a2,2 a2,3 a3,1 a3,2 a3,3    =    CθCψ SθCψ −Sθ SθSφCψ−CφSψ SθSφSψ+CφCψ CθSφ SφSψ+SθCφCψ SθCφSψ−SφCψ CθCφ   , (3.4.3)

whereS is the sine function andC the cosine function. The Euler angles are calcu-lated as φ= arctan 2 a2,3 a3,3 ! , θ= arctan 2   −a1,3 q a2 1,1+a21,2  , and ψ= arctan 2 a1,2 a1,1 ! . (3.4.4) θ θ φ φ ¯ zC ¯ xC ¯ yC ¯ y0 ¯ z0 ¯z0 ¯ y0 ¯ xC ¯ zB ¯ x0 ¯ zB ¯ yB ¯ xB ¯ x0 ψ ψ ¯ y0

Figure 3.5– Euler 1-2-3 rotation

Mathematical singularities occur when using Euler angles to represent large rota-tions. When both a1,1 and a1,2 in Equation 3.4.2 are zero, the expressions for ψ

(36)

and θ are undefined. This is known as gimbal lock, where the changes in the first and third Euler angles are indistinguishable when the second angle nears a critical value. Alternatively, the DCM can be described using quaternions, which do not have these singularities. The quaternion rotation in Figure 3.6 is expressed by the Euler axis ¯e= [ex, ey, ez]T and an angle θ,

q=      q1 q2 q3 q4      =      cos(θ/2) exsin(θ/2) eysin(θ/2) ezsin(θ/2)      . (3.4.5) ¯ y ¯ x ¯ z ¯ e θ

Figure 3.6 – Quaternion rotation

The DCM as a function of a quaternion set is expressed as,

AB_C =    q₁2+q₂2−q₃2−q₄2 2(q2q3−q1q4) 2(q1q3+q2q4) 2(q1q4+q2q3) q12−q22+q32−q42 2(q3q4−q1q2) 2(q2q4−q1q3) 2(q1q2+q3q4) q21−q22−q23+q24   . (3.4.6)

Using the normalisation constraint, q₁2+q₂2+q2₃+q2₄ = 1, the DCM is simplified to,

AB_C =    1−2q2₃−2q₄2 2(q2q3−q1q4) 2(q1q3+q2q4) 2(q1q4+q2q3) 1−2q22−2q42 2(q3q4−q1q2) 2(q2q4−q1q3) 2(q1q2+q3q4) 1−2q22−2q23   . (3.4.7)

The body-fixed angular rates of the target in the CRF,ωB_C, is expressed as a function of quaternion rates by,

ωBC =    ωbx ωby ωbz   = 2    −q2 q1 −q4 q3 −q3 q4 q1 −q2 −q4 −q3 q2 q1         ˙ q1 ˙ q2 ˙ q3 ˙ q4      . (3.4.8)

(37)

CHAPTER 3. MODELLING 26      ˙ q1 ˙ q2 ˙ q3 ˙ q4      = 1 2      0 −ωbx −ωby −ωbz ωbx 0 ωbz −ωby ωby −ωbz 0 ωbx ωbz ωby −ωbx 0           q1 q2 q3 q4      . (3.4.9)

Quaternions will be used throughout this thesis for attitude representations. Qua-ternions do not have ambiguity regarding the order of rotations and the rotation is around a well-defined axis. The sin and cos elements of the rotation matrix are already encoded in the quaternion form of the DCM. Therefore, only one matrix operation is required for attitude transforms, where Euler angles require three.

3.4.2 Dynamics

Newton-Euler equations are used to describe the rotational dynamics of the target and it is applicable to all rigid inertial bodies [36]. For this project, it is assumed that the angular momentum of the target remains constant and is expressed as,

˙

H= dH

dt =Iω,˙ (3.4.10)

where H is the angular momentum and the diagonalised moment of inertia tensor is represented by I. When there are no external forces present, the rotational kin-ematics of a rigid body around its centre of mass and around its principal axis of inertia can be written as,

Ixxω˙x =ωyωz(Iyy−Izz),

Iyyω˙y =ωxωz(Izz−Ixx) and

Izzω˙z =ωxωy(Ixx−Iyy),

(3.4.11)

withIxx,Iyy andIzz constant and dependent on the shape and mass distributions of the body. The magnitudes of the moments of inertia depend on the mass distribution of the target. It is worthwhile to discuss the stability behaviour of an object due to its mass distribution. It is stated by Marsden and Ratiu [37] that the rotation of a free rigid body is stable around its longest and shortest axis and rotation about the middle axis will not be stable. If the energy in the system is assumed to be constant, then the energy is only distributed over the three principal body rates. Spin about the intermediate axis is not stable and over time the energy is dissipated to the major and minor axes. If the angular velocity is about a non-principal axis, a nutation will be superimposed on the rotation.

Newton’s second law of motion describes the linear motion of an object with mass,m

and the exact discrete solutions for the displacementdt and velocityvt is expressed as, dt=dt−1+vtδt+ 1 2mF(t)δt 2 _(3.4.12) vt=vt−1+ 1 mF(t)δt, (3.4.13)

(38)

where F(t) is the force working on the object. Since no external torques and forces are assumed and the mass is unknown, the last term in Equations 3.4.12 and 3.4.13 are eliminated. The linear velocity of the target at the current timestep will thus only depend on the previous velocity.

3.5 Conclusion

A formal definition of the project problem and proposed solution was provided in this chapter. The SLAM approach was discussed and how it is used to solve the pose estimation problem of a rotating and translating rigid body. Recursive estimators and Kalman filters were investigated. Mechanics of moving bodies in 3D space were discussed.

The rotational motion of the target implies that a non-linear model will be required to describe the process update of the system. Using a dynamic process model is not possible without information of the relative inertias. Therefore, a kinematic estimator will be implemented, even though it might lead to slower convergence or delays in rate estimates. Using a dynamic estimator with inaccurate inertia information may lead to severe errors in the estimation of all target states.

This project aims to verify the use of an enhanced SLAM solution to solve the autonomous rendezvous problem for space-based applications. Attention is given to the development of the stereo camera sensor in the next chapter and how features on the target surface can be extracted for use in the EKF-SLAM algorithm.

(39)

Chapter 4

Image Processing

This chapter focuses on sensor modelling and measurement extraction using a stereo vision sensor. A feature-based method is used, where the position of reference points on the target are tracked over time. It is necessary that these distinct image points are extracted consistently so that correspondence can be established. The monocular pinhole camera model is introduced to describe how 3D world coordinates are related to the CRF. Stereo geometry and depth calculation is discussed and a sensor model is derived. The feature class and methods for how features are detected and matched using image processing techniques are introduced.

4.1 Pinhole Camera Model

It is necessary to model the transformation from 3D world points to 2D image plane coordinates. The pinhole camera is an ideal model that adequately represents this perspective transformation. An ideal camera assumes all light enters through a pinhole or single point known as the optical centre. The light produces a vertically flipped image that is projected onto an image plane behind the optical centre, at a focal distance, f. The image plane can also be placed in front of the optical centre which results in an upright projection. The image created in the pinhole camera is perfectly focused, the aperture is infinitely small and no geometric lens distortion exists.

(40)

¯ yc ¯ xc ¯ zc u v p_im p_w c f

Figure 4.1 – Pinhole camera model

Figure 4.1 shows a pinhole camera model with the principal axis in line with the ¯zc -axis and the centre of projection,c at the origin of the coordinate system. A point,

p_w = [xw, yw, zw]T, in 3D space is projected onto the image plane at pim = [u, v]T. Using similar triangles, pim can be described as

p_im= fxw zw fyw zw T . (4.1.1)

Using homogeneous vectors, the perspective transformation fromp_wtop_imis defined as    u v w   = [K 0]      xw yw zw 1      , (4.1.2) where K=    f 0 0 0 f 0 0 0 1   . (4.1.3)

The 3×3 projection matrixK, assumes that the origin of the image coordinates is at the principal point where the ¯zc-axis intersects the image plane. It is not always the case in practical systems and therefore translation elements, px and py, are added to K. Scaling changes between the world and image coordinate systems, mx and

Space debris: pose estimation using stereo vision

Pose Estimation Using Stereo Vision

Willem Carel de Jongh

Plagiarism Declaration

W.C.

de

Jongh

April

201

9

Abstract

Uittreksel

Contents

List of Figures

List of Tables

Nomenclature

Chapter 1

Introduction and Problem

Description

1.1

Problem Background

1.2

Project Aim and Proposed Solution

1.3

Document Outline

Chapter 2

Literature Study

2.1

Introduction

2.2

Space-Based Pose Determination

2.3

Overview of Space Debris

2.4

Active Debris Removal

2.5

Pose Estimation Methods From Previous Research

2.6

Conclusion

Chapter 3

Modelling

3.1

Introduction

3.2

Problem Definition

3.3

Recursive Estimation

3.4

Rigid Body Mechanics

3.5

Conclusion

Chapter 4

Image Processing

4.1

Pinhole Camera Model