4 Navigation & Docking System
4.4 LEDs Detection and Tracking
camera
Target Chaser
4cm 1cm
5cm 2cm
5cm 2cm
visibility cone
Figure 4.17 – Position of the two LED patterns and camera in the target and chaser structures. The dashed lines represent the LEDs visibility cone.
of 41× 29 × 29 mm and 70 × 29 × 29 mm including optics, i.e. 0.03 U. The total mass is 163 g.
Figure 4.18 – Basler ACE camera acA3800-10um and 4 mm focal length optic [293].
The camera sensor has its peak quantum efficiency at 460 nm. LEDs emitting in the blue part of the spectrum are thus selected. To minimize the influence of stray light and potential Sun blinding, a bandpass filter is used. The COTS filter peak transmission is at 470 nm [295]. The selected LEDs have a peak emission at 470 nm, a viewing angle of 80 deg and a luminous intensity of 17 cd at 350 mA.
4.4 LEDs Detection and Tracking
The EKF takes as input the LEDs positions on the CCD which have to be detected on the camera image. Even though several algorithms can be used to detect forms on images, in order to keep the computational load at the lowest possible level, Blob analysis of the MATLAB® Computer Vision System toolbox is selected. This algorithm detects connected regions (blobs) on a binary image without extensive mathematical operations.
The basic working principle of blob analysis is shown in Figure 4.19.
Chapter 4. Navigation & Docking System
(a) Grayscale image. (b) Binary image.
Figure 4.19 – Grayscale to binary image conversion using thresholding and connected region detection.
a value higher than the threshold are converted to white and the other to black. Then, all the connected white pixels are gathered in blobs (or centroids). The LEDs centroid coordinates are then fed to the EKF (4.16) or to the analytical solution (4.22).
The drawback of such a simple algorithm is that there is no a priori way to reject undesired signals. As opposed to the Hough transform or more advanced image processing algorithms, which can detect specific shapes, the connected region algorithm detects any blob of white pixels. Note that the connected region algorithm is capable of characterising the centroid shape, i.e. computing its semi-major and -minor axes. As each LED blob should be close to circular, it should be possible to exclude non-circular centroids. However, as in far-rage each LED only covers a few pixels, and the circularity becomes difficult to assess, this option is discarded.
To be interesting for CubeSats, the proposed VBN should allow docking in any illumination conditions in order to minimise operation time and to increase robustness during final approach. Parasite signals on the image due to Sunlight (reflections on the target structure, blinding, stray light, Earth albedo) or any other signals appearing on the CCD due to radiations, must be expected and a robust detection algorithm is required.
The case when part of the Earth is in the camera FoV can be easily dealt with by discarding blobs connected to the sensor border. The LEDs are surrounded by the CubeSat structure, and because of the FoV size, the LEDs images will never be close to the sensor border.
Two cases have to be considered:
4.4. LEDs Detection and Tracking
1. The VBN filter has converged and its precision allows active tracking of the LEDs.
2. The VBN is not yet initialised or its precision is not within requirements; active tracking of each LED cannot be performed. This can be solved using the geometrical properties of the two patterns .
When the VBN filter has converged, active tracking of the LEDs is achieved with help of the measurement equations (4.42), (4.44), as well as the navigation filter prediction xˆ−k to obtain their a priori estimated position within a certain confidence interval, given by the EKF covariance. The filter covariance defined a Region of Interest (ROI) on the image which will reduce the number of pixels that the connected region algorithm has to analyse (see Figure 4.20). Windowing the image will also reduce the number of parasite signals that the camera may detect. Furthermore, a confidence interval on the predicted LEDs position can be obtained defining ROIs around each LED. This represents their predicted location.
Figure 4.20 – Generated image of the central LED pattern observed by the camera.
The total ROI is displayed as well as each individual LED ROI. The cross markers are the LEDs a priori positions.
Assumingn LEDs have to be observed (xinc,i = 1, ..., n) and m connected regions are detected (xjblob, j = 1, ..., m); if m > n, n blobs among m regions need to be selected.
The a priori position of each LED is provided by the EKF:
xˆinc =h&
xˆ−k'
(4.48)
whereh(x) is the EKF measurement equation (4.36b). The norm between each estimate and each signal can be calculated
ˆxinc− xjblob , i = 1 , ..., n, j = 1 , ..., m (4.49)
Chapter 4. Navigation & Docking System
and is shown in Figure 4.21. The n connected regions which are closest to the a priori positions can then be selected. This tracking method works for both the inner and the outer patterns.
Note that if a parasite signal within a LED ROI is selected by mistake, the navigation solution is marginally impacted as the wrong signal position is in the order of the current estimation error.
ˆx5nc − xlblob ˆx5nc − xjblob
Figure 4.21 – The gray crosses are the a priori estimated LEDs’ positions ˆxinc, i = 1, ..., 5. The black elements are signals detected on the CCD.
If the VBN filter did not converge, the individual tracking of each LED is not possible, and another solution must be used.
4.4.1 Geometrical Features
A first iteration to solve the detection problem involved machine learning. Classifiers, with the number of inputs equal to the number of LEDs’ coordinates on the CCD, were trained and compared among themselves. The best classifier, a decision tree, detected the LEDs in 80% of the cases. The remaining false positives were groups of blobs which had similar patterns but not in the same proportions. It was then decided to extract geometrical features from the patterns and use them to train the classifiers. Better results were obtained, with 93% detection rate using a decision tree. As a decision tree is doing nothing more than comparing features’ values against expected values obtained during the classifier training, and as geometrical features and expected values can be easily computed, it was decided to perform the selection manually. This approach was preferred to the machine learning algorithm as it removed the “black box” disadvantage of a classifier.
Because the two patterns have fixed geometrical properties, features can easily be extracted.
Figure 4.22 shows the features for the external and internal patterns.
For the external pattern (Figure 4.22a), define:
4.4. LEDs Detection and Tracking
Figure 4.22 – Geometrical features of the two different LEDs patterns.
• d1 and d2 the lengths between the central and outer LEDs.
• d3 the length between the two outer LED’s.
The features are then:
1. α, the angle between d1 and d2.
2. β, the angle between d3 and the sensor horizontal coordinate.
3. d1/d2, the ratio between the outer and central LEDs.
4. d3 the pattern size.
As the perspective effects caused by control and navigation errors are small, the angleα is expected to be always close to π. The maximum expected attitude errors are known and boundaries forα can be predicted. The distance ratio between the outer LEDs and the central LED,d1/d2, is expected to be close to unity. The pattern should be aligned with the camera sensor horizon, andβ must be close to zero. Finally, the expected maximum and minimum number of pixels between the two outer LEDs can be obtained using (4.1).
The upper bound for this value is given at 5 m and the lower bound at a 10 m range.
The feature vector is thus:
For the central pattern, shown in Figure 4.22b, the features extracted are similar than for the outer pattern. The following values are defined:
Chapter 4. Navigation & Docking System
• d2,4 and d1,3 the lengths between the outer LEDs.
• d2,4 and d1,3 the pattern size.
• γ the angle between d1,3 and the sensor horizontal coordinate.
• d2,4 and d1,3 the pattern size.
• d2,4/d1,3 the length ratio between the outer LEDs.
The eight length ratiosd1/d2, ...,d3/d4, andd2,4/d1,3, are expected to be close to one.
The anglesα1,α2, as in the case of the outer pattern, are expected to be close to π. β should be close to π2 andγ close to zero. Finally to bound the maximum and minimum pattern size, d2,4 and d1,3 can be computed at 5 m and at docking using (4.1). The feature vector is thus:
Note that the features are dependent on the LEDs selection order. The LEDs designation is shown in Figure 4.23. For the central pattern, the LEDs will always be numbered from 1 to 5, and for the outer pattern, the LEDs designation are: a, b, and 5. This sequence described and shown is imposed.
Figure 4.23 – LEDs numbering for the inner and outer patterns.
4.4. LEDs Detection and Tracking
For the internal and external patterns, the features values can be bounded using the expected performances of the GNC plus margins, and the LEDs measurement equations (4.16).
Using the GNC requirements (5% of the range relative position accuracy and 2 deg relative attitude), the features for the inner and outer pattern can be generated. In order to have sufficient margins, a relative lateral position corresponding to 15% of the range and a relative attitude error of 5 deg around each axis is considered. The features have been generated systematically by scanning sets of extreme error values using the requirements mentioned above plus margins. For the outer pattern, values at 12 m, 8 m, and 4 m range have been generated. For the inner pattern, values at 6 m, 5 m and 7 cm range have been generated. Taking the extremal values of each feature allows defining upper and lower bound on douter and dinner.
To differentiate the unwanted signals from the LEDs’, groups of three blobs are constructed for the external pattern and groups of five blobs for the internal one. For each group, the features are extracted and tested against their expected values. If the features are within the bounds, LEDs are considered detected. This method has the inconvenience that potentially more than one pattern per image can be detected.
4.4.2 Detection Algorithm Performances
To assess the performances of the proposed detection method, images of patterns at various relative positions and attitudes have been generated using MATLAB® Simulink.
Fake signals, representing reflections have been randomly added to the images. The LEDs appear as circular marks on the image whereas the false signals have rectangular shapes.
Recall that the signal shape is not used by the connected region algorithm to pre-filter the output centroids.
The VBN starts at 10 m range, and the 3 LEDs pattern needs to be detected. This is shown in Figure 4.24. The detection algorithm can always successfully recognise the LEDs and reject perturbations. The case where a reflection is present between the LEDs has not been simulated explicitly as in far range, most perturbations come for the surrounding environment. Note that such extreme cases are expected only when first initialising the VBN EKF. Once converged, the filter covariance will be used to define ROIs and thus limit significantly the number of perturbations to reject.
Figure 4.25 shows the central LEDs pattern at a range of 1 m. In Figure 4.25a all the perturbations are successfully rejected. The only case in which the LEDs could not be measured is when a fake signal borders an LED. In this case, the combined LED+perturbation centroid is too far from the accepted value, and the pattern is rejected.
This could typically happen if the Sun was reflected on the pattern edge. Note that the
Chapter 4. Navigation & Docking System
(a) 3 LEDs pattern at 10 m. (b) Zoom on the 3 LEDs pattern.
(c) 3 LEDs pattern at 5 m. (d) 3 LEDs pattern at 5 m.
Figure 4.24 – 3 LEDs pattern observed at a range of 10 m and 5 m. The detection algorithm can reject all the perturbations 100% of the time.
footprint on the CCD is simply smaller.
The detection becomes more complicated when the fake signal distribution is comparable to the central pattern size. This typically happens just before docking, at a range of 7.5 cm. In Figure 4.26, two examples of successful detection are provided. Although the LEDs are surrounded by fake signals, they do not fit the features boundaries, and only the pattern is detected.
4.4. LEDs Detection and Tracking
(a) Pattern detected. (b) Pattern not detected.
Figure 4.25 – 5 LEDs pattern observed at a range of 1 m. In b) the pattern is not detected as a perturbation blob is connected to the LED blob.
(a) Pattern detected. (b) Pattern detected.
Figure 4.26 – 5 LEDs pattern observed at a range of 7.5 cm. In b) the pattern is not detected as a perturbation blob is connected to the LED blob.
Not detecting the LEDs never happened. However, it occurred several times that a nearby fake signal was taken for an LED, leading to more than one possible pattern. This is illustrated in Figure 4.27.
In Figure 4.27a, LED and fake signal blobs merged. In Figures 4.27b, 4.27c, and 4.27d, one or two fake signals were mistaken for LEDs, leading to several potential patterns.
Chapter 4. Navigation & Docking System
(a) Fake signal and LED merged blobs. (b) Two patterns detected.
(c) Two patterns detected. (d) Three patterns detected.
Figure 4.27 – 5 LEDs pattern observed at a range of 7.5 cm. In a), the LED blob and a fake signal one merged into one centroid. In b), c), and d), more than one pattern are detected.
Two or more patterns which are similar in shape are detected, and this would lead to small navigation errors.
In the extreme cases as shown in Figure 4.28, the wrong pattern would induce considerable errors in the filter – typically several degrees of error in the LoS determination – and an additional method to distinguish the possible patterns must be implemented.
The solution to this problem is to combine the active tracking of the LEDs with the geometrical features algorithm. The distance between each potential pattern and the
4.4. LEDs Detection and Tracking
Figure 4.28 – Six possible patterns detected.
VBN EKF a priori solution, h(ˆx−k), is computed. The pattern which minimises this norm is then selected, as discussed in Section 4.4. In this case, even if the wrong pattern was selected, the navigation errors will remain consistent with the EKF current accuracy level.
It is not possible to use the filter information to accurately select the LEDs before the first initialisation of the VBN EKF. However, it is possible to use the relative GNSS position and the target/chaser absolute attitude navigation solutions to obtain the LEDs’
Table 4.3 – LEDs detection and tracking algorithm for the inner and outer patterns.
Prediction Use the a priori navigation solution to estimate an ROI and the LEDs position.
Centroid detection 1. Threshold the image.
2. Detect all centroids within the ROI.
Geometrical features 1. Among all detected blobs, create groups of 3 blobs (outer pattern) and 5 blobs (inner pattern).
2. For each group of blobs, compute the geomet-rical features.
3. Select all blobs satisfying the geometrical features boundaries.
LEDs selection 1. For all possible detected patterns, compute distances to the a priori pattern.
Chapter 4. Navigation & Docking System
a priori position. The proposed method combining the geometrical features and active tracking can, therefore, be used at any time. The complete detection and tracking process is summarised in Table 4.3. This algorithm can be used at any range for the inner and outer pattern.
The VBN EKF performances will be provided in Chapter 6. Four examples are provided in Figure 4.29. Note that in these examples the fake signals size has been reduced to avoid overlaying with the LEDs. The blue dots are the LEDs a priori position. The blobs highlighted in green represent possible signals belonging to the patterns, detected using the geometrical features. Computing the distance between each pattern and LEDs a priori position allows selecting one solution which is then highlighted in red and used in the navigation filter. The correct pattern is detected in all cases. Even when a fake signal borders an LED, modifying the shape of the blob, as shown in Figure 4.27a, the navigation solution is not affected. The filter performances will be discussed later on.
Note that in practical laboratory experiments, no reflections have been observed near the central pattern. Both patterns, are visible all the way from 10 m to docking, with always only one pattern detected. This is discussed in the next section.