3.4 Conclusions and Future Work
4.2.13 Accuracy Assessment Using Verification Points
The method used to assess the accuracy of the derived models in comparison to the total station survey is to report the difference between the precisely-surveyed VPs and their identified location in the derived point cloud. In PhotoScan, this is done by placing a non-ground control marker in the centre of the target in multiple images, and PhotoScan reports the difference between the estimated position of those image marker points in the model to the supplied precise survey coordinate (as X, Y and Z error). In each GCP density scenario, only the chosen set of GCPs were activated as ground control.
7Under Optimization at
Chapter 4
The remaining GCPs were ignored in these scenarios. In each GCP accuracy scenario, the GCP coordinates provided were either the set of precise survey coordinates or the DGPS equivalents. For each scenario, a set of VP errors were exported and used to derive metrics for assessing accuracy in X, Y, Z, XY and XYZ (RMSE, mean, median, standard deviation, minimum and maximum).
4.3
Results and Discussion
The mapping accuracy achieved for each of the scenarios is summarised in the following figures and tables. Refer to Table 4.1 for a summary of the coded scenarios.
4.3.1 Calibration Options
Pre-calibration based on a target field (PS13GCP2mm/PS13GCP2mmObl and Pre13GCP2mm/ Pre13GCP2mmObl), on-the-job self-calibration (Self13GCP2mm/Self13GCP2mmObl)
and pre-calibration derived from a checker board pattern (Lens13GCP2mm and Lens13GCP2mmObl) are compared in Figure 4.4 and Table 4.2. The results indicate that the checker
board calibration performed the most poorly. This is particularly evident in the ver- tical accuracy statistics, with substantially lower precision and significant bias. Pre- calibration solutions perform marginally worse than on-the-job self-calibration solutions, particularly in terms of vertical accuracy. The control in these scenarios is very pre- cise, and with the exception of models that employ the checker board pre-calibration (Lens13GCP2mm/Lens13GCP2mmObl), this leads to precise models with no evidence of significant systematic errors. An on-the-job self-calibration that includes oblique photography (Self13GCP2mmObl) results in the most accurate model. That accuracy degrades when the oblique imagery is not included in the solution and results in a model with accuracy comparable to the robust pre-calibration. Measured in terms of achieved precision, the ranking of choices is:
1. On-the-job calibration using a network that includes oblique photography.
2. Either an on-the-job calibration using only nadir photography, or a robust pre- calibration.
Chapter 4
Figure 4.4: Box plots of the four calibration options forσ = 2 mm and with and without oblique imagery.
Table 4.2: RMSE for each of the four calibration options tested forσ= 2 mm and with and without oblique imagery.
Scenario RMSEXY RMSEZ
(mm) (mm) Lens13GCP2mm 8.8 41.0 Lens13GCP2mmObl 8.7 39.3 PS13GCP2mm 4.2 8.3 PS13GCP2mmObl 4.1 8.1 Pre13GCP2mm 7.3 7.1 Pre13GCP2mmObl 7.1 7.2 Self13GCP2mm 5.1 6.4 Self13GCP2mmObl 3.2 7.8
Chapter 4
4.3.2 GCP Accuracy (GCP σ)
In assessing the impact of GCP accuracy, we will first consider the 13 GCP scenarios (Figure 4.5) before comparing and examining the impact of reducing the number of GCPs to five in Section 4.3.3 (Figure 4.6). The resulting RMSEs are summarized in Table 4.3.
Table 4.3: RMSE for pre-calibration and on-the-job self-calibration for three different GCPσ
scenarios when using a strong control network (13 GCPs) and a sparse control network (5 GCPs).
Scenario RMSEXY RMSEZ Scenario RMSEXY RMSEZ
(mm) (mm) (mm) (mm) Pre13GCP0mm 3.6 5.8 Self13GCP0mm 1.4 5.1 Pre13GCP0mmObl 3.5 5.8 Self13GCP0mmObl 1.3 5.9 Pre5GCP0mm 7.0 7.3 Self5GCP0mm 2.7 5.8 Pre5GCP0mmObl 6.7 7.1 Self5GCP0mmObl 2.1 6.3 Pre13GCP2mm 7.3 7.1 Self13GCP2mm 5.1 6.4 Pre13GCP2mmObl 7.1 7.2 Self13GCP2mmObl 3.2 7.8 Pre5GCP2mm 7.1 7.4 Self5GCP2mm 6.0 13.6 Pre5GCP2mmObl 8.8 7.8 Self5GCP2mmObl 4.3 11.5 Pre13GCP22mm 9.1 12.4 Self13GCP22mm 10.3 16.6 Pre13GCP22mmObl 9.1 12.6 Self13GCP22mmObl 7.0 11.9 Pre5GCP22mm 8.7 20.0 Self5GCP22mm 10.5 19.8 Pre5GCP22mmObl 8.6 20.0 Self5GCP22mmObl 7.3 15.9
Similar to the findings reported in Section 4.3.1, when control is precise (σ≤2 mm), then
on-the-job self-calibration (Self13GCP0mm/Self13GCP0mmObl and Self13GCP2mm/Self13GCP2mmObl) and pre-calibration (Pre13GCP0mm/Pre13GCP0mmObl and Pre13GCP2mm/Pre13GCP2mmObl)
both produced very accurate models, and including oblique imagery did not significantly impact the results. Fixing the marker accuracy setting in PhotoScan at 0 mm produced more accurate models than those that used the 2 mm setting, particularly for the on- the-job self-calibration scenarios.
When GCP precision was degraded to 22 mm, the positive impact of including oblique imagery became more apparent in the self-calibration scenario (Self13GCP22mmObl), which was shown to be the most accurate 22 mm scenario. The pre-calibration RMSEXY = 9.1 mm, and on-the-job self-calibration RMSEXY = 7.0 mm. There was a 3.3 mm horizontal and 4.7 mm vertical improvement for the on-the-job self-calibration scenario with oblique imagery (Self13GCP22mmObl) versus the on-the-job self-calibration sce- nario without oblique imagery (Self13GCP22mm). The implication is that if the GCP survey is undertaken using DGPS (σ = 22 mm), then on-the-job self-calibration with oblique imagery produces the most accurate model.
Chapter 4
Figure 4.5: Pre-calibrationversus on-the-job self-calibration scenario comparison using a strong control network (13 GCPs), with and without oblique imagery.
Figure 4.6: Pre-calibrationversus on-the-job self-calibration scenario comparison using a
Chapter 4
4.3.3 GCP Density (GCP Count)
The impact on model accuracy of reducing the number of GCPs to five will be assessed (Figure 4.6), and then the 13 GCP (dense) and five GCP (sparse) scenarios will be compared (Figure 4.5versus Figure 4.6). Table 4.3 summarises these results.
When the GCP density was sparse (five GCPs), the overall accuracy was degraded, par- ticularly in the vertical, regardless of calibration choice. When control was precise (σ≤
2 mm), the bias in the pre-calibration solutions (particularly in XY) implies that on-the- job self-calibration produced the most accurate models, and including oblique imagery had little impact. When control was less accurate (σ= 22 mm), all solutions show bias, and on-the-job self-calibration with oblique imagery produced the most accurate model (particularly when comparing vertical accuracy). Once again, on-the-job self-calibration with oblique imagery was the best option whenσ= 22 mm and when control was sparse. As in the dense control scenarios, including oblique imagery provided little or no benefit when the control was precisely surveyed.
When comparing dense (13 GCPs) and sparse (five GCPs) GCP density scenarios, the most accurate models were produced when using a higher number of GCPs. The impact was smaller when control was precise, particularly when using pre-calibration. The vertical accuracy was more greatly influenced than the horizontal accuracy when the number of GCPs was degraded, particularly when σ = 22 mm and when σ = 2 mm. When there were only five GCPs, fixing the control (settingσ= 0 mm) had a significant impact on accuracy in the on-the-job self-calibration case, particularly in the vertical. The same impact was not seen in the 13 GCP scenarios. When GCPs were less accurate (σ = 22 mm), the addition of oblique imagery improved model accuracy in both the 13 and five GCP cases. Reducing the number of GCPs degraded the vertical accuracy of the models, whereas the horizontal accuracy of the models was not adversely impacted by this reduction in GCP density. In this study, five GCPs represents a practical minimum, since fewer would result in significant areas of terrain without nearby ground control, and 13 is likely to be a practical maximum, since residuals are approaching the limits of measurement precision. The findings of this study do suggest that there is scope for undertaking a similar study over a larger area in order to produce more ‘scalable’ rules for camera calibration and GCP distribution.
The overarching goal of this research is to better understand the implications of UAV survey design on the capacity to reliably measure topographic change, such as occurs in eroding coastal landscapes.
Chapter 4
The precise survey used to validate our model demonstrates that UAV-MVS has provided sub-centimetre accuracy point clouds from 25–30 m AGL, which, in turn, allows change detection at the centimetre level.
The investigation demonstrates that self-calibration is comparable to pre-calibration when the GCP survey is designed with careful consideration of GCP survey accuracy, dis- tribution and density, coupled with a well-designed camera network. Including oblique imagery may improve the accuracy of the results, and for change detection studies, these oblique images better ensure that terrain complexity is mapped.
A future study will investigate further the spatial distribution of errors. In this study, no doming, such as reported by James and Quinton (2014), Javernick et al. (2014), and Woodget et al. (2015), was evident in either the sparse or the dense GCP density scenarios.
This study helps to inform operational decisions in the survey design process and to provide insight into the impact of calibration choices, oblique imagery inclusion, ground control accuracy and ground control density on the accuracy of the resultant photogram- metric model.