Culvenor (2002) states that achieving the successful delineation of trees is problematic. Outlining trees from homogenous groups, without explicitly quantified GR data can lead to repeated errors. The aim of this study was to develop a framework for objectively quantifying the agreement between two datasets, focussing on common commission errors in RS data, with increased data noise and data population differences. The ARBOR framework was developed and then applied to real-world data to quantify the commission agreement between four different ITC delineation techniques and GR datasets (Figure 20). This type of analysis is frequently absent from RS studies that utilise ITC delineation techniques, which instead, rely upon arbitrary height or other cut-off thresholds to infer the level of agreement (Næsset 2002, Listopad, Drake et al. 2011, Hyyppa, Yu et al. 2012). However, the findings from this research indicates that simple measures, thresholding and not accounting for the biophysical parameters of trees leads to low levels of true-positive match-pairing between GR and RS-derived data (Figure 18).
Throughout Figure 18a-f, there is a general tendency of higher match-pairing performance at lower noise levels, with a diminishing of NRMSE as noise levels increase. Concurrently, increasing data loss, from 0 to 50%, further impacts on the efficacy of the match-pairing. In all cases, noise affecting the data has the greatest effect, while data loss, less so. What is clear is that introducing data noise alters the
biophysical parameters that the trees are being matched on, and therefore, assessment of these parameters should always be included as variables when seeking ITC delineation agreement with GR data. Figure 18a-c shows that match-pairing methods are sensitive to shifts in the biophysical tree structure under analysis. The data losses, or differences in tree population numbers between the two datasets, has a different effect.
Where data in the observed dataset B (e.g. LiDAR) has fewer trees, poorer matches are achieved as the limited tree population will have greater tree numbers available for matching in the opposing dataset A (e.g. GR). Using some methods, such as Hausdorff distance, unmatched tree data is discarded from the analysis when all trees in dataset B are matched. Without measuring the dataset size, the match-pairing analysis declares a successful match even where there are fewer trees in one set than the other. This creates a false positive result, where changes in the data population and quantification of the unmatched pairings is not reported (Figure 18d-e). Furthermore, this analysis has shown that the frequently used match-pairing method, Hausdorff distance, significantly underperforms in reaching agreement between GR and RS datasets, particularly when exposed to increasing data noise and losses, as readily occurs in real-world RS data (Figure 18a & d). However, through the creation of the ARBOR framework, a demonstrably robust framework has been established to quantify agreement between GR and RS-derived data.
The approach used to develop the ARBOR framework was similar to Ole Ørka, Næsset et al. (2009) where a synthetic testing environment was used to replicate complex RS tree datasets, with naturally occurring variations in tree size, shape and location. During early iterations of metric testing, it was recognised that each tree in
the two datasets must achieve a bilateral matching agreement. However, this was problematic as it was observed that this lead to ‘hugging pairs’ within the data assignment. Specifically, where once assigned a matched pair, e.g. SYTree A1 to SYTree B1, the assignment excluded any other potential match even where a subsequent potential match was better suited. Further analysis showed that the order of the match-agreement process is a relevant factor in achieving high agreement match-pairing. To overcome this problem, the Hungarian combinatorial optimisation algorithm was used to search through all the potential combinations in the parallel dataset. An advantage of the Hungarian algorithm is the optimising nature of the routine where the algorithm cannot reach completion with an unsuitable data assignment. Therefore, the algorithm attempts all possible data combinations between the two datasets and completes only when the fullest level of agreement is reached.
The AMPS index quantifies the similarity between the datasets as a measure of the biophysical tree properties agreement, represented as Gaussian overlap (Figure 15), while the DSS index provides a measure of population size estimates from ITC delineations. Contrary to the views of Kaartinen, Hyyppä et al. (2012) who state that the comparison of delineation results between different datasets cannot be achieved due to the variability in crown structures of different species, this research demonstrates that by using GR representations of trees as simple objects (with location, height and area), and matching these objects to ITC delineations using a Gaussian curve model and the Hungarian algorithm, accuracy assessment becomes possible (Figure 20). Therefore, the ARBOR framework provides a new opportunity for quantifying the confidence of ITC delineation techniques in RS investigations. Figure 20 and Table 10 demonstrate that recommendations can be given about the efficacy and suitability of different ITC delineation techniques applied to remotely-
sensed data. We can define optimal ITC delineation methods, as shown by the AMPS and DSS values calculated within the ARBOR framework.
The principal emphasis of this work was to enable the quantification of pairwise match agreement between GR and RS-derived datasets. However, we also recognise there are opportunities for the ARBOR framework to quantify other types of data agreement, for example, tree delineations derived from aerial photography matched with those from aerial or terrestrial LiDAR. Due to the modular nature of the ARBOR framework, it can be adapted, as is required in future studies, to include a range of different match-pairing metrics not incorporated into this study and to generate alternative statistical measures of ITC delineation accuracy. Furthermore, in this study the ARBOR framework was used for quantifying the accuracy of ITC delineation in a complex semi-natural temperate broadleaved woodland. Given the demonstrable robustness of the tree matching technique and sensitivity of the accuracy metrics, the ARBOR framework holds potential as an objective and transferable tool that can be applied across the full range of forest types.
To enable the distribution and further application of the ARBOR framework, a portal has been developed to allow the uploading and analysis of match-pairing data, to provide objective quantification of the accuracy of ITC delineations.