Sensor Model Adjustment Through Tie Points

In order to achieve a high positioning accuracy during the geo-referencing process, each parameter of the utilized physical sensor model has to be carefully determined. Commonly, the satellite position and the interior orientation of the camera system can be determined with a high-precision. The relative alignment between the body and the sensor coordinate system on the other hand, causes in most cases pointing errors, mainly due to inaccurate measurements of the satellite attitude and thermally affected mounting angles. As a consequence, additional data in form of well measured GCPs is required in order adjusting the corresponding parameters (the boresight angles) of the physical sensor model. By reformulation Equation 4.15the following system can be derived

Jx(ε) = r11(xoe−xse) +r12(yoe−yse) +r13(zoe−zse) r21(xoe−xse) +r22(yoe−yse) +r23(zoe−zse) −tan Ψ_x Jy(ε) = r21(xoe−xse) +r22(yoe−yse) +r23(zoe−zse) r31(xoe−xse) +r32(yoe−yse) +r33(zoe−zse) −tan Ψy, (4.16)

whererijrepresents an elements of the matrixRsensor_Earth(t) =Rsensorbody (ε)−1·REarthbody(t)

−1_,rEarth object=

(xoe, yoe, zoe)T, rsensorEarth = (xse, yse, zse)T and r_objectsensor = (xos, yos,1)T = (tan Ψx,tan Ψy,1)T.

The three unknown boresight angles εare estimated by minimizing the cost functions Jx

andJy from Equation 4.16through an iterative least squares adjustment. In order to remove

outliers from the given set of GCPs, and hence estimate the unknown angles εas precise as possible, an iterative blunder detection is integrated into the least squares adjustment. Here, outliers are defined as GCPs with a residual greater than a certain threshold (usually 1 to 2 pixels), where the residuals are the 2D deviation at the GCPs in image space. A detailed description of the blunder detection step is provided [95].

After estimating the boresight angles and adjusting the sensor model parameters, the improved model and a corresponding DEM are utilized for the orthorectification of given optical images (level-1 products). Through this procedure new orthorectified optical images with an improved absolute geo-localization accuracy can be achieved. Note that in contrast to [95], where the GCPs are generated from optical images, we are using tie points generated by the methods described in the Subsections4.2 and4.3. The results of the described sensor model adjustment procedure applied on a set of optical test images and automatic generated tie points are evaluated and discussed in Subsection5.3.3and 5.2.4.

92 4. Deep Learning-based Optical and SAR Image Registration

4.5 Summary

In this section we presented a novel and automatic optical and SAR satellite image registration framework and the associated absolute geo-localization accuracy enhancement of optical images. The three main components of the framework are:

1. Selection of suitable matching areas in order to eliminate geometric differences between optical and SAR images through a semi-automatic process.

2. Generation of a reliable and accurate set of tie points through a deep learning-based matching of optical and SAR image patches cropped from pre-selected areas.

3. Adjustment of the physical sensor model parameters through the generated tie points in order to register optical and SAR images and therefore enhance the absolute geo- localization accuracy of the corresponding optical images.

In contrast to traditional approaches our developed framework provides the following theo- retical benefits:

• Through the pre-selection of suitable areas the existence of salient features can be guaranteed and on the other hand areas containing elevated objects and therefore exhibit different geometric properties in optical and SAR images can be eliminated. As a consequence, the risk for our matching approaches to produce total mismatches is reduced and the quality and reliability of the obtained tie points with regard to their geo-localization is increased.

• If the cGAN-based matching approach is utilized for the tie point generation, radio- metric differences between arbitrary optical and SAR image pairs can be reduced to a minimum through the generation of artificial SAR-like patches. As a consequence, the application of traditional matching approaches for the tie point generation becomes feasible. In addition, the image generation process is independent of handcrafted feature detection and extraction algorithms and not limited to particular features. This circumstance enables its applicably to a wide range of image scenes.

• If the Siamese neural network-based matching approach is utilized for the tie point generation, no handcrafted feature detection, extraction and matching algorithms are required for a single step and new tie points can be generated within seconds. Furthermore, the end-to-end training over a large dataset and the particular design of our network enable the application to a wide range of images acquired over different scenes, at different times of the year, with different resolutions and image sizes. In order to assess the proposed framework, we will perform an excessive evaluation of the tie point generation methods and their abilities for a geo-localization accuracy improvement for a set of optical test images in the Sections 5.2 and 5.3 of the following Chapter. Beforehand, the specifics of the utilized optical and SAR images will be presented and the training, validation and test dataset derived from our semi-automatic area selection process will be described in Section 5.1.1. In a final step, the two tie point generation concepts of our registration frameworks will be compared and their strength, weaknesses and potential for future developments will be discussed in Section 5.4.

5

Results and Discussion

In this chapter the proposed concept for the registration of optical and SAR images through tie points, automatically generated over pre-selected image regions, is tested and evaluated on several image pairs spread across Europe. The main focus of our investigation lies on the evaluation of the two novel tie point generation methods and their ability to generate reliable and accurate tie points. Therefore, the experimental setup with the image characteristics, pre-processing steps and the final datasets for the training, validation and testing of our deep learning based approaches is introduced and an overview of the utilized statistical measures and baseline methods is provided. Then, both tie point generation approaches are consecutively tested on the same test set and compared with state-of the art approaches with regard to their potential for an accurate and precise tie point generation and for an absolute geo-localization accuracy enhancement of optical images. At last, a detailed comparison of the advantages, disadvantages, strength and limitations of both methods is carried out.

Contents

5.1 Experimental Setup. . . 94 5.2 Optical and SAR Image Registration Through Artificial Image Matching 100 5.3 Optical and SAR Image Registration Through Siamese Neural Networks 118 5.4 Comparison of the Image Registration Frameworks . . . 130

94 5. Results and Discussion

5.1 Experimental Setup

This section forms the basis for the evaluation of both tie point generation methods (outlined in Chapter 4) and their abilities for the geo-localization accuracy enhancement of optical images. Therefore, the image specifications and pre-processing steps of the utilized optical and SAR image pairs are described in Subsection 5.1.1. Then, the final training, validation and test sets obtained from the semi-automatic matching area selection procedure described in Subsection 4.1.1 are presented in Subsection5.1.2. Finally, a description of the statistical measures and the baseline methods on which our evaluation are based on is provided in Subsection 5.1.3and 5.1.4, respectively.