Contributions

The following list summarizes the main contributions.

• We present a SLAM system that fuses information from both a monocular camera and an RGB-D camera.

• We propose to generate multiple virtual images from each wide-angle monocular image for improving feature matching and loop closer detection.

• We present an MST-based algorithm for connecting the frames and finding a good initial solution, which is later refined by BA.

6.1.2

Related Work

Monocular SLAM

A body of related work exists in the field of monocular SLAM, also known as structure from motion. Davison et al. [179] proposed one of the first extended Kalman filter (EKF) based monocular SLAM solutions. They constructed a map by extracting sparse features of the environment using a Shi and Tomasi operator [48] and matched new features to those already observed using a normalized sum-of-squared difference correlation. Since an EKF was used for state estimation, only a limited number of features were extracted and tracked in order to manage the high computational cost of the EKF.

PTAM is another well-known method proposed by Klein and Murray [180], in which they pioneered the idea of running camera tracking and mapping in parallel threads. Unlike Davison et al.’s filtering based method, PTAM was optimization based and utilized BA for the estimation of its parameters. Despite its success, PTAM had several limitations, such as the restriction to map small environments, the lack of a large loop closure detection system, and the low invariance to viewpoint change since it is based on the correlation between low resolution images of the keyframes. Both of the aforementioned methods are feature based, as they rely on extracting and tracking a sparse set of salient image features. Most recently, due to the increase in computational capability, direct methods such as LSD-SLAM [82] have been proposed. The direct methods exploit every pixel in the image to produce an estimate of the camera pose relative to a 3D map, but are still unstable in scenes with limited textures, common in indoor environments.

RGB-D SLAM

As we previously mentioned in Chapter 1, many researchers have utilized RGB-D sensors for solving challenging SLAM problems such as [30][65][66][69][181][70]. However, all of the aforementioned RGB-D methods are constrained by the RGB-D camera’s limitations such as having a narrow FOV and limited depth range, leading to failures when distant frames are registered. Most recently, Endres [178] outlined the problem of the restricted field of view of RGB-D cameras for SLAM applications. He proposed the use of multiple RGB-D cameras and demonstrated that this can result in substantial benefits for the reconstruction accuracy. In contrast, we aim to rectify this problem by aiding the RGB-D sensor with a wide-angle

6.1 Introduction 125

monocular camera, providing additional information that allows correct 3D registration in situations where the RGB-D camera fails.

Monocular-RGBD SLAM

Hu et al. [182] addressed the problem of not having sufficient depth information in large areas due to the limitations of RGBD cameras. Their method heuristically chose between an RGBD SLAM approach and an 8-point RANSAC based monocular SLAM depending on the availability of depth information in the scene, and merged the two maps generated by the two individual SLAM approaches. Zhang et al. [183] addressed the issue of using a heuristic switch and proposed a single method to handle sparse depth information by combining both features with and without depth. In their method, depth was associated to the features in two ways, from a depth map provided by the RGBD camera and by triangulation using the previously estimated motion for features lacking depth information. One of the shortcomings of their method is that it is a visual odometry method, which lacks a loop closure system and would not achieve global consistency in large scale environments.

Ataer-Cansizoglu et al. [184] used both features with and without depth in a SLAM frame- work as well as in postprocessing. As opposed to these methods using only an RGBD camera, we use a separate wide-angle monocular camera along with the RGBD camera for obtaining more constraints using RGBD-to-monocular registration.

RGBD-to-monocular registration was exploited in [185] for calibrating RGB cameras that might have non-overlapping FOVs using a map obtained with an RGBD SLAM system, but the map was assumed to be fixed for the RGBD-to-monocular registration. In contrast, we use RGBD-to-monocular registration to extend the mapped regions and to improve the registration accuracy.

The method presented in [186] fuses the information obtained by both a monocular camera and a laser range-finder for performing SLAM in dynamic environments. Their method incorporates both a monocular and a LASER EKF-SLAM, and by fusing the aforementioned approaches, the localization errors are reduced. In our approach, we mainly focus on the limitations associated with RGB-D sensors such as the Microsoft Kinect.

RGBD frames Monocular frames Perform RGBD-to-RGBD sequential matching Construct initial graph

Perform VLAD based RGBD-to-monocular &

RGBD-to-RGBD matching

Update the graph and find MSTs Calculate global

poses by traversing the

tree Prune edges with

inconsistent poses Run bundle adjustment on the graph Output 3D model Find MSTs

Figure 6.2: Overview of the proposed system.