Impiego di 3DF Zephyr nel rilievo da APR

(1)

Impiego di 3DF Zephyr

nel rilievo da APR

Andrea Fusiello (andrea.fusiello@uniud.it) -- DIEGM University of Udine

(2)

3DF Zephyr

Software per la ricostruzione 3D da fotografie

Sviluppato da 3Dflow srl spinoff UniUD (ex UniVR)

Gestisce blocchi non strutturati Gestisce camere eterogenee con calibrazione ignota

Adatto sia per la ricostruzione di piccoli oggetti che per rilievi fotogrammetrici

(3)

Image acquisition

3DF Zephyr pipeline

3d photo-consistency

from images 3d photo-consistency 3d surface from Image orientation 3DF Samantha (structure and motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe,

2006]

Images from Vogiatzis, 2010

(4)

Image acquisition

3DF Zephyr pipeline

from images 3d photo-consistency 3d surface from Image orientation 3DF Samantha (structure from motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe,

(5)

Structure from motion

(6)

2-‐d %e-‐points matching object 3-‐d points

Structure from motion

(7)

Structure from motion

Motion = exterior orientation

(8)

Orientation of a block

¤  Block: set of overlapping photographs

¤  Independent models block adjustment

¤  Create independent stereo-models with relative orientation for each pair of overlapping photographs

¤  Simultaneous transform the models into the ground coordinate system with absolute orientation (given known GCP).

¤  Bundle block adjustment

¤  Compute directly the relations between image coordinates and object coordinates, without introducing model coordinates as intermediate step.

(9)

Bundle block adjustment

Bundle adjustment & self calibration

Simultaneousestimation of:

1) the camera location and orientation 2) the 3D coordinates of the image points

Requirement: initial values

3) camera calibration parameters

Associated concepts:

Optimization methods

h i l/ h i l d l Mathematical/stochastical model Reduced Normal Equation (RNE) Datum definition

Total Error Propagation (TEP) Total Error Propagation (TEP) Limited Error Propagation (LEP) Blunder detection

(10)

Resection-intersection cycle

¤  1. Initialize:

¤  (a) Select two suitable photographs;

¤  (b) Solve relative orientation and form the stereo-model (intersection)

¤  2. For every additional photo (following a suitable order),

¤  (a) Solve exterior orientation (resection)

¤  (c) Refine the existing model (intersection);

¤  3. Transform the final model into the ground coordinate system with

absolute orientation (given known GCP).

¤  This is also known in CV as sequential structure from motion (à la Bundler)

(11)

Hierarchical variation

¤  The previous method can be generalized by organizing the photographs on a tree instead of a chain.

¤  The tree is produced by hierarchical clustering the photographs according to their overlap (the overlapping relationship is indeed

not a chain)

(12)

Hierarchical variation (Samantha)

¤  1. Form many independent stereomodels from photo pairs at the leaves of the tree

¤  2. Traverse the tree; in each node one of these operations takes place:

¤  Grow one model by adding one photo with resection followed by intersection;

¤  Merge two independent model with absolute orientation;

¤  3. Transform the final model into the ground coordinate system with

absolute orientation (given known GCP).

¤  Note:

¤  If the tree reduces to a chain, the algorithm is a sequential SfM;

¤  If the tree is perfectly balanced, only the Merge step is taken, and the resulting procedure resembles the IMBA.

(13)

Tree traversal

Relative Orientation Exterior Orientation Absolute Orientation

(14)

Comments

¤  CV methods are designed for irregular and unknown block

structure. Recovering block structure and tie-points matching is fundamental (see forward)

¤  CV methods do not make use of GCP but at the end (free solution) because they are compulsory.

¤  The gold standard is bundle adjustment, the other methods are seen as tools to get an approximate solution.

¤  The hierarchical method is more effective than the sequential one in drift containment.

(15)

Pre-processing summary

¤  Keypoint extraction

¤  Matching - broad phase: select O(n) views to be matched

¤  Matching – narrow phase: match keypoints between pair

¤  Clustering: determine processing order (can be on-line)

(16)

Keypoints

¤  Detector: scale-space extrema of the scale-normalized Laplacian [T. Lindeberg, 1994]. SIFT is an variation on this theme.

¤  We use a 8-level scale-space and in each level the Laplacian is computed by convolution (in CUDA) with a 3 × 3 kernel.

¤  Key: multiresolution pyramid based on derivative operator. ¤  Descriptor: 128- dimensional radial descriptor based on the

accumulated response of steerable derivative filters. ¤  Key: derivatives, directions histogram,

(17)

Which images are to be matched?

¤  Recover the image graph, i.e., the graph that tells which image overlaps (or can be matched) with which other.

¤  For each key-point descriptor its approximate k nearest neighbours in feature space are computed (via ANN)

¤  A 2D histogram is then built: increment bin(i,j) whenever a keypoint of image i has a keypoint of image j in its k-neighbourhood;

(18)

Which images are to be matched?

¤  Image graph G = (V,E) where V are views and the weighted adjacency matrix is the 2D histogram.

¤  This graph has | V | = O(n2_{). The objective is to extract a subgraph} G’ with a number of edges that is linear in n.

¤  In graph theory, a graph is k-edge-connected if it remains connected whenever fewer than k edges are removed.

¤  We devised a strategy that builds a subgraph G’ of G which is

(19)

Matching

¤  Match keypoints (images connected in the graph)

¤  Validate matching with relative orientation (RANSAC)

¤  Non linear refinement of relative orientation

(20)

Tracks

¤  From pairwise matches to tracks

(21)

Clustering

¤  Agglomerative, hierarchical clustering problem

¤  Simple (complete, average, ward’s) linkage

¤  Distance needed:

¤  Common matches

¤  Good coverage

(22)

Image acquisition

3DF Zephyr pipeline

(23)

Multi-view stereo

(24)

Depth-map fusion

1. Compute depth hypotheses

2. Volumetrically fuse

the depth-maps

(25)

Computing depth maps

¤

 

For each pixel in the current image

¤

 

March along the back-projected ray of the pixel inside

the bounding volume

¤

 

For each depth value d, project the 3D point in all the

neighbouring views and compute NCC

¤

 

Equivalently, we work in image space on rectified

images (easy to code on GPU)

¤

 

The mapping between epipolar lines is a 1-d

homography

(26)

(27)

Aggregating NCC scores

¤  Several NCC profiles must be aggregated to extract the depth of the given pixel.

¤  All local maxima could be good hypotheses.

¤  How do we pick right one?

Images from Vogiatzis, 2010

(28)

MRF estimate

¤

 

Voting: each local peak casts a vote (weighted by its

score value) on a discrete histogram along the optical

ray.

¤

 

Winner take-all: the most voted value is the depth of the

point

à

can do better

¤

 

MRF relaxation:

¤  the k bins of the histograms with the highest score are as the candidate depths for the pixel.

¤  minimization selects the best depth among several hypotesis taking neighbourhoods into account.

(29)

MRF result

Winner-take-all Depth map after MRF

(30)

Merge depth maps

¤

 

Depth maps are lifted in 3D space to produce a

photoconsistency volume, represented by an

octree that accumulates the scores coming from

each depth map.

(31)

Photo-consistency of a 3d point

A

B

A is photoconsistent

B is NOT photoconsistent

(32)

Visibility

¤

 

The photoconsistency volume at

this stage contains spurious

points, which do not belong to a

real surface.

¤

 

They are characterized by two

features:

¤  their photoconsistency is generally lower than actual surface points

¤  they usually occludes actual surface points.

(33)

Visibility

¤

 

This observation leads to an iterative strategy where

the photoconsistency of an occlusor is decreased by a

fraction of the photoconsistency of the occluded point.

¤

 

Points with negative photoconsistency are eventually

removed.

(34)

Image acquisition

3DF Zephyr pipeline

(35)

3D surface from points

¤

 

Compute an indicator function from oriented

points by solving a Poisson problem

[Kazhdan et. al]

(36)

Alcuni risultati: 3DF Zephyr con foto da APR

(37)

Nettuno

Fotografie nadirali (15) e oblique (28), con due fotocamere diverse (compatte).

Tutto il blocco di 43 fotografie viene orientato da Zephyr.

Punti di legame: 13K

(38)

Nettuno

(39)

Cerere

Fotografie nadirali (52), oblique (24) e terrestri interno/esterno (212), con tre fotocamere diverse (compatta su APR, reflex digitale a terra). Tutto il blocco di

288 fotografie viene orientato da Zephyr.

(40)

Cerere

(41)

Ventimiglia

Fotografie nadirali (253) e oblique (224), con stessa fotocamera (reflex digitale).

Tutto il blocco di 447 fotografie viene orientato da Zephyr.

(42)

Ventimiglia

Dopo multiple-view stereo si ottengono 2.715M punti (vertici mesh)

(43)

Ventimiglia

(44)

Ventimiglia

Ortofoto

(45)

Accuracy analysis

2 4 3 5 6 7 8 9 1011121314151617181921222324 0 0.02 0.04 0.06 [m]

Distance to ground control points

2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.02 0 0.02 0.04 x [m]

Difference with ground control points on each dimension

2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.04 −0.02 0 0.02 y [m] 2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.1 −0.05 0 0.05 0.1 z [m]

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

(46)