Pairwise registration of TLS point clouds by deep multi-scale local features

(1)

ContentslistsavailableatScienceDirect

Neurocomputing

journalhomepage: www.elsevier.com/locate/neucom

Pairwise

registration

of

TLS

point

clouds

by

deep

multi-scale

local

features

R

Wei Li

a ,b

_{, Cheng Wang}

a ,b ,∗

_{, Chenglu Wen}

a ,b

_{, Zheng Zhang}

a

_{, Congren Lin}

a

_{, Jonathan Li}

a ,b ,c

a_Fujian_Key_Laboratory_of_Sensing_and_Computing_for_Smart_City,_School_of_Informatics,_Xiamen_University,_Xiamen,_China b_Digital_Fujian_Institute_of_Urban_Traﬃc_Big_Data_Research,_Xiamen_University,_Xiamen,_China

c_Department_of_Geography_and_{Environmental}_Management,_University_of_Waterloo,_Waterloo,_Canada

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received24September2018 Revised29October2019 Accepted18December2019 Availableonlinexxx

CommunicatedbyDr.ZhuJianke

2018MSC:

00-01 99-00

Keywords:

MSSNet

Pointcloudregistration Terrestriallaserscanning(TLS) Dataaugmentation

Geometricconstraints

a

b

s

t

r

a

c

t

BecauseofthemechanismofTLSsystem,noise,outliers,variousocclusions,varyingclouddensities,etc. inevitablyexistinthecollectionofTLSpointclouds.ToachieveautomaticTLSpointcloudregistration, manymethods,basedonthehand-craftedfeaturesofkeypoints,havebeenproposed.Despitesigniﬁcant progress,thecurrentmethods stillfacegreat challengesinaccomplishingTLSpoint cloudregistration. Inthispaper,weproposeamulti-scaleneuralnetworktolearnlocalshapedescriptorsforestablishing correspondencesbetweenpairwiseTLSpointclouds.Totrainourmodel,dataaugmentation,developed onpairwisesemi-synthetic3Dlocalpatches,istoextendournetworktoberobusttorotation transfor-mation.Then,basedonvaryinglocalneighborhoods,multi-scalesubnetworksareconstructedandfused tolearnrobustlocalfeatures. Experimentalresultsdemonstratethatourproposed methodsuccessfully registerstwoTLSpointcloudsandoutperformsstate-of-the-artmethods.Besides,ourlearneddescriptors areinvarianttotranslationandtoleranttochangesinrotation.

1. Introduction

With the rapiddevelopment oflaser scanning technology, 3D pointcloudsacquiredfromLiDAR,Kinect,RangeCameras,etc.,are becomingmoreandmoreprevalent.Asatechnologyforcollecting pointclouds,aTerrestrialLaserScanning(TLS)system[1] provides an array ofcapabilitiesintermsofinstrumentrange,scan speed, ﬁeldofview,sizeandportability.Atpresent,TLSiswidelyapplied tomanypracticalapplicationssuchaspreservingculturalheritage, engineering surveying, and manufacturing. TLS point clouds reg-istration is the foundationfor3D reconstruction, objectretrieval, objectposeestimation,andcameralocalization.

Classicmethods,suchastheIterativelyClosestPoint(ICP) algo-rithm[2] anditsvariants[3–6] ,whichrelyontheinitialposition, cannotsatisfytheregistrationofreal-world3Dpointclouds. Exper-imental results show the effectiveness of the 4-PointsCongruent Sets-based(4PCS)methods[7 ,8] .However,asaresultofoperating atapointlevel,thesemethodsaresub-optimalinthedirectionof

R _Fully_documented_templates_are_available_in_the_elsarticle_package_on_CTAN_. ∗ _{Corresponding}_author_at:_Fujian_Key_Laboratory_of_Sensing_and_Computing_for SmartCity,SchoolofInformatics,XiamenUniversity,Xiamen,China.

E-mailaddress: [email protected] (C.Wang).

slippage.Manyhand-crafted3Dlocaldescriptors,suchasthoseby Rusuetal.[9] ,Zhangetal.[10] ,andZouetal.[11] ,havebeen pro-posedforpointcloudregistration.Thesemethodsperformwellin the3D modelshavingmanifoldcompletedsurfaces,butare insuf-ﬁcientlyrobust to real-world 3D point clouds.Although carefully followingtheparadigmshifttodeepneuralnetworks[12] and[13] , recenttrendstrytonaivelyextendthenetworksfroma2Dtoa3D domain.Duetosparserenderingdata,lotsofspatialdetailsarelost whichleadstosub-optimal.Therefore,TLSpointcloudregistration stillpresentsa challengebecauseoflow sceneoverlapsevere oc-clusion and self-occlusion, and without prior positional informa-tion.Themaindiﬃcultiesaresummarizedasfollows:

Data size: TLS point clouds can be acquired at a speed of 300,000pts/ssothatlarge-scalepointcloudscanbecollected eas-ily.Thus,manytypicalmethodsareinvalidonsuchlargeamounts ofdata.

Noise and outliers: Noise is presented as a form of randomly ﬂuctuating data,outliers are considered as those pointsfar from thesurface.Bothnoiseandoutliersarecommonandunavoidable.

Various occlusion: Because of different scanning views, data is often missing and incomplete when the objects are the same. Thus, the descriptions from corresponding keypoints are often inconsistent.

(2)

featureswithincreasingcontextualscales.Inspiredbythemethod ofPointNet++[15] ,wedesignanovelmulti-scalenetworktolearn thedescriptionoflocalneighborhood.Specially,by varying neigh-borsizesaround apoint, a novelmulti-scale learningnetwork is developedforrobust estimationof localshape descriptors overa range of TLS point clouds. The key here is that local shape de-scriptorscanberobustlyestimatedbysuitablyaccountingfornoise margin,occlusion,andvaryingdensities.

The main contributions of this work are summarized as fol-lows: (1) Based on varying local neighborhoods, we propose a Multi-ScaleSiamese Network (MSSNet), which directlyconsumes unorderpointclouds,tolearnlocalshapedescriptors.(2)Tobuild atrainingset,wedevelopanoveldata-augmentedmethodby ran-domly removing points, adding noise, and rotating for 3D local patches.(3)Geometricconstraintsofmatchingareusedtoextract morecorrectcorrespondingpairs.

The remainder of this paperis as follows. Section 2 provides a briefreview on the representative works of registration meth-ods including ICP-like, RANSAC-like, and features of keypoints-based.Section 3 introducesthepipelineofourregistrationmethod, including learning local features by MSSNet and rejecting false correspondences with geometric constraints. Section 4 presents experimetal results and analyses. Section 5 contains some conclusions.

2. Relatedwork

Usingtransformingparameterswithsixdegreesoffreedom,3D pointcloudregistrationisusually consideredasrigidregistration. Manyrelatedmethodshavebeenproposedfortherelated applica-tions.InthisSection,we brieﬂy reviewthe representativerelated workinrigidregistrationofpointcloud.

ICP [2] , one of most popular methods, alternates between estimatingthepoint correspondence andthe transformation ma-trix. Besides, many variations of this method [16–19] based on differentapplications havebeenproposed.However, suchICP-like methodsﬁrstrelyontheassumptionthatallpointshavepairwise counterparts between two sets. Furthermore, the methods are very sensitive to a giveninitialization. Yang et al.[20] proposed a globally optimal solution to ICP in 3D Euclidean registration, whichcombinesICPwithaBranch-and-Bound(BnB)scheme. Sim-ilarly,CampbellandPetersson[21] replacedtheobjectivefunction with a convolution of Gaussian Mixture Models (GMMs). How-ever,thesemodels cannot guarantee the globaloptimality ofthe solutions. To describe point cloud and surface normal densities, Straubetal.[22] proposedamethodofBayesiannon-parametrics, which includes BnB optimization to estimate relative transfor-mation. Because BnB is exponentially complex, these methods arecomputationally expensive, andare sensitive tomissing data. Fanetal.[23] proposed aConvex Hullindexed GaussianMixture Model (CH-GMM) by incorporating proximity, area conservation andprojectionconsistency.TheCH-GMMalgorithmworkswellon

algorithmforregistration.However, dueto theirpoint-level oper-ation, these RANSAC-likemethods are easy to sub-optimal when computingtheir transformationrelations. Theproblemof varying density in the TLS point clouds makes the performance of the 4PCS-basedmethodevenworse.

Many 3D local descriptors, based on hand-crafted methods, havebeenproposed. JohnsonandHebert[25] constructed alocal shapedescriptor,whosepointsareorientedwithassociated direc-tions, and used it to match surfaces. Tombar et al. [26] pointed out the key issues of uniqueness and repeatability of the local reference frame, and encompassed a new unique, repeatable lo-cal referenceframe as well as a new 3D descriptor. Tooptimize initial alignment, Rusu etal. [27] proposed a Sample-Consensus-based method that combines the local features calledFast Point Feature Histograms (FPFH), and Signatures of Histograms. How-ever,their methodstillcannotguaranteeglobalconvergence.Yang and Zang [28] extracted crest lines as matching primitives and then proposed a deformation energy model to ﬁnd correspon-dences. In their experiments, they obtained accuracy registration results.Toreducetherequiredprocessingdramatically, Kechagias-StamatisandAouf[29] proposea3Dcovariancedescriptor,which overrides thenecessityofa localreferenceframe oraxis(LRF/A). Yangetal.[30] proposedaregistrationmethodbasedonsemantic feature points extractedfrom large-scale urban scenes. Zai etal.

[31] proposed an adaptive covariance (ACOV) descriptor that, in someTLSpointclouds,isinvarianttorigidtransformationand ro-bust tonoise and varying resolutions.However, therobust ACOV descriptors relies on RGB colorand intensity ofthe points. Yang etal.[32] proposed a localreferenceframe (LRF)together witha tripleorthogonallocaldepthimages(TOLDI)representation. Navar-rete et al. [33] proposed a 3D compression and decompression method using GMMs for registration of 3D point clouds. Yang etal.[34] revealedthatdifferentspatialinformationencoding ap-proaches would bring signiﬁcant effecton a local shape descrip-tor.However,Allofthemanually designeddescriptorsareusually basedon thepropertyof thedataitself andworkwell on small-scalemodels.Thus, itisdiﬃcultto adapttolarge-scaleTLS point clouds.

Recently, withthe adventof deep learning, multi-scale infor-mation has been exploitedboth in 2D and3D keypoint descrip-tions.Tooptimallycombinefeaturemapsfromdifferentscalesfor visualcorrespondence,Wangetal.[35] proposedascale-attention network.However,thismethodislimitedto2D keypoint descrip-tion. Fathyet al.[36] proposed a CNN-based hierarchical match-ingframework for2D and3D matching.However, tocompare ex-tractedfeaturevectorsandestablishcorrespondences,thetraining processofthismethodrequiresstrongsupervisionintheformof per-pixel ground-truthlabels.Besides,thetasksof2D and3D in-terest pointmatching andreﬁnement arevery differentfromour 3D point cloud registration. Learning local geometric representa-tionon3Dpointcloudsisbecomingmoreandmorepopular.Some works tried to naively extend the networks, usually in form of a binary-occupancy grid, from 2D to 3D domains. This idea was Pleasecitethisarticleas:W.Li,C.WangandC.Wenetal.,PairwiseregistrationofTLSpointcloudsbydeepmulti-scalelocalfeatures,

(3)

Fig.1. Algorithmframework,consistingoftesting(registration)andtraining.Intheregistrationsection,pairwisepointcloudsPandQareusedastheregistrationinput. Then,theHarrisalgorithmisusedtodetectkeypoints,andthetrainedMSSNetisusedtodescribethecorrespondinglocalpatches.Finally,matchesareextractedbased onthesimilaritymeasureoflocalfeaturesandgeometricconstraints,andrigidtransformationparametersareestimated.Inthetrainingsection,ﬁrst,atrainingdataset, includingmatchingandnon-matchingpatches,isconstructed.Then,MSSNetisusedtotrainthedataset.Finally,forTLSpointcloudregistration,thetrainedMSSNet,isused totransformthe3Dlocalpatchesintorobustlocalfeatures.

quicklyextended tomore informativeencoding such asTDF[37] , TSDF[38] (TruncatedSigned DistanceField).Becausemainlyused inthecontext of3Dretrieval,entire3D objectswererepresented withsmallvoxelgrids303_limited_by_the_maximum_size_of_the_3D

convolution kernels.Zeng et al.[38] presented3DMatch, a data-drivenmodelthatlearnsalocalvolumetricpatchdescriptorfor es-tablishingcorrespondencesbetweenpartial3DRGB-Ddata.Huang etal. [13] proposed a learningmethodof localshape descriptors using multi-view convolutional networks. Bothof these methods ignoretherawnatureoftheinput:sparsityandunstructured-ness. The methods usedense local grids and3DCNNs tolearn the de-scriptor. Thus, they fall short in performance of training/testing performanceandrecall.Elbazetal.[39] proposedanauto-encoder deeplearningnetworktolearnlocalfeaturesforregistration. How-ever,such a methodcannot adoptthisapproachforsimilar sized pointcloudswithpartialoverlap.

Comparedwithcurrentmethodsofdeeplearning,inthispaper, weproposeanovelmulti-scalenetworkthatdirectlyconsumes3D rawpointset,tolearnrobustlocalshapedescriptors.

3. Theproposedmethod

InspiredbythemethodsofPointNet[14] andPointnet₊₊[15] , which directly consumeunordered 3D point sets, we designeda MSSNet networkto learn robustlocal shapefeatures. Then, com-bining corresponding learned local shape features, the keypoints wereusedtoconstructcorrespondencesbetweentwopartialpoint clouds. Here,the pairwise matching of localdescriptors is trans-formed into classifying as well as corresponding interest points (match) andnon-correspondinginterest points(no-match) by Eu-clidean distance. As shown in Fig. 1 , the framework of our pro-posedalgorithmcanbesummarizedmainlyasthefollowingthree sections:constructingtrainingdatasets,traininglocalfeatures and pairwise registering of point clouds. First, training datasets were

builtfroma semantic3d dataset,which provides alarge quantity oftrainingdatasets.Second,trainingdatasetswereconstructedand fedintotheMSSNetnetwork.Third,superiorlocalfeatures, gener-atedbythetrainedMSSNetnetwork,wereusedforTLSpointcloud registration.

3.1. Constructingtrainingdatasets

Becauseeachpointcloudislarge-scaleandcontainsmillionsof pointsinthe semantic3ddataset,1 _it_can_provide_a _large_quantity

of3D local patches to train ofour MSSNetnetwork. Constructed like3DMatch[38] ,pairwise3Dlocalpatchesfrommany-view ob-servations ofa scenecan providemore reliable trainingdatasets. However,becausethereisonlyasmallnumberofdifferent obser-vationsofthe samesceneina semantic3ddataset,it cannot pro-videsenoughtrainingsamples.Therefore,wedevelopedamethod ofdataaugmentationtoconstructtrainingsamples.Especially,we ﬁrst extracted keypoints from each point cloud by the 3D Har-ris method [40] . Second, one of the n0 nearest neighbor points

(n0=10inourexperiments)ofeachkeypointisrandomlyselected

asthe matchingkeypoint. Any randompoint from anotherpoint cloudisselectedasanon-matching keypoint.Third,basedonthe above keypoints (match and non-match), we randomly sampled 1024pointswithinagivenneighborhoodwitharadiusof0.5m.

Toacquire robust 3Dlocaldescriptors,we constructeda semi-synthetictrainingdataset thatincludes processingasfollows: (1) addingGaussianrandomnoisebyrandomlyvibrating5,10and30 percentof thepoints;(2) randomlyrotating thepairwise match-inglocalpatches(0◦–360◦)overtheZ₋axis;(3)simulating occlu-sionsbyremoving15,25,45percentofthelocalpatches.SeeFig 2 .

1_{http://www.semantic3d.net/}_.

(4)

Fig.2. simulatingnoise,densitychange,andocclusions.(a)View1ofa3Dlocalpatchp1.(b)view2ofthesamelocalpatch.(c)Anew3Dlocalpatchwhichisadded

Gaussiannoise(σ= 0.06).(d)The3Dlocalpatchsimulatingocclusionsandnoiseisusedasmatchingpatchofp1.

Fig.3. BasedontheSiamesearchitecture,wepresentaMSSNet,consistingofthreesubnetworksineachbranch,tofusethefeaturesofdifferentscales.Specially,inthe DESCsection,basedonvaryingneighborhoodradii,weﬁrstconstructthreesubnetworkstogeneratemulti-scalefeatures.Second,allthefeatureswereconcatenatedineach branchoftheSiamesearchitecture.

Thesemethods providealargeanddiverse(match ornon-match) numberlocalpatchesfortraining.

Insteadofthe3Dvolumetricrepresentationofmulti-view pro-jectedorvoxel grid,each3D localpatchis theoriginal represen-tation(unorderedpointclouds).Givena3Dpointcloud P,we ex-tracted3D Harriskeypoints

{

c_i

}

K

k=1 andobtainedthe

correspond-ing 3D local patches, respectively. In the TLS coordinate system, each3Dlocalpatchpn

i=1,whichistranslatedaccordingtothe

cor-responding keypoint, uses ci asthe origin, andis normalized by

thedistanceofthefurthestpointfromtheorigin. Thetranslating formulateisasfollows.

p _i= p i−c k (1)

wherepirepresentscoordinatesofapointinthelocalcoordinate

system.

3.2.Networkarchitecture

Based onthe PointNet-likeencoder, MSSNetdevelopsa multi-scalefusingapproachtolearnlocalfeatures.Itisdesignedto oper-ateonthreesubnetworkswithvaryingneighborssize,as summa-rizedinFig. 3 .

TheglobalfeaturesacquiredbythePointNet-likeapproach sig-nify that thelarger theneighbor size, themore stabletheglobal information. Given different neighbors of a keypoint, however, there exists great shape differences. Therefore, ifa PointNet-like methodisused,thelargerneighborsizeleadstoconsiderablyless localinformation.Itismeaningfultofusethefeaturesatdifferent neighbor size to generate more signiﬁcant andstable feature. As seenintherighthandsegmentofFig. 3 ,localpatchpairs (includ-ing matchandnon-match) are fed into ourMSSNet ata ratioof 1:1.

To fuse the geometric characteristics of varying neighbors,as shownintheleftofFig. 3 ,foreachbranch,threelocalpatchesare sampledbyvaryingneighborhoodsizesr1,r2 andr3.More

specif-ically, a three-layer, point-wise Multi-Layer Perception (MLP) fol-lows the input layer and subsequently a max-pooling is used to generate a local feature using skip-links. This results in a more powerfulrepresentation.Toenhance the signiﬁcanceoflocal fea-tures, we concatenate the above three local features to a new global feature (384 dimension). Then, the global feature is used to concatenate the following two-layer MLP in each subnetwork, respectively. In the end, the local features are generated by Max poolingandconcatenatedtoacodeword(896dimension).

(5)

In ourexperiments, aSiamese network basedon PointNet (S-PointNet)wasconstructedfrom1024kernelswithaneighborhood size(r1)of1.0m.A3-PointNet-likeSiamesenetwork,which neigh-borhoodsizes(r1,r2,andr3)of0.3m,0.6mand1.0m,was con-structedfrom128,256,and512kernels,respectively.Additionally, we implement ourMSSNet intensorﬂow anduse ADAMto train ournetwork for10,000iterationsusinga baselearningrate10−3

andaninitialmomentum0.9.Wedecaythelearningrateat10,000 stepsanddecayingrateissetto0.7.

3.3. Lossfunction

Weformulatedmatchingissuesasabinaryclassiﬁcation includ-ing matchandnon-match.Ourobjectiveistoreduce thecost be-tweentwo matched3Dpatches,and,conversely,increasethe dis-similarity betweentwonon-matched 3D patches. Tolearn robust local features, we constructed a multi-scale network based on a Siamese fashion.Tomeasure thesimilarityof3D patchpairs,the

L2 norm asa metric,modeled bya contrastivelossfunction, was

used. To avoidthe phenomenon of over-ﬁtting, we added the L2

regularizationtermtotheobjectivefunction.Thelossfunction for-mulaEq. (2) isconstructedasfollows:

J

2 (2)

where i represents the i-th pairwise local patches, n represents the number of training pairs of local patches; y(i) _is _a _matching

label that representsthe matching ofthe pairwise points, where

y(i)₌₁_if_the_pair-wise_points_match_and_y(i)₌₀_if_the_pair-wise

pointsnon-match.diisEuclideandistance,

ω

representsthe

amongF

(

Q

)

neighbor of F

κ

II.

Third, forthe tuplestest, three correspondencepairs

(

p1,q1

)

,

(

p₂_,q₂

)

_,

(

p₃_,q₃

)

areselectedrandomlyfrom

κ

III,andcheckifthe

tuples

(

p1,p2,p3

)

and

(

q1,q2,q3

)

are compatible. In detail, we

=0.9,pi,pj∈Prepresentsthekeypointsfrompointcloud

P andqi,qj∈Qrepresentsthekeypointsfrompoint cloudQ.

∗

Fig.4. Evaluationofourmethodcomparedwithsomerepresentativemethodson 3DMatch’sbenchmark,whichcontains10,000pairsofsampleswithmatchornot.

represents the L2 norm. This test veriﬁes if the correspondences

arecompatible.

4. Experimentanalysis

In this section, ﬁrst, we evaluate the quality of our learned local shape features, which lies in the analysis of varying noise, varyingneighborhoodsize,andvaryingrotation.Second,compared with the state-of-the-art methods, our learned 3D local features achieved superior registration results for 3D point clouds.Third, weshow severalpairwise registrationresultsofpointclouds.Our trainingmethods of localpatches were used withtensorﬂow 1.7 and cuda 9.0. Registration experiments were conducted in C++, andon a PC withubuntu 16.04,Intel Core(TM) i5-4460 3.2GHz CPUand16.0GBRAM.

4.1. Dataset

Tovalidateourmethod,ourevaluationsforkeypointsmatching areagainstthediverse3DMatchRGBDbenchmark[38] inwith62 differentreal-worldscenesretrievedfromthepoolofdatasets.This collectionis split into 2 subsets with54 scenes for trainingand validationand8scenes fortesting. The datasettypically includes indoor scenes likeliving rooms,oﬃces, bedrooms, tabletops,and restrooms.

ForTLSpointcloudregistration,ourmethodwastestedmainly in TLS point clouds. As a huge dataset of 3D local patches, the trainingdataset, labeled matchand non-match from Semantic3d

[42] ,wasbuilttotrainourdescriptorseffectively.Thedataset, con-tainingﬁfteen outdoor TLS point clouds from the trainingset of semantic-8,andfourfragmentsfromthetestingsetofsemantic-8, wereusedtoconstructlocaltestingpatches.Specially,wesampled keypointsfromthe TLSpoint clouds,andobtained200,000 pairs of3Dlocalpointclouds(i.e.localpatches)atagivenneighbor ra-dius(0.5m).Fortraining,allthelocalpatchpairsincludingmatch ornon-match, were labeled ata 1:1 ratiofor training, Similarly, 10,000pairsoflocalpatcheswerelabeledfortesting.

ForregistrationtestofTLSpointclouds,eightfragmentsofTLS pointcloudswereselectedfromtheSemantic3d,wheretwoscenes includingfourfragments(SeeFig. 10 )wereselectedforevaluating ofregistration.Theinformationoftheevaluatedfragmentsis sum-marized in Table 2 , the average resolution r¯ (the average of the distances of all two adjacent points in the point cloud) of each Pleasecitethisarticleas:W.Li,C.WangandC.Wenetal.,PairwiseregistrationofTLSpointcloudsbydeepmulti-scalelocalfeatures,

(6)

Fig.5. (a)PRCurvesachievedbyvaryingneighborsize.(b)PrecisionCurveachievedbyRecallratio= 0.5of(a).

fragment.Additionally,another twofragments(See Fig. 11 (g) and (h))wereselectedfortheregistrationtest.

4.2.Keypointmatching

We used Precision–Recall (PR) curves to evaluate the distinc-tiveness of these descriptors. The Precision and Recall are com-putedasfollows:

Precision =T P/

(

TP+F P

)

Recall = T P/

(

T P+F N

)

(4)

whereTPisthenumberoftruepositivematches,FPisthenumber offalse positivematches, andFN isthe numberoffalse negative matches.ThePRCurveisgeneratedbytuningtheparameterratio. To demonstratethe superiority ofour learneddescriptors, we focus on evaluating the different parameters of noise, varying neighborhoodradii,andvaryingrotationofpairwise3Dlocalpoint clouds.Comparedwithseveralrepresentativedescriptorsincluding RoPS[43] ,SHOT[26] ,FPFH[9] ,ACOV[31] ,and3Dmatch[38] ,we testedtheabilityofourdescriptorstodistinguishbetweenmatch andnon-match. Implementation of ROPs wasachieved based on the open-source MATLAB code.2 _SHOT _and _FPFH _were

imple-mentedbased ontheopen-source PointCloudLibrary (PCL)[44] . Thedefault parameters inPCL were selected inour implementa-tion. We ﬁrst tested our descriptors on the benchmark of Zeng etal.[38] that contains10,000pairs of localRGB-D patches and theirlabelsofgroundtruth correspondence(binary“1” formatch and“0” fornon-match).AsseeninFig. 4 ,thetrainingandtesting datasetsarefromreference[38] .Ourlearneddescriptorsachieved thebestperformance.

To test the inﬂuence of neighborhood size, we evaluated the quality of the descriptors at different neighborhood sizes. First, keypointswererandomlyextractedfromtrainingandtestingpoint clouds.Then,thecorrespondinglocalpatcheswithdifferent neigh-borhoodsizesweresampledbyKNN. Wedesignedﬁvegroup ex-perimentswithdifferentneighborhoodsizes(0.4m,0.5m,0.6m, 0.7m,0.8m)tovalidatetheperformanceofkeypointmatching.As showninFig. 5 ,the larger theneighborhood size, the better the performance. However, a larger neighborhood size requiresmore computational time. Therefore, in the following experiments,the neighborhoodradiusforinitialcalculationissetto0.5m.

2_{http://yulanguo.me/img/RoPS.rar}_.

Totesttherobustperformance,wedesignedsixdifferentlevels ofGaussiannoise(

σ

=0.00,0.05,0.10,0.15,0.20,0.40)foreach3D localpatch. Fig. 6 showsthevariationinalocal pointcloudwith varyinglevelsofGaussiannoise.AsshowninFig. 7 (a–f),the train-ingandtestingdatasetsareconstructedlikethemethodsofFig. 6 . It isobservedthat ourlearneddescriptors achievedthebest per-formance.Especially,ourlearnedlocaldescriptor hassigniﬁcantly betterperformancethantheotherdescriptors.

Totestthetoleranceofvaryingrotations,weconductedtwelve pairsofexperimentsonvaryingrotations.Allthe3Dlocalpatches are ﬁrst decentralizedby the corresponding keypoints,thus, it is invariant to the shifting transformation of the 3D local patches. Speciﬁcally,wealsobuilt10,000pairsof3Dlocalpatches(P1 and P2). P2 wascopiedandrotatedatevery

π

/12 radianto build

an-other 3D local patches P2i. Similarly, aseries of3D local patches

{

P2i

}

12_i₌₁,were generated andtestedby ourlearned detectors.We

useprecision4toevaluatethe,performance.AsFig. 8 ashows,the accuracyofourmethodoscillatedbetween0.8and0.9.Therefore, theexperimental results indicatethat dataaugmentation extends ournetworkto berobust torotationtransformation.Additionally, as seen from Fig. 8 b, the lowest precision is when the 3D local patchesarerotated

π

.

4.3. Registration

To evaluate the registration performance, we ﬁrst tested our proposed method on the public Geometric Registration Bench-mark3_._As_seen_in_{Table 1}_,_our_proposed_method_achieved_the

high-est Precisionandthe second highestRecall onthe public bench-markof 3DMatch.Therefore, itshows that ourproposed method hasacertainsuperiorityinRGB-Ddataregistration.ForTLSpoint clouds,wedesignedseveralexperimentsinvolvingpointcloud reg-istration.Especially,asseeninFig. 9 ,twoscenes,labeled“Scene1” and“Scene2”,wereselectedforpairwiseTLSpointcloud registra-tion. The “Scene1” contains two TLS point clouds denoted by P1

andQ1,andthe“Scene2” alsocontainstwopoint cloudsdenoted

byP2andQ2.Theinformationpertainingtothetestedpointclouds

is given in Table 2 . Especially, the point clouds P1 and Q1 have

36.3%overlap,thepointcloudsP2 andQ2 have35.3%overlap.

Be-sides,tomakethealgorithmmoreeﬃcient,wedetectedkeypoints by3DHarrisdetectors[40] .

3_{http://3dmatch.cs.princeton.edu/geometric-registration-benchmark}_.

(7)

Fig.6. Thevariationsofa3Drawlocalpatch.(a)A3Drawlocalpatch.(b–f)DifferentlevelsofGaussiannoise(σ= 0.05,0.10,0.15,0.20,0.40).

Fig.7. PRCurvesachievedbysevensetsofcomparativetestsonourbenchmark,whichcontains10,000pairsofsampleswithmatchornot.(b–f)showthecorresponding PRCurvesofdifferentlevelsofGaussiannoise.

(8)

Fig.8. (a)Rotationtolerancetest.PRCurvesachievedbyvaryingrotation.(b)EvaluationofrotatingvariationachievedbyRecallratio= 0.5ofFig.(a).

Fig.9. Twoscenesincludingtwopairwisepointcloudswereselectedforregistrationevaluation.(a,b)TheﬁrstpairwisepointcloudsdenotedbyP1andQ1areacquired

from“Scene1”.(c,d)ThesecondpairwisepointcloudsbyP2andQ2areacquiredfrom“Scene2”.

Table1

The performanceofourproposedmethodis listedontheBenchmarkLeaderboard.

Method Recall(%) Precision(%) MSSNet 52.9% 58.9% 3DMatch 66.8% 40.1% Spin-Images 51.8% 45.8%

FPFH 44.2% 30.7%

Table2

Informationofthetestedpointclouds.

Model Numberofpoints Resolutionr(m) Overlap(%)

P1 824,509 0.0411

Q1 13,986,927 0.0351 36.3

P2 2,018,953 0.0201

Q2 2,271,520 0.0211 35.1

Table 3 showsthetimecostofeachstepinourproposed regis-trationframework.Thekeypointsextractionsteprequiresthe ma-jority of time. As seen in Table 4 , the comparative performance

ofsomerepresentativemethodsislisted intermsofaverageRMS distance andcomputational time. The proposed MSSNet was de-signed with a 3-scale subnetwork. As seen in Fig. 10 , compared withfeatures-basedmethodsincludingFPFH,ACOV,3DMatchand S-PointNet,(a-f) presentthe registrationresultsof“Scene1”,(g–l) presenttheregistrationresultsof“Scene2”.RANSACbasedon ge-ometricregistrationwasusedtoaligntheTLSpointclouds.In ad-dition,ourmethod,combinedwithgeometric constraints,isused to reject false correspondences. in terms of both registration er-ror and computational time, our method outperforms the other methods.However,thesemethods,proposedbyFPFH[9] andACOV

[31] (noRGB colorandintensity),failtoregisterpairwiseofpoint clouds. Especially, Fig. 10 (f) and (l) present the registration re-sultsofourmethodcombinedwithICPalgorithm,whichachieved the ﬁne registration.Therefore, the experimental results demon-stratethatourlearnedlocaldescriptorsaremorerobustthanother methods.

To further demonstrate the feasibility and effectiveness, two pairs of TLS point clouds fromSemantic3D and one pair of TLS pointcloudsacquiredbyRIEGL VZ-1000system, wereselectedas testingdata.Theinformationofpointcloudsforthefurthertestis

Table3

Timecostofeachstepinthetestedpointclouds.

Step Keypointsextraction 3Dlocalpatchesgeneration Descriptorscalculation Correspondencegeneration Totaltime(s)

Scene1 154.0 12.0 26.5 5.2 197.7

Scene2 33.0 6.0 10.3 5.0 54.3

(9)

Fig.10. RegistrationresultsonthepointcloudsofmodelSScene1andScene2.(a),(g)AchievedbyFPFH.(b),(h)AchievedbySuper4PCS.(c),(i)AchievedbyACOV.(d),(j) Achievedby3DMatch.(e),(k)AchievedbyMSSNet.(l)AchievedbyMSSNet+ ICP.

Table4

Comparedwithsometypicalmethods.

FPFH Super-4PCS ACOV 3Dmatch Ours Ours+ ICP AverageRMS1(cm) – 7.81 – 5.88 5.18 2.90 Overlap1(%) – 28.3 – 32.5 35.0 36.0 Totaltime(s) – 1500.3 – 327.8 197.7 218.8 AverageRMS2(cm) – 6.33 – 5.39 4.51 2.52 Overlap2(%) – 33.4 – 31.3 33.9 35.0 Totaltime(s) – 1008.1 – 356.8 54.3 77.8

showninTable 5 .Theﬁrstpairofpointclouds,P3andQ3,covering

a rangeof approximately30m by 25m, are showninFig. 11 (a) and(b).Thetrueoverlaprate,computedbymeansofmanual reg-istrationandtheICPalgorithm,is40.5%.Fig. 11 (c)showsthe suc-cessfulregistrationresultwithanoverlaprateof40.6%.Thesecond pairofpointclouds,P4 andQ4,coveringarangeofapproximately

40m by 15m, areshown inFig. 11 (d)and(e). The trueoverlap rateis20.9%.Fig. 11 (f)showsthesuccessfulregistrationresultwith anoverlaprateof20.6%.Thethirdpairofpointclouds,P5 andQ5,

covering a range of approximately80 m by 20 m, are shown in

Fig. 11 (g) and(h).The trueoverlaprateis44.5%. Fig. 11 (i)shows the successful registration result with the overlap rate of 43.7%.

(10)

Fig.11. Registrationresultsonthepointcloudsofmodels.(a)and(b)aretheﬁrstpairwisepointclouds,(c)registrationresultoftheﬁrstpairwisepointclouds.(d)and(e) arethesecondpairwisepointclouds,(f)registrationresultofthesecondpairwisepointclouds.(g)and(h)arethethirdpairwisepointclouds,(i)registrationresultofthe thirdpairwisepointclouds.

Table5

Informationofpointcloudsforfurtherregistrationtesting. Model Numberofpoints Resolutionr(m) Overlap(%)

P3 618,521 0.0335 Q3 83,925 0.0551 40.7 P4 13,789,069 0.0212 Q4 7,514,469 0.0121 21.0 P5 8,787,836 0.0143 Q5 8,318,075 0.0166 45.1

Therefore,based on the localdescriptors learnedby MSSNet, we obtainsuccessfulregistrationresults.

5. Conclusion

WeproposedanovelmethodtoregisterTLSpointclouds.Based on inspired PointNet architecture, our method directly consume pointclouds.Differentfromexisting methods,3D localshape de-scriptors,whichlearnbyfusingmulti-scalesubnetworks,aremore robustandreliable.Especially,comparedwiththestate-of-the-art methods, first, the experiments involving public data sets show thatthelearnedlocalshapedescriptorsaremoreaccurateand ro-bust to large noise; Second, the learned local shape descriptors are invariant to translation, and are tolerant to changes in rota-tion;Third,ourmethodperformsaccurateandefficientregistration evenonvery challengingscenes withoutanyestimationofinitial position.Overall, ourproposedmethodoutperformsexisting local shapedescriptorsandregistrationmethodsbyasignificantmargin.

Inthispaper,toextendthesizeoftrainingdata,data augmen-tationsuch asdatarotationis used.The aimofrotation isto in-creaserotationinvarianceinthefeatureextraction.However,data augmentation, especially the rotation, decreases the capability to describe features. Therefore, our method would be improved by afuturestudyofhow toperformdataaugmentationwithout de-creasingthecapabilitytodescribefeatures.

DeclarationofCompetingInterest

Theauthorsdeclarethattheyhavenoknowncompeting ﬁnan-cialinterestsorpersonalrelationshipsthatcouldhaveappearedto inﬂuencetheworkreportedinthispaper.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grants U1605254 ,

41471379 and theFujian Provincial Department of Education Sci- ence and Technology project JA14292 .The authors would like to acknowledge the anonymous reviewers for their valuable com-ments.

References

[1]M. Lemmens, Terrestrial Laser Scanning, Springer Netherlands, Dordrecht, pp.101–121.

[2] P.J.Besl,Amethodforregistration3-Dshapes,IEEETrans.PatternAnal.Mach. Intell.14(2)(1992)193–200.

[3] Y.Liu,ImprovingICPwitheasyimplementationforfree-formsurface match-ing,PatternRecognit.37(2)(2004)211–226.

(11)

[4] K.-H.Bae,D.D. Lichti, A methodfor automatedregistration ofunorganised pointclouds,ISPRSJ.Photogramm.RemoteSens.63(1)(2008)36–54. [5] J.Yang,H.Li,Y.Jia,Go-ICP:solving3dregistrationeﬃcientlyandglobally

op-timally,in:ProceedingsofIEEEInternationalConferenceonComputerVision, 2013,pp.1457–1464.

[6] Y.Liu,H.Liu,R.R.Martin,L.D.Dominicis,R.Song,Y.Zhao,Accurately estimat-ingrigidtransformationsinregistrationusingaboosting-inspiredmechanism, PatternRecognit.60(2016)849–862.

[7] D.Aiger,N.J.Mitra,D.Cohenor,4-Pointscongruentsetsforrobustpairwise surfaceregistration,ACMTrans.Graph.27(3)(2008)1–10.

[8] N.Mellado,D.Aiger,N.J.Mitra,Super4pcsfastglobalpointcloudregistration viasmartindexing,in:ProceedingsofComputerGraphicsForum,33,Wiley OnlineLibrary,2014,pp.205–215.

[9] R.B.Rusu,N.Blodow,M.Beetz,Fastpointfeaturehistograms(FPFH)for3D registration,in:ProceedingsofIEEEInternationalConferenceonRoboticsand Automation,2009,pp.3212–3217.

[10] Z.Zhang,S.H.Ong,X.Zhong,K.W.C.Foong,Eﬃcient3Ddentalidentiﬁcation viasignedfeaturehistogramandlearningkeypointdetection,PatternRecognit. 60(C)(2016)189–204.

[11] Y.Zou,T.Zhang,X.Wang,Y.He,J.Song,BRoPH:Acompactandeﬃcient bi-nary3Dfeaturedescriptor,in:ProceedingsofIEEEInternationalConferenceon RoboticsandBiomimetics(ROBIO),2016,pp.1093–1098.

[12] G.Elbaz,T.Avraham,A.Fischer, 3Dpointcloudregistrationforlocalization usingadeepneuralnetworkauto-encoder,in:ProceedingsofIEEEConference onComputerVisionandPatternRecognition(CVPR),2017,pp.2472–2481. [13] H.Huang,E.Kalogerakis,S.Chaudhuri,D.Ceylan,V.G.Kim,E.Yumer,Learning

localshapedescriptorsfrompartcorrespondenceswithmulti-view convolu-tionalnetworks,ACMTrans.Graph.37(1)(2017)1–14.

[14] R.Q.Charles,H.Su,K.Mo,L.J.Guibas,PointNet:deeplearningonpointsets for3Dclassiﬁcationandsegmentation,in:ProceedingsofIEEEConferenceon ComputerVisionandPatternRecognition(CVPR),2016,pp.77–85.

[15] C.R.Qi,L.Yi,H.Su,L.J.Guibas,PointNet++:deephierarchicalfeaturelearning onpointsetsinametricspace,in:ProceedingsofAdvancesinNeural Infor-mationProcessingSystems30,CurranAssociates,Inc.,2017,pp.5099–5108. [16] P.J.Besl,N.D.Mckay,Amethodforregistrationof3-dshapes,IEEETrans.

Pat-ternAnal.Mach.Intell.14(2)(2002)239–256.

[17] K.H.Bae,D.D.Lichti,Amethodforautomatedregistrationofunorganisedpoint clouds,ISPRSJ.Photogramm.RemoteSens.63(1)(2008)36–54.

[18] A.Gressin,C.Mallet,J.Demantk,N.David,Towards3Dlidarpointcloud reg-istrationimprovementusingoptimalneighborhoodknowledge,ISPRSJ. Pho-togramm.RemoteSens.79(I-3)(2013)240–251.

[19] J.Stechschulte, C.Heckman,Hidden Markov RandomFieldIterativeClosest Point,CoRRabs/1711.05864 (2017).

[20] J.Yang,H.Li,D.Campbell,Y. Jia, Go-ICP:agloballyoptimalsolutionto3d icppoint-setregistration,IEEETrans.PatternAnal.Mach.Intell.38(11)(2016) 2241–2254.

[21] D.Campbell,L.Petersson,GOGMA:globally-optimalgaussianmixture align-ment, in:Proceedings ofIEEEConference on ComputerVisionand Pattern Recognition,2016,pp.5685–5694.

[22] J. Straub, T. Campbell, J.P. How, J.W. Fisher, Eﬃcient global point cloud alignmentusing Bayesiannonparametric mixtures, in: Proceedings ofIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp.2403–2412.

[23] J.Fan,J.Yang,D.Ai,L.Xia,Y.Zhao,X.Gao,Y.Wang,Convexhullindexed Gaus-sianmixturemodel(CH-GMM)for3Dpointsetregistration,PatternRecognit. 59(C)(2016)126–141.

[24] P.W.Theiler,J.D.Wegner,K.Schindler,Keypoint-based4-pointscongruentsets automatedmarker-lessregistrationoflaserscans,ISPRSJ.Photogramm. Re-moteSens.96(11)(2014)149–163.

[25] A.E. Johnson, M. Hebert, Usingspinimages for eﬃcientobject recognition incluttered3dscenes, IEEETrans.PatternAnal.Mach.Intell. 21(5)(1999) 433–449.

[26] F.Tombari,S.Salti,L.DiStefano,UniqueSignaturesofHistogramsfor Local SurfaceDescription,Springer,Berlin,Heidelberg,pp.356–369.

[27] R.B.Rusu,N.Blodow,Z.C.Marton,M.Beetz,Aligningpointcloudviewsusing persistentfeaturehistograms,in:ProceedingsofIEEE/RSJInternational Confer-enceonIntelligentRobotsandSystems,2008,pp.3384–3391.

[28] B.Yang, Y. Zang, Automated registration ofdenseterrestrial laser-scanning pointcloudsusingcurves,ISPRSJ.Photogramm.RemoteSens.95(3)(2014) 109–121.

[29] O.Kechagias-Stamatis,N. Aouf,Histogramofdistancesforlocalsurface de-scription,in:ProceedingsofIEEEInternationalConferenceonRoboticsand Au-tomation(ICRA),2016,pp.2487–2493.

[30] B.Yang,Y.Zang,Z.Dong,R.Huang,Anautomatedmethodtoregisterairborne andterrestriallaserscanningpointclouds,ISPRSJ.Photogramm.RemoteSens. 109(2015)62–76.

[31] D.Zai,J.Li,Y.Guo,M.Cheng,P.Huang,X.Cao,C.Wang,Pairwiseregistration ofTLSpointcloudsusingcovariancedescriptorsandanon-cooperativegame, ISPRSJ.Photogramm.RemoteSens.134(SupplementC)(2017)15–29. [32] J.Yang,Q.Zhang,Y.Xiao,Z.Cao,TOLDI:aneffectiveandrobustapproachfor

3Dlocalshapedescription,PatternRecognit.65(2017)175–187.

[33] J.Navarrete,D.Viejo,M.Cazorla, Compressionandregistrationof3Dpoint cloudsusingGMMs,PatternRecognit.Lett.110(2018)8–15.

[34] J.Yang,Q.Zhang,Z.Cao,Theeffectofspatialinformationcharacterizationon 3Dlocalfeature descriptors:aquantitative evaluation,PatternRecognit. 66 (2017)375–391.

[35]S.Wang,L.Luo,N.Zhang,J.Li,Autoscaler:Scale-attentionNetworksforVisual Correspondence,arXivpreprintarXiv:1611.05837 [cs.CV] (2016) 1–10. [36] M.E.Fathy,Q.-H.Tran,M.ZeeshanZia,P.Vernaza,M.Chandraker,

Hierarchi-calmetriclearningandmatchingfor2dand3dgeometriccorrespondences, in:ProceedingsoftheEuropeanConferenceonComputerVision(ECCV),2018, pp.803–819.

[37] S.Song,J.Xiao,DeepSlidingShapesforAmodal3DObjectDetectioninRGB-D Images139(2)(2015)808–816.

[38]A.Zeng,S.Song,M.Niebner,M.Fisher,J.Xiao,T.Funkhouser,3DMatch: Learn-ingLocalGeometricDescriptorsfromRGB-DReconstructions(2017)199–208. [39] G.Elbaz,T.Avraham,A.Fischer,3Dpoint cloudregistrationforlocalization

usingadeepneuralnetworkauto-encoder,in:ProceedingsofIEEEConference onComputerVisionandPatternRecognition,2017,pp.2472–2481.

[40] I.Sipiran,B.Bustos,Harris3D:arobustextensionoftheharrisoperatorfor interestpointdetectionon3dmeshes,Vis.Comput.27(11)(2011)963. [41] Q.-Y.Zhou,J.Park,V.Koltun,Fastglobalregistration,in:Proceedingsof

Euro-peanConferenceonComputerVision,Springer,2016,pp.766–782.

[42] T.Hackel,N.Savinov,L.Ladicky,J.D.Wegner,K.Schindler,M.Pollefeys, SE-MANTIC3D.NET:anewlarge-scale point cloudclassiﬁcationbenchmark, in: ProceedingsofISPRSAnnalsofthePhotogrammetry,RemoteSensingand Spa-tialInformationSciences,IV-1-W1,2017,pp.91–98.

[43] Y.Guo,F.Sohel,M.Bennamoun,M.Lu,J.Wan,Rotationalprojectionstatistics for3dlocalsurfacedescriptionandobjectrecognition,Int.J.Comput.Vis.105 (1)(2013)63–86.

[44] R.B.Rusu,S.Cousins,3Dishere:pointcloudlibrary(PCL),in:Proceedingsof IEEEInternationalConferenceonRoboticsandAutomation,2011,pp.1–4.

Wei Li receivedthe M.Sc.degreeincollege ofScience from Jimei University, China, in 2013. He is currently workingtowardsthePh.D.degreeinschoolof informat-icswiththeFujianKeyLaboratoryofSensingand Com-putingforSmartCities,XiamenUniversity.Hiscurrent re-searchinterestsincludecomputervision,3Dpointcloud processing,registrationof3Ddata,objectdetectionand extraction,mobilelaserscanningdataprocessing.

ChengWang(M’11-SM’16)receivedthePh.D.degreein information and communication engineering from Na-tionalUniversityofDefenseTechnology,Changsha,China, in2002.HeisaProfessorwiththeFujianKeyLaboratory ofSensingandComputingforSmartCitiesandan Asso-ciateDeanoftheSchoolofInformatics,Xiamen Univer-sity,China.Hiscurrentresearchinterestsincluderemote sensingimageprocessing,mobileLiDARdataanalysis,and multi-sensorfusion.

Dr.Wanghascoauthoredabout150paperspublished inrefereedjournalssuchastheIEEETRANSACTIONSON GEOSCIENCEANDREMOTESENSING,IEEETRANSACTIONS ONINTELLIGENTTRANSPORTATIONSYSTEMS,IEEE GEO-SCIENCEANDREMOTESENSINGLETTERS,IEEEJOURNALOFSELECTEDTOPICSIN APPLIEDEARTHOBSERVATIONSANDREMOTESENSING,andISPRSJournalof Pho-togrammetryandRemoteSensingandconferencessuchasIGARSSandISPRS.He istheChairoftheISPRSWorkingGroupI/6onMulti-sensorIntegrationandFusion (2016–2020).HeisaCouncilMemberoftheChineseSocietyofImageandGraphics. ChengluWen(M’14-SM’17)receivedthePh.D.degreein mechanical engineeringfrom ChinaAgricultural Univer-sity,Beijing,China,in2009.SheiscurrentlyanAssociate ProfessorwiththeFujianKeyLaboratoryofSensingand Computing forSmart Cities,SchoolofInformatics, Xia-menUniversity,Xiamen,China.Shehascoauthoredmore than 40research paperspublishedinrefereedjournals and proceedings.Hercurrent research interestsinclude 3D pointcloudprocessing,machinelearning,androbot vison.SheistheSecretaryoftheISPRSWGI/6on Multi-SensorDataFusion(2016–2020),andAssociateEditorof IEEE-GRSL.

Zheng Zhang received bachelor’s degree in School of FuzhouUniversity,FuZhou,China,in2016.

Heis currentlyamaster studentinSchoolof Infor-mation Scienceand Engineering,XiamenUniversity.His currentresearchinterestsincludepointcloudregistration andmobilescanningdataprocessing.

(12)