Robust 3D face capture using example based photometric stereo

AUTHOR QUERY FORM

Journal:

COMIND

Please e-mail or fax your responses and any corrections to:

E-mail:

[email protected]

Article Number:

2453

Fax:

+353 6170 9272

Dear Author,

Please check your proof carefully and mark all corrections at the appropriate place in the proof (e.g., by using on-screen

annotation in the PDF file) or compile them in a separate list. Note: if you opt to annotate the file with software other than

Adobe Reader then please also highlight the appropriate place in the PDF file. To ensure fast publication of your paper please

return your corrections within 48 hours.

For correction or revision of any artwork, please consult

http://www.elsevier.com/artworkinstructions

.

Any queries or remarks that have arisen during the processing of your manuscript are listed below and highlighted by flags in

the proof. Click on the ‘

Q

’ link to go to the location in the proof.

Location in

Query / Remark:

click on the Q link to go

article

Please insert your reply or correction at the corresponding line in the proof

Q1

Please confirm that given names and surnames have been identified correctly.

Q2

Please check the telephone number of the corresponding author, and correct if necessary.

Q3

Fig. 1 will appear in black and white in print and in color on the web. Based on this, the respective

figure caption has been updated. Please check, and correct if necessary.

Q4

Please provide complete details for Ref. [15].

Q5

Please supply the name of the city of publication for Ref. [47].

Please check this box or indicate your approval if

you have no corrections to make to the PDF file

[image:1.842.54.558.324.531.2]

1

2

Robust

3D

face

capture

using

example-based

photometric

stereo

3 Q1

Rafael

F.V.

Saracchini

a,

*

,

Jorge

Stolfi

a

,

Helena

C.G.

Leita˜o

b

,

Gary

A.

Atkinson

c

,

4

Melvyn

L.

Smith

c

5 a

InstituteofComputing,StateUniversityofCampinas,Brazil

6 b

InstituteofComputing,FederalFluminenseUniversity,Brazil

7 c

MachineVisionLaboratory,UniversityoftheWestEngland,UnitedKingdom

8

9 1. Introduction

10 Captureofthethree-dimensionalgeometryofthehumanfaceis 11 a computer vision problem with many applications, including 12 facial recognition and mood detection, computer animation, 13 plasticsurgeryandautomatedsculpture.Inparticular,ithasbeen 14 showninrecentyearsthattheuseofthe3Dgeometryoftheface 15 improves the robustness of facial recognition methods under 16 variationsinillumination,poseandperspective[1–4].

17 Several techniques have been used for 3D facial geometry 18 capture, including laser ranging [5], structured lighting [6], 19 geometric stereo [7] and shape-from-shading [8]. Shape-from-20 shading (with a single illumination) is limited to objects with 21 uniformcolorandfinish,andistoounreliableforpracticalfacial 22 recognition. Currentlytheonly methodsthat are beingusedin 23 commercialsystemsarelaserrangingandgeometricstereo,with 24 or without structured lighting. However these methods often 25 requirethetargettoremainstillduringthescanning,andrequire 26 bulky and expensive specialized equipment. These factors 27 significantlylimittheirapplication.

28 Morphablemodels[9]havebeenproposedasawaytohandle 29 variationsinillumination,poseand perspectivewithoutfull3D 30 geometrycapture.However,thosemethodsareinherentlylimited 31 intheiraccuracyandrobustnessunderperturbations,likeglasses 32 orfacialhairthatarenotpreviouslyincludedin themorphable

33 model;andareunabletocapturefinedetailsofskin.Theyalsohave

34 a somewhat high computational cost, which clearly limits

35 application.

36

Variablelightingphotometricstereo,here calledsimply

photo-37

metric stereo(PS),is theextensionof shape-from-shadingusing

38 multipleimageswithasingleviewpointbutdifferentillumination

39 conditions. As a potential 3D facial capture method, its main

40 advantages are that it requires very simple and inexpensive

41 equipment, can be used in ordinary environments without

42 hampering the subject’s motion or demanding his cooperation,

43 andcancapturehigh-resolution3Ddatainafractionofasecond

44

[10–12].Indeed,Broadbentetal.demonstratedaphotometricstereo 45 system that captures 640480 depth maps at video rates (15

46 frames-per-second)usingaPCwithapopulargraphicscard[13].

47 Photometric stereo does not obtain the depth information

48 directly; insteadit measurestheaverage normalof thesurface

49 within each image pixel. The slope data is then integrated to

50 providetherelativedepthsofdifferentpartsoftheobject.These

51 integrateddepthsaresomewhatlessaccuratethanthoseobtained

52 withlaserrangingandstructuredlight,butareaccurateenoughfor

53 facialrecognition,consideringthenaturalvariationofhumanface

54 geometryovertime.Indeed,theslopedatacanbeuseddirectlyin

55 facialrecognitionalgorithms[14,15].

56 Photometricstereo requiresinformation onthefinish ofthe

57 surfaceandon theilluminationconditionsofeachinputimage.

58 Specifically,itneedsforeachimagetheshadingfunctionthatmaps

59 asurfacenormaltothecorrespondingshadingfactor.Italsoneeds

60

aweightmap,amaskthatidentifiestheimagepixelswherethe

ARTICLE INFO

Articlehistory:

Availableonlinexxx

Keywords:

Photometricstereo Shape-from-shading 3Dsurfacereconstruction Facialgeometrycapture Imageprocessing

ABSTRACT

Weshowthatusingexample-basedphotometricstereoispossibletoachieverealisticreconstructionsof thehumanface.Themethodcanhandlenon-Lambertianreflectanceandattachedshadowsaftera simplecalibrationstep.Weusesphericalharmonicstomodelandde-noisetheilluminationfunctions fromimagesofareferenceobjectwithknownshape,andafastgridtechniquetoinvertthosefunctions andrecoverthesurfacenormalforeachpointofthetargetobject.Thedepthcoordinateisobtainedby weighted multi-scale integration of these normals, using an integration weight mask obtained automaticallyfromtheimagesthemselves.WehaveappliedthesetechniquestoimprovethePHOTOFACE

systemofHansenetal.(2010)[10].

ß2013PublishedbyElsevierB.V.

*Corresponding

Q2 author.Tel.:þ552191975833.

ContentslistsavailableatSciVerseScienceDirect

Computers

in

Industry

j ou rna l h ome p a ge : w ww . e l se v i e r. co m/ l oc a te / c om pi nd

0166-3615/$–seefrontmatterß2013PublishedbyElsevierB.V.

61 brightnessdataisnotreliableorrelevantforsomereason(e.g.for 62 being part of the background, for being affected by projected 63 shadows,or for straddling silhouette edgesof theobject).This 64 maskisessentialforaccurateintegrationoftheslopemaps[16]. 65 Hereweconsiderspecificallyexample-basedphotometricstereo 66 (EBPS),avariantofthemethodwheretheshadingfunctionsare 67 obtainedby analyzingimages ofanexampleobjectwithknown 68 geometry and the same finish as the scene, under the same 69 illuminationconditions.Themaincontributionsofthispaperare 70 (1)theuseofhigh-ordersphericalharmonicstomodeltheshading 71 functions and remove noise from the example images; (2) an 72 unsupervisedmethodtoconstructtheweightmaskofeachimage; 73 and(3)theuseofthesemethodstoimprovethePHOTOFACE

real-74 timefacecapturesystemofHansenetal.[10]. 75 2. Basicconceptsandnotation

76 Thebasicproblem ofphotometricstereo istodetermine the 77 orientationofthesurface (thatis,thesurface normal)at every 78 visiblepointofanopaqueobject,givenm3digitalimagesS1,..., 79 Smof it, all takenwiththe samepose and viewpoint but with

80 distinctilluminationconditions.Weassumethattheimagesare 81 geometricallyalignedandphotometricallycorrected,sothatthey 82 havea commondomainSR2,andthat thesamplesS1[p], ..., 83 Sm[p]atthesamepoint p2Saretheapparentradiancesofsame

84 pointP[p]on thevisiblesurface ofthetargetobject,under the 85 variouslightings.Thegoalisthentodeterminesurface’snormal 86 ~s½patthatpoint.

87 Forsimplicityweassumethattheimagesaremonochromatic. 88 We also assume that the camera’s field of view is sufficiently 89 narrowand thelight sourcesin each image aresufficiently far 90 awaythatthelightingandviewingdirectioncanbeassumedtobe 91 uniformovertheentiretargetobject.Themethodcanhoweverbe 92 extendedtocolor images, non-uniformlight fields,and conical 93 insteadofparallelimageprojection.

94 Inthesimplestversionofthephotometricstereoproblem,one 95 also assumes that the surface’s finish is known, isotropic and 96 uniform.Moreprecisely,oneassumesthatthesurface’s bidirec-97 tional radiance distributionfunction (BRDF)

s

[p] at P[p]is the 98 productof a known isotropic and absorption-free BRDF

b

, the

99 surfacefinish,andanunknownfactorˆs½p2½0;1,thealbedoatthat

100 point(alsocalledtheintrinsiccolororlightabsorptioncoefficient).It 101 followsthattheintensitySi[p]ofeachimagepixelcanbeanalyzed

102 astheproductofthealbedoˆs½pandashadingfactorthatdepends 103 onlyontheimageindexiandonthesurface’snormal~s½patthat 104 point.Specifically,

Si½p¼ˆs½pLið~s½pÞ (1)

105

106 Here,eachLiistheshadingfunctionforimageSi,thatmapsaunit

107 vector~n2S2totheapparentradianceofawhitesurfacewithBRDF 108

b

,orientedwithnormal~nunderthelightingconditionsofimageSi.

109 Theshading functions are related tothe finish BRDF

b

by the 110 equation

Lið~nÞ¼

Z

S2

F

ið

~uÞ

b

ð~n; ~u;~

v

Þd~u (2) 111

112 where~

v

istheviewingdirection(fromthepointP[p]towardthe 113 camera),and

F

ið~uÞistheintensityofthelightflowinthedirection

114 ~u(thatis,theradianceofthe‘‘sky’’inthedirection~u)prevailingin 115 imageSi.Iftheilluminationforimageiwasprovidedbyadistant

116 pointsourceinthedirection~ui,theintegralreducesto Lið~nÞ¼

F

i

b

ð~n; ~u

i;~

v

Þ (3)

117

118 wherethefactor

F

i quantifiestheintensityofthatlightsource.

119 Notethatweareincludingthegeometricfactormaxf~n~u;0ginthe

120 finishBRDF

b

,andweusetheseequationsonlywhen~n~

v

>0.Note

121 alsothat,bytheuniformlightingassumption,theshadingfunction

122

Lidoesnotdependonthepositionp,exceptthroughthenormal

123 ~s½p.

124 Thislightingmodelallowsattachedshadows,andisadequate

125 forscenesconsistingofasinglemostlyconvexobject.Ontheother

126 hand,thismodelcannotaccountforprojectedshadows,radiosity

127 effects,orsourceswithunevenlightdistribution.

128 Withthesedefinitions,wecanformallystatethebasicproblem

129 ofphotometricstereoasfollows:giventheintensitiesS0[p], ...,

130

Sm[p]forapixelp,findthenormalvector~n½pandthealbedoˆs½p

131 thatsatisfyEq.(1)foralli.Thisproblemgenerallycanbesolvedif

132 theilluminationsaresufficientlyvariedandthefinishBRDF

b

is

133 dominated by wide-angle scattering: that is, more like the

134 LambertianBRDFthanthatofamirroredorglossyblacksurface.

135

2.1. Observationvectors

136

Theobservation vectorof a pixel p2Sis them-vectorof its

137 radiancesinalltheimages,thatis,

S½p¼ðS1½p;S2½p;...;Sm½pÞ (4)

138 139 Wedefinetheshadingvectorfunctionasthelistofallmshading

140 functions,thatis,thefunctionLfromS2toRmsuchthat

Lð~nÞ¼ðL1ð~nÞ;L2ð~nÞ;...;Lmð~nÞÞ (5)

141 142 The basic problem of photometric stereo can be stated more

143 succinctlyas:giventhevectorS[p]ofapixelp,findtheunitvector

144 ~

n2S2suchthatthevectorLðn~ÞispracticallycollinearwithS[p],that 145 is,the anglebetweenthemis nearlyzero.(Needless tosay,the

146 inevitablemeasurementerrorsintheradiancesSi[p]willintroduce

147 randomperturbationsintheobservationvectorS[p],soonecannot

148 expectexactcollinearity.)Thenwecaninferthatthesurfacenormal

149 ~s½p atpis~n,andthatthe albedoˆs½pis theratiokS½pk=kLð~nÞk

150 betweenthem-dimensionalEuclideannormsofthetwovectors.

151

2.2. Observationsignatures

152 Wecanremovethealbedosfromtheproblembynormalizing

153 theobservationvectorsandtheshadingvectorfunction.Namely,

154 we define the observed signature s[p] of a pixel p as being its

155 observation vector normalized to unit length; and the shading

156

signaturefunctionlastheshadingvectorfunctionnormalizedin

157 thesameway.Thatis,

s½p¼ S½p

kS½pk; lð~nÞ¼

Lð~nÞ

kLð~nÞk (6)

158 159 Notethats[p]isavectoronthesphereSm1,andlisafunctionof

160

S2 to Sm1. Then the photometric stereo problem reduces to 161 computing the functional inverse of the shading signature

162 function; that is, find~s½p2S2 such that lð~s½pÞ is as close as

163 possibletos[p]inthenorm||||(whichisamonotonicfunctionof

164 theanglebetweenthevectors).

165

2.3. Thesufficientdatahypothesis

166 The photometric stereo approach will fail if the shading

167 signaturefunctionisnotinvertible,thatis,iftherearetwonormal

168 directions~n0; ~n00 withcollinearshadingvectorsLð~n0Þ¼

a

Lð~n00Þfor

169 somescalar

a

.Toavoidthisproblem,theilluminationconditions

170 must be sufficiently varied to break any such ambiguities. In

171 particular,thelightsourcesusedinallimagesmustnotbeallinthe

172 sameplane,andeveryvisiblepointofthetargetobjectmustbe

173 illuminatedonatleastthreeoftheimages.Wewillassumethat

175 On theotherhand,example-basedPS can inprinciple work 176 withnon-LambertianBRDFsandarbitrarylighting,aswellaswith

177 attached shadows and penumbras, since these effects do not

178 destroytheproportionalitybetweenthevectorsS[p]andLð~nÞ.In 179 particular,thereisnoneedtoidentifytheimagesandpixelswhere 180 attachedshadowsoccur.(Castshadowsandscene-scatteredlight, 181 however,arestillaproblem.)

182 3. Relatedwork

183 Variable-lighting photometric stereo was first studied as a 184 computervisionproblembyWoodhaminthelate1980s[17].In 185 his pioneer work, Woodham demonstrated that is possible to 186 recovertheinclinationofthesurfaceusingatleast3non-coplanar 187 point-likelightsources.

188 Shadowsandglossyreflections:In thefollowingyears,

Wood-189 ham’s results were improved and extended in many ways. 190 Woodham originally assumed that the scene had Lambertian 191 finish,andconsideredonlypointsthatwerefullyilluminatedbyall 192 three sources with known directions. In that case the surface 193 normalvectorcanbecomputedfromthethreeintensityvaluesof 194 thepixelbyasimpleanalyticformula.BarskyandPetrou[18]later 195 showedhowtohandleglossyhighlightsandshadowsusingonly4 196 images,providedthatsuchanomaliesaffectthevalueofapixelin 197 atmostone ofthefourimages. Yu etal. [19]thenusedlinear 198 programming toextend this result for an arbitrary number of 199 lightings, allowingmultiple anomalies in each pixel. Due their 200 simplicity and low processing cost, these methods are now 201 commonlyusedforreal-timeandfastcaptureapplications[11,13].

202 Unknownlighting:In practiceoneoftenfacesamoredifficult

203 versionoftheproblemwherethelightingconditionsofeachimage 204 arenotknownaprioriandmustbedeterminedfromtheimages 205 themselves.Hayakawa[20] addresseda limited version of this 206 problem,assumingthateachimagewasilluminatedbyadistant 207 pointsourcewithunknowndirection.Heclaimedthatthenormal 208 directionscouldbeobtained throughsingular-value decomposi-209 tionoftheinputdata,viewedasmatrixwhereeachrowisapixel 210 andeachcolumnaninputimage.Thissolutionwaslaterimproved 211 byYuilleetal.[21,22].Basrietal.[23,24]furthergeneralizedthe 212 solutionusingsphericalharmonicstohandleambientand semi-213 diffuselighting.However,Hayakawa’sapproachislimitedbecause 214 ofinherentambiguitiesintheproblem,andrathersensitivetothe 215 arrangementoflightsources.Theresultingnormalsareaffectedby 216 indeterminatefactorsthatmustbedeterminedbyothercriteria 217 andexplicitlycorrectedfor.

218 Unknownsurfacefinish:Aldrinetal.[25]andGoldmanetal.[26]

219 consideredthemoredifficultproblemwherethesurfacefinish

b

is 220 unknown.Theyassumedthat

b

ofthesurfacewassomeunknown 221 linearcombinationofafinitelibraryof‘‘model’’BRDFs (Lamber-222 tian, glazed, etc.), with different coefficients in each pixel. 223 McGunnigleet al. [27]were able torecovergood estimates of 224 surfaceofmetallic objectsunderverystrictlightingconditions. 225 Higo[28]doesnotattempttomodeltheBRDF,assumingonlythat 226 itismonotonicandisotropicandachievesgoodresultsinpresence 227 ofglossyhighlightsandshadows.Thesemethodsrequiredozensof 228 inputimages,andtheirhighcomputationalcostpreventstheiruse 229 inreal-timeapplications.

230 Example-based photometric stereo: In spite of the advances

231 described above, the problem of photometric stereo with an 232 unknownfinishBRDF and/orunknownlightingremainsfraught 233 withpracticalandcomputationaldifficulties.Theexample-based 234 approach to photometric stereo sidesteps these difficulties by 235 extracting the shading functionsLi directly from m calibration

236 images G1, ...,Gm ofa referenceobjectwithknown shape and

237 albedo, taken from the same viewpoint and under the same 238 lightingsastheimagesS1,...,Sm.

239 ThisapproachwasintroducedbyWoodhamin1989[17]and

240 furtherexploredrecentlybyHerztmannandSeitz[29,30]andby

241 us[31]Theinconvenienceofhavingtoplaceareferenceobjectin

242 thesceneiscompensatedbythefactthatrecoveryofthesurface

243 normals with very complex BRDFs and arbitrary lighting is

244 possible. Moreover, the shading functions Li, are inherently

245 smoother(andthereforeeasiertomodel)thanthefinishBRDF

b

246 andthelightflows

F

i.Theexample-basedapproachisalsovery

247 efficientandusuallyproducesgoodresultswithonly6–12images.

248

3.1. Slopeintegration

249 In thecontext of this paper,‘‘3D geometrycapture’’ means

250 determining for each pixel p2S the heightz[p]of the surface

251 visibleinp,relativetosomearbitraryreferenceplane

perpendicu-252 lartotheviewingdirection~

v

.Photometricstereodoesnotyield

253 directlythisinformation,butonlythesurfacenormalvector~s½p

254 within that pixel.The normal canbe triviallyconverted tothe

255

heightgradientorslopevector~_rz½p,thevectorofR2consistingof

256 theXandYderivativesoftheheightmapz.Thegradientmap_r~_z

257

thenmustbeintegratedtoyieldtheheightmapz.

258 The integration of gradient maps obtained by photometric

259 stereoisnotatrivialtask.Foronething,thegradientdataisnot

260 continuous,butdiscretized,thatis,availableonlyinthecenterof

261 pixels,formingaregularorthogonalgrid.Moreover,thegradient

262 data _r~_{z½p is contaminated by errors caused by camera noise,}

263 violationsofthephotometricstereoassumptions(opaquesurface,

264 uniformlighting,constantfinish,etc.)andapproximationsinthe

265 photometricstereoalgorithmitself(suchastheerrorduetothe

266 useofafinitetabletoinverttheshadingsignaturefunction).

267 Forsomepixels,themagnitudeofsuchperturbationsmaybeso

268 high that the gradient value~_rz½p ispractically unknown. That

269 happens, in particular,in backgroundareasthatareoutsidethe

270 rangeofthelightsources,orwheretheimageisverydark,orwhere

271 the surfaceiscoveredbyhairorothernon-trivial3Dtexture,or

272 wherethepixelstraddlesadiscontinuityintheheightfunction,such

273 as a silhouette edge (the boundary of the projection of some

274 foregroundobject).Oneshouldnotethatthegradientdata_r~z½pfor

275 apixelpisanon-linearfunctionofthepixelradiancesS1[p],...,

276

Sm[p];andeachSi[p]istheaverageoftheradianceofthesurfaceover

277 somefiniteregioncorrespondingtop,whichinturnisanon-linear

278 functionofthesurface’struegradient.Therefore,ifthetruesurface

279 gradient variesconsiderablywithin thepixel,thegradient_r~_z½p

280 computedbyphotometricstereomaybesubstantiallyincorrect.

281 These unavoidable errors in the photometric gradient _r~_z

282 requiretheuseofspecializedintegrationalgorithms.A

compara-283 tivesurveywaspresentedbyusinapreviousarticle[16].Some

284 integratorsthatarestillwidelyusedinthisproblem,suchaspath

285 integration [32] and Frankot–Chellapa’s Fourier-based method

286

[33]simplyignorethoseerrors,andmayproduceveryincorrect 287 heightmaps.SeeFigs.1and2.

288

WeightedPoissonintegrators:Theonlyintegratorsthatcancope

289 with missing and unreliable gradient data are the weighted

290 Poisson-basedmethods[34,35,16,36].Thosemethodsrequirean

291 extrainput,aweightmapthatspecifiesthereliabilityw½pofeach

292 gradientvalue~_{rz½p,asanumberbetween0(‘‘meaningless’’)and}

293 1 (‘‘maximallyreliable’’). Thoseintegrators set up a system of

294 linear equations that relate the given gradients_r~_{z½p to finite}

295 differencesofunknownheightsz[p0_{]ofadjacentpixels.Thelinear}

296 systemisthensolvedtoobtaintheheights.

297

Fast system solving: The linear system built by Poisson

298 integratorsis sparsebut verylarge:ithasoneunknownheight

299 andoneequationforeachpixelwithnon-zeroweight,andeach

300 equationtypicallyhasfivetoninenon-zerocoefficients.Solvingit

301 by Gauss elimination requiressuper-linear space; solving it by

303 multi-scaleiterativesolverthatwedevelopedspecificallyforthis 304 problemthatusesalinearamountofmemoryandrunsinlinear 305 timefortypicalinstances[16].

306 3.2. Photometricstereofor3Dfacecapture

307 Due toitsnon-intrusive natureand lowercost,photometric 308 stereo hasreceived considerableattention in recent years as a 309 methodforthecaptureoffacialgeometrythatmaybeoptimalfor 310 certainapplications.ItsviabilitywasdemonstratedbyYuilleetal. 311 [22],usingtheirproposedSVDmethodforfaceimagescaptured 312 under light sources with unknown direction. However, their 313 methoddemandedalargenumberofimagestoovercomeerrors 314 introduced by deviations of the Lambertian reflection model. 315 Georghiadesetal.[37]managedtoimprovethisresultforhuman 316 facesusingonly7distinctlightsources,bydiscardingvaluesthat 317 wereunder or over a predeterminedthreshold. Lee et al. [38]

318 extended his approach, using 3 known light sources and an 319 arbitrarynumberofimageswithunknownlighting.

320 ThePHOTOFACEsystembuiltbyHansenetal.[10]demonstrated 321 theuseofphotometricstereofornearlyinstantaneousfacecapture 322 ofpeopleinmotion.Thissystemisdescribedinmoredetailin

323 Section4.Itusesfourpoint-likelightsources,andcomputesthe 324 normalsbyasimpleanalyticmethodthatassumesaLambertian 325 surfacefinish.Thenormalsareusedasinputforfacerecognition 326 algorithms.

327 A major difficulty in this application is that skin has a 328 complicatednon-Lambertianfinish,duetoitsstructureofmultiple 329 translucentlayers.

330 Real-timefacecapture:Thecaptureoffacial3Dgeometryinreal

331 time, possibly at video frame rates, has received significant 332 attentioninthelastfewyears.Forgreaterspeed,someofthese 333 systems usemulti-spectral method, namely colorcameras and 334 coloredlights,tocapturethreeormoreimagesSiatthesametime.

335 ThistechniquetoowaspioneeredbyWoodham,in1994[39]. 336 UsingamodifiedversionofHayakawa’sSVDmethod,Schindler 337 [40]achievesreal-timecaptureusingacolormonitorasa multi-338 spectral the light source. The captured geometry is not very 339 accurate,beingintendedtobeusedinfacemodelingand video-340 chats. Vogiatzis et al. [12] uses a combination of shape-from-341 motionandmulti-spectralphotometricstereo,allowingforglossy 342 reflectionsusingamodifiedPhongmodel.Fyffeetal.[41]usesa

343 customized 6-channel camera to recover albedo and surface

344 inclinationwithasinglepicture.

345

4. ThePHOTOFACEsystem

346 Our work builds on the PHOTOFACE project, a hardware and

347 software system developed in 2010 by Hansen, Atkinson, and

348 othersattheMachineVisionLaboratory(MVL)oftheUniversityof

349 the West of England (UWE) [10]. PHOTOFACE was designed for

350 automatic,near-instantaneous,non-intrusive3Dfacecapture of

351 peoplewalkingthroughaninstrumentedbooth.Itsdemonstration

352 prototypeconsistedofaaluminumframestructurewitha

high-353 speedphotographiccameraandfournear-infraredlightsources.

354 ThesystemwastriggeredbyanultrasoundsensorasthePerson

355 wasabout2mawayfromthecameraandwalkingtowardit.The

356 camerasnapped fourphotos oftheperson inquick succession,

357 whileeachlightwasflashedinturn,andthenonemorewithall

358 lightsturnedofftorecordtheambientlight.SeeFig.3.

359

4.1. Specifications

360 AllPHOTOFACEdevicesarecontrolledbyastandardPC.Thelight

361 sourcesandtheultrasoundtriggerareconnectedtothecomputer Fig.2.Aheightmapwithdiscontinuities(left),itsheightgradientmap(middle)asitwouldbecomputedbyphotometricstereowithoutanynoise,andtherecoveredheight map(right)computedfromthegradientbytheFrankot–ChellapaFourier-basedintegrator[33].

Fig.3.ThePHOTOFACEprototypebuiltatUWEMVL.

Fig.1.Asimpleheightmap(left),itsheightgradientmapwithsomenoiseadded(middle)andtherecoveredheightmap(right)producedfromthegradientbytheFraile–

Hancocktree-pathintegrator[32].(Inthegradientmap,theXandYderivativesarecombinedinto

Q3 colorvalues.)(Forinterpretationofthereferencestocolorinthisfigure

362 viaanNIPCI-7811DIOcard.AnNIPCIe-1426framegrabberisalso 363 connectedtotheDIOcardviaaRTSIbusfortriggeringpurposes, 364 andtothecameraviaaCameraLinkß_{interface.Thelatterisusedto}

365 sendthetriggeringsignaltothecameraandtostoretheframedata 366 inthecomputer.ThecameraisalsodirectlyconnectedtotheDIO 367 card.AllinterfacingcodeiswritteninNILabVIEWandcoordinates 368 alltheimagecapturesequence.

369 Lightsources:Eachofthefournear-infraredlightsourcesisa

370 VIS080IR7-LEDcluster,whichemitslightat 850nm.Thefour 371 lightsarearrangedinairregularrectangularshapepointingtoward 372 the general area where the person will walk through. This 373 dispositionensuresthatthelightsourcedirectionswouldnotbe 374 co-planar, in order to obtain good quality images for each 375 illuminationtriplet.

376 Camera:ThecameraisaBasler504kcmodelwitha55mm,f5.6

377 Sigmalenspositionedinthecenteroftherectangleformedbythe 378 lightsources.Itisabletocapturean128010248-bitRGBimage 379 inlessthan5ms.

380 Trigger:Thesystem’striggerisanultrasoundsensor,ahighly

381 directionalBaumerproximityswitch that is activatedwhenits 382 beamisbrokenwithinadistanceof70cm.

383 4.2. Imagecapture

384 Toensurethealignmentoftheimages(requiredby photomet-385 ricstereomethods), thecapture shouldbeperformedin avery 386 smalltimeinterval.Itwasverifiedexperimentallythatisnecessary 387 at least a framerate of 150frames-per-second and a accurate 388 synchronizationoflightsources.

389 Theframegrabbingprocessisstartedwhenapersoncrossesthe 390 beamoftheultrasoundsensor.Uponreceiptofthesensor’ssignal, 391 theDIOtells thecameratostarttheframeintegrationprocess. 392 Oncethecameraconfirmsthattheintegrationhasstarted,theDIO 393 cardflashesoneofthelamps,andwaitsforthecameratosignalthe 394 endofframecapture.TheDIOcardrepeatsthissequenceforthe 395 otherthreelights,andthenoncemorewithalllightsturnedoff.

396 Theentirecaptureprocesstakesabout50ms,whichseemstobe

397 instantlytoahumanobserver.

398

4.3. Imageprocessing

399 The ‘‘dark’’ image is subtracted from the four illuminated

400 imagestoremovethecontributionofambientlighting.Sincethe

401 subjectsmayvarysubstantiallyinheight,thecameraissetupto

402 capture anarea many timeslarger than theperson’s face. The

403 imagesarethereforecroppedtotheface’sapproximatebounding

404 box(typically 400 500 pixels),whichisdeterminedbythe

405 algorithmofLienhartandMaydt[42].SeeFig.4.

406 AsdescribedbyHansenetal.,thePHOTOFACEsoftwareusedan

407 analyticmethodtocomputethesurfacenormalateachpixel.The

408 methodassumedaLambertiansurfacefinishandasingledistant

409 point-like light source in each image. First, their algorithm

410 computed a tentative normal~s0½p by assuming that the pixel

411 wasilluminatedbyall foursources. Underthis assumptionthe

412 shading Eqs. (1)–(3) reduce toan overdetermined system of 4

413 equationson3unknowns,thatwassolvedwiththeleastsquares

414 criterion.

415 Asacheckforthepresenceofshadows,theythencomputeda

416 second normal ~s00½p by the same method but excluding the

417 image Si with smallest Si[p]. If ~s

00

½p made an obtuse angle 418 with thedirectionof the excludedlightsource, theyassumed

419 thatthe pointwasin thatlight’sshadowandset~s½p to~s00½p,

420 otherwisetheyset~s½pto~s0½p.Heightmapswerethencomputed

421 from the normals usingthe Frankot–Chellapa integrator [33].

422 SeeFig.5.

423

Calibration:Theanalyticmethodusedrequiredthedirectionof

424 each light source relative to the person’s face. Since the light

425 sourceswerenotpermanentlyfixedtotheframe,acalibrationstep

426 wasperformedonceafterassemblytoobtainthatinformation.For

427 thatpurpose,acapturesequencewasperformedwithareflective

428 sphere inplace oftheperson’sface. Thedirectionofeachlight

429 sourcewasthen determinedbythelocation ofits reflection (a

Fig.4.FourimagescapturedbyPHOTOFACEaftersubtractionofambientlightingandtrimmingbythefacedetectionalgorithm.

430 brightspot)onthesphere.Thefourlightsourceswereassumedto 431 havethesameintensity.

432 4.4. Ourimprovements

433 TheanalyticmethodusedinthefirstversionofPHOTOFACEwas

434 veryfast,butsinceitassumedaLambertianfinishfortheskinit 435 yielded rather inaccurate normals wherever the image was 436 affectedbyglossy reflections.Themethodalsofailedusually in 437 those regions (mainly under the chin and nose) which were 438 illuminatedbyonlytwoofthefourlights.

439 TheFrankot–Chellapaintegratortoowaschosenforitsspeed:it 440 tookonlyafewsecondstoprocesseachnormalmap.However,since 441 itusestheFastFourierTransformalgorithm,it necessarilygives 442 equalweighttoallpixels.Therefore,theintegratedfacegeometry 443 was often distorted by the spurious gradient values in the 444 backgroundandshadowedareas.Thesedistortionspreventedthe 445 useoftheheightmapsforfacerecognition;theslopemapswere 446 usedinstead[10].

447 Intheremainderofthisarticlewedescribeanimprovedimage 448 processing system we developed for the PHOTOFACE hardware. 449 Namely,wereplacedtheanalyticnormalcomputationalgorithm 450 bytheexample-basedmethod,whichcaninprinciplehandlethe 451 semi-glossyfinishofhumanskin.Forthispurposewedevelopeda 452 reliablemethodtoextracttheshadingfunctionfromtheimagesof 453 theexampleobjectthatremovesmostofthenoisepresentinthose 454 images.ThismethodisdescribedinSection5.

455 WealsoreplacedtheFrankot–Chellapaintegratorbyour multi-456 scaleweightedPoissonintegrator,describedinapreviousarticle 457 [16].SincethePHOTOFACEsystemismeanttooperateautomatically,

458 wedeveloped an algorithmtoautomatically extractthe weight 459 maskfromthe capturedimages. Thisalgorithmis describedin

460 Section6.

461 The newsoftwaretakesabout3stoobtainthenormal map 462 fromtheacquiredphotos,andanother3stocomputetheheight 463 map,onPHOTOFACE’sPC.Thesetimesarelessthan50%higherthan

464 thoseoftheoriginalPHOTOFACEsoftware.

465 5. TheEBPSalgorithm

466 5.1. Table-basednormaldetermination

467 Inordertoinverttheshadingsignaturefunctionl,weobtaina 468 sufficientlydense setTofsample pairstk¼ð~nk;tkÞ2S2Sm1

469 withk=1,2...,N,wheretk¼lð~nkÞ.Thenforeachpixelpinthe

470 targetobjectwelocateinthistabletheentryð~nr;trÞforwhichthe

471 distance||trs[p]|| is minimum, and return the corresponding

472 normal~nr asthepresumednormal~s½poftheobject’ssurfacein

473 thatpoint.

474 ThisapproachisverysimilartothatusedbyWoodhamin1994 475 [39];exceptthatweusenormalizedsignaturess,linsteadofthe 476 unnormalized observation vectors. This approach is extremely 477 flexible,sinceitcanworkwithanylightsources,concentratedor 478 diffuse,andanyconstantisotropicfinish

b

,aslongasthelighting

479 functions Li are fairly smooth and the signature function l is

480 invertible.NotethatitdoesnotrequiremodelingtheBRDF

b

orthe

481 lightdistributions

F

iexplicitly.

482

5.2. Fasttablelook-up

483 Theaccuracyoftheresult~nrreturnedbytablelook-upmethod

484 dependsonlyonthedensityofthesamplenormals~nkinTandthe

485 amountofnoisepresentinthegivenimages.Asfortheformer,

486 even if the normals in T are uniformly distributed over the

487 hemisphereH2,theangularerrorbetween~nrandthetrueinverse

488

l1_{(s[p])willbeabout1}

:5=pffiffiffiffiNradians.Thus,forexample,inorder

489 tokeepthatpartoftheerrorbelow18,thetablemusthaveatleast

490 8000entries.Forthisreason,thetablelook-upstepdominatesthe

491 computationalcostofphotometricstereo.

492 Computingthedistance||tks[p]||hascostproportionaltom,

493 therefore a simple linear search of the table would have cost

494 proportionaltoNm.Woodham’smethodtospeedupthelook-up

495 wastoquantizetheobservedradiancesSi[p]asb-bitintegers,and

496 use them as indices into and m-dimensional array where the

497 normals~nkwerepreviouslystored.Thismethodreducedthelook-up

498 cost to O(m); however, since the table required 2mb _{entries, it}

499 severelylimitedthenumberofimagesmandtheaccuracyofthe

500 result. Other data structures for fast m-dimensional nearest

501 neighbor searching have been proposedin the following years,

502 suchask-dimensionaltreesearch,Approximatenearestneighbors

503

[29], or Locality Sensitive Hashing [43]. However, while these

504 methods have O(logN) asymptotic look-up cost, they are not

505 effective in this problem because of the so-called curse of

506

dimensionality[44]: namely,thedatabecomessosparseinhigh

507 dimensionalspacesthatthemethodsonlybegintoworkforvery

508 largetables,muchlargerthanthetablesneededforphotometric

509 stereo.

510 Therefore,weuseinsteadatwo-dimensionalhashingmethod

511 that wedeveloped specificallyforphotometric stereo[45].Our

512 methodexploitsthefactthatthesetofallsignatureslð~nÞfor~n2H2

513 isatwo-dimensionalsurfacepatchinthepositiveorthantofRm.It

514 usesatwo-dimensionalhashingarraytoreducethesearchtoa

515 smallnumber(constant,ontheaverage)oftableentries.Thususes

516

O(N)space,andprovidesapproximatelyO(m)averagelook-upcost, 517 independentlyofthetablesizeN.

518

5.3. Thereferenceobjects

519 Thereferenceobjectmustbechosensothatitsnormals~g½qare

520 accuratelyknownandprovideadenseandcompletecoverageof

521

H2. Spherical or hemispherical reference objects with uniform

522 albedoaremostconvenient,sincetheyareeasilyobtainedwith

523 highly accurategeometry,are completelydescribed bya single

524 geometricparameter(theradius),andallow~g½qtobecomputed

525 directly from the coordinates of pixel q by simple algebraic

526 formulas.

527 Since theexample-based approach does not requireexplicit

528 knowledgeofthelightsourcedirections,wereplacedthereflective

529 sphere used in the original PHOTOFACE calibration run by the

530 referenceobject,aspherecoatedwithsemi-glossywhitepaint.See 531 Fig.6.

532 Although no effort was made to match the BRDF of this 533 referenceobjecttotheBRDFofthehumanskin,themereinclusion 534 of a semi-glossy term improved considerably the accuracy 535 ofthe computednormalmaps; especiallyin theforehead and 536 nose,wheretheglossyreflectionsweremostconspicuous. See 537 Section7.

538 5.4. Usingtheexampleobjectimages

539 LetG1,...,Gmbetheimagesofthereferenceobject.Applying 540 Eq.(1)toeachpixelqintheseimagesthatfallsonthereference 541 object,weget

Lið~g½qÞ¼ Gi½q

ˆ

g½q (7)

542

543 where~g½qandgˆ½qarethe(known)surfacenormalandthealbedo 544 ofthereferenceobjectatpixelq.Thus,foreachpixelqoneach 545 imageGiofthereferenceobjectoneobtainsasamplevalueofthe

546 shadingfunctionLiforadirectiong~½q.Therefore,everysuchpixelq

547 providesasamplevalueoftheshadingsignaturefunction:

lð~g½qÞ¼ G½q

kG½qk (8)

548

549 where

G½q¼ðG1½q;G2½q;...;Gm½qÞ (9)

550

551 Thesesamplesoflarelimitedtothevisiblepartoftheobject,and 552 thereforetothedirections~n2S2thatmakeanacuteanglewiththe 553 viewingdirection~

v

.WewilldenotebyH2thatsubsetofS2.

554 5.5. Acquiringtheshadingfunctions

555 ThesignaturetableTcouldbebuiltdirectlyfromtheimagesof 556 thereferenceobject,namely~nk¼~g½qandtk=g[q]foreachpixelq

557 onthereferenceobject.Howevertheserawsamplesignaturesare 558 usually too few to yield the desired precision, and are often 559 contaminatedbyimagingnoiseandbysmallgeometricaldefects 560 (suchasscratchesandbumps)ontheexampleobjectitself.Evenif 561 hardlyperceptibleontheimages,theseperturbationsleadtolarge 562 errorsinthesignatureg[q]orinthenormals~g½q,thusintroducing 563 spuriousdataonthetable.SeeFig.7.

564 Inordertoattenuatesucherrorsandobtainasignaturetable 565 that is dense enough, we fit a mathematical model L˜i of each

566 shadingfunctionLitotherawdatapairsð~g½q;Gi½q=gˆ½qÞ.Wethen

567 re-samplethesemathematicalmodelsatadensesetofdirections 568 inH2toobtainthetableT.

569

5.6. Shadingfunctionmodel

570 WenowdescribehowtoobtaintheapproximationsL˜i.Since

571 thisstepiscarriedoutseparatelyforeachimage,wewilldropthe

572 indexifromLi,L˜i andGiinthissection.

573 We use a linear approximation model with weighted least

574 squarescriterion.Specifically, wechoosea setofbasisfunctions

575

f

1;

f

2:::

f

l:S

2

)R,andlookforanapproximationL˜ofLoftheform

˜ Lð~nÞ¼X

l

r¼1

a

r

f

rð~nÞ (10)

576 577 for each~n2S2,where

a

1, ...,

a

larerealcoefficients chosento

578 minimizetheweightedquadraticerrorQðL˜Þ,namely

QðL˜Þ¼X

q G½q

ˆ

g½qL˜ð~g½qÞ

2

(11) 579 580 Thesumhereistakenoverallpixelsqthatarecompletelyinsidethe

581 outlineofthereferenceobject.Thealbedogˆ½qmustbegiven,and

582 thecoefficientvector

a

canbefoundbysolvingthelinearsystem

Ma¼b (12)

583 584 whereMisallmatrixandbisacolumnvectoroflelements

585 givenby

Mij ¼

X

q

f

ið~g½qÞ

f

jð~g½qÞ

bi ¼

X

q

f

ið~g½qÞG½q=gˆ½q

(13)

586 587 The basis we have chosen for this application consists of the

588 monomials

f

ið~nÞ¼xrysztforallnaturalnumbersr,s,twhosesum

589 iseitherdord1,wheredisachosenpositiveinteger,insome

590 arbitraryorder.Herex,y,zaretheCartesiancoordinatesofthe

591 normalvector~n.Thesemonomialsgeneratepreciselythespaceof

592 allsphericalharmonicfunctionsofmaximumdegreed[46],but

593 they are mucheasier to compute than the standard spherical

594 harmonicbasisfunctionsYrs[47].Thelatterareusuallypreferred

595 because they are orthogonal when integrated over the whole

596 sphere.However,theinnerproductimplicitlyusedinthequadratic

597 error formula (11) is a discrete sums over an irregular set of

598 normals ~g½q2H2. The harmonic basis functions Yrs are not

599 orthogonal in this inner product, and therefore have no clear

600 advantageoverthemonomials.

601

5.7. Virtualreferenceobjects

602 Forcheckingpurposes,thefittedshadingfunctionsL˜ianbeused

603 to produce synthetic images G˜i of a ‘‘virtual’’reference object,

Fig.7.AnimageGiofareferenceobjectandaplotofitsshadingfunctionLiðnÞ~for~n2H2hemisphere.Notethesubstantialamountofnoiseinthelatter thatisnotvisibleinthe

604 whereeachpixelG˜i½qispaintedwiththecolorL˜ið~g½qÞexpected

605 fromitsassumednormal~g½q.Thesenewsyntheticimagescanbe 606 directlycomparedtotherawimagesG[i]ofthereferenceobject. 607 SeeFig.8

608 6. Obtainingtheweightmap

609 We now consider the problem of obtaining the gradient 610 reliabilityweightmapthatisrequiredbytherobustintegrators. 611 Thereareseveralpublishedmethodsthattrytoderivetheweights 612 fromthegradientmapitself,byusingthefactthatthegradientofa 613 functionmustbeacurl-freevectorfield.Namely,theY-derivative 614 oftheX-slopemustbeequaltotheX-derivativeoftheY-slope;the 615 differencebetweenthetwobeingthecurlofthefield.Ifthecurlis 616 notzero,thegradientmapcannotbeintegrated.Thosemethods 617 generallymarkasunreliablethosepixelswherethecurlisnotzero, 618 orwherethegradientoftheintegratedmapdoesnotmatchthe 619 givengradient[34,35].

620 However, the zero-curl condition is only necessary, but not

621 sufficient,forthegradienttobecorrect.InFig.2,thecurlis

non-622 zeroalongthesidesoftheramp,butiszeroeverywhereelse.As

623 thatexampleshows,thediscontinuitiescannotbedetectedfrom

624 the gradient map alone. In general, the weight map must be

625 determined by problem-specific methods, for example edge

626 detectionbyprojectedshadows[48].

627

6.1. AmaskingalgorithmforPHOTOFACE

628 Wehavedevelopedanalgorithmforautomaticextractionofthe

629 reliabilityweightmaskfromthePHOTOFACEimagesets.Themethod

630 aimstoexcludetheareasofthescenewherephotometricstereo

631 cannotbeapplied,suchasthedistantbackground,hair,eyepupils,

632 andareaswhichareilluminatedbyonlytwoofthefoursources

633 (rightbelowthenoseandchin,andonthetemples).Themethod

634 also uses specific heuristics to exclude clothing and other

635 disconnectedbitsofsurface,suchaspartsoftheear.

Fig.8.Syntheticimageofavirtualreferenceobject(left)usingasmoothedshadingfunctionL˜i(right)fittedtothedataoftherealreferenceobjectimageofFig.7.

Fig.9.Constructionoftheweightmaskforacapturedimageset.Topleft:theinitialweightsw½pcomputedfromthesignaturematchqualityks½ptrkandfromthe

636 Thereare several publishedalgorithmsfor the detectionof 637 humanskininphotographs.[49,50].However,thosealgorithms 638 generallyrelyoncolorinformationwhichisnotavailableinthe 639 near-infrared monochromatic images captured by PHOTOFACE. 640 Instead,we usefour maincriteria: theself-consistency ofthe 641 photometric stereo computation, the estimated albedo of the 642 surface,theanglebetweenthecomputednormalandtheviewing 643 direction,spatialcoherence,andcontinuity.

644 Specifically,wefirstsettheweighw½pbytheformula

w½p¼kS½pk2e1=2d2

(14) 645

646 where

d¼ks½ptrkkS½pk

s

(15)

647 648 andtristhesignaturefromthetableTthatbestmatchesthepixel’s

649 signatures[p].Thefirstfactoreliminatesinformula(14)darkareas

650 wherethesignatures[p]istoocontaminatedbynoise.Thesecond

651 factorpenalizespixelswheretheobservedsignatures[p]deviates

652 from the expected signature for the recovered normal, and is

653 therefore likely to becontaminated bynoise, castshadows, or

654 otherun-modeledeffects.

655 Next, we set w½p to zeroif the Z component ~s½p~

v

of the

656 computednormal~s½pislessthanafixedthreshold

e

1(currentlyset

657 to0.03).Thisstepeliminatespartswherethesurfaceseemstobe

658 nearlyperpendiculartotheviewingdirection,andthereforecannot

659 be simultaneously illuminatedby threelight sources. Then the

660 weight w½p issettozeroif itis lessthananotherthreshold

e

2

661 (currently set to 0.2), in order to completely remove fromthe

662 computation those pixels whose height is expected to be too

663 unreliabletouse.

Fig.10.Toprow:imageS0fromeachofthefourtestdatasetscapturedwithPHOTOFACE,withnear-infraredlighting,afterambientlightsubtraction.Bottomrow:the correspondingweightmasksforintegration.

664 Next,weapplyagray-scalemorphologicalopeningoperation(an 665 erosionfollowed byadilatation)with a55circularkernel, to 666 removesmallisolatedpixelclumpsandbreakanynarrow‘‘bridges’’, 667 aswellasremovingpixelsnearproblematicareassuchasnostrils 668 andeyes.Thenweidentifytheconnectedcomponentsoftheimage 669 (separatedbyareasofzeroweight),anddiscardallbutthemost 670 centralone(whichisassumedtobetheface,consideringhowthe 671 imagewascropped).Finally,weapplyagray-scalemorphological 672 closingoperation(adilationfollowedbyanerosion)witha66 673 circularkernel,meanttopreservethe‘‘holes’’inthemaskattheeye 674 pupils,butcloseothersmallholesandgaps.SeeFig.9.

675 7. Tests

676 To illustrate the changes, we show below the height maps 677 obtained with the old and new PHOTOFACE algorithms, on the

678 same four datasets labeled face0,face4,face6,face7. See

679

Fig.10.

680 Thenormalswerecomputedfromthesedatasetswiththeold

681 PHOTOFACEanalyticalgorithmandwithournewEBPSalgorithm.For

682 thelatter,shadingfunctionswerefittedtothefourimagesofthe

683 referenceobject(Fig.6)usingthemonomialbasiswithmaximum

684 degree 6. These fitted functions were then re-sampled at

685 approximately 10,000 evenly distributed normal directions in

686

H2toformthesignaturetableT.

687 Bothnormalmaps,oldandnew,werethenintegratedwithour

688 multi-scalePoisson-basedintegrator.Theresultingheightmapsare

689 shown in Figs.11–14. For comparison, we alsoshow the facial

690 geometries of those same four people, captured in separate

691 occasionsbythe3DMDfacescanner[6]andcroppedtoaboutthe

692 samepartoftheface.(Unfortunately,numericalcomparisonwith

693 thelatterisnotviableduetodifferencesinfacialexpressionand

694 perspectivedistortion.)

Fig.12.Heightmapscomputedforthetestdatasetface4.

695 8. Conclusions

696 Weshowedhowexample-basedphotometricstereoisaviable 697 alternative for the capture the 3D surface geometry of human 698 faces,withuniqueadvantagesincludinglowcost,highspeed, non-699 intrusiveness,flexibility,highresolution,largeworkingdistance, 700 andindifferencetoalbedoandambientlighting.Wealsopointed 701 outtheinadequacyofthepopularFrankot–ChellapaFourier-based 702 gradientmapintegratorcomparedtoPoisson-based integrators. 703 Wedescribedanalgorithmforautomaticgenerationoftheweight 704 maskneededbythoseintegrators.

705 Inparticular,weshowedthattheaccuracyandrobustnessof 706 PHOTOFACE,astate-of-theartphotometricfacecapturesystem,are

707 significantlyimproved,withlittleextracomputationcost,when 708 theof popularanalytic algorithmsfor normal computation are 709 replacedbyEBPS,andtheFrankot–Chellapaintegratorisreplaced 710 byourmulti-scaleintegratoretal.[16].Eventhoughnoeffortwas 711 madetoreproducetheBRDFofthehumanskininthereference 712 object, the mere inclusion of a glossy term (which cannot be 713 handledbyanalyticmethods)wasenoughtoremovemostofthe 714 distortions created by glossy reflections when using the old 715 algorithms.

716 Evenwithourimprovements,theheightmapsobtainedwith 717 currentversionPHOTOFACEhavenoticeableproblems,especiallyin

718 theregionofnostrilsandunderthechin.Thesedefectsarenotdue 719 toalgorithmlimitationsbutrathertoinsufficientilluminationin 720 those areas. The addition of two more light sources to the 721 prototype would suffice to ensure the basic requirement of

722 photometricstereo,namelythateverypointofthetargetsurface 723 beilluminatedbyatleastthreelightsources.

724 References

725 [1]W.Zhao,R.Chellappa,P.J.Phillips,A.Rosenfeld,Facerecognition:Aliterature

726 survey,ACMComputingSurveys35(2003)399–458.,http://dx.doi.org/10.1145/ 727 954339.954342.

728 [2]K.W.Bowyer,K.Chang,P.Flynn,Asurveyofapproachesandchallengesin3dand

729 multi-modal3dþ2dfacerecognition,ComputerVisionandImage

Understand-730 ing101(1)(2006)1–15.,http://dx.doi.org/10.1016/j.cviu.2005.05.005http:// 731 www.sciencedirect.com/science/article/pii/S1077314205000822.

732 [3]S.Berretti,A.DelBimbo,P.Pala,3dfacerecognitionusingisogeodesicstripes,IEEE

733 TransactionsonPatternAnalysisandMachineIntelligence32(12)(2010)2162– 734 2177.

735

[4]Y.Wang,J.Liu,X.Tang,Robust3dfacerecognitionbylocalshapedifference

736

boosting,IEEETransactionsonPatternAnalysisandMachineIntelligence32(10) 737 (2010)1858–1870.

738

[5]Konica Minolta Sensing, 2012, http://sensing.konicaminolta.us (accessed

739

22.08.12).

740

[6]3dmdface system, 2010, http://www.3dmd.com/3dmdface.html (accessed

741

22.04.10).

742

[7]Dimensionalimaging’sdi4d4dfacialcapturesystem,2013,

http://www.di3d.-743

com/(accessed03.03.13).

744

[8]W.Smith,E.Hancock,Facialshape-from-shadingandrecognitionusingprincipal 745 geodesicanalysisandrobuststatistics,InternationalJournalofComputerVision

746 76(2008)71–91.

747

[9]V.Blanz,T.Vetter,Amorphablemodelforthesynthesisof3dfaces,in:

Proceed-748

ings ofthe 26thAnnual Conferenceon ComputerGraphics andInteractive

749

Techniques,SIGGRAPH’99,ACMPress/Addison-WesleyPublishingCo.,NewYork,

750

NY,USA,1999,pp.187–194.,http://dx.doi.org/10.1145/311535.311556.

751

[10]M.F.Hansen,G.A.Atkinson,L.N.Smith,M.L.Smith,3dfacereconstructionsfrom

752

photometricstereousingnearinfraredandvisiblelight,ComputerVisionand

753

ImageUnderstanding(2010)inpress.

754

[11]V.Nozick,Pyramidalnormalmapintegrationforreal-timephotometricstereo, 755 EAMMechatronics2010(2010)128–132.

756

[12]G.Vogiatzis,C.Herna´ndez,Self-calibrated,multi-spectralphotometricstereofor 757 3dfacecapture,InternationalJournalofComputerVision(2011)1–13.

758

[13]L.Broadbent,K.Emrith,A.R.Farooq,M.L.Smith,L.N.Smith,2.5dfacialexpression 759 recognitionusingphotometricstereoandtheareaweightedhistogramofshape

760

index,in:IEEERO-MAN,2012,2012,490–495.

761

[14]L.Broadbent,K.Emrith,A.Farooq,M.Smith,L.Smith,2.5dfacialexpression

762

recognitionusingphotometricstereoandtheareaweightedhistogramofshape 763

index,in:IEEERO-MAN,2012,2012,490–495., http://dx.doi.org/10.1109/RO-764 MAN.2012.6343799.

765

[15]S.N.Kautkar,G.A.Atkinson,M.L.Smith,Facerecognitionin2dand2.5dusingQ4 766

ridgeletsandphotometricstereo,PatternRecognition.

767

[16]R.F.Saracchini,J.Stolfi,H.C.Leita˜o,G.A.Atkinson,M.L.Smith,Arobustmulti-scale 768 integrationmethodtoobtainthedepthfromgradientmaps,ComputerVisionand

769 ImageUnderstanding116(8)(2012)882–895.

770

[17]R.J. Woodham, Determining surfacecurvaturewith photometric stereo, in: 771 Proceedingsofthe1989IEEEInternationalConferenceonRoboticsand

Automa-772 tion,vol.1,1989,pp.36–42.

773

[18]S. Barsky,M.Petrou, The4-source photometricstereo techniquefor three-774 dimensionalsurfacesinpresenceofhighlightsandshadows,IEEETransactions

775 onPatternAnalysisandMachineIntelligence25(10)(2003)1239–1252.

776

[19]C.Yu,Y.Seo,S.Lee,Photometricstereofrommaximumfeasiblelambertian 777 reflections,in:K.Daniilidis,P.Maragos,N.Paragios(Eds.),ComputerVisionECCV

778 2010,vol.6314ofLectureNotesinComputerScience,Springer,Berlin,

Heidel-779 berg,2010,pp.115–126.

780

[20]H.Hayakawa,Photometricstereounderalightsourcewitharbitrarymotion, 781 JournaloftheOpticalSocietyofAmericaSeriesA(JOSAA)11(11)(1994)3079–

782 3089.

783

[21]A.Yuille,D.Snow,Shapeandalbedofrommultipleimagesusingintegrability,in:

784

ProceedingsoftheIEEEConferenceonComputerVisionandPatternRecognition 785 (CVPR97),1999,pp.158–164.

786

[22]A.Yuille,D.Snow,R.Epstein,P.Belhumeur,Determininggenerativemodelsof

787

objectsundervaryingillumination:Shapeandalbedofrommultipleimagesusing

788 svdandintegrability,InternationalJournalofComputerVision35(1999)203–

789 222.,http://dx.doi.org/10.1023/A:1008180726317.

790 [23]R.Basri,D.Jacobs,Lambertianreflectanceandlinearsubspaces,IEEETransactions 791 onPatternAnalysisandMachineIntelligence25(2)(2003)218–233. 792 [24]R.Basri,D.Jacobs,I.Kemelmacher,Photometricstereowithgeneralunknown

793 lighting,InternationalJournalofComputerVision72(3)(2007)239–257. 794 [25]N.Alldrin,T.Zickler,D.Kriegman,Photometricstereowithnon-parametricand

795 spatially-varyingreflectance,in:IEEEConferenceonComputerVisionandPattern

796 Recognition,2008,CVPR2008,(2008),pp.1–8.,http://dx.doi.org/10.1109/CVPR 797 .2008.4587656.

798 [26]D.Goldman,B.Curless,A.Hertzmann,S.Seitz,Shapeandspatially-varyingbrdfs

799 fromphotometricstereo,IEEETransactionsonPatternAnalysisandMachine

800 Intelligence32(6)(2010)1060–1071.,http://dx.doi.org/10.1109/TPAMI.2009. 801 102.

802 [27]G.McGunnigle,Photometricstereowithgradatedextendedsourcesforrecovery

803 ofspecularsurfaces,JournaloftheOpticalSocietyofAmericaA27(5)(2010)

804 1127–1136. , http://dx.doi.org/10.1364/JOSAA.27.001127 http://josaa.osa.org/ 805 abstract.cfm?URI=josaa-27-5-1127.

806 [28]T.Higo,Y.Matsushita,K.Ikeuchi,Consensusphotometricstereo, in:IEEE Com-807 puterSocietyConferenceonComputerVisionandPatternRecognition,2010, 808 1157–1164.

809 [29]A.Hertzmann,S.M.Seitz,Example-basedphotometricstereo:shape reconstruc-810 tionwithgeneralvaryingBRDFs,IEEETransactionsonPatternAnalysisand 811 MachineIntelligence27(8)(2005)1254–1264.

812 [30]A.Hertzmann,S.M.Seitz,Shapeandmaterialsbyexample:aphotometricstereo

813 approach,in:ProceedingsIEEECVPR2003,vol.1,2003,pp.533–540.http:// 814 citeseer.ist.psu.edu/article/hertzmann03shape.html.

815 [31]H.C.G. Leita`o, R.F.V. Saracchini, J. Stolfi, Matching photometricobservation 816 vectorswithshadowsandvariablealbedo,in:Proceedingsofthe21stBrazilian

817 SymposiumonComputerGraphicsandImageProcessing(SIBGRAPI),2008,pp. 818 179–186.

819 [32]R.Fraile,E.R.Hancock,Combinatorialsurfaceintegration,in:Proceedingsofthe

820 18thInternationalConferenceonPatternRecognition(ICPR’06),vol.1,2006,pp. 821 59–62.

822 [33]R.T.Frankot,R.Chellappa,Amethodforenforcingintegrabilityinshapefrom 823 shadingalgorithms,IEEETransactionsonPatternAnalysisandMachine Intelli-824 gence10(4)(1988)439–451.

825 [34]A.Agrawal,R.Raskar,R.Chellappa,Whatistherangeofsurfacereconstructions 826 from a gradientfield?in: Proceedingsof the9th EuropeanConference on

827 ComputerVision(ECCV),vol.3951,2006,pp.578–591.

828 [35]D.Reddy,A.Agrawal,R.Chellappa,Enforcingintegrabilitybyerrorcorrection 829 using‘1-minimization,in:Proceedingsofthe2009IEEEConferenceonComputer

830 VisionandPatternRecognition,2009,pp.2350–2357.

831 [36]R.F.V.Saracchini,J.Stolfi,H.C.daGamaLeita˜o,G.A.Atkinson,M.L.Smith, Multi-832 scaleintegrationofslopedataonanirregularmesh,in:PSIVT(1),2011,109–120. 833 [37]A.Georghiades,P.Belhumeur,D.Kriegman,Fromfewtomany:illuminationcone 834 modelsforfacerecognitionundervariablelightingandpose,IEEETransactionson

835 PatternAnalysisandMachine Intelligence23(6)(2001)643–660.

836 [38]S.-W.Lee,P.S.Wang,N.Y.Svetlana,L.Seong-whan,Noniterative3dface recon-837 structionbasedonphotometricstereo,InternationalJournalofPattern Recogni-838 tionandArtificialIntelligence22(03)(2008)389–410.

839 [39]R.J. Woodham,Gradientandcurvaturefrom thephotometricstereomethod 840 includinglocalconfidenceestimation,JournaloftheOpticalSocietyofAmerica, 841 SeriesA11(11)(1994)3050–3068.

842 [40]G.Schindler,Photometricstereoviacomputerscreenlighting,in:Proceedingsof 843 the 4thInternational Symposium on3D Data Processing,Visualization and 844 Transmission,2008.

845 [41]G.Fyffe,X.Yu,P.Debevec,Single-shotphotometricstereobyspectral

multi-846 plexing,in:IEEEInternationalConferenceonComputationalPhotography(ICCP), 847 2011,2011,1–6.,http://dx.doi.org/10.1109/ICCPHOT.2011.5753116.

848 [42]R.Lienhart,J.Maydt,Anextendedsetofhaar-likefeaturesforrapidobject 849 detection, in: Proceedings ofthe 2002 InternationalConference on Image

850 Processing,vol.1,2002,pp.I-900–I-903.,http://dx.doi.org/10.1109/ICIP.2002. 851 1038171.

852 [43]L. Zhong,J.J.Little,Photometricstereovia localitysensitivehigh-dimension 853 hashing,in:in:ProceedingsoftheSecondCanadianConf.onComputerand 854 RobotVision(CRV’05),2005,pp.104–111.

855 [44]P.Indyk,R.Motwani,Approximatenearestneighbour:towardremovingthecurse 856 ofdimensionality,in:Proceedingsofthe13thAnnualACMSymposiumonTheory 857 ofComputing(STOC’98),1998,pp.604–613.

858 [45]R.F.V.Saracchini,J.Stolfi,H.C.daGamaLeitao,Auniformgridstructuretospeed

859 upexample-basedphotometricstereo,IEEETransactionsonImageProcessing20

860 (12)(2011)3495–3507.,http://dx.doi.org/10.1109/TIP.2011.2159386.

861 [46]A.Gomide,J.Stolfi,Basesfornon-homogeneouspolynomialCksplinesonthe

862 sphere,in:ProceedingsofLATIN’98-LatinAmericanTheoreticalInformatics 863 Conference,vol.1380ofLectureNotesinComputerScience,Springer,1998,pp.

864 133–140.

865Q5[47]C.Mu¨ller,SphericalHarmonics,vol.17, Springer,1966.

866 [48]M.Chandraker,S.Agarwal,D.Kriegman,ShadowCuts:photometricstereowith

867 shadows,in:CVPR07,2007,1–8.

868 [49]C.Garcia,G.Tzirita,Facedetectionusingquantizedskincolorregionsmerging 869 andwaveletpacketanalysis,in:IEEETransactionsonMultimedia,vol.3,1999,

870 264–277.

871 [50]R. lienHsu, M. Abdel-mottaleb, A.K. Jain, Face detection in color images, 872 IEEE Transactions on Pattern Analysisand Machine Intelligence 24 (2002) 873 696–706.

874

875 876 RafaelF.V.SaracchinireceivedtheB.Sc.degreefrom

877

theFluminenseFederalUniversity,Nitero´i,Brazil, in

878

2007,andthePh.D.degreefromStateUniversityof

879

Campinas, Campinas, Brazil, in 2012. His research

880

interest include 3-D data recovery through digital

881

imagesusinggeometricandphotometricstereofrom

882

non-Lambertian and glossy/shiny surfaces and its

883

applicationsin medicine, industry, andsecurity. In

884

2012RafaelmovedtotheTechnicalInstituteofCastilla

885

y Leo´n to develop Computer Vision aided systems

886

focusedinassisted livingto elderlypeopleand 3D

887

reconstructionofindoorsenvironments.

888

889 890 Jorge Stolfi received the B.E. degree in electrical

891

engineeringandtheM.Sc.degreeincomputerscience

892

fromtheUniversityofSaoPaolo,SaoPaolo,Brazil,in

893

1973and1979,respectively,andthePh.D.degreein

894

computersciencefromStanfordUniversity,Stanford,

895

CA,in1989.HeiscurrentlyaFullProfessorwiththe

896

InstituteofComputing,StateUniversityofCampinas,

897

Campinas,Brazil.Hisresearchinterestsincludenatural

898

languageprocessing, computationalgeometry,

com-899

putergraphics,numericalanalysis,imageprocessing,

900

andapplications.

901

902 903 HelenaCristinadaGamaLeita˜oreceivedtheB.Sc.and

904

M.Sc.degrees incomputersciencefromtheFederal

905

UniversityofRiodeJaneiro,RiodeJaneiro,Brazil,in

906

1988and1993,respectively,andthePh.D.degreefrom

907

theStateUniversityofCampinas,Campinas,Brazil,in

908

1999.SheiscurrentlyanassociateProfessorwiththe

909

InstituteofComputing,FluminenseFederalUniversity,

910

Nitero´i,Brazil.Herresearchinterestsincludepattern

911

recognitionforarcheological,medical,andmolecular

912

biologyapplications.

913

914 915 GaryAtkinsoncompletedanM.Sc.degreeinphysicsat

916

theUniversityofNottinghamin2003.Upongraduation,

917

hemovedtotheUniversityofYorktostudyforaPh.D.

918

degreeintheDepartmentofComputerScience,underthe

919

supervision of Edwin Hancock. His research was

920

concernedwithimprovingshaperecoveryalgorithms

921

andreflectancefunctionestimationforcomputervision.

922

Mostof his work involved theexploitation of the

923

polarizingpropertiesofreflectionfromsurfaces.In2007,

924

GarymovedtoUWEtoworkonfacereconstructionand

925

recognition research in collaboration with Imperial

926

CollegeLondon.GaryalsohastieswiththeUniversity

927

ofCampinas,theUniversityofYork,theUniversityof

928

CentralLancashireandseveralindustrialpartners.He

929

supervisedMarkHansen’sPhDinfacerecognitiontocompletioninMarch2012.In

930

additiontohiscomputervisionresearch,forwhichhehaspublishedovertenmajor

931

peer-reviewedinternationaljournalpapers,Garyhasworkedinthemedicalfieldto

932

develop a new means of assessing head deformations in babies affected by

933

plagiocephaly.In2010,GaryactedastheeditorandUKandEuropeco-convener

934

for the International Conference and Exhibition on Biometrics Technology in

935

Coimbatore, India. Gary teaches undergraduate and postgraduate courses in

936

ComputerVisionandEngineering.’’

937

938 939 Melvyn Smith is Professor of Machine Vision and

940

DirectoroftheCentreforMachineVisionintheBristol

941

RoboticsLaboratory(BRL)apartnershipbetweenthe

942

UniversityofBristolandtheUniversityoftheWestof

943

England. He received his B.Eng. (Hons) degree in

944

MechanicalEngineeringfromtheUniversityofBathin

945

1987,M.Sc. inRoboticsandAdvancedManufacturing

946

SystemsfromtheCranfieldInstituteofTechnologyin

947

1988 and Ph.D. in Computer Vision and Graphical

948

ModelingfromtheUWEin1997.Heactsasassociate

949

editorforfourleadinginternationaljournals,including

950

ComputersinIndustry,andregularlyactsasaprogram

951

committeememberforinternationalconferencesinthe

952

computervisionfield.Todatehehaspublishedinexcess

953

of150articles,including4patentapplications,abookand2bookchapters,edited2

954

journalspecialissues,42refereedjournalpapers,59refereedconferencepapersas

955

wellasnumerousnon-refereedpapersandspecialreports.Heregularlypresentshis

956

workthroughinvitedseminars,aswellasarticlesandinterviewstothepopular

957

media.HehasbeenamemberoftheEPSRCPeerReviewCollegesince2003,servedasa

958

EuropeancommissioncandidateevaluatorfortheSixthFrameworkandiscurrentlya

959

programandevaluator/reviewexpertandmonitoringexpertfortheEUFramework7

960