Gregory G.Slabaugh
ThomasMalzbender, W. Bruce Culbertson y
School of Electricaland Comp. Engineering Client and Media Systems Lab
Georgia Institute of Technology Hewlett-Packard Laboratories
Atlanta,GA 30332 Palo Alto,CA 94306
Abstract
Starting with aset of calibrated photographs taken
ofascene,voxelcoloringalgorithmsreconstruct
three-dimensionalsurfacemodelsonanitespatialdomain.
Inthispaper,wepresentamethodthatwarpsthevoxel
space, so that the domain of the reconstruction
ex-tends to an innite or semi-innite volume. Doing
soenables the reconstructionofobjectsfar away from
the cameras,aswellasreconstructionofabackground
environment. Newviewssynthesizedusingthewarped
voxel spacehaveimprovedphoto-realism.
1 Introduction
Voxelcoloringalgorithms[7,5,2]reconstruct
three-dimensional surfaces using a set of calibrated
pho-tographs taken of ascene. When working with such
algorithms, onetypicallydenesareconstruction
vol-ume,whichisaboundingvolumecontainingthescene
that is to bereconstructed. Oncedened, the
recon-struction volume is divided into voxels, forming the
voxel space in which the reconstruction will occur.
Voxels that are consistent with the photographs are
assigned acolor,andinconsistent voxelsareremoved
(carved)fromthevoxelspace[7].
These algorithmshavebeenparticularlysuccessful
inreconstructingsmall-scalescenesthatarerestricted
toanitedomain. Applyingthemtolarge-scalescenes
can become challenging, since one must use a large
reconstruction volume to contain the scene. Such a
largereconstructionvolumecanconsistofanunwieldy
numberofvoxelsthatbecomesprohibitivetoprocess.
Inaddition,itisunnecessarytomodelfarawayobjects
with high resolutionvoxels. Ideally,onewouldlikea
spatially adaptivevoxelsizethat increasesawayfrom
thecameras.
Furthermore,voxelcoloringalgorithmsarenotwell
suitedtocapturingtheenvironment(sky,background
objects,etc.) ofascene. Typicalreconstructions are
y
photo-realistic in the foreground, which is modeled,
butemptyinthebackground,whichisunmodeled. As
a result, synthesized new views can have large
\un-known" regions, as shown in black in Figure 1. For
somescenes, such asanoutdoorscene, wemightlike
toreconstructthebackgroundaswell,yieldingamore
photo-realisticreconstruction.
To address these issues, we propose a warping of
thevoxelspacesothatsurfacesfartherawayfromthe
cameras can be modeled without an excessive
num-ber of voxels. In addition, our proposed warping of
the voxel space can extend to innity along any
di-mension, so that innite (all of R 3
), orsemi-innite
(suchasahemispherewithinniteradius)
reconstruc-tion volumes can be dened. The latter might best
model anoutdoor scene. As will beshown in
subse-quentsectionsofthispaper,wedevelopahybridvoxel
spaceconsistingofaninteriorspaceinwhichvoxelsare
notwarped,andanexteriorspaceinwhichvoxelsare
warped. Thevoxelsarewarpedso that thefollowing
criteriaaremet:
1. Nowarpedvoxelsoverlap.
2. Nogapsform betweenwarpedvoxels.
3. The warped reconstruction volume is at least
semi-innite.
Avoxelcoloringalgorithmisthenexecutedusingthe
warpedreconstructionvolume.
Thelayoutofthispaperisasfollows. First,we
ex-ploresomerelated work. Then, weintroducea
func-tionthatwarpsthevoxelspacesubjecttothecriteria
enumeratedabove. Next,wediscusssome
implemen-tationdetailsthatarisewhenperforminga
reconstruc-tion in warped space. We then present results that
demonstratetheeectivenessofourapproach.
2 Related Work
Figure 1: Unknown regions due to reconstruction on a nite domain. A photograph of our \bench" scene is
shown in (a), withthe reconstruction volume superimposed. Onlyvoxelswithin the reconstruction volume are
considered in voxelcoloringalgorithms. Thescene containsmany objects outsideofthe reconstructionvolume
thatarenotreconstructed,resultinginunknownregionsthatappearasblackinaprojectionofthereconstruction,
shown in (b). The ideaspresentedin this paper warp the voxel space, so that thereconstruction volume can
becomeinnite,andthebackgroundsceneandenvironmentcanbereconstructed.
scene reconstruction problem. Seitz and Dyer's [7]
voxel coloring technique exploits color correlation of
surfaces to nd a set of voxels that are consistent
with the photographs taken of a scene. Kutulakos
and Seitz [5] develop a space carving method that
extends voxel coloring to support arbitrary camera
placement via amulti-sweep algorithm. Culbertson,
Malzbender,andSlabaugh[2]presenttwogeneralized
voxel coloring (GVC) algorithms, which, like [5]
al-low for arbitrarycamera placement, and in addition
usetheexactvisibilityofthescenewhendetermining
if a voxel is consistent with the photographs. These
threemethods,referredtocollectivelyas\voxel
color-ing algorithms", havebeenquite successful in
recon-structing three-dimensionalscenes on anite spatial
domain. Inthispaper,weextendthesethreemethods
in orderto reconstruct sceneson aninnite or
semi-innitedomainbywarpingthevoxelspaceusedinthe
reconstruction. Doing so enables the reconstruction
ofnearbyobjects,far-awayobjects,andeverythingin
between.
SaitoandKanade[6],andlaterKimura,Saito,and
Kanade[4]specifyavoxelspaceusingtheepipolar
ge-ometryrelatingtwo[6]orthree[4]basisviews,for
vol-umetric reconstruction using weakly calibrated
cam-eras. Intheirapproach,avoxeltakesonanarbitrary
hexahedral shape, a consequence of their projective
space. In our approach, we intentionally warp
exte-rior voxels into arbitrarily shaped hexahedra. In [6]
and [4], a voxel's size is solely based on its location
approach, avoxel's size is instead based on its
loca-tionin auser-dened voxelspace. In[6] and[4], the
reconstruction volume is nite, and only foreground
surfaces are reconstructed. In contrast, our method
warps the voxel space to innity so that objects far
fromthecamerascanbereconstructed,inadditionto
foregroundsurfaces.
In the computer graphics domain, innite scenes
have been modeled and rendered using environment
mapping. This methodprojectsthebackgroundonto
theinteriorofasphereorcubethatsurroundsthe
fore-groundscene. Blinn and Newell [1] use such a
tech-nique tosynthesize reectionsof theenvironmento
ofshiny foregroundsurfaces,aprocedure also known
as reection mapping. Greene [3] additionally
ren-derstheenvironmentmap directly to generate views
of the background. This approach is quite
eec-tiveat producing convincing synthetic images.
How-ever, sincethe foreground and backgroundare
mod-eled dierently, separate mechanisms must be
pro-vided to create and render each. Furthermore, the
three-dimensionalityoftheenvironmentislost,asthe
backgroundisrepresentedasatexture-map. Like
en-vironmentmapping, thetechniquesdescribedin this
paper seek an eÆcient mechanism to represent the
backgroundscene. Ourwarpedvolumetricspace
pro-videsthis inasingleframeworkthat canmoreeasily
accommodate surfaces that appear both in the
fore-groundand background. In addition, we reconstruct
com-representaninniteorsemi-innitevolumewitha
-nitenumberofvoxels,whilesatisfyingtherequirement
thatnovoxelsoverlapandnogapsexistbetween
vox-els. Therearemanypossiblewaystoachievethisgoal.
Inthissection,weusethetermpre-warped toreferto
the volumebeforethe volumetricwarpingfunction is
applied.
Thevolumetricwarpingmethodpresentedhere
sep-arates the voxel space into an interior space used to
modelforegroundsurfaces,andanexteriorspaceused
to model backgroundsurfaces, as shown in Figure 2
(a). Thevolumetricwarpdoesnotaectthevoxelsin
the interior space, providing backwardcompatibility
withpreviousvoxelcoloringalgorithms,andallowing
reconstruction ofobjectsin theforegroundat axed
voxelresolution.
Voxelsintheexteriorspacearewarpedaccordingto
awarpingfunction thatchangesthesize ofthe voxel
based on its distance from the interior space. The
furtheravoxelintheexteriorspaceislocatedfromthe
interiorspace,thelargeritssize,asshowninFigure2
(b). Voxels on the outer shell of the exterior space
havecoordinateswarpedto innity,andhaveinnite
volume. Note that while the voxels in the warped
space haveavariable size,the voxel spacestill hasa
regular3Dlatticetopology.
Tohelp further limitthe classof possiblewarping
functions, weintroduce thefollowingdesirable
prop-ertyofawarpedvoxelspace:
Constantfootprint property: For each image,
voxels project to the same number of pixels,
independentofdepth.
Figure 3showsanexampleofavoxelspacethat
sat-ises the constant footprint property for two
cam-eras. Assumingperspectiveprojection,avoxelspace
that satises this property has a spatially adaptive
voxel size that increases away from the cameras, in
a manner perfectly matched with the images. While
a useful conceptualconstruct, the constantfootprint
property cannot in general be satised when more
than n cameras are present in R n
space. Thus, for
three-dimensionalscenes,avoxelspacecannotbe
con-structedthatsatisesthepropertyforgeneralcamera
placement when there are more than three cameras.
Sincereconstructionusingthreeorlesscamerasis
lim-iting, weinsteaddesignourvolumetricwarping
func-tion to approximate the constant footprint property
tionthat is used to warp theexteriorspace. We
de-veloptheequationsandguresin twodimensionsfor
simplicity;theideaeasilyextendstothreedimensions.
The frustum warp assumes that both the interior
space and the pre-warped exterior space have
rect-angular shaped outer boundaries, as shown in
Fig-ure4. Thepre-warpedexterior spaceis dividedinto
four trapezoidal regions, bounded by(1) lines l
con-nectingthe fourcornersof theinteriorspaceto their
respectivecornersoftheexteriorpre-warpedspace,(2)
theboundaryoftheinteriorspace,and(3)the
bound-aryofthepre-warpedexteriorspace. Wedenotethese
trapezoidalregions as x, and y, based on the
re-gion'srelativepositiontocenteroftheinteriorspace.
TheseregionsarealsoshowninFigure 4.
Let (x;y) be a pre-warped point in the exterior
space,andlet(x
w ;y
w
)bethepointafterwarping. To
warp(x;y),werstapplyawarpingfunctionbasedon
theregioninwhichthepointislocated. Thiswarping
functionisappliedonlytoonecoordinateof(x;y). For
example,supposethat thepointis locatedin the+x
region,asdepictedin Figure5. Pointsin the+xand
x regionsarewarpedusingthex-warpingfunction,
x w =x x e x i x e jxj ; (1) where x e
is the distance along the x-axis from the
center of theinterior spaceto the outer boundary of
theexteriorspace,andx
i
isthedistancealongthe
x-axisfromthecenterof theinteriorspacetotheouter
boundary of the interior space, shown in (a) of
Fig-ure 5. A quick inspection of this warping equation
reveals itsbehavior. Forapoint onthe boundary of
theinterior space, x = x
i
, and thus x
w =x
i
, sothe
pointdoes notmove. However, pointsoutsideof the
boundarygetwarpedaccordingtotheirproximityto
the boundary of the exterior space. For a point on
the boundary of the exterior space, x = x
e
, and so
x
w =1.
Continuing with the above example, once x
w is
computed, we nd the other coordinate y
w
by
solv-ingalineequation,
y
w
=y+m(x
w
x); (2)
wherem istheslope oftheline connectingthepoint
(x;y)withthepointa,shownin(b)ofFigure5. Point
aislocatedattheintersectionofthelineparalleltothe
Figure2: Pre-warped(a)andwarped(b)voxelspacesshownintwodimensions. In(a),thevoxelspaceisdivided
into two regions; an interior space shown with dark gray voxels, and an exteriorspace shown with light gray
voxels. Both regions consist of voxels of uniform size. The warped voxel space is shown in (b). The warping
doesnotaectthevoxelsintheinteriorspace,whilethevoxelsintheexteriorspaceincreasein sizefurtherfrom
theinteriorspace. Theoutershell ofvoxelsin(b)arewarpedtoinnity,and arerepresentedwitharrowsinthe
gure.
Figure3: Exampleofa2Dvoxelspacethat satisestheconstantfootprintpropertyfortwoimages. Noticethat
the two lled in voxels project to the same number of pixels in the right image, regardlessof theirrespective
distance from the camera. Note thatthis gureis solely used to illustratethe constant footprintproperty; the
(a)
(b)
Figure5: Finding thewarpedpoint. Thex-warpingfunctionisappliedtothex-coordinateofthepoint(x;y),as
the pointislocatedin the+x region. Thisyields thecoordinate x
w
,shownin (a). In(b), theothercoordinate
y
w
is foundbysolvingthelineequationusing thecoordinatex
w
of theinteriorspace.
As shown above,theexteriorspaceisdividedinto
four trapezoidalregionsforthetwo-dimensionalcase.
In three dimensions, this generalizes to six
frustum-shaped regions, x, y, z; hence the term
frus-tum warp. Therearethreewarpingfunctions,namely
thex-warpingfunction asgivenabove,andy-and
z-warpingfunctions, y w = y y e y i y e jyj (3) z w = z z e z i z e jzj ; (4)
In general,theprocedure towarpapointin the
pre-warpedexteriorspaceisasfollows.
1. Determine in which frustum-shaped region the
pointislocated.
2. Applytheappropriatewarpingfunctiontooneof
thecoordinates. Ifthepointistheinx region,
apply the x-warping function, if the point is in
theyregion,applythey-warpingfunction,and
ifthepointisthezregion,applythez-warping
function.
3. Find the other two coordinates by solving line
equationsusingthewarpedcoordinate.
Afterreconstruction,weintendthemodeltobeviewed
fromnearorwithintheinteriorspace. Forsuch
view-points,voxelswill projectto approximatelythesame
footprintin eachimage.
3.2 Other warping functions
Thefrustum warppresentedaboveis nottheonly
possiblewarp. Anywarpthatdoesnotmovetheouter
boundary of the interior space, and warps the outer
boundary of the pre-warped exterior space to
inn-ity,whilesatisfyingthecriteriathatnogapsform
be-tweenvoxels,andthatnovoxelsoverlap,isvalid.
Fur-thermore, itisdesirabletochooseawarpingfunction
thatapproximatestheconstantfootprintpropertyfor
the camerasused in thereconstruction aswellasthe
cameraplacementsduringnewviewsynthesis. An
ex-ample of an alternative warping function is onethat
warps radially with distance from the center of the
reconstruction volume.
4 Implementation Issues
Reconstructing asceneusing awarped
reconstruc-tion volumeposes somenew challenges, describedin
PerhapsthemostdiÆcultchallengeisthat of
hav-ing the cameras embedded inside the reconstruction
volume. Typically, when one uses a standard voxel
coloringalgorithm,thecamerasusedtotakethe
pho-tographsofthesceneareplacedoutsideofthe
recon-struction volume, so that at least two camerashave
visibility of each voxel. The photo-consistency
mea-sure used in voxel coloring algorithms, qualitatively,
determinesifallthecamerasthatcanseeavoxelagree
onitscolor. Thisphoto-consistencyispoorlydened
whenavoxelisvisible fromonlyonecamera.
Since the warped reconstruction volume can
oc-cupyallspace,camerasgetembeddedinsidethevoxel
space,asshownin(a)ofFigure6. Ourreconstruction
algorithminitiallyassumesthatallvoxelsareopaque.
Therefore, cameraviews are obscured,and the
cam-erascannotworktogether to carvethevolume. This
poses a problem, since to be properly dened, the
photo-consistencymeasure requires that at least two
camerashavevisibility ofa voxel. Consequently, the
voxel coloring algorithm cannot proceed, and
termi-nateswithoutremovinganyvoxelsfromthevolume.
Toaddressthis issue, wemust remove(pre-carve)
asectionofthevoxelspacesothatinitially,each
sur-facevoxelisobservedbyatleasttwocameras,
validat-ingthephoto-consistencymeasure,asshownin(b)of
Figure6. Thereare avarietyof possible methods to
achieve this result. A generic method is to have a
useridentify regions of the voxelspace to pre-carve.
Obviously, the pre-carved regions must only consist
of empty space, i.e. not contain any scene surfaces
tobereconstructed. Whileeective,thismethod
pre-cludesafullyautomaticreconstruction.Alternatively,
onecan pre-carvethe volume using a heuristic. For
example, if appropriate, one could require that the
camerashave visibility of the boundarybetween the
interior space and the exterior space. Other
heuris-tics are possible. Once the pre-carving is complete,
weexecuteastandardvoxelcoloringalgorithm using
thewarpedvoxelspace.
4.2 Preventing visible holes in the outer
shell
Due to errors in camera calibration, image noise,
inaccurate color threshold etc., voxel coloring
some-times removesvoxels that should remain in the
vol-ume. Thus, it is possible that voxels on the outer
shell of the voxel space will be deemed inconsistent.
Removingsuchvoxelscanresultinunknownblack
re-gionssimilartoFigure1duringnewviewsynthesis,as
pix-Figure 6: Pre-carvingoperation. Reconstruction inthewarpedspacecausesthecamerasto beembedded inthe
voxelspace,asshownin (a). Formanycameraplacements,itwouldbeimpossibletocarveanyvoxels,sinceno
voxelisvisibletomorethanonecamera. Weexecuteapre-carvingstepin(b)sothatcamerascanworktogether
to carvethevolume.
innity, wedo not carve voxelson theouter shell of
thevoxelspace,independentofthephoto-consistency
measure.
5 Results
We have modied the GVC and GVC-LDI
algo-rithms [2] to utilizethewarpedvoxelspace. We
cre-ated a synthetic data set, called \marbles",
consist-ing of twelve 320 x 240images of ve small texture
mapped spheresinside amuchlargerspheretextured
witharainbow-likeimage. Wereconstructedthescene
usingavoxelspacethatconsistedof48x48x48
vox-els, of which the inner 32 x 32 x 32 were in the
in-terior space and unwarped. The voxel space wasset
upsothatthevesmalltexturemappedsphereswere
reconstructed in the interior space, while the larger
sphere,makingupthebackground,wasreconstructed
intheexteriorwarpedspace. Sampleimagesfromthe
data set are shown in (a) and (b)of Figure 7. A
re-construction was performed using the warped voxel
space. Thereconstructionwas projectedtothe
view-pointsof(a)and(b), yielding(c)and (d). Notethat
thebackgroundenvironmentwasreconstructed using
ourwarpedvoxelspace.
eldofview)photographsofaquadrangleatStanford
University, using aPanoscan 1
digitalcamera. These
photographshadresolutionofabout2502x884pixels.
OnephotographfromthesetisshowninFigure8(a).
Wehavefoundthat when reconstructingan
environ-ment,itispreferabletouselargeeldofviewimages,
asobjects far from the cameras are visible in many
photographs. This achieves a suÆcient sampling of
the scene with fewer photographs. A voxel space of
resolution300x300x 200voxels, ofwhichthe inner
200x 200x 100were interior voxels, was pre-carved
manually by removing part of the voxel space that
containing the cameras. Then, the GVC algorithm
wasusedtoreconstructthescene. Figure8(b)shows
thereconstructedmodelreprojectedtothesame
view-pointasin (a). Notethat objectsfar awayfrom the
cameras,suchasmanyofthebuildingsandtrees,have
beenaccuratelyreconstructed. Newsynthesizedviews
areshownin(c) and(d)ofthegure.
Despitethesuccessesofthisreconstruction,itisnot
perfect. Theskyisveryfarawayfromthecameras(for
practicalpurposes, at innity), and should therefore
be represented with voxels on the outer shell of the
less, cusping [7] occurs, resulting in inaccurate
com-puted geometry,apparentinananimatedsequenceof
new views of the reconstruction. Reconstruction of
outdoorscenesischallenging,assurfacesoftendonot
satisfy the Lambertian assumption. To compensate,
we used a higher consistency threshold [7], also
re-sulting in some inaccurate geometry. On the whole,
though,thereconstructionisreasonablyaccurateand
producesconvincingnewviews 2
.
6 Conclusion
Inthispaperwehaveproposedextensionstovoxel
coloring that permit reconstruction of a scene using
awarpedvoxelspace,in aneorttocomprehensively
reconstruct objectsbothnear and far awayfrom the
camerasused tophotographthescene. Wehave
pre-sented a frustum warp function, which describes a
methodtowarpthevoxelspacetomodelinnite
vol-umeswhilemaintainingtherequirementsthatno
vox-elsoverlapandnogapsformbetweenthewarped
vox-els. Wehavepresented resultsshowingthe ability of
this approach to reconstruct a background
environ-ment,in additiontoaforegroundscene.
7 Future Work
Since voxelscanwarpto points innitelyfarfrom
the camera centers, using z-values (such as in a
z-buer) to establish depth order can be problematic
due to a computer's nite precision. We are
inter-estedinexploringalternatemethods,suchaspainter's
algorithms,todeterminedepthorderofvoxelsduring
reconstruction andrendering.
Acknowledgments
WeareindebtedtoPanoscanforacquisitionofthe
panoramicimagesofStanford'scampus,andtoMark
LivingstonandIrwinSobelforcalibrationofthese
im-ages. Wefurther thankMarkLivingstonfor
develop-ing a viewer that renders a reconstructed scene
us-ing volumetricwarping. We would alsoliketo thank
SteveSeitz andChuckDyerfornumerousdiscussions
regardingvoxelcoloring. Finally,wethank Fred
Kit-sonand RonSchaferfortheircontinued support and
encouragementofthisresearch.
References
[1] J. Blinn and M. Newell, \Texture and
Reec-tiononComputerGeneratedImages,"
Communi-cations of ACM, Vol. 19, No. 10, pp. 542 - 547,
Oct.1976.
2
An animation showing new synthesized views
of our Stanford scene is available online at
Slabaugh, \Generalized Voxel Coloring," Vision
Algorithms Theory and Practice (ICCV 1999
Workshop), Springer-Verlag Lecture Notes in
ComputerScience, toappearin2000.
[3] N.Greene,\EnvironmentMappingandOther
Ap-plicationsofWorldProjections,"IEEE Computer
GraphicsandApplications, pp.21-29,Nov.1986.
[4] M.Kimura, H. Satio, and T. Kanade, \3D Voxel
ConstructionBasedonEpipolarGeometry,"Proc.
ICIP,pp.135-139,1999.
[5] K.KutulakosandS.Seitz,\ATheoryofShapeby
SpaceCarving,"Proc. ICCV,pp.307-314,1999.
[6] H. Saito and T. Kanade, \Shape Reconstruction
in Projective Grid Space from Large Number of
Images,"Proc. CVPR,pp.49-54,1999.
[7] S.SeitzandC.Dyer,\PhotorealisticScene
Recon-struction byVoxelColoring,"International
Jour-nal of Computer Vision, Vol. 35, No. 2, pp.
(c) (d)
Figure7: Originalimagesofthemarblesdatasetareshownin(a)and(b),andareconstructionprojectedtothe
(b)
(c)
(d)
Figure8: ResultsfortheStanfordscene. Oneofthetenpanoramicphotographsisshownin(a). Thereconstructed
