6. Conclusion and Future Work
6.2 Future Work
Wide baseline stereo is a new research area that attracts much attention from the computer vision community in the last decade or so. There are many possible ways to extend the ideas and methods proposed in this research.
For feature points matching or sparse matching, we may want to enhance the proposed method so that it can be used to detect more matches in difficult situations. A possible way is to employ local image normalization. If an image can be segmented into several parts of continual depth changes, we can apply different affine transformations on different areas of interest, which will make the algorithm more adaptive and thus combine the advantage of global topological analysis and local geometric analysis.
For the projective rectification part, the topology of the triangle net can be optimized by appropriately selection of the distribution of the feature points. The experiments show that the results of rectification are affected by the topology of the triangle net. If we can select the control points before the construction of the triangle net, the performance of rectification may be further improved.
Dense matching may be enhanced with the introduction of local descriptor matching rather than just the matching of intensity value. This will remove some ambiguity and give more accurate details in the disparity map.
As a whole, the system can be deployed in more real world wide baseline applications such as three-dimensional wide area surveillance for further test and enhanced.
The disparity maps can be used to generate 3D model of the scene if calibration information can be obtained. This is a very interesting topic for further study.
[Agr-03] M. Agrawal and L. Davis. “Camera calibration using spheres: A dual-space approach”, Research Report CAR-TR-984, Center for Automation Research, University of Maryland, 2003.
[Alo-90] J. Y. Aloimonos. “Perspective approximations”. Image and Vision Computing, 8(3):177–192, 1990.
[Al-S 00] K. A. Al-Shalfan, J. G. B. Haigh, S. S. Ipson, “Direct algorithm for rectifying pairs of uncalibrated images”, Electronics Letters 36 (5):419–420,2000.
[Ana-02] P. Anandan and M. Irani. “Factorization with uncertainty”. International Journal
of Computer Vision, 49(2/3):101–116, 2002.
[Arm-96] M. Armstrong. “Self-Calibration from Image Sequences”. PhD thesis, University of Oxford, England, 1996.
[Ast-98] K. Astrom and A. Heyden. “Continuous time matching constraints for image streams”. International Journal of Computer Vision, 28(1):85–96, 1998.
[Avi-98] S. Avidan and A. Shashua. “Threading fundamental matrices”. In Proc. 5th
European Conference on Computer Vision, pp. 124–140, 1998.
[Bai-99] C. Baillard and A. Zisserman. “Automatic reconstruction of piecewise planar models from multiple views”. In Proc. IEEE Conference on Computer Vision and Pattern
Recognition, pp. 559–565, 1999.
[Bar-92] E. B. Barrett, M. H. Brill, N. N. Haag, and P. M. Payton. “Invariant linear methods in photogrammetry and model-matching”. In J. L. Mundy and A. Zisserman, editors, Geometric invariance in computer vision. MIT Press, Cambridge, 1992.
[Bas-98] B. Bascle and A. Blake. “Separability of pose and expression in facial tracing and animation”. In Proc. International Conference on Computer Vision, pp. 323–328, 1998.
[Bas-99] R. Basri and D. Jacobs. “Projective alignment with regions”. In Proc. 7th
International Conference on Computer Vision, pp. 1158–1164, 1999.
[Bat-76] K-J. Bathe and E.Wilson. “Numerical methods in finite element analysis”. Prentice Hall, 1976.
[Bea-78] P. Beaudet. “Rotationally invariant image operators”. Proc.4th Int. Joint
Conference on Pattern Recognition, pp. 579-583.1978
[Bea-92] P. A. Beardsley, D. Sinclair, and A. Zisserman. “Ego-motion from six points”. Insight meeting, Catholic University Leuven, February 1992.
[Bea-94] P. A. Beardsley, A. Zisserman, and D. W. Murray. “Navigation using affine structure and motion”. In Proc. European Conference on Computer Vision, LNCS 800/801, pp.85–96. Springer-Verlag, 1994.
[Ber-08] M. de Berg, O. Cheong, M. van Kreveld and M. Overmars, “Computational Geometry: Algorithms and Applications”, Springer-Verlag, 3rd edition, 2008.
[Bar-82] S. T. Barnard and M. A. Fischler, “Computational Stereo”, ACM Computing
Surveys, 42(14):553-572.1982.
[Bay-08] H. Bay, A. Ess, T. Tuytelaars, L. V. Gool, "SURF: Speeded Up Robust Features",
Computer Vision and Image Understanding, 110(3):346-359. 2008
[Bir-98] S. Birchfield. “An Introduction to Projective Geometry (for Computer Vision)”. Stanford University, March 1998.
[Boy-01] Y. Boykov, O. Veksler and R. Zabih, "Fast approximate energy minimization via graph cuts". IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11):1222–1239. 2001.
[Bro-91] D.C. Brown, “Close-Range Camera Calibration”, Photogrammetric Engineering, 37(8):855-866, 1991.
[Bro-03] M.Z. Brown, D. Burschka, G.D. Hager, “Advances in Computational Stereo”.
IEEE Transaction PAMI, 25(8):993-1008, 2003.
[Bro-05] M. Brown, R. Szeliski, and S. Winder. “Multi-image matching using multi-scale oriented patches”. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 510–517, San Diego, CA, 2005.
[Bru-03]V. Bruce, P. R. Green, M. A. Georgeson. “Visual Perception: Physiology, Psychology and Ecology”, Psychology Press, 2003.
[Cap-90]B. Caprile and V. Torre, “Using Vanishing Points for Camera Calibration”,
International Journal of Computer Vision, 4( 2):127-140, 1990.
[Col-96] R. T. Collins. “A space-sweep approach to true multi-image matching”. In Proc.
IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.
358–363, San Francisco,CA, 1996.
[Dan-99] K. J. Dana et al. “Reflectance and texture of real world surfaces”. ACM
Transactions on Graphics, 18(1):1–34, 1999.
[Deb-99] Debevec, P. E.. “Image-based modeling and lighting”, Computer Graphics, 33(4):46–50, 1999.
[Dho-89] U. R. Dhond, J. K. Aggarwal. “Structure from stereo—A review”. IEEE
Transactions on Systems, Man, and Cybernetics,19(6):1489–1510, 1989.
[Dre-82] L.S. Dreschler and H.-H. Nagel, “On the selection of critical points and local curvature extrema of region boundaries for interframe matching”, In Proc. Inter. Conf. on
Pattern Recognition, pp. 542–544, 1982.
[Eld-01] J. H. Elder, J. H. and R. M. Golderg. “Image editing in the contour domain”. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 23(3):291–296, 2001.
[ Fai-75] W. Faig, “Calibration of Close-Range Photogrammetry Systems: Mathematical Formulation”, Photogrammetric Engineering and Remote Sensing, 41(12):1,479-1,486, 1975.
[Fau-92] O. Faugeras. “What can be seen in three dimensions with an uncalibrated stereo rig?”, ECCV 1992, Springer Verlag, LNCS 588:563–578, 1992.
[Fau-93] O. Faugeras. “Three-Dimensional Computer Vision: A Geometric Viewpoint”. MIT Press, 1993.
[Fau-95] O. Faugeras. “Stratification of 3-D vision: projective, affine, and metric representations”. Journal of the Optical Society of America,12(3):465–484,1995.
[Fau-01] O. Faugeras, Q. Luong. “The Geometry of Multiple Images”. The MIT Press, 2001.
[Fol-95] J. D. Foley, van Dam, A., Feiner and J. F. Hughes. “Computer Graphics: Principles and Practice”. Addison-Wesley, Reading, MA, 2 edition.1995.
[Fro-96] T. Frohlinghaus and J. M. Buhmann. "Regularizing phase-based stereo". In Proc.
Inter. Conf. on Pattern Recognition, Vol. A, pp. 451–455.1996.
[Fus-00] A. Fusiello, E. Trucco, and A. Verri, "A compact algorithm for rectification of stereo pairs," Machine Vision and Applications, 12(1):16–22, 2000.
[Fus-08] A. Fusiello and L. Irsara, “Quasi-Euclidean Uncalibrated Epipolar Rectification”, In Proc. Of the IEEE Inter. Conf. Pattern Recognition, pp 1-4, 2008.
[Glu-01] J. Gluckman and S. K. Nayar, "Rectifying transformations that minimize resampling effects," Proc. of the IEEE Conference on CVPR, pp. I:111-117, 2001.
[Gol-96] G. Golub and C. van Loan, Matrix computations, 3rd edition, The Johns Hopkins University Press. 1996.
[Gon-07] M. Gong et al , "A performance study on different cost aggregation approaches used in realtime stereo matching". International Journal of Computer Vision, 75(2):283– 296.2007.
[Har-88] C. Harris and M. Stephens. "A combined corner and edge detector". Proceedings
of the 4th Alvey Vision Conference. pp. 147–151.1988.
[Har-99a] R. Hartley, “Theory and practice of projective rectification”, International
Journal of Computer Vision, 35 (2):115–127, 1999.
[Har-99b] R. Hartley. “Camera calibration and the search for infinity”. in Proceedings of
the 7th International Conference on Computer Vision, pp. 510-517, Greece, September
[Har-04] R. Hartley and A. Zisserman, “Multiple View Geometry in Computer Vision”. Cambridge University Press, 2nd edition, 2004.
[Hir-09] H. Hirschmuller and D. Scharstein,"Evaluation of stereo matching costs on images with radiometric differences", IEEE Transactions on Pattern Analysis and Machine
Intelligence, 31(9):1582–1599,2009.
[Int-03] Matlab Camera Calibration Toolbox included in Intel OpenCV library.
http://www.intel.com/reseatch/mrl/research/opencv
[Isg-99] F. Isgro, E. Trucco, “On projective rectification”, in Proceedings IEE Conference
on Image Processing and Analysis, pp. 42–46,1999.
[Jur-04] F. Jurie and C. Schmid, "Scale-Invariant Shape Features for Recognition of Object Categories", in Proc. CVPR (2), pp.90-96, 2004.
[Kad-04] T. Kadir, A. Zisserman, and M. Brady, “An affine invariant salient region detector”. In ECCV04, pp. 404-416, 2004.
[Kan-07] J. Kannala and S.S. Brandt, "Quasi-dense wide baseline matching using match propagation", in Proc. of IEEE Conference on Computer Vision and Pattern Recognition ,pp.1-8, 2007.
[Koe-84] J. Koenderink, "The structure of images", Biological Cybernetics, 50:363-370, 1984.
[Kol-06] V. Kolmogorov, et al. "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation". IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9):1480–1492, 2006.
[Kos-93] A. Koschan, “What is New in Computational Stereo Since 1989: A Survey of Current Stereo Papers”, Technical Report 93-22, Technical University of Berlin, 1993.
[Kos-08] A. Koschan and M. Abidi, "Digital Color Image Processing," Wiley, Hoboken, New Jersey, April 2008.
[Lei-04] Bastian Leibe and Bernt Schiele. "Scale-Invariant Object Categorization using a Scale-Adaptive Mean-Shift Search" in DAGM’04 Pattern Recognition Symposium, Tubingen, Germany, Aug 2004.
[Lie-98] D. Liebowitz and A. Zisserman. “Metric Rectification for Perspective Images of Planes”. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 482–488.1998.
[Lin-93] Tony Lindeberg. “Detecting salient blob-like image structures and their scales with a scale-space primal sketch: A method for focus-of-attention”, International Journal
of Computer Vision, 11(3):283-318, 1993.
[Lin-98] Tony Lindeberg. "Feature detection with automatic scale selection". International
Journal of Computer Vision, 30 (2): 77-116.1998.
[Liv-08] M. Livingstone. “Vision and Art: The Biology of Seeing”. Abrams, New York, 2008
[Loo 99] C. Loop, Z. Zhang, “Computing rectifying homographies for stereo vision”, in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol.1
pp. 125–131.1999.
[Low-04] D. G. Lowe, "Distinctive image features from scale-invariant keypoints,"
[Luc-81] Bruce D. Lucas and Takeo Kanade. “An Iterative Image Registration Technique with an Application to Stereo Vision”. International Joint Conference on Artificial
Intelligence, pp. 674–679, 1981.
[Luo-96] Q-T Luong, O. D. Faugeras, "The fundamental matrix: Theory, algorithms, and stability analysis," International Journal of Computer Vision, 17(1):43-75, 1996
[Luo-97] Q.T. Luong and O. Faugeras. “Self-calibration of a moving camera form point correspondences and fundamental matrices”. International Journal of Computer Vision 22(3), pp.261-289,1997.
[Mar-82] D. Marr. “Vision-A computational Investigation into the Human Representation and Processing of Visual Information”. Freeman, San Francisco, 1982.
[Mal-05] J. Mallon and P. F. Whelan, "Projective rectification from the fundamental matrix," Image and Vision Computing, 23(7): 643-650, 2005.
[Mat-02] J.Matas, O. Chum, M. Urban, and T. Pajdla, “Robust wide baseline stereo from maximally stable extremal regions”. In British Machine Vison Conference. pp.384-393, 2002.
[Mat-04] J. Matas et al.. “Robust wide baseline stereo from maximally stable extremal regions”. Image and Vision Computing 22(10):761–767.2004.
[Mid-03] MiddleBury Stereo Vision Research Page. http://cat.middlebury.edu/stereo/ retrieved since 2003.
[Mik-04] K. Mikolajczyk and C. Schmid, “Scale and Affine invariant interest point detectors”. International Journal of Computer Vision 60(1):63-86, 2004.
[Mik-05a] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir and L. Van Gool, “A comparison of affine region detectors”.
International Journal of Computer Vision 65(1/2):43-72, 2005.
[Mik-05b] K. Mikolajczyk, C. Schmid, “A performance evaluation of local descriptors”. In
IEEE Transaction on PAMI 27(10):1615-1630, 2006.
[Moh-96] R. Mohr and B. Triggs. “Projective Geometry for Image Analysis”. In
International Symposium of Photogrammetry and Remote Sensing, Vienna, July 1996.
[Ng-02] K. C. Ng, M. Trivedi, H. Ishiguro, “Generalized multiple baseline stereo and direct virtual view synthesis using range-space search, match, and render”, International Journal
of Computer Vision 47 (1-3):131–147, 2002.
[Pal-99] Stephen E. Palmer. “Vision Science - Photons to Phenomenology”. MIT Press, Cambridge, MA. 1999
[Pap-06] D. V. Papadimitriou, T. J. Dennis, “Epipolar line estimation and rectification for stereo image pairs”, IEEE transactions on image processing, 15 (4):672–676, 2006.
[Pil-97] M. Pilu, “A Direct Method for Stereo Correspondence Based on Singular Value Decomposition,” In Proc. IEEE Computer Vision and Pattern Recognition Conf., pp. 261- 266, 1997.
[Pol-99a] M. Pollefeys, “Self-Calibration and Metric 3D Reconstruction from Uncalibrated ImageSequences”. PhD thesis, ESAT-PSI, K.U. Leuven, 1999.
[Pol-99b] M. Pollefeys, R. Koch, L. V. Gool, “A simple and efficient rectification method for general motion”, In Proceedings of the International Conference on Computer Vision, Vol. 1, pp. 496–501,1999.
[Pol-08] M. Pollefeys, D. Nister, et al, “Detailed Real-Time Urban 3D Reconstruction From Video”, International Journal of Computer Vision 78(2):143- 167, 2008.
[Sai-99] H. Saito and T. Kanade. “Shape reconstruction in projective grid space from large number of images”. In Proc. of IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, pp. 49–54,1999.
[Sch-02a] F. Schaffalitzky, A. Zisserman. “Multi-view Matching for Unordered Image Sets or How Do I Orangnize My Holiday Snaps?”, In Proceedings of the 7th European
Conference on Computer Vision-Part I, 2002.
[Sch-02b] D. Scharstein and R. Szeliski, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms," International Journal of Computer Vision 47(1-3):7-42, 2002.
[Sch-00] C. Schmid, R. Mohr and C. Bauckhage. “Evaluation of Interest Point Detectors”.
International Journal of Computer Vision 37(2):151–172, 2000.
[Sco-91] G. Scott and H. Longuet-Higgins. An algorithm for associating the features of two patterns. In Proc. Royal Society London, vol. B244:21-26, 1991.
[Sei-96] S. Seitz, C. Dyer. “View Morphing”, SIGGRAPH, 1996.
[Sei-97] S. M. Seitz, “Image-Based Transformation of Viewpoint and Scene Appearance”. Ph.D. Dissertation, Computer Sciences Department Technical Report 1354, University of Wisconsin - Madison, October 1997.
[Sem-79] J.G. Semple and G.T Kneebone. “Algebraic Projective Geometry”. Oxford University Press, 1979.
[Shi-94] J. Shi. and C. Tomasi. “Good features to track”. In Proceedings of the Conference
on Computer Vision and Pattern Recognition, pp. 593–600.1994.
[Sin-08] S.N. Sinha, D. Steedly, R. Szeliski, M. Agrawala, and M. Pollefeys. “Interactive 3D architectural modeling from unordered photo collections”. ACM Transactions on
Graphics, 27(5),2008.
[Str-03] C. Strecha, T. Tuytelaars, L. V. Gool, “Dense Matching of Multiple Wide-baseline Views”, In Proceedings of the IEEE International Conference on Computer Vision, pp. 1194-120, 2003.
[Sun-03] J. Sun, N. Zheng and H. Shum, "Stereo matching using belief propagation".
IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7):787–800. 2003.
[Sze-99] R. Szeliski and P. Golland. “Stereo matching with transparency and matting”.
International Journal of Computer Vision, 32(1):45–61. 1999.
[Sze-11] R. Szeliski, “Computer Vision: Algorithms and Applications”, Springer, 2011.
[Tol-10] E. Tola, V. Lepetit, P. Fua. "daisy: an efficient dense descriptor applied to wide baseline stereo", IEEE Trans. on Pattern analysis and MachineIintelligence 32(5):815-830, 2010.
[Tom-91] C. Tomasi and T. Kanade. “Detection and Tracking of Point Features”. Carnegie Mellon University Technical Report CMU-CS-91-132, April 1991.
[Tom-08] F. Tombari et al., "Classification and evaluation of cost aggregation methods for stereo correspondence". In Proc. of IEEE Computer Society Conference on Computer
[Tri-95] B.Triggs, “The geometry of projective reconstruction I: Matching constraints and the joint image”, in Proceedings of the 5th International Conference on Computer Vision,
pp.338-343, Boston, MA, June 1995.
[Tri-98] B.Triggs, “Autocalibration from Planar Scenes”, Proc. Fifth European Conf.
Computer Vision, pp. 89-105, June 1998.
[Tsa-86] R. Y. Tsai “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision”, in Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition, pp. 364-374, 1986.
[Tru-98] E. Trucco and A. Verri, “Introductory to techniques for 3D Computer Vision”. Prentice Hall, 1998.
[Tuy-04] T.Tuytelaars and L. Van Gool, “Matching widely separated views based on affine invariant regions” . In International Journal of Computer Vision 59(1):61-85, 2004.
[Tuy-08] Tuytelaars and K. Mikolajczyk , “Local Invariant Feature Detectors – Survey”. In CVG, 3(1):1-110, 2008.
[Vai-06] V. Vaish et al.,"Reconstructing occluded surfaces using synthetic apertures: Shape from focus vs. shape from stereo".In Proc. IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pp. 2331–2338, 2006.
[Vla-09] Daniel Vlasic, et al. “Dynamic Shape Capture using Multi-View Photometric Stereo”. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 28(5), 2009.
[Wit-84] A. Witkin, "Scale-space filtering: A new approach to multi-scale description," in
[Wu-05] H. P. Wu and Y. Yu, "Projective rectification with reduced geometric distortion for stereo vision and stereoscopic video", Journal of Intelligent and Robotic Systems, 42(1): 71 – 94, 2005.
[Zha-98a] Z. Zhang. “Determining the Epipolar Geometry and its Uncertainty: A Review”,
The International Journal of Computer Vision, 27(2):161–195, 1998.
[Zha-98b] Z. Zhang. “A Flexible New Technique for Camera Calibration”. Technical Report MSRTR-98-71, Microsoft Research, December 1998.
[Zhe-07] K. C. Zheng, S. B. Kang, M. Cohen and R. Szeliski. “Layered depth panoramas”. In Proc. of IEEE Computer Society Conference on Computer Vision and Pattern
Vita
Wei Hao was born in Hubei province, China. He attended Tianjin Polytechnic University where he majored in Electrical Engineering and received a Bachelor of Science degree in 1997. After that, He continued to pursue his Master of Science degree in Tianjin University with a major in Pattern recognition and Intelligent System. After year 2000, he worked as a software engineer in the telecommunication and Internet industries. He came to University of Tennessee in fall 2003 and joined the Imaging, Robotics, and Intelligent Systems Laboratory afterward, where he completed his Doctor of Philosophy degree in 2011.