In Figure 17, we compared images which are considered likely by the different models. In Figure 22, we show samples that we drew from the models using Markov chains (Hamiltonian Monte Carlo). Since the models are defined on a sphere, we constrained the Hamilitonian dynamics by projecting the states after each leapfrog step back onto the sphere. The number of leapfrog steps was set to 100, and the rejection rate to 0.35 (Neal, 2010, Section 4.4, p.30). The top row shows the most likely samples while the bottom row show the least likely ones. The least likely samples appear similar for all models. For the more probable ones, however, the two-layer models lead to more structured samples than the one-layer models.
Laplacian
Thresholding Spline
(a) One-layer models
Refinement Spline Thresholding
(b) Two-layer models
Figure 22: Sampling from the learned models of natural images. Figure (a) shows samples from the one-layer models, Figure (b) shows samples from the two-layer models. The samples are sorted so that the top ones are the most likely ones while those at the bottom are the least probable ones. See caption of Table 1 in Section 5.5 for information on the models used. Samples of the training data and the noise are shown in Figure 9 in Section 5.1.
References
C. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
C.J. Geyer. On the convergence of Monte Carlo maximum likelihood calculations. Journal of the Royal Statistical Society, Series B (Methodological), 56(1):261–274, 1994.
M. Gutmann and A. Hyv¨arinen. Learning features by contrasting natural images with noise. In Pro- ceedings of the 19th International Conference on Artificial Neural Networks (ICANN), volume 5769 of Lecture Notes in Computer Science, pages 623–632. Springer Berlin / Heidelberg, 2009. M. Gutmann and A. Hyv¨arinen. Noise-contrastive estimation: A new estimation principle for un- normalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR W&CP, pages 297–304, 2010.
T. Hastie, R. Tibshirani, and J.H. Friedman. The Elements of Statistical Learning. Springer, 2009. G. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation,
14(8):1771–1800, 2002.
A. Hyv¨arinen. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, 6:695–709, 2005.
A. Hyv¨arinen. Optimal approximation of signal priors. Neural Computation, 20:3087–3110, 2008. A. Hyv¨arinen, P.O. Hoyer, and M. Inki. Topographic independent component analysis. Neural
Computation, 13(7):1527–1558, 2001a.
A. Hyv¨arinen, J. Karhunen, and E. Oja. Independent Component Analysis. Wiley-Interscience, 2001b.
A. Hyv¨arinen, J. Hurri, and P.O. Hoyer. Natural Image Statistics. Springer, 2009.
Y. Karklin and M. Lewicki. A hierarchical Bayesian model for learning nonlinear statistical regu- larities in nonstationary natural signals. Neural Computation, 17:397–423, 2005.
D. Koller and N. Friedman. Probabilistic Graphical Models. MIT Press, 2009.
U. K¨oster and A. Hyv¨arinen. A two-layer model of natural stimuli estimated with score matching. Neural Computation, 22(9):2308–2333, 2010.
J. L¨ucke and M. Sahani. Maximal causes for non-linear component extraction. Journal of Machine Learning Research, 9:1227–1267, 2008.
R.M. Neal. Handbook of Markov Chain Monte Carlo, chapter MCMC using Hamiltonian Dynam- ics. Chapman & Hall /CRC Press, 2010.
B.A. Olshausen and D.J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996.
S. Osindero and G. Hinton. Modeling image patches with a directed hierarchy of Markov random fields. In Advances in Neural Information Processing Systems 20, pages 1121–1128. MIT Press, 2008.
S. Osindero, M. Welling, and G. E. Hinton. Topographic product models applied to natural scene statistics. Neural Computation, 18 (2):381–414, 2006.
M. Pihlaja, M. Gutmann, and A. Hyv¨arinen. A family of computationally efficient and simple esti- mators for unnormalized statistical models. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI), pages 442–449. AUAI Press, 2010.
M.A. Ranzato and G. Hinton. Modeling pixel means and covariances using factorized third-order Boltzmann machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2551–2558, 2010.
C.E. Rasmussen. Conjugate gradient algorithm, Matlab code version 2006-09-08. Downloaded from http://learning.eng.cam.ac.uk/carl/code/minimize/minimize.m. 2006.
C.P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer, 2nd edition, 2004.
N.N. Schraudolph and T. Graepel. Towards stochastic conjugate gradient methods. In Proceed- ings of the 9th International Conference on Neural Information Processing (ICONIP), volume 2, pages 853–856, 2002.
W. Sun and Y. Yuan. Optimization Theory and Methods: Nonlinear Programming. Springer, 2006. Y. Teh, M. Welling, S. Osindero, and G. Hinton. Energy-based models for sparse overcomplete
representations. Journal of Machine Learning Research, 4:1235–1260, 2004.
T. Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gra- dient. In Proceedings of the 25th International Conference on Machine Learning, pages 1064– 1071, 2008.
J. H. van Hateren and A. van der Schaaf. Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London. Series B: Biological Sciences, 265(1394):359–366, 1998.
Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004. L. Wasserman. All of Statistics. Springer, 2004.
L. Younes. Parametric inference for imperfectly observed Gibbsian fields. Probability Theory and Related Fields, 82(4):625–645, 1989.