Qualitative study

6.4.1 Associative memories

In past work on HMAX we assumed that the hierarchical architecture performs a kind of preprocessing of an image to provide, as result of the computation, a vector (that we called “signature” here) that is then input to a classifier. This view is extended in this paper by assuming that thesignature vectoris input to an associative memory so that a number of properties of the image (and asso-ciations) can be recalled. Parenthetically we note that oldassociative memories can be regarded as vector-valued classifiers – an obvious observation.

Retrieving from an associative memory: optimal sparse encoding and recallThere are interesting estimates of optimal properties of codes for associative mem-ories, including optimal sparsness (see [49, 54]). It would be interesting to connect these results to estimated capacity of visual memory (Oliva, 2010).

Weak labeling by association of video framesAssume that the top associative module associates together images in a video that are contiguous in time (apart when there are clear transitions). This idea (mentioned to TP by Kai Yu) relies on smoothness in time to label via association. It is a very biological semi-supervised learning, very much in tune with our proposal of the S:C memory-based module for learning invariances to transformations and with the ideas above about an associative memory module at the very top.

Space, time, scale, orientationSpace and time are in a sense intrinsic to images and to their measurement. It seems that the retina is mainly dealing with those three dimensions (x, y, t), thoughx, yare sampled according to the sampling theorem in a way which is eccentricity-dependent forcing in later cortical layers the development of receptive field with a size which increases with eccentricity (spacing in the lattice and scale of receptive fields increase proportionally to

∼logr).

The theory assumes that at each eccentricity a set of receptive fields of dif-ferent size (egσ) exist during development at the level of developing simple cells, originating a set ofscales. It is an open question what drove evolution to discover multiresolution analysis of the image. Given finite channel resources – eg bandwidth, number of fibers, number of bits – there is a tradeoff between size of the visual field and scale (defined as the resolution in terms of spa-tial frequency cutoff). Once multiple scales are superimposed on space (eg a lattice of ganglion cells in eachx, y) by a developmental program, our theory describes how the orientation dimension is necessarily discovered by exposure to moving images.

6.4.2 Visual abstractions

• Concept of parallel linesConsider an architecture using signatures. Assume it has learned sets of templates that guarantee invariance to all affine transformations. The claim is thatthe architecture will abstract the concept of parallel lines from a single specific example of two parallel lines. The argument is that according to the theorems in the paper, the signature of the single image of the parallel lines will be invariant to any affine transformations.

• Number of items in an imageA classifier which learns the number five in a way which is invariant to scale should be able to recognize five objects independent of class of objects.

• Line drawings conjectureThe memory-based module described in this pa-per should be able to generalize from real images to line drawings when exposed to illumination-dependent transformations of images. This may need to happen at more than one level in the system, starting with the very first layer (eg V1). Generalizations with respect to recognition of objects invariant to shadows may also be possible.

(A)

Figure 45: For a system which is invariant to affine transformations a single training example (A) allows recognition of all other instances of parallel lines – never seen be-fore.

. 6.4.3 Invariance and Perception

Other invariances in visual perception may be analyzed in a parallel way. An example is color constancy. Invariance to illumination (and color opponent cells) may emerge during development in a similar way as invariance to affine transformations.

The idea that the key computational goal of visual cortex is to learn and ex-ploit invariances extends to other sensory modalities such as hearing of sounds and of speech. It is tempting to think of music as an abstraction (in the sense of information compression a’ la PCA) of the transformations of sounds. Clas-sical (western) music would then emerge from the transformations of human speech (the roots of western classical music were based in human voice – Gre-gorian chants).

6.4.4 The dorsal stream

The ventral and the dorsal streams are often portrayed asthe what and the where facets of visual recognition. It is natural to ask what the theory described here implies for the dorsal stream.

In a sense the dorsal stream seems to be the dual of the ventral stream:

instead of being concerned about the invariances under the transformation in-duced by a Lie algebra it seems to represent (especially in MST) the orbits of the dynamical systems corresponding to the same Lie algebra.

6.4.5 Is the ventral stream a cortical mirror of the invariances of the physical world?

It is somewhat intriguing that Gabor frames - related to the “coherent” states and thesqueezed statesof quantum mechanics - emerge from the filtering op-erations of the retina which are themselves a mirror image of the position and momentum operator in a Gaussian potential. It is even more intriguing that in-variances to the groupSO2×R²dictate, according to our theory, the computa-tional goals, the hierarchical organization and the tuning properties of neurons in visual areas. In other words: it did not escape our attention that the theory described here implies that the brain function, structure and properties reflect in a surprising direct way the physics of the visual world.

AcknowledgmentsWe would like to especially thank Steve Smale, Leyla Isik, Owen Lewis, Steve Voinea, Alan Yuille, Stephane Mallat, Mahadevan, S. Ullman for discussions leading to this preprint and S. Soatto, J. Cowan, W. Freiwald, D. Tsao, A.

Shashua, L. Wolf for reading versions of it. Andreas Maurer contributed the argu-ment about small apertures in section 4.1.1. Giacomo Spigler, Heejung Kim, and Dar-rel Deo contributed several results including simulations. Krista Ehinger and Aude Oliva provided to J.L. the images of Figure 3 and we are grateful to them to make them available prior to publication. In recent years many collaborators contributed indirectly but considerably to the ideas described here: S. Ullman, H. Jhuang, C. Tan, N. Edelman, E. Meyers, B. Desimone, T. Serre, S. Chikkerur, A. Wibisono, J. Bouvrie, M. Kouh, M. Riesenhuber, J. DiCarlo, E. Miller, A. Oliva, C. Koch, A. Caponnetto, C.

Cadieu, U. Knoblich, T. Masquelier, S. Bileschi, L. Wolf, E. Connor, D. Ferster, I. Lampl, S. Chikkerur, G. Kreiman, N. Logothetis. This report describes research done at the Center for Biological and Computational Learning, which is in the McGovern Institute for Brain Research at MIT, as well as in the Dept. of Brain and Cognitive Sciences, and which is affiliated with the Computer Sciences and Artificial Intelligence Labora-tory (CSAIL). This research was sponsored by grants from DARPA (IPTO and DSO), National Science Foundation (NSF-0640097, NSF-0827427), AFSOR-THRL (FA8650-05-C-7262). Additional support was provided by: Adobe, Honda Research Institute USA, King Abdullah University Science and Technology grant to B. DeVore, NEC, Sony and especially by the Eugene McDermott Foundation.