The temporal continuity typical of objects has been used in an associative learning rule with a short-term memory trace to help build invariant object representations in the networks described previously in this paper.Stringer et al. (2006)showed that spa- tial continuity can also provide a basis for helping a system to self-organize invariant representations. They introduced a new learning paradigm“continuous spatial transformation (CT) learn- ing” which operates by mapping spatially similar input patterns to the same post-synaptic neurons in a competitive learning sys- tem. As the inputs move through the space of possible continuous transforms (e.g., translation, rotation, etc.), the active synapses are modified onto the set of post-synaptic neurons. Because other transforms of the same stimulus overlap with previously learned
exemplars, a common set of post-synaptic neurons is activated by the new transforms, and learning of the new active inputs onto the same post-synaptic neurons is facilitated.
The concept is illustrated inFigure 36. During the presenta- tion of a visual image at one position on the retina that activates neurons in layer 1, a small winning set of neurons in layer 2 will modify (through associative learning) their afferent connections from layer 1 to respond well to that image in that location. When the same image appears later at nearby locations, so that there is spatial continuity, the same neurons in layer 2 will be activated because some of the active afferents are the same as when the image was in the first position. The key point is that if these afferent con- nections have been strengthened sufficiently while the image is in the first location, then these connections will be able to continue to activate the same neurons in layer 2 when the image appears in overlapping nearby locations. Thus the same neurons in the output layer have learned to respond to inputs that have similar vector elements in common.
As can be seen inFigure 36, the process can be continued for subsequent shifts, provided that a sufficient proportion of input cells stay active between individual shifts. This whole process is repeated throughout the network, both horizontally as the image moves on the retina, and hierarchically up through the network. Over a series of stages, transform invariant (e.g., location invari- ant) representations of images are successfully learned, allowing the network to perform invariant object recognition. A similar CT learning process may operate for other kinds of transformation, such as change in view or size.
Stringer et al. (2006)demonstrated that VisNet can be trained with continuous spatial transformation learning to form view- invariant representations. They showed that CT learning requires the training transforms to be relatively close together spatially so that spatial continuity is present in the training set; and that the order of stimulus presentation is not crucial, with even inter- leaving with other objects possible during training, because it is spatial continuity rather the temporal continuity that drives the self-organizing learning with the purely associative synaptic modification rule.
Perry et al. (2006)extended these simulations with VisNet of view-invariant learning using CT to more complex 3D objects, and using the same training images in human psychophysical investiga- tions, showed that view-invariant object learning can occur when spatial but not temporal continuity applies in a training condition in which the images of different objects were interleaved. How- ever, they also found that the human view-invariance learning was better if sequential presentation of the images of an object was used, indicating that temporal continuity is an important factor in human invariance learning.
Perry et al. (2010)extended the use of continuous spatial trans- formation learning to translation invariance. They showed that translation-invariant representations can be learned by continu- ous spatial transformation learning; that the transforms must be close for this to occur; that the temporal order of presentation of each transformed image during training is not crucial for learn- ing to occur; that relatively large numbers of transforms can be learned; and that such continuous spatial transformation learning can be usefully combined with temporal trace training.
FIGURE 36 | An illustration of how continuous spatial transformation (CT) learning would function in a network with a single-layer of forward synaptic connections between an input layer of neurons and an output layer.Initially the forward synaptic weights are set to random values. The top part(A)shows the initial presentation of a stimulus to the network in position 1. Activation from the (shaded) active input cells is transmitted through the initially random forward connections to stimulate the cells in the output layer. The shaded cell in the output layer wins the competition in that layer. The weights from the active input cells to the active output neuron are then strengthened using an associative learning rule. The bottom part(B)shows what happens after the stimulus is shifted by a small amount to a new partially overlapping position 2. As some of the active input cells are the same as those that were active when the stimulus was presented in position 1, the same output cell is driven by these previously strengthened afferents to win the competition again. The rightmost shaded input cell activated by the stimulus in position 2, which was inactive when the stimulus was in position 1, now has its connection to the active output cell strengthened (denoted by the dashed line). Thus the same neuron in the output layer has learned to respond to the two input patterns that have similar vector elements in common. As can be seen, the process can be continued for subsequent shifts, provided that a sufficient proportion of input cells stay active between individual shifts. (AfterStringer et al., 2006.)
5.11. LIGHTING INVARIANCE
Object recognition should occur correctly even despite variations of lighting. In an investigation of this,Rolls and Stringer (2006) trained VisNet on a set of 3D objects generated with OpenGL in which the viewing angle and lighting source could be indepen- dently varied (seeFigure 37). After training with the trace rule on all the 180 views (separated by 1˚, and rotated about the ver- tical axis inFigure 37) of each of the four objects under the left lighting condition, we tested whether the network would recog- nize the objects correctly when they were shown again, but with the source of the lighting moved to the right so that the objects appeared different (see Figure 37). With this protocol, lighting invariant object recognition by VisNet was demonstrated (Rolls and Stringer, 2006).
FIGURE 37 | Lighting invariance.VisNet was trained on a set of 3D objects (cube, tetrahedron, octahedron, and torus) generated with OpenGL in which for training the objects had left lighting, and for testing the objects had right
lighting. Just one view of each object is shown in the Figure, but for training and testing 180 views of each object separated by 1˚ were used. (AfterRolls and Stringer, 2006.)
Some insight into the good performance with a change of light- ing is that some neurons in the inferior temporal visual cortex respond to the outlines of 3D objects (Vogels and Biederman, 2002), and these outlines will be relatively consistent across light- ing variations. Although the features about the object represented in VisNet will include more than the representations of the out- lines, the network may because it uses distributed representations of each object generalize correctly provided that some of the fea- tures are similar to those present during training. Under very difficult lighting conditions, it is likely that the performance of the network could be improved by including variations in the lighting during training, so that the trace rule could help to build representations that are explicitly invariant with respect to lighting.
5.12. INVARIANT GLOBAL MOTION IN THE DORSAL VISUAL SYSTEM
A key issue in understanding the cortical mechanisms that under- lie motion perception is how we perceive the motion of objects such as a rotating wheel invariantly with respect to position on the retina, and size. For example, we perceive the wheel shown in
Figure 38Arotating clockwise independently of its position on the retina. This occurs even though the local motion for the wheels in the different positions may be opposite. How could this invariance of the visual motion perception of objects arise in the visual sys- tem? Invariant motion representations are known to be developed in the cortical dorsal visual system. Motion-sensitive neurons in V1 have small receptive fields (in the range 1–2˚ at the fovea), and can therefore not detect global motion, and this is part of the aper- ture problem (Wurtz and Kandel, 2000b). Neurons in MT, which receives inputs from V1 and V2, have larger receptive fields (e.g., 5˚ at the fovea), and are able to respond to planar global motion, such as a field of small dots in which the majority (in practice as few as 55%) move in one direction, or to the overall direction of a moving plaid, the orthogonal grating components of which have motion at 45˚ to the overall motion (Movshon et al., 1985; Newsome et al., 1989). Further on in the dorsal visual system, some neurons in macaque visual area MST (but not MT) respond
to rotating flow fields or looming with considerable translation invariance (Graziano et al., 1994;Geesaman and Andersen, 1996). In the cortex in the anterior part of the superior temporal sulcus, which is a convergence zone for inputs from the ventral and dor- sal visual systems, some neurons respond to object-based motion, for example, to a head rotating clockwise but not anticlockwise, independently of whether the head is upright or inverted which reverses the optic flow across the retina (Hasselmo et al., 1989b).
In a unifying hypothesis with the design of the ventral cortical visual systemRolls and Stringer (2007)proposed that the dorsal visual system uses a hierarchical feed-forward network architec- ture (V1, V2, MT, MSTd, parietal cortex) with training of the connections with a short-term memory trace associative synaptic modification rule to capture what is invariant at each stage. The principle is illustrated inFigure 38A. Simulations showed that the proposal is computationally feasible, in that invariant representa- tions of the motion flow fields produced by objects self-organize in the later layers of the architecture (see examples inFigures 38B–E). The model produces invariant representations of the motion flow fields produced by global in-plane motion of an object, in-plane rotational motion, looming vs receding of the object. The model also produces invariant representations of object-based rotation about a principal axis. Thus it is proposed that the dorsal and ventral visual systems may share some unifying computational principlesRolls and Stringer (2007). Indeed, the simulations of Rolls and Stringer (2007)used a standard version of VisNet, with the exception that instead of using oriented bar receptive fields as the input to the first layer, local motion flow fields provided the inputs.
6. LEARNING INVARIANT REPRESENTATIONS OF SCENES