nth power diverges whenn grows. In a purely linear network, this phenomenon would lead to a blow-up of the power spectral density, in agreement with the fact that activity in a linear network is unbounded forg> gc. Ifφ is a compressive nonlinearity however, the coefficient anwill tend to zero for growingn, counterbalancing the unbounded growth of
g2| ˜χ110 ( f )|2n. Notice that this constraint on the coefficientan is necessary independently of the truncation of the series in Eq. (4.13), since all the neglected terms are positive and would not provide a different mechanism for contrasting the growth of the first term. Based on Eq. (4.17), we would predict that all the modes for which| ˜χ11
0 ( f )|2> 1/g2will get amplified over multiple iterations,
while all the other modes will get suppressed. While this is a highly simplified description, the suppression and the amplification of modes is clearly visible when comparing the dynamics of the self-consistent solution (Fig. 4.1C,F) to the corresponding linear response function (Fig. 4.1B,E). When truncating the series after the first order however, the mean-field network does not admit a self-consistent solution, for which we need to retain also higher order terms. The presence of those terms will be reflected, among others, in the interference among amplified modes. We now consider also higher order terms of the sum, which allow the existence of a self-consistent solution and that are responsible for the formation of harmonics. For example, the second order term in Eq. (4.13) is given by
1 2 * φ(x1)+2Cx1(τ) 2 F T −−→1 2s2 ∞ −∞S 11 x ( f)d f S11x ∗ S11x ( f ) (4.18) wheres2is defined analogously tos1. In general, higher-order terms will contain convolutions of
the power spectral density with itself, which are responsible for the creation of higher harmonics. Indeed, if a function has a bump-shaped profile, then itsn-times self-convolution shifts the center of the bump to thenthmultiple of the bump center. This implies that if the power spectral density is resonant, i.e. if it has a peak at a nonzero frequency, then to be self-consistent it should also exhibit harmonics.
In Fig. 4.1A,B,C we consider a three-dimensional rate model that has the global maximum at zero frequency and a local maximum at a higher frequency. For the weakest connectivity strength (g= g1), only the modes near the global maximum are amplified (Fig. 4.1C), in agreement with
our qualitative analysis. At the other extreme, i.e. for very strong connectivity (g= g3), also
the modes near the second local maximum survive in the self-consistent solution. Finally, for intermediate connectivity strength (g= g2), the interference among modes is strong enough to
prevent the amplification of the modes close to the local maximum, so that only those next to the global one survive. Similar observations can be made for the four-dimensional model shown in Fig. 4.1D,E,F.
4.5
Robustness of the iterative method
The formalism we presented is independent of the choice of the nonlinearity and the only assumptions that we made in deriving the existence of a microscopic fixed-point in zero is that
Chapter 4. Dynamics of multi-dimensional rate units
Figure 4.1 – Two examples of multi-dimensional rate models. A-B-C: Analysis of a three- dimensional rate model. Eigenvalue spectra (A) corresponding to the coupling valuesg1= 1.28,
g2= 1.4andg3= 2. The dashed line indicates the imaginary axis. In B we plot the linear response
function of the single unit| ˜χ110 ( f )|2(solid line), and the instability threshold corresponding to the three coupling valuesg1,g2andg3(dashed lines). In C we plot the solution of the mean field
theory obtained with the iterative method for the three values ofg,g1= 1.5,g2= 2andg3= 3.
D-E-F: Same as A-B-C, but for a four-dimensional rate model.
4.5. Robustness of the iterative method
Fig. 4.1A,B,C Fig. 4.1D,E,F Fig. 4.2
⎛ ⎝ 0.1−1 −0.1−1 1.7−1 0.1 −0.4 −0.5 ⎞ ⎠ ⎛ ⎜⎜ ⎜⎝ −1 −1 −1 −1 1 −0.5 −0.65 −0.6 1 0.35 −0.05 −0.57 1 0.35 0.28 −0.005 ⎞ ⎟⎟ ⎟⎠ −1 −1 0.25 −0.25
Table 4.1 – Parameters of the models in the examples. MatrixAdefining the rate model for the different example in Fig. 4.1 and 4.2.
φ(0) = 0. In solving the mean-field theory for the examples presented in Fig. 4.1, we used a piecewise-linear approximation of the hyperbolic tangent, given by
φP L(x)= ⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩ −1 for x< −1 x for − 1 ≤ x ≤ 1 1 for x> 1 . (4.19)
The self-consistent solution that we obtain does not vary qualitatively when considering similar gain functions, such as the hyperbolic tangent or a cubic approximation of it (shown in Fig. 4.2A, for the two-dimensional model studied in chapter 3). However, since the cubic gain function is unbounded, the dynamics will be unstable above a certaingc∗, which is different fromgc in general.
The evolution of the power spectral densityS11x ( f )over iterations is shown in Fig. 4.2C, where the formation of harmonics over iterations is clearly visible. While we have no guarantee that the iterative method converges, empirical tests of the method indicate convergence properties that match the one of stability of the network itself, i.e. if the network converges, so does the iterative method. The mean-squared distance between the power spectral density at consecutive realizations decreases approximately exponentially after an initial transients (Fig. 4.2B) and saturates at a value dependent on the numerical error made in performing the nonlinear pass. The iterative method is also robust over different initializations ofS11x ( f ). To follow the evolution ofS11x ( f )over iterations, we measure total area under it, i.e.
"+∞
−∞S11x ( f )d f =Var(x1)and the maximum height of the power spectral densitymaxf(S11x ( f )). Despite different initializations, the trajectories in the subspace of these two measures converge quite rapidly to a one-dimensional sub-manifold (Fig. 4.2D).
Discussion
Chapter 4. Dynamics of multi-dimensional rate units
Figure 4.2 – Stability of the iterative method for a two-dimensional rate model. A: Power spectral densityS11x ( f )for three different nonlinearities, as indicated in the legend. B: Mean- squared distance (MSD) between two consecutive iterations ofS11x ( f ). Conventions are the same as in A. Notice that the curve corresponding to the hyperbolic tangent is saturating at a much higher value than for the other nonlinearities, which is due to the sampling method used to evaluate the nonlinear pass. C: Evolution of the power-spectral density over iterations, shown for piecewise linear nonlinearity. D: Evolution of the total area under the power spectral density Var(x)and of the maximum amplitude ofSx( f ), over iterations. Shown for piecewise linear nonlinearity.