Lyapunov exponents

As described in §4.1 above, in a chaotic system, two trajectories that have sepa- ration δ0 at time 0 will diverge over time. If δt is the separation at time t, and

|δt| ' |δ0|eλt then λ is known as the Lyapunov exponent, and it measures the

exponential rate at which the two trajectories will diverge (Strogatz, 1994). Alter- natively, in a stable system, λ will be negative, indicating the rate at which two nearby trajectories converge to an attractor.

In reality, for an N-dimensional system, there are actually N Lyapunov ex- ponents. The spectrum of Lyapunov exponents of an N-dimensional dynamic system can be conceptualised by imagining the time evolution of an infinitesimally small, N-dimensional sphere. Over time the sphere will become an ellipsoid and if we let δk(t), k = 1, . . . , N denote the kth principal axis of the ellipsoid, then

|δk(t)| ' |δk(0)eλkt| where λk are the Lyapunov exponents, each describing the

expansion or contraction of the ellipsoid in the n dimensions. Over time, the di- ameter of the ellipsoid will be dominated by the most positive λk, therefore λ is

the largest Lyapunov exponent (Wolf et al., 1985). From this point on in this thesis, any mention of the Lyapunov exponent will refer to the largest Lyapunov exponent:

λ=lim(t → ∞)log(δt

δ0

) (4.3)

The magnitude of the Lyapunov exponents provides a quantitative picture of a system’s dynamics in information theoretic terms, measuring the rate at which systems create or destroy information (Wolf et al., 1985). In a practical sense, the magnitude of a positive exponent corresponds to the time scale on which a system’s dynamics become unpredictable. The magnitude of a negative exponent corresponds to the rate at which a system approaches an attractor.

While it is possible to calculate the spectrum of Lyapunov exponents directly from a set of differential equations, the difficulty of doing so increases with the size of the system under consideration (Wolf et al., 1985). Fortunately, knowledge of the largest Lyapunov exponent is sufficient to identify the qualitative behaviour of a system, and several methods exist for estimating the value of this exponent from time series data (Wolf et al., 1985, Bryant et al., 1990, Sprott, 2003). This technique has previously been applied to the analysis of high-dimensional neural

4.2 Study 1: Tools for exploring network dynamics 65

networks (Dechert and Gencay, 1992, Albers et al., 1998, Albers, 2004). In partic- ular, Albers (2004) carried out a substantial investigation of the transition from order to chaos in a parameterised class of high-dimensional dynamic systems. The procedure used to estimate the value of the largest Lyapunov exponent in this study was based on that described by Sprott (2003) (pp.116–117):

1. The state of the networkI was initialised to (I0, . . . , IN−1).

2. The network was iterated for 1,000 steps to ensure that the trajectory was located on an attractor, rather than a transient.

3. A duplicate network was created and its activation state (s0) was perturbed such that its separation from the state of the unperturbed network (s) was

δ0.

4. Both networks were iterated for a single step.

5. The new distance, δ1, between the states of the original and perturbed net-

works was measured as:

δ1 = [(s00−s0)2, ...,(sN0 −1−sN−1)2]1/2.

6. The log of the ratio of the two distances,rt, was calculated and recorded as:

rt =ln(

|δ1|

|δ0|

)

7. The state of the perturbed network was modified such that the direction of its trajectory was unchanged, but its distance from the trajectory of the original network was restored toδ0(see Figure 4.5) by adjusting the activation of each

node as follows:

s0_n(t) =sn(t−1) +

δ0

δ1

[s0_n(t−1)−sn(t−1)]

8. Steps 4 to 7 were repeated fort= 500 iterations and the value of the largest Lyapunov was estimated by taking the average of the log ratios:

66 The Dynamics of Cell Differentiation

Figure 4.5: A graphical representation of the procedure used to estimate the largest Lyapunov exponent. The continuous line represents the original (unperturbed tra- jectory). d0 is the original separation between the unperturbed and perturbed tra- jectories. d1 is the separation between the two trajectories after a single iteration. After each iteration, the perturbed trajectory is adjusted so that its separation is

d0 in the direction of d1. (Figure redrawn from Sprott, 2003, p. 117)

λ= 1 t t X i=1 rt

The number of time steps required to assure sufficient accuracy in steps 2 and 8 was found to vary with the size of the system. Preliminary trials were used to determine the number of time steps required to ensure that the network was located on an attractor (in step 2) and that the value of the Lyapunov exponent had converged (in step 8). As with the orbit diagram, this procedure was repeated for 100 values of W in the range [0.1,20.0] and the values of λ obtained were plotted (Figure 4.4).

Figure 4.4 provides a complementary view of network dynamics to that pro- vided by the orbit diagram shown in Figure 4.3. Whereas Figure 4.3 showed the locations of attractors, Figure 4.4 shows their stability. The nature of the attrac- tors can now be verified:

1. When W is below 1.0 and the system is stable, the Lyapunov exponent is negative;

2. Around W = 1.0, the attractor bifurcates and the Lyapunov exponent ap- proaches 0.0.

4.2 Study 1: Tools for exploring network dynamics 67

3. When the system is chaotic, the Lyapunov exponent is positive. Positive Lyapunov exponents are much noisier due to the non-repeating nature of the trajectory; the exact value of the exponent varies throughout the trajectory. AsW becomes very large, the activation function saturates, the system begins to approximate a Boolean network (Kauffman, 1993) and the Lyapunov exponent drops below 0.0. At this stage, the network is very robust to small perturbations to activation.

Even when there is no qualitative change in the type of an attractor, its stability changes asW is varied and the location of the attractor changes. The sudden jumps in the location of the attractor that were observed in the bifurcation diagram are matched by sudden changes in the value of the Lyapunov exponent. One limitation of the method described above is that, unless the DRGN contains only a single attractor, only a portion dynamic space is being measured (that of the basin containing the initial state I). The following section addresses this shortcoming by sampling more widely from the set of initial conditions.