In the large y limit the external entropy Σ (D, y) is found to be negative for all values of the parameters, thus canceling out the unphysical results shown above. This signals a problem with the RS Ansatz, and implies that we should instead consider replica-symmetry-broken solutions. In geometrical terms, the interpretation is as follows: the RS solution at y → ∞ implies that the typical overlap between two different reference solutions ˜Wa and ˜Wb, as computed by ˜q = 1
N
P
iW˜iaW˜ib, tends to 1, and therefore that there should be a single solution of maximal local entropy density. The fact that the RS assumption is wrong implies that the structure of the configurations of maximal density is more complex, and that, at least beyond a certain y, the geometry of the reference configurations ˜W breaks into several clusters.
Still, for a large range of values for α this limit is in good agreement with the more involved analyses and can provide some insight into the physical phenomena under study.
4.4
Finite y
Before giving up with the RS Ansatz, we can try to study the problem at finite
y [1]: a motivation is given by the fact that the replica symmetric scenario is
more directly comparable with the typical Franz-Parisi potential, providing the most straightforward way to demonstrate the radically different picture about the nature of the solutions painted by the large deviations analysis and the equilibrium case. Moreover, from the technical point of view, the 1RSB equations will produce a larger system of equations, involving multiple nested integrals that are very computationally expensive for arbitrary y.
In order to obtain a two-dimensional phase diagram, we need to choose a criterion for fixing the value of y while α and D are varied. We have seen that Σ (D, y) needs to be positive in order to get physically meaningful results: the systems are discrete, so this quantity measures the logarithm of the number of reference configurations ˜W (at the given y and D) divided by N, and negative
values would indicate rare events instead of typical instances [73].
It turns out that, for all values of α and D, there is a value of y beyond which Σ (D, y) < 0. Therefore, we can search for the value y⋆ = y⋆(α, d) at which Σ (D, y⋆) = 0, i.e. the highest value of y for which the RS analytical results are consistent, representing structures that appear O (1) times in the space of solutions (as we expect the dense cluster to do). It is worth noting that following this criterion we still get ˜q < 1, which implies that the number of reference solutions ˜W is larger than 1. For each couple of α and D, the
sought value of the inverse temperature can be found by interpolating between different saddle point solutions at varying values of y.
From the results (shown in figure 4.4), we observe that up to a certain αU (where αU ≃0.77 in the classification case and αU ≃1.1 in the generalization case), the S (D) curves are monotonic in D. Beyond αU, there is a transition in which there appear regions of D (dotted in figure 4.4) which are not correctly described by the RS Ansatz (since geometric bounds are violated, see the discussion in the SM for details), and must be described at a higher level of replica symmetry breaking (RSB). We speculate that this transition signals a change in the structure of the space of solutions: for α < αU, the densest cores of solutions are immersed in huge connected structure; for α > αU, this structure fractures and the dense cores become isolated and hard to find (see a sketch of this transition in figure 4.5).
Using the vanishing complexity criterion is sufficient to derive results which are geometrically valid across most values of the control parameters α and D. There are two exceptions to this observation, though, both occurring at high values of α and in specific regions of the parameter D. Let us indicate with [DL, DR] these regions, with 0 < DL< DR <1:
• The most obvious kind of problem occurs occurs at α ≳ 0.79, where S(D, y) < 0 for D ∈ [DL, DR]. The standard treatment for this kind of problem is to break the replica symmetry, until the process reaches a level
Fig. 4.4 Local entropy curves at varying distance d from the reference solution ˜W
for various α (classification case). Black dotted curve: α = 0 case (upper bound). Red solid curves: RS results. Up to α = 0.77, the curves are monotonic. At α = 0.78, a region incorrectly described within the RS Ansatz appears (dotted; geometric bounds are violated at the boundaries of the part of the curve with negative derivative). At
α = 0.79, the solution is discontinuous (a gap appears in the curve), and parts of
the curve have negative entropy (dotted). Blue dashed curves: equilibrium analysis (typical ˜W ) [22] (dotted parts are unphysical): the curves are never positive in a
neighborhood of d = 0. Inset: zoom of the region around d = 0 (notice the solution for α = 0.79, followed by a gap).
Fig. 4.5 Sketch of the phase space of the random binary Perceptron, as described by the large deviation analysis. The dense region of solutions disappears before the SAT/UNSAT transition.
where the local entropy remains positive or null for any α and D (in the equilibrium calculations a single step of symmetry breaking is sufficient, but this is not guaranteed to hold for this large deviation analysis). • Another type of transition occurs between α ≃ 0.77 and α ≃ 0.79,
where the ∂
∂DS(D, y) ≤ 0 in [DL, DR]. A closer inspection of the order parameters reveals that, in this interval, q1 ≥ S. At the transition points
q1 = S, which is manifestly unphysical: paradoxically any of the solutions
W (which are exponential in number, since S > 0) could play the role of
the reference solution ˜W, yet the number of ˜W should be sub-exponential
at Σ = 0. Clearly, those regions are inadequately described within the RS Ansatz.
As for the parts of the curves which are outside these problematic regions, the results obtained under the RS assumption are reasonable, and in very good agreement with the numerical evidence. In order to assess whether the RS equations are stable, further steps of RSB would be needed; unfortunately, this would multiply the number of order parameters and complicate the system of equations.
Since the extremal case is found only in the limit of y = ∞ (where the RS solution is inadequate) the values found for S (Σ = 0) might be seen as lower bound. However, when D → 0 the sought value y⋆ → ∞. The same happens when D → (1 − qRS) /2, where the distance constraint starts including the typical equilibrium solutions of the standard analysis and S becomes equivalent to the standard entropy of the equilibrium ground states.