A final question that must be answered is how the number of evaluations performed by MOSES corresponds to actual computational complexity and runtimes. To summarize, the computational complexity of MOSES within a particular deme may be bounded by O(N ·n3), where N is the population size, and n is the size of the representation; n2 for modeling8 and n for the number of generations. The number of scoring function evaluations here will be O(N · n). We can let N = O(2k· n1.05), where k is the highest degree of subproblem interaction, and n = O(a · l), where a is the arity of the problem’s function and terminal set, and l is the size of the exemplar program.9 Letting c(l) be the problem-dependent cost of scoring a program of size l, we get:
8
As a technical note, this assumes that the depth of the decision trees (local structures) learned by the hBOA may be bounded by a constant. For hierarchical problems an additional factor of log(n) or n may need to be introduced here, and the ensuing calculations modified accordingly. In practice however, a constant bound on the decision tree depth seems reasonable.
9
This is the relation which perhaps can and should be improved most critically in future design develop- ments – see the next chapter.
0 2 4 6 8 x 105 0 20 40 60 80 100 Number of Evaluations Program Size 0 2 4 6 8 10 12 x 104 5 10 15 20 25 30 35 40 45 Number of Evaluations Program Size 0 0.5 1 1.5 2 2.5 3 3.5 x 104 0 20 40 60 80 100 120 Number of Evaluations Program Length 0 1 2 3 4 5 6 x 105 0 20 40 60 80 100 Number of Evaluations Program Size
Figure 6.6: Exemplar program size vs. number of evaluations for 5-parity (top left), 11- multiplexer (top right), n = 40 ORDER (bottom left), and n = 42 TRAP (bottom right). The dotted lines are the minimal optimal solution sizes (approximated for parity and mul- tiplexer).
Table 6.9: For various problems, the percentage of total MOSES execution time spent on the hBOA, on representation-building, and on program evaluation. The last row shows computational complexity (per generation), for comparison. N is the population size, l is the program size, a is the arity,and c is the number of test cases.
Problem hBOA Representation-Building Program Evaluation
5-Parity 28% 43% 29%
11-multiplexer 5% 5% 89%
CFS classification 80% 10% 11%
Complexity O(N · l2· a2) O(N · l · a) O(N · l · c)
O(N (nc(l) + n3)) = O(2kn2.05· (c(l) + n2) = O(2k(a · l)2.05(c(l) + (a · l)2)).
(6.3)
A conjecture for future investigation is that when representation-building is prop- erly configured for a domain at hand, the above equation will hold as a bound on the computational complexity of the entire MOSES procedure, taking n as the size of the tar- get program to be learned. This involves assuming no deception on the deme level, as described in Chapter 5. In this case, the complexity of the final deme optimization problem should dominate the cost of any single optimization problem encountered previously. For problems that may be solved via a few demes such as the artificial ant or the multiplexer, this argument is sufficient. But what about problems such as parity that require significant neutral drift and consequently many successive demes to solve, such as parity? In this case, I would further conjecture that the optimal configuration of MOSES is with asymptotically smaller populations than the above equation would suggest for individual demes, leaving the relation itself to hold in aggregate. Clearly, there is much work here still to be done.
The implementation of MOSES used for all experiments in this dissertation has been written in high-level C++, compiled with g++ version 3.4.4. Runs have been executed on machines with between two and four gigabytes of RAM, which is more than adequate; the code is typically CPU-bound rather than memory-bound. Profiling was carried out for the 5-parity, 11-multiplexer, and CFS classification problems; results are shown in Figure 6.9. In comparison, simpler program evolution systems such as GP typically spend nearly all cycles on program evaluation (cf. [48]).
The high percentage of time spent on representation-building was somewhat surpris- ing; in retrospect, this appears to be due to suboptimal implementation decisions rather than any intrinsic computational bottleneck (in practice or asymptotically). I believe that a
superior implementation could render the cycle counts for representation-building (at least on Boolean formulae) insignificant relative to the hBOA and program evaluation.
Which computational task is most intensive varies quite understandably by problem in accordance with the computational complexities given and problem parameters. For 11- multiplexer c = 211= 2048, and so program evaluation is dominant. For CFS classification the arity is relatively high (56), and so the cost of hBOA, which is quadratic in arity (a), dominates. 5-parity has relatively low arity, and c and l are of roughly the same magnitude; there is no clear dominating factor here in comparing the costs of hBOA and program evaluation.
Actual runtimes of MOSES were measured on a 2.2 GHz AMD Opteron machine with 4GB RAM, running GNU/Linux. The longest were the 6-parity runs, typically clocking in at around twelve hours. Runs for 5-parity, 11-multiplexer, and supervised classification typically ran for between one and three hours. Runs for other problems took less than one hour (per run).
Chapter 7
Towards a Theory of Competent
Program Evolution
Empirical and theoretical results have demonstrated that competent optimization algo- rithms such as the hBOA can learn a wide range of nearly decomposable functions. They are robust to deception, noise, exponential scaling of subproblems, etc., and scale poly- nomially with problem size. To a great extent, their success has followed from the clear articulation of exactly what problem classes they are trying to solve [28]. The corresponding effort to describe an interesting class of “nearly decomposable” program evolution problems is just beginning.
I have attempted to advance this effort by first describing the salient features of programs in particular that make them difficult to evolve in a direct fashion, as well as those that might be exploited as inductive bias (Chapter 1). Chapter 2 then outlines a general search mechanism that exploits this inductive bias. Chapters 3, 4 and 5 show, concretely, how program spaces may be studied to uncover the aforementioned features and transformed to exploit the aforementioned biases via representation-building. Finally, Chapter 6 presents and analyzes the results of applying this approach to solve problems.