Learning in feature spaces

We compare experiments performed under two conditions. In the original condition, the GAPS algorithm was used with a lookup table representation with a single pa-rameter θ(o, a) for each possible observation-action pair. For the task of locomotion by self-reconfiguration, this constituted a total of 2⁸ × 9 = 2304 parameters to be estimated.

In the log-linear function approximation condition, the LLGAPS algorithm was run with a set of features determined by hand as the cross product of salient aspects of neighborhood observations and all possible actions. The salient aspects were full corners and straight lines, and empty corners and straight lines. The features were obtained by applying partial masks to the observed neighborhood, and selecting a particular action, with response fields as shown in figure 3-5. This gave a total of 144 features, the partial masks for which are fully detailed in figure 3-6. In all cases, modules were learning from the space of reactive policies only.

4The rewards are most probably not normally distributed: indeed, the distribution will be mul-timodal, with “good” and “bad” trials clustered together. It may therefore be more appropriate to use the nonparameteric Kruskal-Wallis test, which does not make the normal distribution as-sumption. The Kruskal-Wallis test also reveals a significant difference in reward as a function of the learning algorithm (χ²(2, 27) = 16.93, p < .01). However, the KW-test does not transfer well to multi-way analysis (Toothaker & Chang 1980), which will be necessary in further experiments. We will therefore use ANOVA, because it is robust to some assumption violations, and because we are only interested in large effects. But see also Brunner & Puri (2001) for a framework for multi-way nonparametric testing.

(a) (b)

Figure 3-5: 6 modules learning with LLGAPS and 144 features. (a) One of the features used for function approximation in the LLGAPS experiments: this feature function returns 1 if there are neighbors in all three cells of the upper left corner and a_t = 2(NE). (b) Smoothed (100-point moving window), downsampled average rewards over 10 runs, with standard error: comparison between original GAPS and LLGAPS.

Hypothesis: LLGAPS’s compact representation will lead to faster learning (better convergence times), as compared to standard GAPS.

Figure 3-5b presents the results of comparing the performance of LLGAPS with the original GAPS algorithm on the locomotion task for 6 modules. We see that both algorithms are comparable in both their speed of learning and the average quality of the resulting policies. If anything, around episode 1,500 basic GAPS has better performance. However, a two-way mixed-factor ANOVA (repeated measures every 1000 episodes, GAPS vs. LLGAPS) reveals no statistical significance. Our hypothesis was wrong, at least for this representation.

However, increasing the number of modules reveals that LLGAPS is more prone to finding unacceptable local minima, as explained in section 4.3.2. We also hypothesized that increasing the size of the observed neighborhood would favor the feature-based LLGAPS over the basic GAPS. As the size of the observation increases, the number of possible local configuration grows exponentially, whereas the features can be designed to grow linearly. We ran a round of experiments with an increased neighborhood size of 12 cells obtained as follows: the acting module observes its immediate face neighbors in positions 1,3,5, and 7, and requests from each of them a report on the three cells adjacent to their own faces⁵. Thus the original Moore neighborhood plus four additional bits of observation are available to every module, as shown in figure 3-7a. This setup results in 2¹²× 9 = 36, 869 parameters to estimate for GAPS. For LLGAPS we incorporated the extra information thus obtained into an additional 72 features as follows: one partial neighborhood mask per extra neighbor present,

5If no neighbor is present at a face, and so no information is available about a corner neighbor, it is assumed to be an empty cell.

Full corner Full line Empty corner Empty line

- current acting module - neighbor present

- no neighbor (empty cell) - neighbor or empty cells

Figure 3-6: All 16 observation masks for feature generation: each mask, combined with one of 9 possible actions, generates one of 144 features describing the space of policies.

15 modules 20 modules

GAPS 0.1±0.1 5±1.7

LLGAPS with 144 features 6.6±1.3 9.7±0.2

Table 3.1: Mean number of configurations that do not correspond to locomotion gaits, out of 10 test trials, for 10 policies learned by GAPS vs. LLGAPS with 144 features, after 6,000 episodes of learning, with standard error. We will use the mean number of non-gait configurations throughout this thesis as one of the performance measures for various algorithms, representations and constraints. The standard error ˆσ/p(N) (N is the number of sampled policies, here equal to 10) included in this type of table both gives an estimate of how spread-out the values from different policies are, and is a measure of quality of the mean value as a statistic.

and one per extra neighbor absent, where each of those 8 partial masks generates 9 features, one per possible action. We found that increasing the observation size indeed slows the basic GAPS algorithm down (figure 3-7b, where LLGAPS ran with a much lower learning rate to avoid too-large update steps: α = 0.005 decreasing to α = 1e⁻⁵ over the first 1,500 episodes and remaining at the minimal value thereafter).

During the first few hundred episodes, LLGAPS does better than GAPS. However, we have also found that there is no significant difference in performance after both have found their local optima (mixed-factor ANOVA with repeated measures every 1000 episodes, starting at episode 500 reveals no significance for GAPS vs. LLGAPS but a significant difference at 99% confidence level for the interaction between the two factors of episodes and algorithm). Table 3.1 shows that the feature-based LLGAPS is even more prone to fall for local optima.

Two possible explanations would account for this phenomenon. On the one hand, feature spaces need to be carefully designed to avoid these pitfalls, which shifts the hu-man designer’s burden from developing distributed control algorithms to developing sets of features. The latter task may not be any less challenging. On the other hand, even well-designed feature spaces may eliminate redundancy in the policy space, and therefore make it harder for local search to reach an acceptable optimum. This effect is the result of compressing the policy representation into fewer parameters. When an action is executed that leads to worse performance, the features that contributed to the selection of this action will all get updated. That is, the update step results a simultaneous change in different places in the observation-action space, which can more easily create a new policy with even worse performance. Thus, the parameters before the update would be locally optimal.