Comparison of Performance Between Assorted Benchmark Problems

The benchmark testing by [17] compared the Hybrid Ant System for the QAP (HAS-QAP) with the genetic hybrid method of [37], the reactive tabu search [38], a tabu search of [18], and simulated annealing [39]. Of these tests, Taillard’s tabu search [18] will be run alongside the TabuACO solvers on the same machine, matched by iteration count rather than runtime. By comparing Taillard’s tabu search to the TabuACO, it provides a benchmark comparison to all of the solvers tested by [17].

3.3.7. Data. Two sets of experiments were performed. One experiment tested the TabuACO against a conventional ACO. The other tested the TabuACO against other solvers.

3.3.7.1. Performance comparison using attractive-only verses attractive+repulsive pheromones. The entire suite from Christofides and Beavent was tested to compare the TabuACO to the reference ACO solver. The results are summarized in Table 3.4.

Table 3.4. Results Comparing the Performance of the TabuACO to the Reference ACO

Puzzle

Table 3.4. Results Comparing the Performance of the TabuACO to the Reference ACO

3.3.7.2. Comparison to other solvers. The puzzles in Table 3.5 were solved by the RTS, TabuACO, and ACO solvers. In most cases the RTS solver drove the puzzle to the best-known minimum. In some cases, the RTS solver could not achieve this minimum. It was allowed to run its course, and the number of iterations used to achieve its best score were recorded. This same number of iterations were then used to allow the TabuACO and reference ACO solvers to achieve the best score they could attain. The percentage above ideal is expressed in the two rightmost columns. Ten trials were conducted and averaged.

Table 3.5. Comparison of RTS to TabuACO and Reference ACO Solvers

Table 3.5. Comparison of RTS to TabuACO and Reference ACO Solvers (con’t)

tai80b 818415043 343559 0.02 34 34

wil50 48816 84827 0.02 9 9

3.3.8. Discussion of Results. Two different experiments are discussed. One pits a conventional ACO against the TabuACO. The other compares the TabuACO to other solvers.

3.3.8.1. Performance comparison using attractive-only pheromones verses attractive+repulsive pheromones. The study shows that (for the puzzles studied) the TabuACO with both attractive and repulsive pheromones outperformed the solver with attractive pheromones only [30]. When a short run was performed, it was difficult to detect a difference in performance between the two. A longer run was often necessary to bring out the differences in performance. It is believed that this is due to the need for a significant amount of information to be present in the environment before the ant’s behavior becomes significantly affected. Compare the puzzle with a small number of runs to the same puzzle with a large number of runs. Quite often, a statistically significant difference³ is not apparent until after ten thousand sorties.

It should be noted that the TabuACO solver is intended to work with computationally constrained applications which are unable to store the entire model in memory. If the important and relevant portions of the model are retained, convergence upon the optimum should be possible. If important portions of data are missing from the

3 Significance being quantified here by the student’s T-test.

model, the solver will have difficulty finding the optimal solution and may malfunction altogether.

With the QAP established as outlined in Figure 3.6, a large search space is created. Only a portion of the travelled edges are retained in memory. An analysis of a completed run typically shows a count of fullyExplored edges to be approximately 0.02%. The memory manager recycles fullyExplored nodes. Yet, due to their strategic location, these remaining edges served to influence the outcome of the TabuACO solver.

The test results show that in every case which is driven toward a result, there is a statistical advantage to using the repulsive pheromone information in addition to the attractive pheromone information.

3.3.8.2. Comparison to other solvers. The analysis in this paper compared the RTS solver to the TabuACO solver, and by extension, to all of the other solvers tested in [17]. The comparison in this case was based on iteration equivalents rather than computational time. The RTS solver beat the TabuACO solver in every case. The TabuACO and conventional ACO did fairly well on the entire “bur” series of puzzles, but not as well on other series. It was expected that the RTS (as a specialized solver) would outperform the more general-purposed ACO and TabuACO solvers. The TabuACO and conventional ACO both deposited attractive pheromone and attempted to converge to a solution by exploring combinations of numbers that appeared to yield good scores. The TabuACO outperformed the conventional ACO because it prevented previously explored portions of the graph from being reexplored. The Robust Taboo Solver (RTS) however made use of a prioi information. It was able to eliminate many combinations of numbers from the search space by exploiting properties of the QAP equation itself.

3.3.8.3. Memory management. The solver retained only 1000 of the most valuable nodes (as determined by the scores leading to them). For any puzzle studied, this is a remarkably small percentage of the solution space. Even for the smallest puzzle, less than 2 part per million of the solution space was retained in memory. For the largest puzzle, 1000 nodes represent a mere 1:10²² portion of the solution space. The results show a statistical difference in performance between single and dual pheromone operation. It is believed the memory manager must be doing an effective job ranking node value, retaining important nodes and discarding less important ones.