Edit distance v Jaccard similarity

In Section 2.5.4 was stated that the edit distance could be used to quantify the distance between solutions to the VRP, thus its use is considered here. With the purpose of evaluating the proposed Jaccard similarity against the edit distance, both were implemented in BiEA, allowing a comparison of the results from both techniques. As was stated earlier, BiEA chooses the second parent according to

solution similarity. For this, we have at least two options when the set of T size individuals has been selected to compete in the tournament: it can be chosen the individual that is the least similar to the whole population (W) or the least similar to the first parent (P). The algorithm which implements Jaccard similarity and selects the individual that is the least similar to the whole population will be identified as BiEA-JW, and as BiEA-JP the one which selects the least similar to the first parent. Analogously, the algorithms which use edit distance will be labelled as BiEA-EW and BiEA-EP.

With the aim of making a proper comparison, the outcome non-dominated solu- tions from each implementation of BiEA were evaluated using the three perfor- mance metrics reviewed in Section 2.3.2, i.e. coverage, MC in (2.7), convergence, MD

in (2.9), and hypervolume, MH in (2.10). In order to apply the coverage metric,

for each given instance and ordered pair of implementations BiEA-X and BiEA-Y, M_{C(BiEA-Xi, BiEA-Yj), ∀ i, j = 1, . . . , 30, that is 900 MC values, were computed.} BiEA-Xi refers to the outcome set from the i-th execution of BiEA-X. After these

computations, the MC(BiEA-Xi, BiEA-Yj) values were averaged (MC) over all the

instances within each set category, and the resulting 900 values were collected to- gether. These MC values are presented in Figure 4.8 as box-and-whisker diagrams,

which represent the distribution of the MCvalues for each ordered pair of algorithms.

Each cell, which range is 0 at the bottom and 1 at the top, contains six box-and- whisker plots, corresponding to categories C1, C2, R1, R2, RC1, and RC2 from left to right, referring to the average coverage of the algorithm in the corresponding column by the algorithm in the corresponding row. Each box indicates where the middle 50% of the data is located, on which the central mark is the median and the lower and upper edges are the first and third quartiles respectively. Dashed lines specify the most extreme data values that are not considered as outliers.

BiEA-EW

BiEA-EP

BiEA-JW

BiEA-JP

Figure 4.8: Box-and-whisker plots representing the distribution of the MCvalues

for each ordered pair of the implementations EW, EP, JW, and JP of BiEA.

We observe that, for categories C1 and C2, all four algorithms found the optimum solutions for almost all the instances, this is why the plots corresponding to these categories, the two leftmost boxes on each cell, show similar heights. For the remai- ning categories, we can observe that the height of the medians corresponding to the coverage by BiEA-EP and BiEA-JP, second and forth rows, is always lower than 0.5, and those by the implementations BiEA-EW and BiEA-JW, first and third rows, are higher than 0.5, with some exceptions in the cells between these two. This means

C1 C2 R1 R2 RC1 RC2 0 5 10 15 Set category Average convergence (x10 −3

) BiEA−EW BiEA−EP BiEA−JW BiEA−JP

Figure 4.9: Bar plots representing the MD values, averaged over instance ca-

tegory, for the results obtained by the implementations of BiEA with similarity and selection settings EW, EP, JW, and JP.

that the latter methods have a higher coverage of the former than the inverse cases. Finally, for the implementations BiEA-EW and BiEA-JW, the coverage between them appears to be similar.

In order to compare the algorithms using the convergence metric, it is necessary to have a reference set for every instance since the true Pareto fronts are not known. For each algorithm and instance, the overall non-dominated solutions were extracted from the 30 Pareto approximations. Then, a composite non-dominated reference set R was found using the overall non-dominated sets from the four algorithms. Afterwards, for each implementation BiEA-X, MD(BiEA-Xi, R), ∀ i = 1, . . . , 30,

were computed. Finally, the MD values were normalised according to the distance

from point z = (N, Dmax_{) to the origin, where z corresponds to the solution with}

largest number of routes, i.e. number of customers N , and longest travel distance Dmax_{equal to twice the sum of the distances of all customers from the depot. These}

normalised MD values were grouped by the instances in each set category and the

average MD and standard error were calculated. Figure4.9 presents these results

as bar plots, which heights represent the averages. We can see that solutions from BiEA-EP and BiEA-JP, second and fourth bars on each group, are the farthest

to the reference set in five of the six categories, while those from BiEA-EW and BiEA-JW, first and third bars, are the nearest and present similar performance. On the other hand, the hypervolume metric has been utilised to compare only algorithms BIEA-EW and BiEA-JW, since they have been the most competitive methods regarding the other two quality indicators. Computing the hypervolume metric MH requires an appropriate reference point z to be set. As was done for

the normalisation of the convergence metric, each instance has an obvious maximal solution, that with largest number of routes N and longest travel distance Dmax_,

hence the reference point for each instance was set at z = (N, Dmax_{). For each}

implementation BiEA-X, MH(BiEA-Xi, z), ∀ i = 1, . . . , 30, were computed. Then,

the MH values were normalised according to the space defined between point z and

the origin. These MH values were grouped by the instances in each set category and

the average MH was computed.

The details of the hypervolume metric, along with those of the coverage and conver- gence metrics, comparing the results from BiEA-EW and BiEA-JW are presented in Table 4.3. The averages MC, MD and MH over the instances in each category

are shown, along with the number of instances, in brackets, where the algorithm performs better than the other at the 95% significance level. The statistical si- gnificance for each instance was determined by applying a two-tailed t-test for two samples with unequal variance to the results of MC(BiEA-EWi, BiEA-JWj)

and MC(BiEA-JWj, BiEA-EWi), MD(BiEA-EWi, R) and MD(BiEA-JWj, R), and

M_{H(BiEA-EWi, z) and MH(BiEA-JWj, z), ∀ i, j = 1, . . . , 30. The result of the t-} test specifies, with 95% of confidence, if the true means of the data do differ. In Table 4.3 we see that the averages of the coverage, convergence, and hypervolume metrics present a small difference between the results from BiEA-EW and BiEA-JW for all categories, which may suggest that both algorithms perform similarly. Ho- wever, despite the narrow gap, BiEA-EW have a significantly better performance,

Algorithm Metric C1 C2 R1 R2 RC1 RC2 BiEA-EW MC 0.85 (5) 0.73 (5) 0.47 (4) 0.51 (5) 0.47 (2) 0.53 (5) MD(×10−4) 14.80 (0) 7.684 (0) 58.69 (1) 82.72 (0) 48.57 (0) 92.01 (2) MH(×10−2) 76.95 (0) 87.30 (0) 66.39 (1) 78.46 (3) 69.69 (1) 80.44 (4) BiEA-JW MC 0.81 (2) 0.64 (1) 0.48 (7) 0.47 (3) 0.49 (3) 0.39 (1) MD(×10−4) 17.63 (0) 8.099 (0) 56.44 (0) 82.05 (1) 48.65 (1) 99.61 (0) MH(×10−2) 76.92 (0) 87.29 (0) 66.43 (1) 78.39 (0) 69.64 (0) 80.21 (0)

Table 4.3: Averages MC, MD and MH over instance category for the solutions

obtained by BiEA-EW and BiEA-JW. Shown in brackets are the number of instances for which the result is significantly better than the other approach.

C1 C2 R1 R2 RC1 RC2 0 200 400 600 800 1000 1200 Set category

Average execution time (s)

BiEA−EW BiEA−EP BiEA−JW BiEA−JP

Figure 4.10: Execution time, averaged over instance categories, of the imple- mentations of BiEA with similarity and selection settings EA, EP, JA, and JP.

regarding the coverage metric, for more instances in set categories C1, C2, R2, and RC2, while BiEA-JW for more instances in categories R1 and RC1. We can also ob- serve that BiEA-EW delimits a significantly larger hypervolume for some instances in categories R2 and RC2.

Lastly, Figure 4.10represents the average execution time, corresponding to each set category, of the four implementations of BiEA. It is clearly visible that BiEA-JW is the quickest method, while BiEA-EW is the slowest, the latter taking at least 300% more time in executing 500 generations than the former, and more than 500% on average. This is due to the fact that the edit distance has a quadratic time complexity, in contrast to the linear behaviour of the Jaccard similarity.

Based on the analysis of the three performance metrics, we conclude that imple- mentations BiEA-EW and BiEA-JW have a better performance that the implemen- tations BiEA-JP and BiEA-EP, since the former obtain better results for all three quality indicators. Considering execution time, BiEA-JW is the quickest algorithm, running, on average, 500% faster than BiEA-EW, the slowest. This last fact make us believe that the saving in time is worth the minimal overall differences in the performance metrics between algorithms BiEA-JW and BiEA-EW. However, both edit distance and Jaccard similarity will be tested later with the final algorithm in order to analyse if they present the same behaviour. Meanwhile, the following experiments and analysis will be made considering the results from BiEA set to use Jaccard similarity, and hereafter BiEA-JW will be simply called BiEA.