The results of the parallel processing tests are shown inFigure 29, Figure 30, Figure 31, and Figure 32. They correspond to performance characteristics of PERSEUS’s social navigation support, PERSEUS’s concept-based navigation support, NavEx’s concept-based navigation support, and PERSEUS’s topic-based navigation support, respectively.
Each figure has 5 (4 in the case of Figure 32 for topic-based navigation support) graphs each corresponding to different request complexities (AKA request sizes). These graphs are percentile plots where the horizontal axis denotes time taken to complete requests and the vertical axis denotes percent of the requests complete. On each graph there are several curves, each corresponding to a different load condition (measured in milliseconds between requests).
These graphs offer a limited view on the timeline of the request completion. The horizontal axis is capped at 2 seconds – the maximum response delay – according to (Nielsen, 1993) – that would not interrupt the user’s flow of thought when working with a system. Although (Shneiderman, 1984) gives a smaller delay of only 1 second, 2 seconds seems like an acceptable compromise.
The general rule for interpreting percentile graph is to look at the point of the 95th percentile and at the general shape of the curve in the graph. A percentile curve that corresponds to an acceptable performance is expected to be oriented approximately vertically. As soon as the time values abruptly grow, the performance degrades. To conclude that the system copes with the load successfully (sometimes the term scalable is used here), the curve is expected to be approximately vertical until the 95th percentile.
In our case, we are adding a 2-second cap on the percentile values. We are not interested in the orientation of the curve that much. Our criterion is that the curve stays within the 2-second boundary as much as possible. For this reason we decided to lower the percentile criterion from 95 to 90 percent. Thus, to conclude that a certain implementation of an adaptation technique successfully copes with a particular load, we require 90 percent of the requests to complete in less than 2 seconds, i.e., the 90th percentile point to be on the graph. Because we are most interested in whether 90th percentile point is within 2 second boundaries, we are only displaying percentiles starting with 80th in Figure 29, Figure 30, Figure 31, and Figure 32.
Figure 29. Percentile plots for PERSEUS’s social navigation support service capacity-planning/soak tests. Loads
Figure 30. Percentile plots for PERSEUS’s concept-based navigation support service capacity-planning/soak tests.
Figure 31. Percentile plots for NavEx’s concept-based navigation support service capacity-planning/soak tests.
Figure 32. Percentile plots for PERSEUS’s topic-based navigation support service capacity-planning/soak tests.
Loads meeting the cap criteria are marked by call-outs
Table 9 provides a summary for in Figure 29 through Figure 32. Here for each of the tested adaptation technique implementations and across all request complexities, the highest load that is successfully tolerated is given. Shaded cells correspond to request complexities that are more probable to occur (judging from our experience).
We can see that social navigation support, as expected, copes with performance challenges the best among other PERSEUS-based adaptation methods. It successfully tolerates loads of 40 and 80 ms between requests for 10 and 20 resources per request, respectively. PERSEUS’s concept-based navigation support is slightly less scalable. It successfully copes with a load of 80 ms between requests for both 10 and 20 resources per request, while for higher request complexities its performance is visibly worse.
Table 9. Maximal successfully tolerated loads for tested adaptation techniques. Shaded cells mark expected request
complexities.
Adaptation technique Maximal tolerated load, ms between requests Request complexity 5 10 20 30 40 PERSEUS’s social 20 40 80 80 80 PERSEUS’s concept-based 40 80 80 160 160 NavEx’s concept-based 20 40 80 80 80 Request complexity 5 (50)* 10 (100) 15 (150) 20 (200) PERSEUS’s topic-based 40 80 160 160
* Here the first-order object is not a resource, but a folder. The total number of resources in folders is given in brackets.
At first glance, PERSEUS’s topic-based navigation support is the least scalable. As request complexity grows, its maximal successfully tolerated load quickly decreases. However, we should bear in mind that, while the nominal request complexities are from 5 to 20 folders, the number of underlying resources is from 50 to 200 and for an expected request complexity of 10 folders (100 resources), maximal successfully coped with load of 80 ms between requests is a high mark to score. Also, since topic-based navigation support is intended for index (folder) nodes of the hyperspace that are accessed less frequently, the numbers of supported users are actually the highest among all tested adaptation techniques.
NavEx’s implementation of concept-based navigation support, as expected, wins by one step of the used load scale over PERSEUS’s implementation of the same technique and is
identical to the characteristics of PERSEUS’s social navigation support technique. The reasons for this not very large advantage are discussed in Section 5.2.
An important component of our evaluation is not only quick responses to requests for adaptation, but also the validity of the responses. Figure 33 provides a summary of the number of erroneous responses to adaptation requests. Four sub-graphs correspond to four different techniques tested. Each sub-graph visualizes the number of errors across load intensities (delays between requests) and request sizes. Shaded areas correspond to the most expected request sizes and the load intensities that were successfully tolerated (as summarized in Table 9).
Figure 33. Summary of the errors for all adaptation techniques across request sizes and loads (delays between
requests). Shaded regions correspond to expected request complexities and load characteristics up to maximal successfully tolerated ones. Only errors for requests with below 90th perentile delay are considered.
As it can be seen in Figure 33, PERSEUS’s social navigation support and topic-based support and NavEx’s concept-based adaptation support are predominantly error-free until the 90th percentile of the request delays – our cap. PERSEUS’s concept-based adaptation support
has a fair amount of errors, especially for tougher loads and more complex requests. However, all of the targeted request complexities and successfully tolerated loads are error-free (until the 90th percentile of the request delays).