Discussion

This thesis has been undertaken with two research objectives in mind. With respect to the OH research, the work was aimed at providing a new application of the link augmentation technique by looking at a different view of representing a linkbase which stores link structures more effectively than traditional linkbases and solving the link overload problems caused by the conventional link augmentation technique. Secondly, regarding the AH research, this work attempted to deal with some of the AH criticisms that users do not understand or have control over adaptation behaviour of the system by faciliting user’s control over personalisation.

The system implementation has established the affirmation that the MDL concept has presented a different view of representing a linkbase for link personalisation, which resulted in additional functionality to support the process of inserting additional links into the body of a document. The main emphasis of the evaluation study was therefore carefully chosen and designed to prove that by allowing users to have control over personalisation of links, IPNS would give the affordances the user expected from the adaptation. In this context, ‘control’ is enabling users to see the working behaviours of the system, by means of the direct manipulation of the MDL and other linkbases. In doing this, redundant links are removed and therefore the link overload problem is reduced.

The studies were carried out to reflect the above mentioned principle of evaluation of IPNS. The heuristic evaluation reviewed by experts was purposely assigned to examine the strengths and weaknesses of the applicability of the user- controlled adaptation provided by the MDL concept as applied in the IPNS, and the empirical study was deliberately designed to study the usefulness of the user adaptation provided by the MDL concept using usability as the evaluation criteria. The

experiments were intended to contribute these evaluation rationales by comparing the prototype system with non-personalised systems which served as control systems.

First, the result of the heuristic evaluation revealed that the user-controlled adaptation provided by the MDL concept and its IPNS prototype conformed to its requirements and most of its heuristics. There was no major error or problem with the integration of the MDL concept into the working prototype. The review has provided a rational indication that the user-controlled adaptation provided by the MDL concept was applicable, flexible and efficient, and far from being too abstract. However, no system is yet said to be perfect or error free. This inevitably encompasses our prototype system as well. Informative comments by the experts were elicited to give an insight into the implementation of the MDL concept on a bigger scale as an application.

Secondly, concerning the empirical study, the objective and subjective data were gathered and analysed by statistical techniques. The IPNS prototype was compared against other two systems serving as non-personalised systems (‘control systems’) –

system Non link (i.e. system with no additional links given) and system All links (i.e. system with additional link insertion but users have no control over its presentation). The result for the first experiment has suggested that users could benefit from the IPNS

in navigation, as the percentage of task completed was significantly improved by the set of links presented in comparison to navigation without the presence of personalised features (system Non link).

However, an insignificant difference between the IPNS and system All links in the second experiment was discovered, meaning that the set of links presented in IPNS did not increase the percentage of navigation completed (speed of navigation). This could possibly be due to one of the followings reasons:

• When a subject domain, like our domain, is not so large, the system with non- personalised link augmentation (‘system with automatic-all-links insertion’) might not yet appear problematic to some users. Some users commented that they preferred to see all links, hoping that the links would stand out as the answers themselves, or lead to some relevant piece of information. However, if the domain was larger and it was distributed across Web services environment, the problem with having no control over link presentation and personalisation might come into view more obviously, when every keyword could become a link.

• Another approach might have been to allocate users more tasks to perform in order to gain a significant result in capturing the percentage of navigation completed (speed of navigation), i.e. more terms or phrases needed to be assigned for users to search for (we asked users to look up only four keywords in the experiment conducted).

• The speed of navigation might not have been an appropriate dependent factor to measure in our studies. A better measurement might be to count the number of links the users used in finding the answers for the task. The more links given, the more likely that the user would need to spend time looking; hence, reducing their speed of navigation.

Having said this, the user-controlled adaptation provided by the MDL concept as applied in the IPNS allowed the presentation of links to be personalised (based on user’s expertise dimensions and preference) with the options of having all links visible, only basic links or advanced links appearing, or no additional links inserting into a Web page at all. As a matter of fact, the function that the system All links provided was just a subset of all other functions the IPNS could perform.

With respect to the subjective comments, they were gained from the subjects mentally comparing the three systems (Experiment 2(b)), and from the attitude questionnaire which the users were requested to merely focus on the IPNS prototype and judged its conformity on the Likert scale in relation to the scales of evaluation (Experiment 3). Hypotheses proposed in the study and the two experiments were tested and statistical models were used in verifying the data. Based on the ‘one-sample Chi- Square’ test used for tables containing counts, with restrictions that Expected Value (E) should be five or above for the results of a χ2 test to be valid (Diamond and Jefferies, 2001), the statistical results pointed that users significantly favoured the user-controlled adaptation provided by the MDL concept as applied in the IPNS to the non- personalised systems. Although there were a number of users who would rather use a non-personalised system (i.e. using ‘table of contents’) than using the tools provided in IPNS to locate the documents and perform required tasks due to its simplicity and straightforwardness, their fondness was not significant when comparing to the IPNS’s users.

Similarly, the users reflected that using IPNS with the link presentation and personalisation tools was significantly useful as it allowed them to make selection of

the links to be displayed, and statistically users felt they were in control of link presentation and personalisation in the IPNS.

Positive results were also obtained for the attitude questionnaire on SUMI scales and scales for evaluation of industrial hypermedia. Most of the statements were favourable, particularly the ‘Comprehension’ scale, where 87.5% (twenty one users) – ‘best result’- agreed that the systems tools were satisfactorily presented. Few users mentioned the general user interface matter (which was not in the scope of this work, but it certainly could be improved in the future for a more usable and user-friendly system) and the issue that ‘too many dimensions and too much setting’ might be cumbersome and cause users to be reluctant to use the system. This issue will be discussed futher in the next Chapter. With regard to the statistical analysis of data, the significant results pointed to the users enjoying interacting with the system, the user- controlled adaptation provided by the MDL concept as applied in the IPNS was useful and helpful, and the user adaptation provided by the MDL concept as applied in the prototype allowed users to have control over links presentation and personalisation. Overall reactions gained from the users were impressive and statistically easy and satisfactory.

On the whole, the studies revealed that the user-adaptation approach provided by the MDL concept facilitated the user’s control over personalisation of links. That is, IPNS enables the user to see the working behaviours of the system by means of the direct manipulation of the MDL and other linkbases. Through this adaptation process, irrelevant links are removed and therefore the user-controlled adaptation provided by the MDL concept can help to reduce the link overload problem.

8.6 Summary

This chapter has presented the evaluation that was conducted with the objective of examining the applicability and usefulness of the user-controlled adaptation provided by the MDL concept as applied in the IPNS prototype. The data was gathered objectively and subjectively, and analysed by means of statistical techniques. The results of this evaluation showed a significant level of acceptance from the ‘applicable’ and ‘useful’ aspect of the user-controlled adaptation provided by the MDL concept and its counterpart the IPNS prototype.

The following chapter provides a summary of the research undertaken in this thesis. The contributions this work has made as well as key research issues raised are particularly discussed. The chapter then concludes with the highlights of the possible directions this work could be continued.