Experiment 2(b): To examine if users prefer the user-controlled

provided by the MDL concept as applied in the IPNS more than non- personalised systems

Task: The participants were requested to provide subjective feedback about the three systems they had been using, make a rational comparison between them, and answer eleven questions by which to emphasise their opinions. Each group was assigned the same task as shown in Table 8-12. The questionnaires for Experiment 2(b) can be found in Appendix B (III-b: Q18: page 198) and (IV-b: Q19: page 206) for the Group 2.1 and 2.2 subjects, respectively.

Group Task Group 2.1

and Group 2.2

Make a subjective comparison between the three systems (system Non link, system All links, IPNS)

that have been used. Answer the questionnaire.

Table 8-12: The allocation of the subjects in the Experiment 2(b)

Result for Experiment 2(b):

Appendix C (IV: page 218) presents all the data from the trial for the experiment 2(b).

Hypothesis 3:

H₁: Users prefer the user-controlled adaptation provided by the MDL concept as applied in IPNS than non-personalised systems.

H₀: Users have no preference between the user-controlled adaptation provided by the MDL concept as applied in IPNS and non-personalised systems.

To test this hypothesis, the question (‘I would prefer to use the following systems (system Non link, system All links, and IPNS) to locate the documents and perform all required tasks in the future’) in the questionnaire was analysed.

The ‘one-sample Chi-square’ was employed for the purpose of data analysis.

Chi-square is a statistical model used to make predictions about categorical variables, or counting how many different subjects will fall into one or more categories, where each subject can be assigned to only one category (Greene and D’Oliveira, 1999), and the ‘one-sample Chi-Square’ is used to compare observed frequencies with what would be expected if the frequency was equal for all events (Foster, 2001). As shown in Table 8-13, the null hypothesis for Hypothesis 3 was rejected (p = 0.039, i.e. p < 0.05), indicating that the obtained frequencies differed from those expected under the null hypothesis by more than could happen by chance (Howell, 2002). That is there was significant user preference in using the system (amongst the three systems).

Subjective feedback on the user preference for a system

Observed N Expected N Residual

System Non link 4 5.3 -1.3

System All links 2 5.3 -3.3

IPNS 10 5.3 4.7

Total 16

Chi-Square(a) 6.500

Df 2

Asymp. Sig. 0.039

a. 0 cells (.0%) have expected frequencies less than 5. The minimum expected cell frequency is 5.3.

Table 8-13: Result produced by SPSS for the three systems in Experiment 2(b)

To further investigate by comparing between the non-personalised systems (grouping together system Non link and system All links) and the IPNS, Table 8-14 summarises this finding.

System Observed

(O)

Expected

(E) O-E (O-E) 2

(O-E)2/E χ2 Non-Personalised systems 6 10.66 -4.66 21.716 2.037 6.1298

IPNS 10 5.33 4.67 21.809 4.092

Table 8-14: The statistical result for non-personalised systems and the IPNS

As can be seen, the value of Chi-square (χ2) (hand calculated) = 6.1298 on 1 df, whereas from the χ2 distribution table, χ2 (0.05) (1 df) = 3.84 (i.e. cuts off the upper 5%

of the distribution)(Howell, 2002). As χ2 obt > χ2 crit (i.e. 6.1298 > 3.84 ), the null

hypothesis for Hypothesis 3 was therefore rejected which indicated that users significantly preferred the MDL concept as applied in the IPNS than non-personalised systems.

Hypothesis 4:

H₁: The IPNS with the links presentation and personalisation tools is useful as it allows the selection of links to be displayed based on users’ preference.

This hypothesis was tested by taking into consideration the question (‘I found the IPNS prototype was useful as it allowed me to select the links to be displayed on my preference’) in the questionnaire. The ‘one-sample Chi-Square’ model was chosen as a statistical model instead of the ‘one-sample t test’, as the author was purely interested in the data frequency counts and in comparing observed frequencies with expected ones (Foster, 2001; Diamond and Jefferies, 2001). However, the one-sample t test might have been more appropriate if the emphasis had been on the ordering of the agreement, how the score obtained varied from the neutral point (the test value), or the relationship amongst the agreements.

As shown in Table 8-15, the ‘one-sample Chi-Square’ technique resulted in the rejection of the null hypothesis for Hypothesis 4, i.e. Chi-Square = 5.281, p = 0.022 < 0.05. This indicates that the IPNS with links presentation and personalisation tools was significantly useful as it allowed the selection of links to be displayed based on users’ preference.

Observed _Value Expected _Value Residual _(O-E) Chi-Square_(a) df Asymp. _Sig. Agree 15 5.33 4.33

Not sure/ Disagree

1 10.66 -4.33

5.281 1 0.022

(a) 0 cells (.0%) have expected frequencies less than 5. The minimum expected cell frequency is 5.3.

Table 8-15: Result produced by SPSS for the subjective feedback on the ‘usefulness of the IPNS’

Hypothesis 5:

H₁: Users find that they have control over link presentation and personalisation in IPNS.

H₀: Users find that they do not have control over link presentation and personalisation in IPNS.

The question (‘I found the IPNS enabled me to have control over the link presentation and personalisation, that is, I can select the links to be presented’) in the questionnaire was analysed for this purpose. Again, the ‘one-sample Chi-Square’ test was chosen, and its result, as shown in Table 8-16, revealed that the null hypothesis for

Hypothesis 5 was rejected, i.e. Chi-Square = 12.500, p = 0.000, i.e. p < 0.05. Statistically, this confirms that users felt they were in control of link presentation and personalisation in the IPNS.

Observed Value Expected Value Residual (O-E) Chi-Square (a) df Asymp. Sig. Agree 12 5.33 6.67 Not sure/ Disagree 4 10.66 -6.66 12.500 1 0.000

(a) 0 cells (.0%) have expected frequencies less than 5. The minimum expected cell frequency is 5.3.

Table 8-16: Result produced by SPSS for the subjective feedback on the ‘user control’ aspect of the IPNS

Other statistical results from the trial can be summarised in Table 8-17. The column 1 df represents the result gained from grouping ‘not sure and disagree together’ and compared it against the ‘agree’ opinion.

Statistical results Subjective feedback

1 df (**) I found the IPNS prototype helped me find the document. Chi-Square = 21.125

p=0.000 I found that there were too many links in System All links and

some of these links were what I had already known.

Chi-Square = 9.031 p=0.003 I think the MDL concept and the Personalised Links assistant

interface was useful as it allowed a same keyword to become different links based on the user’s selection.

Chi-Square = 16.531 p=0.000

I think the MDL concept and the Personalised Links assistant interface could solve some of the problems of too many additional links inserting into the document, whereby these links

might not be of concerns, not only in this specific domain, but also in bigger hyperspace.

Chi-Square = 16.531 p=0.000

I think the links presentation and personalisation interfaces were user-friendly and easy to use.

Chi-Square = 6.125 p=0.013 I would prefer to user the following system (non-personalised

system, IPNS, none) for links presentation

• IPNS

Chi-Square =21.125 p=0.000

(**) It is correct to carry out a one-tailed chi-square only when there is just 1 df (Howitt and Cramer, 2005)