Study 4 Survey Comments

Additional user feedback was collected via the Study 4 survey questionnaires, which allowed the participants to comment on each of three website versions they interacted with. Fifty seven participants provided comments for at least one of the website versions they evaluated.

Manual analysis of the textual data revealed that users were almost entirely focused on the availability of cancer-related content or lack thereof - possibly as website content may be the target users’ main concern. They, hence, often failed to pay attention to the main aim of the experiment - to notice the distinguishing levels of personalisation. One of the

163

participants (Par8) summed up the main issue – the need to have a longer exposure to the personalised website, in the comment for the EXT website version:

“I don't think I had enough time to understand the types of personalisation that were being provided to me. I think I would need more time in the system, and more 'real- world' usage in order to experience the changes in the content.”

Participant comments clearly show they failed to distinguish between the websites; they would either give one comment referring to all three versions, or three comments of similar nature. Often, the feedback was very generic, and addressing aspects that were the same for all website versions, for example: lack of content, content mainly in Bosnian, the website being inactive, or it being a simple website, easy to navigate, with a nice design, etc. Some participants commented on how satisfied they were with the content on one website version, but not on the other. For example, Par1 (translated from Bosnian):

Comment for EXT: “...the amount of content is minimal. I expected more information about the illness itself, symptoms, treatment and patient experience.”

Comment for PRE: “I am pleasantly surprised by the more extensive content on this website.”

Moreover, comments related to the lack of content were more frequent for the personalised website versions, where it would be expected that content recommendations would imply greater relevance and accessibility of content. Nevertheless, in designing the Study 4 experiment, it was ensured that all website versions had the same amount and type of content, so that it would not be the factor affecting the difference in perception about the website versions.

The participants who stated they preferred the website version with emotion-based personalisation (Phase III PORT website) gave the following reasons (direct quotations): Par9: “After going through all websites I definitely have to say that grey one [EXT] was

164

I like how I could see forum and chat immediately because I think this is one of the most important things when it comes to these and similar websites.”

Par10: “I found this version of the web site far more useful. The best features for me include background changing color, making it more pleasant and filtering the articles based on my stated interest.”

Par11: “I liked the expanded emotion tool...”

Par12: “The good news is that the interface itself is extremely well-designed, and very user friendly with loads of potential to be a fantastic website.

Some of the participants also pointed out what they perceived were the negative aspects

of the EXT version (Phase III PORT website). Hence, Par12 further continues in the comment:

“Unfortunately, that is where the positive feedback ends. The article recommendations in my profile had absolutely nothing to do with my previous searches, and even when I deselected the option to show information about women's cancer, the *only* recommendation in my profile remained an article about food related specifically to women and cancer. This was, overall, a very frustrating task12 that took **significantly longer** than anticipated... Moreover, 90% of the time I updated my emotions and click "save" -- it did *not* correctly save, the panel immediately popped back up, and I had to redo it again.”

Interestingly, even though the comment gives an impression the user is frustrated, Par12 was the only participant that additionally contacted me via e-mail and offered his assistance in identifying the system bugs, due to his interest in this type of a cancer website. Indeed, the Phase III PORT website had several bugs in the functionality of the Emotion Tool. Due to this, a few users claimed to have experienced the Emotion Tool to reappear immediately after having reported emotions. The Emotion Tool issues were repaired in the Phase IV website version. Moreover, the interests-related optimisation of

165

content recommendations was introduced in Phase IV. In Phase III, the filtering of recommendations based on user interests was not as strictly defined.

Overall, user comments were helpful for identifying website services which needed further improvement, which was accomplished in Phase IV of website development. Moreover, the comments highlighted areas of the experiment design which needed adjustment. Hence, several refinements were made in Study 5. For example, Study 5 experiments included a statement informing participants that the website’s content is not the focus of the evaluation, but its features.