Setup of User Study

3.2.1 Design of User Experiments

Thirty subjects were recruited to participate in a user study through in-person contacts or via ads distributed in email lists. Information seeking behavior of users can be affected by many environmental factors in the medical domain. This makes conducting evaluations associated with serendipity a challenge. In order to reduce the impact of users’ heterogeneity, the participants recruited in this study were college staff members because they came from a group of people who not only had similar experience but also shared common campus environments. The total time allocated to complete the study was roughly 1.5 hours. As compensation for the time spent, each participant received $40. The study was divided into two parts. The main part involved the presentation and interaction with news items. The other part consisted of the pre-session and post-session surveys. Considering the stringent requirement related to security of health-related information, this study adopted de- identification strategies. Each participant was given one of 30 unique identification codes (UIC) which are alphanumeric and 10-digital, which were generated and randomized prior to the study. Any of RA, KA and KAA methods can be enabled by using the 10 identification codes. This setting was adopted to ensure that each serendipity level had the same number of participants. Participants’ data, such as results of questionnaires and logs of interface actions, were linked using the identification code. The MedSDFilter system and the database were

installed in a laptop with password protection. It was brought to the testing sites convenient to the users on the UNC-CH campus. When the test was completed, the laptop was brought to the researcher’s office with password-protected access, which is located in the Laboratory of Applied Informatics Research (LAIR), RM. 300 Manning hall on the UNC campus.

The study was conducted under a given scenario. At the beginning of the study, the participants were requested to read through the description of the scenario as shown below.

“For this study, we would like you to imagine that you are interested in learning about new medical information, especially as it relates to you, your family, and friends. Since there are a lot of medical-related news articles, you have decided to use a filtering system that will recommend articles of possible interest. The system allows you to indicate areas of interest and it will recommend articles that you can read during your free time.”

The scenario of this study is based on typical filtering settings in which news articles are ranked according to the strength of users’ interests on article topics. From the scenario description, users may know that filtering system delivers relevant information in a ranked order based on their interests in medical topics. This implies that users are not completely blind to the utility and application of topic ranking. A static ranking of interesting topics is likely to enhance the possibility that users find the patterns of how articles are sorted. In many situations, the designer of user study wants to prevent users from finding patterns through strategies like randomness. One concern of these designers is that users may apply the found pattern to guide their information seeking behaviors and that this changes the initial context of information seeking. However, it is not a problem in this study because the pattern

of topic ranking in personalized information filtering may be expected by users, and we reinforce the utility of the profile by presenting content based on it.

3.2.2 Procedure of User Experiments

Before each experimental session, a consent form was presented to the subjects. After they reviewed and signed the form, a one-page instruction was shown. The instruction provided a description of the study scenario and the experiment procedures. The subjects were told that if they have any issue after reviewing the instruction, they can ask the researcher to clarify it. Then, the subjects accessed MedSDFilter system and entered unique identification code to start the experiment. Each user was asked to complete 10 filtering sessions. One short survey was given before and after the experimental session. The consent form, pre-session survey (2c, 2d and 2e in Table 3.3), in-session ratings, and post-session survey (2j, 3a and 3b in Table 3.3) were adapted from previous established user experiments (Fan et al., 2012). They were modified to fit the current study needs. The process flow for the experiments is presented in Table 3.3.

It is common that that people follow different paces in reading medical articles. Therefore, to make the study realistic, each session was limited to 10 minutes (i.e. an estimate based on average reading time from the pilot study). When a session times out, it is switched to the next one automatically. If a subject completed reading all the articles of interest ahead of time (i.e. <10 minutes), that individual could go to the next session by clicking on the “Go to next session” link at the top right corner of the window. In order to ensure users completed reading each article of interest, the system double-checks by generating a popup box to collect his or her affirmation. The session switch occurs only if

users check the box to indicate that they have completed reading all the articles of interest. Participants were made aware of the time limit per session and the session switch feature during the introduction.

Table 3.3: Process Flow for User Experiment

Before the experiment on MedSDFilter system

1a. Review and sign consent form

1b. Review study instruction (scenario description included)

Conducting the experiment on MedSDFilter system

2a. Launch MedSDFilter system (and its database) 2b. Enter UIC to load one serendipity model

2c. Questionnaire to obtain user’s basic information 2d. Questionnaire to capture user’s interest topics 2e. Questionnaire to capture user’s interest strength 2f. Start one filtering session

2g. Read and rate articles based on user’s interest 2h. Complete the filtering session

2i. Repeat [2f, 2h] for nine additional times 2j. Questionnaire to capture user’s current interest 2k. Log out MedSDFilter system

After the experiment on MedSDFilter system

3a. Questionnaire to capture user’s perceptions on serendipity feature

3b. Interview to clarify user’s inputs in the questionnaire 3c. Compensation, receipt