Pre and post-task questionnaires

questionnaire measured items such as participants’ knowledge of the task topic and their perceptions of how difficult it would be to search for information about the topic. Responses were measured on a five-point Likert-type scale in which the score of 1 indicated the lowest or most negative responses (e.g., “nothing,” “not at all,” etc.) and scores of 5 indicated the highest or most positive responses (e.g., “very much,” very often,” etc.). Table 8 shows the means and standard deviations of participants’ responses.

Table 8. Means and Standard Deviations for Pre-Task Questionnaire Responses

Task Topic

Pre-task Question Reverse Mortgage Payday Loan Student Loan Total Knowledge of Topic 1.88 (.97) 2.12 (1.27) 2.83 (1.42) 2.28 (1.28) Relevance of Topic 1.88 (1.15) 2.10 (1.28) 3.62 (1.58) 2.53 (1.55) Interest in Learning 2.86 (1.14) 2.98 (1.22) 3.64 (1.30) 3.16 (1.26) Frequency of Searching 1.21 (.57) 1.45 (.83) 2.40 (1.45) 1.69 (1.14) How Difficult to Search 2.48 (.83) 2.83 (1.01) 2.19 (.83) 2.50 (.93)

A one-way analysis of variance (ANOVA) was calculated on the pre-task question item to determine if there were task effects. The one-way ANOVA results shown in Table 9 uses the more robust Brown-Forsythe F-ratio because several of the pre-task questionnaire response data

83

sets violated the homogeneity of variance assumption (Field, 2005, p. 347). There was a main effect for task topic on participants’ knowledge of the task topic, F(2, 114) = 6.87, p < .001, η2 ₌ .101. Bonferroni post-hoc analysis7 indicated that participants’ rated their knowledge of the student loan task topic higher than both the reverse mortgage and payday loan task topics. There was a main effect for task topic on participants’ perception of the relevance of the topic to their own lives, F(2, 114.75) = 20.72, p < .001, η2 = .252. Bonferroni post-hoc analysis indicated that participants rated the student loan task topic as most relevant to their lives, over both reverse mortgage and payday loan task topics. There was a main effect for task topic on participants’ interest in learning about the task topic, F(2, 121.53) = 5.04, p = .008, η2 = .076. Bonferroni post-hoc analysis indicated that participants rated their interest in learning about the student loans topic as greater than both the reverse mortgage and the payday loan task topics. There was a main effect for task topic on how frequently participants reported having ever searched online for the task topic, F(2,79.52) = 16.07, p < .001, η2 = .207. There was a main effect for task topic on participants’ perceptions of how difficult it would be to search on the task topics, F(2, 118.66) = 5.43, p = .006, η2 = .081. Bonferroni post-hoc analysis indicated that participants expected the payday loan task topic to be more difficult to search for than the student loan task topic.

7 _{When using the Brown-Forsythe F-ratio, the degrees of freedom for the post-hoc analysis are taken from} the Robust Tests of Equality of Means.

84

Table 9. Results of ANOVA for Pre-Task Questionnaire, Using Brown-Forsythe F-ratio

Pre-task Question SS dfm dfr F η2

Knowledge of Topic 20.64 2 114.0 6.87** .101

Relevance of Topic 75.44 2 114.7 20.72*** .252

Interest in Learning Topic 15.06 2 121.5 5.04** .076

Frequency of Searching 33.33 2 79.5 16.07*** .207

Difficulty in Searching Topic 8.71 2 118.7 5.43** .081

Note. ** p < .01; *** p < .001

After completing each search task, participants filled out an online questionnaire that contained three post-task questions (Appendix F) and the six-question NASA-TLX mental workload index (Hart & Staveland, 1988) (Appendix G). The questionnaire items were

measured on a 10-point Likert-type scale. During the data analysis, in order to compare the pre- task “difficulty” question with the post-task “difficulty question,” the 10-point scale of the post- task responses was transformed to a 5-point scale, using SPSS to calculate the linear

transformation. Linear transformation approximates the interval scale to the same degree after the transformation such that the interval and other properties of the distributed are unaffected8_.

This enabled comparison for paired sample t-tests of the two items. Table 10 shows the results of the 5-point transformation of the post-task questionnaire items. Questions ask how difficult it was for participants to find relevant items (Difficulty Finding Documents, 1=Very easy, 5=Very difficult), how participants rated their own skills at finding relevant items (Your Ability to Find Relevant, 1=Not good, 5=Very good), and how participants rated the system’s ability at

retrieving relevant items (Rate System’s Ability, 1=Not good, 5=Very good). Across the three

85

tasks, participants rated the difficulty in finding relevant documents an average of 3.85 on the scale from 1 to 5, their ability to find relevant documents an average of 3.9, and the system’s ability to retrieve relevant documents an average of 4.29.

Table 10. Means and Standard Deviations for Post-Task Questionnaire Items

Task Topic Post-task Question Reverse Mortgage Payday Loan Student Loan Total Difficulty Finding Documents 3.71 (1.10) 3.85 (1.13) 4.00 (1.07) 3.85 (1.10) Ability to Find Relevant Documents 3.80 (1.04) 3.81 (1.19) 4.07 (.92) 3.90 (1.06) System’s Ability to Retrieve 4.25 (.77) 4.39 (.56) 4.24 (.79) 4.29 (.71)

A one-way ANOVA was run to determine if there was a task effect on any of the post- task questions. The ANOVA used the Brown-Forsythe F-ratio and indicated no significant differences across the three tasks on the post-task questions, which meant that participants did not find any of the tasks more difficult than others, nor did they believe they performed differently on any of the tasks versus the others, nor did they think the system performed differently for any of the tasks versus the others.

On the pre-task and post-task questionnaires, a question was asked about the difficulty of the tasks. For the pre-task questionnaire, this item was worded, “How difficult do you think it

will be to search for information about this topic?” with 1 = very easy and 5 = very hard. For the post-task questionnaire, the item was worded, “How difficult was it to find relevant

documents?” with 1 = very difficult and 5 = very easy. To understand if there was a difference between the amount of difficulty participants expected searching to be before the task and the how difficult they felt the task was after they completed the task, a paired-samples t-test was conducted to compare the pre- and post-task responses. There was a significant difference in the

86

scores for difficulty in the pre-task (M = 2.48, SD = .833) and post-task (M = 3.71, SD = 1.100) questions for the reverse mortgage task topic, t(41) = -5.047, p < .001. There was a significant difference in the scores for difficulty in the pre-task (M = 2.83, SD = 1.010) and post-task (M = 3.85, SD = 1.134) questions for the payday loan task topic, t(41) = -3.937, p < .000. There was also a significant difference in the scores for difficulty in the pre-task (M = 2.19, SD = .833) and post-task (M = 4.00, SD = 1.065) questions for the student loan task topic, t(41) = -7.705, p < .001. In all cases, participants experienced searching for the task topics as easier than they originally expected.

5.2.2. Cognitive abilities. There were two cognitive abilities measured: perceptual speed and memory span. Perceptual speed was measured using the Finding A’s test from the Ekstrom Kit of Factor-Referenced Cognitive Tests (Ekstrom et al., 1976a). Table 11 shows descriptive statistics for perceptual speed. The average score on the perceptual speed test was 63.1 (SD = 18.2) out of a possible score of 200. The median score for perceptual speed was 59 which was used as the cut-point score for the median split for low and high perceptual speed groups. Participants who scored less than or equal to 59 were grouped in the low perceptual speed group (N = 23) and those who scored higher than 59 were group in the high perceptual speed group (N = 19).

Table 11. Means and Standard Deviations of Perceptual Speed Test (N=42)

Cognitive Test n Min Max M SD

Perceptual Speed 42 30 109 63.10 18.17

87

Figure 14. Score distribution for participants on the perceptual speed test.

Working memory was measured using a memory span test from CogLab 2.0 (Francis et al., 2008). Table 12 shows the descriptive statistics for the memory span test. The average score on the memory test was 5.65 (SD = .644) out of a possible score of 10. The median score was 5.70 which was used as the cut-point score for the median split for low and high working memory groups. Participants who scored less than 5.70 were grouped in the low working memory group (N = 21) and those who scored higher than 5.70 were grouped in the high working memory group (N = 21).

Table 12. Means and Standard Deviations of Memory Span Test (N=42)

Cognitive Test n Min Max M SD

88

The distribution of memory span scores is shown in Figure 15.

Figure 15. Score distribution for participants on the memory span test.

5.2.3. Financial knowledge. Table 13 shows participants’ scores on the financial