5.2 Pilot Studies
5.2.2 Pilot 2: Disguising the Cognitive Reflection Test
To the best of my knowledge, the CRT has not been used in any longitudinal studies before now. A particular concern is that the ‘trick’ nature of the ques- tions might make them more memorable than other measures at post-test and that this will affect the way in which participants (at least those who inhibited their intuitive response and are thus aware of the ‘trick’) respond to them.
An attempt to address this issue was to mix the CRT questions in with non-trick mathematical word problems, so that the trick might be less salient at repeated testing points. However, it is possible that this could alter the test properties in some way, perhaps meaning participants no longer process the CRT questions in the way they otherwise would have. The aim of the pilot study reported here was to assess whether this was the case. Participants either saw the three CRT questions followed by the three non-trick problems or they saw the six items in a random order. The three non-trick questions, shown in Figure 5.3, were taken from the Woodcock-Johnson III Applied Problems subtest and were selected for being mathematically simple and of a similar length to the CRT questions. If mixing the questions does not alter the test properties then scores should not significantly vary between groups.
Participants Participants were recruited, without payment, through websites that advertise internet based research studies where they saw a brief description and could open the study webpage. Fifty-four participants completed all six questions and were included in the analysis. The participants were aged 18- 59 (M =29.60, SD =10.40), and 23 were male and 31 were female. Twenty-
1. If a girl saved £1 each week for 1 year, how much money would she have at the end of that year?
2. If a dog can run two and a quarter miles in one hour, how long would it take the dog to run four and a half miles at that same rate?
3. Mileage varies from car to car. Judy's car gets 22 miles to a gallon of gas, and Bob gets 35 miles to a gallon of gas. How many miles can Judy drive on six gallons of gas?
Answers: Q1 = £52, Q2 = 2 hours, Q3 = 132 miles
Figure 5.3: The three items from the Woodcock Johnson III Applied Problems subtest that were used in Pilot Study 2.
seven were randomly allocated to the mixed condition and 27 to the non-mixed condition when they opened the webpage.
Procedure Participants first saw a page providing information about the study. They were told that if they took part they would be asked to answer six arithmetic word problems which would take no more than 10 minutes. They were also told that their data would be kept confidential and used for research purposes only. They were asked to select whether they wanted to seriously participate or just browse the pages before continuing to the study, and only those who wanted to seriously participate were included in the analysis.
Next they were asked to report their sex, age, degree subject (if applicable), and whether their native language was English or non-English. The six questions were each presented on a separate page and required participants to type their answer into a blank response box. In the mixed condition, the six questions were presented in a random order. In the non-mixed condition, the three CRT questions were presented first in a random order, followed by the three non- trick questions in a random order. Finally, participants were thanked for their participation and given my email address in case they wanted to request further information or to withdraw their data (none did).
Two Mann-Whitney U tests were used to compare performance across the two conditions: one analysing number of correct responses to the three CRT ques- tions, and one analysing number of intuitive responses (it is possible, but rare, to give a non-intuitive but incorrect response meaning that these measures are
Figure 5.4: Mean number of intuitive responses in the mixed and non-mixed conditions (error bars represent ±1 SE of the mean).
not exact inverses of each other).
Number of correct responses were not significantly different in the mixed (M =2.00, SD =0.73) and non-mixed groups (M =1.93, SD =1.07), U (54)=358.0, z = −.12, p=.904, r = 0.02. Similarly, number of intuitive responses were not significantly different in the mixed (M =0.89, SD =0.70) and non-mixed groups (M =0.81, SD =1.04), U (54)=417.00, z = −.98, p=.325, r = 0.13, see Figure 5.4.
Discussion and Implications
The aim of this pilot study was to assess whether inter-mixing the three CRT questions with three non-trick questions would affect the way in which parti- cipants respond to the CRT questions. There was no evidence that this was the case. Neither the number of correct responses nor the number of intuit-
ive responses given to the CRT questions significantly differed when they were inter-mixed with non-trick questions compared to when they were seen first.
One thing this result suggests is that the reflective level determines on an item-by-item basis whether the algorithmic level should be used. It is not the case that answering an item with no intuitive answer which thus requires al- gorithmic level processing (any of the Woodcock-Johnson questions) sets the reflective level to a ‘use algorithmic processing’ mindset. If this were the case the presence of an item that requires algorithmic processing should mean that the intuitive answer is inhibited for subsequent CRT questions, which was not the case here. This fits with the established finding that participants do not tend to get all CRT questions right or wrong, rather they may get one or two right and give the intuitive responses to the rest (Frederick, 2005).
Returning to the purpose of this pilot study, the results indicate that the three CRT questions and three Woodcock-Johnson questions can be randomly inter-mixed in the longitudinal study without affecting the way participants re- spond to the CRT questions, which are the real measure (responses to Woodcock- Johnson questions will not be analysed). The Woodcock-Johnson questions ap- pear to be simple enough as to not influence the reflective level in a task-general manner. It is hoped that their inclusion will reduce repeat-testing effects on CRT performance.