Testing and modifying the SE-IAT website

Before using the newly developed website to collect data, it is important to test the practicality and face validity of the implemented study materials online. A focus group was organized to go through all the measures in the website using their own computers to see if they can successfully complete the study online.

4.4.3.1. Participants

A focus group of 6 participants was organized to test the link. According to Morgan (1997), the optimum size of a focus group should be 6 to 10. Moreover, Krueger and Casey (2009) have also urged caution of the use of homogeneous group - a group of people who have a common background, position or experience – because homogeneous groups have their own well-established dynamics that can influence

contributions. Considering these requirements, the gender and academic majors of the participants in the focus group are balanced. Moreover, the focus group must be able to represent participants in the main study. Given that I proposed to investigate stereotypes of empathy in scientists among UK university students, the final participants in my focus group were 6 students from the University of Cambridge. The information about age, gender, major subjects and nationality of the 6 participants is shown in Table 4.6.

Table 4.6 Demographics of the focus group for testing the initial IAT website

No. Gender Age Subject Nationality

1 Female 32 Pathology Chinese

2 Female 26 Education British

3 Female 27 Psychology British

4 Male 23 Economics Chinese

5 Male 21 Mathematics British

6 Male 21 Medicine British

4.4.3.2. Procedure

The focus group was required to go through all the study materials implemented in the website using their own personal computers. The main purpose is for them to review the practicality and face validity of the online SE-IAT. Face validity is the degree to which a measure appears to be related to the specific construct it is supposed to measure, in the judgment of non-experts such as test takers (Furr & Bacharach, 2008). Before participants start the test, they were informed of the research aim and were encouraged to give detailed advice on how to improve the test website in terms of revising the content of SE-IAT as well as making the whole procedure easier to self-administer from the perspective of test takers. Participants were asked to give answers to the question "Did you have any difficulty completing the study? If so, what would you suggest the

stimulating discussion among participants to bring up any important issues that had not been identified in the design phase. As a result, several problems were pointed out and modifications were made accordingly. Any item that the focus group suggested to be unrelated to the supposed construct were dropped or revised. Instructions that were found unclear were rephrased. The detailed updates are presented in the following paragraphs.

4.4.3.3. Results and updates

The newly developed online SE-IAT was successfully run with all participants' personal computers with different operating systems including Microsoft Windows and Mac OS. Reaction time was recorded accurately in milliseconds. However, there are six issues observed and the website was updated accordingly.

4.4.3.3.1. Items for Science and Liberal arts were changed from occupations to subjects

After discussion of the focus group, items representing target categories were changed back from occupations to subjects for two reasons. First, the test takers reported that they found it easier for them to associate subjects than occupations with target categories of Science and Liberal arts. One of the test takers reported: "I would prefer

subjects to occupations as the stimulus items. I think the subjects are related to the label Science and Liberal arts more directly. Instead, occupations are more likely to be related to the label Scientists and Artists." Moreover, some occupations were found too

ambiguous for test takers to sort into just one category. For example, one test taker who studies mathematics reported: "I found Philosopher could be sorted into both Science

and Liberal arts because several mathematicians were also known as philosophers, such as Bertrand Russell and Rene Descartes." On the contrary, he said "But I would have no problem sorting Philosophy into the category Liberal arts given that philosophy is generally known as a subject in liberal arts." In this case, the stimulus

items for Science and Liberal arts were changed from occupations to subjects, which are again in line with the items in the well-established Science-Gender IAT and Science-Rationality IAT.

4.4.3.3.2. Ambiguous items with double membership were identified and dropped

In addition, discussion of the focus group also revealed certain ambiguous items that could be sorted into more than one category. The item anthropologist and educator were reported to be confusing because some anthropology research also draws upon physical and biological sciences and some educators can be specialized in science education. Therefore, these two items were replaced with items of little confusion such as Literature and Fine arts to represent the target category Liberal arts. Test takers reported no confusion with stimulus items for attribute categories of empathy and

rationality.

4.4.3.3.3. Wording of the attribute items for Empathy and Rationality was polished up

Regarding wording and phrasing of the test, English native speakers in the focus group also gave three suggestions. First, all items for attribute categories of Empathy and

Rationality were unified into adjectives. For example, the word emotion was adjusted

to emotional to be identical with all the other adjective items representing the category

Empathy. Second, the compound word "perspective-taking" was replaced by its

synonym "thoughtful" for it was the only two-word phrase, making it different from all the other one-word items that might lead to construct-unrelated variance. Third, one of the test takers pointed out that the adjective "affective" in the Empathy category was easily mistaken for "effective", which could be sorted into the opposite Rationality category. To avoid a confounding effect, the word "affective" was replaced by a more frequently-used word "affectionate". Table 4.7 displays the updated items for the SE- IAT.

Table 4.7 Updated items for the preliminary SE-IAT (2nd _version)

Science Liberal arts Rationality Empathy

Chemistry Fine arts Consistent Affectionate

Physics Linguistics Coherent Considerate

Mathematics Philosophy Deductive Caring

Engineering History Logical Emotional

Computer science Literature Organised Empathetic

Astronomy Sociology Rational Feeling

Biology Politics Reasoning Thoughtful

Geology Music Systematic Sensitive

4.4.3.3.4. Device requirement was added in the instructions

Some participants tried to click on the test link with their tablets and found it unable to be run on tablets or smartphones. Therefore, a requirement specifying the valid devices for the test was added in the instructions. Participants are asked to use desktop or laptop computers to complete the test. Smartphones or tablets without a physical keyboard were warned to be invalid for the test.

4.4.3.3.5. A test progress indicator was added for participants to track their progress

When completing 200 trials in the SE-IAT, participants reported feeling bored and impatient during the process for they had no idea how long it would last. A test progress indicator (e.g.,"Block/Question 1 of 7"), which shows the current phase of the study, was added for participants to track their progress during the test in order to reduce the fatigue effect and improve concentration on the study.

4.4.3.3.6. A question for academic discipline identification was added

A question asking the participants' self-identification with science or liberal arts was added considering the situation that some participants may major in integrated subjects. It has been considered not only inappropriate but also impossible for the researcher to judge the academic discipline that the participants affiliated to. Therefore, it is essential for participants themselves to report their identified academic discipline.