Once our survey was designed, prepared and piloted, it thus needed to be given to as wide a participant pool as possible. There were a number of ways through which we recruited participants, who ultimately came from a range of different demographic groups.
6.5.1 Recruitment
University MemosOur primary method of advertising the study was by ‘Wednesday Memos’, a system provided by the University of St Andrews whereby individuals are each provided with a weekly list of announcements. These lists are prepared separately for undergraduate students, postgraduate students and staff, but with no more personalisation than that. In order to attract as wide a range of individuals as possible, we requested that our adver- tisement be sent to all three groups.
Our intention with such a broad advertisement was to give as little information as possible, to minimise self-selection biases, while ensuring participants knew all relevant practical details. Nonetheless we felt it necessary to include at least a general domain area, as not doing so would be unorthodox for such memos and might, as such, put off a number of readers. We mentioned the financial incentive available in order to provide an obvious reason to participate other than the subject matter.
The three memos were identically worded (and included a minor accidental punctua- tion error), as follows:
Participants wanted: native English speakers for language survey
Are you a native English speaker? Would you like a £5 Amazon voucher? Participants are needed for a one-hour study on automatically produced translations within the School of Computer Science North Haugh). For more information contact Martin McCaffery at [email protected]. (Ethi- cal approval: CS12370)
When interested parties got in touch by email, they were informed that the study would likely take somewhat less than an hour, but were given little more information beyond that which was needed to organise a time to meet in person. They then arrived and were given the various information sheets and the survey; after completion of all of these, no further communication was made. Note that the vast majority of partici- pants took significantly less than the stated hour to complete the survey, with most taking approximately 30-40 minutes.
Using the Memos system, we were technically able to reach the entire body of ap- proximately 10,660 students [Higher Education Statistics Agency, 2016] and 2,485 staff members [University of St Andrews, 2015] (approximated using data published for the academic year 2014/15). In practice, many may not have received or read the memos for a variety of reasons, but we have no means of estimating the number of such cases. Of
6.5. PARTICIPATION 101
Group Advertised to Responded Participated University members (via Memos) 13145 (approx.) 29 18
St Andrews Quaker Meeting 30 6 5
University of St Andrews Jujitsu Club 6 6 5
Other individuals 8 8 8
Table 6.4: Breakdown of the various groups of whom members participated in the survey those who did read the memos, 29 individuals expressed an interest, of whom 18 actually participated in the survey.
Word of Mouth
In addition to the University’s Memos system, a number of participants were recruited by simple word of mouth. This was in order both to increase the quantity of judges and judgments we received, and to broaden the scope of our participants beyond those University members who were likely to read the memos they received. Only one group was truly unrelated to the University, while the personal interaction with the others was in addition to, rather than instead of, them.
Several groups were approached, fitting into three broad categories. First, the St Andrews Quaker Meeting is a religious group, unrelated to the University and primarily composed of individuals aged over 30. Next, the University Jujitsu Club is a martial arts group affiliated to the University. Very few members of the former group would have received the University memos, while all of the latter group would have done. Finally, a small number of individuals were approached personally about their participation; these were all members of the University. The numbers within each group can be found in Table6.4.
6.5.2 Demographics
While the original memos publicised to the entire University restricted participants to being native English speakers, no other communication forms made this restriction. This was because while we expected the majority of our participants to be found within the University, and thus wanted to restrict that pool to those who would have the most reliable insights into the English language, we nonetheless wished to have a number of judges for whom English was not their native language for the sake of flexibility, as described in Section6.2.1. In the end, we had two participants for whom English was not their native language, with participant IDs of 17 and 38.
Similarly, while no explicit restrictions were made on the grounds of disability, we had only a single judge who declared any form of disability which might have affected their reading ability; this participant, with ID 32, has dyslexia.
Our participants ranged in age from 18 to 74, with the vast majority being at the younger end of this range. The vast majority considered themselves simply British, while a handful considered themselves specifically Scottish. We had five more female participants than males. Details on all three of these aspects can be found in Figure6.5.
0 1 2 3 4 5 20 40 60 Age in years Number of par ticipants Age of participants 0 5 10 15 20 Female Male Gender Number of par ticipants Gender of participants 0 5 10 15 Amer icanBritish
British & Amer ican
British & A ustralian British ColombianBritish − P
akistaniDanishFinnish Irish Norw egian Scottish [Not pro vided] Self−identified nationality Number of par ticipants