Declared pilot test – stage two

Stage two of the pilot test elicited more formal responses to the complete questionnaire. Second year Financial Accounting students from the researcher’s own class were informed about the pilot study during a lecture and invited to provide contact details if they were interested in participating. These students would not be part of the main study. Thirty students volunteered, which represented over half of those present at the lecture. The administration of the pilot survey was strategically delayed a further three weeks thereby helping to create some distance in time and sense of commitment for those who had volunteered. The plan was to allow volunteers ‘a cooling off’ period. When they received the email three weeks later they would have had time to re-consider whether or not they still wished to complete the survey and whether or not it was convenient to do at the time (as it would be for the actual test group). Eighteen of the original thirty volunteers completed the survey (a 60% response rate).

The pilot survey was administered online, using SurveyMonkey, as was planned for the main study. Students also tested the link to a separate page where survey respondents would indicate their preference to participate further with interviews. This separate webpage asked the pilot study participants to note the time it took them to complete the survey and for general feedback about the survey instrument itself. Fourteen students completed this additional form with five of them offering to participate in interviews. The goal for the main study was to interview eight students from each research site so the pilot study response of 28% (5/18) of the surveyed sample was pleasing and indicated that the stated aims of the full study were achievable.

The pilot study data also suggested that the aim of achieving maximum variation in demographic characteristics of participants could also be achieved. The pilot study participants comprised 50% males and 50% females (Table 5.2). Participants were also evenly split between ESL students and English first language students (50/50). Notably,

108 around 45% of all participants had previously deferred their university studies resulting in a higher median age range for second year undergraduate students (22-25 years). There was a slightly higher proportion of younger female students (89% compared to 67% of males who were under 26 years of age) and no students over the age of 29 years. The pilot study group confirmed that group work was regularly undertaken in accounting subjects (often/always 66.7%), but students rarely or occasionally (88.9%) received any training or help with group processes. Despite the lack of guidance, the majority (88.9%) thought that group work should be part of the accounting curriculum.

The purpose of testing the instrument as a whole, in this second stage, was to check the changes previously made as well as evaluating the flow, question skips, timing, and respondent interest (Converse & Presser, 1986; de Vaus, 2002). A couple of small technical issues with the online software were identified (eg. in the use of question skips and prompts for missed responses). It was decided not to force a response to the scaled item questions because ‘missing’ answers may in themselves provide an indication of an underlying problem with the question.

Furthermore, forcing a response can potentially introduce measurement errors into the data set (Neuman, 2004). Dillman (2012) also questions the ethical legality of forced questions since all survey responses should be completed voluntarily. The pilot tested version of the survey instrument contained a total of 71 questionnaire items (arranged into 15 questions or sets of questions) and 15 demographic items. Missed items for the stage two pilot study numbered only four of the total 1638 (91x18) data points tested. As each of the four missed items differed, it was considered a positive response to the overall instrument.

Table 5.2 Pilot study stage 2 participants

Variable Male Female Total Percentage of total Age: 18 - 21 3 3 6 33.3% 22 - 25 3 5 8 44.4% 26 - 29 3 1 4 22.2% Total 9 9 18 100% ESLa _{4 5 9 50%} English-1st _{lang. 5 4 9 50%} Total 9 9 18 100% School to uni 5 5 10 55.6% Deferred uni. 4 4 8 44.4% Total 9 9 18 100%

109 The time taken to complete the questionnaire ranged from 10 minutes to 30 minutes as noted by 11 students on a separate feedback sheet. The average was 18 minutes. Two of the three students who spent 30 minutes answering the questions identified themselves as Chinese. One of these students made a comment that the survey was “quite long” (PS student 6). To ensure the survey was not unnecessarily biased against ESL students, further analysis was performed on the entire pilot study sample. Using the start and finish time stamps recorded on the online SurveyMonkey database, the average time for all international student participants was 18 minutes as was the overall mean for all 18 students. This finding reflected the accuracy of the self-reported data. Noting that computerised time stamps are based on actual login and logout times, and do not account for interruptions, it appeared reasonable to budget for between 15 and 20 minutes to complete the survey.

One pilot survey student noted that there were “too many group questions…got disinterested” (PS student 11). This is a valid observation. de Vaus (2002) suggests that removing, moving or providing a greater variety of questions may help avoid such monotony or disinterest. However in the earlier development stage of the survey a conscious effort had been made to vary the types of questions and include single five- point scaled questions with grouped seven-point scaled items. The length of the survey (Converse & Presser, 1986; Dillman, 2000), as well as question format and survey topic (Millar & Dillman, 2012; Neuman, 2004), can impact on the way students might respond to a survey. So too can the individual’s personality, gender, economic status, and/or social engagement (Porter & Whitcomb, 2005). An apparent variation in student attitudes and responses can be gleaned when comparing another student’s response to the pilot study final question: ‘Having completed this questionnaire are you now more likely to think about group work differently?’ Commenting on their involvement in the pilot study the student responded, “[I] understand the group work much deeper than before, this is an excellent experience, thanks” (PS student 12).

Seven of the 18 pilot study participants provided additional comments regarding their general experience of group work. All comments related to the impact that group formation and composition had on their experience. Coding responses to open-ended questions in stage two of the pilot test can help highlight potential difficulties for coding later on, or identify areas of concern not adequately covered by the survey (de Vaus, 2002). On this occasion, the open-ended responses provided affirmation for the structure

110 of the survey instrument which grouped together sets of questions about group formation, group processes, individual accountability, and interdependence, in that order. Notably, the area of most concern for students was addressed early in the survey. After further analysis of the instrument structure it was decided to add the numerical codes for each of the matrix (or scale) question choices (i.e. 1.Very strongly disagree…7.Very strongly agree) to enhance the understandability for students making a choice between them. No further alterations were made to the instrument. Stage one and two had met the recommendations for pre-testing questionnaires of at least 20 people from 3 different groups (Frazer & Lawley, 2000) (in this case academics, accounting students, and other university students).

However, as became evident, pilot tests are subject to a number of limitations. They are generally limited by time and funding restrictions, and different pre-test methods identify different types of problems (Presser et al., 2004). Presser et al. (2004, p. 111) further acknowledge that ‘computerized modes of administration pose special challenges for pre-testing’. In the current study, two additional problems were identified with the implementation of the online survey: a low response rate to email invitations; and a severe response bias in the characteristics of respondents. The outcome was the abandonment of the data set for the purposes of the main study, instead presenting the valuable lessons learned here, as stage three in the pilot test. The following section discusses response rates and the concept of non-response error and outlines the process which led to the stage three pilot study. Significant procedural changes resulted.