Pilot Study (Appendix 1.3)

The pilot study had three main objectives: a) to test the experiment procedure and detect any flaws in it; b) to determine the success of the manipulations; and c) to validate the accuracy of the computer programmes written to obtain the recognition memory score and working memory score by using the procedures explained previously in Sections 4.3.3.2 and 4.3.4.2 of this chapter.

A total of 24 participants were expected to participate in the pilot study based on the heuristics that suggested 12 participants per group for a pilot study of two or more groups would be adequate (Ailawadi et al., 2001; Engle et al., 1999). Accordingly, 37 undergraduates of the University of Canterbury participated in this phase.

The outcome of this study did not indicate any concerns regarding the execution of the experiment except for minor suggestions made to improve the understanding of some of the instructions. Furthermore, the participation through sending the email invitations in Stage II also appeared to be successful. The functionality of the computer programmes indicated no logic errors and therefore was considered to be reliable.

Additionally, Manipulation checks were also conducted to determine whether the treatments were able to generate the intended levels of stimulation. Thus, two separate independent sample t-tests were carried out and the results indicated that mean differences for music (𝑡 = −2.46, 𝑝 = .02) and complexity (𝑡 = −6.89, 𝑝 = .00) manipulations were statistically significant. The summary of these results are shown in Table 4-12 and Table 4-13.

120

Table 4-12 - Descriptive Statistics for Music Congruence and Message Complexity Manipulations

Manipulation Mean Std Dev

Music Congruence

Congruent 4.06 1.76

Incongruent 2.81 1.33

Message Complexity

Complex 5.74 0.83

Simple 3.23 1.35

Table 4-13 - t-test Results for Manipulation Checks

Manipulation t sig

Music Congruence -2.46 .02

Message Complexity -6.89 .00

Since the pilot study did not indicate any major issues with the experiment procedure and its intended manipulations, the final data collection phase was conducted subsequently.

4.6 SAMPLE

The sample of this experiment was an online consumer panel registered with Amazon Mechanical Turk (www.mturk.com). One of the main reasons for using an online panel was to increase the external validity of the research. As mentioned in Section 4.2.1 of this chapter, external validity was a concern in a study that uses experiment methodology. Hence, in an attempt to reduce such limitations, a generic sample with a mixed set of participants was used.

4.6.1.1 Sample Size

It is considered that the minimum sample size for a between-subject experiment would be 30 subjects per group. However, given some provisions for (a) withdrawal of participants, (b) incomplete data sets, and (c) making the group sizes equal, 50 participants per group were considered prudent. However, considering the nature of this study, where participants were expected to return after approximately 48 hrs, a

121

significant margin was added to the sample size of a group. Hence, 100 participants per experimental condition were targeted for the main data collection phase. From there onwards, a total of 403 participants took part in the study.

4.6.1.2 Sampling Procedure

The sample was recruited by posting an invitation in Mechanical Turk. Initially, it was planned to adopt a ‘pulsing’ recruitment procedure where the recruitment invitation would be re-posted every two days. However, due to very high response rate for the first post, the expected number of responses was met without the need to re-post in subsequent times.

A part of the invitation contained a description of the study which included a general introduction to the study, what was expected of participants, special requirements of the study, details of the two stages, and the duration of each stage. However, as mentioned earlier, the actual objectives of this experiment were kept hidden and the prospective participants were told that it was to obtain feedback on a few advertisements.

Furthermore, the payment for each stage and the eligibility requirement for especially Stage II were also mentioned. Finally, the link through which they were redirected to the experiment site developed in Qualtrics (www.qualtrics.com) was included.

Two separate payments were made for participation. The payment for Stage-I was

$0.50 and those who returned for Stage-II was paid an extra $1.00. Such discrimination was made to encourage the return of the participants for Stage-II. Further, the participation in Stage-II was controlled by an access code given to them on successful completion of Stage-I. The same was also deployed in the email inviting the eligible participants for Stage-II. This code served two purposes. First, it prevented anyone attempting Stage-II without completing Stage-I and second, it was used to match the participants in both stages for the purpose of merging the data of Stage-II with those of Stage-I, as a repeated measure.

122 4.7 ETHICAL CONSIDERATION

This experiment was conducted in accordance with the ethical guidelines provided by the University of Canterbury Human Ethics Committee. Accordingly, the participants were clearly notified about their involvement in the experiment and its anonymity and confidentiality. Thus, the participants were informed that the study was confidential but was not anonymous due to the fact that Stage-II required contacting participants personally after approximately 48 hours. Thus, they were notified at the beginning about collecting their email addresses at the end of Stage-I for contacting them subsequently for Stage-II. To ensure confidentiality, the procedures recommended by the Ethics Committee were adhered to and the contact details of participants were not kept longer than necessary and were not used for any purpose other than what was specified in the research.

Participants were given the right to withdraw at any stage of the study without any penalty and this was notified to the participants at the first possible instance.

Additionally, in line with the law that requires participants to be over 18 years old to take part in a study of this nature, a screening question was inserted at the beginning of the study that prevented underage participants from participating. Furthermore, the active consent of the participants was also obtained before the actual test began. To ensure active consent, participants had to click on “…I am willing to participate in this study” button to proceed.

In order to minimise contamination of the memory tests and the other cognitive measurements by participants’ guessing the hypotheses, the actual objectives were kept hidden. Such deception was allowed, but the Human Ethics Committee made it a requirement that in such cases, the participants should be debriefed (see Appendix 7, p.

311) about the actual objectives at the earliest possible time. Hence, the debriefing was given to all the participants at the completion of Stage-II. This included an explanation for the deception, the stimulus manipulations in the target advertisements, different types of measurements obtained, and a statement enabling participants to withdraw their responses if they did not wish their data to be used in the analysis. Finally, all the collected data was stored in secured places with restricted access. They were encrypted and password protected in addition to being physically kept in a locked cabinet.

123

This research was reviewed and approved by the University of Canterbury Human Ethics Committee before collecting data. The approval letter is in Appendix 8 (p. 314).