Small-scale pilot studies

The final stage of pre-testing consists of a small-scale pilot study (Artino Jr. et al., 2014). This stage is of utmost importance since a successful pilot study is a good indicator that the measurement instrument works. This phase can identify specific issues regarding problems with the method of administration, or any respondent issues when completing the

questionnaire. Furthermore, some initial inferences can be illustrated regarding potential attrition rates for the questionnaire, which is a significant issue to understand for the present study since expected attrition rates are not known. Also, of paramount importance is to understand the dynamics of sales self-efficacy26, understanding how frequently changes take place. Consequently, the pilot study questionnaires were distributed weekly, helping to understand the temporal dynamics of the variables included in the present study. 4.6.3.1 Pilot study 1

An initial pilot study aiming to obtain 15 participants in the final wave was conducted for three reasons, to (1) ensure the validation of measures, (2) help establish a picture of how self-efficacy fluctuates over time, and (3) generate a vague expectation of attrition levels for the main study. Regarding the first reason, all measures demonstrated adequate Cronbach Alpha’s (>0.7), and adequate loadings (>0.5). A lack of variance was identified as a potential problem for internal locus of control, but despite this potential problem, the sample size used to generate these results was small (<50), and thus was not a big concern. Concerning the second reason, the initial study obtained weekly measures of self-efficacy understand how often self-efficacy fluctuates. It was found that measurements of self-efficacy identified non- significant change (i.e. less than 10% of variance was within-person), whereas there was

26 _{The temporal dynamics of salesperson performance is also of interest. However, there is already}

132

significant change in at a 1-month time-lag27, thus it was decided that a 1-month time lag is appropriate for the main study.

One grave concern was also identified from the pilot study. Specifically, much higher

attrition levels occurred than expected. The sample size at wave 1 began at 49, which reduced to 45 for wave 2 (9% attrition), 7 at wave 3 (85% attrition), and 3 at wave 4 (48% attrition). At this point it was decided to terminate the pilot study with so little participants left.

Ultimately, 6% of participants who completed the survey at wave 1 completed wave 4, with a major dropout occurring after the second wave, indicating potential respondent burnout, or a lack of engagement. Furthermore, an error occurred with Qualtrics programming, meaning some respondents took 2 surveys within 3 days, compromising the required time-lag. Discussion took place between the research team and Qualtrics to identify ways to reduce attrition levels. Changes to the research design included (1) giving an extra incentive to participants to complete follow-up waves to incentivize participants to return to future questionnaires, (2) changing the time-lag between waves to one-month to reduce

questionnaire burnout of participants28, and (3) adding a further note to participants that they must be available for the full duration of the study to be able to participate. Moreover, Qualtrics assured the researcher that the system error regarding respondents completing the questionnaires at inappropriate time-lags was resolved. It was agreed that a second pilot study must be undertaken to determine the effectiveness of these changes.

The negotiations also meant a concession from both sides regarding expectations. Qualtrics admitted they would have to sample many more participants than they had expected at time one to reach the desired aims of the research team. Qualtrics informed the research team that this would be too expensive for them to run. Consequently, the research agreed to reduce the re-contacts required by one wave (to four) to continue the project. The reduction by one wave still allows the researcher to test for quadratic trends in the data but prevents them from being able to test for cubic trends29 (Hoffman, 2015).

4.6.3.2 Pilot study 2

27 _{Although only a small sample size was obtained (N=3).}

28 _{Alongside the aforementioned self-efficacy temporal dynamics conundrum}

29 _{Cubic trends are not expected, and therefore this concession does not represent a problem for the}

133

The second pilot study was undertaken with one core objective. Specifically, it was important to understand how the changes made from pilot study 1 impacted attrition levels. The sample size at wave 1 began at 50, which reduced to 10 for wave 2 (80% attrition), 3 at wave 3 (70% attrition), and 2 at wave 4 (33% attrition). Ultimately, 4% of participants who completed the survey at wave 1 completed the study, outlining that the major issue of attrition had not been resolved. Further discussions between Qualtrics and the research team took place to establish the issues and what needed to be changed.

After discussions, further changes were made to the study to try and reduce attrition levels. Firstly, it was decided to explicitly inform participants of the expected time it would take them to complete the questionnaire, and how much they will receive for completion for the present, and further, questionnaires. Secondly, additional incentives were increased as it was agreed that the present level of incentives was not enough to entice participants to continue participation. Thirdly, the time follow-up e-mails are sent to participants will vary on

different days to account for respondents who prefer to respond at different times of the day. Additional confirmation at the beginning and end of survey explicitly revealing the

participants progress concerning the full study duration, and what they are to be paid for the further participation in the study was also included. It was agreed that a third pilot study should be undertaken, since the issues regarding attrition were still unresolved.

Concerning the low variance in the internal locus of control measures, the second pilot study sample identified higher levels of variation and thus it was decided that this is not a major concern for the main study.

4.6.3.3 Pilot study 3

The third pilot study was undertaken for the same core reason as the second, namely reducing attrition. The sample size was 40 at wave 1, 20 at wave 2, 14 at wave 3, and 12 at wave 4. The attrition rates were 50%, 30%, and 14% accordingly, representing a significant decrease in attrition compared to the previous pilot studies. For the main study Qualtrics made aware to the researcher that they would be recruiting more panels to meet the criteria agreed but assured the researcher that nothing would change. No other issues with the data were apparent, and although the attrition levels were higher than for other studies conducting repeated measures on salespeople, it is believed that salespeople in the Qualtrics panel are less likely to continue retaking surveys than for studies supported by their company. Thus,

134

the research team and Qualtrics agreed to begin commencement on the full study at the end of the following month

4.7 Main study

Many researchers are beginning to utilize services such as Qualtrics to collect survey data, with such data being accepted in highly ranked ABS journal articles (e.g. Arens & Hamilton, 2018; Grégoire et al., 2018). However, to the knowledge of the author, no present studies utilize their services for longitudinal data, and is not expected to cause any additional issues beyond those discussed. One drawback of utilizing such as service is an ability to conduct a non-response bias test (Armstrong & Overton, 1977). Despite this, individuals previously utilizing the service assured the researcher of the credibility of the data which they had received previously from Qualtrics.

4.7.1 Cover letter

To appeal to participants, and to alleviate social desirability bias by assuring complete confidentiality and anonymity (Nederhof, 1984; Wiseman, 1972), a small cover note is presented at the beginning of the survey (see Figure 4.2). ‘Greetings’ is chosen to open the questionnaire, since individuals are expected to undertake the questionnaire at various points in the day. A small background behind the purpose of the study is given to help make

participants feel comfortable with providing personal data. It was agreed between Qualtrics and the research team that all that could be done, had been done, to maximize the success potential of the project.

135 Figure 4.2. Survey cover note

4.7.2 Response rates

The three pilot studies provided some valuable feedback regarding attrition. Using

calculations from the attrition rates from the third pilot study, it was estimated that, to obtain 120 at wave 4, that 400 participants would have to be contacted on the first wave. Table 4.18 provides an overview of the responses achieved at each wave. 417 responses were obtained, but 31 were deleted as there was obvious signs of participants being ‘speeders’ or ‘straight- lining’, leaving the present study with 386 useable responses.

Considering the third pilot study’s attrition level, it was expected that roughly 194

participants would respond to the second survey, but only 153 responses were achieved. This Greetings,

This research is sponsored by Loughborough University UK and is part of a PhD project investigating salesperson performance. Complete confidentiality and anonymity is guaranteed, so please do not worry when providing honest answers. We would be very grateful if you could share your expertise by completing this survey, and you may receive an executive summary from the findings of this study if you so wish by

contacting [email protected].

Thank you for your invaluable contribution towards this research and doctoral study.

Yours sincerely,

Dayle Childs PhD researcher

136

representing an increase in the rate of attrition by 11% (61% in total) from waves one to two (compared to the third pilot study). For wave three 105 participants responded to the survey, there was an increase in attrition of 2% as opposed to the third pilot study (32% in total). In keeping with this trend, attrition from waves three to four rose by 16% (30% in total)

compared to the pilot study. Overall only 19% of participants who completed the first survey completed the entire study, 11% less than indicated by the pilot study. Ultimately the study finished with 75 participants completing the entire study, 45 less than anticipated. Although not ideal, the sample size was considered adequate for the present study’s objectives, with data collection ceased at this point.

Table 4.18 Response rates for each wave for the main study

Wave Number Total responses achieved Response rate

1 386 N/A

2 153 39%

3 105 68%

4 75 71%