Questionnaire Development and Methodological Considerations

(a) Pre-Testing the Survey Instrument

In order to minimise concerns of response error bias, as it relates to problems in instrument design, a two-step process was adopted. First, the completed questionnaire was pre-tested by experts in the field of tax compliance research,89 following a process recommended by Dillman (2000). All comments and errors identified were addressed and corrected.

The revised questionnaire was forwarded to 18 accounting and 21 engineering postgraduate students,90 all from the University of Canterbury. The postgraduate students were asked to assess the ambiguity, readability, comprehensibility, the ease of completing the questionnaire, and to measure the time taken to complete the task. The exercise provided useful feedback, including suggestions for improving the questionnaire. The comments and suggestions received were incorporated into the modified questionnaire. The questionnaire was then re-submitted to the two academic experts for a final review. Once this was completed, the questionnaire for the mail survey was prepared for distribution.

In terms of the web-based questionnaire, these were forwarded to a number of volunteer testers from around New Zealand.91 Problems encountered in the testing process were forwarded to the IT consultant who attempted to address these problems as they were

88

Demographic variables have, nonetheless, been effectively used as mediating variables in prior studies (Hite, 1997; McKerchar, 2003a; and Devos, 2008).

89

Experts included the senior supervisor, co-supervisor and selected members of the IRD.

90

It was assumed that engineering students would have minimal knowledge of accounting and tax terms.

91

Students, friends and IRD colleagues from various parts of New Zealand tested the questionnaire on their personal computers.

126

identified. Once these were resolved, the questionnaire was linked to the web-page set up for this study.

(b) Self-Reports

Self-reports are commonly used in data collection, and are widely used to measure individuals‟ subjective evaluations of their attitudes, beliefs, feelings, experiences and behaviours. Self-reports are an effective way to obtain information that cannot be observed or obtained objectively.

Hite (1988) argues that most survey results based on self-reports suffer from biases due to a number of factors. These may include respondents misunderstanding the questions, not remembering the information requested, not having any cognisance of the issue, concealing information or even deliberately giving misleading or untruthful answers (Hite, 1988; Neuman, 2006; and Fowler, 2009).

A number of studies have provided support for the accuracy of self-reports (Clark & Tifft, 1966; Tittle & Hill, 1967; Cahalan, 1968; Clausen, 1968; and Sheffrin & Triest, 1992); other evidence suggest self-reports to be imperfect representations of actual behaviours (Goddard et al., 1961; Cannell & Fowler, 1967; Manfredo & Shelby, 1987; and Elffers et al., 1992). The findings to date are still inconclusive.

Despite this, a literature review undertaken by Tittle (1980) indicated that self-reports are generally 80 to 90 percent accurate. A later study by Elffers et al. (1987) compared self- reports of evaders and non-evaders with government data; however, no significant correlation was found between the self-reported and government data. Elffers et al. (1987) found that some compliers admitted to noncompliant behaviour, whereas some noncompliers deny underreporting their income. The authors attribute this outcome to the fact that government statistics tend to underrepresent offences, however, the authors were also unable to explain why tax evaders would deny their past noncompliant behaviour. Perhaps this outcome could be attributed to the fact that the interviews were conducted personally. Surveys conducted in person or over the telephone are more likely to receive less truthful responses compared to self-administered mail or web-based surveys (Neuman, 2006). This may suggest the existence of self-presentation bias in Elffers et al.‟s (1987) study. Another view attributes this to the differences of opinion between auditors and taxpayers, explaining that in the Tax Compliance Measurement Program (TCMP) audits, a significant proportion of taxpayers made mistakes that overstate their taxable income (Kinsey, 1988). Conversely, auditors may also not detect

127

unreported income, which would also result in a discrepancy between auditor views and self- reports (Sheffrin & Triest, 1992).

A key concern with tax compliance research is that, due to rights of privacy, the accuracy of most taxpayer survey data cannot be directly established. Therefore, despite the shortcomings associated with self-reports, this method of data collection is the best alternative for collecting sensitive and personal data, such as individual‟s tax compliance data.

(c) Method Bias

Method bias, also known as “methods halo” or “methods effects” may occur when data is collected via only one method (Campbell & Fiske, 1959) or via the same method but only at one point in time (Marsh & Hocevar, 1988). Arguably, the problem may occur when items measuring a particular constructs are placed together, which could influence respondents to give a similar response to these items (Straub et al., 2004). Issues could also arise when the wording of a survey question influences respondents to give a particular response.

Straub et al. (2004) are critical of instruments which lack random ordering, arguing that the extremely high Cronbach alphas, in the upper .90 range found throughout the TAM research stream may be attributed to the non-random ordering of items in the survey instrument.92 Further, Straub et al. (2004) suggest that Cronbachs of greater than .95 in some studies are highly suspicious for this reason.

The current study involved collecting data via the self-administered survey only, and the survey data was collected at one point in time, and therefore may be susceptible to method bias. Hence, steps were taken during the development of the questionnaire to ensure that any potential effects from method bias are minimised. Dillman‟s (1978; 2000) tailored design

method was adopted in the development of the questionnaire. This approach included

randomly ordering non-TPB questions, and reviewing each question to ensure they are not likely to influence any of the other responses.93

(d) Ethical Issues

The key concerns with behavioural surveys, which are likely to contain personal and sensitive questions, relate to anonymity of respondents, the safeguard of the data gathered and the unlawful use of the data. All University of Canterbury research students dealing with

92

TAM refers to the areas of study involving the Technology Acceptance Model.

93

It was necessary to place some TPB questions together, especially those multiple questions measuring a composite variable (Ajzen, 2002).

128

human subjects are required to obtain prior approval from the Human Ethics Committee before proceeding with the research (refer to Appendix 1 for letter of approval). This is to ensure that researchers will conduct their research with appropriate regard to ethical principles. An application was lodged with the Human Ethics Committee outlining the objectives of the research, the steps taken to ensure the anonymity of respondents, and the proposed use and storage of the data. The covering letter to respondents informed respondents that ethical clearance was given by the Human Ethics Committee to undertake the research (refer to Appendix 4).

(e) Survey Distribution

(i) Mail Survey

The survey package contained a copy of the survey instrument, a covering letter explaining the nature of the research and soliciting their participation, a form requesting a summary of findings and two prepaid self-addressed envelopes.94 In addition, a small token of appreciation in the form of a tea-bag,95 was included with a note thanking the recipient for their participation. These were mailed out to the 1,000 subjects randomly selected from the New Zealand Electoral Roll. A wide body of literature suggests multiple follow-ups in order to increase the response rate of mail surveys (Dillman, 1978; Schaefer & Dillman, 1998; and Dillman, 2000). However, this would considerably increase the cost of the study and the author decided instead on only one complete follow-up.96

(ii) Web-Based Survey

The link to the web-based survey, together with an electronic copy of the cover letter, was forwarded by the Information Technology Department of the University of Canterbury from the University‟s website. Prior notice was given to NZICA and NZLS of the survey. On receiving the link and copy of the letter, NZICA emailed the link and letter to 1,000 of its members who were randomly selected from the NZICA membership database that contains email addresses. The letter reassured all respondents‟ complete anonymity and explained that their responses could not be linked to them individually or to their organisation.

94

To assure respondents of anonymity, one envelope was for mailing back the completed questionnaire, while the other was for posting the request form for a copy of the summary of findings.

95

Studies have documented that the inclusion of a premium (token) improves response rates, although some critics have argued that it may also induce a response bias. Whitmore (1996) found that small non-monetary tokens do not lead to any premium inclusion bias or any systematic bias.

96

Steps taken to safeguard the anonymity of respondents meant that non-respondents could not be identified. The follow-up to taxpayers therefore included everyone on the list.

129

A similar process was adopted for NZLS. However, NZLS‟s membership database revealed that only 266 of its members spend more than 50 percent of their time on tax work. The electronic letter and the link to the survey were forwarded to those members previously identified. No follow-up reminders were issued to selected members, as a result of both organisations‟ policy on unsolicited emails. The NZLS, however, inserted a reminder in their membership newsletter, which did not result in any additional responses.