Integration - Role of experiment, pre-experimental interviews and observations

This research is of exploratory nature and employs a quasi-experiment method.

Experimental studies in social sciences are often criticized in respect of the issue of realism (e.g., Hogarth, 1993; Peters, 1993; Gibbins, 2001). The level of real world complexity that experimental settings can incorporate without compromising the validity of the result has always been a challenge confronting researchers. To better address the issues in audit judgment, qualitative methods are utilized, serving a confirmatory purpose, to grasp background knowledge of audit judgment.

5.3.1.1 The issue of realism in experimental studies

The issue of realism emerged as the most vexed problem in experimental research in audit and accounting since 1980s and debate often centres on the robustness of the findings.

Swieringa and Weick (1982) distinguished between two types of realism: ‗mundane realism‘ - regarding the representation of laboratory events, and their resemblance, or similarity, to those in the real world; and ‗experimental realism‘, which is concerned with the sufficiency and validity of the laboratory setting – ‗whether the laboratory events are believed, attend to, and taken seriously‘ (Swieringa and Weick 1982, p 57).

The issue of mundane realism concerns audit researchers. Experiments in audit JDM often use settings abstracted away from real contexts, and rarely mirror a specific real audit judgment or decision process. Peters (1993) sees it as a problem that most studies in this area are based on simple experimental settings and usually rely on statistical tests to validate their models. This line of criticism therefore focuses on issues such as the extent to which findings on biases in judgments and choice can be generalized beyond the simple laboratory conditions to real audit decision-making practice (Hogarth 1993). In 1980s,

Swieringa and Weick (1982) assessed over 100 experimental studies published in Accounting, Organizations and Society, The Accounting Review, and the Journal of Accounting Research. They found that those studies usually concluded with weak statements like ‗some do, some don‘t‘, ‗the differences are very great‘, and ‗it‘s more complicated than that‘ (Swieringa and Weick, 1982). Moreover, Gibbins (2001) argues that the value of studying auditors depends on the richness of the study context. Psychologists and economics are likely have an advantage in abstracting from the applied setting;

whereas accountants have comparative advantages in understanding the applied setting.

Other researchers take a contrary view and consider mundane realism ‗unnecessary and insufficient for internal validity or external validity‘ (Peecher and Solomon, 2001, p 197).

More specifically, the success of an experimental setting lies in its simplification of the real world and the task. Criticism on the simplification of experiments substantially makes a list of factors from the reality that have been purposely omitted for greater control. Therefore,

‗attempts to elevate external validity at the expense of internal validity are a grave mistake‘

(Peecher and Solomon, 2001, p 198).

The high level of complexity of audit judgment and confidentiality issue has restricted the access to real life information that researchers can obtain. It is often difficult to test more complete theories of accounting decision-making statistically. Hence, the generalizability, the external validity, of experimental findings often appears open to challenge. In respect of mundane realism, verification and discovery might become more difficult and less instructive. Swieringa and Weick (1982) argued that experiments can also serve important purposes of discovery and theory development:

‗Experiments can be used to create conditions that do not exist now and to address

―what if‖ questions. For developing and testing theory, the artificiality of an experiment may facilitate a clean test of a theory, lack of random sampling may not be a disadvantage because it is the theory that facilitates generalization across actors and settings, and triviality becomes a substantive rather than a methodological issue.‘

(Swieringa and Weick, 1982, pp. 57)

Moreover, there is a tendency to assume that practitioners always make the best participants and the use of practitioner participants ensures external validity of the study (Peecher and Solomon, 2001). Thus it has been widely asserted that students are inappropriate and misrepresentative surrogates for practitioners. The use of students as experimental subjects raises concerns, for some commentators, regarding mundane realism

(Swieringa and Weick, 1982). However, due to the unavailability of practitioners, the use of student subjects has become the practical option for many experimental studies (Liyanarachchi, 2007). Liyanarachchi (2007) defends the feasibility of using students as surrogates in experimental studies. He notes the pragmatic and financial reasons (e.g., time required for participation) for using student surrogates. More significantly, he argues that gaining access to accounting practitioners is often quite difficult and that certain compromises (e.g, non-random sampling of subjects) may be forced by their use that eventually undermine the utility of the experiments (Liyanarachchi, 2007, p 50). Students might be different from practitioners in many respects, but whether these differences severely limit the appropriateness of the use of surrogates is difficult to establish, and ultimately is an empirical question and dependent on the particular features of the case and context. In considering the likely impact of the use of surrogates, it is important to distinguish between essential features affecting the validity of experimental findings from non-essential features (p 49). Although studies have attempted to empirically test the validity of surrogates (e.g., Abdel-khalik, 1974; Ashton and Kramer, 1980; Gorden, Slade and Schmitt, 1986; Liyanarachchi and Milne, 2005), it remains unclear whether, in the audit context, differences such as skills and age override certain psychological properties in judgment. Besides, differences between student subjects and practitioners may be a theoretical interest to some researcher (Peecher and Solomon, 2001).

Furthermore, accounting students and practitioners share a similar cognitive structure as practitioners (Hodge, 2001; Liyanarachchi, 2007). Libby et al. (2002) argued that student subjects are ‗entirely appropriate‘ in studies that focus on general cognitive abilities (Libby et al., 2002, p 803). In his view, experiments should avoid using professional subjects due to increasing researchers‘ own time and expense, unless it is necessary for achieving research objectives (p 803).

Mundane realism can only be obtained by converging results of many studies in one area (Liyanarachchi, 2007). There is a danger of generalizing from one single study and therefore replications are important to establish ‗significant sameness‘ from a series of related studies. It was also noted that, due to the limited availability of professional subjects, such replications becomes extremely difficult if relying entirely on practitioner subjects when conducting experiments.

5.3.1.2 Methodology of the experiment and the quasi-experiment method The method of science relies on ‘observation of the world’. Many psychologists identify their discipline as natural science and their essential methodology as the experiment (Hood, 2013). The most important criterion of method of science is a scientific procedure that data can be repeatedly obtained to enable public replication of causal claims from research.

Data and theory are the most important elements of science and they interlink in complex ways. Theory is the ‘organization of concepts that permit prediction of data’ (Elmes et al., 2003, p 34) and data is the source of theory. A deductive study is a process that hypotheses and predictions on specific event are generated based on derivation from theory. Inductive approach is the process of generalizing theory from observations of specific events.

Psychology research is mainly conducted in the form of experiments in which ‗the data are collected and presented in the form of numbers – average scores for different groups on some task, percentages of people who do one thing or another, and so on‘ (Goodwin 2002, p 75). This thesis is a hypothetico-deductive study that follows the tradition of audit JDM research.

Experimental approach is characterized by great control over the research environment and some variable are manipulated to observe their effect on other variables (Kothari, 2004, p5).

It is the most appropriate method to examine causal relationship between variables (McIntyre, 2005).

‗They (psychologists) seek …general laws that govern psychological phenomena.

These laws are discovered by means of true experiments, with participants randomly assigned to experimental and various comparison groups. At least one independent variable is introduced in the experimental group, and differences in outcome are assessed on one or more dependent variables. The variables are measured and the data analysed by statistical methods. Hence, the establishment of a causal linkage between one or more independent and one or more dependent variables in a research design that is specified in enough detail so that independent scientists can replicate the results‘ (Hood, 2013, p 1318).

For a true experiment, random assignment of participants is essential. Audit JDM research often targeted on a specific group of research subjects that share certain qualification and characteristics, which mean random, in strict sense, can not be achieved. Hence, true

experimental methods cannot be used in audit JDM research. In such case, quasi-experimental method is at researchers‘ disposal. Quasi-quasi-experimental design facilitates meaningful comparison between convenience samples that differ on some significant variables (Hood, 2013).

5.3.1.3 The integration of experiments and qualitative pre-experimental work Qualitative and quantitative methods and their underlying presuppositions have been increasingly debated since the early 1980s as though one or the other should eventually emerge as superior (Newman, 1998). The partnership between experimental and qualitative methods was recommended by methodologists, by which more complexity can be built in (Peters, 1993; Bryman, 2006). To address the research topics, qualitative methods has been employed to support the design of experiments and the development of simulation cases.

The research interest of this study centers on the persuasion aspects of the audit review process. Interviews of audit practitioners and observations of auditors‘ fieldwork help to update the descriptive data from the audit review process and to gather more detailed information of modern practice. By means of qualitative methods, the hypotheses and the construct of laboratory events are developed with more confidence. It is expected that this integrative research design will provide an alternative option for future research.

Bryman (2006)’s reviewed research studies that combined quantitative and qualitative methods. Studies were categorized according to their primary discipline. The results strongly suggested that mixed method is more commonly practised in some disciplines than others¹⁴, and it is not popular in accounting and auditing study. In fact, there are few instructions available regarding ‘how, when and why different research methods might be combined’ (Bryman, 1998, p 155). There are various ways of integration of quantitative and qualitative methods. The outcomes of integration of quantitative and qualitative methods are not always predictable. ‘While a decision about design issues may be made in advance for good reasons, when the data are generated, surprising findings or unrealized potential in the data may suggest unanticipated consequences of combining them’

(Bryman, 2006, p 99).

14 Among the sample gathered in Bryman (2006), sociology contributes the largest amount of mixed method studies, which constitute 36% of the sample. For the rest, 27% of the articles are in social psychology, 23%

are from management and organizational behaviour studies, 8% in geography and 7% are media and cultural studies.

Some researchers attempted to formalize the integration in order to provide guidance for other researchers in respecting the ways to combine quantitative and qualitative methods.

Attention has been drawn to issues such as the data collection strategy – data to be collected simultaneously or sequentially; priority of data collected (quantitative or qualitative); the function of integration; and, stage where integration takes place.

To justify the adoption of combined methods, one of the methodological decisions researchers need to make is the rationale and the function of the integration (Bryman, 2006, p 105). Greene et al. (1989) devised a scheme isolates five justifications for combining qualitative and quantitative research: triangulation, complementarity, development, initiation, and expansion. Integration as a function for triangulation is to corroborate results from different methods. As a function of complementarity, the integration is to ‘seek elaboration, enhancement, illustration, clarification of the results from one method with the results from another’ (Bryman, 2006, citing from Greene et al., 1989, p 259). For development purpose, integration seeks to use the results from one method to help develop or inform the other method (Greene et al., 1989; Bryman, 2006).

Integration may also serve the purpose of initiation discovery of new perspectives and contradictions of results from different methods. It can also help to expand research in breath and range of enquires. The integration adopted in this thesis is to assist the development of research so that qualitative data gleaned can help strengthen the experimental design.

Another predetermined issue of adopting integration methods is concerned with the stage in the research process when integration applies. Several writers have pointed out that quantitative and qualitative research can be combined at different stages of the research process: e.g., formulation of research questions; sampling; data collection; and data analysis. The pre-experimental qualitative part of this thesis is to support the design of the research and to assist the formulation of research question.