Methodological considerations

As mentioned previously, developing a research methodology and selecting an appropriate data collection method should be driven by the type of data needed to address the theoretical propositions. For the present research on the effects of managerial overconfidence and regret aversion, most importantly, the claims about the existence and relevance of these two phenomena in the chosen context need to be tested. Overconfidence was shown for a range of different professionals (see literature review), so there is some hope that evidence on this bias will also be found for my target group of managers who are in charge of capital investment decisions; yet this

Chapter 4: Data collection methods

cannot be assumed to be given, and for the case of regret aversion, no prior study has demonstrated its existence among corporate finance decision makers. Second, data on the independent variables in the model, the managerial decision behaviour, must be obtained. Observations of project selection, individual effort and project evaluation decisions are thus to be collected alongside the psychometric data. Third, there ought to be measures that can be used to evaluate the role of external factors such as moral hazard.

The ideal method would thus need to gather individual psychological data on a large number of corporate finance managers or other managers responsible for capital investment decisions, in real-time, and guarantee the absence of any disrupting alternative drivers of the observed behaviour. In Bavaria, a country with a strong pastoral culture, people refer to such an ideal as a 'milk-producing, woolly pig that can lay eggs'⁴⁷. Just as with productive livestock, no single data collection method seems able to offer all properties desirable. The review of the related literature with regard to data collection methods has shown that experiment and survey appear to be dominant in research on individual investment decisions; with the exception of the survey by Glaser et al. (2003), overconfidence and regret aversion have so far largely been investigated experimentally. In the following, I therefore examine these observations more closely.

4.2.1 Methodological considerations relating to surveys

A survey⁴⁸ potentially offers some advantages that would be beneficial to my investigation. Most importantly, a survey questionnaire permits a standardized recording of first-hand data by directly collecting feedback from practitioners. In research that makes propositions regarding managerial decision behaviour, such insights are obviously very valuable. It has also been suggested that surveys are ideal for the investigation of variable associations in a likely multi-causal situation (Oppenheim, 1992:21); hence, a survey could explore the potential role of several factors like overconfidence, regret aversion, and moral hazard in capital investment decisions quite efficiently. In addition, there are a number of practical advantages to

47 The infamous "Eierlegende Wollmilchsau".

48 Although a survey can also consist of a series of interviews, the term is used here to describe questionnaire mailings only.

Capital investment decisions with managerial overconfidence and regret aversion

conducting a survey that are relevant to the present research. Administering a questionnaire is possible with limited resources, and represents an efficient way of gaining access to top-level corporate decision makers who are normally very busy, because respondents can complete the survey questionnaire by themselves and at a time that suits them best. Theoretically, this should increase the willingness of practitioners to contribute to academic work.

However, surveys are also criticized for being only able to record beliefs at best, and

‘socially desirable’ comments that do not reflect the facts at worst (Aldridge & Levine, 2001:103). Clearly, the fact that the researchers does not know if responses reflect sincere opinions, and even if they do, whether they are accurate of reality, poses a limitation to what can be inferred from survey data. However, the collection of purely quantitative firm-level financial data may also be distorted by certain accounting methods, so that it can be argued that empirical data at the corporate level will always be subject to some distortion.

Another problem with surveys is that there may be a divergence in the interpretation of questions between the researcher and the respondent, aggravated by the fact that, in case they do not understand a question, respondents will rarely turn to the researcher for clarification for reasons of time and lack of interest. It is therefore very important to formulate questions such that respondents will understand what the researcher wants to know; in other words, the ambiguity in survey questionnaires strictly must be minimized.

This requirement attaches great importance to the design of individual questions and the construction of the questionnaire as a whole. Even though Malhotra et al. (2003) note that “there are no scientific principles that guarantee an optimal or ideal questionnaire” (p.326), they offer several useful insights regarding the design of a questionnaire. On the issue of ambiguity, they caution that some questions may be perceived as leading to a certain response option and that this should be avoided (Malhotra et al, 2003:328). Moreover, since a crucial component of respondents’

willingness to participate in a survey is the time needed to respond to all questions, questionnaires should be relatively efficient to go through; hence, the questionnaire should rather be shorter than longer, and only questions that are truly needed in the research should be featured.

A further point raised by Malhotra et al. (2003) is the importance of structure in improving response rates: Questions that may seem threatening to participants (which

Chapter 4: Data collection methods

is arguably the case for inquiring about overconfidence or regret aversion and inefficient decision making) should be placed towards the end of the questionnaire in order to establish a certain rapport with the respondent (Malhotra et al., 2003: 328).

4.2.2 Methodological considerations relating to experiments

In experiments, a sample group of subjects is submitted to a treatment (independent variable) and the resulting effect on the dependent variable(s) is observed and measured (Hussey & Hussey, 1997:60). The purpose of experiments is thus to study the influence of one variable on one or several others in order to make inferences about possible causal relationships. Experiment as a research methodology originates from the natural sciences, where laboratory tests of cause-effect relationships between variables under controlled environmental conditions are well established. According to Malhotra et al. (2003:206), for conducting social experiments, three requirements for making inferences must be met: (i) the variables must vary together (concomitant variation), (ii) the suspected cause (treatment) must occur prior to or simultaneously with the suspected effect, (iii) and extraneous factors must be controlled for. Clearly the most significant benefit of an experimental methodology lies in the possibility of studying the effect of only one treatment in relative isolation (Johnson et al., 2000:45), which is owed to the mentioned controlled decision environment.

However, when experiments are criticised, it is particularly this aspect of control that is targeted. The typical criticism is that experiments are too artificial to be meaningful (Oppenheim, 1992:12). As a consequence, the laboratory set-up might cause the behaviour of subjects to be influenced by their role as subjects in an experiment, as opposed to the presumed more natural behaviour in the real world decision making (Rosnow & Rosenthal, 1997:8). These arguments may be valid to some extent, but they do not provide a basis for discarding experiments as a useful research method in testing cause-effect relationships in general (Dobbins, Lane, & Steiner, 1988). It is precisely the ‘artificiality’ which enables researchers to isolate specific variables of the causal relationship under investigation (e.g. Nelson, Krische, & Bloomfield, 2003:505). In a corporate context, a manager may take a particular decision not because he is biased but because his superior has expressed a preference for a given strategy, and the manager does not wish to counteract. The risk of such spurious relationships should thus be lower for experimental data.

Capital investment decisions with managerial overconfidence and regret aversion

Another frequently mentioned demand on experiments is that they must be ethical, particularly as they involve human beings (Blaxter, Hughes, & Tight, 2001:75). In particular, subjects should not be misled or deceived in the experiment, as Cadsby et al. (1998:281) argue, because this might entail damaging repercussions for the reputation of experiments in general. On the other hand, and especially in the case of studying the role of cognitive biases, subjects should not be fully informed about the purposes of the experiment or different parts of it either, because this might cause behaviour to be 'unnatural'. Further points of consideration in the design of an experiment mentioned by Cadsby et al. (1998) include deciding which experimental materials should be made available for subjects, the type and number of treatments to be administered, the subject group, and the use of incentives for participants.

Regarding whether subjects should be given incentives to introduce an incentive for them to perform well in an experimental game, Cadsby et al. (1998:277) find that “the average payment per subject ranges as high as $165 for 2h to no payment at all” but argue, though, that rewards are essential to make participants take real economic decisions (p.286).

Finally, concerns are frequently voiced as to whether students do not make good experimental subjects because of differences compared to managers that would impede external validity (Remus, 1996; Tosi, Brownlee et al., 2003; Chang et al., 2004).

However, not least because they are most conveniently available at any academic institution, students are the obvious choice for experimental subjects, and consequently are employed by the majority of experimental studies in Finance I encountered. Even if conducting an experiment with students were to limit to some extent the generalisability of the findings, obtaining managers’ participation is prohibitively difficult. To address this concern, a good compromise may be to ensure a certain suitability of the students; for instance, Schwarzkopf (2003) selects for his experiment students who have prior investing experience (p.96).

4.2.3 Choice of methodology

The choice of how to best collect data on the existence of managerial overconfidence and regret aversion, and the hypothesized effects of these biases on capital investment decisions should take into consideration the arguments and insights presented in this chapter so far, and evaluate them in the light of the requirements of the present research. Essentially, my methodology should be somewhat hybrid: Establishing the

Chapter 4: Data collection methods

existence of managerial overconfidence and regret aversion, and the predicted inefficient behaviour can only be done with observations of real-world practice; on the other hand, in order to investigate the proposed association between the biases and certain decision behaviour properly, a relatively controlled environment as in experiments would be ideal. As the analysis of the existing literature in this chapter has shown, experiments are particularly popular for investigating decision processes.

In view all of these considerations, I decided to use a dual approach to data collection by carrying out a survey and conducting experiments as well. Such a strategy is hoped to seize on the benefits associated with either of the two methods in order to meet the challenging data requirements imposed by the research topic, whilst minimizing the individual disadvantages of either experiment or survey alone. Since it was considered a priority to ascertain relatively early on in the research process that a case for managerial overconfidence and regret-aversion could be made, the survey was launched first. In particular, I decided to conduct an internet-based⁴⁹ survey in order to capitalize on the benefits associated with electronic communication. For example, mailing costs were effectively halved since there are no fees associated with the posting of online questionnaires.

In addition, the data from an online survey is available for download in a format that can be directly imported into statistics software such as SPPS. This feature obviously also eliminates any errors that may occur if the responses need to be entered manually into the computer. Furthermore, an electronic questionnaire can be programmed so as to have greater control over the way in which it is completed; for example, participants in the survey can be automatically alerted if any question was not completed properly.

In addition, past concerns about internet access and computer literacy inducing a sample bias do not seem so relevant nowadays. Once the first-hand data obtained in the survey seemed to support some of the assumptions and predictions of the model, I began to design experiments to further obtain evidence on the proposed effects of overconfidence and regret aversion on project selection, effort, and project evaluation decisions.

As the earlier review has revealed, experiments have also become more widely accepted in economics and finance in recent years, with the survey of the use of

49 Please refer to appendix F for a description of how the survey questionnaire was created and made available to participants.

Capital investment decisions with managerial overconfidence and regret aversion

experiments in this area by Cadsby & Maynes (1998) containing a surprisingly large number of references. An experimental framework was deemed to be fitting for the studied decision stages for at least three reasons. For one, the questionnaire-related hindsight problem, which was already noted for the research by Cooper et al. (1995), could be reduced, as instead of a report of past decision making, experimental data would pertain to behaviour 'as it happens'. In addition, a number of potentially disturbing factors could be excluded. For example, when subjects make decisions on their own account, agency cost problems should be negligible. Reputation or career-related concerns should equally be minimized in an experimental setting. Third, by asking managers questions of the type “do you often make mistakes?” the survey responses could be potentially somewhat insincere with managers wishing to protect their self-esteem. The direct observation of actual behaviour where decisions do not have to be justified should present a useful addition to the survey. In the remainder of this chapter, I now present the survey questionnaire used in my empirical research, as well as the two experiments I conducted.