RESEARCH DESIGN

Research conducted to test theories characteristically investigates hypothesized relationships between variables. Such research is first concerned with whether a relationship exists at all and then with the causal nature of that relationship. Research focused on the existence of a relation-ship is relatively easy to conduct; however, research into the causal problem is clearly much less tractable.

The study of causation typically requires the collection of data over time, on the premise that the cause must be shown to precede the effect. There are now techniques, however, known col-lectively as causal modeling approaches, that under appropriate circumstances can be used with data collected at one time, as well as longitudinally. These techniques have expanded in number, in complexity, and in explanatory power over the past twenty years. Their use is increasing rap-idly, and they appear to offer considerable promise in evaluating causal hypotheses (Williams, Edwards, and Vandenberg 2003).

THE CONDUCT OF RESEARCH AND THE DEVELOPMENT OF KNOWLEDGE 21 A second factor that makes identification of causal relationships difficult is the necessity for establishing adequate controls. Control may be accomplished statistically through the use of pro-cedures that measure unwanted variables and then remove their effects from the relationship under study. However, these statistical techniques require that the data satisfy certain assump-tions, and in many cases it is not at all clear that these assumptions can be met. The alternative is to control variables through the original design of the study. That is not always easy.

Laboratory Experiments

Much of the research on causal relationships has been done in the laboratory. An extreme instance of this laboratory research is computer simulation in which no real subjects are involved. More frequently, the experiment is of the small group or group dynamics type; experimental variables are introduced among subjects, often college sophomores, and the results are measured under highly controlled conditions. Because the study is conducted outside the real world of ongoing organizations, it is easier to use longitudinal measures and to control unwanted variables. Yet even here major difficulties in maintaining controls exist. Furthermore, the results are very much a function of the variables considered (this is particularly true of computer simulations). If the real world is not effectively modeled in the laboratory, or at least the key elements of that world, the results of laboratory experiments will not transfer.

This said, it appears that in many areas such transfers do occur (Locke 1986). Laboratory studies often appear to be well conducted, or conceivably field research is deficient in important areas, with the result that similar results are obtained. In any event, the evidence to date is that laboratory research, with its greater control, is much more valid than previously anticipated. There may be conditions under which this is not true. A degree of field research on laboratory findings still seems warranted. But, assuming initial confirmation, the need for extensive reiteration of these initial results does not seem as great as previously thought.

Field Experiments

The ideal situation is to take the techniques of sample selection, repetitive measurement, and variable control associated with laboratory research into the real world and conduct the same kind of research with ongoing organizations. In such a context the myriad variables that may be impor-tant do in fact operate, and any results obtained there can be expected to characterize the actual organizations to which any meaningful theory is addressed. The problem is that all the difficulties of designing and conducting good experiments that were so easily handled in the laboratory now become overwhelming. Real organizations have innumerable ways of resisting and undermining objective scientific research—not out of contrariness, but because the goals of the real world and the laboratory are different.

The difficulties of conducting causal research in organizations may be illustrated by a study by Belasco and Trice (1969) on the effects of a particular management development program. The study utilized 119 managers divided into four groups. Managers were assigned to each group on a random basis within sex, type of work supervised, and division groupings. In this manner, as many factors as possible were held constant across the four groups to control for spurious factors that might contaminate the findings and make causal attribution difficult.

One group of managers was pretested, trained, and posttested on knowledge, attitudes, and behavior. The objective was to see if a change occurred on any of these factors.

A second group took the pretest, received no training, and then took the posttest. If this group

22 SCIENTIFIC INTRODUCTION

changed as much as the first, clearly the training was not the cause of change. If this group did not change as much as the first, the training remained a strong contender as a cause.

A third group underwent no pretest, received training, and took the posttest. By comparing the posttest result for the third group with that for the first group, it was possible to identify any apparent change due to a sensitizing effect of the pretest (the groups were similar in all other respects). The problem addressed here is control for any effects the pretest may have had in alerting the managers to what they were supposed to learn later in training.

The fourth group received no pretest, no training, and only the posttest. This group, in com-parison with the others, yields a measure of the effects of the passage of time only, and therefore it isolates time from either repeated measurement or training as factors.

Clearly, this kind of research requires a large number of subjects, the opportunity to assign them to groups as desired for research purposes, and extensive collaboration from the sponsoring organization throughout the study. And, as elaborate as the research plan is, it could be argued that a fifth group, undergoing some training of a relatively neutral nature, should have been included to create a placebo situation and cancel out any so-called Hawthorne effect produced by receiving special attention. Thus, even this very complex experiment cannot be said to have achieved the ideal in terms of control. Such studies are very difficult to conduct, yet they continue to appear in the literature (e.g., see Probst 2003).

Quasi-Experimental Designs

Realistically elegant research designs with all possible controls are unlikely to be implemented in many organizations, and if an organization does decide to go this route, it may well be an atypical organization. Accordingly, certain variants have been proposed (Cook, Campbell, and Peracchio 1990; Evans 1999). These designs represent major advances over the noncausal, correlational analyses, but no one such study answers all questions. Basically, these studies utilize as many components of the ideal experimental design as possible, while recognizing that it is better to conduct some kind of research related to causes than to do nothing. Hopefully, the larger number of research investigations carried out will compensate for the relative relaxation of control re-quirements. Accordingly, several interlocking investigations should develop the same level of knowledge as one very elegant study. On the other hand, it is easy to relax scientific standards to the point where replication is not possible and thus not obtain scientific knowledge that can be substantiated. Some trends in qualitative research on organizations show this tendency. It is im-portant to maintain a clear distinction between scientific research and personal narrative in testing organizational behavior theories.

A number of examples of well-conducted quasi-experiments exist in the recent literature. The typical design calls for some combination of the elements considered in the previous section (e.g., see Markham, Scott, and McKee 2002). A particularly good discussion of the limitations that may be inherent in the quasi-experimental design is contained in Morgeson and Campion (2002).

Descriptions of how quasi-experimental designs may be utilized in studying promotion effects are presented in a series of studies conducted within an international bank based in Hong Kong (see in particular Lam and Schaubroeck 2000).

Common Method Variance and Bias

Common method problems can arise from having a common rater provide the measures of vari-ables, a common measurement context, a common item context, or from characteristics of the

THE CONDUCT OF RESEARCH AND THE DEVELOPMENT OF KNOWLEDGE 23 items in a measure. Of these, obtaining measures of both the predictor and criterion within the same study from the same person produces the most pronounced such results; these biases can be quite substantial (Podsakoff, MacKenzie, Lee, and Podsakoff 2003). Thus, when the same person reports on the two types of variables, that person may change the correlations in an attempt to maintain logical consistency. The results are a function of the measurement method rather than of the underlying constructs.

In expectancy theory (Chapter 7), cross-sectional rather than longitudinal designs are often used. Accordingly, individuals’ reports of their internal states (such as expectancies) are obtained at the same time and from the same person as reports of past behavior related to these internal states. As a result of a desire to maintain cognitive consistency, these correlations can be inflated substantially (Lindell and Whitney 2001). This bias is introduced because of the measurement approach taken and the failure to use more appropriate designs.

Solutions to this type of problem, as is typical in organizational behavior research, focus on designing the problem away or controlling it with statistics. In the past, however, many studies have been conducted that did neither of these, thus simply ignoring the problem.

What is needed is to separate the measures of the variables involved by using different sources, and thus different research designs. An alternative is to use measures of variables that are not self-evident (such as projective techniques), so that the individual cannot mobilize attempts to attain cognitive consistency. Attempts to solve common method problems through the use of statistical approaches have been numerous, but as yet no widely accepted solution has emerged.

Requirements for Conducting Experimental Research

Blackburn (1987) has set forth a list of what he labels the ten commandments for conducting experimental research. These can serve as a guide in assessing research used to test theories in the organizational behavior field.

1. Thou shalt assess the extent to which the change actually took effect.

2. Whenever possible, thou shalt use multiple measures.

3. Whenever possible, thou shalt use unobtrusive measures.

4. Thou shalt seek to avoid changes in measurement procedures.

5. Thou shalt endeavor to use a randomized experimental design whenever possible.

6. In the absence of random assignment, thou shalt not select experimental or control groups on the basis of some characteristic that the group may possess to some unusual degree.

7. Thou shalt use appropriate statistical analyses to examine the differences between the experimental and control groups.

8. Whenever possible, thou shalt collect time-series data.

9. To the greatest extent possible, thou shalt protect the employee, the organization, and the experiment in that order.

10. Thou shalt report fully and honestly the procedures and results of the research.

Many of these points are illustrated in a book edited by Frost and Stablein (1992), which provides detailed descriptions of what actually happened in connection with seven research stud-ies. This book is also a good source of information regarding ways in which qualitative research may be employed for purposes of inductive theory development.

24 SCIENTIFIC INTRODUCTION

THEORETICAL KNOWLEDGE OF ORGANIZATIONAL BEHAVIOR