Causality, Generalizability, Transferability and External Validity 135 

As this was a mixed methods study it was important to reconcile the concepts of causality, generalizability and validity that are typically associated with quantitative research and transferability which Jensen (2008) and Patton (2002) associated with the ability to apply qualitative research findings to situations beyond the context of the original research. Inherent in transferability is rich description whereby readers are able to determine the applicability of the research to their own situations (Patton, 2002). However, it was impossible to discuss external validity in a quantitative environment, particularly in experimental research, without contemplating the specific participants, treatment or intervention, the outcomes and the setting all of which affect external validity (Cronbach, 1982). Thus, Teddlie and Tashakkori (2009) suggested that external validity, a quantitative concept, and transferability, a qualitative concept, are similar in that defining the context is paramount to both transferability and external validity.

Because the quantitative investigation was not only dominant but had the ability to stand on its own, the concepts of causality, generalizability and validity remained relevant to that portion of the mixed methods study. The advantages of a true experiment related to its strength in asserting causality, the claim that the intervention, treatment or independent variable was the cause of the change in the dependent variable. However, claims of causality and generalizations from the random experiment to the population were dependent on the external validity of the research, the inferences about the extent the causal relationship is appropriate across various person, treatments, settings and outcomes (Shadish et al., 2002).

Experimental research has often been based on a single program, single treatment, single setting or single experiment for which the participants could not be selected randomly from the population. As with any research conducted on a single program at a single period in

time the generalizability of the findings may be impaired (Weiss, 1972). However, it would have been almost impossible to select participants for a leadership development intervention from the population of emerging leaders because programs have typically solicited their own participants under certain selection guidelines. Furthermore, programs have often been constrained as to the number of participants they could reasonably accommodate. The capacity of the program may not have been sufficient to meet required sample sizes. Such was the case with the Northern Exposure to Leadership Institute (NELI) and thus, a

limitation of this research. However, it would have been just as difficult to randomly select people with a certain medical condition to receive an experimental treatment. The ethics of selection must be considered. In either instance, the nature of the experiment has made generalization of causality difficult. “A conflict seems to exist between the localized nature of the causal knowledge that individual experiments provide and the more generalized causal goals that research aspires to attain” (Shadish et al., 2002, p. 19).

The problems associated with random selection of program or experiment participants and thus with generalizability, have often been encountered in evaluation. Evaluation

theorists have agreed that the problem can be addressed by conducting multiple studies or by including multiple programs in the evaluation. This solution did not fit this investigation as it would have sacrificed causality for generalizability. It made little sense to be able to

generalize the results if causality was not possible to determine. Presuming that additional leadership development programs could have been recruited to participate and could have been enticed to allow random assignment of their participants to groups, it might have been possible to improve generalizability. However, each of these programs was slightly different and would have introduced new extraneous variables that would have required control. It

would have been impossible to control for all the variations and accommodate the

differences in the programs within an experimental design. Thus causality would not have been determinable. The solution of including multiple programs would have worked in an instance where there were multiple geographic locations running the same program so that the variances between the instances of the program were minimized and potentially controllable. That was not the case in the realm of leadership development programs.

Shadish et al. (2002) provided some relief to the problem of external validity in single experiments with the Grounded Theory of Causal Generalization. The theory suggests that scientists and others assess the generalizability, validity and value of research against a set of practical principles that are related to the persons, settings, treatments and outcomes of the research but not necessarily rooted in statistical validity. While random selection methods are always desirable, the set of principles suggests that random selection is not the only means by which external validity can be established. Purposive sampling was thus applicable to both participants and programs; either or both of which could be selected on the basis of typical or heterogeneous instances without compromising validity. The five principles that address construct validity, in addition to external validity, are set out by Shadish et al. (2002):

1. Surface selection - assessing the similarities between the operational characteristics of the research with those of the program

2. Ruling out irrelevancies - identifying attributes related to the specific operational characteristics, that is, the persons, setting, treatment or outcomes that would not change a generalization and thus, would be irrelevant

3. Making discriminations – determining factors related to the persons, setting, treatment or outcomes that would preclude or limit generalization

4. Interpolation- Extrapolation – uses the results to infer intermediary results and those beyond the range of the research

5. Casual Explanation – develops and tests theories that explain and relate to the target of generalization.

Experimental methodologists suggested that these principles would work in tandem with purposive sampling to address issues related to external validity and were particularly relevant when the target of the generalization was a construct (Shadish et al., 2002). Such was the case with this investigation. Generalizations have been made about the effect of a leadership development intervention, a treatment, on the construct of leadership self- efficacy. Shadish et al. (2002) also acknowledged that it is common to explore only one typical instance relative to treatments and settings. This was also the case in the research as the Northern Exposure to Leadership Institute was the one typical instance.

However, these principles imply an ability to discern which operational characteristics and variances therein actually make a difference. That process of evaluation uses knowledge and relies on prior knowledge creation or research by which to inform judgement.

Paradoxically, this research responded to gaps, such as those that related to program content and attributes, and thus provided the very knowledge that would have informed the

evaluation and discernment related to some operational characteristics.

In advocating for an integrated philosophy and application within mixed methods

research, Teddlie and Tashakkori (2009) suggested that both the concepts of external validity and transferability be referred to as inference transferability. It was evident that a robust

description of the research context was necessary to satisfy both transferability and the five principles associated with external validity. The importance of providing context to the research was consistent with the pragmatic paradigm and its application in mixed methods research. Thus, in this study, given the pragmatic research paradigm, inference

transferability was the goal of the integrated quantitative and qualitative findings. With the rich description of the context of both the quantitative and qualitative elements of this

research, potential users will be able to make their own determination as to the comparability and applicability of the findings. Thus, the research has value but the inference

transferability will be dependent on the context of the potential user. Support for this assertion is found in the Grounded Theory of Causal Generalization (Shadish et al., 2002). The characteristics of the people, setting, treatment and outcomes had to be defined to enable potential users of the research to conduct a proper assessment. Thus, the detailed description of the Northern Exposure to Leadership Institute and justification as to why it represents a typical instance both as a setting and intervention was essential to inference transferability in this study.

3.11. Research Context - Northern Exposure to Leadership Institute (NELI)