The Theoretical Framework: Evaluation Research 120 

Evaluation theory was particularly relevant to this research as the subject of study was a leadership development program. The research problem arose, in part, because of the lack of leadership development program evaluation and the corresponding gap in research on the relationships between leadership development and self-efficacy. To make sense of, assess and incorporate as applicable, the evaluation research that existed on leadership development required an understanding of the underlying theory of program evaluation. Because the practical application of the research will be in the realm of program development and evaluation, the context of this use and the theoretical foundations for the evaluation environment informed the research analysis and interpretation.

Evaluation theorists have been divided as to whether evaluation constitutes practical research or is distinct from research, representing an alternate activity. In either case,

evaluation has been replete with theory and theorists, albeit the same theorists who also have worked in a purely research domain. One group of theorists suggested that research produces new knowledge that may have no specific reference to any practical decision, while

evaluation is deliberately undertaken as a guide to action (Wolf, 1990). Wolf (1990) argued that the purpose of research differs from that of evaluation and thus there can be a

fundamental difference in the generalizability of the results produced by each type of activity. Evaluation seeks to produce knowledge that is particular to a specific setting or context. The knowledge may not be generalizable. Furthermore, research is always based on world views or paradigm with corresponding methodologies to generate knowledge while

evaluation often fails to consider paradigms and methodological consistency (Wolf, 1990). It appeared that this characterization of evaluation was a positivistic reflection of the state of what often is, rather than what other theorists suggested evaluation was intended to be.

Conversely, Weiss (1972), in declaring similar positivist preferences, deliberately connected evaluation and research by using the nomenclature ‘evaluation research’ to describe a process that employs the rigor and objectivity of research to establish a purpose, collect objective and representative evidence, compare the evidence against established criteria, and draw conclusions as to the effectiveness, merit or success of the phenomena. Weiss argued that by casting evaluation as research, evaluation would benefit from the rigor associated with research and the validity of the conclusions would be more widely accepted. The purpose of evaluation research is to measure the performance of a program against established criteria, to enable subsequent decision making for that program (Weiss, 1972).

Both characterizations of research and evaluation suggested that all theorists have agreed that research and evaluation can be differentiated, not by their subject matter, but by the purpose for which they are done. While the theoretical aspects of this investigation are guided by research, the practical applications are appropriately guided by evaluation theory. Often the purpose of evaluation has been directly related to the intended users of the

evaluation. Greene (2007) identified the four purposes of program evaluation which have been generally accepted by evaluation and research theorists:

 to inform decision making or provide accountability information (policy analysts and decision makers)

 to develop a better understanding of the program from multiple contexts (policy makers, administrators, participants)

 to promote greater equity and fairness in the program (participants).

The four purposes suggested a practical application to evaluation. The practical elements are necessary to respond to the purpose of the evaluation. Evaluation can be further

differentiated from pure research as it has the following unique characteristics:  Evaluation is intended for use by a decision maker.

 Evaluation research questions are those of the decision maker, not the researcher.  Evaluation compares what is with what ought to be. “The element of judgment

against criteria is basic to evaluation and differentiates it from other kinds of research” (Greene, 2007, p. 7).

 Evaluation takes place in the setting of the program.

 Evaluation and program administration have different imperatives such that evaluation may need to defer to the needs or preferences of the program and is administrators.

 Evaluation results are often not published nor made public, but retained by the decision makers.

 Evaluation can be perceived as less than objective because of the conflicting allegiances the evaluator inherently has to the decision makers, the program administrators and society (Weiss, 1972).

As a result of these differences, and the related debate as to whether qualitative or quantitative methods are more appropriate, evaluation has over time developed its own theoretical framework founded on the knowledge base of evaluation. Evaluation, through

experience, has evolved as a methodological speciality (Shadish et al., 1991). The theoretical framework for the evaluation of social programs consists of five elements, each of which continues to evolve as evaluation evolves: social programming, knowledge construction, valuing, knowledge use, and evaluation practice (Shadish et al., 1991).

Evaluation theories can be categorized as fitting into one of four stages or generations. The first three generations have been identified by their primary orientation. That is, the first generation was measurement oriented, the second-generation description oriented, and the third generation judgment oriented. Scientific enquiry pre-dominated the first three stages. The fourth generation has been characterized as subscribing to a constructivist worldview and a negotiation orientation (Guba & Lincoln, 1989).

The moniker ‘current theory’ adopted by Shadish et al. (1991) cannot be interpreted to suggest that a single theory of program evaluation currently exists. Rather, theorists and practitioners subscribe to the theories or some modification of the theories previously described. ‘Current theory’ is a collection of the beliefs that are held in common by most of the theorists and practitioners in the field. Thus, it is less of an integrated theory but an assemblage of the agreed upon constructs in each of the five key elements of theory. There have been as many if not more areas of disagreement in each construct.

Relevant ‘current theory’ was espoused by Shadish et al. (1991) and can be summarized as follows:

Social Programming

 The utility and quality of an evaluation is dependent on the definition of the social problem and what programs or program components are to be evaluated.

Theory of Use

 It is incumbent on evaluators to ask the right questions to ensure that their work will be used and disseminated.

 Evaluation can be useful both as a basis for making changes to program priorities, service mix etc. and as a method of understanding the theory and operation. Theory of Valuing

 Evaluation cannot be value free.

 Consider all stakeholder perspectives rather than a subset of those related to the program.

Theory of Knowledge

 Epistemology and methodology are essential to evaluation.  Evaluation is an empirical endeavour.

 Knowledge of many kinds should be considered in most evaluations, thus the use of mixed methods evaluation research is appropriate.

 Studies that encompass multiple programs or multiple studies are preferred so as to address issues of generalizability and reliability.

 The quality of the knowledge increases with critical public scrutiny thereon. Theory of Practice

 The single evaluation study is inevitably flawed.

 Evaluation typically occurs under time and resource constraints that influence the design and methodology.

The four stages of theory have not specifically addressed generalizability of the evaluation research, although it has been the concern of theorists and practitioners as

evidenced in the ‘current practice’ section. In evaluation, like pure research, generalizability or external validity varies with the research paradigm. Experimental research in a single instance suffers from limitations in generalizing to units, treatments, variables and settings not directly observed (Shadish et al., 2002). Clearly, evaluation of a single program suffered from the same limitations. Thus, the challenges of attaining external validity were the same in both domains. Suffice to say however, that the overriding criterion of ‘purpose’ and the focus on the user suggests that generalizability, while highly desirable, has not always been an overriding concern in evaluation.

Program evaluation methodology of the third and fourth generations is particularly suited to complex environments where context is often important. As a social science, the study of leadership is not only complex but leadership development within the context of a program is also inherently complex; and context, as noted by the third and fourth generation theorists, is very important. In evaluation, as in pure research, the use of mixed methods to address multiple questions is appropriate to the study both of self-efficacy development and leadership development.

Pragmatism, the philosophical stance adopted for this mixed methods study, suggested a practical applicability to research stemming from the focus on the research question. There was nothing more practical than the application of the research to support program

evaluation through a better understanding of the outcomes and interrelationships of the program content and leader development. Moreover, existing leadership development programs could only be improved and new programs enhanced if research existed to inform decision makers. Whether the research was undertaken as pure research or under the guise of evaluation, would be irrelevant to its usefulness and significance as long as the theoretical

foundations for the research or evaluation were consistent and understood. Thus, the practical applicability of evaluation theory was particularly relevant to this research on leader development that was within the context of a leadership development program.