Adversary-Oriented Evaluation (2)

SGDC 5023 SGDC 5023 EVALUATION OF TEACHING EVALUATION OF TEACHING GROUP 5 GROUP 5

ADVERSARY-ORIENTED EVALUATION

ADVERSARY-ORIENTED EVALUATION

APPROACHES

APPROACHES

ARWINA SYAZWANI BINTI GHAZALI

ARWINA SYAZWANI BINTI GHAZALI

820403-11-5560

820403-11-5560

W

WAN ANISAH AN ANISAH BINTI ABDUL KADIRBINTI ABDUL KADIR

810522-11-5572

810522-11-5572

NOOR FAZLINA YATI BT MAT RAZALI

NOOR FAZLINA YATI BT MAT RAZALI

820208-11-5338 820208-11-5338 NORIHAN BT. ENDUT NORIHAN BT. ENDUT 791223-11-5408 791223-11-5408

Prepared By:

Prepared By:

ADVERSARY-ORIENTED

ADVERSARY-ORIENTED

EVALUATION

EVALUATION

APPROACHES

APPROACHES

ADVERSARY-ORIENTED

EVALUATION

 Adversary-oriented evaluation is based on the judicial

metaphor. It is assumed here that the potential for

evaluation bias by a single evaluator cannot be ruled

out, and, therefore, each “side” should have a separate

ADVERSARY-ORIENTED

EVALUATION

Purposes of Evaluation:

 Providing a balanced examination of all sides of

controversial issues or highlighting both strengths and

weaknesses of a program.

ADVERSARY-ORIENTED

EVALUATION

Distinguishing Characteristics:

Use of public hearing, use of opposing points of view,

decision based on arguments heard during

 proceedings.

DEVELOPERS OF

ADVERSARY-ORIENTED EVALUATION

APPROACHES

AND THEIR CONTRIBUTIONS

GUBA [1965]

OWENS [1971]

WOLF [1973]

HISCOX & OWENS [1975] WOLF [1975] STENZEL [1976] LEVINE [1976] NORTHWEST REGIONAL EDUCATIONAL

DEVELOPERS OF

ADVERSARY-ORIENTED EVALUATION

APPROACHES

AND THEIR CONTRIBUTIONS

OWEN [1971]

GUBA [1965] Suggested that educational evaluation might use aspects

of legal paradigm

The first real adversary evaluation in education was conducted “modified judicial model”

THREE GENERAL APPROACHES TO

ADVERSARY-ORIENTED EVALUATION

The Judicial Evaluation Model and other „Pro and Con” Adversary

Hearing

Adversary Hearing with More than Two Opposing Views

Adversary Debates and Other Forensic Structures

THE CHARACTERISTICS OF

The Judicial Evaluation Model and other ‘Pro

and Con” Adversary Hearing

1. Procedural rules must be flexible

2. There are no strict rules for the assessment of evidence. 3. The only requirement is that the judge(s) must determine

 before hand whether evidence is admissible or not.

4. The parties may be asked before the hearing to present all relevant facts, pieces of evidence and names of

witnesses/experts to the judges

5. A copy of the complaint must, before the public hearing takes  place, be committed to the judge(s) and the defence. The

defence may plead guilty to some charges and deny others.

THE CHARACTERISTICS OF

The Judicial Evaluation Model and other ‘Pro

and Con” Adversary Hearing

6. Witnesses are able to speak freely and may be subjected to cross-examination.

7. Experts may be summoned for a statement before or during the hearing.

8. Meetings of all parties involved with the judge(s) prior to the  public hearing tend to soften the debate and can be conducive

to a joint striving to get to the truth of the matter on the basis of relevant facts.

9. Besides the two parties involved, other stakeholders may also  be allowed to participate.

4 STAGES OF

ADVERSARY-ORIENTED

EVALUATION

THE ISSUE SELECTION STAGE

THE PREPARATION OF ARGUMENTS STAGE

THE HEARING STAGE ITSELF

THE ISSUE GENERATION STAGE

Wolf (1979) and Thurston,propose the following four stages for adversary evaluation:

STEPS OF

ADVERSARY-ORIENTED EVALUATION

THE ISSUE GENERATION

STAGE Should the program laid off

and pick some new programs suitable

alternative?

At this stage, a broad range of issues are identified. Thurston recommends that issues which reflect those perceived by a variety of persons involved in, or affected by the program in question, are taken under consideration in the

THE ISSUE SELECTION STAGE

This stage consists of issue-reduction. Wolf (1979) proposes that issues on which there is no debate, should be eliminated. Thurston states that this

reduction may involve extensive analysis (inclusive of content, logic and inference). The object of debate should also be defined and focused during this

STEPS OF

THE PREPARATION OF ARGUMENTS STAGE

PRO _CON

This stage consists of data collection, locating relevant documents and synthesising available information. The data or evidence collected should be

relevant to the for and against arguments to be deployed in the hearing

STEPS OF

THE HEARING STAGE ITSELF

This stage may also be referred to as the clarification forum and involves public presentation of the object of debate (Wolf, 1979). This is followed by

the presentation of evidence and panel or jury deliberation.

STEPS OF

THE CHARACTERISTICS OF

Adversary Hearings With More Than Two

Opposing View

1. Some of the review panels discussed in chapter 8 may hold

 public hearings to collect information pertinent to their charge

2. Appointed commission charged with the resolution of

controversial issues frequently hold hearings to obtain evidence and opinions relevant to their mission.

3. House (1980) has cited as one example the frequent use in

England of commission and councils headed by prominent citizens to provide guidance to government policymakers.

4. Several types of committee hearings are structured to identify

and explore all the points of view represented in particular context.

THE CHARACTERISTICS OF

Adversary Hearings With More Than Two

Opposing View

5. Although not adversarial in the strict sense of world because:

  Smith (1985) has noted they explore a variety of positions.  prefer to include them because:

a. they reflect multiple viewpoints

 b. they frequently use hearing processes, questioning, cross-examination,

interaction concerning alternate viewpoints and summary statements of the various positions.

THE CHARACTERISTICS OF

Adversary Hearings With More Than Two

Opposing View

6. St John (undated) referring to such hearings as the committee

approach to evaluation:

 All of those with a stake of evaluation

 A public hearing with testimony, questioning,

cross-examination, and summary statements

 The hearing method consists of public, verbal, face to face

interaction.

 A process of communication and education occurs, and

evaluation makes its impact as it is happening.

THE CHARACTERISTICS OF

Adversary Hearings With More Than Two

Opposing View

1. Several approaches that qualify as adversary-oriented do not

employ hearing processes.

2. Kourilsky (1973) proposed that pro and con arguments be

 presented to a decision maker. ( who would examine the

evidence and question the presenter, ultimately arriving at the decision that seemed fair given both positions)

THE CHARACTERISTICS OF

3. Kourilsky and baker (1976) described an adversary model in which:

a. two teams prepared, respectively, affirmative and negative appraisals of that which was evaluated,

 b. met to present the views to one another,

c. Cross-examining and critiquing one another‟s continents on  prespecified criteria

d. Engaged in open ended discussions until reconciliation of views was attained and translated into written

recommendations in a single report

THE CHARACTERISTICS OF

4. Levine (1974) proposed that a resident adversary or critic

might be assigned to a research project to challenge each bit of information collected, searching for other plausible

explanations.

5. The Stake and Gjerde (1974) strategy of having two

evaluators prepare separate reports summing up opposing  positions for and against the program- yet another variant

adversarial approach that does not depend on hearing format.

THE CHARACTERISTICS OF

6. Donmoyer (undated) proposed “deliberatively” approach to

evaluation which focused on assessing and balancing alternative conceptions of reality and the differing value  positions underlying these conceptions.

7.  Nafziger and others (1977) described an adversary evaluation

design employing a modified debate model of presenting data collected in a comprehensive evaluation to ensure that both sides of controversial issues were illuminated.

THE CHARACTERISTICS OF

Clyne ( 1982 ) summarized

:-1 •

Summative evaluation

2 •

Formative evaluation

3 •

Social Science making

4 •

Policy analysis and debate

5 •

School governance and local decision making

THE STRENGTHS OF

THE ADVERSARY- ORIENTED

EVALUATION APPROACH

1. Due to the public nature of the evaluation, openness

and transparency regarding the object of evaluation is encouraged.

2.  As the model takes into account multiple forms of

data (inclusive of statistical fact, opinions,

suppositions, values and perceptions), it is argued to do justice to the complex social reality which forms part of the evaluation.

3. The judicial nature of this approach may reduce

political controversy surrounding an object of

STRENGTHS OF

4.  As both sides of an argument are presented, the risks

of tactical with holding of information should be minimized.

5. This approach allows for the incorporation of a

multitude of perspectives, this should promote a more holistic evaluation.

6. The presentation of pro and con evidence and a

platform which allows for cross-examination, permits public access to various interpretations of the

evidence introduced into the evaluative context .

STRENGTHS OF

STRENGTHS OF

ADVERSARY-ORIENTED EVALUATION

7. The presentation of rival hypotheses and explanations

may enhance both quantitative and qualitative approaches.

8.  All data must be presented in an understandable and

logical way in order to persuade the jury. Dependent on the jury in question, this can make the data

presented more accessible to the public and other stakeholders involved in the evaluation.

9. Finally, this approach is suitable for meta-evaluation

and may be combined with other approaches which are participatory or expertise- oriented.

SUMMARY The object of the evaluation affects many people Controversy about the object of the evaluation has created wide interest Decisions are summative Evaluators are external Clear issues are involved Administrators understands the intensity Resources are available for additional expends required by adversarial strategies

THE LIMITATIONS OF

THE

ADVERSARY-ORIENTED EVALUATION

APPROACH

THE LIMITATIONS OF

ADVERSARY-ORIENTED EVALUATION

 According to Smith (1985), many of the limitations of this approach relate to its competitive nature, the complexity of the process, and the need for skilled individuals willing to perform the various roles needed for a hearing. Listed are the main limitations of the adversary evaluation:

1. This form of evaluation may provoke venomous debate and conflict may have a negative impact on the outcome of the evaluation.

2. The focus of the evaluation may shift to assigning blame or guilt, rather than optimizing policy.

3. As adversary-advocate models are conflict-based, possibilities for reaching an agreeable outcome are

THE LIMITATIONS OF

ADVERSARY-ORIENTED EVALUATION

4. Key stakeholders are not always equally skilled, and articulate individuals are placed at an advantage.

5. This method can be time-consuming and expensive (Owens, 1973)

6. It is sometimes difficult for hearing members to develop specific, operational recommendations (Wolf, 1979.

7. Time-limitations may only allow for a narrow focus.

THE APPLICATION OF

THE

ADVERSARY-ORIENTED EVALUATION

APPROACH

THE APPLICATION OF

ADVERSARY-ORIENTED EVALUATION

 Although currently out of favour, this approach has been used quite extensively in the field of educational evaluation (Owens, 1973). It has also been applied to ethnographic research (Schensul, 1985) and the evaluation of state employment agencies (Braithwaite & Thompson, 1981). Crabbe and Leroy contend that an adversary approach to evaluation should be beneficial when:

1. The program being evaluated may affect a large group of people;

2. when the issue in question is one of controversy and public attention;