Program evaluation through a curriculum framework

When determining the effectiveness of a program, one should look to the definition of curriculum or ‘plan for learning’ (van den Akker, 2003). van den Akker (1998, p. 421) described curricula representations as:

 Ideal: the original version underlying a curriculum (basic philosophy, rationale or mission);

 Formal: the vision elaborated in a curriculum document (with either a prescribed/obligatory or exemplary/voluntary status);

 Perceived: the curriculum as interpreted by its users (especially teachers);  Operational: the actual instructional practices in the classroom, as guided

by previous representations (also often referred to as the curriculum- in-action or the enacted curriculum);

 Experiential: the actual learning experiences of the students;  Attained: the resulting learning outcomes of the students.

This framework could also be translated to specialized programs. The ideal curriculum, in the instance of this study, would be the program created by the Deans of the seven Southeastern engineering colleges in 1975 and their mission to increase the minority presence in engineering. The formal program was realized with the formation of a competition program and alliance with local member universities in the various participating school districts. The perceived curriculum would actually become two-pronged. At the district level, by the organizers and at the school-site, by the teachers who implemented the program with the students. The operational would be accomplished by both the district staff and the school-site teacher coordinators of the program. The experiential program occurred when the students, teachers, and parents participated in the Saturday Engineering Design Seminars and competition offered by the district office and at the school-site. The attained program results would prove to be the most difficult to ascertain. Attitudinal surveys and/or achievement tests could provide implications for what was attained as a result of the program, however, ideal would be a longitudinal study that would track students from K – 16 through career selection.

This study implemented the broader definition of the aforementioned curricula representation into the intended, implemented, and perceived/achieved program because the effects of the program delivered from the district perspective was the intended exploration. Other studies have used this four-part curriculum framework to identify best teaching practices and curriculum design (Hartley, 2002; Treagust, 1989). A description of each follows.

Table 2.1 Typology of curriculum representations (from van den Akker & Voogt, 1994)

Ideal Vision (rationale or basic philosophy underlying a curriculum)

INTENDED

Formal/Written Intentions as specified in curriculum documents and/or materials

Perceived Curriculum as interpreted by its users (especially teachers)

IMPLEMENTED

Operational Actual process of teaching and learning (also: curriculum-in-action)

Experiential Learning experiences as perceived by learners

ATTAINED

Learned Resulting learning outcomes of learners

Intended

There often exists a gap between the intended curriculum or program structure envisioned by the organizing entities and the experiential curriculum or program that is experienced by the students (van den Akker & Voogt, 1994). Often entities such as organizations or school districts create programs or curricula that are intended to meet certain needs, but by the time it is interpreted or translated into documents for dissemination and interpreted by teachers, then delivered to students, the experiences that students receive may not be the exact intent. There is a need for fidelity of implementation of programs and curricula if they are to be evaluated as effective. However, because there are so many variables involved, especially the human variable, it is often difficult, if not impossible, to stick entirely to the “script” of an intended curriculum or program because each individual teacher or presenter has his/her style and/or interpretation of what and how the material is delivered.

Therefore, it is essential that professional development is provided up-front to introduce the philosophy and goals of the program and/or curriculum as well as to set

clear expectations of its implementation. In this study, the intended curriculum is the district model of implementation of a precollege engineering program. It must be stated, however, that this model may vary slightly from the originally intended program established by the national founding universities. These differences would be a result of the same issues that plague all new curricula and/or programs.

Implemented

When teachers implement the curriculum or program, they rely on their pedagogical preferences, content background, and inherent biases. They must consciously be aware of their practices and constantly reflect on whether these practices are in alignment with the intent of the program and/or curriculum that they are implementing. This proves to be the tension between their practices and fidelity to the intended curriculum/program. Instructors typically do not change their practices until their ideologies align with a new program/curriculum and they have had ample opportunity to internalise it (van den Akker, 1998).

Attained

How the targeted audience responds or reacts to the curriculum and/or program introduces an additional variable and potential loss of intent, since, once again, attainment is subject to the recipients’ interpretation and internalization of what is implemented.

2.6 Summary

The presence and persistence of minorities in science and mathematics, the influences that affect minority achievement and gender comparisons as they relate to academic achievement are factors that contribute to the achievement gap. Unless all students, especially minority students, were encouraged to stay in school and provided with the resources to pursue advanced mathematics and science courses, the United States would be without the local talent to meet the needs of its science, technology, engineering, and mathematics (STEM) workforce, not to mention, maintain its competitive edge globally.

Special attention focused on minority education that eradicates the achievement gap is critical, as the minority will eventually become the majority. The United States census predicts that minorities will represent 50 percent of the population by 2050.

After-school and enrichment programs that provide students with safe, motivational, academically rich learning environments may be the resource to provide the positive academic reinforcement for minority students.

Walberg’s theoretical framework categorised by student aptitude, instruction, and psychological environment examined specific factors associated with achievement and learning outcomes. The nine educational productivity factors that influence each other and subsequently student learning encompassed ability/prior achievement, motivation and attitude, quantity of instruction, quality of instruction, home environment, classroom environment, peers, and use of out-of-school time (Thomas, 2000).

Specialised precollege programs such as SECME have been developed to help close the achievement gap and encourage students to participate in and successfully complete postsecondary studies in STEM. The program seems to address many of the educational productivity factors. Further analysis, through a curriculum research framework evaluation (van den Akker, 1998), would reveal the intent, implementation and attainment of the program goals. The intent of the SECME program in the school district involved in the study was to increase historically underrepresented and underserved student populations’ achievement in science and mathematics while providing precollege engineering exposure and experiences to better prepare them for postsecondary education in these areas. The program was directed at K–12 students and their parents, and at the same time provided teachers with professional development, instructional resources and support for effective implementation.

at district sponsored engineering eventsfrican-American male students, would provide exposure to role models and experiences that they may not typically be involved in at home or at school. An analysis of quantitative and qualitative data would reveal the implementation and the attainment of the intent of SECME. However, it should be noted that as indicated by Thomas (2000), for minority students, standardized test achievement scores may not serve as a sufficient predictor of mathematics achievement. As Rosenbaum (2004) indicated, the strongest predictor of academic preparation for college was the enrolment in rigorous and

intensive courses in senior high school, which may prove to be an important future analysis of the SECME program through a longitudinal study.

CHAPTER 3