Research Framework

A Research Framework for Curriculum Evaluation was used to evaluate the effectiveness of SECME since, in essence, it is a “specialised curriculum.” The intent of the program, how it was implemented, and the perceptions and achievement of the students involved in the program were used to frame how the research questions were answered. The intended curriculum was documented in the focus and mission of SECME. The implemented curriculum was evidenced through observations of the activities provided to the participants. The perceived and achieved curricula were evaluated by questionnaires and assessments such as the Test of Science Related Attitudes (TOSRA), Test of Mathematics Related Attitudes (TOMRA), researcher- developed questionnaires, and the Florida Comprehensive Assessment Test (FCAT), respectively. The TOSRA and the TOMRA were both modified due to the length of the original questionnaires and the relevance of only some of the scales to this research. Both instruments had been administered to middle school students in other studies (Ogbuehi & Fraser, 2007; White & Richardson, 1993).

The intent of SECME nationally and locally were ultimately the same, to increase historically underrepresented and underserved student populations’ achievement in science, technology, and mathematics while providing precollege engineering exposure and experiences through an alliance with industry, government and industry partners. The main difference between the national and local SECME program intentions was that, locally, the focus was directed at K–12 students and their parents, and at the same time providing teachers with professional development, resources and support. At the national level, the focus was primarily the teacher. However, it must be reiterated that the end result of increased student participation in STEM was the intended outcome for both. The ideal implementation of SECME in the school district was accomplished by schools having teams of teachers, counselors and administrators working with students and parents.

Ultimately, the program would be implemented as part of the classroom curriculum and as an after-school enrichment program that provided STEM opportunities and exposure to historically underserved and underrepresented students in those fields. Schools would host engineering design seminars for students and provide in-house competitions. A school climate of high expectations and encouragement of students to excel in mathematics and science would exist among the entire staff. Teachers would target minority students and actively recruit them into the program. There would also be collaboration among neighboring elementary, middle, and high school SECME students to articulate and transition smoothly into already existing SECME schools. The students would participate in school-site as well as district sponsored activities and competitions.

3.2 Research Design

Primarily, the research was carried out through the use of a mixed method (Anderson, 1998, Cohen, Manion & Morrison, 2002) that included a pre-posttest design using surveys and questionnaires of 12 active SECME middle schools. At a workshop, teachers participating in the SECME program were introduced to the modified TOMRA and TOSRA pre/posttest survey. Participants were informed that the surveys would be sent to their schools in the fall, between October and November. Directions were included with the modified TOMRA and TOSRA to ensure that surveys would be administered with fidelity and equity (see Appendices

A2, B and C).

The researcher collected the completed surveys from each school and compiled the data. In the spring, the posttests were sent to schools and collected in June. Pre and posttest data were combined and analysed. Interviews, observations, and researcher- developed questionnaires were included as part of the data collection.

More in-depth case studies were conducted with three targeted schools from the SECME program. The literature suggested that using case studies as a research method was valuable and appropriate, especially with respect to developing a deeper understanding of a phenomenon such as the effects of specialised programs on student achievement in mathematics and science. Anderson (1998) explained that a case study allowed for intensive analysis, therefore, limiting the researcher overlooking critical information. As a part of the case study, face-to-face interviews, questionnaires, and observations in natural school settings provided a better understanding of student motivations and their responses to and interactions with teachers and peers during SECME meetings and SECME Saturday Engineering Design Seminars.

A triangulation strategy was used in this study to increase the validity of research. Mathison (1988) described two types of triangulation strategies that would be used during this program evaluation. Data triangulation as “the inclusion of more than one individual as a source of data....understanding a social phenomenon requires its examination under a variety of conditions” (p. 14). In this research, multiple students and teachers were observed in a variety of settings. Methodological triangulation uses multiple methods, such as interviews, questionnaires, and observations to examine the phenomenon of the case study. “The rationale for this strategy is that the flaws of one method are often the strengths of another: and by combining methods, observers can achieve the best of each while overcoming their unique deficiencies” (Denzin, 1978, p. 302).

Test scores attained from the Florida Comprehensive Assessment Test (FCAT) from the current year of study were analysed and compared to the district and state average mean scale scores. Since Punch (1998) notes that in comparative case studies

“...the focus is both within and across cases” (p. 152), participants for the case study were selected from three different middle schools. This use of both data triangulation and methodological triangulation (Mathison, 1988) were used to gain a more complete examination of the SECME program.