Data Analyses

Research Question 1 asks whether authentic science pedagogy varied over the school year. To answer this, each of the 22 proficiency assessment tasks in Table 5.2 is analyzed to document the features of authenticity students’ experience in this aspect of the teacher’s pedagogy. Table 5.3 shows how the three component and Combined Authenticity scores are calculated. A descriptive analysis documents the patterns found in the presence of these features of authenticity across the study period and in the levels of authenticity between assessment tasks. Patterns in individual task authenticity indicators (TAIs) reveal details of the teacher’s expectations of students.

The scoring guide for authenticity of the assessment tasks is based on the presence of authentic requirements within a task rather than the task as a whole (Appendix A). Therefore, tasks with many parts that provided multiple performance opportunities to demonstrate authenticity were biased to score higher. This applies to two assessment tasks at the end of the school year. The Wind Turbine Engineering and Testing task stretched over three and a half weeks of class time, had many parts and the student artifacts were 10-20 pages.

By contrast, one week was devoted to the Energy Concepts unit, which was assessed by the Energy Concept Quiz task requiring only a 1 page artifact. Therefore, the

84

Wind Turbine Engineering and Testing task was divided into three parts based on natural divisions within the overall task and on the number of performance opportunities in each section. Each of these parts was scored independently using the assessment task scoring guide (Appendix A). The scores reported for the task as a whole are the means of the scores on these three parts of the overall task.

The Engineering Challenge 1 unit was shorter, but the task’s structure was similar to the Wind Turbine Engineering and Testing task. It had similar performance

opportunities. Therefore, the reported scores are the means of the same three parts as the Wind Turbine task.

5.1.3.2 Determining mean authenticity levels for classroom instruction

Measures of the mean authenticity of classroom instruction for the entire study period are determined to assess this second aspect of the authentic pedagogy students experienced. The total instructional time of lesson segments with high, moderately high, moderately low, and low authenticity during the entire study period are also calculated.

Measures of the authenticity of classroom instruction were also determined for two or three selected instructional units from each trimester. All the measures of authenticity for individual lessons during these instructional units were aggregated and compared. The mean Combined Authenticity and the levels of four components of authentic pedagogy are contrasted between units and across the study period.

Lesson segment lengths varied from just a few minutes to 75 minutes. When calculating the location measures of authenticity of classroom instruction for each instructional unit and the entire study period, the combined and component scores for

each lesson segment were weighted by the length of the lesson segment in minutes using the SPSS Weight Cases procedure. This treats longer lesson segments proportionally to their frequency in the sample, giving greater construct validity to central tendency measures for these variables. Without weighting each lesson segment by its length, shorter lesson segments make up a greater proportion of the means only because they are more frequent.

Table 5.5 lists the selected seven instructional units from Table 5.4 that were observed and scored for authentic pedagogy during at least 60% of the class days associated with these units. Three hundred sixty-four (73%) of the 500 lesson segments scored are from these units. This is 123 hours of class time. The levels of authentic affordances experienced by students in the teacher’s classroom instruction and assessment tasks were determined for each of these instructional units. These affordances include each of the variables in Table 5.1.

These instructional units were chosen to be observed often enough to have a rich enough observational record for a reliable estimate of measures of central tendency for the authenticity of the classroom instruction during them. The observations for these instructional units included most of the learning activities the teacher designed to build knowledge and skills devoted to preparation for and completion of the assessment tasks the teacher used to assess student proficiency and assign grades. They also included most of the instructional patterns, such as, teacher presented task instructions, direct

instruction, individual seatwork, small group work, and student-directed project work, during each of these units.

86

The RNA Fingerprinting Lab unit was included in the analysis because it was observed at least half of the days devoted to the unit. However, closer inspection during post hoc analysis shows that the sample of lesson segments observed isn’t as

representative as the other six units. It did not include the full range of major activities in this unit. Fewer lesson segments and less class time was observed than in the other units. Only 5.5 hours of class time was observed, one-third of the mean class time observed per unit. Days students were performing the RNA Fingerprinting lab activity were observed, but the class time focused on interpreting the lab results were not observed at all.

Based on examination of the teacher’s lesson planning and class activity

documents, and patterns in the teacher’s instruction in other units, it is likely the sample of lesson segments scored for the RNA Fingerprinting Lab unit biased the mean

Construction of Knowledge component score to be lower than would have been measured if some of the missed lesson segments had been included. It is less likely that the mean Science Meaning-Making Processes and Value Beyond School component scores are inaccurate.6 _{Interpretation of these results for this unit take this into account.}

Because of the ordinal nature of the underlying Lesson Authenticity Indicators, the convenience samples used, and the absence of interrater reliability, an appropriate approach to describing and detecting differences in mean authenticity levels for classroom instruction for each unit is the one adopted for the assessment tasks. The normalized component and combined mean scores are categorized as: not authentic (0- 0.19), to moderately low authenticity (0.2-0.39), moderate authenticity (0.4-0.59), moderately high authenticity (0.6-0.79), and highly authentic (0.8-1).

5.1.3.3 Determining authentic pedagogy levels for selected instructional units Table 5.5 shows that some of these instructional units were assessed by multiple assessment tasks. The teacher’s flexibility in designing the curriculum to meet

instructional goals and respond to student needs required special treatment of the assessment task scores from three of the seven instructional units, so that mean task authenticity was a measure of student experience for the instructional unit as a whole.

First, Table 5.5 shows two assessment tasks were used by the teacher to assess student proficiency for the RNA Fingerprinting Lab Analysis instructional unit. However, scores for the lab analysis assessment task were used as the measures of authentic affordance for this unit instead of the RNA Fingerprinting Quiz because the quiz was only made available to students at the end of the trimester as an additional opportunity to demonstrate proficiency. The RNA Fingerprinting Lab Analysis task was the task the teacher was preparing students for during the classroom instruction scored for authenticity in this instructional unit.

Next, Table 5.5 shows the Immune System & Flu Vaccines instructional unit was assessed by four tasks that the teacher used to assess two Learning Targets. Even though the teacher assessed proficiency on the two Learning Targets independently, the teacher taught this content in a sequential, connected context focused on understanding the decision whether to get an influenza vaccine. Therefore, all the lesson segments addressing both these Learning Targets are considered one instructional unit. Authenticity scores for classroom instruction from all these lesson segments are

88

The teacher focused his instruction on the public service announcement task in class, but made it clear to students that the quiz format would be an option for them. Students freely chose between these options. Students worked on either the PSAs or the quizzes during most of the lessons observed. Based on field notes, about 90% of the students chose the PSA option.7

Therefore, observations of student behavior and classroom instruction during this instructional unit were primarily influenced by the PSA tasks for Learning Targets 1.4 and 1.5. Post hoc it was decided that the most valid measure of the authenticity for these tasks comparable with the measure of the authenticity of classroom instruction for this instructional unit was the mean of the authenticity scores for only the two PSA tasks.

Lastly, Table 5.5 lists two assessment tasks and two Learning Targets for the Human Characteristics Lab Analysis unit. Data display in graphs and tables is assessed by the first of these Learning Targets. The interpretation of scientific data is assessed by the second Learning Target. This unit included both of these. In this case, the teacher judged that the first of the tasks for the unit in Table 5.5 was completed by students before sufficient instruction on all the content.

What the teacher originally intended as a combined assessment became a

formative assessment. The second task was a second opportunity to show proficiency on the same Learning Targets after all of the content in both Learning Targets was taught. Therefore, the mean of the scores from the two similar tasks is used as the measure of the authenticity of the assessment tasks in this unit. This mean was compared with the mean

of authenticity scores from all the lesson segments scored for classroom instruction in the unit.

The other four instructional units were each assessed by one assessment task. The authenticity scores for that task were compared with those for classroom instruction from all the lesson segments in the unit.

5.1.3.4 Interpreting classroom instruction and assessment task authenticity levels Scoring some of the TAIs and LAIs include common elements, as can be seen in the scoring guides in Appendix A and Appendix B, respectively. The scoring guides for LAIs include both specific observations of teacher and student behaviors and judgments on the cognitive complexity afforded by the teacher’s instruction observed during each lesson segment.8 _{The judgment of cognitive complexity involved in scoring LAI 1, for}

example, may be influenced by this investigator’s judgment of cognitive complexity when he scored TAI 1 and TAI 2 for the assessment task(s) for the same instructional unit.9 _{Some correlation between the two measures for the same instructional unit is an}

artifact the construction and implementation of the measurements.

While not completely free of overlap with the task authenticity indicators (TAIs), the Lesson Authenticity Indicators consider many factors that are not part of the TAIs. Central tendency measures of collections of lesson segment scores are also weighted by the time students experience instruction at different LAI levels, which is independent of TAI scoring judgments. The instrument for classroom instruction was designed to produce a valuable measurement of student experience of affordances for constructing

90

demonstrating success in school. It is believed that the methodology used produces a measurement with value beyond that in the measurement of the same affordances in the assessment tasks.

This study’s measures of two aspects of authentic pedagogy are designed to consistently implement the common theoretical model from the Center for Organization and Restructuring of Schools (CORS) underpinning both measures. Both measures construct the components of authenticity in the CORS instruments from discrete indicators, but there are differences.

This study’s scoring guide for the assessment tasks has the same indicators with the same scoring scale as the CORS scoring guide (Table 5.3). They were just adapted for science and engineering. However, the indicators and scoring protocol for the instrument for authenticity of classroom instruction is quite different. As described above, it uses the set of discrete indicators in Table 5.6 to score the same four authenticity components in the CORS instrument. Component scores are determined from algorithms described in Table 5.7 and designed to simulate the set of levels in the CORS scoring guides. This is different from choosing a level from one scoring guide for each component in the CORS instrument.

There are also differences in level of analysis. Each assessment task measurement describes that task. Table 5.5 shows when the mean of two assessment tasks are used to characterize the authentic affordances in those tasks for each of the seven selected

instructional units. However, levels of authentic affordances in classroom instruction for each instructional unit are the mean authenticity scores from many lesson segments

scored during that unit. These means also take into account class time in each lesson segment to better reflect student experience of authenticity.

There is a theoretical basis for concluding that the two instruments measure the same constructs on a scale from high authenticity to moderate to low to no authenticity. However, the differences in the instruments – and for classroom instruction, the

convenience samples and time-weighting of lesson segment scores – mean that this study will not claim the scales are quantitatively calibrated. Still, large differences within and between the measures suggest real differences.