Analyzing and triangulating data across multiple sources required several steps. First, interview data were coded using a framework developed and refined by the
research team that was aligned with the study’s conceptual framework. Second, multiple observations were consolidated for each teacher to identify patterns or characteristics of individual teachers’ practice that were observable across observations. Third, using a combination of coded interview data and observation write-ups, profiles of assessment practice were constructed for each teacher in the study. Fourth, teacher responses to
the two misconception scenarios (fall and spring interviews) were analyzed to construct a typology of teacher responses to student misconceptions or errors. Finally, teacher profiles and responses to misconceptions were analyzed in relation to individual teacher MKT scores to explore relationships between teachers’ mathematical content knowledge and their assessment practices.
Interview data. All interviews were professionally transcribed and analyzed using Atlas.Ti qualitative data analysis software. The study’s conceptual framework was used to develop an extensive set of codes, which were used to sort interview data into descriptive categories. This code set had five primary domains. The first three domains related to steps of the cycle of instructional improvement as presented in our conceptual framework: data collection, data analysis, and action. The fourth domain focused on position-based roles (e.g. principal, instructional coach) that were instrumental to the interim assessment process. A fifth domain captured data related to how the use of interim assessments was supported through professional development, coaching, or other processes; and a sixth domain focused on curriculum. Additional codes were added to mark specific segments of interviews (e.g. data or misconception scenarios); to capture specific participant expectations, satisfaction, or desires about the use of interim assessments; or to identify other important school or district factors (e.g. descriptions of the accountability system or school leadership) that might influence the use of interim assessments.
Each code was clearly defined by the research team; definitions were refined and modified through multiple rounds of trial coding. In this process, all members of the research team coded a single interview independent of one another, and then worked collaboratively to refine and clarify the codes to ensure greater consistency across analysts. Once an acceptable level of consistency was reached, individual researchers were assigned groups of teachers, for which they coded all three interviews.
Once the entire data set was coded, data could be retrieved using a query function that could include a single code, or multiple codes in relation to one another. In addition, the data set could be filtered by individual, school, district, position and grade level, allowing for targeted retrieval of data from subsets of the study sample.
Teacher profiles. Teacher profiles were constructed for all teachers for which multiple interviews and observations were available (n=39). The profiles consolidated data from interviews and classroom observations, focusing on how individual teachers collected, interpreted, and acted on assessment information in three domains:
• Benchmark assessments/practice tests: interim-scored assessments
designed to measure the progress of an entire class over an extended period of time.
• Short-cycle assessments: practices employed by teachers within a single class period to determine the extent to which students grasp a specific concept or task. • Teacher-developed assessments: tools developed or adopted by teachers to
gauge student understanding. While some teacher-developed assessments may also be short-cycle assessments, others may extend across multiple class periods.
Teacher profiles were constructed using a two-step process. First, the data base was filtered by teacher and several specific queries were run. In total, data were
retrieved for codes related to all aspects of interim assessment use, short-cycle assessment, teacher-administered tests and quizzes, interpretive or diagnostic processes, action based on analysis of assessment data (all types) and response to student misconceptions. These data were used to populate a matrix organized by steps in the cycle (collection, interpretation, action) and formative assessment type (interim, short-cycle, and teacher-developed). Second, classroom observation notes were
reviewed to identify tendencies, routines, or practices that were consistent across multiple observations, and to cross-reference those with the categorized and reduced interview data.
The profiles served three important purposes within the analysis. First, they consolidated and reduced a large amount of teacher-level data around themes and categories most central to an analysis of classroom practice. Second, they provided a more detailed view of teacher assessment practice as a whole, rather than focusing solely on interim assessments. This allowed the research team to analyze how interim assessments were situated with a broader range of teacher routines and practices. Finally, teacher profiles served as an important point of comparison with other classroom level measures such as the MKT and misconception scenarios.
Misconception scenarios. As mentioned above, the goal of the misconception scenarios was to learn: a) whether or not teachers could identify student errors in
mathematics and what these errors told them about students’ thinking, b) what questions they would ask students to learn the extent to which their own interpretations were correct, and c) what instructional steps they would take to address particular
misconceptions. An, Kulm, and Wu (2004) devised scenarios and a categorization rubric to examine teacher response to student understanding of mathematical concepts. Based on a four-part conceptual framework of Knowing Students’ Thinking, these researchers describe ways in which teachers report that they would: a) address students’
misconceptions, b) engage students in math learning, c) build on students’ ideas, and d) promote students’ thinking in mathematics. This analysis rubric is designed to identify and describe practices that have been hypothesized to contribute to “learning
mathematics with understanding,” (Carpenter & Lehrer, 1999), the primary goal of the National Council of Teachers of Mathematics (NCTM, 2000).
Because our research goals and scenario items are nearly identical to those of An, Kulm, and Wu, we used their rubric to analyze the data from our misconception scenarios. At first, we anticipated that we would need to slightly adapt their rubric for elementary school teachers, but it turned out that this was not necessary as their categories were directly relevant to our teachers’ responses, and no additional categories were needed to describe our teachers’ thinking. As shown in Table 2.1, teachers’ responses to the misconception scenarios were coded using one or more of the following components from An, Kulm, and Wu (2004). The number of practices reported as addressing, engaging, building, or promoting was summed for each teacher. Quality of response was not coded in this analysis, although in order for a practice to be counted, the teacher had to provide enough detail so that the particular instructional practice could be understood. For example, a simple response of “I use manipulatives” would not be counted, while a response of “I use base-10 blocks” or “I use paper cutting to focus their attention” would be counted.
Table 2.1. Codes for the Misconception Scenarios (in abbreviated form)
Components of Knowing Student Thinking
Codes
a) Addressing students’ misconceptions
• Address or identify students’ misconceptions • Use questions or tasks to correct misconceptions • Use rule or procedure
• Draw pictures or table • Connect to concrete model b) Engaging students in math learning • Manipulative activity
• Connect to concrete model • Use one representation
• Use more than one representation • Give example
• Connect to prior knowledge c) Building on students’ math ideas • Connect to prior knowledge
• Use concept or definition • Connect to concrete model • Use rule or procedure d) Promoting students’ thinking about
mathematics
• Provide activities to focus on students’ thinking • Use questions or tasks to help students progress in
their ideas • Use estimation • Draw picture or table
• Provide opportunity to think and respond
One member of the research team coded the misconception scenarios. Because prompting was inconsistent across the interviews, only teachers’ initial responses were coded; in other words, teachers’ responses that followed a researcher’s prompt were not coded for this analysis. However, if a teacher responded to a researcher’s request for clarification, that clarification was coded.
After the data set was coded, data could be retrieved using a query function that could include a single code, or multiple codes in relation to one another. In addition, the data set could be filtered by individual, school, district, position and grade level, allowing for targeted retrieval of data from subsets of the study sample. The resulting codes were also entered into SPSS to facilitate linking with other quantitative measures and to
improve data presentation. The descriptive statistics for these codes are presented in Table 2.2.
Table 2.2. Descriptive Statistics of KST Practices (n=31)
Minimum Maximum Mean SD
Addressing 1 8 5.13 1.65 Engaging 0 6 2.67 1.71 Building 0 6 2.35 1.62 Promoting 0 5 2.45 1.73 Total KST practices 3 20 11.80 4.41
MKT scoring. The returned CKT-M surveys were scored and double checked by two members of the research team. The data was entered into SPSS. In keeping with recommendations from the CKT-M development team, the number of items each
teacher got correct was transformed into a z-score, indicating his or her rank among the sample. The use of z-scores is appropriate in this instance because our analytic goal is to explore the possible reasons for and consequences of variation in MKT within our sample and not compare individuals or groups in our sample to any external criterion. We have found that this method of scoring has made the administration of the CKT-M more friendly for teachers, administrators, and unions, who may be understandably reticent to participate in any process that resembles a non-contractual performance evaluation.
Because the number of items administered was adequate only to form three categories of teachers (see above section on Data Collection), the distribution of z- scores was then examined for potential groupings (see Figure 2.3). This examination revealed a slightly bimodal distribution with peaks both below -1.00 SD and above 1.00 SD. This led us to create a categorical variable for each teacher indicating low (below - 1.00 SD), medium (between -1.00 and 1.00 SD), or high (above 1.00 SD) levels of MKT, with 8 teachers in the low group, 17 teachers in the medium group, and 7 teachers in the high group.
References
An, S., Kulm, G., & Wu, Z. (2004). The pedagogical content knowledge of middle school mathematics teachers in China and the U.S.. Journal of Mathematics Teacher Education, 7, 145-172.
Carpenter, T.P., & Lehrer, R. (1999). Teaching and learning with understanding. In E. Fennema & T. Romberg (Eds.), Mathematics classrooms that promote
understanding, (pp. 19-32). Mahwah, NJ: Erlbaum.
Hill, H.C., Rowan, B., & D.L. Ball. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42, 371-406.
Hill, H.C., Schilling, S.G., & D.L. Ball. (2004). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105, 11- 30.
National Council of Teachers of Mathematics (NCTM). (2000). Principles and Standards for School Mathematics. Reston, VA: NCTM.
Stecher, B., Le, V-N., Hamilton, L., Ryan, G., Robyn, A., & Lockwood, J.R. (2006). Using structured classroom vignettes to measure instructional practices in