Given the complex nature of ABV inquiry, a pre-research pilot was used to refine the surveys and other study instruments and to provide interview practice for the investigator in order to help reduce research bias and for the purpose of best addressing the research questions.
The data collection in the four phases of the study was aimed at validating the TPA as a measure of readiness. The researcher collected four candidate surveys: pre-TPA experience, during- TPA experience, post-TPA submission experience, and post-ST experience. In addition, the
researcher and a second interviewer conducted case-studies with six candidates during the pilot. Case study candidates were interviewed following the same phase schedule for data collection–pre, during, post-submission and post-ST. Mentors, supervisors and faculty were also surveyed and interviewed. Finally, West E and TPA raw data were scored by an external scoring agent, Pearson.
1. Pre-TPA: The primary purpose of data collection prior to experience with the TPA was to establish a base-line for both attitudes about ST and the TPA and candidates’ self-reports regarding their current abilities and weaknesses.
2. During-TPA: Data collected during the taught segment and TPA writing up process reveals candidate thinking about the instrument during completion. In particular, candidates are asked to provide information about areas of struggle and mastery.
3. Post-Submission: This data provides the initial reaction to the instrument after submission. Questions emphasized the formative and summative learning experience provided.
4. Post-ST: Data collected at the culmination of the term asked participants to review the TPA with respect to the entire experience of ST.
A quantitative component included the collection of data using ST evaluations from mentor(s) and supervisors which also included observation ratings on the entire sample of candidates, score reports for the fifteen sub-traits of the TPA, four surveys administered to
candidates (one in each phase), three surveys administered to mentors and supervisors (beginning, post-TPA, post-ST), and university summative instruments. Where possible, scores on the fifteen sub-traits of the TPA were compared with the corresponding variables in ST evaluations and
observation ratings to see whether there were any systematic differences between the two rankings of candidate abilities. In addition, the responses of candidates were analyzed to examine the factors that impacted performance on the TPA. The sample sizes for each of the quantitative instruments are found in Table 3.3.
In analyzing responses from the surveys and case study interviews, differences between the undergraduate and graduate programs were found. These differences may be a result of program components or they could be attributable to age and experience, length of placement, and comfort with placement setting or other factors. While all candidates had similar coursework, there were some significant variances in programs, most importantly the length of time spent in the placement. Most elementary candidates in the undergraduate cohort and candidates in the graduate program had two full semesters (32 weeks) at their placement site working with the same mentor and classroom culture. This placement started out part-time and moved to full-time in the second semester. Secondary candidates in the undergraduate cohort had only twelve weeks of ST in one
semester (starting with one day a week and developing to full-time for ten weeks). Most
importantly, there were differences in the districts and school contexts for ST placements. Some candidates were at high-need, high-risk schools and others were placed at schools from backgrounds of socio-economic advantage. Variables in cohort experiences will be discussed below and in more detail in the analysis chapter (Chapter four) and were taken into account when selecting participants for the case studies.
The qualitative component of the study included case study interviews of six candidates. Qualitative methods are useful for identifying patterns and casual linkages that are not apparent in the quantitative data. Three case study candidates participated from the graduate program and three from the undergraduate program. Cross-case studies from the same institution allowed for an examination of both the meaning candidates made of the TPA (motivation, view of consequences of scores) and the placement contexts and preparation factors that better prepared or prevented candidate learning during the teaching event. The primary purpose of the case studies was to closely examine the experience candidates underwent as they completed the portfolio assessment and whether they believed the assessment accurately measured their readiness and preparation to teach (for licensure). The second purpose of the case studies was to identify the contextual factors that impacted the experience, and ultimately, the outcomes. If placement and context of ST proved to be a significant factor in TPA success, as measured by scores on sub-traits, but those factors were perceived beyond the control of the candidate, it would raise questions about the consequential validity of the operationalized construct of the TPA. Finally, case studies offered the IHE
opportunities to examine the types and levels of support needed for future candidates. Because the TPA and the length of ST were theorized to have an influence on teacher learning (Chung, 2005), case study subjects were selected based on literal replication logic (cases are expected to produce similar results) as well as theoretical replication logic (cases are expected to produce contrasting results but for predictable reasons) (Yin, 1994). Based on program membership, placement levels, and TPA subject, these six candidates represent multiple levels of placement
diversity. For each program, one elementary, one secondary and one middle school candidate was selected in order to evaluate long and short-term placement length and program differences on TPA experiences and performance. Table 3.15 lists the case study subject, their program, and placement. Table 3.15
Case Study Subjects and their Placements
Elementary Middle School Secondary
Undergraduate Year Placement Jason (4th grade) Undergraduate Semester
Placement Jane (7th grade Math) Jill (9-12 English / Theatre Arts) Graduate
Year Placement
Jackie (1st grade) Jamie (5th grade) Jennifer (10th grade Social Studies / Science)
Differences in programs and placements will be further examined in Chapter four. Case study participants were intentionally selected based on these criteria (program and placement). In addition, placements spanned three different school districts and also varied by the number of mentors assigned to each candidate. Differences between districts and schools are discussed below.