the previous section was to determine what TCs identified in terms of features of AE and to what extent they incorporated those features of AE into their lesson plans after they attended the training session. First, TCs made an extremely large number of
identifications on the pre-test, post-test, lesson planning worksheet, and lesson plan. I needed to devise a way of organizing these identifications in a way that would help me to make sense of what TCs had learned about AE and what they might still find challenging.
As I examined what TCs included on these documents, I noticed that there were different types of identifications. Some of the identifications that TCs made were names or descriptions of features of AE which matched the description of the term “language feature” provided on the pre-test and post-test. Language features were described as “vocabulary words and phrases”, “sentence structures (simple, compound, complex) and
62 other grammatical features (such as verb tense, prepositional phrases, passive voice)”, and “the amount of language students need to use and the structure of language (e.g. a four-step procedure, a bulleted list, a paragraph with a topic sentence, three details, and a concluding sentence)”. These names or descriptions of features of AE that TCs included could then be further distinguished into specific and non-specific categories. In specific identifications, features were named or described in a way that demonstrated that the TCs could identify what the language feature looked like, such as, “Tier 3 words: asyndeton, metonymy, polyptoton, alliteration, elision, anaphora”. An identification that said only “Tier 3 words” was not considered specific. Although the identification “Tier 3 words” did demonstrate that a TC understood the concept of “language feature”, it did not
specifically name an actual feature of AE. TCs also included identifications that were not names or descriptions of features, which I categorized as “not features of AE”.
By sorting TCs’ identifications into the categories of specific features, non- specific features, and not features of AE, I was able to organize a large amount of data. This organizational scheme allowed me to both count the number of identifications of features of AE that TCs made and describe the types of features that TCs identified. Also, in categorizing the identifications in this way, patterns could be discerned in terms of the identifications that TCs made which were not features of AE, patterns which might be helpful in determining what TCs found challenging about identifying features of AE. In Figure 3.2, a visual model is shown to demonstrate how the identifications were sorted into the three categories.
63
Figure 3.2. Visual model of the process for categorizing identifications.
On the pre-test, post-test, and worksheet, TCs were asked to classify their identifications as word, sentence, or discourse level features of AE. For this reason, identifications of specific features of AE on these documents were also evaluated as to whether they were listed correctly as word, sentence, or discourse level features. As explained on the pre-test and post-test, words and phrases were word level features, sentence structures and grammatical constructions were sentence level features, and descriptions of amount or structure of language to be used were discourse level features.
I designed two documents on which to record the categorization of the
identifications. To assess TCs’ identifications of features of AE on the pre-test and post- test, I designed scoring guides. Figure 3.3 shows the scoring guides used to score the sections in which TCs’ named features of AE on the pre-test and the post-test. There is a category on the scoring guide in which to mark any identification that TCs should have made but did not. Any identifications that TCs made on the pre-test or post-test that were
64 not names or descriptions of features were recorded in a blank space below the scoring guide. I field tested the pre-test scoring guide using the pre-tests of TCs who had chosen not to participate in the research. . One TC’s post-test and a completed scoring guide are included in Appendix D to illustrate how the scoring guide was used.
Figure 3.3. Scoring guides for the pre-test and post-test.
Evaluating the lesson planning worksheets and lesson plans that TCs submitted was a bit more complex than evaluating the pre-tests and post-test, because, unlike on the
65 pre-test and post-test, I could not predict which features of AE TCs might name or how many features TCs might name on the worksheets or incorporate into their lesson plans. I created a recording sheet on which the identifications that TCs made on the lesson
planning worksheet and the lesson plan could be documented. On these sheets, there were spaces to record exactly what TCs wrote on these documents as well as charts in which the number of identifications they made could be recorded. The charts had separate spaces for identifications made at the word, sentence, and discourse level. On the
worksheets and lesson plans, a set or list of identifications of the same type of feature was counted as one identification. This procedure was followed in order to maintain
consistency in counting identifications across documents, as groups or sets of features were counted as one identification on the pre-test and post-test (e.g. photosynthesis,
decompose, atmosphere, and fossil fuels were all counted as one identification, called
“science vocabulary”). In Figure 3.4, a section from the recording sheet used to document identifications from the worksheet and a section used to document identifications from the lesson plan are shown. The sections used with the worksheet needed to include space for identifications of specific language features that were classified correctly and
incorrectly. Appendix D includes one completed example of each section of the recording sheet to illustrate how these sections were used to organize identifications that TCs made on the lesson planning worksheet and the lesson plan.
66
Figure 3.4. The recording sheet.
Using the recording sheet also allowed me to determine the extent to which TCs incorporated features of AE into their lesson plan. TCs had been instructed in the training session to name specific features of AE in the language objective(s) and then describe how they would teach those features to students in the lesson procedure section. If, in a
67 lesson plan, a TC had named specific features of AE, I noted on the recording sheet if the TC included the features of AE in the language objective(s), in the lesson procedure section, or both. I was then able to sort the lesson plans into groups based on the extent to which features of AE were incorporated into the lesson plan.
Since TCs were asked to supply their own identifications, the TCs’ responses on the pre-test, post-test, lesson planning worksheet, and lesson plan varied greatly in wording and specificity, leaving room for judgment in terms of categorizing these
identifications. I employed a second rater to work with me in evaluating these documents. The second rater was a fellow doctoral student with a background in linguistics and some familiarity with the concept of AE from her experience both as a research assistant and as a grader for the teaching CLD students course. We met three times, once to score the pre- tests, once to score the post-tests, and once to evaluate the lesson planning worksheets and lesson plans. At each meeting, we followed the same procedure. We reviewed the criteria used to categorize identifications, and then we used the criteria to evaluate the identifications on one document together. We discussed what features the TC named in the identification, in what category we would place the identification, and for what reason we would categorize it in that way. Once we were in agreement on the scoring of that document, we would then fill out the scoring guide or recording sheet for that document. Then we would evaluate a set number of documents independently and stop to compare categorizations when we finished. Discrepancies in categorization were discussed, and, for each discrepancy, we explained our reasons for placing the identification in that category. We discussed previous decisions regarding categorization and referred back to
68 the explanations from the pre-test and post-test to help us in coming to consensus on a category.
When we had evaluated the identifications on all documents, I calculated the scores on the scoring guides and counted up the numbers of types of identifications on the recording sheets. I entered the numerical data from the scoring guides and recording sheets, along with the demographic information for all participants, into statistical analysis software. I used the software to calculate descriptive statistics and conduct statistical analyses of the data.