5.3 TYPES OF FEEDBACK PROVIDED
5.3.2 Results of Comment Coding
Students made a total of 3,038 feedback comments in 9th and 12th grade by a total of 163
students. Because a focus of this study was to analyze the change in student feedback comments from 9th to 12th grade, it is important that analyzed feedback comments come from students who participated in providing feedback in both 9th and 12th grade. Therefore, I eliminated students who did not provide feedback in 9th or 12th grade from the analysis. Thus, I created a subset of 74 students who provided feedback in both 9th and 12th grade. When compared to the students with writing whom raters scored using the analytic rubric, this group of 74 students includes all 21 students whose essays I used for in-depth analysis of writing features, and six students whose writing was scored on the analytic rubric, but not used for in-depth writing analysis. The number of opportunities this subset of students had to provide peer feedback in all their classes from 9th to 12th grade ranged from a low of 47 peer feedback assignments to a high of 72 peer feedback assignments (M=60.85; SD=7.58). Typically, Mr. Zain asked students to provide feedback on the writing of at least three peers per assignment. However, I found many students did not provide feedback on every paper they were assigned for review. This was especially true in 9th grade; students in 9th grade frequently provided feedback to only one peer and not to the other peers assigned to them for review in SWoRD. This subset of students generated 648 comments in response to the 9th-grade feedback prompts and 1,062 comments in response to the 12th-grade prompts.
I coded each feedback comment provided in 9th and 12th grade as being one or more of the following: high critique, explanation, low critique, vague, no critique, or wrong (see feedback coding description in Table 3, chapter 3). Figure 4 shows the distribution of types of feedback comments in both 9th and 12th grade.
Figure 4. Percentage of Comment Types Made in 9th and 12th Grade
I measured improvement in feedback quality by comparing the percentage of individual student comments coded as high critique, explanation, low critique, vague, and wrong in 9th grade to the percentage of each type of comment an individual student made in 12th grade. I measured improvement in feedback quality by a positive change in the percentage of comments coded as high critique and explanation from12th grade to 9th grade and a decrease in the
percentage of comments coded as no critique, low critique, vague, and wrong. Overall, students’ feedback comment quality improved from 9th to 12th grade, both in terms of an increase in high critique comments and a decrease in three of the other categories. T-tests were used to determine if there was a significant change in the types of feedback comments that students provided from 9th to 12th grade.
There was a statistically significant decrease (p<.001) in the percentage of comments coded as no critique from 9th grade (M=0.67; SD=0.18) to 12th grade (M=0.52; SD=0.23). The decrease in the percentage of comments coded as no critique may indicate that students provided
less praise to their classmates in 12th grade. However, the decrease may also be related to the prompts that students were asked to use when providing feedback. One prompt at 9th grade asked students to copy and paste the thesis statement from their classmate’s writing and only to provide a comment if the thesis statement was unclear. Some students provided a comment along with the copied sentence; however, many students only copied and pasted the sentence they
understood to be the thesis without providing a critique. Coders coded copied and pasted sentences without critique as no critique.
There was a statistically significant decrease (p<.001) in the percentage of comments coded as vague from 9th grade (M=0.24; SD=0.17) to 12th grade (M=0.12; SD=0.11). The
decrease in comments coded as vague indicates that students’ comments in 12th grade were more specific, identifying a problem that was specific to the essay under review and locating that problem as opposed to providing a general comment that could be applied to any essay (i.e., use spellcheck). The decrease in comments coded as vague may be a result of the way that Mr. Zain wrote several of his comment prompts in 12th grade. The comment prompts Mr. Zain asked students to use for the dystopian essay included the word “explain,” which may have lead students to not only explain their thinking but to be specific about problems when writing comments to their peers.
There also was a statistically significant decrease (p=0.03) in the percentage of comments coded as wrong from 9th grade (M=0.03; SD=0.07) to 12th grade (M=0.01; SD=0.02). The
percentage of comments coded as wrong was small in 9th grade, and students wrote these comments in response to the comment prompt that asked them to copy and paste their peers’ thesis statements. Comments coded as wrong typically had the wrong sentence from the introduction or incorrectly stated that there was no thesis statement. The small amount of
comments coded as wrong in 12th grade ranged from comments such as, “They didnt put no direct quotes at all so no evidence to support” when the student’s essay used textual evidence, to comments that incorrectly stated that the focus of the essay was not clear, “it didnt tell me which prompt they were writing about it was just talking about dystopia.”
The change in the percentage of comments coded as low critique was not statistically significant (p=0.37) from 9th grade (M=0.20; SD=0.16) to 12th grade (M=0.22; SD=0.15). The percentage of comments coded as low critique increased in 12th grade. I did not expect that the percentage of low critiques would increase over time; however, the increase may have been a result of students no longer being asked to copy and paste sentences from their peers’ essays as opposed to providing comments that focused on content, grammar, language, and organization. Therefore, there were more opportunities for students to provide comments that could be considered low critique in 12th grade.
There was a statistically significant increase (p< .001) in the percentage of comments coded as high critique from 9th (M=0.14; SD=0.17) to 12th grade (M=0.23; SD=0.19). Students in 12th grade provided more comments on their peers’ papers about specific content issues (ideas and reasoning), that if revised would improve the overall quality of the writing, than they did in 9th grade. The increase of comments coded as high critique indicates that over time, students improved in their ability to identify content issues in their peers’ writing. In contrast, there was no statistically significant change (p=0.37) in the percentage of low critique comments made from 9th (M=0.20; SD=0.16) to 12th grade (M=0.22; SD=0.15). The lack of a significant change in comments coded as low critique indicates that although students improved at providing comments on content issues, comments coded as high critique, they continued to provide a large amount of editing comments, comments coded as low critique. Although editing comments are
useful in polishing a paper, these are not comments that would help a writer improve content during a first revision.
Additionally, there was no statistically significant change (p=0.86) in the percentage of comments that provided an explanation from 9th grade (M=0.14; SD=0.18) to 12th grade (M=0.14; SD=0.19). The lack of a significant change in comments coded as explanation
indicates that students did not improve their ability to provide an explanation in their comments. However, there was a statistically significant increase (p=0.001) in the percentage of comments coded as high critique and explanation from 9th grade (M=0.02; SD=0.07) to 12th grade (M=0.08; SD=0.14). Students moved from providing broad critiques about the content of their peers’ essays (i.e., “Your thesis statement is wrong.”) to providing critiques that utilized explanations to show the writer the specific problem and suggest how that problem may be corrected (i.e., “Your thesis statement only says the theme is ‘mother and son relationships’; but your conclusion is more towards how not all mother-son relationships are happy. I think the second one relates more towards the paper as a whole, and you should use that one.”).
Research has shown that receiving high critique comments with explanation lead to students making revisions that improve the overall quality of their writing (Cho & MacArthur, 2010; Patchan, Schunn, & Clark, 2011). Feedback is more likely to be implemented if the writer understands the problem being identified by the reviewer. A comment about a content issue with an explanation of the problem has the potential to give the writer insight into why the reviewer took issue with a specific portion of an essay. Without an explanation, students may ignore the comment because they may not see it as being correct or because they do not have enough information to address the problem (Nelson & Schunn, 2009). In their study of middle school science students, Hovardas, et al. (2014) found that students who received peer feedback
comments that explained critiques were more likely to revise their writing and utilize the comment provided than students who received feedback without explanations. Tseng & Tsai (2007) found similar results in their work with high school students utilizing peer feedback for revision. Both studies found a significant increase in writing scores when students utilized effective feedback comments for revision.
5.4 CORRELATION BETWEEN WRITING SCORE AND TYPE OF FEEDBACK
PROVIDED
In this section, I will explore the relationship between students’ writing scores and the types of feedback that they provided to their peers. I used Pearson’s r to assess the relationship between students’ writing scores in 9th and 12th grade and the types of comments students made in those respective grades. For this analysis I used the same subset of 21 students who had both writing and peer feedback in SWoRD as was used in chapter four. I used this subset because it allows me to look at relationships of writing score and feedback type for students who participated in the writing and feedback assignments in both 9th and 12th grade as opposed to students who may not have had full participation.