Performance of Writing and Reading Tests by Method across Studies

The results in this section addresses research question 2: Did any of the three

methods lead to better performance in character writing and reading? The data

were from the writing and reading scores obtained across three phases: immediate tests in the main study, one-week follow-up tests, and four-week follow-up tests. Forty-eight participants completed all the tests after one student withdrew from the follow-up tests. Therefore, the data from 48 participants were included in the analysis. A descriptive analysis offered the means and standard errors of the writing and reading scores that can be compared across three phases. Statistical significance of the effect of method on performance was run through repeated measures ANOVA, and pairwise comparisons of statistical significance were further conducted among the three methods. The research question was addressed through three sub-questions (sub-question 2.1, 2.2 and 2.3).

Sub-question 2.1: Did the pictorial method produce higher scores than the English and verbal methods in the immediate and follow-up writing tests of the 18 characters learned in the teacher-select session? Figure 4.2 presents the means and standard errors of the writing performance by method across study phases. To maintain consistency for the writing across the study phases, only 18 characters in the teacher-select session were included in the analysis. The writing scores were evaluated by three evaluators.

The result shows that the writing scores generated from the pictorial method achieved the highest among the three methods, averaged 27.49 (SE=.45), in the immediate recall. The verbal method scored closely behind (M=27.35, SE=.46), and the English method (M=25.98, SE=.66) scored relatively lower than the other two methods. These results align with the participants’ perceptions of IBES use and their behavioral responses discussed in the last research question, indicating accuracy of the participants’ perceptions of their abilities to immediately recall character writing.

However, their perceptions did not seem to apply to the retention tests. The two follow-up writing tests yielded almost the same results in the writing retention rates. The means dropped to 8.11 (SE=.46) and 7.95(SE=.50) in the one-week and four-week follow-up tests, respectively, in the English method, with an average

decrease of 69% across phases. In the pictorial method, the means dropped to 7.23 (SE=.26) and 7.07 (SE=.33) in the two follow-up tests, the highest average

decrease of 74%. The verbal method showed the least decrease, with the means hit 15.16 (SE=.82) and 14.57 (SE=.76), a 46% decrease. However, what is interesting to notice are the relatively larger standard errors in the verbal method. The errors may indicate that there may be a greater variability in the participants’ abilities to take advantage of the verbal method for character learning and memorization. The inconsistency of the performance in the pictorial method between the immediate writing and two follow-up writing results suggest that the pictorial method worked the best only for character immediate recalls; the effect attenuated sharply in the retention tests.

Table 4.3 shows statistical significance results of the effect of method on writing performance across three phases. Specific information about differences between each method is also presented.

Table 4.3.

Repeated Measures ANOVA and Bonferroni Pairwise Comparisons of the Effect of Method on the Writing Performance across Three Study Phases

Immediate Follow-up 1 Follow-up 2

Task Method df F p df F p df F p

Writing Overall 2 5.93 .00* 2 84.65 .00* 2 79.38 .00*

E-P .01* .17 .11

E-V .63 .00* .00*

P-V 1.00 .00* .00*

Note. * The mean difference is significant at the .05 level. E=English method; P=pictorial

The results show that methods significantly impacted the participants’ writing performance at all three phases, F(2, 48)=5.93, 94.65, 79.38, p.01. Comparatively, methods had a higher significant effect in the retention of writing than immediate writing. To be more specific, in the immediate writing recall, there was a

significantly different effect between the pictorial method and the English method (p=.01) on the writing performance, but the effect was not significant between the English method and the verbal method. There was no significant difference

between the pictorial and verbal methods. However, both the follow-up writing tests showed a significant difference between the verbal method and the other two methods (p.01).

To sum up, the descriptive and significance tests revealed that the pictorial method only had significantly better effect than the English method in the

immediate writing recall. However, this advantage dropped dramatically in the two follow-up tests, down to the lowest writing recall rates compared to the other two methods. In contrast, the verbal method showed significantly better effect on character retention than the other two methods, and the effect lasted for three weeks.

Sub-question 2.2: Did the pictorial method produce higher scores than the English and verbal methods in the immediate and follow-up semantic tests of the 18 characterslearned in the teacher-select session？Figure 4.3

demonstrates the means and standard errors of reading performance for 18 characters by each method across two study phases, i.e., the main study and

one-week follow-up study. The reading scores were obtained from the semantic tests used in the main and the first follow-up studies (Appendix H). Considering a different test used in the second follow-up study, the reading scores in the third phase were not included in the descriptive analysis.

The results in Figure 4.3 show consistent differences in reading performance in the three methods and across study phases. The means in the two phases ranged from 13.67 (SE=.42) to 12.02 (SE=.47) in the English method, an average accuracy rate of 71% with a 12% decrease; and, from 14.38 (SE=.41) to 13.02 (SE=.52) in the pictorial method, an average accuracy rate of 76% with a 9% decrease. The verbal method, interestingly, experienced a very slight increase from 15.02 (SE=.39) to 15.32 (SE=.49), an average accuracy rate of 84% with a 2% increase.

Table 4.4 shows statistical significance results of the effect of the method on reading performance across three phases. Meanwhile, specific information about

differences between each method is also presented. The results in the second follow-up reading test were from the scores of the old-new character identification test (Appendix I) on 18 characters learned in the teacher-select session. The results in the second follow-up test are used to address sub-question 2.3 (i.e., Did the pictorial method produce higher scores than the other two methods in the old-new character identification test?).

Table 4.4.

Repeated Measures ANOVA and Bonferroni Pairwise Comparisons of the Effect of Method on the Reading Scores across Three Study Phases

Immediate Follow-up 1 Follow-up 2 a

Task Method df F p df F p df F p

Reading Overall 2 8.13 .00* 2 25.31 .00* 2 11.74 .00*

E-P .09 .11 .44

E-V .00* .00* .00*

P-V .28 .00* .00*

Note. * The mean difference is significant at the .05 level. E=English method; P=pictorial

method; V=verbal method; a _{The follow-up reading 2 was the results from the old-new} character identification test on 18 characters learned from the teacher-select session.

Significance analysis showed that methods also had a significant impact on reading performance in the main and first follow-up tests, F(2, 48)=8.13, 25.31, p.01. Pairwise comparisons further testified that the verbal method was

significantly different from the English method on immediate reading (p.01), but not significantly different from the pictorial method (p=.28). The English and pictorial methods did not show significant difference during immediate reading (p=.09). However, in the one-week follow-up reading, the verbal method showed significant higher scores than both the English and pictorial methods (p.01).

Again, similar to the writing results, the reading results rejected the

hypothesis that the pictorial method might produce higher reading scores than the other methods. The verbal method actually had significant advantages over the other two methods in character reading performance, even shown from the

immediate reading test.

Sub-question 2.3: Did the pictorial method produce higher scores than the English and verbal methods in the follow-up old-new character

identification test of the 18 characters learned in teacher-select session? The old-new character identification test served to further examine the effect of methods on character retention rates across phases. In Table 4.4, significance analysis shows that methods had significant effects on performance in this test (p.01). Specifically, the verbal method produced significantly higher scores than the English and pictorial method (p.01). What was surprising was that though the verbal method rated higher than the other methods, the average accuracy rates of the three methods were far lower than the previous reading tests, 31% for English method, 26% for pictorial method, and 38% for verbal method. A previous assumption was that the identification test should be easier than character writing recall, but it was not supported.

Likewise, the results did not support the hypothesis in sub-question 2.3. The pictorial method did not produce higher scores in the old-new character identification test; conversely, the verbal method consistently showed significantly

better effect on reading performance in this test than the other two methods, though accurate recall was limited across methods.

Therefore, to summarize for this section, research question 2 can be settled by combining the results from the three sub-questions. The results showed that methods in general had significant effects on the participants’ performance in writing and reading across study phases. Among the three methods, the verbal method demonstrated better effects in both writing and reading performances than the other two methods, and the effects were even more distinguished in the retention tests. Although the pictorial method showed significant effects in the immediate writing recall, the effect attenuated in the retention tests. The English method seemed to have the least advantage in both reading and writing performance.

Section 4: Factor Analyses of the Use of the Imagery-Based Encoding Strategy