Discussion

Implication for rater training: Rater discrimination. This study showed that rater training focused on rater severity is an ineffective method to improve classification accuracy. Test developers and assessment agencies invest enormous amounts of time and energy to train raters using measures of agreement based on rater severity. This study reiterates an important result that has implication for rater training – raters should begin to focus on improving how well they discriminate between latent classes defined by the scoring rubric, because this plays an important role in determining how well raters classify an essay. This finding had been noted in previous studies (e.g., DeCarlo, 2002), but the literature on CR scoring is still dominated by training focused on rater severity.

As this study showed, changes in rater severity affected classification accuracy in only minor ways.

This study is one of the first to examine rater discrimination over time. Not many studies have examined rater discrimination in the context of rater drift. In fact, most rater models such as the FACETS model and the PC model do not estimate rater

discrimination; these rater models constrain rater discrimination to be equal across all raters. Yet, empirical results from both the teacher certification test and the high school writing test showed notable differences in rater discrimination; in fact, rater

discrimination was normally distributed between raters. Moreover, there were differences in rater discrimination for the same rater over time.

As demonstrated in this study, the use of rater discrimination to identify rater behavior is important and cannot be ignored – rater discrimination cannot be assumed to be equal across all raters. Given the significant role that rater discrimination plays on improving the quality of classification, the inclusion of rater discrimination in rater models is both empirically and theoretically motivated.

Classification of constructed responses. This study showed that a latent class SDT framework to study rater drift is useful as it presents additional insights into the behavior of raters. This approach differs from traditional rater models such as IRT that ranks constructed responses into a continuous latent trait. In the latent class SDT model, the interpretations of the latent classes are derived from the scoring rubric, which provides a natural context for conceptualizing CR scoring. For example, if an essay is classified into a “2,” the scoring rubric provides a detailed description of the ability that the examinee demonstrated through the constructed response. The description provided in

the scoring rubric is important, because raters are trained on the basis of its description to score essays. As such, these classifications provide diagnostic feedback to examinees that are reflected in the scoring rubric used by the raters. On the other hand, it may be difficult to interpret latent scores derived from other rater models, because its relationship with the scoring rubric may not be clear.

Another benefit in using the latent class approach is the derivation of intersection criteria locations. As illustrated in the plots of rater criteria estimates in this study, the intersection criteria locations provide a relative guide on the severity of raters. Estimates of the relative criteria above the intersection criteria location may imply stricter rating, while estimates below may indicate lenient scoring. Although this location may be subjective, the close resemblance it showed with parameter estimates in the two real- world examples indicates its usefulness for diagnosing rater severity. These locations cannot be derived using an IRT framework, because the conceptual approach is different in that there are no clear locations to distinguish intersections.

The use of latent classes to examine rater behavior also allows an examination of classification. This measure can be used to compare different patterns of rater drift as demonstrated in this study. In the CR scoring setting, where raters are assumed to classify essays into a score defined by the scoring rubric, classification accuracy provides an important statistic that examines the quality of classification. Given the natural

inclination in CR scoring to measure the classification accuracy of raters, this approach has not been studied in the context of rater drift. In light of these findings, this study adds to the literature in its understanding and implications for rater drift.

Treatment of examinee ability: Discrete versus continuous measures. This study also showed an important distinction between IRT models and the latent class SDT model as models for studying rater drift. Although the main difference between the two models lies in the treatment of examinee ability – whether to treat them as discrete or as a continuous latent measure – this distinction has led to important implications in assessing rater behavior.

This study found that the latent class SDT model is a useful model to examine rater drift. The latent class SDT model was able to detect differences in rater behavior that was comparable to IRT models. However, this study also found that when examinee ability was non-normal, parameter estimates of rater discrimination can lead to greater bias when using IRT models in comparison to the latent class SDT model. In both the latent class SDT model and the GR model, rater discrimination is the slope parameter of examinee ability. When examinee ability is treated as a continuous latent variable, the variance of examinee ability can affect parameter estimates of rater discrimination, which can subsequently also affect estimates of rater severity. As demonstrated in the high school writing test, examinee ability can be non-normal in that there can be a greater concentration of scores in the middle categories. Given that most IRT models assume examine ability to be normally distributed with fixed variance, this assumption must be checked for determining the type of rater model to use.

Studies of rater drift constitute an important and practical aspect of educational measurement. The use of CR items to measure examinee ability is increasing, and an attempt to understand errors resulting from human scoring behavior serves as an important step to refining how CR items should be measured. The study of rater drift is

important with respect to this growing area of CR scoring, because rater errors associated with changes in their behavior and their effect on model-based scores have not been studied comprehensively. With respect to these considerations, this study adds to the growing literature on assessing student ability based on subjective measures of rater scores. As this study concludes, the findings from this study provide new and important understanding of CR scoring and issues that emerge in practice, especially in exploring the effect rater drift has on different rater models.