Controlling for the Influence of Test Attitudes on the Retesting Effect

Although most external influences on retest performance are unsystematic and comprise the random error present with all testing (e.g., test-taker mood, illness, distraction, nuances in test administration), fluctuations in error variance that

59

systematically covary with testing administrations present plausible alternate explanations for the retesting effect (Anastasi, 1981; Messick, 1989). Accordingly, researchers have often explained retest increases by citing a decrease in debilitating test attitudes (e.g., test anxiety; Cassady & Johnson, 2002; Hausknecht, Day, & Thomas, 2004; McCarthy & Goffin, 2005) or, relatedly, increases in facilitating test attitudes (e.g., test motivation, test familiarity; Chan, Schmitt, DeShon, Clause, & Delbridge, 1997; Lievens et al., 2005; Reeve & Lam, 2007). Thus, test attitudes are a source of error variance that suppress initial test scores, but exert less influence at retest, and therefore, increase retest scores and subsequently enhance validity as the error variance is removed.

A substantial body of literature exists investigating the influence of test-taker attitudes on initial test scores, thus it is not unreasonable to posit that test attitudes systematically vary upon retest and similarly affect retest performance (Reeve & Lam, 2007). Van Iddekinge et al. (2011) present evidence that test attitudes covarying with retesting would not only increase test scores but affect validity coefficients as well. Various test attitudes may affect test (and retest) performance, the present discussion is limited to the two test attitudes most likely to covary with retesting, reduced test anxiety (Cassady & Johnson, 2002; Messick & Jungeblut, 1981) and increased test motivation (Hausknecht et al., 2002). Although, it is certainly plausible that test attitudes

systematically covary with retesting, few, if any, studies have directly examined the fluctuation of test-takers’ attitudes at both initial and retest performance (Schleicher et al., 2006) nor how test-taker attitudes directly interact with the retesting effect to

60

influence test score’ validity (Lievens et al., 2007). Thus, to explicate whether the retesting effect reflects either learning or memory requires controlling for the potential confounding influence of test attitudes (Reeve, Heggestad, & Lievens, 2008).

5.7 Summary

The prevalence and influence of the retesting effect on the scientific community and society at large underscore the importance of understanding whether these retest increases reflect either memory effects or learning. It is clear that the retesting effect consistently influences the tests that organizations, educators, and policy-makers use to make decisions across work, educational, and social settings. Despite the divergent implications that a learning versus memory explanation of the retesting effect has for retesting policies and practices in both academic and applied domains, a lack of integrated research between cognitive, education, and personnel researchers limits advancements in this domain and the subsequent inability to make clear evidence-based recommendations. Consequently, examining whether learning or memory best accounts for the retesting effect is the major contribution of the present study to literatures in psychology, education, and management and its findings potentially have implications for test developers, test users, test-takers, policy makers, and society at large. A

summary of the competing hypotheses for the posited memory and learning explanations are presented in Table 1.

61 Table 1

List of Competing Hypotheses

Hypotheses If memory best explains the retesting effect: If learning best explains the retesting effect: 1 (a) Retest scores will be higher than initial

scores; however, the magnitude of this effect will be larger on the same form retest than the alternate form retest.

(b) Retest scores will be higher than initial test scores and the magnitude of this effect will be similar for both the same and alternate form retests.

2 (a) Retest response times will be faster than initial test response times. However, the magnitude of the difference between the initial and retest response times will be larger on the same form retest than the alternate form retest.

(b) Retest response times will be faster than initial test response times. However, the magnitude of the difference between the initial and retest response time for the same and alternate form retest will be small (i.e., similar).

3 (a) Retest score increases from initial test scores will be approximately equal for knowledge and ability; however, the magnitude of this retest increase will be larger on the same form retest compared to the alternate form retest (across construct domains).

(b) Retest score increases from initial test scores will be higher for knowledge than ability, irrespective of whether the same or alternate form retest is administered.

4 (a) The relationship between working memory capacity and retest score increases will be stronger than the relationship between GMA and retest scores increases and be of greater magnitude for the same form retest than the alternate form retests.

(b) The relationship between GMA and retest score increases will be stronger than the relationship between working memory capacity and retest scores increases and be of a similar magnitude between the same form retest and the alternate form retest. 5 (a) Retest score increases will be higher for

the corrective feedback condition compared to the no corrective feedback condition; however, the magnitude of this effect will be larger on the same form retest than the alternate form retest.

(b) Retest score increases will be higher for the corrective feedback condition compared to the no corrective feedback condition and the magnitude of this effect will be similar for both the same form retest and alternate form retest.

62 Table 1

List of Competing Hypotheses (Continued)

Hypotheses If memory best explains the retesting effect: If learning best explains the retesting effect: 6 (a) Retest response times will be faster than

initial test response times for the corrective feedback condition compared to the no corrective feedback condition. However, the magnitude of the difference between the initial and retest response times will be larger on the same form retest than the alternate form retest.

(b) Retest response times will be faster than initial test response times for the corrective feedback condition compared to the no corrective feedback condition and the magnitude of this effect will be similar for both the same form retest and alternate form retest.

7 (a) Retest score increases will be approximately equal for ability and knowledge, but exhibit an interactive effect between receiving corrective feedback conditions and retest form, such that receiving corrective feedback will increase retest scores at a greater magnitude for the same form retest than the alternate form retest.

(b) Retest score increases will be higher for knowledge than ability and exhibit an interactive effect with corrective feedback condition, such that receiving corrective feedback will increase retest scores at a greater magnitude for knowledge than ability, but the magnitude of this effect will be similar for the same form retest and alternate form retest.

8 (a) A stronger relationship will exist between working memory capacity and retest score increases compared to GMA and retest score increases for the corrective feedback condition compared to the no corrective feedback condition, and the magnitude of this relationship will be larger on the same form retest than the alternate form retest.

(b) A stronger relationship will exist between GMA and retest score increases compared to working memory capacity and retest score increases for the corrective feedback condition compared to the no corrective feedback condition, and the magnitude of this relationship will be similar for both the same form retest and alternate form retest.

63

6. METHOD