Washback effect

The concept of washback has been considerably discussed and its

existence is asserted most widely in the context of language testing or high- stakes educational assessments (Alderson and Walker, 1993; Hughes, 2003; Cheng et al., 2004). For example, the washback effect of Zhongkao and Gaokao has been discussed extensively (e.g. Gao, 2010; Cong, 2014; Zhang, 2016). Although PISA is not explicitly high-stakes for schools and students, since it does not report individual students’ scores, it is considered high-stakes for policymakers in PISA participanting education systems (Sellar et al. 2017). The concept of “washback ” has also been introduced to discuss PISA effects as well (e.g. Baird et al., 2016; Maddox and Addey, 2016).

Regarding washback, at the time when it emerged to be a research interest, it specifically refers to test impact on teaching and learning (Buck, 1988; Hughes, 1989; Shohamy, 1992; Hamp-Lyons, 1997). Since discussions on

washback most focus on the processes of teaching and learning in which not only individuals, but also educational and societal systems, are involved, some researchers consider washback as “effects on individuals, educational systems, and society at large” (Bachman and Palmer, 1996, p.30). Along with the development of studies on washback, now the latter’s point of view has been widely recognised (Hughes, 2003). Sometimes, washback is used as a synonym of impact as well (Wall, 1997). For instance, Scott (2014) classifies washback into two types: “micro washback” which directly affects individuals; “macro washback” which directly affects institutions and systems in which individuals are influenced subsequently. This definition is similar to that stated by Bachman and Palmer (1996) for “impact”. It is generally

argued that washback could be either positive or negative, depending on the

administration and use of the test within a particular educational context (Cheng et al., 2004).

To shape the structure of this empirical investigation on the impact of PISA on student learning, I employ the basic washback model suggested by Hughes (1993, cited in Bailey, 1996, p.262) as the basic frame. In his unpublished manuscript4_{, Hughes (1993, cited in Bailey, 1996, p.262)}

distinguishes the washback on participants, process and product in teaching and learning, believing that all three could be influenced by the nature of a test. His trichotomous washback model (Hughes 1993, cited in Bailey, 1996, p.262) is as Table 3.1 shows.

Table 3.1 Hughes’ washback model

Trichotomy Signified

participants students, classroom teachers, administrators, materials developers and publishers, whose perceptions and attitudes towards their work may be affected by a test

process any actions taken by the participants which may contribute to the process of learning, including materials development, syllabus design, changes in teaching methodology, the use of learning and/or test-taking strategies, etc.

product what is learned (facts, skills, etc.) and quality of the learning (fluency, etc.) Hughes (1993, cited in Bailey, 1996, p.262) notes that participants’

perceptions and attitudes towards teaching and learning would be firstly

4 _{I contacted Bailey by email, enquiring whether the copy of Hughes’ (1993)} manuscript is still available. However, she said “I don’t have this paper any longer”.

affected by test nature, their changed perceptions and attitudes may then affect what participants do in their work so as to affect the process of students’ learning, and subsequently influence the learning outcomes. He considers that improved learning is the ultimate product of positive

washback. This model suggests that to investigate PISA’s impact on students’ learning: (1) participants who are involved in the process of students’ learning through taking actions in response to PISA shall be

recruited; (2) their perceptions of PISA, the actions or initiatives motivated by PISA should be identified; (3) the changes in the process of students’

learning and subsequent changes of students’ learning outcomes could be seen as the evidence of the existence of PISA’s impact on students’ learning. As this basic model suggests, the process of students’ learning is complex since various participants are involved, albeit at different magnitudes and in different ways. Hence, employing qualitative data and quantitative data is suggested to be used to triangulate and support a balance of evidence for investigating washback (Bailey, 1999). This underpins the significance of employing mixed methods in my research. For example, besides qualitative data on the perceptions of participants about PISA’s impact on the process of student learning and learning outcomes, evidence based on quantitative data are also needed for triangulation. From policy initiatives motivated by PISA to school-implementation of these initiatives, different participants involved in the process of students’ learning may have different perceptions and interpretations of PISA and its related initiatives. Therefore, it also suggests the need of accounting for different stakeholders’ perspectives (Cheng et al., 2015).

To examine PISA’s impact on the processes and outcomes of students’ learning, it is necessary to further clarify the possible aspects of the changes brought by PISA and PISA-motivated initiatives. From the perspective of empirically examining how washback occurs in the process of teaching and learning, Alderson and Wall (1993, p.120-121) propose a list of possible washback hypotheses, as shown below.

(1) A test will influence teaching. (2) A test will influence learning.

(3) A test will influence what teachers teach; and

(4) A test will influence how teachers teach; and therefore by extension from (2) above:

(6) A test will influence how learners learn.

(7) A test will influence the rate and sequence of teaching; and (8) A test will influence the rate and sequence of learning. (9) A test will influence the degree and depth of teaching; and (10) A test will influence the degree and depth of learning.

(11) A test will influence attitudes to the content, method, etc., of teaching and learning.

(12) Tests that have important consequences will have washback; and conversely

(13) Tests that do not have important consequences will have no washback. (14) Tests will have washback on all learners and teachers.

(15) Tests will have washback effects for some learners and some teachers, but not for others.

The above hypotheses provide insights into my research in terms of identifying possible categories and themes evidencing PISA’s impact, for example, learning content, teaching practices, teaching quality, attitudes towards learning, and cognitive performance, which can also be classified as processes and outcomes as defined in the education effectiveness model employed in PISA assessment framework (OECD, 2013a). They also suggest that the impact may be different across teachers and students due to individual differences between them (Alderson and Wall, 1993).

As discussed previously (see Section 3.2.4), the interpretation and

translation of PISA data in domestic education systems involves a process of de-contextualisation and re-contextualisation, with the agencies in different contextual levels (e.g. national, local, regional and international) (Carvalho and Costa, 2015). It can be further contextualised during school enactment (Braun et al., 2010; Braun et al., 2011). Similarly, washback effect also suggests that, interpretations and uses of a test “are value laden and situated in political, social, cultural, and educational contexts” (Cheng et al, 2015, p.464). These various contextual factors may mediate the

washback effect and should be evaluated when investigating the impact of a test (Bachman and Palmer, 1996; Cheng et al., 2004; Shih, 2009; Cheng et al., 2015). Hence, it is reasonable to conceive that contextual considerations are also essential for investigating and understanding PISA’s impact on students’ learning brought by the interpretation and use of PISA in a specific context. Although the above listed hypotheses about washback particularly focus on the impact of test at the micro level, that is, effects on teaching and

learning, washback effect can be more complex than those listed, with test nature, educational context, and decisions made according to the test results involved (Alderson and Wall, 1993).

Regarding the contextual factors which may mediate washback, Bachman and Palmer (1996) suggest the goals and values of the society and the educational system wherein the test is used. Cheng et al. (2015) classify them as two types: micro-context factors (e.g. school setting) and macro context factors (e.g. society and educational system). Despite the fact that the studies on washback effect have noted the importance of contextual considerations for investigating the impact of a test, neither the basic model of washback mechanism itself nor the washback hypotheses explicitly include and specify contextual factors in detail to allow for empirically investigating how they link with the trichotomy (i.e. participants, process, products) and how they work in mediating washback. Therefore, to

investigate PISA’s impact on students’ learning, the mechanism of washback which is employed as the framework needs to be expanded to build into contextual considerations.