Challenges in comparative designs

There are inherent risks and limitations in constructing comparative research methodologies, summed up by Hart’s observation that: “Not all things can be compared with all other things.” (Hart, 2009, p. 132). For example, considering national standards as a phenomenon in both jurisdictions, clearly these can be compared. However when seeking to compare the different elements within each phenomenon, it was not always possible, as certain elements were present in one but not present in the other and vice versa. This problem was compounded by how each set of national standards was conceived, organised and constructed, and this challenge applied to the other phenomena compared in the research.

A related challenge was that of selectivity, essential to any successful compar- ative analysis since, when selecting, choices have to be made regarding which elements to compare. Clearly the choices need to be clear, explicit and justi- fied. However to do this exhaustively would not be tenable or appropriate in the context of an EdD thesis. The comparative analysis was therefore tailored by taking account of the professional readership, and of how salient the con- trasts were between elements of different phenomena. An example of the former consideration being applied is the analysis of how the two different as- sessment systems compared with research findings on assessment system design, while an example of the latter is in the contrasting perceptions of graduates regarding the efficacy of each assessment system. In addition, communicating clearly how the comparative analysis was designed, and how it has been selectively applied in practice, was a critical underpinning to support the reader.

For Lumby, Crow and Pashiardis (2008), comparative analysis in educational research in the 19th and 20th centuries had been based on: “a firm sense of one's own location and traditions as the intellectual point of departure” (Lumby, Crow and Pashiardis, 2008, p. 4). They critique such research as being groun- ded in the notion that it will support the education of the superior by viewing what is worthy elsewhere. This stance has been characterised as “cultural bor- rowing” (Kay and Watson, 1982, p. 23) or “a process of bricolage” (Ball, 1998, p. 126). Despite the declared purpose of increasing global knowledge, under- standing and the cross-fertilisation of ideas, the actual result may have been to further perpetuate and reinforce ethnocentricity, disjuncture, and a hierarchy of value.

To avoid these still present dangers other comparative studies have used a variety of strategies. One approach is to use research to investigate education- al issues that are current globally, and research how practitioners have respon- ded in their own context. An example of this approach was a 2007 study (Bot- tery, Ngai, Wong, P. M. and Wong, P. H.) which considered how principals in England and Hong Kong experience increasing workload and inspection re- gimes. The similarities and differences found illustrate how principals in very different contexts are responding to common issues that affect their profes- sional lives. This approach is of relevance since the key issue being researched here (SLDP assessment systems) is a global phenomena. The research design, rather than simply allowing the reader to learn from what was happening else- where, focused on the effects of cultural and political contexts, and so enabled an examination of the ways in which local conditions interact with global is- sues.

The strategy behind the comparative and evaluative case study research was based on three elements. The first was participation, in that key actors from both jurisdictions would, through using a theory of action, be involved in evalu- ating the ‘site’ of SLDP assessment in their own context. The second was comparison, to reflect similarities and differences in documentary evidence and respondent perception; and the third was qualitative, to reflect the cultural and contextual realities in both jurisdictions. For this study context was treated as

integral to the holistic character of qualitative research. Therefore, as far as feasibly possible, the evaluation of the two assessments systems was groun- ded in the realities faced by providers, assessors and graduates. The interview protocols, research summary sheets and theory of action documentation were all constructed and deployed with this in mind so as to enhance and

strengthen the comparative design.

Arguably fieldwork issues common to all qualitative research are magnified in a cross-country comparative study. In the area of ideas for example an under- standing of such terms as leadership and assessment may well be contested. Another challenge of the research, since it involved school leadership from the Hong Kong research field, was that it accentuated issues of language. There were two specific language issues. The first concerned the varied relationships between the principal indigenous spoken language (Cantonese) together with its written adjunct standard written Chinese, and the second official language (English), plus the main language of the mainland (Mandarin). The second diffi- culty was the relationship between the research discourse conducted in Eng- lish, and what was in effect the day-to-day Cantonese language used by most Hong Kong respondents. The researcher found these issues were much ameli- orated by the strategy of raising the participation and engagement of respond- ents, coupled to researcher reflexivity, though a relative weakness of this study was its tendency, particularly in Hong Kong, to focus on those affected by de- cisions rather than those setting the agenda. (Stephens, 2012). Finally, given the political climate in Hong Kong sensitivity and diplomacy were required when discussing education with those in official positions.

Other issues were thrown up by the decision to seek and integrate knowledge gained form both documentary and real-world sources. These are discussed below, along with an explanation of how they were accommodated.

4.62 Combining an evidence-based best practice summary document with