There are a number of limitations of the methods that should be addressed. First of all, the indeterminacy of some modal items (e.g. can, may and could) due to their poly-semantic nature mean that the interpretations and classifications of potential meanings are highly subjective and not immediately replicable. Although threats to validity were addressed through the use of an independent inter-rater, it was perhaps too much to expect the inter-rater to identify all potential instances of modality, as well as classify them. The result of this exercise was the generation of differences in opinion about how modality is represented in the text at the expense of more detailed comparison of the classifications awarded to each of the instances. In hindsight, it would have been more effective to have randomly selected 100 instances of what I considered to be modality and then asked the inter-rater to only classify those examples. Despite this limitation, the exercise was useful as it provided for a useful exchange of opinions on what constitutes modality and how to classify the modal verbs can and could. It also led to the adoption of the modified system of modality framework based on Iedema, Feez, and White (1994) in order to separate examples of this type for future analysis.
A second limitation of the methods is with regards to the system of colour-coding used to classify over- and under-use of the combined value-orientation results as expressed in the modal matrices. The banding of the colours can be classed as arbitrary, as it does not coincide with any test of statistical significance; it is aimed at providing the reader with a simple yet useful overlay to visually compare the relative distances in frequency of use of the combined value-orientations in the examples of amateur writing, when compared with the expert writer texts. It does not alter or invalidate the results in any way, but it could raise some objections from those who may be more statistically inclined. Ultimately, the use of this method proved to be very effective at pinpointing where potential differences and similarities in use occurred within the types of modality, as well as enhancing the ability to quickly interpret how change in learner language choice occurred across the genres.
An additional caveat to consider when interpreting the results is the influence that task setting (untimed versus timed) has on the comparability of the expert and amateur texts within the genres of discursive essays and business case study reports. It is likely that the effects of
limitations on time, the lack of access to external information, and the limited ability to draft and edit their texts, would have some influence on the amateur writers’ language choices. The timed element of the written assessments would also impact the length of the texts, and this can be seen with the assessments for Modules One to Three, which generated shorter texts than those in the BAWE corpus of the same genres, all of which were untimed. It could be argued that this would invalidate the comparability of the texts; however, the principal aim of this study is not only to compare frequencies of use between types of writers, but more importantly, to consider how learner writers are socialised in the norms of academic writing in multiple genres. The texts that the amateur writers produce are the product of this process of socialisation, and, as such, are integral parts of this analysis irrespective of the differences in word count. It is only possible to be able to make inferences about the overall programme design and to make recommendations for change to the programme managers if evidence is presented from the actual language generated by the students, otherwise, there is the danger that any conclusions made in this research will lack relevance and rigour.
Although text length is an important consideration in comparing texts, so is the issue of the number of texts used in the study. The limitations in the availability of level 4 (Masters) texts in Business and Economics in the BAWE corpus together with the limited availability of amateur writers that could be tracked across all four modules of the pre-sessional programme, and the use of discursive essays in two of the four assessment points, means that there are differences in the number of texts in the genre groupings and corresponding word counts in the EWC and AWC. The small number of texts means that the threshold for any meaningful statistical testing could not be reached and would therefore not generate reliable results in this regard. However, it does provide an opportunity for a more detailed qualitative analysis than would otherwise be available from a larger style corpus study, as has been noted in the literature; for example, Liardét (2016), Takahashi (2009) and Yang, Zheng, and Ge (2015) all carried out effective in-depth analyses of a small number of texts (six, twenty-five and ten, respectively), and therefore, precedence exists.
The limited number of texts also means that there is an imbalance in the L1s of the AWC, creating a dominance of one particular group (that is, Thai, n=5). Much of the research into novice writing of academic discourse is predominantly based on Chinese learners of English given their dominance as the principal type of learner worldwide. However, unlike at many other pre-sessional programmes at other UK universities, students enrolled on this particular programme came from a more diverse L1 background (11 countries). Indeed, during 2011, Thai L1 speakers represented the largest block overall, and represented the largest group of students who agreed to participate in the research from across all the programmes (n=28). The number of Mandarin and Cantonese L1 speakers only represented a total of 18 students, occupying second place. Historical data of student enrolment show that the
dominance of Thai as the main L1 is consistent. It is therefore relevant as the main group for analysis despite the potential for the results to be skewed towards the Thai speakers’ representations of modality. Cross-linguistic analysis of modality (van de Auwera & Ammann, 2013) demonstrates that both Thai and Mandarin/Cantonese have complex systems of modal expression, particularly at the semantic level where modal items can have multiple senses which extend across modalization and modulation, in a similar way to English. Therefore, if L1 interference were going to be an issue, any divergence in language expression would likely occur with L1 specific syntax and sociopragmatic norms.
Pilot exercises were carried out for both the textual analysis and the interview protocol; however, this was not the case in the materials analysis. Instead, an established framework (Littlejohn, 2011) was adopted to help guide the analysis. It may be the case that utilising an independent coder could also have helped validate the framework for analysis in this phase. However, given the diverse range of materials used on the 20-week pre-sessional programme and the variety of methods used to teach modality, each set of texts was going to require in- depth qualitative analysis in order to identify how and, to what extent, modality is taught within them. Littlejohn’s framework provided the necessary flexibility to do that.
The final limitations of the methods are with regards to how the interviews were conducted. Firstly, it should be noted that this was the first time that I had carried out an interview analysis of this type and scale. In preparing for the role I paid special attention to advice from (Kvale, 1996, p.146) on helping interviewees feel at ease during the interviews,
(Cohen, Manion, & Morrison, 2018) in addressing the ‘interpersonal, interactional,
communicative and emotional aspects of the interview’ (p.421), Holstein and Gubrium (1995, p.59) in collaboratively constructing meaning, Whyte's (1984) six-point scale on the level of directiveness to employ, Field and Morse's (1989) methods for anticipating and mitigating potential problems, and Arskey and Knight's (1999, p.53) general advice on techniques for probing for more information. However, I found myself being less flexible than I could have been, sticking to the interview protocol more than was necessary and therefore being less probing than I could have been. As well as thinking about procedural elements of the interview, I was also very aware of time and was keen not to impose on the generosity of the participants too greatly who were already working to demanding work schedules. The result is that despite the collection of valuable insights into tutor cognition of the teaching of modality, I did consider carrying out a further round of interviews. However, given the difficulty in organising the first round, and the lack of time available to do so, which also included the fact that some of the tutors’ contracts were about to come to an end, this proved impractical.
3.9 Conclusion
This chapter aimed to describe the methods that will be used in the data collection phases and the subsequent analyses, and provided rationales for their use. It identified how a triangulation of methods would be used to provide insights into the teaching and learning of modality on a 20-week pre-sessional programme.
Chapter Four will provide a detailed account of the first phase of analysis in order to answer Research Question 1. It will use SFL to analyse modality in expert and amateur texts in three genres of academic writing in the combined disciplines of Business and Economics. It will use a modified version of the CIA to identify writer preferences and compare frequencies of use. This will allow features of over- and under-use in amateur writing to be identified. In addition, extracts from the texts will be used to highlight similarities and differences in frequently used lexicogrammatical features and semantic preferences. This will allow the amateur writers to be located on the interlanguage cline and make inferences about what they know about the use of modality within the three genres of writing.
Chapter Four: Results and Analysis (1) Contrastive Interlanguage Analysis (CIA) of the expression of modality in expert and amateur texts
4.1 Introduction
The research results presented herein are derived from the analysis of the data relevant to the research questions gathered throughout the period of the research. The chapter will be divided into 4 main sections. Sections one to three will focus on the expression of modality by the expert and amateur writers in essays, case studies and research reports, respectively. Each section will be subdivided according to type of modality and then subdivided further into type of writer to provide an on-going comparison. When analysing the expression of modality, the focus will be on the values (commitment) and the orientations (responsibility) of modality and consider how they impact on the intersubjective stance between writer and reader. The fourth section will provide a discussion and overview of the findings and will be used to answer the first main research question and three subsidiary research questions:
Research Question 1: How do expert writers (successful Masters students) and amateur
writers (pre-sessional students) express modality in their academic writing?
Subsidiary Questions:
1.1 How do the expert and amateur writers express modality in discursive essays? 1.2 How do the expert and amateur writers express modality in business case study reports?
1.3 How do the expert and amateur writers express modality in research reports?
Importantly, it should be noted that the example excerpts taken from the amateur and expert texts in the analysis, are extracted as is, without any grammatical or orthographical alterations applied. It should also be noted that despite using the label ‘expert’ writer, it does not automatically apply that this means a positively good writer.