Language programme evaluation: the context adaptive model

As one purpose of this study was to evaluate the effectiveness of in-sessional pro- grammes available at universities across the UK, the first step was to choose a programme evaluation framework. The evaluation model used in this study was informed by the context-adaptive model (CAM) formulated by Lynch (1996). As briefly mentioned in the previous chapter, the adaptive and flexible nature of the context-adaptive model for language programme evaluation allows for the combina- tion of methods from both positivist and naturalistic paradigms. Since one aim of this study was to investigate the effectiveness of current EAP in-sessional provision across British universities through stakeholders’ perceptions of the provision, and by collecting data from several sources, CAM model which allows for such trian- gulation of data was employed. Unlike purely positivistic or naturalistic models to programme evaluation (e.g. quasi experimental pre-test post-test and responsive model) CAM allows for collection of both quantitative and qualitative data. In addition, given that this study aimed at investigating effectiveness of in-sessional

programmes from a macro perspective, focusing on general features of in-sessional provision across the sector, the adaptable nature of CAM made this evaluation model suitable for this study. This framework was therefore used as a guide to shape all phases of the mixed method research here, and the different stages included in each phase (Mertens, 2003).

According to CAM, seven steps were defined and used in conducting the in- sessional programme evaluation. Figure 3.1 provides an overall picture of the steps taken in this research study by embedding them into CAM.

As is shown in Figure 3.1, the first step in CAM concerns identifying the audience and goals for the evaluation. This step entails identifying the stakeholders who have a central interest in the findings of the evaluation because they are somehow affected by the in-sessional programme. In the case of this study, three primary stakeholders were identified; namely, university students, subject-specialists, and in-sessional staff (further information about each of these stakeholders is provided in the following sections).

In addition to identifying the stakeholders, determining the evaluation goals and purpose is another component of step one of CAM. This includes specifying reasons for conducting the evaluation, and the information it will provide. In the case of this study, this included exploring the stakeholders’ perceptions of the effectiveness of the in-sessional programmes and the criteria they invoked in their evaluation.

Step two of CAM involves acontext inventory which includes characterizing fea- tures of the programme being evaluated. This evaluation model suggests a checklist of dimensions of the programme to be defined in the context inventory. The adap- tive nature of the checklist allows for adaptation and tailoring needed for different programme settings. Given that this research study was an endeavour to provide a holistic impression of in-sessional provision currently offered across British universi- ties, a checklist of general features of the in-sessional programmes was developed in order to investigate features of this provision. This information was collected from the in-sessional programmes’ directors. (Details of the checklist and the features in- vestigated regarding the characteristics of the in-sessional programmes are provided in section 3.5.3).

Step three consists of a preliminary thematic framework which determines the focus of the evaluation. This includes conceptualising the programme at hand in terms of ‘salient issues and themes’ emerging from identifying the audience and the context (ibid; 6). This framework is then used as a guide in determining the focus of ‘the collection and analysis of evaluation data’ (ibid; 6). In the case of the current study, a preliminary thematic framework was developed consisting of five evaluation

Figure 3.1: Language programme evaluation: The Context Adaptive Model (CAM) Taken from Lynch (1996)

criteria to investigate the effectiveness of the academic English in-sessional provision. Section 3.5.3 provides detailed information regarding these scales and how they were developed.

Once the stakeholders, goals, context and evaluation criteria are defined, the next step in CAM is to determinedata collection design/system. This is done in step four where questions concerning the type of data to collect and methods for gathering such data are addressed. As mentioned earlier, in order to benefit from the merits of both qualitative and quantitative data, this study was conducted within the pragmatic paradigm using a mixed-method approach. The type of data and methods of data collection used in this research study are detailed in the following sections.

Data collection and data analysis form steps 5 and 6 of the Context-Adaptive

Model and include determining ‘appropriate conduct of the data-gathering proce- dures and the interpretation of the results’ (ibid.;7). Details of these steps in the present study are provided in the remainder of this chapter.

Finally, step seven entails creating anevaluation report. This report should pro- vide for effective communication of the results with the intended audience. Regard- ing this PhD research, such a report is provided in the Discussion and Conclusion chapters. In what follows, detailed information is provided regarding how each of the steps explained above was used to guide the current research.