The mixed methods research (MMR) approach (Brown, J. D., 2014; Dörnyei, 2007) is adopted as a macro framework for the data collection and analysis of this study. A combination of qualitative and quantitative methods will be used at different stages for this investigation corresponding to the research questions at different stages of the study. Justifications for why methods are used at different stages are discussed along with the design to argue for the validity or trustworthiness of using mixed methods research. The following sections describe the
assessment context, the research questions, the procedures of data collection, and analysis of the data from the qualitative and quantitative phases of the study.
Assessment Contexts and Assessment Task
This section describes the contexts of the chosen performance data from a peer group roleplay assessment task. Set in the context of English as a foreign language curriculum for undergraduate level at a university in central Thailand, this study is aimed at exploring individual and co-contributions within EFL IC in group oral tasks using role-play simulation in order to inspect, expose, improve, and validate the standards of IC that are imposed on and oriented to the students in response to a socializing in semiprofessional environment. The course in which this
socializing task is situated is a 16-week-long English for engineering course titled
Communication and Presentation Skills. This course is a required undergraduate course offers
twice a year to students in the faculty of engineering from all of its majors. Most students take this course during their second year, but some may take the course in their third year depending on their majors. The course objectives are targeted at helping improve students’ oral
problem-solving discussion during meetings, job interviews, and presentations on engineering related topics. The class meets three hours per week, during which class of instruction and assessments take place. The course content is divided into four units corresponding to the curriculum goals: socializing, meeting discussion, job interviews, and presentations.
The primary data for this study come from the first unit on socializing talk. The
summative test at the end of the unit is designed to elicit group interaction, specifically everyday small talk for working and networking purposes in an engineering context. The activity lends itself well to an examination of interactional competence for a specific institutional purpose.
The assessment task. As specified in the syllabus, the students were to create a persona
and play the role of a representative of a chosen company as they attended an international trade show in Sydney, Australia. The scenario required that they were meeting at a pre-conference reception and that none of them was from Thailand. In preparing their own roles for the role- play, each student was required to choose a name, the company they were going to represent, their job position, and their responsibilities in their respective companies. They also had to research other relevant information such as their hometown, education, etc., as part of their preparation for the roleplay assessment task (see Figure 4.1 for the set-up sheet.)
On the day of the test, five to six examinees were randomly assigned to form a group that took the test together. Each group had 15 minutes for their preparation before performing the roleplay in front of an examiner for 12 – 15 minutes. During that time, the students engaged in and elaborated on the activities specified on the set-up instructions: they conducted small talk, introduced themselves, talked about their work and the company they represented, and
exchanged their name cards as part of making new contacts. The original assessment rubric was available to the students and was used by their teachers. It divided the scores into group scores and individual scores. The group score was graded on a 1-5 scale for "group collaboration." The
individual score was composed of three criteria, content, language and pronunciation, and delivery, each graded on a 1-5 scale, totaling 15 points. (See Figure 4.2 for the original rubric.)
Research Questions
Through reviewing the literature on interactional competence (IC), interactional competence development, and the validation framework for performance assessment, it has become quite clear that there is a ripen possibility to construct and validate an assessment instrument that targets IC through merging applied conversation analysis and the data-driven approach to rubric construction for performance assessment. To this end, the current research proposes an activity-based framework in operationalizing IC as the assessment construct in a peer group roleplay performance data. Through Kane's (2006) argument-based approach to assessment validation, this study is aiming to explore empirical evidence in supporting the claims that the proposed rubric and rating scale are defendable in providing a valid evaluation of IC. The current study's research questions are listed below.
1. From the students’ roleplay performance data, what are the constitutive interactional phenomena in the form of actions or courses of action which can be established as the targets for comparing IC across the dataset?
2. What are students’ methods, in varying degrees of success, in accomplishing the actions or courses of action identified as the targeted interactional phenomena? 3. How can the rich description of students’ task performance inform the data-driven
construction of an IC assessment rubric?
4. Given the proposed rubric for assessing IC in this roleplay task, how reliable is the rating process in applying the scale to rate the students’ performances?
5. Given the proposed rubric for assessing IC in this roleplay task, are there any
6. Through mixed methods research, to what extent can the current study argue for the validity of the proposed rubric and rating scale for assessing IC in this context? How do the findings from mixed methods help to strengthen the validity argument?
MMR Study Design
For our current study, we adopted the developmental sequential mixed methods design. Developmental design always implements sequential timing in collecting multiple data for analysis in which the result of one type of analysis is then used to inform the development of the next method (Greene et al., 1989). For rating scales used in domain-specific tests, the use of performance data-driven approach to scale construction is frequently encouraged. Students’ performance data are treated as the primary resource for rubric construction. During the first phase of this study, the goal was to obtain student performance on the task in which students were to display their IC. The performance was analyzed through the lens of conversation analysis (CA) and applied conversation analysis to provide a structural explanation on how student participants organized their interactions in order to complete the role-play, and how higher competent students handle this social activity compared to lower competent students in this dataset. The summary of the qualitative findings in Phase I was organized into a rubric, which was implemented and analyzed in the quantitative phase of the study.