This qualitative phase is designed to address RQs 1-3: “From the students’ roleplay performance data, what are the interactional phenomena in the form of actions or courses of action which can be established as the targets for comparing IC across the dataset?”, “What are students’ methods, in varying degrees of success, in accomplishing the actions or courses of
action identified as the targeted interactional phenomena?”, and “How can the rich description of students’ task performance inform the data-driven construction of the IC assessment rubric?”
Students' performance on the Socializing task was video recorded for the current research and analyzed using qualitative methods of conversation analysis (CA). The data collection in this phase of this study sampled student test taker performance in February 2016 during the first semester of the same academic year.
Student participants. One hundred and eighty students participated in this study and 34
group oral roleplay performances were collected through the help of the course coordinator of the course who has also been part of the material and curriculum development from the start. These students were second-year and third-year undergraduate students from different
engineering majors offered at this university: Computer Engineering, Mechanical Engineering; Industrial Engineering, Civil Engineering, Chemical Engineering, and Nuclear Engineering. One hundred and sixty student participants were male (approximately 89%), and 20 were female. This ratio represents the current demographic of the engineering job market in the country. Descriptive statistics of their scores based on the original rubric are shown in Table 4.1 below. Across all four criteria, students were rated quite high. For three of the criteria (group
collaboration, content, and delivery), means were more than four out of five, with a small spread around 0.5 across all four criteria. The most difficult criterion from the original rubric was the language and pronunciation criterion, followed by delivery, content, and group collaboration. In terms of distribution of scores, group collaboration and content scores were negatively skewed, meaning that more students scored above the means on these categories. The language and pronunciation category appeared to be the most normal. The delivery scores, on the other hand, appeared slightly leptokurtic, which means that many students scored at the mean, resulting in a distribution that is taller than normal.
Table 4.1
Descriptive Statistics of Participants’ Scores on the Original Rubric
N Min Max Mean SD Skewness S.E.S Kurtosis S.E.K
Group collaboration 180 3 5 4.20 0.53 -0.493 0.18 0.223 0.36
Content 180 3 5 4.17 0.55 -0.434 0.18 -0.043 0.36
Language and pronunciation 180 2.5 5 3.88 0.56 -0.169 0.18 -0.245 0.36
Delivery 180 2.5 5 4.02 0.60 -0.199 0.18 -0.557 0.36
The performances of 34 group interactions of the students who gave their consent to participate in the study were video recorded. The group sizes ranged from four to six students per group. Video data were deemed essential in this study since embodied actions are an integrated part of IC in co-present interaction that would not be available in audio recorded data.
Data analysis. The video data were reviewed and transcribed following conversation
analysis (CA) conventions (Atkinson & Heritage, 1984) to reveal the detail how students organize their turns and the sequence through which they organized their actions during their roleplay performance.
The reasons why CA was chosen to provide an analytical description of how students perform the roleplay are as follow. First, CA is uniquely conceptualized to capture the
organizations and structures of social interaction in both of its procedures and outcomes. Second, it takes into account the co-constructed nature of talk-in-interaction, the manner of which can be used to provide a specific description of how co-construction can be operationalized for language testing and assessment of oral communication skills. With CA, the questions of what has been individually achieved or collectively achieved, by what resources, and in what manners can be explicitly addressed. Thirdly, because CA takes an emic perspective in analyzing what and how actions and accomplishments have been made in interaction, it can be illuminating to ascertain how student accomplishments match with those expected by teachers and users of test outcome. Finally, adhering to the principles of using CA as a lens for qualitative observation will help us steer away from treating IC and co-construction as fixed patterns of interaction which reduced IC
into a certain kind of products which rid IC of its rich procedural component that we also need to measure.
The goals of the qualitative analysis are to identify recurring actions or courses of action throughout the roleplay data, which can provide a consistent observational basis for the
assessment of IC. Given the fine granularity which actions and courses of action can differ, to establish a common target for comparison of student IC, this study is attempting to adhere to the principles for quantifying talk-in-interaction from Schegloff (1993), which have been adapted into practical recommendations for longitudinal developmental studies summarized by Pekarek Doehler et al. (2018) and Pekarek Doehler and Berger (2018).
This fine-grained qualitative observation of interactional accomplishments and interactional resources making such accomplishment possible will lay a foundation for constructing a measurement scale for assessing IC in the subsequent stage. As Bond and Fox (2015) note on the nexus between qualitative observation and quantitative observation, all observation starts off qualitatively, and counting starts off with repeated observations before those observations can be quantified. Actions or courses of action identified as the target interactional phenomena for the assessment task will be treated as items against which each person’s performance will be coded for success or failure for each item under the rubric guidelines.