Online Transcripts Analysis

Content analysis technique was used in analysing online discussion transcripts. It is a technique that enables the researcher to study human behaviour in an indirect way, through an analysis of their communications (Fraenkel & Wallen, 2006) that includes the process of coding, transcribing, analysing and verifying online transcripts before a holistic picture of the intervention can be reported. Anderson et al. (2001) argue that content analysis is a technique that “builds on procedures to make valid inferences from text” (p.10). While content analysis has been frequently distinguished as either qualitative or quantitative, this research used quantitative and qualitative measures of content analysis for analysing students’ online interactions in the intervention activities. Content analysis can be used to qualify and quantify the discourse of online applications especially with educational content (Hara, Bonk & Angeli, 2000; Schwandt, 2001; Neuendorf, 2002; Anderson & Kanuka, 2003; Gerbic & Stacey, 2005; Bélanger, 2006). Anderson and Kanuka (2003) argue that content analysis can be used with “any type of artefact of human discourse or activity” and is “often associated with the analysis of text documents, and in e-research investigations” (p. 174). The purpose of using both quantitative and qualitative content analysis in this research was to reveal “information that was not situated at the surface of the online transcripts”, to be able “to provide convincing evidence about the learning and knowledge construction” (Wever, et al., 2006, p.7) and to “capture the richness of student interaction” (Hara et al., 2000, p.119).

One of issues of content analysis in online learning research is the choice of the unit of analysis (Wever, et al. 2006). Basically, there are five types of units of analysis as distinguished by Rourke, et al. (2001), from large to small units such as message (e- mail or forum contribution), paragraph (section), ‘unit of meaning’ (or thematic unit), sentence (or syntactical unit) and illocution (Rourke, et al. 2001; Stribos, et al., 2006; Wever, et al. 2006). This research employed the thematic unit as the unit for content analysis representing a single idea, argument, topic or information, or event to which they referred regardless of its length in online discussion transcripts (Henri, 1992;

Lally, 2001; Rourke, et al. 2001; Stribos, et al., 2006). Wever et al. (2006) note that there is no real agreement on how a researcher comes to choose the unit of analysis. The choice for a unit of analysis is dependent on the context and on the research purpose and question (Wever, et al., 2006). Furthermore, content analysis is subjective and as a result some interpretations may not be easily justified or validated when challenged (Ho, 2002). Previous research found the thematic analysis unit to be useful in investigating collaborative learning through computer conferencing (Henri, 1992), social construction of knowledge (Gunawardena, et al., 1997, 2001), critical thinking (Newman, et al., 1995; Bullen, 1997), social presence (Stacey, 2005) and group dynamics (McDonald & Gibson, 1998).

Previous research suggests that instead of developing new coding schemes, researchers should use schemes that have been developed and used (Rourke & Anderson, 2004; Wever, et al., 2006). Stacey and Gerbic (2003) argue that applying an existing instrument fosters replicability and validity of the instrument. One of the advantages of applying well-developed coding schemes is that the researcher could support the accumulating validity of an existing procedure, and the possibility to use and contribute to a growing catalogue of normative data (Rourke & Anderson, 2003). According to Wever et al. (2006), many researchers do create new instruments, or modify existing instruments. This research adopted and modified Henri’s (1992) analytical instrument to analyse students’ interactions within online group discussions. Based on the literature, Henri’s (1992) analytical instrument is the most cited instrument in online learning research and is often used as a starting point in many Computer Supported Collaborative Learning (CSCL) studies (Wever, et al., 2006). It can be considered as pioneering work and has been the base for subsequent research (Wever, et al., 2006). The limitation of Henri’s model, as pointed out by McLaughlin and Luca (1999), is that it was designed for contexts where there was a strong teacher presence, and is not readily applicable to a learner-centred conferencing environment. However, McKenzie and Murphy (2000) argue that Henri’s model could be more easily applied to structured, problem-solving online tasks than to a less-structured online discussion. In accord with the McKenzie and

Murphy (2000) argument, this research used three structured online discussions which were based on the structured online intra and inter-group discussions on solving problems online via eLearning (Moodle).

The original analytical framework of Henri (1992) was based on five dimensions: participative, interactive, social, cognitive, and meta-cognitive. The participative dimension measures overall participation (which is the total number of messages and accesses to the discussion) and active participation (the number of statements directly related to learning made by learners and educators). The interactive dimension is divided into two parts, interactive versus non-interactive (independent) statements, and explicit versus implicit interactions. The social dimension measures all statements or parts of statements not related to the formal content of the subject matter. The cognitive dimension comprises fives categories, namely, (1) elementary clarification: observing or studying a problem, identifying its elements, and observing their linkages in order to come to a basic understanding, (2) in-depth clarification: analysing and understanding a problem which sheds light on the values, beliefs, and assumptions which underlie the statement of the problem, (3) inference: induction and deduction, admitting or proposing an idea on the basis of its link with propositions already admitted as true, (4) judgment: making decisions, statements, appreciations, and criticisms, and (5) strategies: proposing coordinated actions for the application of a solution, or following through on a choice or a decision. Furthermore, surface processing is distinguished from in-depth processing, in order to evaluate the skills identified. The meta-cognitive dimension measured meta-cognitive knowledge and meta-cognitive skills. Meta-cognitive knowledge is declarative knowledge concerning the person, the task, and the strategies, while meta-cognitive skills refer to ‘procedural knowledge relating to evaluation, planning, regulation and self-awareness (Henri, 1992).

Pozzi et al. (2007) argue that the five dimensions of Henri’s (1992) original model do not necessarily imply the use of all five dimensions. Instead, the researcher is free to decide which dimensions are relevant depending on the specific aims of the research

and the context of the learning experience. Of the five original analytical dimensions, only four were used and considered to accommodate the data collected in this research; they were participative, interactive, social and cognitive dimensions. The researcher added several categories and examples from the literature to the framework, as previous research (Hara et al., 2000) found that adding several categories to the existing framework would be useful in overcoming the lack of precise evaluation criteria to judge each of the categories. The researcher employed several categories from Pozzi et al. (2007) for analysing participative (level of participation and viewing), interactive (types of interaction), social (types of social presence), and cognitive dimensions (types of cognitive presence); and an analytical framework for deep and surface learning from Gerbic and Stacey (2005) in order to elicit more information about students’ participation and interaction during the intervention. The four modified analytical dimensions with added categories are elaborated upon as follows:

 The participative dimension categories were modified to include categories based

on the level of participation determined through students’ number of postings and viewings (Pozzi et al., 2007). These categories were based on two types of indicator of students’ active and passive participation. Active participation was measured through the number of postings students made in the online discussion while passive participation measured the frequency of students viewing particular posts in the online discussion.

 The interactive dimension categories were modified to include categories based on thematic units referring to physical aspects of the online communication such as the frequency of explicit and implicit (or collaborative) interactions, and independent (or cooperative) statements (Ingram & Hathorn, 2004). The research also considered the qualitative aspects of students’ interactions by identifying students’ ways of interacting online (such as used in this research: providing information, sharing views, sharing experiences, agreeing and disagreeing, posing

questions, suggesting new ideas, giving feedback, and clarifying ideas) during the intervention activities (Pozzi et al., 2007).

 The social dimension categories were modified to include categories based on thematic units characterised by affection and cohesiveness exhibited during communication in online discussions (Pozzi et al., 2007). Thematic units characterised by affection include the use of emotional expressions (such as used in this research: emotion icons or emoticons) and thematic units characterised by cohesiveness including the use of social cues (such as used in this research: greetings, salutations, concern, encouragement, apology, jokes and humour, and thanking).

 The cognitive dimension categories were modified to include categories based on

cognitive presence revealed by thematic units referring to (1) revelation (renamed as clarification) that is, recognizing a problem, explaining or presenting a point of view; (2) exploration (renamed as judgment) that is, expressing agreement or disagreement, argumentation, exploring or negotiating; (3) integration (renamed as inference) that is connecting ideas, making syntheses and creating solutions; (4) resolution (renamed as strategies) that is, reflecting on real-life application suggestions or references to real-life solutions (Pozzi et al., 2007).

 The information processing (e.g. surface and deep) categories were modified to include categories based on thematic units referring to (1) surface learning that includes reproducing an approach (not wanting to understand the issue or finish with minimum of effort); or staying inside course boundaries (repetition of what is being discussed or required); or an unthinking approach (jumps to a conclusion with an uncritical acceptance of ideas); or fear of failure (focus on negative aspects of the coursework); or extrinsic motivation (more concerned about passing the assessment than learning); and (2) deep learning includes looking for meaning (focus on what is signified, asking questions to understand new information); or relating ideas (relating ideas to previous information or

knowledge to generate new ideas); or using evidence (finding alternative ways of interpreting information or justifying with an example); or intrinsic motivation (desiring to learn more about the topics) (Gerbic & Stacey, 2005)

The overall steps of conducting the content analysis in this research began with the postings of the students in online discussions within each group and close reading of each posting was established. Next, the researcher coded each unit of analysis starting with participative followed by interactive, social, cognitive and information processing (surface and deep). The researcher established the counting of the number of postings for each category in each dimension. In order to safeguard credibility and to validate the coding procedures of the modified categories from Henri’s (1992) model, intra-rater and inter-rater coding was employed. Intra-rater was conducted by the researcher as ‘coder agreeing with his self (coding) over time’ (Wever, et al., 2006). This was done by running the coding multiple times before reaching coding stability. In this research the coding was reviewed more than three times by the researcher to compare and contrast in order to achieve coding consistency. The overall coding was also reported for reviewing by other experienced researchers (in this case the researcher’s supervisors).

The inter-rater reliability (the ability of multiple and distinct groups of researchers to apply the coding scheme reliably) was also conducted between two independent coders agreeing with each other (Wever, et al., 2006). Guidelines for coding were formulated stating clearly what comprises a unit, and descriptions of all categories. Two Malaysian PhD researchers from Massey University were asked to help with the coding. Before they conducted the coding process, the guidelines and instructions were introduced to them. A one-hour training session was held during which these guidelines were explained. After that, one transcript from each mode of discussion was randomly selected (altogether totalling approximately 10% of online group discussions) and coded separately by the two coders and they then compared their results. The result across all categories reached a Cohen’s Kappa value of 0.81 compared with individual categories such as interactive with 0.84, social with 0.74,

cognitive with 0.71 and information processing (surface and deep) with 0.72. According to previous researchers (Rourke et al., 2001; Neuendorf, 2002; Wever, et al., 2006) a value above 0.75 (sometimes 0.80) is considered to be excellent agreement beyond chance; a value below 0.40 indicates poor agreement beyond chance; and values from 0.75 to 0.40, represent good to fair agreement beyond chance. This study’s 0.81 Cohen’s Kappa value for the consistency of inter-raters’ agreement can be considered highly reliable (Wever, et al., 2006).

Finally, the analysis of types of engagement within each online group was conducted. Four types of engagement were pre-identified from the literature instead of emerging from the analysis of the students’ interactions. However, there was considerable consistency and relationship between the categories of analysis of students’ participation level and their ways of interacting online in the online discussion during the intervention based on the overall triangulation of data (interviews, pre-post questionnaire and final grades). An example of the overall analytical process is depicted in Figure 4.2.