The next step in the coding procedure involved determining SA and TMS cognitive behaviors, task or non-task related communications and speech acts in team members’ communication activity. The communication data were coded applying the data coding method employed by two coders. This approach classified communication activity of team members according to pre-established categories. Communications were segmented into
99
utterances. Here, the sentential construction that referred to a distinct cognitive process. Each utterance was given a separate code. The central purpose via this method was to annotate the communication accordingly, mapping the utterances according to categories and examine the frequency of each requisite indicator.
4.4.1 Coder Training
The coders trained as a group on understanding the coding scheme, their definitions, and application on how to code communication using the data coding method. The coders practiced using the coding system during the pilot testing phase of data coding. During training, the two coders coded team communication data from two-three member teams not included in the present data set. Also, coders practiced individually by coding communication of an excluded four-member team. Coders discussed their respective coding with one another to calibrate frequency annotated for all categories. The team assigned two coders to do all team coding of communication utterances. Coders defined each utterance as the message sent by a particular team member. Following the training period, coders independently coded the communication data, subsequently reviewed their coding, calculated the percent agreement, and resolved any differences in data coding.
4.4.2 Coding Procedure
The communication data between team members were time-stamped chat logs. The resulting transcripts provide a realistic example of an inter-collaborative team response to a simulated emergency. Two coders then coded all messages that the system automatically logged to establish the reliability of the coding method. Coders also included chat message utterances in the coding scheme. Coders first categorized task communication for task-related and not-task related communication. If utterances were identified as not-task related,
utterances were not coded further. If utterances were identified as task-related, the coding included identification and classification of speech act and cognitive behavior indicator in utterances. Coders had first to classify utterances according to the speech act scheme. Next, coders distinguished between two main cognitive behavior indicators SA and TMS with their cognitive behavior sub-codes. A single utterance could involve multiple, code-time
classifications if the message utterance content contained these speech act or cognitive behavior aspects. Where multiple codes applied, a code identified each part of the utterance (Kennedy and McComb, 2014). One sub-code of cognitive or speech act behaviors was applied per utterance to have independence of codes between sub-codes. In cases in which two utterances referred to the same cognitive process (i.e., sub- codes), the coder had to
100
weight the content and assign the most applicable in that certain context to retain independence of observations. Figure 8 presents a flowchart of the process applied.
Figure 8. Flowchart of the Coding Process.
YES NO
Continue with next category Has Information regarding task
been communicated?
Declare Code Declare if utterance
contains verb
Continue with next category Declare Speech Act
Behaviors
Decide on Task Communication If YES
Declare Cognitive Behaviors Declare Reflexivity
Declare tense (present, past, future)
101
4.4.3 Interrater Reliability
After coder assigned codes, their agreement and overall reliability was calculated. To determine the overall agreement between the two coders, the qualitative categorical statistic Cohen’s Kappa was used. Cohen’s Kappa accounts for the fact that each coder may agree by chance and not strictly, because coders chose the same selection of codes. Therefore, Cohen’s Kappa was chosen as the preferred statistic over Chi-square as kappa tests for agreement whereas Chi-square tests for association (Thomas & Hersen, 2003). Because Kappa has its limitation, Krippendorff’s alpha (α; 2004) was computed as it is referred to as standard reliability statistic for content analysis and similar data making efforts (Hayes & Krippendorf, 2007). Krippendorff’s alpha counts category pairs coders assigned to utterances and lets coders be unaffected by their numbers. It bootstraps the distribution of a sample from the reliability data to avoid assuming approximations. The bootstrap sample of 10,000 was chosen to gain accurate inferential statistics. A bootstrap sample larger than 10,000 was found to add little additional precision to the data (Hayes & Krippendorff, 2007). After inter- rater reliability was computed and variations identified, both coders discussed differences and came to a complete agreement. This step was necessary for further analysis.
4.4.4 Pilot Testing
Before launching the full-scale content analysis with the newly developed coding scheme, test coding of a sample of the communication to be analyzed was conducted to reveal inconsistencies and inadequacies in the category construction. It also helped in establishing the reliability of the coding scheme. For the pilot testing percent agreement of the ratio between the number which was agreed upon (agree + disagree) of codes were computed (De Wevers, Schellens, Valcke, & Van Keer, 2005). Coders had 27.76% variation in their agreement for coding 2809 chat messages (i.e., training, scenario 1, discussion phase, and scenario 2). Coder 1 had 27.94% variation in agreement from coder 2 for 1121 chat messages of 10 teams; coder 3 had 32.95% variation in agreement from coder 1 for 926 chat messages of five teams; and coder 4 had 22.40% variation in agreement from coder 1 for 762 chat messages of five teams. After test coding was conducted, category construction of the coding scheme was refined to improve reliability.
102