Reliability of the coding schemes used

The rationale for seeking high categorising reliability is similar to the rationale explained in the previous paragraphs about unitising reliability. To swiftly iterate, reliability of coding schemes is necessary, if the research method of coding is to be consistently applied throughout the data, taking care on eliminating any serious sources of bias and eventually allowing for the replication and verification of the results obtained (Folger, et al., 1984).

The most widely used reliability metric for coding schemes is Cohen’s Kappa (Cohen, 1960). Cohen’s Kappa has been used for obtaining reliability measures for codes that were later used in both sequential and static analyses (Poole & Dobosh, 2010; Poole & Roth, 1989a & 1989b). It is obtained by the following equation:

Kappa = (P’ - Pc ) / (1 - Pc )

96

where Pc is the proportion of chance agreement.

In the same spirit as when calculating the unitising reliability, three samples of 500 units each, were drawn from the start, middle and end of workshop across cases.. For the coding schemes employed in this research the following interrater reliabilities for categorising were achieved. For this part of calculating interrater reliabilities two independent coders were used and their coding was compared with the researchers’ coding. One needs to remember that the reliability scores were achieved after both coders were given the coding manuals presented in the appendices, and after they received a training tutorial of approximately 1-hour.

For the GWRCS, the Cohen’s Kappa calculated averaged .85% for the first coder and 80% for the second.

For the MACS, the Cohen’s Kappa calculated averaged 91% for the first and .82% for the second.

For both GWRCS and MACS the differences were discussed and notes were taken so as to allow the coding of the rest of the transcripts.

All the reliability results for all the coding schemes fall well within the region of acceptable degree of reliability (Fleiss, 1981).

Once sufficient levels of interrater reliability have been achieved, the rest of the coding was performed by me. This is an acceptable way for coding data in exploratory research. The justification offered from a related research conducted by Sambamurthy & Poole (1992) is that “The coding was done by the first author after an adequate level of interrater reliability was attained with an independent coder.[...] In view of the fact that this was an exploratory study with no strong a

97

priory expectations about the results, the first author’s service as a coder did not seem to present serious threat to the study’s validity” (p. 239).

In the next section the issue about whether the coding schemes were of a univocal versus a multifunctional nature is explored.

3.4.5.1 Univocal versus multifunctional coding.

Univocal coding implies that the coding schemes applied contain coding categories that are mutually exclusive. Still, social life and particularly group interaction is much richer than what a meaningful single coding scheme could ever hope to capture. What I observed is that while the freedom to code in a multifunctional manner was given to the coders, the unitisation according to thought units and the strict coding manuals allowed for little differences in interpretation. Therefore the coding scheme that used the thought unit displayed a univocal type of coding. The exception to that was the GWRCS in which multifunctional coding was observed.

The application of the coding schemes used in this research followed the rationale that instead of using one coding scheme that would attempt and grasp multiple dimensions, it would be more fruitful to use a number of coding schemes each intended to capture a single dimension of the interaction. This is the main reason leading me to use two distinct coding schemes in a ‘layered’ coding fashion, instead of trying to capture the richness of the data in a single coding scheme. Such a process is well documented in the literature and is suggested as a possible avenue of overcoming both increased coding scheme complexity (thus running the risk of encountering lower interrater categorising reliability), and the limitations in

98

capturing the required level of richness from the data (Poole & Roth, 1989a, Poole et al., 2000:145-147, Franco & Rouwette, 2011).

3.4.5.2 Domain of meaning to be coded.

In general, there are two broad domains of meaning capturing for which codes and the process of coding can be applied. These are observer privileged meanings and subject-privileged meanings (Poole et al., 2000). Observer privileged meanings are meanings that an ‘outsider’33 could access. Subject privileged meanings are meanings that require an ‘insider’s’ knowledge of the details and intricacies surrounding the social status-quo of the group as a whole and for each participant individually, as well as the nature and history of the task that the group is faced with.

The coding schemes adopted or developed for this research, intend to capture only observer privileged meanings.

3.4.5.3 Validity of the coding schemes used

In this section the issue about the face and construct validity of the coding schemes used will be explicated.

As important as it is to have confidence that the data have been coded consistently in a reliable manner throughout, equally important is to have a certain degree of confidence in knowing that the codes assigned do capture the phenomena in question (Poole & Folger, 1981; Angoff, 1988:25-27).

Both coding schemes adopted have been validated by previous research and application. Construct validity of the coding schemes has been demonstrated in previous research bearing similar applications of the coding schemes (For MACS:

33

An ‘outsider’ being a person with no specific knowledge about the nature and content of the group studied as well as no knowledge in terms of the research questions explored.

99

DeSanctis & Poole, 1994; Poole & DeSanctis, 1992. For GWRCS: Folger, Hewes & Poole, 1984; Poole & Roth, 1989a & 1989b; Poole & Dobosh, 2010; Sambamurthy & Poole, 1992).

As previously explicated, the coding schemes intend to capture only observer privileged meanings. As such, it can be stated that face validity has been ensured through the high reliability scores for each of the coding schemes (Franco & Rouwette, 2011:173).

Essentially, no coding schemes were developed anew in this research, rules for coding and coding manuals were refined and developed where appropriate but no meaningfully new categories were introduced in any of the coding schemes used.

3.4.5.4 Colour coding used

As a visual aid the resulting phases were colour coded in the following manner.

3.4.5.4.1 Universal colour codes:

Yellow: for complex coded phases displaying more than 2 different behaviours.

Light Blue: for coded phases that display dual equal codes (i.e. both codes of a given phase start with ‘1’. For example, 1FW-1CW. It should be noted that for dual equal codes the first step is to go back to the raw data and re- read it while also watching the video and audio data, then, if possible, make a decision in terms of code’s importance (i.e. examine intensity, tone, body language, overall atmosphere) and assign the colour of the most dominant code. In the cases where no clear decision can be made, assign the light blue colour.

100

Black: Periods of Typing. It should be noted that for complex coded phases that display more than 2 different behaviours the yellow colour should be assigned even if the complex coded phases start with typing (T).

The rest of the colours used to denote each phase on the phasic timelines can be viewed in table 3.9

3.4.5.4.2 Phasic timelines colour codes Table 3:9 Phasic Timelines Colour Coding

GWRCS: MACS: FW: Light Green CW: Dark Violet OPP-OD: Red OPP-CAP: Red OPP-TAB: Red INT: White AFF: Pink COMB: Orange CONS: Estoril Blue CONT: Green DIR: Red NEG: Brown NEUT: Maroon NMA: Grey SUB: Dark Green UNF: White

ENL: Purple34

So far I have indicated a number of transformations and data manipulations for deriving the phasic timelines (Poole & Dobosh, 2010; Poole et al. 2000:229-262; Poole & Roth, 1989a). Phasic timelines served two purposes. First, they [i.e. the phasic timelines] were utilised as a data reduction technique for data collected at the micro-level being overly rich and detailed. Second, they were utilised as a flexible mapping technique allowing for the assessment of model appropriation complexity, as will be further explicated in the Analysis chapter.