Agreement for Intensity Judgments

intensity of expressive subjective elements. For the experiments presented later in Chapter5, I merge the high and extreme intensity classes because of the rarity of the extreme class (only 2% of sentences in the corpus contain an annotation with extreme intensity). Thus, when

calculating agreement for the various intensity judgments, I also merge the high and extreme ratings, to mirror their treatment in the experiments.

Included in the judgment of intensity is a determination of whether a private state is being expressed at all. That is, when an annotator chooses to mark an expression as an objective speech event as opposed to a direct subjective annotation, the annotator is in essence making a judgment that intensity is neutral. Thus, to accurately measure agreement for intensity, I consider direct subjective annotations and objective speech annotations together. The value of the intensity for all objective speech events is neutral. For all objective speech events that are not implicit, expression intensity is also neutral.

The classes used for intensity judgments represent an ordinal scale; this raises the ques- tion of which agreement metric is appropriate for evaluating intensity. For the combined direct subjective and objective speech event annotations, the rating scale for both intensity and expression intensity is neutral, low, medium, and high. For expressive subjective ele- ments, the rating scale for intensity is low, medium, and high. Agreement metrics such as Cohen’s κ treat all disagreements equally, which is suitable for discrete classes. However, with the ordinal nature of the intensity judgments, not all disagreements are equal. For example, a disagreement about whether intensity is neutral or high is more severe than a disagreement about whether it is medium or high. Cohen’s κ, therefore, is not a suitable metric.

There is an adaptation of Cohen’s κ called weighted κ (Cohen, 1968), which is for use with ordinal data. Weighted κ assigns weights that allow for partial agreement. However, the weights are calculated based on the number of categories. The intensity scale used for direct subjective and speech event annotations is slightly different than the one for expressive subjective elements, which doesn’t include the neutral class. This means that with weighted κ, the weights for expressive subjective elements will be different than the weights for direct subjective and speech event annotations. Because of this, weighted κ is also inappropriate.

The metric that I use for agreement for intensity judgments is Krippendorff’s α (Krip- pendorff, 1980; Krippendorff, 2004). Like Cohen’s κ, Krippendorff’s α takes into account chance agreement between annotators, but it is more general. It can be used to calculate agreement for both discrete and ordinal judgments, and its method of weighting disagree-

ments does not depend on the number of categories. In its most general form, α is defined to be

α = 1 − Do De ,

where Do is a measure of the observed disagreement and Deis a measure of the disagreement that can be expected by chance. Krippendorff’s α ranges between 0 and 1, with α = 1 indicating perfect agreement and α = 0 indicating agreement that is no better than chance. With α, a distance metric is used to weight disagreements. Different distance metrics are used for different types of data. For intensity, the ratings map naturally to the scale [0,1,2,3], where 0 represents neutral and 3 represents high. Using this scale, I can use the distance metric that squares the difference between any two disagreements. Thus, the distance weight is 1 for any disagreement that differs by one (e.g., neutral-low), the distance weight is 4 for any disagreement that differs by two (e.g., neutral-medium), and the distance weight is 9 for any disagreement that differs by three (e.g., neutral-high).

I measure agreement for the intensity of the combined direct subjective and speech event annotations using the set of matching frames identified in Section 3.2.3.4 (the matching frames in S1intersectionand S2intersection). This is the same set of annotations that are used for calculating agreement for distinguishing between objective speech event and direct subjective frames. To measure expression-intensity agreement, I also use this set, with the exclusion of the matching frames that are marked with the implicit attribute.

Table 3.8gives the pairwise α-agreement values for the intensity and expression intensity judgments of the combined direct subjective and objective speech event annotations. For comparison, the absolute percent agreement is also given. In interpreting α, Krippendorff (2004) suggests that values above 0.8 indicate strong reliability and values above 0.67 are sufficient for tentative conclusions. Using this scale, we see that the α scores for the intensity judgments of direct subjective and speech events are good.

For expressive subjective elements, I again identify the set of matching annotations that were marked by both annotators. Table 3.9 gives the pairwise α-agreement for the intensity of this set of expressive subjective elements, along with absolute percent agreement for comparison. Unlike the agreement for the intensity judgments of direct subjective and speech

Table 3.8: α-agreement and percent agreement for intensity judgments for the combined direct subjective and objective speech annotations

Expression Intensity Intensity Annotator Pair α % α % A & M 0.79 0.73 0.76 0.66 A & S 0.81 0.75 0.76 0.63 M & S 0.76 0.76 0.73 0.59 average 0.79 0.75 0.75 0.62

Table 3.9: α-agreement and percent agreement for expressive subjective element intensity judgments Intensity Annotator Pair α % A & M 0.40 0.49 A & S 0.52 0.56 M & S 0.46 0.54 average 0.46 0.53

event annotations, agreement for the intensity judgments of expressive subjective elements is not high. A look at the disagreements shows that many of them are influenced by differences in boundary judgments. Although annotations are considered matching as long as they have overlapping text spans, differences in boundaries can affect how intensity is judged. Example 3.12 below shows how the same subjective expression was judged by two annotators.

(3.12)

A: <high>imperative for harmonious society</>

M: <medium>imperative</> for <medium>harmonious</> society

Both annotators recognized that the above phrase is subjective. However, while the first annotator marked the entire phrase as a single expressive subjective element with high intensity, the second annotator marked particular words and smaller phrases as expressive subjective elements and judged the intensity of each separately.

A severe type of disagreement between annotators is a difference in intensity ordering, i.e., annotator A rating expression 1 as more intense than expression 2, and annotator B rating expression 2 as more intense than expression 1. Fortunately, there are few such disagreements. On average, only 5% of all possible pairings of matching annotations result in disagreements in the ordering of intensity.

3.2.3.7 Agreement for Intensity of Sentences As with subjective and objective