Discussion

During the annotation of our data and while analyzing the other corpora, we have encountered some more general issues and open questions, which we shortly highlight in the following. We focus on the following issues for this discussion: The definition of what is a comparison and what is not (in three different flavors), aspects versus entities, and discontinuous arguments.

Definition of a comparison. Looking at our experiments on Amazon Mechanical Turk and our agreement study, we can confirm what previous work has already discussed: that the decision of what is a comparison and what is not is difficult even for humans.

92 4. A corpus of comparisons in product reviews

There are of course many sentences that are very obviously comparisons (e.g., those from the introductory examples) or non-comparisons (e.g., “I bought A last week”, “B is fantastic”). Some other categories of sentences that look like comparisons but are not (e.g., idioms, correlatives, see also Sentence 2.19 in Section 2.5.1) may confuse annotators at first, but can be excluded with clear annotation guidelines. But even after this, there are still some questionable items:

(4.12) a. “this is my first digital camera.”

b. “The 2008 Subaru Impreza WRX STI is based on the Impreza WRX hatchback . . . ” c. “The images were great both indoors and out.”

d. “. . . athough I would like to see it a little faster.”

e. “It mirror flip doesn’t sound like a mechanical camera . . . ” f. “I had to compare this camera with the Nikon D80.”

Sentence 4.12a is from the J&L data and many similar sentences with the predicate “first” have been annotated in this data set. Sentence 4.12b from batch 5 of the JDPA cars data set has the word “based” annotated as the comparative predicate. In each case, the other data set does not annotate similar sentences, nor do we in our data. Sentences 4.12c to 4.12f are from the set we used to calculate annotator agreement, which we did after training our annotators and after they had already gained some annotation experience. Sentence 4.12c is a comparison between different usages of the same product, Sentence 4.12d is a wish, Sentence 4.12e is a description of the sound the camera makes. Sentence 4.12f states that there is going to be a comparison, but this is not (yet) it. None of these sentences compare two entities and none should be annotated as a comparison. We have updated the annotation guidelines to explicitly give examples of these categories of sentences to be excluded, but there are probably more such cases that need to be specified.

Limitations of the annotation scheme. Now let’s turn to the other side of the coin, statements that are comparisons, but cannot be annotated in the current scheme. Our working definition of a comparison has been “any statement about the similarity or difference of two entities”, which would include the following examples from our data:

(4.13) a. “. . . it simulates iso 100 whereas my D70s only did iso 200.”

b. “Although it yields nearly the same image quality as the D200, it’s unfortunately twice as expensive, lacks environmental seals, . . . ”

c. “It looks closer to film than any digital camera I’ve owned.” d. “Pricewise, the H2 slots in between Canon S3 IS and .”

4.5. Analysis of the data 93

As two examples that do not fit the current model, Jindal and Liu (2006b) have already discussed juxtapositions (Sentence 4.13a) and non-gradable comparisons of an aspect (here: “environmental seals”) that one entity has and the other does not (Sentence 4.13b). In order to annotate these examples, we would need two different aspect slots. Entities can be problematic as well. Sentence 4.13c does compare two entities (“it”, “any digital camera”), but there is third entity (“film”) involved as well. Sentence 4.13d even more clearly describes a relation between three entities (“H2”, “Canon S3 IS”, [missing token]). This type of relation cannot be put into our current two-entity model and we would need additional comparison types to describe the relations as well. Finally, sometimes the problem is the choice of predicate. Sentence 4.13e expresses an equative comparison between the two cameras in the aspect “cheaply made” and would be easy to annotate if it contained the word “both”. As it is, there is no good choice for a predicate. It is unclear which of the above examples should be included and in what way. For now, we can only note that the current annotation scheme is not able to capture all possible statements of similarity or difference of two entities.

Implied comparisons and sentiments. It has been noted that every value statement about an entity contains an implicit comparison to some sort of internal standard (Hud- dleston, 2002). People only note that something is “good” or “bad” because they compare it against some sort of general standard or expectation about how things should be. This implicit comparison is more overtly present when an adjective is used, e.g., in construc- tions like the elative “X is very good”, the excessive “X is too good”, or the assetive “X is good enough”. While we certainly would not want to treat every sentiment expression as an implicit comparison, in some cases it may be warranted:

(4.14) “[D70]E1 beats [EOS 300D]E2 in almost [every category]A, EXCEPT ONE.”

While the comparison at the predicate “beats” is annotated, one might argue that there is another comparison implied in the second part of the sentence, namely that the “EOS 300D” is at least as good or maybe even better than the “D70” in one category (which presumably is further elaborated on in the following sentences of the review). While this is certainly an interesting point for future work, it seems hard to formulate clear guidelines of when such implicit comparisons should be considered and how exactly they should be annotated.

Aspect versus entity. Conceptually, a comparison contains two entities that are com- pared either in their totality or in some shared aspect. In practice, the distinction between entity and aspect is sometimes hard to make, because aspects form a hierarchy

94 4. A corpus of comparisons in product reviews

and aspects of aspects are compared in arbitrarily deep nestings. Consider the following example sentences (annotations from the JDPA camera data):

(4.15) a. “[This camera]_E . . . its [screen]_A is much bigger than the [400D]_E.”

b. “. . . its [screen]E is [smaller]A than the [ones]E on some competing models . . . ”

The two sentences are very similar, but the annotators took different stands on whether “screen” is an entity or an aspect. The annotation of “this camera” as an entity allows us to connect it back to some product and list the comparison under its aspect “screen”. The annotation of “screen” as an entity on the other hand defines what is syntactically com- pared (the camera’s screen, not only the camera) and allows us to annotate comparisons of aspects of aspects in a parallel way:

(4.16) “. . . its [screen]E has smaller [resolution]A than the [ones]E on . . . ”

In our data, we instructed annotators to annotate syntactic entities, i.e., “its screen” for the example sentence, and additionally annotate the type of entity as “aspect”. This allows us to treat arbitrarily deep aspect hierarchies and still retain the information that what is syntactically the compared entity in this sentence, is conceptually an aspect of the product we are interested in. Still, even with these guidelines the decision is sometimes hard to make in practice, especially if no other aspect is present in the sentence, and we have discussed many of these cases during annotation. Additionally, the annotation scheme does not capture information about the internal structure of “its screen”, where “its” refers to the product and “screen” to the aspect of the product.

While for training a system and detecting the components, ultimately the exact an- notations may not matter so much, the question is very relevant when the detected comparisons are used for aspect-based summaries that group sentiment analysis results for every entity by the recognized aspects. Linking back every annotated entity to some predefined hierarchy of products and their aspects could solve the problem of unambigu- ously identifying what aspect is talked about, but introduces other problems, namely that we need to create such a resource and that we limit the annotators to the aspects that are contained in this resource. Alternatively, post-processing is necessary in order to be able to generate usable aspect-based summaries from the annotations we have.

Discontinuous arguments. We have discussed multiword predicates and introduced the argument type “scale” for the purpose of modeling them, but arguments can also consist of multiple words that do not always have to be adjacent. Consider the following example from our data:

(4.17) “Realistically [Micro-cam batteries]E1 (since they must be very small) can’t [store]A? as

4.6. Summary 95

In the example, “power” has been annotated as the aspect, but in reality the discussed aspect is “store power”, as in contrast to “needs power”, “provides power” or any other possible aspect of “power”. In our annotation scheme it would have been possible for the annotators to annotate two aspects for one comparisons, but they correctly did not do that as conceptually the annotation of two aspects is intended to denote that the entities are compared in two different, independent aspects, such as in this example:

(4.18) “However [it]E1 isbetter in so many ways such as [image quality]Aand [handling]A. . . ”

It remains an open question how frequent such discontinuous aspects are and whether it is worth the effort of defining more complex and more flexible representations of comparisons to deal with them.

4.6. Summary

In this chapter, we presented a dedicated gold standard corpus of comparison sentences from English camera reviews. For our purposes we define a comparison as any statement about the similarity or difference or two entities which covers a wide variety of expres- sions. For each sentence we have annotated detailed information about the comparisons it contains: The comparative predicate as the anchor that introduces the comparison, the type of the comparison, the two entities that are being compared with their entity type, and the aspect they are compared in. We have described our annotation process and given an overview of our annotation guidelines. The results of our agreement study showed that the decision whether a sentence contains a comparison is difficult to make even for trained human annotators. Once that decision is made, we can achieve consis- tent results for detailed annotations. In total, we have annotated 2700 comparisons in nearly 2200 sentences from camera reviews which makes our data the largest resource of comparisons in English reviews currently available. The annotations and guidelines are publicly available on our website10_.

5. Detecting comparisons in