Filtering of generated training examples

The set of automatically extracted PRCs is rather noisy. Part of the reason is that the training examples for (in)consistent words are automatically extracted and contain a considerable amount of noise. We experiment with different filtering methods to create a cleaner set of training examples and resulting PRCs. These filters are based on an analysis of wrongly extracted examples, but address general issues that are common in the area of sentiment analysis. Our filters are designed for high precision on filtering out incorrect examples rather than recall.

The first and most important issue affecting the extraction of training examples is subjectivity (cf. Section 2.2.1). Sentiment words do not express sentiment in every given context. In a preliminary study on a subset of sentiment words extracted as training examples, we found that about 50% of words extracted as inconsistent training examples

50 3. Polarity reversing constructions

did in fact not express sentiment in the sentence context. Consider the following example with the marked sentiment words:

(3.5) “easypos to hold steady when usingslowerneg shutter speeds.” (positive)

The word “slower” does not express sentiment in this context, rather it describes a setting. Identifying and discarding non-subjective phrases like these would improve the classification results as well as the quality of the extracted polarity reversing construc- tions. We implement several filters to tackle this issue.

Different words have different degrees of subjective usages2. In one of the dictionaries we are using, two categories of sentiment words were distinguished and manually anno- tated: weakly and strongly subjective words. Strongly subjective words are subjective in most contexts, whereas weakly subjective words only have certain subjective usages but may have other non-subjective uses (Wilson et al., 2005). To get a higher percentage of actually subjective usages in the training examples, we use only strongly subjective sen- timent words from the dictionary. Some examples of strongly subjective words include “excellent”, “great”, “disappointment”. Examples for weakly subjective words are “slow”, “easy”, “lose”. We call this filter Subjstrength.

Some sentiment words from the dictionary can occur with a POS that never expresses sentiment such as interjection. Consider these examples of extracted training examples for the positive sentiment word “okay”:

(3.6) a. “pictures are okaypos,JJ but not great.” (negative)

b. “Okaypos,UH, now to the cons: The battery life could definitely be better.” (negative)

In the first example, the word occurs as an adjective (JJ) and expresses an opinion. In the second example in occurs as an interjection (UH) to introduce a new part of the review and does not express any opinion, so no training example should be extracted. There are several such POS that never express sentiment which we ignore: interjection (UH), preposition (IN), determiner (DT) or cardinal number (CD). Note that we are not removing the words from the dictionary, we only ignore occurrences in the data with these specific POS. We call this filter POSignore.

Another reason that a sentiment word may not express an opinion is that sometimes sentiment words occur in an objective context as part of aspects or environment descrip- tions. A few examples from our data:

(3.7) a. “Very minor chromatic distortion with widepos angle . . . ” (negative)

b. “in lowneg lightpos conditions images tend to get blurred.” (negative)

3.2. Methods 51

POS0 POS1 POS2 Example aspects

NN NN “battery life”, “picture quality” (CD) JJ NN “optical zoom”, “mega pixel” (JJ) JJ NN “sound quality”, “wide angle” (IN) JJ NN “bright sunlight”, “low light”

Figure 3.3.: Patterns used for aspect bigram extraction. The token matching POS0 is

not extracted as part of the bigram.

c. “The compact flashneg card is easy to use and seems more durable to me that the

smartpos media card.” (positive)

While a camera having a “wide angle” expresses positive sentiment about the camera, in this case the “wide angle” only refers to a setting for the camera. Other similar cases are environment descriptions (“in bright light”) or names of camera parts or accessories (“flash card”, “smart media card”).

This can be viewed as sentiment words occurring as part of a product aspect3 which should be ignored. As we do not have a list of aspects from our domain, we extract aspects automatically. Following Glaser and Schütze (2012), aspects are most often nouns or contain a noun. Single noun aspects are a problem of the dictionary, at this point we are interested in bigram aspects. To extract these, we define a list of extraction patterns on POS as shown in Figure 3.3. In trigram patters only the last two tokens are extracted as a bigram, but the POS of the word before the extracted bigram has to match the first POS of the pattern. We extract the most frequent bigrams as aspects for our domain. A manual inspection of the extracted bigrams confirms the claim of Glaser and Schütze (2012) that nearly all frequent bigrams in a domain corpus are aspects, so no further processing is performed. Once created the list of aspects, we ignore sentiment words that occur as part of an aspect in the text. We call this filter Aspects.

Finally, another reason that a sentiment word does not express sentiment is that it is acting as an intensifier for another sentiment word. Consider the following examples:

(3.8) a. “It is ridiculouslyneg easypos to use.” (positive)

b. “digital zoom is pretty_pos useless_neg because of quality.” (negative) c. “The sound quality is vastly_pos improved_pos.” (positive)

Even though “ridiculous” on its own is a negative sentiment word, in the example it only serves to intensify the positive sentiment of “easy” and should not be extracted

3_{While settings and parts certainly qualify as product aspects, environment descriptions would proba-}

52 3. Polarity reversing constructions

as a reversed training example. For our filter, we ignore adverbs that modify another sentiment word (of all POS tags, these will be adjectives or verbs). Adjectives can also intensify nouns, but the situation is less clear, so we currently restrict ourselves to adverbs. We call this filter Intensifier.

Besides words that do not express sentiment, we have a second issue that produces errors in the extraction of training examples: words that express a different polarity than the one contained in the dictionary. The domain-specificity of sentiment words is well-recognized in sentiment analysis (Pang and Lee, 2008). The sentiment dictionary we are using has been created for the domain of news texts, but we are analyzing product reviews. Consider the following examples:

(3.9) a. “photos are sharpneg with acurate color.” (positive)

b. “technical supportpos for both samsungsprint pcs are in the dark.” (negative)

c. “you have the options of taking black_neg and white photographs and there are some color filters available as well.” (positive)

The word “sharp” is contained in the dictionary as a negative sentiment word, but the usage in our domain in the context of pictures is always positive. Another example is “support” which the dictionary assumes to be a positive sentiment word, but most occurrences in our data are neutral, e.g., “technical support” or “customer support”.

We want to remove words that are not sentiment words in our domain or mostly express a different polarity, but we do not have a domain-specific list and do not want to invest the manual annotation effort. As we have lots of annotated data with sentiment on the document-level, we can train a classifier with unigram features in the domain. The unigram features with high weight for a class are good indicators for the class in the domain. But not all features are sentiment words, e.g. “pda” has high weight for the positive class in our domain, because it is something special and good for a phone to be usable as a PDA (personal digital assistant)4_{. Using the classifier features directly as}

sentiment words would not give very good results. We therefor use the classifier features only to filter the manually created sentiment dictionary. Words from the subjectivity clues list are only included in our list of sentiment words if they are among the c features with highest weight for the class. We call this filter Domain.

Finally, instead of filtering the training examples or the dictionary, we can also filter the resulting set of PRCs. One case that creates false positive PRCs is a fixed word combination where one of the word is a sentiment word and the combination often occurs in the combination in sentences with the opposite polarity. An example is the