Human-generated Context

We seek to more reliably incorporate context using human computation. We propose a more structured and useful definition of context that is based on two concepts:

• A phrase is an expression that can convey sentiment, such as good or small.

• A context is an expression (possibly an empty word), in whose presence the phrase conveys a concrete sentiment, such as hardly any or canister.

We rely on many human workers to acquire a context model that enumerates phrase and context pairs, along with their positive or negative polarities (Table 4.9 shows sample word combinations selected with human computation).

We use this human-generated model to help sentiment classification become competitive on broad domains. One option is to directly use it to label documents, by applying the generic classification procedure described above. More specifically, at Step 3, we can compute a review sentiment score by summing the polarities of phrase and context pairs quantified by their frequency in the text:

• If the context is not empty, we compute the frequency of a phrase and context pair as the number of times they appear together in the sentences of the review. However, we distinguish the cases in which the phrase and the context are separated by at most three words from other co-occurrence patterns. More specifically, we count the former case as a full occurrence that we quantify with a weight of one. We consider the other cases as partial occurrences that we quantify with a smaller weight, which decreases as the distance between the phrase and the context increases. Similar to Ding et al. [25], this allows us to give less importance to farther away contexts that are less likely to target the phrase in question. Note that, in initial experiments, we found that setting the distance threshold to three gives good results.

• On the other hand, if the context is an empty word, we compute the frequency of the phrase as the number of times it appears in the text outside any of its known contexts. Therefore, we use the phases’ context-dependent polarities wherever possible and revert to the context-independent ones otherwise.

A context model does not have full coverage, given that it contains longer word combinations that do not always appear in texts. What we really want is to integrate it into lexicon and supervised methods, which contain individual words on which we can fall back when context features cannot be used. We suggest three ways in which context can be used to extend lexicon and supervised methods. We extend the lexicon by merging it with a context model and using the resulting union to classify reviews. We extend the supervised method with two approaches. One is an ensemble that separately uses the supervised and context models to obtain two sentiment scores, then combines the two to reach a final classification. Another one retrains the supervised method on an extended feature space that includes all the elements in the supervised and context models. Note that this is similar to Kennedy and Inkpen [50], who described how to extend lexicon and supervised methods with negations, intensifiers, and diminishers from a predefined list, and also proposed to combine the two in an ensemble method. We detail our three approaches in what follows.

Sentiment Model Extension

To improve the lexicon method, we apply the extension at the level of the sentiment model, in Step 2. Given a lexicon and a context model, we obtain their union, then use the latter to classify reviews. We merge the two models as follows: for every word in the lexicon model, we add it to the union by pairing it with an empty context component and assigning it the polarity indicated by the lexicon; for every element in the context model, we add it to the union, unless its context is an empty word and the phrase already belongs to the lexicon model. Therefore, the union refines the context-independent polarities in the lexicon model with the context-dependent ones in the context model. We use this union to classify reviews by adding feature polarities quantified by their frequency in the text, as explained above.

Sentiment Score Extension

We cannot apply the previous extension to a supervised model. Lexicon and context models both contain discrete, positive or negative polarities and can thus be merged. However, supervised models contain small continuous polarities that are perturbed when overridden with the discrete ones in a context model. Instead, we improve the supervised method by applying the extension at the level of the sentiment score, in Step 3. We classify a review by separately using the supervised and context models to obtain two sentiment scores. We then combine the two scores, hoping to rectify some of the errors produced by the supervised model. To make the two scores compatible, we use a parameter that scales down the discrete one obtained with the context model. A good starting point to choose a value for this parameter is the average polarity magnitude in the supervised model. In our initial experiments, we find that a value of 0.08 works well.

3.2. Document-level Sentiment Classification

Feature Space Extension

The previous approach integrates context by combining two sentiment scores separately obtained with a supervised model and with a context model. We expect that context can be more effective if it is integrated in the training process, when we can rely on the SVM to find suitable polarity scores for all the features involved. Therefore, to further improve the supervised method, we propose to apply the extension at the level of the feature space, in Step 1. We extend the feature space so that it includes elements from both the supervised and the context models. We then retrain an SVM on this extended feature space and obtain a new supervised model.

To represent a review in the extended feature space, we take two approaches. When we want to extend a supervised model containing only individual words, we obtain a feature space that simply unites all the words and word combinations in the supervised and context models. We represent each review sentence as follows:

• We first find all the elements in the context model for which both the phrase and the context appear in the sentence. For every match, we mark the words involved and output a feature that concatenates the phrase and the context.

• We then find all the words in the supervised model that appear in the sentence and are still unmarked. For every match, we output a feature capturing that word.

For instance, let us assume we have a sentence: I hate this vacuum, it has a very small canister. A supervised model that contains the words hate, very, small, and canister will interpret the sentence as [hate][very][small][canister]. However, when we extend this feature space with a context model that contains the phrase very small in the context canister, the resulting supervised model will interpret this sentence as [very small canister] [hate].

When we want to extend a supervised model that also contains bigrams, the approach above can output many superfluous features that often overlap with one another. Instead, we obtain a feature space that contains the individual words and the bigrams in the supervised model, but we restrict only to those bigrams that also appear in the context model. We represent each review sentence as follows:

• We first find all the bigrams in the supervised model whose both constituent words appear in a phrase and context combination from the context model. We output every such bigram that appears in the sentence.

• We then find all the words in the supervised model that appear in the sentence. For every match, we output a feature capturing that word.

For example, a supervised model that contains the features hate, this vacuum, a very, and small canister will interpret the sentence from before as [this vacuum][a very][small canister]

Figure 3.1: Context acquisition. Main game interface

[hate]. However, we can intersect the bigram space with a context model that contains the phrase very small in the context canister. The resulting supervised model will interpret this sentence as [small canister] [hate], thus effectively pruning some of the irrelevant bigrams.