Position Based Summarization Approaches

According to [13], there is an empirical observation that reviewers tend to summarize their overall feeling in a sentence or in a short paragraph, placed either in the beginning or at the end of the review. Ohana and Tierney [67] conducted experiments aimed at proving the existence of areas within documents which tend to carry more opinion context, and specifically focus on the closing remarks of the author at the end of the document. Working on this assumption, we select sentences from different positions, first taking the first sentence, as has been implemented in traditional summarization approaches, then we select just the last sentence, then the last 3 sentences and the first 3 sentences. We have designed these experiments in a similar manner to the work of [13], which we have specified that our work bears the most similarity to. Only the polarity scores derived from SentiWordNet for the words from the sentences were considered here.

The position based summaries we implemented were the first and last sentences of the documents to represent the whole, and also the N-closing and N-opening sentences approach. We did not consider sentences in the middle of these documents as a separate test because from our observation, most times, the bodies of movie reviews focus on the movie plot, than on sentiments. This however, is a test we would carry out in the future, to test if determine the validity of this observation.

Table 4.4: First and Last Sentence Approach Results

Precision Recall Accuracy First Sentence Positive Reviews 50% 52%

50% Negative Reviews 50% 48%

Last Sentence Positive Reviews 55% 59%

55% Negative Reviews 55% 47%

4.5.1.1 First and Last Sentence Approach

In this set of tests, the first and the last sentence of the negative, as well as positive reviews were considered. This gave 1000 first sentences and 1000 last sentences. These sentences were used to represent each of the documents which they were taken from. The purpose of this experiment was to assess how efficient it would be to detect the polarity of a document, using only the first sentence of the review as a summary of the entire review, and also, using just the last sentence of the review as a summary of the entire review. Each document was first represented by just its first sentence, and the scores of each of the words in the sentence was derived from SentiWordNet using the overallPolarity approach. The polarity scores of the words in this sentence were then summed up. If the positive score was higher than the negative score, then the sentence, and hence the document was classified as positive. Otherwise, it was classified as negative. The same experiment was repeated using just the last sentences. The results from the experiment are presented in Table 4.4.

We observed that the accuracy obtained from these two types of summaries were not very impressive. The accuracies are lower than those that were obtained from the approaches in section 4.2. The precision and recall values have also been affected and are lower than those reported in that section. It appears that these summaries are not appropriate representations of the full documents, and they do not appear to solve the problems outlined in the error analysis performed on the classification results in the previous section.

4.5.1.2 N-Closing and N-Opening Sentences Approach

the next type of summaries to be tested are the N-block summaries. We decided to use ’3’ as the value of our ’N’ as this was the value used in [13], and hence we would have some source with which to compare our results with.

For the closing sentences, we decided to take the last-3 sentences of the each review, similar to that tried by [13], who carried out extractive summarization in their work and stated that based on empirical observation, reviewers tended to summarize their overall feeling in a sentence or a paragraph, placed at the beginning or end of the review.

We went on to implement three variations of the last sentences summarization. We call these variations Test 1, Test 2 and Test 3. These three tests have never been tried before, to the best of our knowledge, and the aim for attempting them here is to discover if there are certain sentences within the last sentence segment of a review which are more definitive, where sentiment polarity is concerned.

Test 1 In Test 1, we took the average of the scores of the last three sentences. The positive scores of the three last sentences were added and divided by 3, and the same was done for the negative scores. The resulting values were assigned as the positive and negative score of the document, respectively. The higher polarity score of the two was assigned as the polarity of the document. The motivation behind this design of Test 1 was to enable us obtain the overall sentiment of that section of the review. Our aim was to obtain the average negativity or positivity expressed in these section of the review.

Test 2 In Test 2, we selected what we refer to as the most popular polarity, as the overall polarity for the summary. The three sentences’ polarities were inspected, and the polarity occurring the most number of times was selected as the overall polarity. The assumption behind this selection technique was that there would probably be a cluster of the sentiment being conveyed by the review in these sentences, so making a selection based on the most frequently expressed sentiment could lead to predicting the overall polarity of the review.

begin

For all sentences in document (i)

Add all positive scores of words in (i)

Store result in PosSum

Posscore of Document (i) = PosSum/3 Add all negative scores of words in (i)

Store result in NegSum

NegScore of Document (i) = NegSum/3

Compare PosScore with NegScore If NegScore > PosScore

(i) is a negative document else,

(i) is a positive document end

Figure 4.8: Steps in carrying out Test 1

The point to note in this test was that there were few occurrences where one sentence was classified as positive, one as negative, and the third as neutral. In such a case, the solution was to assign the polarity with the highest score as the overall polarity.

Test 3 In Test 3, we selected the highest recorded score among the three sentences as the overall polarity. The positive and negative scores were compared for the three sentences, and the polarity with the highest score was assigned as the overall polarity for the document. The motivation behind this choice was to capture strong opinions which could have been expressed in the closing sentences in order to drive home a point. The results for these tests are shown in Table 4.5.

We repeated the same set of experiments for the first-3 sentences as well. The results obtained from these tests are shown in Table 4.6. Same as with the last-3 sentences, our objective was to detect if the opening section of the review was actually where the reviewers had summarized their overall feeling.

begin

For all sentences in document A

Neg = 0 Pos = 0

While sum of NegScores in Sentence j in A!= sum of Positive Scores if sum of PosScores of words in j > sum of NegScores of words in j

Sentence j in A is positive Pos = Pos + 1 else Sentence j in A is negative Neg = Neg + 1 end if Neg > Pos

Polarity of document A = Negative else

Polarity of document A = Positive end

end

Figure 4.9: Steps in carrying out Test 2

Table 4.5: Position Based Summary Approach Results -Last 3 Sentences Precision Recall Accuracy

Test 1 Positive Reviews 59% 67% 60% Negative Reviews 61% 53%

Test 2 Positive Reviews 57% 70% 58% Negative Reviews 59% 49%

Test 3 Positive Reviews 57% 61% 57% Negative Reviews 58% 54%

Table 4.6: Position Based Summary Approach Results -First 3 sentences Precision Recall Accuracy

Test 1 Positive Reviews 60% 54% 55% Negative Reviews 55% 56%

Test 2 Positive Reviews 48% 56% 53% Negative Reviews 53% 51%

Test 3 Positive Reviews 54% 51% 54% Negative Reviews 54% 57%

Given document A with n sentences

begin

Negscore = 0 Posscore = 0

For each sentence in document A

Negscore = sum of all negative scores of constituent words Posscore = sum of all positive scores of constituent words end

For sentence 1 to n compare NegScores Temp = max [NegScores] compare PosScores Temp2 = max[Posscores] if Temp > Temp 2 Document is Negative else Document is Positive end end end

Figure 4.10: Steps in carrying out Test 3