Comparison with Publicly Available Sentiment Classifiers

There are some sentiment analysis tools that are publicly available on the Internet. This section presents the evaluation of several free versions of them on 177 negative and 182 pos- itive hand-labeled tweets. We discuss their characteristics and compare their performance results with the results of our sentiment classifier on the same test dataset.7

AlchemyAPI8 _{is a service, which performs sentiment analysis, keyword extraction, en-}

tity extraction, image extraction and tagging, etc. It employs natural language processing techniques and machine learning algorithms. Concerning sentiment analysis, this service is able to perform document-level, user-specified target, entity-level, and keyword-level sentiment analysis from a public webpage, HTML, or text document. It can perform sen- timent analysis for a text written in English or German. To test this service, we used their free starter API, which offers all service’s functions, but it is limited to 1,000 transactions per day and 5 concurrent requests. We used the Text API for sentiment analysis and classified tweets from the test set. The API returns, among other information, a polarity result of sentiment classification in the form “positive”, “negative,” or “neutral”, and a

6

The LATINO library uses the Snowball English stop word list, which is available at http://snowball. tartarus.org/algorithms/english/stop.txt.

7

Since the tweets in the test dataset were already HTML encoded, we decoded them in order to give the original texts of the tweets to the tested tools. For example, ‘&’ is transformed to the ‘&’ sign. We tested several public tools also without the HTML decoding and differences in the performance were negligible.

32 Chapter 3. Sentiment Classification

numerical sentiment strength value. We classified the tweets from the test dataset and achieved the accuracy of 83.57%.

Repustate9 provides sentiment analysis and text analytics for several languages, i.e., English, French, Spanish, Italian, German, Chinese, and Arabic. Sentiment analysis can be performed at the document, topic, or sentence level from a given text, an URL, or a list of URLs. The free usage enables 1,000 API calls per month. API response provides a numerical score of the sentiment classification, where a positive number represents positive sentiment and a negative number represents negative sentiment. In the documentation it is written that a score close to zero can be interpreted as the neutral sentiment. In our experiments, we interpreted exact zero score as the neutral sentiment. To test this service we used version 2 of the API and classified the tweets from the test dataset. In this experiment, we achieved the accuracy of 66.57%.

Text-processing10 is a free and open public service for text mining and natural lan- guage processing. It provides sentiment analysis, stemming, lemmatization, POS tagging, chunking, phrase extraction, and named entity recognition. The public API is intended for non-commercial usage, and it is limited to 1,000 calls per day per IP (Internet Protocol) ad- dress. For higher limits or commercial usage, one should use the Mashape Text-Processing API.11 We used the free public Sentiment API, which returns the sentiment label of an analyzed text in the form “pos”, “neg,” or “neutral”. It also returns probabilities for each of the sentiment labels. Using hierarchical classification, the algorithm first determines whether the analyzed text is neutral. If it is not neutral, it continues to the next step to determine the positive or negative sentiment polarity of the text. The sentiment API can perform sentiment analysis for text written in English, Dutch, or French. We classified the tweets from the test dataset using this service and used only the information about the final sentiment label. We achieved the accuracy of 61.00%.

Sentiment14012 is a sentiment analysis tool by Go et al. (2009). The service can perform sentiment analysis for text written in English or Spanish. We used their Simple Classification Service API and classified tweets from the test dataset using the HTTP GET requests. The API response returns sentiment polarity values in the form: “0” for negative, “2” for neutral, and “4” for the positive sentiment. On the test set we achieved the accuracy of 45.96%.

Finally, SentiStrength13determines the strength of the positive (on the scale from 1 to 5) and negative (on the scale from -1 do -5) sentiment in short texts. The SentiStrength algorithm uses a sentiment word strength dictionary and handles a range of non-standard spellings and approaches for expressing sentiment in text (Thelwall, Buckley, Paltoglou, Cai, & Kappas, 2010). Initially, it was implemented for English texts and optimized for short social Web contents, but it can be used also for a number of other languages, i.e., Finnish, German, Dutch, Spanish, Russian, Portuguese, French, Arabic, Polish, Persian, Swedish, Greek, Welsh, Italian, or Turkish. We registered and obtained the free version of SentiStrength in order to classify tweets from the test dataset. If the output sentiment strengths for the positive and negative class were the same, we classified the corresponding tweet as neutral. If the strengths were not equal, the tweet was classified as positive or negative, depending on which class got the higher score. We achieved the accuracy of 69.92% on the test set.

Summary of the evaluation performance of the tested publicly available sentiment ana- lysis tools is given in Table 3.4. As can be seen from the table, the best result (the accuracy

9 https://www.repustate.com/. 10 http://text-processing.com/. 11_{https://www.mashape.com/japerk/text-processing/pricing#!.} 12 http://www.sentiment140.com/. 13_{http://sentistrength.wlv.ac.uk/.}