Context-Based News Headlines Analysis: A Comparative Study of Machine Learning and Deep Learning Algorithms

(1)

Syeda Sumbul Hossain^*, Yeasir Arafat^†and Md. Ekram Hossain^‡ Department of Software Engineering

Da®odil International University Dhaka, Bangladesh

Received 16 January 2020 Accepted 25 January 2021 Published 23 April 2021

Online news blogs and websites are becoming in°uential to any society as they accumulate the world in one place. Aside from that, online news blogs and websites have e±cient strategies in grabbing readers' attention by the headlines, that being so to recognize the sentiment orientation or polarity of the news headlines for avoiding misinterpretation against any fact. In this study, we have examined 3383 news headlines created by ¯ve di®erent global newspapers. In the interest of distinguishing the sentiment polarity (or sentiment orientation) of news headlines, we have trained our model by seven machine learning and two deep learning algorithms. Finally, their performance was compared. Among them, Bernoulli naïve Bayes and Convolutional Neural Network (CNN) achieved higher accuracy than other machine learning and deep learning algorithms, respectively. Such a study will help the audience in determining their impression against or for any leader or governance; and will provide assistance to recognize the most indi®erent newspaper or news blogs.

Keywords: Sentiment analysis; opinion mining; semantic orientation; sentiment polarity detection; news headline; text mining.

1. Introduction

Over the years, news headlines are being used as the key benchmark of any news.

Readers become fascinated with any news by reading the headlines of any news instead of going through the whole. Most often we do not go through the entire news as in our view, headline is the essence of any news Ref. 1. We know there are no frontiers in circulating news around the world with the instantaneous involvement of social-networking. Most of the time, we are being misguided by rendering the caption

This is an Open Access article published by World Scienti¯c Publishing Company. It is distributed under the terms of the Creative Commons Attribution 4.0 (CC BY) License which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Vietnam Journal of Computer Science Vol. 9, No. 1 (2022)

#.c The Author(s)

DOI:10.1142/S2196888822500014

1 Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 134.122.89.123 on 09/22/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(2)

of any news that very often causes social-political collisions on any tribe. In Ref. 2, this is investigated how headlines in°uenced the readers' reasoning and deceiving them towards the interpretation of statements. In such a case, it is obligatory to identify the sentiment polarity of the news headlines or caption for avoiding mis- conception Ref.1.

Based on the context of any news, sentiment polarity would be di®erent Ref.1.

Moreover, sometimes any particular news is demonstrated in di®erent ways in var- ious online news sources. Thereby, it is important to ¯nd the semantic orientation or polarity of any news caption or headline. There is a de¯ciency in studies to ¯nd out the semantic orientation of news headlines in reliance on the context of news.

However, the consequences of news on social-political, economic, and business behavior have been reported in numerous studies. How the commodity price is a®ected by the news is presented in Ref.3. How the market behavior instinctively interlinked and in°uenced by business and economic news is claimed in Refs.4and5.

This study trails a comparative analysis of machine learning and deep learning techniques to detect the semantic orientation of any news headline. We have used Naïve Bayes Ref.6, MNB Ref.7, Bernoulli Naïve Bayes Ref.8, Logistic Regression Ref.9, Stochastic Gradient Descent (SGD) Ref.10, SVC Ref.11and Nu SVC Ref.12 and Convolutional Neural Network (CNN) Ref. 13, Long short-term memory (LSTM) Ref.14to learn our models. The result of this study will provide assistance to recognize the most indi®erent newspaper or news blogs by detecting the semantic orientation of any news titles or headlines.

This paper is organized in the following manner. Related work is portrayed in Sec.2followed by Research Methodology and Result & Discussion in Secs.3and4, respectively. Section5 depicts our contribution.

2. Related Work

Numerous publications have been done on product evaluation applying sentiment analysis. It is revealed in Ref.15that a large number of sentiment analyses were done on product assessment or review data. Besides news data, social media data and web blogs data are also used in opinion mining. In Ref. 16, Naïve Bayes classi¯cation, maximum entropy classi¯cation and Support Vector Machines (SVM) were applied to conduct sentiment analysis on movie review in which they compared the human generated and machine learning results. The study reveals that SVM gives the best accuracy while Naïve Bayes gives the worst. SVM and Particle Swarm Optimization (PSO) are used in Ref. 17 for analyzing ¯nancial news. In Ref. 18, a survey was conducted to determine sentiment analysis constraints or challenges relevant to di®erent methodologies. A feature-based opinion recapitulating technique is presented in Ref.19of product reviews using natural language processing. A system Opinion Observer Ref. 20 is implemented using holistic lexicon-based approach Ref.21. A lexicon-based approach is also applied in Ref.22to distinguish the positive or negative orientation of the commercial or ¯nancial news. A large-scale sentiment Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 134.122.89.123 on 09/22/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(3)

analysis system is presented in Ref.23to point out a positive and negative view of news and blogs by assigning a score. In Ref.24, a sentiment analysis was performed by indicating positive, negative or neutral sentiments with an accuracy of 66%.

Computational linguistics is used in Ref.25to predict the impact of news on public impression towards political candidates. Concentrating on the lexical patterns, an election prediction system (Crystal) was presented in Ref. 26 where they get the result of predicting future election with an accuracy of 81.68% with SVM. Opinion Lexicon-based algorithm and Naïve Bayes algorithm are used in Ref.27 for sentiment analysis of Malaysian ¯nancial news headlines. Sentiment analysis is done in Ref.28by mining code repositories to measure the e®ectiveness of review comments done by the reviewers.

During the recent years, deep learning is also applied for sentiment analysis.

Reference 29 used a deep learning technique for predicting polarities of tweets at both message level and phrase level. Micro-blogging and movie review data sets are used in Ref.30to assess the performance of sentiment analysis using deep learning. In Ref. 31, authors provide an overview of deep learning approaches for sentiment analysis and also suggest some mitigation to address the challenges.

In sentiment portrayal, the all-out methodology speaks to emotional states as a few discrete classes, for example, binary (i.e., positive and negative) or as multiple classi¯- cations, for example, see Ref.32six essential emotions (anger, happiness, fear, sadness, disgust and surprise). The dimensional technique addresses emotional states as con- tinuous numerical characteristics in di®erent estimations, for instance, the valence- arousal (VA) space. By and large, CNN is equipped for separating local information;

however, it may disregard to get long-separation dependency Ref.33. On the other hand, LSTM can address this constraint by successively demonstrating writings across sentences Ref.33. LSTM outperforms better accuracy in analyzing sentiment of long sentences Ref.13Fundamentally, LSTM is utilized for local (regional) information inside sentences and long-separation reliance across sentences can be considered in the ex- pectation procedure.

CNN is used in predicting di®erent aspects of sentiments like subjectivity detection, polarity detection and irony detection of Italian Twitter messages Ref.34. In Ref.35, it has been shown that CNN achieves better accuracy on Twitter data than traditional machine learning techniques like NB or SVM. In Ref.36, word2vec and CNN are used to apply a seven layers architecture model to analyze sentences. This model achieves better performance than some other neural network models like Recursive Neural Network (RNN) and Matrix-Vector (MV RNN). Di®erent algorithms word2vec, CNN, Gated Recurrent unit (GRU) and LSTM are compared for analyzing sentiments of Russian tweets Ref.36. A coupled CNN and RNN architecture has been proposed for analyzing the sentiment of short text. This study presents that the proposed architecture performs better than the individual CNN and RNN in sentiment classi¯cation Ref.37.

Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 134.122.89.123 on 09/22/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(4)

3. Research Methodology

In this section, we brie°y sketched the methodology that we have chosen from our prior work Ref. 1for the analysis. To perform the overall analysis in this study, at

¯rst we collected data from ¯ve di®erent international online news sources and built the data set. Then we preprocessed the data sets and categorized them in di®erent contexts. After inferring the sentiments, we trained our model with machine learning and deep learning techniques. Then we evaluated our model and analyzed the results for comparison. Figure1presents the overall methodology adopted by us.

3.1. Building the data set

We pulled data from ¯ve di®erent international news sources. The di®erent news sources are: Daily Star,^aDhaka Tribune,^bThe New York Times,^cThe New Age^dand The Daily Observer.^eAll of them are globally acknowledged online news media with millions of readers every day. For Daily Star, Dhaka Tribune and The New York Times, we have closely observed their RSS (Rich Site Summary) feed daily for a period of four months. While performing the crawling process, we focused only on the top news headlines. For the crawling process, we use the respective new portal's API to extract the data. The data set is also used in our prior work Ref. 1. Table 1 presents an overview of di®erent news sources data volume.

Fig. 1. Research methodology.

ahttps://www.thedailystar.net/.

bhttps://www.dhakatribune.com/.

chttps://www.nytimes.com/.

dwww.newagebd.net/.

ewww.observerbd.com/.

(5)

3.2. Categorizing the data

After collecting data sets, we categorize the headlines based on di®erent contexts.

We have mainly considered only ¯ve categories such as politics, business, world or international, health, and sports. We have basically crawled the metadata of each headline's URL (Uniform Resource Locator). From the meta of each headline, we have extracted the headlines along with the categories. For this exercise, we have classi¯ed the headlines into the above-mentioned categories. Table2depicts a short de¯nition of each category that we have identi¯ed.

The distribution of the number of news items for each newspaper is shown in Table3.

3.3. Preprocessing the data set

For analyzing our data set, we preprocessed all the headlines. First, we de-capitalized all the words, as a common feature of news headlines is capitalized words. Then we tokenized the headlines and removed the noise words considering the selected POS tag.

Table 2. Categories of news headlines.

Category Description

Politics News related to politics, government, political leader Business Focusing on market and policy news relevant to business World National and International news (other than above) Health Covering news on Health, Health care, Medical science Sports All games, athletics

Table 1. Statistics of data sets.

News source Time length Headlines Ratio (%)

Daily Star Nov, 2018 to Feb, 2019 1345 49.76

Dhaka Tribune Nov, 2018 to Feb, 2019 681 20.13 The New York Times Nov, 2018 to Feb, 2019 390 11.53

The New Age Nov, 2018 to Feb, 2019 694 20.51

The Daily Observer Nov, 2018 to Feb, 2019 273 08.07

Total 3383 100

Table 3. Distribution of news headlines based on context.

Context

Daily Star (%)

Dhaka Tribune (%)

The New York Times (%)

The new Age (%)

The Daily Observer (%)

Politics 41 26 09 77 18

Business 39 19 43 10 24

World 13 19 36 13 18

Health 02 17 10 22

Sports 05 18 02 18

(6)

3.3.1. Tokenization

We have split every headline into multiple text segmentation using spaces and punctuation marks. The process of chopping up a sentence into multiple pieces is called tokenization. However, we make sure that short forms such as \don't", \I'll",

\she'd" will remain as one word. An example of tokenization is shown in Table4.

After the completion of tokenization, all the words together form a bag of words (BoW).

3.3.2. Noise reduction

The news article contains many parts of speech that are irrelevant to detect semantic orientation in our work. We have considered only JJ (Adjective), VB (Verb, base form) and RB (Adverb) Parts of Speech (POS) tag from Penn Treebank annotation Ref.38.

3.4. Inferring sentiment

To identify the actual sentiment, we put sentiment or polarity score associate with every headline. We quantify the polarity of headlines on a scale of 0 as Negative and 1 as Positive. The entire headline data set is often inferred as positive or negative from the polarity score. Table 5shows an example of headlines polarity score as per our consideration.

3.5. Training model

For training our model, we have used seven machine learning algorithms and two deep learning algorithms. Used algorithms are: Naïve Bayes Ref. 6, MNB Ref. 7, Bernoulli Naïve Bayes Ref.8, Logistic Regression Ref.9, SGD Ref.10, Linear SVC Ref.11, Nu SVC Ref.12, CNN Ref.13, and LSTM Ref.14. The data set for training model is available at https://github.com/SshShamma/Comparative-Study-of-Ma- chine-Learning-and-Deep-Learning-Algorithms.

Table 4. Tokenization.

Headlines Tokens

Together we'll ¯nd a way out of this crisis Together, we'll, ¯nd, a, way, out, of, this, crisis

Table 5. Polarity or sentiment score of headlines.

Scale Polarity News headline

1 Positive House democrats prepare to scrutinize DeVos's education department 0 Negative Sanders returns to NY roots, says he can defeat trump Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 134.122.89.123 on 09/22/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(7)

We have trained our corpus data sets with our chosen training models. Figure2 shows the training process of our model.

3.6. Validation

3.6.1. Validation accuracy

The corpus dataset we used to train the models contains 10,664 lines (positive-5332 and negative-5332 lines). We have used k-fold cross-validation where k is considered 5 to test our dataset. The test dataset is used to provide an unprejudiced evaluation of a ¯nal model to ¯t the training dataset. Table6 shows the validation results of di®erent learning models.

3.6.2. Learning curve

To assume the error of our training model, we also check the learning graph. Figure3 presents the learning graph of Naïve Bayes classi¯er.

Fig. 2. Training process of models.

Table 6. Cross validation results of di®erent classi¯ers.

Algorithm Accuracy (%)

Naïve Bayes 75.50

Multinomial Naïve Bayes (MNB) 75.40

Bernoulli Naïve Bayes 82.68

Logistic regression 74.55

Stochastic gradient descent (SGD) 74.86 Linear support vector classi¯er (SVC) 75.90 Nu support vector classi¯er 75.75 Long short-term memory (LSTM) 70.19 Convolutional neural network (CNN) 75.64 Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 134.122.89.123 on 09/22/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(8)

3.6.3. Receiver operating characteristics

To measure the performance of the classi¯er, we have also generated ROC curve.

Figure4illustrates the ROC curve of our validation.

4. Result and Discussion

In this section, we explain the classi¯cation results, the evaluation contexts and the polarity estimation of our research.

4.1. Classi¯cation results

To classify the news headlines as positive or negative, we used nine di®erent classi-

¯cation algorithms. Table7shows the accuracy of di®erent classi¯ers we used. It is

Fig. 3. Learning curve of Naïve Bayes classi¯er.

Fig. 4. Receiver operating characteristics.

(9)

overwhelming to see that some of the widely used algorithms fail to achieve satis- factory performance in this case. Most of the machine learning classi¯ers gave an average accuracy of around 76% while Naïve Bayes and SGD gave an average of 80.57% and 74.55%, respectively. On the other hand, from deep learning classi¯ers, CNN and LSTM have shown the accuracy of 75.64% and 70.19%, respectively. We can state that Bernoulli Naïve Bayes classi¯er gives the best accuracy among all of the classi¯ers.

4.2. Polarity estimation

In this section, we explain the Sentiment analysis results, the evaluation contexts, and the polarity estimation of our research.

4.2.1. Polarity consideration

We present the words polarity in Table 8. Our train model gives the most informative words in trained data sets. In the ¯rst instance, the model gives informative words based on our sentiment ratio score.

Table 7. Accuracy of di®erent classi¯ers.

Algorithm Accuracy (%)

Naïve Bayes 80.57

Multinomial Naïve Bayes (MNB) 76.51

Bernoulli Naïve Bayes 82.68

Logistic regression 76.05

Stochastic gradient descent (SGD) 74.55 Linear support vector classi¯er (SVC) 75.90 Nu support vector classi¯er 75.75 Long short-term memory (LSTM) 68.82 Convolutional neural network (CNN) 70.33

Table 8. Sentiment ratio.

Words Sentiment Sentiment ratio

Engrossing Positive 19.0

Boring Negative 14.8

Inventive Positive 13.7

Refreshing Positive 12.5

Stupid Negative 12.1

Warm Positive 11.7

Wonderful Positive 11.4

Refreshingly Positive 11.4

Dull Negative 11.4

Realistic Positive 11.0

(10)

4.2.2. Polarity of news headlines against news sources

Our research group members read these 3383 news headlines and created ¯les for each of the news headlines which included details on whether the headlines are positive or negative. This data set is then used to outcome the model for classi¯cation and also for evaluation mining tasks. Figure5 shows the polarity of news headlines against di®erent news sources. From this, we can easily summarize that most news headlines of di®erent news sources are presented as negative news.

4.2.3. Polarity of news headlines as per news context

In this section, we present the polarity of news headlines as per news context.

Figure6shows the polarity of news headlines as per news context.

Fig. 5. Polarity of news headlines against news sources.

(a) New York Times (b) Daily Star

Fig. 6. Polarity of news headlines as per news context.

(11)

4.3. Precision, recall and F-measure for positive and negative reviews data sets

As we found the highest accuracy with Naïve Bayes classi¯er among di®erent machine learning and deep learning classi¯ers, we trained our model with Naïve Bayes classi¯er. This time, we run a random shu®le in our trained date sets (positive and negative). This classi¯er gives the accuracy of measuring the trained data set. We collect the reference values and observed values for each label (positive or negative), then use those sets to calculate the precision, recall and F-measure of the Naïve Bayes classi¯er. The resultant confusion matrix is shown in Table9that is often used to describe the performance of a classi¯cation model on a set of test data for which the true values are known. It also allows the visualization of the performance of an algorithm.

. Every positive-trained data in the data sets is correctly identi¯ed with 85% recall.

This means very few false negatives in the positive class.

. As positive data while classifying is only 65% likely to be correct. So, poor precision with 35% leads to false positives for the positive data set.

Table 9. Some metrics from confusion matrix.

Sentiment Precision Recall F-measure

Positive 0.65 0.85 0.78

Negative 0.96 0.58 0.64

(c) Dhaka Tribune (d) The New Age

(e) The Daily Observer

Fig. 6. (Continued )

(12)

. Any trained data identi¯ed as negative is 96% likely to be correct (high precision).

This means very few false positives for the negative data set.

. Hence, many negative data sets are incorrectly classi¯ed. Low recall causes 42%

false negatives for the negative data set.

. F-measure provides no useful information. There is no insight to be gained from having it, and we would not lose any knowledge if it will be taken away.

4.4. Sentiment polarity considering procedure

In this section, the trained model gives sentiment against our news headlines. The instance model gives headline sentiment based on our con¯dential score. A con¯- dential score between 0.1 and 1.0 is considered a negative sentiment, and also for a con¯dential score between 0 and 1.0 is considered positive sentiment. Table 10 shows some of the con¯dential score of news headlines.

4.5. Comparison of machine learning and deep learning

In this study, we have used seven machine learning algorithms and two deep learning algorithms. Among them, BNB and CNN show the better accuracy of 82.68% and 70.33%, respectively, from machine learning and deep learning algorithms. This study shows that we can obtain better accuracy of 82.68% from machine learning algorithms than other deep learning algorithms. From this study, we can state that machine learning algorithms outperform deep learning techniques.

5. Conclusion

Headlines are the in°uential elements of any news. In our prior work, we have worked with only machine learning algorithms. In this paper, we have proposed a technique to identify the neutral newspaper or micro news blogs in terms of semantic

Table 10. Con¯dential score's of headlines.

Headlines Sentiment Con¯dential score

Sanders returns to NY roots, Negative 0.6

Says he can defeat trump

Trump delivers a slashing speech Negative 0.6

That rouses the right

Klobuchar and Kennedy Kid Negative 0.6

Washington Elite at Gridiron

Taskin happy with his recovery so far Positive 1.0

Morata ¯nally scores, Positive 0.8

Atletico keeps pace with Barcelona

Stakes high for England after Grenada washout Negative 1.0 House democrats prepare to scrutinize Positive 1.0 DeVos's education department

(13)

orientation using machine learning and deep learning approach. This study also shows that machine learning algorithms outperform deep learning algorithms. The objective is to ¯nd out the context-based tagging of news headlines to avoid prejudicing the readers. Our work will mostly help the public to make a decision based on reading news headlines at a glance by avoiding misconceptions against any leaders or governance.

In the future, this technique can also be implemented in a tool or a plug-in for identifying the most neutral newspapers or news blogs. Moreover, this work can be extended to analyze the news headings and readers' review comments together to

¯nd out the most authentic news on social media as well as the readers' perception in an essence.

References

1. S. Rahman, S. S. Hossain, S. Islam, M. I. Chowdhury, F. B. Ra¯q and K. B. M.

Badruzzaman, Context-based news headlines analysis using machine learning approach, in Int. Conf. Computational Collective Intelligence (Springer, 2019), pp. 167–178.

2. U. K. H. Ecker, S. Lewandowsky, E. P. Chang and R. Pillai, The e®ects of subtle mis- information in news headlines, J. Exp. Psychol. Appl. 20(4) (2014) 323.

3. S. Feuerriegel, S. F. Heitzmann and D. Neumann, Do investors read too much into news?

how news sentiment causes price formation, in 2015 48th Hawaii Int. Conf. System Sci- ences (IEEE, 2015), pp. 4803–4812.

4. M. Nardo, M. Petracco-Giudici and M. Naltsidis, Walking down wall street with a tablet:

A survey of stock market predictions using the web, J. Econ. Surv. 30(2) (2016) 356–369.

5. M. Alanyali, H. S. Moat and T. Preis, Quantifying the relationship between ¯nancial news and the stock market, Sci. Rep. 3 (2013) 3578.

6. I. Rish et al., An empirical study of the naive bayes classi¯er, in IJCAI 2001 Workshop on Empirical Methods in Arti¯cial Intelligence Vol. 3, 2001, pp. 41–46.

7. J. D. Rennie, L. Shih, J. Teevan and D. R. Karger, Tackling the poor assumptions of naive bayes text classi¯ers, in Proc. 20th Int. Conf. Machine Learning (ICML-03), 2003, pp. 616–623.

8. A. McCallum et al., A comparison of event models for naive bayes text classi¯cation, in AAAI-98 Workshop on Learning for Text Categorization Vol. 752 (Citeseer, 1998), pp. 41–48.

9. T. K. Ho, J. J. Hull and S. N. Srihari, Decision combination in multiple classi¯er systems, IEEE Trans. Pattern Anal. Mach. Intell. 1 (1994) 66–75.

10. L. Bottou, Large-scale machine learning with stochastic gradient descent, in Proc.

COMPSTAT'2010 (Springer, 2010), pp. 177–186.

11. S. R. Gunn et al., Support vector machines for classi¯cation and regression, ISIS Tech.

Rep. 14(1) (1998) 5–16.

12. B. Sch€olkopf, A. J. Smola, R. C. Williamson and P. L. Bartlett, New support vector algorithms, Neural Comput. 12(5) (2000) 1207–1245.

13. D. Li and J. Qian, Text sentiment analysis based on long short-term memory, in 2016 First IEEE Int. Conf. Computer Communication and the Internet (ICCCI) (IEEE, 2016), pp. 471–475.

14. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9(8) (1997) 1735–1780.

(14)

15. W. Medhat, A. Hassan and H. Korashy, Sentiment analysis algorithms and applications:

A survey, Ain Shams Eng. J. 5(4) (2014) 1093–1113.

16. B. Pang, L. Lee and S. Vaithyanathan, Thumbs up?: Sentiment classi¯cation using machine learning techniques, in Proc. ACL-02 Conf. Empirical Methods in Natural Language Processing Vol. 10 (Association for Computational Linguistics, 2002), pp. 79–86.

17. R. Chiong, Z. Fan, Z. Hu, M. T. P. Adam, B. Lutz and D. Neumann, A sentiment analysis-based machine learning approach for ¯nancial market prediction via news dis- closures, in Proc. Genetic and Evolutionary Computation Conf. Companion (ACM, 2018), pp. 278–279.

18. D. M. E.-D. M. Hussein, A survey on sentiment analysis challenges, J. King Saud Univ.- Eng. Sci. 30(4) (2018) 330–338.

19. M. Hu and B. Liu, Mining and summarizing customer reviews, in Proc. Tenth ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (ACM, 2004), pp. 168–177.

20. B. Liu, M. Hu and J. Cheng, Opinion observer: Analyzing and comparing opinions on the web, in Proc. 14th Int. Conf. World Wide Web (ACM, 2005), pp. 342–351.

21. X. Ding, B. Liu and P. S. Yu, A holistic lexicon-based approach to opinion mining, in Proc. 2008 Int. Conf. Web Search and Data Mining (ACM, 2008), pp. 231–240.

22. T. L. Im, P. W. San, C. K. On, R. Alfred and P. Anthony, Analysing market sentiment in

¯nancial news using lexical approach, in 2013 IEEE Conf. Open Systems (ICOS) (IEEE, 2013), pp. 145–149.

23. N. Godbole, M. Srinivasaiah and S. Skiena, Large-scale sentiment analysis for news and blogs, Icwsm 7(21) (2007) 219–222.

24. D. Maynard, K. Bontcheva and D. Rout, Challenges in developing opinion mining tools for social media, in Proc. NLP Can U Tag# usergeneratedcontent, 2012, pp. 15–22.

25. K. Lerman, A. Gilder, M. Dredze and F. Pereira, Reading the markets: Forecasting public opinion of political candidates by news analysis, in Proc. 22nd Int. Conf. Computational Linguistics Vol. 1 (Association for Computational Linguistics, 2008), pp. 473–480.

26. S.-M. Kim and E. Hovy, Crystal: Analyzing predictive opinions on the web, in Proc. 2007 Joint Conf. Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), 2007, pp. 1056–1064.

27. S. M. Shuhidan, S. R. Hamidi, S. Kazemian, S. M. Shuhidan and M. A. Ismail, Sentiment analysis for ¯nancial news headlines using machine learning algorithm, in Int. Conf.

Kansei Engineering & Emotion Research (Springer, 2018), pp. 64–72.

28. S. S. Hossain, Y. Arafat, M. E. Hossain, M. S. Arman and A. Islam, Measuring the e®ectiveness of software code review comments, in Int. Conf. Advances in Computing and Data Sciences (Springer, 2020), pp. 247–257.

29. A. Severyn and A. Moschitti, Twitter sentiment analysis with deep convolutional neural networks, in Proc. 38th Int. ACM SIGIR Conf. Research and Development in Informa- tion Retrieval (ACM, 2015), pp. 959–962.

30. O. Araque, I. Corcuera-Platas, J. F. Sanchez-Rada and C. A. Iglesias, Enhancing deep learning sentiment analysis with ensemble techniques in social applications, Exp. Syst.

Appl. 77 (2017) 236–246.

31. D. Tang, B. Qin and T. Liu, Deep learning for sentiment analysis: Successful approaches and future challenges, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 5(6) (2015) 292–303.

32. P. Ekman, An argument for basic emotions, Cogn. Emotion 6(3–4) (1992) 169–200.

33. J. Wang, L.-C. Yu, K. R. Lai and X. Zhang, Dimensional sentiment analysis using a regional cnn-lstm model, in Proc. 54th Annual Meeting of the Association for Compu- tational Linguistics Vol. 2, Short Papers, 2016, pp. 225–230.

(15)

34. X. Ouyang, P. Zhou, C. H. Li and L. Liu, Sentiment analysis using convolutional neural network, in 2015 IEEE Int. Conf. Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing;

Pervasive Intelligence and Computing (IEEE, 2015), pp. 2359–2364.

35. S. Liao, J. Wang, R. Yu, K. Sato and Z. Cheng, CNN for situations understanding based on sentiment analysis of twitter data, Procedia Comput. Sci. 111 (2017) 376–381.

36. J. Tro¯movich, Comparison of neural network architectures for sentiment analysis of Russian tweets, in Proc. Comput. Linguistics Intellectual Technology, Int. Conf. Dialogue (RGGU), 2016, pp. 1–10.

37. N. Chen and P. Wang, Advanced combined lstm-cnn model for twitter sentiment analysis, in 2018 5th IEEE Int. Conf. Cloud Computing and Intelligence Systems (CCIS) (IEEE, 2018), pp. 684–687.

38. A. Taylor, M. Marcus and B. Santorini, The penn treebank: An overview, in Treebanks (Springer, 2003), pp. 5–22.