Sentiment Analysis with Active Learning Workflow

In this section we present a workflow in the ClowdFlows platform, which performs Twitter sentiment analysis with active learning. The workflow is available at: http://clowdflows. org/workflow/1422/ and is presented in Figure 6.3.

As can be seen from Figure 6.3, the workflow is similar to the one in Figure 6.2 with the only difference being that instead of the “Tweet Sentiment Analysis” widget there is the “Active Learning Tweet Sentiment Analysis” widget. The name of the new widget indicates that it performs sentiment analysis with the enhancement of the active learning approach. The workflow collects Twitter messages based on a user specified query, filters them by language, applies selected preprocessing techniques, performs sentiment analysis, and presents the results in form of individual tweets, aggregated sentiment graphs, and word clouds of positive and negative tweets. Additionally, as a part of the active learning process, the user has the possibility to manually label a selection of tweets given by the algorithm in order to improve the performance of the sentiment classifier. The labeling interface is shown in Figure 6.4. Tweets are initially labeled as neutral, while the user can manually label them also as positive or negative.

The workflow from Figure 6.3 is a result of the study presented in Kranjc et al. (2014). It can be observed that there is no widget for employing the neutral zone, but it can be easily added and applied, as described in Section 6.1.1.

As a result of new experiments and results in the dissertation, we have implemented the second version of sentiment analysis and active learning widget in ClowdFlows (Active Learning Tweet Sentiment Analysis v2), which is different than the one described by Kranjc et al. (2014). Regarding the sentiment analysis in the new version of the active learning widget, the choice of the training dataset, standard and Twitter-specific preprocessing setting, and the algorithm for implementing the Twitter sentiment classifier are based on the discussions and experimental results presented in Chapter 3 of this dissertation. On the other hand, the implementation of the active learning approach is based on the experimental results in Section 5.3. More details are provided below.

The approach to sentiment analysis and active learning was developed as a Web service and imported in ClowdFlows. The service offers operations for classifying a collection of

6.1. Implementations in the ClowdFlows Platform 65

Figure 6.4: The labeling interface in the ClowdFlows platform as a part of active learning. Tweets are initially labeled as neutral, while the user can manually label them also as positive or negative. Twitter usernames are blurred in order to hide personal information.

tweets, providing a set of tweets for hand-labeling as a part of the active learning pro- cess, and updating the sentiment classifier with obtained hand-labeled tweets. It supports multiple workflows and builds a separate sentiment model for each of them. The initial sentiment model is the same for all workflows, but differences occur later, when active learning is applied. Namely, the updated models for workflows are different since every workflow obtains a distinct collection of additional hand-labeled training tweets. Differ- ences between the hand-labeled collections may arise as a consequence of different queries for collecting the tweets, or, if the queries are the same, a different amount of hand-labeled tweets or different hand-labels of the same tweets in the collections. In order to distinguish between the different workflows, they are given unique identifiers.

The default query strategy in the active learning workflow is the one that performed best in the experiments in Section 5.3, i.e., the strategy that divides tweets from the data stream into batches of 100 tweets out of which 10 tweets from each batch are selected for hand-labeling using the active combination querying strategy (combination of 80% “Clos- est to the neutral zone” query strategy with the 20% random strategy). Therefore, out of 10 tweets for hand-labeling the algorithm selects 8 tweets whose classification reliability is closest to the reliability threshold and puts them into the pool of query tweets, so that the top-most are the ones which are closest to the reliability threshold. The value for the reliability threshold is set to 0.1. The other two tweets for hand-labeling are chosen randomly from the batch and put into a separate pool of random tweets. With time, as new tweets arrive, the pools are updated. Whenever the user decides to conduct manual labeling of the tweets, she is presented with a set of tweets to label, which contains 8 that are closest to the reliability threshold and 2 random ones from the pool of random tweets. The hand-labeled tweets are placed in the pool of labeled tweets. Once a day, using the

66 Chapter 6. Implementations and Applications

Figure 6.5: Default parameters for the active learning widget in ClowdFlows, which can be changed by the user.

initial set of training data and manually labeled positive and negative tweets from the pool of labeled tweets, the sentiment models are retrained in every workflow which obtained new hand-labeled tweets.

Besides using the default setting for the active learning query strategy, we enabled the user also with the possibility to change the following parameters:

• batch size,

• number of tweets for hand-labeling chosen with the “Closest to the neutral zone” query strategy, and

• number of tweets for hand-labeling chosen with the random strategy.

Figure 6.5 presents a screenshot of a window obtained with double-click on the newest version of the active learning widget. There one can observe several default active learning parameters, which can be changed.

We applied the relative neutral zone from Section 4.1.1.2 in order to calculate classi- fication reliability and to detect the neutral tweets. In order to adjust the neutral zone, i.e., the average distances of positive and negative training examples, to the changes in the data stream, the formula for calculating adaptive average distances of positive and negative training examples, given in Equation 5.1, is applied every time after the sen- timent model is updated. We set α parameter from the equation to 0.1, meaning that new manually labeled tweets for a specific workflow have influence of 10% on the average distances.