Stochastic Local Search Algorithm

Searching for an optimal feature subset is an NP-hard problem. The stochastic local search (SLS) approach has proven to be highly competitive for solving a range of hard computational problems including satisfiability of propositional logic formulas [76] as well as computing the most probable explanation [77] and the maximum a posteriori hypothesis [78] in Bayesian networks.

Our proposed SLS algorithm is described in Algorithm 3, where c indicates an employed machine learning classifier. F denotes the feature candidate set. Stserves as a tabu list that records all the visited states so that the algorithm does not consider a state repeatedly. The parameters θu and θcotare inputs to the function INITIALIZATION(θu, θcot) and re- spectively control how many percentage of unigram candidates in Fuand CoT candidates in Fcotthat are included in an initial state by random selection. The parameter λ is input to the function NEXTSTEP(λ), which decides the next step to be a noise step with probabil-

ity of λ or a greedy step with probability 1− λ. The NEIGHBOR(s, St) function returns a random neighbor state of s that is not recorded in the tabu list St.7The parameter κ limits how many neighbor states are evaluated in a greedy step. The parameter R defines how many steps are performed in each try, and MAX-TRIES defines the maximum number of tries before termination performed by the algorithm.

Algorithm 3 works in the following way. In each try, an initial state is randomly created. The search begins with this initial state. Then the algorithm goes through an iterative hill- climbing process in R steps. Each step of the process is either a noise step or a greedy step. If it is a noise step, the search moves to a neighbor state. If it is greedy step, the search test κ neighbors of the current state and goes to a neighbor state with maximum score. In each step, if the score p in that step is not worse than the recorded score p∗, then p∗ is updated with p and the recorded state s∗is updated with s. After algorithm finishes with R steps in MAX-TRIES tries, the recorded state s∗is returned. In this way, the correspond feature subset Fs∗is optimized for the classifier c.

CHAPTER 4. RESEARCH RESULTS AND EVALUATION 49

4.3.3

Evaluation

In this subsection, we present experiments to evaluate the performance of the proposed EFS framework. In the evaluation we compare our proposed approach with existing meth- ods on two standard datasets, i.e., the datasets DM1400 and DM2000, which are from the movie review domain and were originally introduced by Pang et al. in 2002 [10] and 2004 [12] respectively.

Metrics

To evaluate the performance of sentiment classifiers, we adopt the evaluation metric ac-

curacy which is the percentage of correctly labeled reviews out of total reviews and is

generally used in most of the previous work:

accuracy =#correctly labeled reviews_{#total reviews} .

Performance Comparison

Table 4.3 chronologically summarizes results on the two standard datasets reported in recent ten years. The column “Ex-Efforts" indicates whether the related work uses extra human efforts as part of their proposed methods. Performance above 90% on each dataset is bolded. Best performance on each dataset is underlined. From the Table 4.3, we can see that our approach achieves the best performance (90.13% accuracy) on DM1400. Among the six methods that achieve classification performance above 90% on DM2000, most of them require extra human efforts, e.g., manually built lexicons [9], predefined extraction patterns [80, 92], and preselected feature categories [84], as inputs to their methods. In comparison, our proposed EFS framework is a fully automatic process. Although Bai’s method [82] currently is best on DM2000, our approach shows comparably good (92.70% v.s. 92.37%) on the same dataset. In addition, our approach beats Bai’s method [82] on the DM1400 with more than 11% accuracy, which suggests our approach is more robust than Bai’s approach [82]. Therefore, through performance comparison on two standard datasets, we conclude that our proposed EFS framework is generally superior to existing state-of-the-art approaches in that our approach is high accuracy, more robust, and needs small human efforts.

4.3.4

Summary

The paper P3 is a further development on the contribution of the paper P2. In the paper P3, we propose an Effective Feature Search (EFS) framework to enhance the performance of sentiment classifiers. The proposed EFS framework takes advantages of CoT patterns and

50 CHAPTER 4. RESEARCH RESULTS AND EVALUATION Table 4.3: Performance Comparison

Dataset Work Year Ex-Efforts Accuracy

DM1400 Pang et al. [10] 2002 No 82.90%

DM1400 Mullen&Collier [79] 2004 Yes 86.00% DM1400 Riloff et al. [80] 2006 Yes 82.70%

DM1400 Zhai et al. [81] 2010 No 84.30%

DM1400 Bai [82] 2011 No 78.08%

DM1400 our EFS framework 2012 No 90.13% DM2000 Pang&Lee [12] 2004 No 87.20% DM2000 Whitelaw et al. [9] 2005 Yes 90.20% DM2000 Kennedy&Inkpen [83] 2006 Yes 86.20% DM2000 König&Brill [14] 2006 Yes 91.00% DM2000 Zaidan et al. [13] 2007 Yes 92.20% DM2000 Abbasi et al. [84] 2008 Yes 91.70% DM2000 Martineau&Finin [85] 2009 No 88.10% DM2000 O’Keefe&Koprinska [86] 2009 No 87.15% DM2000 Taboada et al. [7] 2011 Yes 76.63% DM2000 Pak&Paroubek [87] 2011 Yes 85.10% DM2000 Saleh et al. [88] 2011 No 86.19% DM2000 Heerschop et al. [89] 2011 Yes 81.00% DM2000 Mejova et al. [90] 2011 Yes 87.50%

DM2000 Maas et al. [91] 2011 No 88.90%

DM2000 Abbasi et al. [92] 2011 Yes 89.65%

DM2000 Bai [82] 2011 No 92.70%

DM2000 our EFS framework 2012 No 92.37%

search for effective features in an SLS process. Performance comparison on two standard datasets shows that our proposed EFS framework is comparatively superior to existing state-of-the-art approaches in that our EFS framework is highly accurate, robust, and needs small human efforts. The result of this paper answers questions of RQ3 discussed in the Section 1.4.

My contribution: I was the first author of the paper P3 and did the implementation of the work, analysis of the results, and writing the paper. Ole J. Mengshoel gave suggestions and feedback on the development of the SLS model and the algorithm. Jon Atle Gulla on result analysis and writing on improving the paper.

CHAPTER 4. RESEARCH RESULTS AND EVALUATION 51