The Evaluation of TSTS usage with Conversational Agent Log Files

The aim of the evaluation is to find how well TSTS performs with Thai sentence pairs. This following section will provide detail of the experimental methodology, results and discussion.

167

8.4.1

Methodology

To achieve the aim of the evaluation of the TSTS, 15 specific sentence pairs are selected. The 15 sentence pairs can be separated into three groups: High similarity group, Medium similarity group, and Low similarity group. These 15 sentence pairs are chosen from English Conversational Agent debt adviser log files. These log files come from a real-life system. The semantic similarity rating calculated from TSTS is also separated into three groups as follows:

• High similarity group (rating between 0.750-1.00)

• Medium similarity group (rating between 0.25-0.749)

• Low similarity group (rating between 0.00-0.249).

The Medium similarity group has a bigger range because the Medium similarity group contains both Medium-High similarity and Medium-Low similarity groups.

The results are shown in Section 8.4.3.

8.4.2

Materials

The chosen 15 sentence pairs are shown in Table 8.3. These 15 sentence pairs are translated into Thai by a native Thai speaker who has a BA in Language and Culture. Column SP is the sentence pair number. Column Prediction is the prediction similarity group of sentence pairs (based on my personal judgement).

Table 8.3: The Chosen 15 Sentence Pairs

SP S1 S2 Prediction

1 ฉันสับสน ฉันยังสับสนอยู่ High

I am confused I am still confused

2 ฉันไม่เข ้าใจ คุณไม่เข ้าใจ High

I cannot understand this You do not understand

3 ฉันเป็นคนติดการพนัน ฉันเป็นคนติดพนัน High

I am a gambling addicy I have a gambling addiction

4 ฉันต ้องการเงิน ฉันต ้องการเงินสด High

I want money I need cash

5 ก็ได ้ฉันจะจ่ายสามส่วน ฉันจ่ายได ้แค่สามส่วน High All right I can pay third I can only pay third

6 ฉันยังต ้องจ่ายหนี6อยู่ไหม ฉันไม่มีปัญหาหนี6สิน Medium Do I still have to pay my debt I do not have a debt problem

7 ฉันสับสนกับตัวเลือกพวกนี6ไปหมดแล ้ว ฉันรู ้สึกว่าเหตุผลพวกนี6สับสน Medium I am confused by all these choices I find all the reasons confusing

8 ไม่สามารถจ่ายเงินได ้ เงินค่าที%อยู่อาศัย Medium

168

9 นิยามการวัดผลให ้ฉันที ฉันยังไม่ได ้รับการวัดผลเลย Medium Define assessment for me I did not get my assessment yet

10 ฉันยังไม่ได ้เงิน ฉันต ้องการเงิน Medium

I did not get the money I want money

11 ฉันไม่อยากบอกชื%อกับคุณ ฉันยื%นเรื%องขอรับเงินไป Low I do not want to tell you my name I applied for the money

12 ฉันซื6อรถมา ไม่สามารถจ่ายเงินได ้ Low

I bought a car Cannot afford payment

13 ฉันต ้องจ่ายที%ไหน ฉันเป็นคนติดพนัน Low

Where do I pay I have a gambling addiction

14 ฉันต ้องการความช่วยเหลือ ฉันไม่แน่ใจว่าควรทําอย่างไร Low I need your help I am not sure what I should do

15 ก็ได ้ฉันจะจ่ายสามส่วน คุณไม่เข ้าใจ Low

All right I can pay third You do not understand

8.4.3

Results

The experiment result is shown in Table 8.4. Column SP is the sentence pair number. Column TSTS is the machine rating for the Thai sentence pairs using the algorithm described in Section 8.2. Column Group is the similarity group of TSTS rating. Column

Prediction is the prediction similarity group of sentence pair from Table 8.3. Column Result is the result of the prediction for each sentence pair.

Table 8.4: The Results of the Chosen 15 Sentence Pairs

SP TSTS Group Prediction Result

1 0.922 High High Correct

2 0.894 High High Correct

3 0.908 High High Correct

4 0.953 High High Correct

5 0.843 High High Correct

6 0.527 Medium Medium Correct

7 0.612 Medium Medium Correct

8 0.224 Low Medium Wrong

9 0.467 Medium Medium Correct

10 0.726 Medium Medium Correct

11 0.186 Low Low Correct

12 0.309 Medium Low Wrong

13 0.226 Low Low Correct

14 0.324 Medium Low Wrong

169

8.4.4

Discussion

According to the results shown in Table 8.4, TSTS predicts the High similarity group correctly. For the Medium similarity Group, TSTS predicts 4 from 5 Medium sentence pairs accurately, while the Low similarity Group TSTS predicts 3 from 5 Low sentence pairs accurately. Thus, TSTS predicts 12 from 15 sentence pairs accurately; i.e. 80%. The Spearman rank correlation between human prediction and TSTS prediction is:

• Spearman’s ρ = 0.864 (P-Value < 0.01)

One of the wrongly predicted sentence pairs from Table 8.4 is sentence pair 8; this sentence pair is meant to be in the Medium similarity group. However, TSTS produced the rating for sentence pair 8 as 0.224, which is in the Low similarity group. This happens because after translating into Thai, the word ‘payment’ in the sentence ‘Cannot afford payment’ is translated into word ‘เงิน’ (Money). Therefore, the word ‘payment’ in the sentence ‘My accommodation payment’ is translated into ‘เงินค่า’ (Payment). This makes the meanings in Thai and English more different because the word ‘เงินค่า’ (Payment) in Thai can only be used in some specific content, while the word ‘เงิน’ (Money) is generally used .

A Conversational Agent has the capacity to go back, correct and disambiguate use of any incorrect sense misunderstandings through dialogue, which is why 80% accuracy is approaching workabl,e whereas it would be too low for applications such as information retrieval. Therefore, this supports the view that TSTS could be used to create the Thai semantic-based Conversational Agents.

8.5

Conclusion

The aim of the research is to propose a Thai sentence semantic similarity measure (TSTS). This chapter described how the first sentence semantic similarity measures works and also discussed the experiment result and how correlations coefficient of 0.809 (P-Value < 0.01) was obtained. This measure performs better than STASIS when used with the Thai sentence pairs and should be useful in Thai semantic similarity. TSTS is considered to be a starting point for a Thai sentence measure which can be used to create semantic-based Conversational Agents in future. However, there are a number of aspects concerning the algorithm which can still be improved, discussed in Chapter 9.

170

Chapter 9