Analysis of Triplet Extraction

This analysis is based on the NLP component of the proposed system as described in Section 3.6. The principal objective of this analysis is to identify for what extent triplet extraction of the system performs well. Triplet is considered an actor, action and object which are taken as the input of reasoning.

6.4.1 Analysis of actor

The Advanced filtering method discussed in Section 3.6.5, the system identifies a valid actor word should be a noun which is extracted based on Stanford Dependency Parser [56]. From the sample data set identification of actor as a noun was 89% and 11% was identified as not a noun.

The nature of an actor word being either the noun or the noun phrase depends upon the context of the software product as mentioned in actor filtering topic under Section 3.6.5.

According to the data set, valid actors for the system can be calculated by the following equation.

Valid Actor for the system = 𝑵𝒐.𝒐𝒇 “𝒄𝒐𝒓𝒓𝒆𝒄𝒕𝒍𝒚 𝒂𝒅𝒅𝒓𝒆𝒔𝒆𝒅” 𝒂𝒄𝒕𝒐𝒓𝒔

𝑵𝒐.𝒐𝒇 𝒂𝒍𝒍 𝒂𝒄𝒕𝒐𝒓𝒔 ×

𝟏𝟎𝟎%

No. of “correctly addressed” actors: Actor words that do not affect the functionality of software product and identified from the noun of noun phrase in user story.

No. of all actors: Actor words that identified as a noun from the user story dataset

According to the results, 65% actor words have been observed as a valid input to the proposed solution. However, 35% of actor words have affected the functionality of the software. This situation will affect the generation of test cases because the extracted actor word may or may not exist in the Ontology.

Chapter 6. Results and Evaluation 86

6.4.2 Analysis of action extraction

According to the comparison between extraction of action from the implemented NLP component and manual identification of action from user stories, the accuracy of the correctly extracted actions can be identified as follows.

From the comparison of actual action of user story with implemented NLP component results,

Action Accuracy =

𝑵𝒐. 𝒐𝒇 𝒂𝒍𝒍 𝒂𝒄𝒕𝒊𝒐𝒏𝒔 ×

𝟏𝟎𝟎% = 88%

No. of Correct actions: Total number of actions correctly extracted by NLP component as obtained in user stories which defined the functionality of the requirement

No. of all actions: Total number of actions obtained in user stories which defined the functionality of the requirement.

The system captures the action words as the root node of the given sentence as discussed in Section 3.6.3. There are words having multiple expressions with different meanings that cause ambiguity. Actual action word should be a verb but there are cases where actual action word has more than one meaning with different POS tags. In such cases the whole sentence becomes ambiguity and the dependency parser does not identify the word that system needs as the root relation. Therefore such scenarios that makes a sentence ambiguity has not taken care through this method.

Examples: As an admin, I want to view users of the system.

Here the substring of “view users of the system” is ambiguous because of the word

‘view’. The word ‘view’ can be taken as either noun or a verb and it depends on the sentence meaning that want. Though the word ‘view’ should be the action word of this sentence, because of the ambiguities it has not taken care through the system.

6.4.3 Analysis of object extraction

According to the comparison between extraction of object from the implemented NLP component and manual identification of object from user stories, the accuracy for the correctly extracted object can be identified as follows.

Chapter 6. Results and Evaluation 87

From the comparison of actual object of user story with implemented NLP component results,

Object Accuracy =

𝑵𝒐. 𝒐𝒇 𝒂𝒍𝒍 𝒐𝒃𝒋𝒆𝒄𝒕𝒔 ×

𝟏𝟎𝟎% = 81%

No. of Correct objects: Total number of object words correctly extracted by NLP component which are directly accessed by extracted action word in user stories.

No. of all objects: Total number of object words which are directly accessed by extracted action in user stories.

As described under object filtering topic in section 3.6.5 it is observed that the accuracy of identification of an actual object of a sentence is directly based on the action word identified in that sentence. In this proposed methodology for object extraction, it only finds for the direct object relation or the dependent relation. Here apart from the ‘dobj’

relation, ‘dep’ relation was used to handle ambiguities between the action word identified and its noun object to some extent. However, still there are cases where more than one dependent words are connected to the action word as objects. Among all these object words the system is not in a possible state to find the exact object word that related to the action word.

Also when the action word is preceded by a preposition word attached to the actual object word then the ‘dobj’ relation is not further identified by the dependency parser and it provides another relation like ‘nmod:for’ which is related to preposition ‘for’ and that noun object ‘order’ word as below example Figure 6.3.

Figure 6.3. Typed dependencies extracted for sentence “pay for my order”

Therefore such cases related with prepositions are not taken care with this proposed methodology of entity extraction.

Chapter 6. Results and Evaluation 88