The introduction chapter explains about the ARIA-VALUSPA project which develops a framework of virtual humans. One work package that is being developed at the University of Twente is called Multi-Modal Dialogue Management for Information Retrieval. The work package is conducted for a virtual human called Alice, who is a representation of the character Alice in the Alice in Wonderland story. One challenge in developing multi- modal dialogue management for information retrieval is preparing the domain knowledge for the virtual human. Question generation is chosen as an approach to create the domain knowledge of Alice.
The concepts of conversational agents and virtual humans are also explained, including dialogue management which is a module in conversational agents or virtual humans that is responsible to manage the dialogue between Alice and its users. Next, the dialogue manager of Alice is explained together with the domain knowledge that is managed by a tool called QAMatcher. The QAMatcher works by matching a user’s question with existing questions by using text processing algorithms. The existing questions in the QAMatcher will be generated by the chosen approach, question generation. The topic of the questions that will be generated focuses on Alice in Wonderland related story.
Question generation is a subject in natural language processing that intends to generate questions from text. Question generation is usually used for helping teachers to make ques- tions for their students. However, recent research show that question generation can also be used for other purposes such as internet security domain and virtual humans. There are two main approaches in conducting question generation research, they are the syntactic ap- proach and the semantic approach. Syntactic approach usually explores the use of syntactic
tools such as Stanford Parser and Tregex while the semantic approach explores the semantic tools such as Stanford Dependency and Semantic Role Labels (SRL). The approach that is chosen for the Alice Question Generation system is the semantic approach.
Alice Question Generation (AQG) is a question generation system that is developed to generate question and answer pairs about Alice in Wonderland. AQG uses SRL as the main task to retrieve the semantic meaning of Alice in Wonderland story. SENNA is the SRL tool that is used to retrieve the SRL. Beside SRL, AQG also uses Stanford Dependency to retrieve the semantic meaning of Alice in Wonderland story. PyStanfordDependencies is the Stanford Dependency tool that is used for the AQG system.
The first phase in building the AQG system is observing the SRL patterns based on the frequency of the pattern occurrences and the consistency of the semantic information conveyed by the pattern across different sentences. Two summaries of Alice in Wonderland from GradeSaver and SparkNotes are used as the training data. Based on the observation, the pattern that consist of 2 or more Arguments, 0 or more Modifiers, and 1 verb, is chosen to be included in the AQG templates. The modifiers that are chosen are adverbials, manners, locatives, and temporals. Next, the dependency labels are observed for sentences that have conjunctions, because sentences with conjunctions are most likely separated into different clauses by SENNA and can lose a complete information from the sentence.
The second phase in building the AQG system is creating the templates. The template creation focuses on the events (actions, happenings) and existents (characters, settings). The questions in the templates ask about the subject, the predicate, and the object of the events and existents. In the initial version of the AQG, there are 25 templates of question and answer pairs that fall under 6 categories. Next, initial evaluation of the templates is conducted by the author. The templates that create too many errors are removed from the AQG system, and other templates are improved. After initial evaluation, there are 19 templates under 6 categories that are included in the system.
The AQG system is next evaluated by 6 annotators by using a 5-scale rating system. The test data is a summary of Alice in Wonderland from Litchart that consists of 69 sentences. The annotators are divided into two groups. The first group consists of 3 annotators and 35 sentences, while the second group consists of 3 annotators and 34 sentences. The first group evaluate 137 question and answer pairs from the 35 sentences, while the second group
evaluate 131 pairs from the 34 sentences. The average score from both groups 3.495 out of 5. A last improvement on the template is conducted before the next evaluation with the QAMatcher.
The QAMatcher is first set up by a follow-up question strategy in order the keep the evaluators to ask questions about Alice in Wonderland only. A pilot evaluation is first conducted and the follow-up strategy is improved. Next, the user study is conducted with 4 evaluators. The evaluators are given about 15 to 20 minutes time to ask about Alice in Wonderland as they want to know more about the story. The result from this user study is that there were more irrelevant answers than relevant ones that were given by the QAMatcher. The current generated QA pairs from the AQG system cannot be used by themselves for the QAMatcher. More varied templates that ask about the same thing are necessary to be created in the future work. The follow-up question strategy and the history of the dialogue between the user and the QAMatcher are important implementation for the current user study and for other purposes in the future work.