Figure 5.7: Google Form for collecting queries
For each of the query sources used, the aim was to generate realistic learner queries, so queries where the user wanted to learn about a technique, for example: “How does cluster analysis work” were used. Other generic or career-related queries such as “What is it like to be a data scientist at Amazon?”, or queries that were out of scope such as “is there any course on ML?” were not included. There were 25 queries generated from learners and 60 queries generated from online sources, resulting in 85 queries. From this set, 15 of the queries were randomly chosen for param- eter tuning while 70 queries were left for the user evaluation. Each query had an Identifier (ID) associated with it. A random number generator was used to generate the IDs of the queries used for parameter tuning.
5.3
Aspects of the Query Refinement Method
In this section we examine two key aspects of the CONCEPTBASED-QR method. First, the vocab- ulary used for refinement. Second, the parameters used within the CONCEPTBASED-QR method. Some queries generated by learners can be high level and the concept labels are also high level. So we investigate the vocabulary that is used for query refinement, to determine if using only concept labels are sufficient or if using a bigger vocabulary would be better. We wish to determine how specific the vocabulary used for query refinement should be. We compare when only the concept
5.3. Aspects of the Query Refinement Method 80
labels are used for refinement with when the descriptions together with the concept labels are used. Using the concept labels only is denoted by Labels only, while using the descriptions and labels is denoted by Descriptions + Labels. These notations are used to refer to the methods in the following sections.
We expect suitable parameters that will allow us to refine queries effectively and enable us to find relevant documents. Two parameters of the query refinement method are: the number of domain concepts selected and the number of highly weighted terms to select from the potential refined query. The first parameter examined is the number of concepts to select for refinement. It is important to choose a suitable number of concepts for query refinement to avoid deviating from a learner’s goal. The number of concepts is needed after an initial query from a learner has been compared with all the domain concepts. The second parameter is the number of highly weighted terms to select from the potential refined query. Selecting a suitable number of terms is necessary in order to avoid using noisy or irrelevant terms for refinement. The selected terms with highest weights would be added to an initial query to generate a refined query.
5.3.1 Experimental Design
The e-Learning recommender presented in Chapter 5 is used for the following experiments. The dataset accessed by the recommender contains 504 chapters of Machine Learning and Data Mining (ML/DM) e-Books. A description of this dataset was presented in §5.1.2. The experiments are evaluated on a collection of 15 learner queries randomly chosen for initial experiments, by using a random number generator.
A document retrieval task is used to evaluate the relevance of the top 3 documents retrieved for each query. The top 3 documents are selected because from our previous document retrieval exper- iments, we found that earlier retrievals are more likely to be relevant, hence we focus on the first 3 retrievals in this task. The relevance of each retrieved document is evaluated by the researcher using a 5-star rating mechanism as recommended in (Weijters, Cabooter & Schillewaert 2010). Where, 1-star is poor and 5-stars are very good. Experiments are run using the CONCEPTBASED- QR method with the number of concepts set at 1, 3, 5, 7 and 10. These are denoted as CB1 to CB10. For example at CB1 for Labels only, the label of the most similar concept to the query is added to the initial query for refinement. For Descriptions + Labels at CB1 the most similar concept containing its pseudo-document and label is retrieved and used for query refinement as described in §5.2.1.
5.3. Aspects of the Query Refinement Method 81
5.3.2 Results and Discussion
Figure 5.8 contains the results of a comparison of the part of the domain concepts to use for refinement. The x-axis shows the number of concepts, while the y-axis contains the average ratings for the top 3 most relevant documents retrieved for each variant of the method. The shaded bars in Figure 5.8 show when Labels only are used for refinement. For using Labels only, there is an increase in the average rating from CB1 to CB3, when two more concept labels are added to an initial query. However, from CB3 to CB10 there is a decrease in the average rating as the number of concepts increase, as CB10 < CB7 < CB5 < CB3. The best performance for using Labels only is at CB3, when the three most relevant concept labels are added to an initial query for refinement.
Figure 5.8: Comparison of the vocabulary to use for refinement
The solid dark bar in Figure 5.8 represents when Descriptions + Labels are used. For De- scriptions + Labels, it is observed that using only 1 concept is not very helpful for generating useful terms to refine the query with. The terms from a single concept may be too limited to in- fluence the retrieval of relevant documents. Better performance is observed from CB3 when the descriptions and labels from 3 concepts are used for refinement. So, using up to 3 concepts pro- vides better coverage of the domain. The performance of CB5 reduces compared to that of CB3, but using the terms from 5 concepts is still better than using the terms from only 1 concept for refinement of a query. The performance of this method continues to fall with higher number of concepts, as the performance of CB7 < CB5, and CB10 < CB7. The performance of CB10 even falls below that of CB1. Using the terms from many concepts such as 10 concepts for refinement