Abstract
The goal of this work is to improve upon existing help in tutoring systems. More specifically, “buggy” messages are improved and compared against existing hints. The machine learning algorithm described in detail in the previous chapter is applied to ASSISTment data and buggy messages are then manually created by the incorrect processes learned from this algorithm. These messages are then run in a randomized control trial in the ASSISTments system to see if they increase student performance compared to just hints.
Introduction
To address several issues with hints mentioned in chapter three, different Buggy Messages are used in this work to provide assistance to students who give a predicted incorrect answer. “Buggy” messages are when a message appears on the screen after a student enters a predicted wrong answer. Messages will only appear if the answer given by the student matches a predicted wrong answer. An example of a problem that has a hint in ASSISTments is shown in Figure 4.1. An example of the same problem that has a buggy message is shown in Figure 4.2. Buggy messages are supported but rarely used in ASSISTments because it takes too much time to predict and enter all the incorrect messages for all possible common wrong answers. The machine learning algorithm in this work can identify most wrong answers and exactly how a student derived them by taking advantage of the existing infrastructure and data in
Figure 4.1: Buggy message example
Figure 4.2: Hint example
Since an introduction was already provided in chapter three, only a brief version is mentioned in the following paragraphs. Buggy messages were first introduced by Brown and Burton (Brown
& Burton., 1978) in 1978 and the early 1980’s with their series of “BUGGY” programs. Work
with these programs was done in (Brown & Burton. 1978; Brown & VanLehn. 1980; Burton. 1982). Their main goal was to build a database of incorrect answers and the process of how those incorrect answers were generated. VanLehn’s early work on bugs has been very influential in the intelligent tutoring field. Others in the AIED community (Jarvis et al., 2004; Matsuda et
al., 2011) have looked at machine learning from examples the bug message. Classical Model
tracing requires authors to write rules.
Jarvis, Jones & Heffernan report on a technique to try to infer these bug rules from small numbers of instance. Heffernan eventually gave up on that line of research, finding the task of
inferring bug rules to be too difficult, but Masuda et al (Matsuda et al., 2007; Matsuda et al., 2011) have push this field further, and reports more success on this challenging task of inferring the cognitive models rules that explain errors..
Systems like Webwork or StudyIsland appear to generate problems from some sort of
algorithmic process. Therefore, our contribution could help any system that has a template like way of generating answers. We image that this algorithm’s results can then be presented to teachers who can then write bug message for that function.
Randomized Control Trial Design
The purpose of this study is to see if specifically targeting a student’s incorrect answer and providing them with immediate specialized help in the form of buggy messages will result in an increased learning rate. To evaluate the effectiveness of the buggy messages a randomized control trial was run on the ASSISTment’s intelligent tutoring system. In ASSISTments two problem sets were chosen to run the randomized control trial on, “Order of Operation” and “Solving Equations”. Both problem sets are typically taught in 7th
grade (student ages 11-12). An example problem for “Order of Operations” with hints is shown in Figure 4.3andwith buggy messages in Figure 4.4. An example problem for “Solving Equations” is shown with hints in Figure 4.5 and with buggy messages in Figure 4.6. A complete list of all the problems in both problem sets can be found at 1 for “Order of Operation”, and 2 for “Solving Equations”. Each of these problem sets were generated by two templates each. 25 instances of each template were generated for a total of 50 different problems that each condition group could potentially get, where instances are generated simply by plugging in different numbers for the variables in the template. Instances that contained one or two of the same number (Ex: “7 + 7 * 7”) were replaced with different instances and duplicate instances were also replaced with different instances.
For each of these problem sets, the students must get three questions right in a row to complete the problem set. Students were randomly assigned to one of the two condition groups. The control group received the existing content in ASSISTments. Typically the standard form of help in ASSISTments is just a single hint consisting of the answer to the problem. However in
1https://sites.google.com/site/selentits2014/home/order-of-operation 2https://sites.google.com/site/selentits2014/home/solving-equations
this experiment those problem sets were not chosen. The chosen problem sets contained customized sets of hints to provide a stronger control group. This method may not be the ideal way to obtain good results, but it was done to compare the effectiveness of buggy messages to the strongest possible existing tutoring in ASISTments. The experiment would not have much external validity if the control group was weak or sub-optimal. The experiment group used the same content as the control group with buggy messages added to the problems.
Figure 4.3: Example of a problem with hints (control group) on “Order of Operations” shown inside the ASSISTment builder.
Figure 4.4: Example of a problem with buggy messages (experimental group) on “Order of Operations” shown inside the ASSISTment builder.
Figure 4.5: Example of a problem with hints (control group) on “Solving Equations” shown inside the ASSISTment builder.
Figure 4.6: Example of a problem with buggy messages (experimental group) on “Solving Equations” shown inside the ASSISTment builder
Buggy messages were used with the existing hints rather than by themselves due to the results of a previously run pilot experiment. Taking hints away from students is risky because it prevents them from continuing on to the next problem until they get the current problem correct. The last hint provides the students with the answer to the problem allowing them to continue on to the next problem. If only buggy messages were used, the students would have to get the correct answer on their own in order to continue on to the next problem. Results from the pilot study on a homework assignment showed that sometimes when the student’s answer is not caught by the buggy message they will get completely stuck and be unable to continue with the assignment. This is the reason why buggy messages were added to the existing content rather than being by themselves. The students need a way to continue in the situations where the buggy messages do not help. This design also makes the condition groups invisible to the users since their problems look exactly the same. It is only different when an incorrect answer is caught by the buggy messages, which is much more subtle than the disappearance of the hint button.
The buggy messages used in the problem sets were generated by the machine learning algorithm in this work. All ASSISTment data (roughly 5 years of data) was supplied to the machine learning algorithm. This consisted of all the incorrect answers for every problem generated by the given template. The algorithm was applied to each of the four templates to generate the generalized incorrect solution paths for the problems. From all these incorrect solution paths, the paths that covered over 40% of the problems and over 1% of the incorrect answers were used to supply incorrect messages for. Each incorrect message was created by me.
Analysis
To analyze the randomized control trial three types of information were examined. Several basic statistics we analyzed to see if the buggy messages made a difference in student learning.
Secondly, a performance curve was looked at to see if there were any increases in learning for a certain number of problems. Lastly, some error analysis was done to get a deeper understanding of what mistakes were being made. All students that did not receive any form of help were removed from the results since they did not use any form of tutoring. For Order of Operation, 71/263 unique students completed the problem set without making any mistakes and were removed. For Solving Equations, 213/378 unique students completed the problem set without making any mistakes and were removed. This left 192 unique students who received some form of help for the Order of Operations problem set and 165 unique students for Solving Equations.
Basic Statistics
A brief description of the statistics analyzed is listed below. The dataset and complete analysis tables can be found at3. The important points of the analysis are summarized in the paper and most of the unimportant results are not shown or mentioned. These statistics are shown Figure 4.7 comparing the control group, which only received hints, to the experimental group, which received buggy messages in addition to the existing hints. How well students performed on the next question after their first mistake was looked at but not included because the trends were similar.
1. Correctness – The average correctness per student. 2. Hints – The average number of hints used per student.
3. Hints per Problem – The average number of hints used per problem per student.
4. Last hint per problem – The average number of last hints used per problem per student. The last hint gives the student the answer to the problem.
5. Attempts per Problem – The average number of attempts per problem per student. An attempt is when a student enters an answer (correct or incorrect).
6. Time per Problem – The average time per problem per student rounded to the nearest second where the median time was taken for each student. Times greater than 20 minutes and less than 0 were removed.
7. Problems per Student – The average number of problems it took a student to complete the problem set per student. This statistic can only be calculated at the problem set level.
Figure 4.7: Hints vs. Hints + Buggy Messages
Similar trends existed for both the Order of Operation and Solving Equation problem sets, therefore both were combined inFigure 4.7. Out of all the statistics examined, the only significance difference was the amount of hints used, with the experimental group using fewer hints. The control group that only had hints used an average of 3.12 (n=172) total hints, where the experimental group that had buggy messages used an average of 2.09 (n=184) total hints. This is a significant difference (p<.03) with an effect size of 0.24. Hints per problem also had a
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 Only Hints Hints + Messages
significant reduction (p=0.03), with an effect size of 0.24, where the group with hints (n=172) used 0.48 hints per problem and the group with buggy messages (n=184) used 0.35 hints per problem. There were no significant differences in correctness, time spent, or number of problems. This means that the buggy messages were able to reduce the amount of hints used without decreasing learning.
Figure 4.8: Average Correctness per Problem
Figure 4.9: Number of student vs. Number of problems done
0 0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 7 8 9 10 A ve rage Cor re ctness Number of Problems Hints Messages 0 10 20 30 40 50 60 70 80 90 1 2 3 4 5 6 7 8 9 10 S tud en ts Problem Number Hints Messages
Learning Curve
The student learning curve is shown in Figure 4.8 for Solving Equations. As the number of problems increases, the number of students decreases, because students either finished the problem set or gave up. This can be seen in Figure 4.9. From these graphs the point to see is what happens from problems 8-10. At problem 8 there are 13 students in both the control group and the experimental group. There is a noticeable difference in the graphs for correctness on these problems. Those students, who received buggy messages and reached 8 problems, finished the problem set within 10 problems 93% of the time. Those students, who did not receive bug messages and reached 8 problems, finished the problem set within 10 problems 54% (7/13 students) of the time. Therefore buggy messages were able to help more than hints for the small subset of <4% of the students.
Error Analysis
From the previous analyses, buggy messages reduced the number of hints used but do not appear to provide any additional help to most of the students. Other than the fact that the existing hints are good, the question of why are the buggy messages not providing additional help arises. The types of errors students made were analyzed. Figure 4.10 shows that students typically do not make the same mistake twice regardless of whether or not they see a buggy message. For all error processes, the control group made the same mistake twice 12% of time and the experiment group made the same mistake twice 15% of the time. This is similar to findings in (VanLehn, 1982) where few students made systematic errors. More data on this analysis can be found at4. If students are not making the same mistakes twice then what mistakes are they making? For each erroneous process a percent of the students making another known process was calculated. Figure 4.10 shows all the known incorrect processes students made for the first problem set. For both the control and experimental group there were eight processes total. The control group errors are represented as the odd process numbers and the experimental group errors are
represented as the even process numbers. This means that processes 1 and 2 are the same where process 1 is in the control group and process 2 is in the experimental group. This is true for the other processes. Combining all the results for both problem sets shows that 63% of the time students make another known process error.
Figure 4.10: This figure shows how many students made the same mistake twice for known incorrect processes in the first problem set. The control group has odd error process numbers and the experimental group has even error
process numbers. Process 1 and 2 are the same but for different groups, as are 3 and 4.
Contributions, Conclusions and Future Work
The contribution that this work makes is a study using buggy messages where we showed that although we did not get an increase in student learning, we did show a positive effect in decreasing the number of hints that students use. This suggests that some of the bug messages were helpful, and thus providing some level of evidence to suggest this algorithm could be used in practice to find common answer patterns.
We also replicated Van Lehn’s work, reporting that we had a large number of common wrong answers that were not systematic, where few students make the same mistake twice. It showed that students rarely make the same mistake twice regardless of whether or not they received specialized feedback on their mistakes. Merely being told they were wrong was enough for students not to make the same mistake twice. It also shows that receiving the specialized feedback reduced the amount of students that asked for help in the form of hints without inhibiting learning.
The process of learning exactly what the student’s erroneous process was and being able to provide specialized help are two separate processes. The machine learning process solves the problem of identifying the erroneous process but does tell the author how to make a good buggy message that will be helpful to students. As future work there needs to be a way to create buggy messages that are known to be good buggy messages, since the ones used in the randomized control trial may, or may not, have been good messages.
0 20 40 60 80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Students Error Process Other Mistakes Mistake Twice