The success of intelligent tutors within the educational field is evident. Examples of these types of systems enhancing students’ learning gains can be found in use within different domains. However, the time and resources required to design, implement, and release those ITSs as finished and stable systems are extremely high. As we discussed in this dissertation, developers of successfully implemented ITSs report that hundreds of hours are required for ITS designers and domain experts to gather the required domain and pedagogical knowledge. Furthermore, considerable time must be invested by ITS developers to convert the collected domain-knowledge into the intended knowledge representation structures (e.g., production rules and constraints statements).
The downside of ITS development regarding time and resources is highly influenced by the amount of knowledge the ITS should learn before starting to perform as an effective instructional tool. Furthermore, the effort to obtain adequate and comprehensive domain and pedagogical knowledge varies depending on the complexity
121 of the domain. Specifically, for ill-defined domains, the time and resources required to
build ITSs tends to increase significantly. As we described in Chapter II, the reason for this greater effort is based on the large number of correct solutions and solution paths characterized by ill-defined problems. The characteristics underlying ill-defined domains lead to two main challenges: First, gathering comprehensive domain knowledge and a solid representation of potential students’ misconceptions from domain experts is a hard task to be performed during the system’s design time. Second, using the conventional knowledge representation structures, developed for ITSs used within well-defined domains, tends to be ineffective in representing knowledge for ill-defined problems.
In this research, we focused on the generation of the domain-knowledge for an ITS intended to support students’ learning in the type of ill-defined domains described previously. In our study, we explored ways to address previously mentioned problems by using the Learning from Demonstration (LfD) technique. We leveraged basic LfD by a mixed-response approach in which the instructor has a permanent participation within the tutoring loop. In addition, we implemented and evaluated two knowledge representation methods, based on Weighted Markov Models and on Weighted Context Free Grammars respectively, which proved to be well suited for modeling languages and sequential data.
To minimize the time required to gather comprehensive domain knowledge during the ITS design time, the presented mixed-response ITS framework based on LfD aims to allow an intelligent tutor to continuously learn from instructor-student interactions at run time. By using an implemented metacognitive skill, the ITS is able to
121 estimate its current knowledge confidence level in order to defer to the human instructor
in cases where the ITS does not have sufficient confidence in the applicability of any of its available tutorial actions. As a result of this, the time required to release an ITS can be shortened by leaving the task of identifying more sophisticated or unexpected situations to the intelligent tutor at run time. Then, by asking for instructor intervention, the domain knowledge will continuously grow from the instructor tutoring demonstrations.
We validated the proposed ITS framework based on six functionality characteristics we envisioned for a modern ITS supporting learning within ill-defined domains: effectiveness, scalability, efficiency, portability,metacognition, and robustness. We established that, in addition to always providing an effective tutoring performance, an ITS framework should offer: (1) scalability in data magnitude, in other words, an ITS should offer a means to expand its knowledge-base and preserve suitable performance; (2) efficiency reducing the time and human effort required to integrate new knowledge into the knowledge-base; (3) portability to be used across multiple domains with nil to minor alteration to the framework functionality; (4) metacognitive skill to infer its knowledge confidence level in order to trigger adequate tutorial responses or request for instructor help; and (5), sufficient robustness to deal with different pedagogical and tutoring criteria used by instructors to teach the intricate skills and knowledge required to solve ill-defined problems.
Furthermore, we introduced the implemented methods (WMMs and WCFGs) and algorithms used for knowledge representation. In our pursuit to choose the best ITS
121 feedback classification model, we conducted five different experiments using various
granularity levels for knowledge representation. We found that by using our WCFG- based classifier in conjunction with a finer granularity level for knowledge representation, the implemented intelligent tutor was able to reach around 97% effectiveness in providing correct feedback when considering its knowledge confidence level. This percentage of effectiveness was accomplished after a training period when the ITS frequently asked for instructor demonstrations. Most of the tutorial actions were provided by instructors within the first 20 to 40 analyzed CSs. The ITS started using the learned tutorial actions after around 50 demonstrations. Then, after approximately 120 or 130 CSs the ITS started providing more feedback and the instructor´s intervention decreased.
Moreover, the performance of the ITS demonstrated its robustness and predictability when applied to different datasets and pedagogical perspectives. Furthermore, contrary to the time required to build production rules reported by previous ITS researchers, a relatively smaller amount of time was reported by instructors using the proposed framework. The time spent by the instructor ranged from 1.0 to 3.0 hours to construct the knowledge-base and required tutorial actions per exercise. Results from our experiments within the SQL-queries domain were similar to the 1.1 hours reported by Mitrovic (1998). However, even though we conducted experiments using data from the same domain (SQL-queries), the level of complexity of exercises and differences between the two implemented approaches prevented us from making a direct comparison regarding implementation time. As mentioned previously, the time spent reported by
121 Mitrovic (1998) building the SQL-tutor is based on the implementation of a constraint
while our reported time is based on the construction of an entire exercise. However, each constraint within the SQL-tutor can be used in multiple exercises. While currently in our case, each new exercise must be constructed from scratch. Finally, the proposed ITS framework proved to perform well when it was applied to a second domain. This is significant because the ITS framework could be directly applied without modifications. However, more empirical research is suggested regarding the portability criteria.