In this section we will supply and justify the criteria for determining how the above-stated research questions can be answered. These criteria will also determine how deeply these questions must be answered to fulfil
the goals of this research project.
RQ1 and RQ2 will be answered if we identify system interventions and ways to organize collaborative learning environments that effectively support students in ethical problem-solving.
RQ3 will be answered when we identify the most accurate way(s) to diagnose students’ arguments that allow the effective interventions to occur. For our proposed interventions (see the next Section 3.3), the diagnosis of students’ arguments is equivalent to the comparison of students’ arguments with each other and with the system’s arguments to determine if the given arguments are focused on the same or different issues. We are seeking ways that result in the highest accuracy in determining if the two given arguments are on the same or different issues as judged by a human expert. One thing to note here is that we do not intend to develop a new algorithm for diagnosing students’ arguments. Our intention is to use already developed text comparison algorithms, such as word-based search, Latent Semantic Analysis, etc., and identify the most effective among them for our needs.
How can we measure the effectiveness of proposed interventions for RQ1 and RQ2? In measuring the effect of the interventions of a new computer-based learning environment, it is common in the AIED/ITS community to talk about learning gains. Students take a pretest, use the new computer environment, and then take a posttest. The difference between students’ results in the pretest and posttest are called learning gains, and attributed to the effectiveness of the new learning environment. This works well for problems with single definite solutions, such as many maths or quantitative physics problems.
However, for ill-defined domains including the ethics domain, this approach for measuring the system’s impact is not possible, because of the absence of formal or well-accepted methods to verify solutions, a lack of criteria by which solutions are judged, and disagreement even among domain experts regarding the adequacy of solutions [53]. Specifically for the ethics domain, methods to assess students’ ability to resolve ethical dilemmas remain largely undeveloped; already existing assessment schemes require trained coders, and have been tested to be sensitive only for measuring learning gains across a whole semester of study [93]. All this makes it impractical to use the standard pretest/posttest procedure to objectively evaluate the learning gains of students for our intended short-run interventions in Umka. Thus, instead we adopted different metrics for the evaluation of the effectiveness of the system interventions:
1. Measuring productive student behaviors. Rather than measuring the students’ learning, we will measure
the extent to which the system was able to create necessary students’ experiences, and foster students’ behaviors that are associated with higher learning. Section 2.2.2 on pedagogy in ethics and moral education listed the behaviors that are associated with students’ moral development: participating in arguments, justifying opinions, responding to counterarguments, re-examining assumptions, analyzing multiple perspectives, etc. Thus, the more the system was able to stimulate students’ productive behaviors, the more effective are the system’s interventions.
These productive behaviors of students are, in fact, metacognitive processes. To analyze a case study, students need to be aware of and regulate their ethical thinking: to question their own assumptions,
analyse their own arguments and motivations, make sure they have covered all the facts, have not factored in their own beliefs or prejudices too strongly, have uncovered all the possible directions for analyzing the case, have taken measures to improve their analysis through interaction with other students. In section 2.1.2 we demonstrated that metacognitive processes are important for solving ill- defined problems. Thus, measuring how strong students’ metacognitive skills are, we indirectly measure the students’ ability to solve ill-defined problems in ethics.
2. Measuring students’ attitudes towards different system features. This is done by asking students to
evaluate helpfulness of various system support features, and tracking students’ usage of them. The students’ frequent usage of certain support types most likely indicates their effectiveness in helping students. A high evaluation of the helpfulness of the support types by the learners can be an additional indicator of their helpfulness.
3. Measuring the changes in a student’s analysis as the result of the system interventions. This can be
accomplished by comparing the student’s initial individual analysis before any system interventions with the student’s final work after the interventions. Based on the discussion in Section 2.2.3 on the evaluation of an ethical analysis, this comparison involves identifying differences in the number of issues and arguments presented, differences in the quality of the discussed issues and arguments, and differences in the justification of the final resolution.
4. Comparing with other systems, or the same system with different types of interventions. Contrast-
ing students’ recorded behaviors, their attitudes from questionnaires, and resulting changes in their analyses across different systems or across different interventions of a given system allows the relative assessment of the effectiveness of given interventions in comparison with other interventions, and allows the identification of the most effective system interventions.