Study 5 utilized the rating scale from Study 3 to investigate the influence of problem familiarity on students‟ learning. This study can be considered as an extension of Soppe et al.‟s (2005) study which investigated the influence of problem familiarity on various aspects of students‟ learning. To do so, they presented 270 students from a psychology course with either a familiar or unfamiliar version of the same problem. The authors defined familiarity level as the extent to which the students could identify with the characters/actors in the story narrated in the problem. Both the versions of this problem focused on the same subject/content matter (reasoning and decision-making). The familiar problem
narrated events about psychology students. On the other hand, the unfamiliar version narrated events involving lawyers and other non- psychologists. Their hypothesis was that students working with the more familiar problem would activate more prior knowledge during the initial discussion. This activation of prior knowledge would stimulate more interest, which will in turn lead to students spending more time on self-study, resulting in acquisition of higher-quality subject matter which is reflected as higher scores on relevant knowledge tests.
Approximately half the 270 students received the familiar version while the other half received the unfamiliar version of problem. Measurements of students‟ perceptions about problem familiarity, problem quality (interestingness, difficulty level and match with prior knowledge), and time spent on self-study were measured. In addition, external measures such as tutor‟s assessment of the quality and quantity of learning issues, as well as students‟ achievement in knowledge tests were taken. Data analysis suggested that students found the familiar problem to be of higher quality and to be more interesting than the unfamiliar problem. However, no corresponding difference was found in terms of the quality and quantity of learning issues generated by students, time taken for self-study, and students‟ achievement in knowledge tests as a result of the working on the two problems. The authors suggested that insignificant differences in some of the measures could be due to the subtleness of the experimental manipulation, and recommended increasing the difference between the familiar and unfamiliar version of the problem.
To address this, we relooked again at the definition of problem familiarity. Although we agree that problem familiarity includes the extent to which students can identify with the characters/actors in the story narrated in the problem, we felt that the notion of “familiarity” has a broader meaning. We define familiarity
as the extent to which the problem matches the students‟ subject matter knowledge, experiential knowledge, and contextual knowledge, in accordance with Dochy and Alexandar‟s (1995) definition of prior knowledge. Like Soppe et al., (2005) we aimed to investigate the influence of problem familiarity on students‟ learning, but this was based on the broader definition of problem familiarity. A caveat to note is that we did not attempt to differentiate between the various forms of prior knowledge associated with problem design. Resonating with Soppe et al.‟s study (2005), our hypothesis was that the level of problem familiarity will influence students‟ learning, in particular their interest, critical reasoning, collaborative learning, and the extent to which students can identify the intended learning issues.
To test the hypothesis, we carried out an experimental study using two problems from the actual curriculum in a naturalistic educational setting. The two problems were identified a priori by the module coordinator to be differing in the extent of problem familiarity to students. The module coordinator selected the problems based on past experience in using the same problems. Of the two problems, one was on “Knowledge and morality” while the other was on “Realism and anti-realism”. Students were expected to be familiar with the problem on “Knowledge and morality” in terms of subject matter. In contrast, they were expected to be less familiar with the subject of “Realism and anti-realism”. The specific questions asked by the study are (1) do students differentiate between familiar and unfamiliar problems?, (2) what are students‟ perceptions of their own learning as a result of working on the familiar and unfamiliar problems, and (3) does the students‟ perceptions of their own learning correspond with their tutors‟ assessment?
To this end, we presented both the familiar and unfamiliar problem as part of the regular curriculum to 172 students on two
occasions, one problem per occasion. A point to note is that the same group of students attempted both the problems on two occasions unlike Soppe et al.‟s (2005) study in which each group of students attempted one of the two versions of the same problem. Students were administered the problem presupposed to be unfamiliar (Realism and anti-realism) first. After the students had worked on each of the two problems, students‟ perceptions of problem familiarity, the extent to which the problem led to the intended learning issues, the extent to which problem triggered interest, the extent to which the problem stimulated critical reasoning, and the extent to which the problem promoted collaborative learning were collated using the rating scale from Study 3. In addition, tutors‟ assessment of students‟ learning was used to verify the influence of the two problems.
Data analysis of the students‟ response revealed that they perceived the two problems to be significantly different with respect to problem familiarity. As presupposed, the problem on “Knowledge and morality” was found to be more familiar. In addition, the results suggest that the students perceived the familiar problem to be significantly more interesting, and more successful in guiding them on the learning issues. Interestingly, they did not find the two problems to be significantly different in stimulating critical reasoning, and in promoting collaborative learning. Tutors who observed and facilitated the participants in the problem solving process assessed that the students‟ individual learning to be significantly better for the familiar problem than for the unfamiliar problem.
Students‟ perceptions of the two problems on the measure of problem familiarity show that they found the problem “Knowledge and morality” to be more familiar (M = 3.51, SD = .45) when compared with the problem on “Realism and anti-realism (M = 3.06, SD = .58). This finding supports the presupposition about
the extent of problem familiarity. Students‟ response on the rating scale (Sockalingam et al., 2008) shows that they attributed this to a better fit of the problem with their prior knowledge associated with past-experience, subject knowledge, and general knowledge. As for the influence of problem familiarity on the measure of problem leading to the formulation of intended learning issues, students who worked on the familiar problem felt that it had clearer instruction, and had more key words or clues embedded to guide them successfully to the intended learning issues (M = 3.68, SD = .53) than the unfamiliar problem (M = 3.33, SD = .46). This result reflects Dolmans et al.‟s (1993) findings that students fail to identify learning issues intended by the tutor if their prior knowledge does not sufficiently match the problem.
Students also felt that the familiar problem (M = 3.66, SD = .52) was significantly more appealing and engaging than the unfamiliar problem (M = 3.51, SD = .58). This is in line with Soppe et al‟.s (2005) and Gijselaers & Schmidt‟s (1990) findings that problems that are familiar in context and problems that match students‟ prior knowledge are found to be more interesting to students. The present study therefore adds further evidence that the extent of problem familiarity positively influences students‟ interest and success in identifying relevant learning issues.
Surprisingly, at first glance, the results seem to suggest that there was no significant difference between the two problems in the extent to which critical reasoning was stimulated (Familiar problem; M = 3.82, SD = .42, Unfamiliar problem; M = 3.88, SD = .45). Based on Hmelo-Silver‟s (2004), description of the PBL process, it is expected that a more unfamiliar problem would trigger more questions and thinking. A closer examination of the subscales demystifies the anomalous findings. The rating scale used in the study (Sockalingam et al., 2008) defines critical reasoning as stimulation of questioning, thinking, and reasoning, and
consideration of the problem from multiple perspectives. A closer examination of the students‟ response shows that the unfamiliar problem had indeed resulted in significantly more questioning, and thinking and reasoning. However, no significant different emerged between the two problems in triggering consideration of multiple perspectives. This could be because both the problems contained goal specifications that required consideration of multiple perspectives.
As for the influence of the problem familiarity on the collaborative learning, no significant difference was found between the two problems (Familiar problem; M = 3.90, SD = .45, Unfamiliar problem; M = 3.86, SD = .51). Similar results have also been reported by Van Berkel and Schmidt (2000), and Gijselaers and Schmidt (1990) that prior knowledge does not necessarily influence collaborative learning. However, no explanation has been proposed for this. The result from this study suggests that unfamiliar problem stimulates significantly more questioning, thinking and reasoning than the familiar problem. Hence, we think that exploring the type and pattern of verbal interaction may provide more insights than considering collaborative learning as a whole. This idea seems to be in line with, Visschers-Pleijers et al.‟s (2005) study which showed that interaction in PBL can be classified into at least two types; exploratory questions and cumulative reasoning. However, their study investigated interaction taking place as a result of students working on one problem only. Hence, a future study that investigates the influence of various types of problems on the pattern of group interaction may be useful.
Overall, the measures of students‟ perceptions suggest that the familiar problem has a more positive influence on their learning. This is also reflected in the tutors‟ assessment of the students‟ learning. According to tutors‟ assessment, the two problems differed significantly in influencing students‟ learning [t(171) = 2.04, p =
.04]. They assessed the students‟ learning to be better for the familiar problem (M = 3.22, SD = .51) than the unfamiliar problem (M = 3.09, SD = .86). In comparison with Soppe et al.‟s (2005) study, the present study has not only considered more aspects of students‟ learning such as critical reasoning and collaborative learning, but it has also included the tutors‟ observation throughout the students‟ learning process which is expected to be more comprehensive than the measure of students‟ achievement through knowledge tests. In addition, this study has analyzed students‟ individual responses, rather than the aggregation of their responses at group level as in Soppe et al.‟s (2005) study.
Nevertheless, the study has a number of limitations. The first limitation is the generalizability of the study. As only two problems were used in the study, the findings need to be tested with more problems, and other PBL contexts. Second, the study does not demonstrate causality between the various variables. A causal modeling approach may be more suitable for that purpose. The experimental/evaluative approach used in this study is instead more suitable to collate information about individual problems. This approach is likely to be useful as formative evaluation of the course material. Third, the study only focuses on the extent of problem familiarity. However, other confounding characteristics of problems such as the level of problem clarity and problem complexity may be very closely associated with problem familiarity and this can influence the students‟ learning as well. Fourth, although this study seems to suggest that problem familiarity leads to better learning, one must be cautious in interpreting the result. It is likely that if a problem is too familiar, students may find it to be boring and not motivating enough, resulting in poor learning. Hence an extension of this study to investigate more problems will be useful.
CONCLUSION
In conclusion, the five studies in the dissertation build on what is found in the literature, and one another to synthesize new knowledge about problems used in Problem-based Learning. The first two studies in the dissertation capture a panoramic view of students‟ perceptions, and provide a comparison between both students‟ and tutors‟ perceptions for the first time. Results from these studies also identify three common characteristics cited across various PBL contexts and a new characteristic which has been neglected in the existing guidelines/principles of problem design. In addition to generating a comprehensive list of problem characteristics, Study 1 attempts to provide a meaningful classification of the characteristics as feature and function characteristics. Study 2 shows that there is no significant differences between the students‟ and tutor‟s perceptions of effective problems, and relates the effectiveness of problems in terms of the eleven identified characteristics to students‟ learning. Overall, Studies 1 and 2 form the foundations of the subsequent three studies in the dissertation.
Study 3 contributes to the existing literature by developing and validating a comprehensive problem quality rating scale measuring five problem characteristics. Study 4 tests the reliability of the rating scale and suggests satisfactory inter-rater reliability of the rating scale measures and offers insights that students‟ perceptions of the problem quality may vary during the learning process. In addition, Study 4 shows that the measures using the problem quality rating scale are generally reliable and provide support for further validation in other PBL contexts. As far as we know, rating scales to measure problem quality have not been tested in different PBL contexts. Therefore this study adds to the current literature by taking the first step to developing a rating scale that is applicable across PBL contexts.
Study 5 demonstrates the utility of the developed rating scale in different contexts as well. The five characteristics measured by the rating scale are not only theoretically grounded (Dolmans et al., 1997; Hung, 2003), and empirically evidence-based (Des Marchais, 1999, Sockalingam & Schmidt, 2007) but they are also aligned with the constructivist principles (Savery & Duffy, 1995). This study also shows that the rating scale can be used to measure more aspects of students learning‟ using the rating scale, instead of being limited to knowledge tests or comparison of student-generated learning issues with faculty-intended learning issues. Furthermore, Study 5 adds evidence to the existing literature (Gijselaers & Schmidt, 1990; Soppe et al., 2005) that the extent of problem familiarity positively influences students‟ motivation and success in identifying the learning issues. It also reveals the unfamiliarity level of the problem may not be all that bad as it leads to thinking, questioning, and reasoning. Finally, it sheds light that more research work is needed to understand the interaction taking place in collaborative learning as a result of problem-solving. In sum, the consistency in the findings through the various studies using different approaches adds validity to the overall findings.
IMPLICATIONS