Since there is a lack of empirical research on characteristics associated with good problems, especially from the students‟ perspectives, the first study explored the students‟ perceptions of good problems based on their experiences with problems in problem-based learning. Despite the existence of principles and guidelines which give list of problem characteristics, most of these tend to be theoretical and not validated. The limited few empirical studies tend to focus on specific characteristics. One rare study which has identified a comprehensive list of problem characteristics using Delphi technique is Des Marchais‟ study (1999). This study identified nine characteristics associated with effective problems. Though useful, this study had not included the students‟ perceptions. Students are the end-users of the problems and can be considered as novices in terms of the content knowledge in comparison with the faculty. Cognitive psychology suggests that experts and novices process information differently (Van der Vleuten, 1996). Hence, it is
possible that tutors‟ perceptions of the quality of problem are different from that of the students‟ perceptions. This motivated us to find out the students‟ perceptions on the characteristics associated with effective problems. To this end, we asked students from a PBL curriculum to reflect and record their perceptions of what makes a good problem in their e-journals. Students‟ submissions were then text analyzed to identify the characteristics of good problems in PBL. Results and conclusion of Study 1 are presented in Chapter 2.
STUDY 2
Study 2 builds on Study 1 and investigated both the students‟ and tutors‟ perceptions of effective problems by means of focus group studies. The objective of this study was to first compare the students‟ and tutors‟ perceptions of effective problems and determine if there is any qualitative difference. As far as we know, most of the studies in PBL literature focused on either the students‟ perceptions or the tutors‟ perceptions of problems. One of the few exceptional studies is the likes of Dolmans et al.‟s (1993) study which compared the student-generated learning issues with the faculty-intended learning issues. However, these studies only focused one problem characteristic; the extent to which the problem led to the intended learning issues. In addition, we also wanted to find out if both these groups held their perceptions when given specific set of problems and whether their ratings of problem correlates with the students‟ grades. As mentioned previously, it is possible that tutors view the effectiveness of problems differently from the students. Furthermore, asking participants to mention desirable characteristics of problems in general may yield different answers from asking them to mention characteristics of specific problems. Finally, with the exception of Soppe et al., (2005), most studies have not tried to relate the problem attributes directly to academic achievement.
Therefore, this study aimed to answer the following questions: Which problem characteristics do students and tutors consider generally as contributive to the overall effectiveness of problems in PBL? Do the students‟ and tutors‟ perceptions of problem characteristics hold across a set of problems? To what extent do students and tutors agree in their judgments of the overall effectiveness of these problems? Does the evaluation of problem effectiveness, based on the identified characteristics, reflect itself in the students‟ academic achievement? To test this, we conducted focus group interviews with students and tutors separately. The focus group studies were carried out in two phases. In the first phase of group-discussion, we sought the students‟ and tutors‟ generalized opinions about characteristics of effective PBL problems. In the subsequent phase, we gathered the students‟ and tutors‟ individual responses regarding the effectiveness of eight familiar sample problems. Analysis of the data, results and conclusion are presented in Chapter 3.
STUDY 3
Study 3 explored whether it is possible to develop and validate a rating scale instrument derived form the characteristics of problems identified in the earlier studies to measure the quality of problems. Generally two approaches are used to assess problem quality in PBL literature. One approach is to compare the student- generated learning issues with the faculty-intended learning issues (e.g., Dolmans et al., 1993, 1995). Though useful, this approach sheds light on only one problem characteristics; the extent to which the problem leads to the intended learning issues. Furthermore, this approach is time-consuming and resource intensive. Therefore, this approach may not be suitable when evaluating several problems. The second approach which addresses these issues involves the use of a questionnaire. Although few questionnaires are reported in the
PBL literature to assess (1) complexity and structuredness of the problems (Jacobs et al., 2003), (2) problem familiarity (Soppe et al., 2005), and (3) the extent to which problem leads to appropriate learning activities (Marin-Campos et al., 2004), most of these questionnaires are not validated. In addition, only few characteristics are assessed. Therefore the objective of the third study was to develop and validate a more comprehensive problem quality rating scale. To this end, this study first piloted a rating scale before refining to measure five problem characteristics. Chapter 4 of this dissertation reports the validity and reliability testing of the rating scale.
STUDY 4
Study 4 attempted to rigorously test the reliability of the rating scale developed in Study 3. Although the earlier study included reliability testing, only internal consistency of the measures was assessed. According to Cohen, Manion, and Morrison (2000) the three principles of reliability are internal consistency, stability, and equivalence. Internal consistency measure of the rating scale indicates the extent of congruency and consistency between the different items of a single scale. On the other hand, stability refers to the measure of consistency over time and across different raters. It is possible that the rating scale is reliable in terms of internal consistency, but only when measured at a certain time-point or when administered to a specific group of students. For instance, high achievers may evaluate the problems differently from low achievers. Hence the objective of Study 4 was to investigate the inter-rater reliability and the stability of the ratings over time, and across different groups of students in using the problem quality rating scale. Chapter 5 presents the findings from Study 4.
STUDY 5
Study 5 attempts to extend on Soppe et al.‟s (2005) work using the rating scale developed and tested in this research (Studies 3 & 4). In 2005, Soppe et al. carried out an experimental study to investigate the influence of problem familiarity on various aspects of students‟ learning. Their hypothesis was that students working with the more familiar problem would activate more prior knowledge during the initial discussion. This activation of prior knowledge would stimulate more interest, which would in turn lead to students spending more time on self-study, resulting in acquisition of higher- quality subject matter which is reflected as higher scores on relevant knowledge tests.
To test their hypothesis, they presented one group of students with a familiar version of a problem and another group with an unfamiliar version of the same problem and measured the students‟ perceptions about problem familiarity, problem quality (interestingness, difficulty level and match with prior knowledge), time spent on self-study using a rating scale. Additionally, external measures such as tutor‟s assessment of the quality and quantity of learning issues as well as students‟ achievement in knowledge tests were used. Their results suggested that students found the familiar problem to be of higher quality and interesting than the unfamiliar problem. Surprisingly, no corresponding difference was found in terms of the quality and quantity of learning issues generated by students, time taken for self-study, and students‟ achievement in knowledge tests as a result of the working on the two problems.
The authors suggested that insignificant differences in some of the measures could be due to the subtleness of the experimental manipulation, and recommended increasing the difference between the familiar and unfamiliar version of the problem. To extend on this work, Study 5 was carried out with some modifications in the methodology. First, two different problems
differing in the familiarity level were used instead of using two versions of the same problem. Second, a validated and reliability- tested rating scale (from Studies 3 & 4) was used to measure the various problem characteristics. Third, the tutor‟s observation of the students‟ behaviour and learning was included. Chapter 6 presents the findings from Study 5.
Overall, Studies 1 and 2 were explorative in nature and attempted to identify characteristics of problems in students‟ and tutors‟ perceptions. Studies 3 and 4 tested the validity and reliability of these characterises while Study 5 attempted to apply the rating scale in investigating the influence of problem familiarity on students‟ learning. Taken together, these five studies not only tried to shed light on the different characteristics associated with problem quality (Studies 1, 2, 3, 4, & 5) but they also attempted to relate the characteristics to students‟ learning (Studies 2 & 5). Findings and discussion of the findings for each of the five studies are presented individually in the next five chapters. Finally, to give a more holistic picture, an overall summary and discussion of the five studies are reported in Chapter 7 of the dissertation.
CHAPTER 2
CHARACTERISTICS OF PROBLEMS FOR PROBLEM-