Structure of the rubric 81

In constructing the rubric, we integrated top-down and bottom-up approaches (Chi et al. 1997). In the top-down approach we constructed an a priori representation of the “ideal knowledge” underlying an expert approach to the problem, and looked for the extent to which each student's approach included certain elements of the “ideal knowledge”. We developed a scoring rubric that allows identifying the gap between the student's knowledge reflected in his/her solution and the "ideal knowledge" needed to solve the problem appropriately.

In representing the expert "ideal knowledge" in the rubric, there are generic versus specific elements. Generic elements include the presence of physical principles within the problem as well as a presentation of the solution, i.e. how well it is communicated and justified. As students in the course were asked to follow a strategic problem-solving approach, we took the presentation of the solution according to such approach as representing a solution “ideally” communicated and justified. Thus, we constructed physics and presentation knowledge

categories in the rubric. The physics category is divided into the subcategories of invoking a physical principle and applying that principle. Each row in each subcategory therefore represents every physical principle that a student will have to invoke and apply to correctly solve the specific problem. The presentation category includes three subcategories. Problem description considers whether the student presented a helpful description of the problem’s situation in terms of physics concepts and principles, e.g., if a diagram is drawn to help visualize the problem. Planning and solution construction evaluates whether the student constructed a good plan for solving the problem with regard to the target quantity and intermediate problem steps needed to obtain this quantity. Evaluation considers whether the student checked the reasonability of his or her answer once it is obtained so as to make sure he or she did the problem correctly (e.g., units, extreme cases, etc.). The physics and presentation subcategories are essentially general and do not have to be changed from problem to problem; only the specific criteria based upon the subcategories need to be changed. Specific elements have to do with the physical principles and presentation steps that the student must invoke and apply in order to solve the specific problem.

However, to uncover how the students are actually thinking and the possibly incorrect mental models that they use to solve the problem, one has to take in parallel a bottom-up approach. To that end, we went over students' work and identified common mistakes in approaches to solve the problem. We represented these common approaches in the rubric under novice “incorrect ideas”. The rubric has additional rows in the physics subsection that tracks if a student invoked an inappropriate principle that doesn’t apply to the problem or applied inappropriately the principles she/he correctly invoked. Such analysis allows us to note a student who realized he/she made some specific mistake, even though he/she can't correct it.

The work of each student was evaluated in three ways. The researcher diagnosis of the student’s quiz solution (RDS) is simply an evaluation of the students’ initial quiz performance according to the rubric (i.e. not based on the TA’s score). The student’s self-diagnosis of his/her solution (SDS) is where we interpreted the self-diagnosis the student provided in terms of the rubric. The researcher’s judgment of this student’s self-diagnosis (RSD) compares the researcher’s diagnosis and the student’s diagnosis of the student solution. To represent these three grading evaluations, we constructed three columns in the rubric.

The method for scoring is as follows: In the RDS and RSD columns, “+” is given if a student correctly performs/identifies a mistake defined by some subcategory. A “-” is given if the student incorrectly performs, fails to identify a mistake, or identifies a mistake incorrectly. If a student is judged to have gotten something partially correct, then the grader may assign “++/-,” “+/-,” or “+/--.” The term “n/a” is assigned if the student could not reasonably address a subcategory given the prior work done. In the SDS column, we report how the student would grade oneself with this rubric. For example, if a student correctly diagnoses a mistake, a “-” is given since this is the grade the student gives himself or herself in the solution. If a student does not refer to a mistake he has made, an “x” is assigned.

The validity of the rubric is determined by the extent to which it indeed allows us to map the student's solution to the expert “ideal knowledge” as well as to novice-like incorrect ideas. The validity was determined by four experts in physics education who perceive it as measuring an appropriate performance of the solution and self-diagnosis. Two researchers performed analysis on the students’ work and any disagreements were discussed and resolved. The inter- rater reliability achieved was 80% of the aggregate of all subcategory items for a sampling of 20% of the students graded by both researchers.