Coding scheme development illustrated using the Lawn Problem

Two alternative viewpoints were employed when interpreting the participants’ solutions to the Lawn and the other problems – one was to adopt an established description of problem solving presented by Mayer (1992) and the other was to start with a blank slate and be open to whatever was contained in the data. The latter is generally more challenging as not

knowing where to start makes it harder to start. The attraction of Mayer’s schema was that it existed and was developed for simple word algebra problems and could be applied to the problems used in this study to develop a set of a priori codes which could be used as data were

examined. The weakness of Mayer’s schema was that it does not allow for variation in

problem solving approaches and representations that may be naturally present in the data. As the data were examined through the lens of the Mayer framework, actions that might not be contained in or consistent with the a priori Mayer schema were also searched for and noted.

Analysis was completed and is presented separately for each problem beginning with the Lawn problem, for which the analytical process is elaborated to a greater extent in order to illustrate the process in detail. The order in which the problem analyses are presented in this chapter and the next is based on the order in which they were presented to the students. The Lawn problem was the first to be analysed and, therefore, was accompanied by a larger learning curve but this ordering of the problems is not important as each problem was analysed

separately and independently. With the exception of the Lawn problem as it used to elaborate the method of analysis these analyses can be read in a different order to which they are presented if the reader so desires. Since this chapter and the next are concerned with the analysis of the problem representation phase data collected from all 115 participants were included as it was assumed that all were capable of attempting to represent the problem even if they could or could not correctly answer the core competency questions. In other words, a lack of ability to complete the solution step was not assumed to hinder an attempt to

complete the preceding representation step. As mentioned in Chapter 5, participants were presented with the full set of problems first and then then the full set of questions.

Step 1, Apply Mayer’s schema to develop the first set of codes

The Lawn problem is presented first to provide some context for the coding schemes:

“A square lawn was extended in width by 2 m and in length by 3 m. The area of the new lawn is twice as big as the area of the old lawn. What are the

measurements of the old lawn?”

The core competency question related to this problem is Question 1, i.e., factorise a quadratic equation:

Find the roots of 2x² + 6x - 8 = 0 using factoring

The Mayer framework (Mayer, 1992) for math problem solving consists of several types of knowledge that are required to solve math problems. In the context of the Lawn problem, the application of each of these types of knowledge by the successful problem solver is outlined in Table 6-1.

Type of knowledge Application to the Lawn problem

Linguistic knowledge ability to understand the words used in the problem, e.g. the lawn is square, area of new lawn is twice the old area, new width equals old width plus 2, new length equals old length plus 3

Semantic knowledge ability to draw on common sense or knowledge that is taken for granted, e.g. a square has four equal sides, a lawn is an area of grass beside a house (this definition was provided in the problem statement) that is 2 dimensional (not provided)

Schematic knowledge ability to draw on knowledge of schema, i.e. that have been previously learnt; in this case, a schema that is required is that area = length x width

Strategic knowledge ability to set subgoals in the problem solving process and to monitor progress, in this case one should develop an equation for one unknown – the side of the original square – and then solve for this unknown using algebra

Procedural knowledge ability to perform standard mathematical procedures; in this case, an important procedure is to factorise a quadratic, (hence the core competency question testing this skill)

Table 6-1. Mayer framework (Mayer, 1992) for math problem solving applied to the Lawn problem.

Several problem solving actions can be performed when drawing on each of these types of knowledge. These actions were listed separately to allow for cases where a participant demonstrated some but not all actions that draw on linguistic knowledge, for example. This led to the creation of a more fine grained set of codes presented in Table 6-2.

No. Code Present if…. (1) Absent if …. (0) Type of knowledge Required to solve problem 1 Square Lawn Discerns lawn is a square Does not, e.g.

rectangle Linguistic

Assignment Yes

2 Area change Discerns ANew=2xAOld Does not Linguistic Relational Yes 3 Width+2 Discerns new width = old + 2 Does not Linguistic Relational Yes 4 Length+3 Discerns new length = old + 3 Does not Linguistic Relational Yes

5 2D object Lawn is a 2D object Does not Semantic Yes

6 A=WxL Includes Area = width x length Does not Schematic Yes 7 Correct

equation The correct equation is provided Not provided Strategic Yes

8 Solve Solve equation correctly Makes error Procedural Yes

9 Check Checks answer Does not Strategic No

Table 6-2. A priori coding scheme for the Lawn Problem based on the Mayer schema.

Step 2, Start reading, coding and reflecting on codes

Each transcript was studied in turn checking for evidence of these actions in each participant’s written solutions. If, on moving to the next solution script, a new action that was qualitatively different to an existing code was identified it was added as a new code to the list. These codes were phrased almost entirely as binary statements or questions, i.e. present or not present in the solutions. If there was more than one component to the code then it was split up to create several related binary codes or, in rare cases, a code was allowed to have several categories.

Hence, the a priori list of codes changed with reflection and careful consideration while reading the solutions in a constant effort to replace, improve or discard codes. When changes occurred they were retrospectively applied to all participants’ solution by returning to the first participant’s solutions and scoring it and each subsequent participant for the modified codes.

In the case of the Lawn problem this process led to the creation of additional codes (Table 6-3) and the deletion of some a priori codes if they appeared to be redundant. For example, since every solution showed evidence of Code 3, ‘Width + 2’, also contained Code 4, ‘Length + 3’, it was possible to delete Code 4 without losing any information. All treated the lawn as a 2 dimensional object so Code 5 was also deleted. Finally, some codes (no. 15 to 18, Table 6-3) were created by combining other codes that connected together to form a coherent approach

to solving the problem such as having all the ingredients to write the correct quadratic equation.

It was difficult to discern strategic knowledge from the answer sheets, particularly the monitoring aspect. Some solutions were very straightforward, simple and error free – there must be monitoring in action here but how could it be discerned? Other solutions started with a rectangular lawn and were then corrected to a square lawn – is this an example of

monitoring sparked by an ‘I can’t solve this, I’ve done something wrong, let me go back’

moment or might the student have just glanced again at the problem and the word ‘square’

caught the eye? Analysis of the data focused on the problem representation step, the topic of interest in this study, rather than this more difficult to interpret aspect of strategic knowledge.

No. Code Present if…. (1) Absent if …. (0) Type of

knowledge Required to solve problem 10 Schema change Changes from incorrect to correct No change Strategic (is

monitoring work) No 11 L=W=x Assigns a variable ID like x to

length/width Does not Strategic Yes

12 Number of equations Has 1 equation for each unknown Does not, e.g. 1 equation for 2 unknowns

Procedural Yes

13 Guess & check Uses guess and check approach Does not Strategic? No 14 Equation equality Both sides of equation are equal Both sides are not

equal Procedural Yes

Table 6-3. A posteriori codes for the Lawn Problem.

Statistical analysis of codes

Since the purpose of this phase of the analysis was to examine variation in approach to problem solving among weak and strong visualizers, the extent to which each of the codes revealed a difference in spatial ability was checked in two ways. First, the sample was grouped by spatial ability into weak and strong visualizers before counting how many of each scored 0 and 1 on each code. For example, the number of weak and strong visualizers who showed evidence of treating the lawn as having a square shape (code 1 = 1) was noted. Second, the

sample was regrouped based on the code being present or not (0 or 1) and the mean and S.D.

of the PSVT:R scores were computed for each group and compared using an independent samples t-test and a Cohen’s d effect size. For example, all those who showed evidence of treating the lawn as a square were placed in one group (code 1 = 1) with the remainder (code 1 = 0) in the second group, the mean PSVT:R scores of each group were then compared using an independent samples t-test to measure the significance of any difference and a Cohen’s d effect size was calculated to indicate the magnitude of the difference. This process was repeated for all the codes and the results are presented in Table 6-4.

The lawn problem contains several statements that define variables or relationships between variables which are contained in codes 1, 2, 3 and 6 and which varied greatly in effect size.

While code 1, ‘the lawn is square’, divided the sample into two large groups, it resulted in a small to medium effect size (d = .28, N.S.) in terms of spatial ability difference between those who did and didn’t treat the lawn as a square when solving the problem. Larger and more significant effect sizes were observed for codes 2 and 3, the relational statements in the problem (d = .64, p < .05 and d = .72, p < .01). Code 6, Area = W x L, produced the largest effect size (d = .93, p < .05) and all but eight participants showed evidence of this code in their solutions. These four codes were needed as a complete set to write the correct quadratic equation, i.e. it was essential to have them as a set to solve the problem. This combination is shown by code 17a which revealed a medium effect size (d= .54, p < .01).

Weak n Strong n Total n PSVT:R

1 Binary version of original code, must have all components to score as 1, e.g. if Code 16 = 3, then Code 16a = 1, else Code 16a = 0

Table 6-4. Statistical data for the codes in the Lawn Problem.

While 52 participants obtained the four ingredients required to write the equation, only 31 were successful in writing the correct equation and, of these, only 23 were successful in the solving the equation. At each of these steps, particularly the first two, strong visualizers were more successful than weak visualizers as shown in Table 6-5. The gap between weak and strong in solving the equation is not that large – if provided with the equation, 6 of 9 weak visualizers can factorise it versus 17 of 22 strong visualizers or 66% v 77%. Question 1, which assessed this same competency, was correctly answered by 41 of 55 weak visualizers and 50 of 60 strong visualizers or 75% v 83%. Most of the attrition occurred before the solution phase in which the core competency was needed.

All 4 ingredients Correct equation Solve equation

Weak visualizers 36%, n = 20 16%, n = 9 11%, n = 6

Strong visualizers 53%, n = 32 37%, n = 22 28%, n = 17

Table 6-5. Distribution of weak and strong visualizers among problem solving codes for the Lawn Problem.

Success on the Lawn problem revealed a medium to large effect size in spatial ability (d = .54, p

< .05) and a very similar result for the corresponding competency question (d = .56, p < .05).

When those who responded incorrectly to the competency question were excluded the effect size was almost unchanged (d = .52, p < .05). With regard to the problem solutions, large effect sizes were observed for codes related to translating the relational expressions in the problem statement (codes 2 & 3 above) and selecting area as the appropriate schema for the problem (code 6). Strong visualizers were more likely than weak visualizers to be successful in gathering the ingredients needed to write the quadratic equation and to then write this equation.