Findings related to the Interpretations of the AMS instructions

affected answer length theme

There were 2 codes related to the Interpretations of the AMS instructions affected answer length theme, and these are presented in Table 5.5 below. This theme relates

to the lengths of the answers to the AMS questions given by the participants, and it was coded 25 times overall. In addition, to facilitate the discussion of this theme,

the mean answer length in words for each of the participants in the study is shown in Figure 5.4 below.

Code Number of times

coded When responding to the information about answer

length, some participants made their answers shorter (C6)

11

When responding to the information about answer length, some participants made their answers longer (C7)

14

Table 5.5: Codes associated with the Interpretations of the AMS instructions affected answer length theme.

The mean length of the answers to the free-response questions varied considerably by participant, as captured by codes C6 and C7. Participants P1, P5, P7 and P8

gave shorter answers, whereas participants P2, P3, P4 and P6 gave longer answers. Some of the reasons for this variation were explained by the participants’ interview

responses. Participants P1 and P3 both deliberated over the level of detail required in their answers, with P1 settling on giving shorter answers and P3 choosing to give

longer, more descriptive answers:

“I felt that I’d better not just give a one-word answer, ‘neither’, although, I suppose

that logically, that should have been adequate. But, I thought I’d better smooth it out a little bit” [P3] (P3 was referring to their answer to Q1. They did indeed type the full

answer “Neither - ignoring air drag, both should hit ground at same time”).

Participant P2 claimed that they wanted to make their answers shorter; in spite of this, they typically gave longer answers in the form of sentences. Similarly, Participant

P4 said that they attempted to modify the length of their answers as they worked through the AMS, but actually gave sentence answers in most cases.

Figure 5.4: Graph showing the mean answer length in words by participant. The red horizontal line represents the mean of these mean answer lengths.

No participant attempted to submit an answer longer than twenty words. However,

participants P5 and P7 used the information given in the instructions that answers of more than 20 words would not be accepted to guide their answer length. In the

case of P5, the guidance made them feel as if the answers that they had given to the free-response questions were not detailed enough:

“I think I was worried about being either kind of vague, or, especially because there was a twenty-word limit, sometimes I felt that I would like to say a bit more, and was

worried about how much depth I needed to go into with my response, I guess” [P5] (P5 gave both short and long answers to the AMS questions. For example, they gave the one-word answer “equal” to Q20, but gave the longer answer “when the top block

moves between 3 and 4 at some point they will be the same speed” to Q22).

On the other hand, P7 interpreted the guidance as advising them to write shorter

answers to the free-response questions. In line with this, P7 gave mostly short answers, with several being only one word in length.

In summary, the length of participants’ answers to the free-response questions were found to vary between participants. The data and responses suggested that

none of the participants struggled with the length of their answers, but some were consciously aware of it. In addition, there was evidence that the participants used the guidance provided to determine what amount of detail was appropriate to present in

their answers. In the cases highlighted by code C6, the participants made their answers

shorter, as the guidance had told them that answers of a few words would be sufficient; whilst in the cases identified by code C7, the participants made their answers longer,

alluding to the 20 word limit that they had been alerted to. It is possible that the participants’ reactions to the guidance were examples of regular exam techniques and

study skills. However, it is also possible that these reactions were enhanced because the participants were entering their answers on a computer, rather than on paper. Further data pertaining to the length of hand-written student responses to a paper-

based version of the AMS would be required to investigate this possibility.

Discussion of answer length by question

A consideration of the actual lengths of the answers given to the AMS questions

is also relevant to the Interpretations of the AMS instructions affected answer length theme. The mean answer length in words for each free-response question on the AMS

given by the participants in the study is shown in Figure 5.5 below. Figure 5.5 shows that Q17, Q22, Q30 and Q32 had the longest answers on average, whereas Q3 and Q25

had the shortest answers on average. The remaining 14 questions had short answer lengths, which is in keeping with the objective of having free-response questions that

can be answered with a few words or a short sentence. As a further reflection of this, the overall mean answer length was 8 words, which is just under half the number of

words permitted for each response.

The question with the shortest average length was Q25, which is worded as follows:

“As the rocket moves from position b to position c does the speed increase, decrease,

or stay the same?”

This question is the second in a sequence of four questions based on the scenario

of a rocket in space. Suggested answers are given within the question itself, and these suggested answers were used in all of the participants’ answers. In contrast, Q30 was

the question that had the longest answer on average. It is worded as follows:

“Now, the external force applied by the woman is suddenly switched off. Describe as

accurately as you can what you think happens to the speed of the box after this event.”

This question is the third in a sequence of three questions based on the scenario of a box being pushed. It instructed participants to describe a situation “as accurately as they could”, which is a likely explanation for the lengthy answers received.

Figure 5.5: Graph showing the mean answer length in words by question. The red horizontal line represents the mean of these mean answer lengths.

Figure 5.6: Q17 of AMS

Another question with longer answers was Q17, shown above in Figure 5.6. This

question is the first of a pair of questions based on the scenario of a car pushing a truck. It is adapted from Q15 of the original FCI, which is considered in the literature

to be a challenging FCI question (Scott et al., 2012), which may explain the length of the answers.

The above examples show that the participants gave answers based on the level of detail that they perceived the question to be asking them for. This is a demonstration of the participants acting as conscientious consumers (Higgins et al., 2002), doing

exactly what they think that they are supposed to do, even if that is not what was

actually asked or intended. More generally, the impact of question wording on the responses given has implications for the design priorities for the authoring of questions

in the future.