continuous grammaticality judgements
6.6.1. Introduction
Experiment 4b was designed to test the effects of violating semantic selection restrictions and of pragmatic anomalies with a task that would discourage subjects from pacing through the sentence, as might happen in the self-paced paradigm used in Experiment 4. In this task (which was also used in Experiments 6 and 7) subjects have to use one of two buttons: a Yes-button, which they are instructed to press when they judge the sentence, up to the current word, to be grammatical, and a No-button, when
they judge the sentence to be ungrammatical. More details of the task and the procedure are presented with Experiment 6.
In this experiment, syntactically violated sentences were used in addition to the normal, semantically violated and pragmatically-odd sentences of Experiment 4. This was done in an attempt to assess the types of information subjects use in making the continuous grammaticality judgements required in this self-paced paradigm. A typical sentence quartet is given in Table 6.9. below.
Table 6.9.: Example stimulus set for Experiment 4b normal:
In the laboratory the chemist thickened the liquid and searched in the cupboard for the right container.
sem-viol.:
In the laboratory the chemist thickened the teacher prag-odd:
In the laboratory the chemist thickened the gravy and searched in the cupboard for the right container.
synt-viol.:
In the laboratory the chemist thickened the too
If making a grammaticality judgement about the sentence presented so far slows down processing (as compared to the self-paced task used in Experiment 4), pragmatic violations should have an effect on the direct object noun itself: the time to make a YES-decision should be longer in the pragmatically odd than the normal condition, even though the object nouns of both conditions are correct syntactically and in terms of the required semantic features. Such a difference would indicate that subjects have build up a representation of the sentence context, and that they cannot disregard pragmatic plausibility even when the task is to judge the grammaticality of a sentence. The number of incorrect No-decisions should not differ for these two conditions.
A comparison of the number of correct NO-decisions in the semantically-violated and the syntactically-violated conditions gives important clues as to how subjects
understand the instruction to judge the grammaticality of a sentence. They were told in the instructions that sentences are ungrammatical "if they are incorrect in any
conceivable context". Furthermore, the practice sentences provided them with feedback (see Procedure section in Chapter 8.3.). Neither the semantically nor the syntactically- violated sentences continued after the violating word. It was expected that the NO- decisions on this word would take longer in the semantically-violated than the
syntactically-violated condition. In the semantically-violated condition, subjects might try to reinterpret the semantically incorrect object noun in some way which would allow them to ignore the violation of the semantic selection restriction. In the
syntactically-violated condition, however, the determiner requires the occurrence of a noun (or adjective or adverb); no word of any other syntactic category can be
reinterpreted to fit the sentence grammatically.
6.6.2. Method
Design
Sentence Type is a repeated measures factor in the by-subjects and the by-items design. It has four levels, i.e. normal (the direct object noun fits the verb semantically and pragmatically), semantically violated (the direct object noun violates semantic selection restrictions of the verb), pragmatically odd (the direct object noun fits the semantic selection restrictions but is unlikely as established in the pretest for
Experiment 4) and syntactically violated (the word following the determiner violates syntactic phrase structure rules). See Table 6.9. above for example sentences.
Four parallel lists were created using a Latin Square, such that each subject would only see one of the sentences from a sentence quartet. Each subject saw five sentences in each of the four conditions, and each sentence was seen by 11 subjects per condition.
Stimuli
Sentence sets were identical to the ones used in the previous Experiment 4, but an additional 5 sentence sets were created resulting in a total of 20. To each of the sentence triplets used in the previous self-paced experiment, a fourth, syntactically violated, sentence was added. In the syntactically violated condition, the critical noun was replaced by a word of a syntactic category that cannot grammatically follow a
determiner, i.e. a verb, a determiner, a pronoun or a preposition. All sentences included in the present experiment had at least six words following the critical noun. A few further minor stylistic changes were made to the sentences, which are indicated in Appendix H in brackets.
The sentences were derived from the stimulus set used in the selection-restriction experiment (Experiment 2). Pretests were run to ensure low Cloze values of the direct object noun in the normal condition, and to establish the naturalness of the sentences in the normal and pragmatically odd conditions (see previous chapter for details).
In the semantically- and the syntactically violated conditions, the sentence discontinued after the ungrammatical words. In the normal and pragmatically-violated conditions, sentences continued after the critical noun. These sentences became ungrammatical a few words later on.
There were 79 filler sentences of various sentence structures, unrelated to the purposes of this experiment, and 13 practice sentences of various sentence structures. There were three practice sentences which were similar to the sentences in the experimental conditions of the present experiment.
For a full list of stimuli, see Appendix H.
Subjects
44 subjects were recruited from among the staff and students of University College London.
6.6.3. Results
Semantically violated and syntactically violated conditions
Data in the two violated conditions were only analyzed on the word which introduced the violation (i.e. the semantically violated noun, or the syntactically impossible word following the determiner after the verb). Paired t-tests indicated that the average time to make a correct No-decision was longer in the semantically-violated (2394 ms) than the syntactically violated condition (1382 ms) [by-subjects: t(40)=6.61, p=<.0005, 2- tail; by-items: t(19)=4.12, p=.001, 2-tail]. Note that three subjects made only incorrect Yes-decisions in the semantically violated condition. Their values were set to missing in this analysis of the correct No-decisions.
The number of incorrect Yes-decisions differed in the two conditions: 41 % of the responses in the semantically-violated condition were an incorrect Yes-decision on the semantically violated direct object noun, as compared to 15.9% of incorrect Yes- decisions in the syntactically violated condition (significant in the by-subjects and by items analysis, Wilcoxon Matched-pairs Signed-ranks Tests: p’s <.0004, 2-tail). The incorrect Yes-decisions in the semantically-violated condition did not come from only a small set of subjects (on average, 2.07 incorrect Yes-decisions were made per subject; only 4 out of the 44 subjects did not make any incorrect Yes-decisions). Neither were the incorrect Yes-decisions limited to only a small set of sentences (the average number of incorrect Yes-decisions per item was 4.55; no item received no incorrect Yes-decision).
Paired t-tests indicated that the average time to make an incorrect Yes-decision in the semantically violated condition (1468 ms) differed significantly from the time to make a correct Yes-decision in the normal condition (822 ms) [by-subjects: t(39)=2.60, p=.013, 2-tail; by-items: t(19)=2.67, p=.015, 2-tail]. Note again that there were four missing values in the by-subjects analysis: 4 subjects did not make any incorrect Yes- decisions in the semantically violated condition. This difference between the normal and the semantically-violated condition was not significant once extremely long times
for making Yes-decisions were taken out before subject-and item-means were
computed (extremely long values were defined as those greater than 1555 ms, which is the cut-off established on the basis of the distribution in the normal condition). Note that the number of responses in the semantically violated condition for incorrect Yes- decisions was quite small once the extremes were taken out: 22 % of the incorrect Yes-responses in the semantically violated condition were extremes (these responses came from 16 different subjects, and were distributed over 13 of the items), as compared to 1% of the conect Yes-responses in the normal condition. Subject-means for this reduced data set in the semantically violated conditon were calculated on average over 2.2 responses, and item-means over 3.8 responses (in the normal condition, the corresponding numbers are 4.6, and 10.2, respectively). In the by subjects analysis there were 11 missing values, i.e. for 11 subjects no data points were left over which to calculate means.
Normal versus pragmatically odd conditions
There were no incorrect No-decisions on the direct object noun in the normal condition, whereas 8% of the responses in the pragmatically odd condition were incorrect No-decisions (they were made by 13 subjects, on 9 different items).
The time to make a correct Yes-decision on the direct object noun was significantly faster in the normal (822 ms) as compared to the pragmatically odd condition (1280 ms), as indicated by paired t-tests [by-subjects: t(43)=7.33, p<.0005, 2-tail; by-items: t(19)=7.12, p<.0005, 2-tail]. The percentage of extremely long decision times (again defined on the basis of the distribution of decisions in the normal condition), i.e. responses greater than 1555 ms, was 7% in the normal condition and 28 % in the pragmatically odd condition. In the normal condition, extremely long responses were distributed over 11 items, and 12 subjects. In the pragmatically odd condition, the extremely long responses were distributed over 31 subjects, and 18 items. When the extremely long responses were removed from the data set before subject-and item- means were computed, the difference between the average decision times for the
normal (733 ms) and the pragmatically odd conditions (882 ms) still differed
significantly, as indicated by paired t-tests [by-subjects: t(43)=4.70, p<.0005, 2-tail; by items: t(19)=3.75, p=.001, 2-tail].
6.6.4. Discussion
Experiment 4b showed effects of Sentence Type immediately on the critical word, unlike Experiment 4, in which the effect was manifest only on the following word and beyond. In Experiment 4b subjects were slowed down by the additional task of having to make a grammaticality judgement on every word of the sentence (response times were about twice as long as in the self-paced task used in Experiment 4). This seems to have provided sufficient time for building up a representation of the nonsyntactic representation of the sentence. Subjects were slower to judge a sentence as
grammatical when the direct object noun was pragmatically odd, given the sentence context, as compared to the normal condition. This increase in processing time was not due to only a few extremely long responses but to a shift of the distribution of
responses in the pragmatically odd condition.
As expected, the time to judge a sentence as ungrammatical was longer in the semantically-violated than the syntactically-violated condition. This probably reflects extra time subjects spent trying to find a possible interpretation of the semantically violated sentences, whereas the syntactic violation signalled more clearly that no grammatical interpretation would be possible.
The results also showed that subjects made more No-decisions in the semantically violated than the pragmatically odd condition (59% versus 8%, respectively). The number of Yes-decisions in the semantically violated condition, however, was quite high, i.e. 41% of all responses in this condition. The time to make such Yes-decisions was longer than the time to make a Yes-decision in the normal condition. The finding that this difference was not significant once extremely long decision times had been taken out should be interpreted with caution: the number of observations over which
subject-and item-means were formed was much smaller in the semantically violated condition (there were 91 Yes-responses, of which 20 were extremely long; for the non extreme data, subject-means were calculated on average over 2.2 responses, and item- means over 3.8 responses) than the normal condition (of the 220 Yes-responses there were 15 extremes; subject-means were formed on average over 4.6 responses, and item-means over 10.2 responses). Nevertheless, these non-extreme response times for making incorrect Yes-decisions on semantically violated nouns probably reflect those cases where the grammaticality judgement is made without an analysis of semantic selection restrictions. Note that these cases only make up 32 % of all (220) responses to semantically violated object nouns.
The few cases of taking an extremely long time to make an incorrect Yes-decision on the semantically violated object noun (22% of all incorrect Yes-decisions in the semantically violated condition, but only 9% of all the responses in this condition) indicate that subjects sometimes noticed the semantic violation but still decided that the sentence was grammatical. This might be because they used an implicit criterion according to which only syntactic violations render a sentence ungrammatical.
Alternatively, they might have succeeded in constructing some metaphorical context in which the violation of semantic selection restrictions had been resolved.
CHAPTER 7 : THE USE OF NON-SYNTACTIC INFORMATION IN