General Method

1.1. Selected Tasks

Within the field of semantic cognition, there are several methods to investigate semantic relations among concepts. As stated above the main underlying idea is that units in the conceptual system are organized and activated according to an associative network. An important goal of the research described in this dissertation is to investigate whether the strengths of semantic relations between paradigmatically and syntagmatically related concepts are different in subjects who use sign language as a significant means of communication. Thus, it was important to employ tasks, which measure the strength of semantic relations of a specific type in detail.

It was decided to use comprehension tasks rather than production tasks, because the responses of the participants can be registered on-line in terms of reaction times and choices of preference. A simple comprehension task is to ask subjects to decide whether a declarative statement is true or false with respect either to general knowledge or whether it fits to a specific visual display. The starting point for this kind of research was the sentence verification task developed by Landauer & Freedman (1968) and Collins & Quillian (1969). In this task a person is presented with simple propositions like "a bird has feathers" or "a fish can swim" and asked to determine as quickly as possible whether a sentence is true or false. For the purpose of this thesis a variant of the sentence verification task - the category verification task – was selected. In this task subjects are given particular signs or words such as ‘Rose’, ‘Apfel’ or 'Taube', and asked to decide whether or not they are related in a certain way to other signs or words (Rosch 1973;

McCloskey & Glucksberg 1978). The underlying idea is that when two concepts are stimulated, activation from each spreads throughout the network until the two concepts get linked. Since normally few errors are made on this task, the response time taken to answer is usually what is measured. This time is thought to reflect the strength of semantic relations between concepts in the semantic lexicon. That is, even though decisions are made very rapidly, the time it takes might be a measure of the strength between different concepts in the semantic system.

1 The studies were part of an interdisciplinary research project at the 'Center for Cultural Research' (University of Cologne) on the 'Mediality of linguistic signs'.

A second class of comprehension tasks is not based on the measurement of response time but rather on the subjective judgment of the degree of semantic relatedness between words or signs. Studies within this approach have taken as a point of departure ratings of similarity in meaning for subsets of the lexicon that form specific semantic domains such as cooking, animal names etc. (Rosch 1973). The basic assumption is that the structure of the internal representation is reflected in this intuitive judgment. In this thesis a variant of the similarity-rating-task – the triad-comparison-task - is used, in which subjects have to decide which one of two instances of a category is more semantically related to a third one, e.g. ‘apple’ and ‘fruit’ are stronger semantically related than ‘banana’. The number of choices (%) are applied to define the strength of relatedness.

Another paradigm in semantic relation studies investigates memory errors in word recognition tasks. In the single-item task, a conventional recognition memory task, participants study a list of words or signs and are tested for their ability to recognize the individual items as previously presented, i.e. as old. There are two types of recognition memory errors, i.e. false hits (or false alarms) and misses. A false hit is the recognition of an item as old when it is in fact new.

A miss is the failure to identify a previously presented item as old. This task can be used to measure semantic relatedness because the two types of errors give an indication of the strength of the relation between two items. For example, Underwood (1965) and others used a continuous recognition memory paradigm and found that subjects made false recognition of words (false hits) that were semantic associates of previously presented words. This false recognition effect was influenced by the semantic relatedness between words, with the highest false memory frequency reported for the words with the strongest semantic relations to the previously studied words.

The described on-line psycholinguistic experiments are very sensitive to experimental variables. Thus, all types of methods, i.e. verification task, triad-comparison-task and memory-recognition task, are utilized in this thesis, in order to study the influence of language modality on conceptual structures.

In Table III.1 an overview about all experiments and methods is given:

Table III.1: Selected tasks

Experiment Method

Verification Task with Signs/Words and Pictures Verifying the semantic relatedness of two single signs/words or pictures

Triad-Comparison Task with Signs/Words and Pictures

Choosing which item out of two is more semantically related to a specific item Verification Task with non-polycomponential and

polycomponential signs

Verifying the semantic relatedness of two single signs

Triad-Comparison Task with

non-polycomponential and non-polycomponential signs

Choosing which item out of two is more semantically related to a specific item Recognition Memory Task with

Nonpolycomponential and polycomponential signs

Memory-recognition: Present different signs in a first trial and ask for recognition in a second one.

Verification Task with iconic Signs/non-iconic Words and Pictures

Verifying the semantic relatedness of a sign/word and a picture

Triad-Comparison Task with iconic Signs/non-iconic Words and Pictures

Choose which picture out of two is more semantically related to a sign/word

Rating task Sign-Iconicity Rating

1.2. Participants

Participants were deaf, bilingual and hearing volunteers who were paid for participation in the experiments. All of the deaf participants were prelingually and profoundly deaf, and they came from hearing families. They were skilled signers and reported German Sign Language (GSL) to be their preferred means of communication with deaf and hearing friends. The bilingual participants were hearing native cross-modal bilingual signers. They learned both German Sign Language and German Spoken Language as a first-native language from their profoundly deaf parents and hearing relatives (CODAS- Children of deaf Adults). Since the bilinguals were all working as sign-language interpreters they used both sign-languages regularly as a means of communication (balanced speech-sign bilinguals). The hearing participants were native speaker of German and had never had any contact with German Sign Language.

All participants had equally high linguistic proficiency in each of their languages and came from the same area in Germany (Nordrhein-Westfalen). Linguistic proficiency of the deaf and the bilingual group in German Sign Language was assessed by a trained deaf sign language lecturer using a subtest of the ATG (Gebärdensprachsinnverständnistest (GSV) of the Aachener Test zur Deutschen Gebärdensprache, Fehrmann, Huber, Jäger, Sieprath & Werth, 1997). Linguistic proficiency of the bilingual and the hearing group in German spoken language was tested by a hearing psychologist with a standardized interview.

1.3. Materials

The material of the experiments consisted of signs, words and pictures, respectively. These stimuli fulfilled the following criteria:

The signs were performed by a deaf native speaker of German Sign Language (GSL), the words were spoken by a native speaker of German Spoken Language (GSpL), and the pictures were drawn by the same artist in order to insure similarity in art and style.

The German signs were required to correspond to German spoken words and thus to be easily translatable (the signs used were all commonly accepted single signs of German Sign Language). It was essential that signs and words were well-known to the participants (subjective frequency of occurrence of signs and words, respectively, was estimated with the computerized procedure VEIP; Grote, 1999). An attempt was made to keep familiarity, iconicity, number of syllables, phonological and morphological similarity as comparable as possible.

A recorded sign was considered to begin when the hand(s) entered signing space and to end when the hand(s) began to move out of the sign configuration and back down to the resting position. The signs were gated at frame intervals of 33 ms.

The spoken words were digitized at a sampling rate of 22 kHz. They were presented acoustically together with a visual marker.

The pictures were black and white line drawings, with the exception of pictures for color-words and -signs (e.g. green, blue, red etc.), which were colored. The pictures were digitized using a Hewlett-Packard scan application, and refined with the editing facilities of MS-Paint. The pictures` objects had a canonical orientation and were presented in black on a white background. The complexity of the pictures differed and since this difference had an impact on recognition and response times, picture-complexity was judged by the participants with the VEIP. (See Appendix for detailed information about videos, audios and pictures).

1.4. General Procedure

All subjects were tested individually in an experimental laboratory at the University of Cologne.

The participants were seated directly in front of the monitor at a distance of about 60 cm with the two index fingers resting on two response buttons of a so-called 'Game Port Checker'. They were informed that they would see several pairs of items. The pairs of items were grouped in different series consisting of a certain number of trials and rest periods. The signs and pictures were

two loudspeakers attached to the computer. In addition to the acoustic presentation of the word, a visual marker (icon of a loudspeaker) appeared on the screen indicating to the subject, which item was the target and which was the stimulus item. The outlines of the sign video, the visual marker and the picture were the same (approximately 12 cm wide and 9 cm high). In order to prevent visual masking, the first item was centered at the top of the monitor, the second one at the bottom. Each item was followed by a grey, blank screen. Reaction times were measured from the onset of the stimulus item. The participants were told to perform the task as accurately and quickly as possible. They could respond anytime from the start of the target item, but fast responses did not alter the inter stimulus interval (ISI). The subjects were instructed by using a computer-based explanation (signed and written vs. spoken and written) and were given practice trials.

The experiment was conducted by a program developed at the University Cologne² and allows measurement of the speed of a human subject's responses to stimuli flashed on the computer screen. Number of trials, inter-trial intervals, ready signal, and randomized foreperiods can be selected from an interactive display (Ini-Data) prior to beginning the experiment. The program does not record premature responses or excessively long latencies. In such cases the program records a non-response. At the end of the selected number of trials, the available data analysis options include type of reaction (yes/no or left/right) and Reaction Time (RT). Raw data can be displayed and a data file is stored automatically. More trials may be added to existing data and the starting point of the experiment can be varied.

All analysis of variance (ANOVA) were carried out as repeated measure ANOVA with task related factors (Semantic Relations, Experimental Condition), as repeated factor and groups as nonrepeated measure factor if not stated otherwise. For all analysis reported, outliers were removed prior to the ANOVA. An outlier was defined as a response time that was two standard deviations from the mean in a given cell for a given subject. Two separate ANOVAs were conducted on subjects and items. Based on these two analysis, min F' was calculated if both subject and item analysis were significant.

2The experiments were carried out on a PC with Intel Pentium II Processor, Intel MMX™ Technology (256 MB RAM).

The software was written in Visual Basics 6.0 with additional High Speed Timer programmed on the basis of Visual C 6.0 and implemented in the Visual Basic program. The control elements for the AVIS (sign-videos), i.e. MS ActiveMovie controls and for sounds (word-audios), i.e. MS Multimedia Sound Controls delivered the responses of the subjects in real time (measurement accuracy = 5 ms) to the High Speed Timer. All controls used DirectX Library. This meets the demands of a Real Time System. The stimuli were displayed on a high-resolution CRT-screen (Iyama Vision Master Pro 510, 21'') with a screen update frequency of 60 Hz and Standard-VAG-Resolution. Responses were recorded via an external manipulandum. A Game Port Checker initialises direct input so that the polling happens with the highest possible frequency.