The Visual Context – Basic Assumptions

3. The Use of Visual Context for Language Processing

When communicating with other people, most of the time we are aware of the visual world that surrounds us. We know where in the world we are situated, we know the communicative setting we are in and we perceive – be it consciously or unconsciously – our immediate visual context even as we are speaking or listening. Moreover, often when we are talking or listening, we make use of this visual information directly for example by referring to something we see, by using visual information as a replacement for a linguistic utterance, or by taking advantage of the visual information to facilitate language comprehension and linguistic ambiguity resolution.

3.1 The Visual Context – Basic Assumptions

Despite this intuitively strong association between a visual context and our understanding and processing of language, as a discipline, psycholinguistics in the past has widely neglected that language comprehension is situated in a rich non-linguistic context and was mostly purely language-driven (Knoeferle, 2015a).

However, during the last 30 years studies have corroborated the view that the linguistic and visual information with which our senses provide us can overlap partially and cannot always be strictly separated. Processing sensory information does not happen discretely but rather continuously. Moreover, our saccades are tightly coupled to our linguistic processing when visual context is present. The signal streams of vision and language are roughly equally important and thus also constrain each other continuously and in real-time (Anderson, Chiu, Huette, & Spivey, 2011).

Using a paradigm dubbed the Visual World Paradigm (Tanenhaus et al., 1995, see also the review of visual-world studies in Sections 2 and in this Section) we can investigate the real-time influence language has on the visual context and vice versa.

Eye fixations are recorded while participants inspect a visual scene and listen to words, sentences or instructions. While doing so, participants tend to rapidly establish a link between the visual and the linguistic input. Moreover, they even do so given complex scenes, high referential density and a fast speech rate. This rapid interplay between language and vision suggests that the linking between the visual and the linguistic input is not stimuli-dependent but a property that is fundamental for the language system (Andersson et al., 2011). This fundamental feature of language understanding makes the use of the Visual World Paradigm for research on on-line

!

language comprehension especially useful. Here, the visual context is readily available for the language user while spoken language is processed in real time.

Furthermore, the linguistic input presented to the subject directs him / her to interact with the visual context, which in turn becomes highly relevant for the comprehension process. As the linguistic message unfolds moment by moment, the listener makes use of this information to reduce referential ambiguity until the linguistic input is specific enough to determine the intended referent (Eberhard, Spivey-Knowlton, Sedivy, &

Tanenhaus, 1995). While doing so, the eye-movements the participants launch can be time-locked to the auditory linguistic input, as they are preceded by a shift in attention towards the assumed visual referent. Thus, we can determine when the auditory input triggered a saccade and which part of this input is responsible for the shift in attention (e.g., Eberhard et al., 1995; Tanenhaus & Spivey-Knowlton 1996; Tanenhaus et al., 1995).

Even though we know that our eye-movements are tightly coupled to, and provide an index of cognitive, linguistic and visual processing in real time (see e.g., Henderson, 2003, Rayner 1998, Rayner 2009), few psycholinguistic accounts attempt to explain language processing in real time and additionally integrate the visual context. One of these few accounts is the Coordinated Interplay Account (CIA, Knoeferle & Crocker, 2006, 2007). However, before we engage in a more in-depth discussion about different language processing accounts and their limitations regarding the integration of the visual context, we must first define what we mean by visual context.

3.1.1 Defining the Visual Context

In order to be used in language comprehension, the visual context has to be available to the comprehender while the linguistic input is processed on-line. This linguistic input then allows the listener to act on, react to, or interact with the visual information. Thus the context is relevant to the language comprehension process in that linguistic information can, for example, be used to constrain which referent (of a set of potential referents) is used. Hence, unambiguous reference is established as soon as the linguistic input has provided enough information to correctly select the intended referent (Eberhard et al., 1995).

!

However, because our visual working memory capacity is limited to about 4 items (Luck & Vogel, 1997), we might not always be able to take all the visual information that is available into account. These items do not have to be individual features, such as colors or shapes but can also consist of integrated objects (Luck & Vogel, 1997).

Depending on the kinds and complexity of the given visual information, we might assume that our brains prioritize some cues over others (as e.g., results by Knoeferle

& Crocker, 2006 suggest that visually depicted cues are preferred over stereotypical knowledge of role relation, see also Section 3.2 for more details) in order to choose which visual cues to integrate into the linguistic input.

Moreover, a set of objects in our visual field is not what we usually would call a visual context in the real world. We rather see different kinds of scenes in front of us.

Henderson and Hollingworth (1999) described a scene as semantically coherent.

According to these authors it is a human-scaled view of the real world and consists of background elements and objects that are spatially arranged with these background elements. Additionally, a well-formed scene has to conform to basic physical and semantic constraints that structure the environment (Henderson & Ferreira, 2004).

Moreover, real scenes often consist of many different kinds of objects. These objects are sometimes very likely to be found in this particular setting (a kettle is, for example, more likely to be found in a kitchen than in a bathroom), but they can also be unpredictable (Andersson, Ferreira, & Henderson, 2011). Just imagine finding your lost car keys in the fridge after you have already given up looking for them in their usual place (e.g., the key holder).

Furthermore, natural real-world scenes are not stable, but are dynamically changing. Objects might be present at one point in time and gone the next moment.

Moreover, when language comes into play and objects in the scene are referenced in speech, language will select only some objects. These objects continue to be relevant while others lose importance. Hence, the comprehender must filter out the irrelevant information, decide what can be used most efficiently from the scene and integrate this information into the process of understanding the linguistic input (Andersson et al., 2011).

!

3.1.2 Present Working Definition

In order to arrive at a more detailed understanding of how language users integrate visual and linguistic information, most psycholinguistic experiments investigating the interface between language and vision take place in a highly controllable laboratory setting. The scenes that the participants of these experiments usually look at while listening to language thus often differ in complexity and naturalness from real-world scenes. In this regard, Henderson and Ferreira (2004) distinguish between a ‘true scene’ and an ‘ersatz scene’. Whereas a true scene is a complete scene (either in the form of a real environment or in the form of a depiction of a scene), an ersatz scene is an incomplete (depicted) scene, which lacks some of the critical characteristics of a normal environment in the real world. These differing and / or missing characteristics in an ersatz scene could for example be a missing background or unusually large depictions of actors in the ersatz scene.

However, even these ersatz scenes, which are often used in psycholinguistic studies, can convey a sense of coherence and do not necessarily have to be composed of discrete and individual objects. Hence, most of the visual scenes used in experiments fall somewhere on a continuum between arbitrarily arranged objects in a visual display and true (depicted) scenes, such as when people are depicted as agents and patients which are actively engaged in an action (Henderson & Ferreira, 2004).

The underlying assumption of using simplified scene depictions is that they are good substitutes of natural real-world scenes. These depictions can capture important properties of real-world objects and scene perceptions like the complexity of the scene, semantic coherence and constraints, and scene meanings, while at the same time allowing the researcher to control for factors that cannot easily be controlled in the natural world and that could lead to confounds due to many possible and uncontrollable influences (Henderson & Ferreira, 2004).

One set of examples of such ersatz scene depictions with semantic coherence and scene meanings are agent-action-patient depictions. Participants can reliably and very quickly extract the necessary information from a given depicted scene that allows them to identify thematic roles if the depicted characters possess typical agent- (Hafri, Papafragou, & Trueswell, 2013), patient-, and action- characteristics. They can moreover use this scene information to correctly interpret a spoken sentence in resolving sentence initial role ambiguities by quickly integrating scene and linguistic

!

information on-line (Knoeferle et al., 2005). The present studies (see Sections 6-9) will make use of these depicted ersatz scenes as a visual context similarly to Knoeferle et al. (2005) in order to investigate the assignment of thematic roles during language comprehension.

However, (depicted) true and (depicted) ersatz scenes are not the only kind of visual contextual information that can be used in order to gain insights about the use of visual information for language processing. More often than not we also use visual information that is not necessarily present in the related scene to interpret linguistic utterances, such as the facial expression (Carminati & Knoeferle, 2013) or the gestures (Wu & Coulson, 2005) of our interlocutor who describes the scene. We know that we extensively use all kinds of available visual information to shape and facilitate the interpretation of the linguistic input (regardless of whether that information is part of a scene or not). Nevertheless, the questions of why we use certain types of information from the visual context as opposed to others or why we prioritize some visual cues whereas others in the same visual field would also have been helpful is not as straightforward as it may seem. These questions are thus addressed further in the next Section (3.2).