Using Simultaneous Verbalisation to Interpret Gaze

The present study follows the McNeillian tradition of interpreting non-verbal behaviour through co-occurring speech. Gestures are “the spontaneous, unwitting, and regular accompaniments of speech that we see in our moving fingers, hands and arms” (McNeill, 2005, p. 3). McNeill (1985) has long contended that gestures are closely tied with our speech. That is, gestures are united with speech during the message transmission process. McNeill distinguishes gesture from language on three levels (McNeill, 2005). In meaning, gestures are global whereas speech is analytic. In creation, gestures are

spontaneously generated in the moment whereas words are formulated in conformity with established rules. In direction, gestures are driven by imagery whereas words are driven by society’s arbitrary rules of word morphology. Together, gestures contrast with words by being instantaneous, global and unconventional. Gestures are therefore the perfect counterpart to spoken language, making gesture and speech well-suited to the

interpretation of each other.

As fully formed language systems, sign language has been used to highlight the close relationship between linguistic and non-linguistic hand movements and, in turn, speech and gesture. Duncan (2005) asked fifteen signers to tell an action-packed story— Canary Row (Freleng, 1950) using their native language, Taiwanese Sign Language. As

participants told this story, hand movements were used to tell the story (i.e., signing) as well as to demonstrate the story’s actions (i.e., enactments). For example, one signer switches from narrating the story to providing an image of it, by putting “the left hand in the half circle form… the right hand in the thumb-and-pinky ‘vertical figure’ handshape. The right hand then moves through and up past the left hand” to show the cat moving through the drainpipe (Duncan, 2005, p. 301). Sandler (2009) too contends for the close relationship between the linguistic and the non-linguistic (or gestural) using Israeli Sign Language, whose culture involves mouth-pointing in spoken discourse. Whereas speakers use hands to supplement the linguistic content of their mouths (i.e., speech), signers in Sandler’s study used the mouth to supplement the linguistic content of their hands. For example, in retelling the story of Canary Row, while narrating with their hands, signers used their mouths to convey the tightness of a drainpipe, the zig-zag shape of the drainpipe and the roundness of a bowling ball. There is a sense that gestures—non-linguistic

behaviour that accompanies linguistic content—are indispensable, such that, even when hands are unavailable for gesturing, gesturing is nonetheless utilised as part of the message-giving process.

The close relationship between speech and gesture has additionally been demonstrated by the synchrony between speech and gestural planning (or the ‘growth point’, McNeill, 1992). That speech and gesture (at least the planning) occur at the same time suggests that they operate in the same mechanism, as one integrated image-language system. To explore this, Church and colleagues distinguished between gesture and action by allowing participants to use objects as part of their activity (e.g., dart-throwing)

description only in the speech-and-action condition (as opposed to the speech-and-gesture condition, Church, Kelly & Holcombe, 2014). By making this gesture-action distinction, Church found significantly greater speech–movement synchrony in the speech-and-gesture

condition, with movement onset. That is, the start of gestured movements coincided with the timings of speech content more than action movements did with speech.

In another study on the integrated system of speech and gesture, Kelly and colleagues used an ‘incongruence paradigm’ to demonstrate that, rather than leaving the spoken message uninterrupted, mismatched speech and gesture disrupts the recipient’s understanding of the message as a whole (Kelly, Özyürek & Maris, 2009). That is, by disrupting the gestured content, the spoken content becomes less accessible, even

distorted. The incongruence paradigm distinguished gesture that were weakly incongruent with speech from those that were strongly incongruent with speech. The incongruent conditions together were compared with the baseline, congruent, condition that had no speech-gesture incongruence. After being shown a video of an action prime (the target behaviour, e.g., ‘chop’), participants watched a video of one of these speech-gesture conditions. Participants were tasked to identify whether the target behaviour was present, yes or now, while their reaction times and error rates were measured. Significantly faster reactions and fewer errors occurred in congruent (baselines) conditions; in the incongruent conditions, reaction times and errors rates increased whether disruption occurred in the speech or the gesture. It was thus shown that speech and gesture are comparably

important in message-giving and that the two modalities belong in one integrated system. Gaze has long been treated as a sub-category of gesture, as researchers have

attempted to decode non-verbal interactions. In a seminal study, Kendon (1967)

documents the way gaze direction and speech work together to show that differing speech content co-occurs with differing gaze directions that are sustained for differing durations. In the dyadic conversations, interlocutors looked away during longer speeches to indicate hesitation, as if to obtain planning time, and looked toward the listener during fluent speech, as if to invite prolonged attention from the listener. As such, gaze direction

partners with speech to achieve the overarching goal of the speaker—to generate planning space, or to command attention. Kendon thus suggested that gaze direction functions like gesture: synchronised with speech timing and content, supporting conveyance of specific messages, emerging from the same unified mechanism—much like the integrated system of hand gesture and speech (Kelly et al., 2009).

In further support for the comparability of gaze with gesture, Jokinen (2009) suggested that gaze and gesture have comparable functions in conversation. During conversation, listeners’ gaze direction, especially eye contact, played a synchronised role with gesture and body posture in non-verbally conveying responses to speakers. For instance, maintaining eye contact was part of the overall non-verbal demonstration of empathy and willingly sustained attention towards the speaker. Jokinen contended that gaze is one part of non-verbal, gestural, support system for speech. Gaze seems to be used together with hand gestures to feed unified non-verbal messages to interlocutors.

That gaze, gesture and speech work closely together suggests that gaze can be analysed in the same way as gesture. Quek and colleagues asked individual participants to explain a given plan of action to a group of listeners, who needed to understand the plan in order for the participant to perform their task successfully (Quek et al., 2000). The most consistent function of gaze direction, especially eye contact, in this task emerged to be checking and maintaining listener understanding. Meanwhile, hand movements were used to point to geographical parts of a display board, thereby having the function of indicating the main point of interest. Similarly, Sidnell (2006) suggested that gaze supports gesture and speech by organising the other two modalities during message-giving. In his analysis of a group conversation, involving a re-enactment of multiple characters with differing perspectives, speakers assumed body movements that appropriated the character they were assuming. Simultaneously, gaze directions also appropriated those of the re-enacted

character, averted away from the conversational partners—even though the whole message (or re-enactment) was for the audience of these conversational partners. In this way, research has shown that gaze and gesture work with—indeed, complement—each other, alongside speech in a co-active manner (Jones & LeBaron, 2002, p. 503).

Neuropsychology research has also given support to the perspective that gaze and gesture are equivalent in the integrated gesture-speech system. In this research, the Grèzes laboratory uses amygdalic reactions as a measure of social judgement, that is the detection of social intention in viewed stimuli. Conty and colleagues showed image sequences of gaze and gesture to explore the differing extent of social intention detected (i.e., amygdalic response) through differing timings of gaze and gesture (Conty, Dezecache, Hugueville & Grèzes, 2012). Authors found that the strongest detection of social intention (e.g., threat or anger) occurred when viewers received combined cues of gaze (i.e., direct gaze) and gesture (i.e., pointing) rather than individual cues involving either gaze or gesture alone. In particular, gaze direction seems to enable the addressee to determine whether the viewed behaviour is relevant to oneself (Conty et al., 2012, Grèzes, Adenis, Pouga & Armony, 2013), such that angry messages only trigger associated neurological responses in viewers when they are accompanied by direct gaze (or eye contact). Moreover, the Grèzes team (laboratory study, Grèzes, Valabregue, Gholipour & Chevallier, 2014) and others (meta-analysis, Hinojosa, Mercado & Carretié, 2015) have also emphasised the overall role of the motor and visual regions of the brain in amygalic activations, whereby the physical movements of others are integrated to make judgements of their social

intentions possible, lending further support to the relationship between message-giving (or message-reading) and gestures (or gesture-reading).

In view of the above evidence for the shared responsibility of gaze and gesture in supporting message transmission, and given that gesture and speech seem to be tightly

linked, the way gesture and speech interpret each other (McNeill, 1985, 2002, 2005) can be extended to expect gaze and speech to interpret each other. Accordingly, the present thesis distinguishes between broad interpretations of gaze using speech that occurs simultaneously with gaze.