Gesture as a “Window” into the Mind

The gestures that children produce are not just important milestones for the de- velopment of their communicative abilities, but can also provide insight into how their cognitive abilities are developing. One of the leading views in gesture stud- ies is that gestures can reveal the mental process of the speaker-gesturer (McNeill, 2000). This view is based on the idea that somehow gesture, speech, and thought are inherently related to each other. This section presents four lines of empirical evidence that support this view and explore the nature of the relationship between gesture, speech, and thought.

First, studies in which gesture was elicited without speech provide a window into children’s level of symbolic understanding. For instance, 3- and 4-year-old chil- dren produce iconic gestures that are less symbolic than iconic gestures produced by 5-year-old children. In a study by Boyatzis and Watson (1993), 3-, 4-, and 5- year-old children were asked to pretend to use eight common objects (e.g., “Use your hands to pretend you are brushing your teeth with a toothbrush”), without those objects being present. There was a developmental shift in the symbolic quality of the produced gestures, where 3- and 4-year-olds used their body parts to repre- sent objects (e.g., using an extended index finger to represent the toothbrush) and 5-year-olds used imaginary gestures (e.g., using a hand shape as if holding a tooth- brush). Body part gestures represent salient aspects of the form of an object and were therefore considered less symbolic than imaginary gestures, which represent the use of an imaginary object but not the form of the object itself. In a follow-up experiment, the same 3-year-old children were asked to imitate the experimenter’s gestures, who produced iconic gestures different from the gestures the children pro- duced themselves in the first part of the task. That is, if the child produced a body part gesture before, then the experimenter would produce an imaginary gesture, and vice versa. Three-year-olds found it particularly difficult to imitate an imaginary gesture if they had produced a body part gesture themselves, which suggests that they struggle with gestural representations that exceed their own symbolic level.

Similarly, in a study by Overton and Jackson (1973), children under the age of 6 years pretended to use objects (e.g., a comb) by producing gestures in which their hands took on the form of the objects itself (e.g., fingers shaped as the teeth of the comb) and children over the age of 6 years produced gestures in which they pretended to hold and use objects (e.g., fingers shaped as if holding an imaginary comb). Taken together, these studies show that gesture offers researchers a win- dow into children’s level of symbolic understanding, by allowing them to “see” what children choose to represent with their hands.

Second, naturalistic observations of co-speech gestures reveal how people in- ternally organize concepts such as left and right. Kita and Eggesbey (2001) observed naturalistic situations of giving route directions in Ghana. In the Ghanaian culture, pointing with the left hand is considered to be a taboo. Especially when talking to strangers or the elderly, there is a politeness convention to hide the left hand from the interlocutor by placing it on the lower back. The study showed that Ghanaian speaker-gesturers often engaged in an anatomically straining position to indicate directions towards the left-hand side with right hand pointing gestures. The key finding was that during verbalization of the concept left in combination with a right- hand finger point, left-hand pointing gestures were not fully suppressed. Ghanaian speaker-gesturers often produced a small gesture with their left hand, which was positioned on their lower back outside the addressee’s view, before they gestured to the left with their right hand. This use of gesture thus shows that concepts such as left and right are largely grounded in sensorimotor experiences. In other words, gesture production is influenced by how people think of using the body to interact with the physical environment (Kita, 2000).

Third, spontaneous co-speech gestures elicited in tasks in which people were asked to describe concepts of time reflect how people internally organize informa- tion about time. A study by Gu, Mol, Hoetjes, and Swerts (2017) demonstrated that Chinese-English bilinguals produce more gestures along the vertical axes when talking about Chinese time references with vertical spatial metaphors than when talking about time conceptions in the English translations, and when talking about Chinese time references with no spatial metaphors. These findings thus show that when speaking Chinese, Chinese-English bilinguals organize their thoughts about time vertically, corresponding with the ancient Chinese measures of time such as the hour glass in which the sand runs from top to bottom, as well as incense, which burns from top to bottom. Thus, this study shows that gesture revealed how time is conceptualized in the mind during the moment of speaking.

to narrate a story reveal how people organize verb clauses. Kita and ¨Ozy¨urek (2003) asked English speakers, Japanese speakers, and Turkish speakers to describe events shown in a short video clip of the cartoon Tweety and Sylvester. In the stimulus video, Sylvester swallowed a bowling ball and rolled down the hill into a bowling alley. Many of the speakers mentioned this event when they narrated the story to an experimenter. Turkish speakers and Japanese speakers described the event of Sylvester rolling down the hill using two verb clauses: one to express the “spin- ning/rolling” motion and another to express the “descending/downward” direction or path. English speakers used only one verb clause to express the same event: “rolling down”. The differences in the number of verb clauses that the speakers used was also reflected in the number of gestures they used. Turkish and Japanese speak- ers produced one gesture per verb clause: one spinning hand or finger movement for the motion and another downward hand movement. English speakers, however, produced one conflated gesture that depicted the spinning motion and downward path simultaneously. Thus, the gestures of Turkish speakers, Japanese speakers, and English speakers reflect how language is organized in their mind during the moment of speaking. These findings are consistent with the idea that gestures, together with speech, can constitute thought (McNeill, 1992). Even if gestures and speech are revealing the same thoughts (i.e., they show equivalent information), it is impor- tant to analyze both modalities because gesture and speech convey information in different representational formats. Gestures present information in an analog, imag- istic format, whereas speech presents information in a discrete, segmented format (McNeill, 1992).

Fourth, gestures can also convey unique ideas that are different from the ideas presented in accompanying speech. In a study by Church and Goldin-Meadow (1986), 5-8-year-old children were asked to judge Piagetian conservation problems. In one of the trials children were asked whether two rows of coins had the same or a different number of coins. This question was asked once after the experimenter had perfectly aligned the two rows of coins in front of the children and once after the experimenter had spread out the coins in one row to make it wider than the other row in front of the children. Most children answered that the number of coins in each row was the same when the rows were aligned, but different when one row of coins was spread out to be wider than the other. When children were asked to explain their second answer they said “because one row is wider”, and they spontaneously produced gestures congruent with their speech (e.g., indicating the width of rows) or incongruent with their speech (e.g, spreading out the fingers). Children who spread out their fingers followed the alignment between the two rows

of coins, which should have led them to the answer that the rows contained the same number of coins. When there was a mismatch between gesture and speech, two beliefs about the same problem were simultaneously expressed–one in gesture (i.e., alignment is key to the answer) and another in speech (i.e., width is key to the answer). Thus, this study shows that gesture can also offer a unique window into a child’s mind, importantly, one that is different from speech. This is important because the simultaneous activation of multiple beliefs suggests that a child is in a transitional state and ready to learn (Goldin-Meadow et al., 1993). In Chapter 7 we present a suggestion for future research which relates this “readiness to learn” conveyed by gestures that children produce themselves to the research presented in this dissertation.

The empirical chapters that follow now are based on self-contained manuscripts which are either published, accepted for publication, under review at a scientific jour- nal, or being prepared for publication. Therefore, each chapter has its own abstract, introduction, and discussion. As most references overlap between the chapters, all references are included in a reference list at the end of this dissertation.