Semiotic interpretation of image and writing My particular research interest is in how children represent/communicate graphically. As I began to study children’s writing and drawing semiotically I became interested in signs beyond those that are traditionally valued. For example, how text had been set out in a particular format appeared to carry meaning. Not attending to the full range of signs made in a graphic text seemed to me to be a partial analysis of all that the sign-maker had represented. I therefore began to draw on the methods of analysis developed by Gunther Kress and Theo van Leeuwen (1996). This provided me with an explicit analytical
‘toolkit’ with which to study children’s texts in a systematic, rigorous and detailed way. Multimodality offered a means of accounting for the multiple modes in which signs can be made and of understanding more about the interrelationships between different modes in graphic texts. Of course, Nathalie’s repre-sentations (examined below) are her interpretations of phenomena just as my ‘reading’ of the signs she made are interpretations. In presenting my work, I found that interpretation beyond or without the authentica-tion of spoken or written words made people nervous. This is symptomatic of the view that language can give full access to meaning. Neverthe-less, the mode of speech in interviewing enabled children to express how they thought about phenom-ena in a different way from diagrammatic drawing (Mavers et al., 2002). It is not that one mode is inferior and another superior but that each communi-cates different aspects of meaning.
Signs in image-based mind mapping
Sign-making on the page is a means of sharing ideas with others. The form – the way in which marks have been composed – carries the meanings of the sign-maker. In her mind map1 Nathalie made a whole range of signs to communicate how she thought about ‘Computers in My World’2. The aim of the task was to gain an understanding of children’s ‘secondary artefacts’ or mental representations of the computer as a tool (Cole, 1999; Wartofsky, 1979). Scripted instructions asked the children to think about types of computers, where they can be found, if they are connected and the people who use them, and how.
Read out by the class teacher, the script informed Nathalie’s class (9-and 10-year-olds) that image-based mind mapping would be a means of communicating with researchers. This framed the task in two ways.
Firstly, the children’s interest was shaped according to the particular communicational need. Implicitly, the inference was that the mind maps would be ‘read’ by unknown others and would therefore need to be readily ‘readable’ by them. An embedded aim was therefore to communicate effectively within a tight time limit of 20 minutes. Secondly, the direction to use image as the primary means of communication was significant for the meanings that could be made.
This both enabled and disabled according to the
‘functional specialization’ (Kress and van Leeuwen, 2001: 64) of drawing. The instructions asked the children to draw ideas quickly as they came into their minds and to link them with lines: ‘The order in which you do the drawings is not important but it is important that you draw lines between the drawings that you feel are linked.’ They were also asked to
‘write a few words to label any of your drawings’ or to write a contents list. The spatiality of the page had implications for the signs Nathalie could make.
Drawing obliged Nathalie to show (Figure 20.1).
Her representation of the Internet, like the website at the top of her map, is generalized rather than specific.
It carries signs that communicate something about her conception of online texts. She shows that they include writing (represented as repeated horizontal text squiggles) and other textual framings (shown as rectangles), presumably images. Note how the ‘email’
and ‘work’ nodes (individual images), in contrast, contain only representations of writing. The implica-tion is that words and image work visually as blocks that can be presented in different ways. This was not an accident. It carries meaning. Drawing compels ways of communicating, and therefore of thinking, in a different way from words. Other children in Nathalie’s class represented the Internet as a surfboar-der, as ‘www.’ inside a computer screen and as a globe. This gave a different emphasis, a different
‘slant’. The drawings do not exclude each child’s knowing about other characteristics of the Internet but signify the foregrounding of a particular idea at a particular moment in time (Marton and Booth, 1997:
123).
Nathalie’s drawings are not reproductions. She transformed the three-dimensional world onto the flatness of the page. Bearing in mind the focus of the task and the intended audience, she chose and 2 0 S O C I A L S E M I O T I C S A N D M U L T I M O D A L T E X T S
Figure 20.1 Nathalie’s mind map
represented those features of electronic games equip-ment which she considered key to conveying her intended meanings (Mavers, 2003). With one excep-tion each node is drawn frontally. Essentially diagram-matic rather than pictorial, this gives her map high modality (a term borrowed from linguistics to denote
‘truth value’). Nathalie’s map comprises 14 nodes.
The majority are drawings of actual ‘things’. However, to the right of her central node is a world image encircled by 10 computers. As a whole, this node is not something she has drawn directly from actuality.
It is a bringing together of drawings of the world and computers to construct a visual metaphor. This carries particular conceptual meanings. Symbolically, Nathalie shows that computers are situated all around the world. Interestingly, there is a shift in how Nathalie made signs through her written label for this node. Whereas all the other labels in her map are identifiers (nouns), ‘round the world’ (preposition, definite article and noun) signifies something different from ‘world’ or ‘our world’. It extends from identifi-cation to explanation. The meanings of this node expand when considered in relation to the central node. The image of the globe displayed on the computer screen suggests that the world is in some way contained in the computer. When these two nodes are looked at together, the joint implication is that computers are around the world and the world is
in the computer; technology is a worldwide resource and the global is accessible electronically.
Nathalie also made signs in her positioning and linking of nodes. The central computer works as the title or superordinate of the map to which all other nodes are linked either directly or indirectly as representations of Nathalie’s ideas about ‘Computers in My World’. With the exception of the scanner, the links to the left of her map connect drawings to the central node individually. However, to the right of her map, Nathalie created two discrete and internally interlinked groupings of nodes. The single links from the superordinate guide the ‘reader’ to these groups.
The grouping to the bottom right suggests locations where computers are used – school, home, possibly the workplace and the world. The interlinked group-ing of nodes to the top right is perhaps less transparent. The sounds of music are depicted through conventional representation as musical notes emanating from a computer speaker. This node is linked to a drawing of a printer with a just-printed image of a face. The text squiggles in the email node suggest writing. At the bottom of the grouping the mouse may imply control. These nodes might show the variety of modes available on computers: writing, sound, image and possibly the actional – their multimodal capacities.
Figure 20.2 Nathalie’s writing
Signs in writing
A 15-minute written task was undertaken on a separate occasion, around one week after the mind mapping. The children were asked to write about their maps but the maps were not returned to them. This was framed as talking to an Alien. The emphasis in the scripted instructions was on description and explanation, the closing words being ‘What would the Alien need to know to understand computer systems in our world and what they can do?’ In response, Nathalie’s writing is shaped as an objective, factual piece. Her opening refers to ‘people’ and ‘they’. On her fifth line (just under a quarter of the way through the text) Nathalie shifts to the personal pronoun ‘you’
which she retains apart from one further use of
‘people’. Never is there slippage to ‘I’ or ‘me’. This is a generalization where she notes regularities in human
practices. It defines the text as an informational report rather than a personalized account.
Nathalie wrote 176 words in black biro on one side of A4 paper (see Figure 20.2). Her punctuation is of significance. She attends to seven different topics, each framed as a sentence ending with a full stop.
There are just two exceptions. On one occasion a word overruns the edge of the page. On her seventh line she made two full stops but followed the first with a lower case letter as if to imply that the two parts of the topic are related (they are both to do with electronic information) but contain separate ideas (how to put a disk in a computer and different types of information for different purposes). She dealt with the wholeness of her opening topic by splitting clauses with commas rather than full stops. This may represent a transcription of speech in response to the 2 0 S O C I A L S E M I O T I C S A N D M U L T I M O D A L T E X T S
task instructions ‘Write down what you would say if you were speaking to the Alien.’
The seven topics Nathalie addresses (communica-tion, locations of use, electronic resources, medium, connectivity, control and information) are similar to the themes of her mind map. However, the mode of writing enabled her to communicate different mean-ings from drawing. Nathalie’s written account in-cludes statements of fact (for example, ‘People can use a thing called a mouse’), descriptions of actions and processes (for example, ‘You can put discs of which will go into the computer inside a little thing that will come out on the computer to put the discs inside’) and explanations of purposes (for example,
‘on these you can Learn things, save work and play games’). For the majority of topics the fact precedes the explanation, for example ‘Computers are connec-ted to one another around the world so they can send messages to each other’. Here ‘so’ implies a causal connection, as does ‘and’ elsewhere. The word ‘can’
appears ten times in Nathalie’s text. On five occasions the succeeding verb is to do with functionality such as operating computer devices (for example, ‘print’,
‘go on a web site’), three occurrences are cognitively related (‘read’, ‘Learn’, ‘find information’) and two suggest the affective (‘play’, ‘like’), ideas particularly well suited to writing.
Issues arising
In both her mind map and her writing Nathalie endeavoured to communicate an accurate, factual account of computers in her world. Both were shaped by the set title and the scripted directions on content, and for the identified audience as she perceived it.
The functional specializations of drawing and writing brought about different representations of meaning.
Drawing obliged her to show meanings in a way different from words. It enabled her to depict ‘things’
as aspects of ‘Computers in My World’ such as electronic equipment and locations. Her spatial ar-rangement through positioning and her links were also a means of communicating aspects of her conceptualization, as in her groupings of nodes.
Something different happens in her writing. Here, she describes actions and processes, and explains reasons why and purposes for. The mode may have been prescribed but Nathalie made choices about how to shape meanings according to her individual interest within the potentialities of drawing or writing. Repre-sentation/communication in drawing and writing enabled Nathalie to make meaning in particular ways.
It is not that one is more trustworthy than the other but that their different affordances enable different signs to be made. Her signs – how she connects signifiers and signifieds – were shaped by her interest and her intent to mean.
Notes
1. ‘Mind mapping’ was the term used to describe the genre in the scripted instructions. In other publi-cations it has been referred to as ‘concept mapping’.
2. This work draws on data from the ImpaCT2 evaluation funded by the UK Department for Education and Skills and managed by the British Educational and Communications Technology Agency. The team included: C. Harrison (Direc-tor), T. Fisher, K. Haw, E. Lunzer (University of Nottingham), P. Scrimshaw, C. Lewin (Open University), B. Somekh, D. Mavers (Manchester Metropolitan University).
Annotated bibliography
Hodge, R. and Kress, G. (1988) Social Semiotics. Cambridge: Polity Press.
In exploring how meaning systems are socially constituted and situated in social contexts, this book addresses power and ideology, space and time, and gender and class.
Jewitt, C. and Kress, G. (eds) (2003) Multimodal Literacy. New York: Peter Lang.
Using a range of diverse examples, this edited book explores how meaning-making is made through a variety of representational and communicational modes such as speech, image, writing, gaze, gesture and movement.
It provides conceptual tools for understanding a multimodal approach to literacy and learning.
Kress, G. (1997) Before Writing: Rethinking the Paths to Literacy. London: Routledge.
In attending to a multitude of ways in which children make meaning, Gunther Kress explores key ideas in a social semiotic approach and challenges how early literacy is understood.
Kress, G. (2003) Literacy in the New Media Age. London: Routledge.
In this book Gunther Kress explores the need for a reconceptualization of what literacy is as a consequence of technological change. As new media signal the dominance of the screen, he considers social and cultural implications.
Kress, G., Jewitt, C., Ogborn, J. and Tsatsarelis, C. (2001) Multimodal Teaching and Learning: The Rhetorics of the Science Classroom. London: Continuum.
This book explores the different communicative modes that are used to make meaning in the secondary science classroom, going beyond language to image, three-dimensional models, gesture and movement.
Pahl, K. (1999) Transformations: Meaning Making in Nursery Education. Stoke on Trent: Trentham Books.
In this book, Kate Pahl explores the multiple ways in which nursery-aged children make meaning as they draw, write, cut out, stick together and model. She uses her detailed observations to explore their developing literacy.
van Leeuwen, T. and Jewitt, C. (eds) (2001) Handbook of Visual Analysis. London: Sage.
With a focus on visual analysis, this edited book covers a wide range of methods such as content analysis, psychoanalysis, social semiotic analysis and ethnomethodology. It provides examples such as advertisements, photographs, children’s pictures and film.
Further references
Burn, A. and Parker, D. (2003) Analysing Media Texts. London: Continuum.
Cole, M. (1999) ‘Cultural psychology: some general principles and a concrete example’, in Y. Engeström, R.
Miettinen and R.-L. Punamäki (eds), Perspectives on Activity Theory. Cambridge: Cambridge University Press, pp. 87–106.
Finnegan, R. (2002) Communicating: The Multiple Modes of Human Interconnection. London: Routledge.
Franks, A. (2003) ‘Palmer’s Kiss: Shakespeare, school drama and semiotics’, in C. Jewitt and G. Kress (eds), Multimodal Literacy. New York: Peter Lang, pp. 155–72.
Jewitt, C. (2003) ‘Computer-mediated learning: the multimodal construction of mathematical entities on screen’, in C. Jewitt and G. Kress (eds), Multimodal Literacy. New York: Peter Lang, pp. 34–55.
Kenner, C. (2003) ‘Embedded knowledges: young children’s engagement with the act of writing’, in C. Jewitt and G. Kress (eds), Multimodal Literacy. New York: Peter Lang, pp. 88–106.
Kress, G. and van Leeuwen, T. (1996) Reading Images: The Grammar of Visual Design. London: Routledge.
Kress, G. and van Leeuwen, T. (2001) Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.
Marton, F. and Booth, S. (1997) Learning and Awareness. Mahwah, NJ: Lawrence Erlbaum Associates.
Mavers, D. (2003) ‘Communicating meanings through image: composition, spatial arrangement and links in student mind maps’, in C. Jewitt and G. Kress (eds), Multimodal Literacy. New York: Peter Lang, pp. 19–33.
Mavers, D., Somekh, B. and Restorick, J. (2002) ‘Interpreting the externalised images of pupils’ conceptions of ICT: methods for the analysis of concept maps’, Computers and Education, 38:187–207.
Scollon, R. and Scollon, S. (2002) Discourses in Place: Language in the Material World. London: Routledge.
Wartofsky, M. (1979) Models: Representation and Scientific Understanding. Dordrecht: Reidel.
2 0 S O C I A L S E M I O T I C S A N D M U L T I M O D A L T E X T S