Multimodal representations in the classroom

Teaching digital literacy in schools has become increasingly important and “… pupils need to be taught now which tools are effective and how to use them responsibly” (e-Safetysupport, 2013). Representing with multimedia in a classroom needs to be taught with the use of suitable technology tools and by providing relevant content. Digital storytelling may become a useful technology tool to help students increase or develop digital literacy skills. Therefore, it is important to identify what makes a representation multimodal.

Over the past decade, the traditional language skills of reading, writing, listening and speaking have been extended to include representing. This additional category in literacy learning is a response to contemporary developments in representation and communication. For Kress (2004), one key aspect of these developments is a move from the dominance of writing as “the culturally most valuable form of representation” (p. 5) towards the new dominance of images. Kress and Van Leeuwen (1996) claimed that “newspaper, magazines, public relations materials, advertisements and many kinds of books today involve a complex interplay of written text, images and other graphic elements, and what is

more, these elements combine together into visual designs, by means of layout” (p. 15).

Barratt-Pugh (2000) emphasised that images help to convey meaning. An image does not replace text, but rather complements it, which results in a blending of modes in multimodal representation or multimodal text. Each mode affords specific potential and limitations for communication (Kress 2004) and images and texts enter into a powerful intersemiotic relationship with each other. For

example, expanding on the work of Barthes (1977), Martinec and Salway (2005) suggested three possibilities for the text–image relationship: text supporting image, image supporting text and the two being equal (that is, the whole image is related to the whole text).

Jewitt (2008) wrote that all modes contribute to the construction of meanings in some way and “no one mode stands alone in the process of making meanings; rather, each plays a discrete role in the whole” (p. 247). In their

Grammar of visual design, Kress and Van Leeuwen (1996) write about the

“overwhelming evidence of the importance of visual communication” and emphasise that effective communication is no longer limited by written texts because of this evolution in the forms of communication: from oral messages in early ages, to printed media, to radio and television, and finally, to the dominance of the Internet as a medium.

Contemporary text and communication are increasingly multimodal. Kress (2004) underlined the move from the dominance of the book (or print-based media in general) to the dominance of the medium of the screen (computer screens in particular). Elsewhere, Kress (2006) writes that the new medium of the screen makes it “easy to use a multiplicity of modes, and in particular the mode of

image—still or moving—as well as other modes, such as music and sound effects for instance” (p. 5). Unlike language-based written texts that have been well researched and have well-established semiotic rules, no corresponding rules exist for multimodal texts (although Kress and Van Leeuwen, 1996, made an attempt to develop what they called the ‘Grammar of Visual Design’). Callow (2003) noted the limited research on what skills students need when they are involved in making meanings from multimodal texts, while for Jewitt (2008), “pedagogical models for print literacy are based on the acquisition and mastery of sets of established practices, conventions and rules” (p. 252), whereas models for digital literacy are incomplete. Therefore, this thesis attempts to understand the digital literacy skills that develop through digital storytelling. Digital storytelling may provide multimodality and interactivity, the two main properties of the process of creating a digital story, which may lead to the development of digital literacy.

Multimodality: Digital text contains language-based text (e.g.,

explanations and discussions, headings and sub-headings, subtitles and labels) and images (e.g., photographs, drawings and illustrations, icons and symbols, aesthetic elements, maps, concept/mind maps, diagrams such as flow charts, schemas and statistical graphs). It may also contain other modalities such as audio (narration, music and sound effects), animation (two- and three-dimensional), video, colour leads, transitions and interactive elements. All of these different modes afford something specific for representations (Kress, 2004) and each communicates certain aspects of the overall display. Together, they blend and the boundaries between them blur and mesh in a new multimodal configuration (Jewitt, 2008). This new multimodal configuration can be represented as a multimedia

podcasts, etc.). For Jewitt (2008), students need to learn how to recognise what is salient in a multimodal text, how to read across the modal elements, how to move from the representation of a phenomenon in one mode to another mode, and how to navigate through the multiple paths of a text. Therefore, a student working with technology needs to have the ability to blend traditional literacy with media literacy, information literacy and technology tools literacy. In other words, a student needs to be digitally literate.

Interactivity: In the context of this study, interactivity involves the reader

or viewer and the author of the text in an interactive exchange around the multimodal texts. Digital literacy learning through the creation of a digital story provides students with an opportunity to explore how to go beyond the content of the multimodal text and engage others in extended meaning making (Ohler, 2006).

These two properties of multimedia representations (multimodality and interactivity) are key elements of digital storytelling, which could become a strategy for digital literacy learning in a classroom. Classrooms could provide students with the opportunity to work on digital storytelling and develop skills required for representing through digital multimodal texts that should not be understood exclusively as a part of language learning. This leads to the question: What is digital storytelling?

2.3.2 Digital storytelling and its affordances in the classroom.