A framework for empirical analysis: the GeM model

In order to develop corpus-based approaches to multimodal documents where we can uncover just what multimodal documents are doing in their own terms, and not in the terms inherited from investigations of language, we need a strong foundation for making the organisation and contents of multimodal documents visible. That is, we require a model that is strong enough to support hypothesis building and analysis, and which also provides an appropriate framework for the construction of multimodal document corpora. The scheme we propose for this is the main result of the project on which this book is based—the GeM project (‘Genre and Multimodality’)—and is accordingly called the GeM model.

The GeM model defines several layers of description for multimodal doc-uments. Our claim is that these layers are the very least that are required to do justice to such documents—there are certainly more, but without the layers we describe, basic components of almost any multimodal document will be left out of the picture. As we will also show in Chapter 6, the formal specification of the layers defined by the GeM model provide the basis for the construction of a corpus of multimodal documents conforming to the most recent recommendations and standards for linguistic corpus design.

This is why we believe that the account developed contributes directly to our stated goal of furthering empirically based research into multimodal meaning.

A schematic overview of the model as a whole is given in Figure 1.4. We see documents formed by configurations of social practices shown in the centre of the diagram. Particular multimodal artefacts are produced with

particular forms and with the help of particular production technologies, or tools, for their construction. Documents are then produced ‘on’ a particular canvas that is to carry the artefact physically. We employ ‘canvas’ as a tech-nical generalisation to refer to whatever medium is adopted as substrate for the artefact at issue. This could be paper of a particular size (and thickness, absorbency, etc.), or a monitor screen (of a particular size and resolution), or an animal skin (of particular texture), and so on.

Figure 1.4 The GeM model

Whatever substrate is selected for an artefact’s canvas brings its own con-straints with it—such as, for example, enforcing fonts to be above a certain minimum size to avoid the characters bleeding into one another due to high absorbency, etc. Possible forms are therefore constrained by the particu-lar ‘canvas’. Forms are also constrained by the technology that is used to produce the artefact. Certain printing presses may only be able to print columns of a certain width, or use a particular restricted range of colours, etc. Certain brushes for painting characters may only allow a particular size of character to be produced, and so on. These restrictions are not strictly imposed by the canvas, but by how the artefact is brought to life on the can-vas. Finally, forms are constrained by their intended uses: if the document is to serve as a quick reference guide, then certain forms (and canvases) will make better choices than others.

Combining these three sources of constraint on forms gives rise to what we term the virtual artefact. Viewed from the outside, i.e., without going into the details of the physical substrate, production technology, and sumption requirements, it may be difficult to disentangle just which con-straints are due to what source. And, indeed, there can be redistributions

among these three sources without the form of the final artefact changing.

For example, early newspaper press technology was not able to produce columns of arbitrary width: columns needed to be short enough both to fit on a particular device and to avoid having their type thrown out by that device’s high-speed rotary action (cf. Hutt 1973, p43). At that time, the constraint on form was a production constraint. Similarly, the use of a particular size of font was a canvas constraint, owing to the lower quality of paper that was available. Nowadays, columns can be of more or less arbitrary width and the fonts used could be very much smaller: but the es-tablished use of the newspaper has brought into being a sufficiently strong (and functionally motivated) tradition that the outward form of the newspa-per has not changed nearly as much as the technology and material used could in theory support. The newspaper is then, in our terms, a ‘virtual arte-fact’ in which the final form is constrained by a combination of physical, production and consumption constraints.

We therefore see documents as being created and used within a configu-ration of constraining influences. Our view of these aspects of documents is particularly inspired by the model of document description developed by Waller (Waller 1987a,b). Although centered broadly on typography and design, Waller draws extensive parallels and connections with linguistic models of communication and so already went a considerable way towards providing a model compatible with the other layers of linguistically moti-vated description that we include. Briefly stated, the GeM model extends and replaces those of Waller’s layers concerned with linguistic description, while adopting and further specifying the layers of his model responsible for capturing design and layout. This results in two broad areas: (i) features describing aspects of the conditions of production/consumption of the doc-ument including its physical manifestation, and (ii) features concerned with the properties inherent in the document being analyzed. The constraints of the first area are summarized in Table 1.1.

Taking the conditions of production for multimodal artefacts seriously as this suggests enables a more realistic appraisal of the precise motivations and reasons for the appearance of documents. Design is often described as a compromise between many competing and sometimes conflicting con-straints and these need to be brought into any discussion of the functional motivation of the resulting artefacts. This issue is equally relevant for all professionally produced multimodal documents. Such documents are typi-cally subject to stringent production constraints ranging over the provision of material, time pressure for decision-making, conforming to the generic design constraints, and so on. Such constraints can make themselves felt in the smallest details of a document—for example, page layout can be a

Canvas constraints

constraints arising out of the physical nature of the object being produced: paper or screen size; fold geometry such as for a leaflet

Production constraints

constraints arising out of the production technology: limit on pages, colors, size of included graphics, availability of photographs; constraints arising from the micro-and macro-economy of time or materials: e.g. deadlines; expense of using color; necessity of incorporating advertising

Consumption constraints

constraints arising out of the time, place, and manner of ac-quiring and consuming the document, such as method of se-lection at purchase point, or web browser sophistication and the changes it will make on downloading; also constraints arising out of the degree to which the document must be easy to read, understand, or otherwise use; fitness in relation to task (read straight through? Quick reference?); assump-tions of expertise of readers

Table 1.1 The primary sources of constraints adopted by the Genre and Multimodality framework

result of negotiations between publisher and advertisement companies and not a matter of strictly functional design at all, or the need to produce a par-ticular length of headline may restrict linguistic choices right into grammar and vocabulary.

The technology used to produce a particular multimodal document can also impose its specific footprint on the artefact as a whole—for example, as print technology changed, newspapers began to use color, but in different ways as the technology developed; analyzing the use or non-use of color im-ages at that time needs to include consideration of this boundary condition.

Not being aware of the production constraints always opens up the dan-ger of over-interpretation because more design freedom, and hence more controllable resources for making meaning, are assumed than are actually available.

Since all of these social practices and their particular contributions to the production of a document are subject to constraints of the types we have suggested, we group these constraints together to give our extended notion of genre, shown to the right of Figure 1.4. Traditional, non-multimodal views of genre would concern themselves primarily with the bottom row:

a social practice employing particular modes of linguistic expression; we open this up both to include the other semiotic modes available and to take into consideration the artefactual nature of multimodal documents.

Content structure

the content-related structure of the information to be communicated—including propositional content Genre structure the individual stages or phases defined for a given genre:

i.e., how the delivery of the content proceeds through particular stages of activity

Rhetorical structure

the rhetorical relationships between content elements:

i.e., how the content is ‘argued’, divided into main mate-rial and supporting matemate-rial, and structured rhetorically Linguistic

structure

the linguistic details of any verbal elements that are used to realize the layout elements of the page/document

Layout structure the nature, appearance and position of communica-tive elements on the page, and their hierarchical inter-relationships

Navigation structure the ways in which the intended mode(s) of consumption of the document is/are supported: this includes all ele-ments on a page that serve to direct or assist the reader’s consumption of the document

Table 1.2 The primary layers of the Genre and Multimodality framework

Each source of constraint (canvas, production, etc.) may bring restric-tions for the concrete forms that may appear in the artefact itself. In order to isolate the effects of genre constraints, therefore, we need to be able to characterise the concrete forms and contents that are found in multimodal documents in sufficient detail to make genre variation visible. This charac-terisation is given in terms of the particular semiotic modes deployed within a document—e.g., in terms of the verbal (linguistic), visual (graphical, di-agrammatic, etc.), and layout (composition). We summarise these main descriptive layers of our account in Table 1.2. This offers a framework for the layered decomposition of any multimodal document that can be used directly for multimodal corpus annotation.