We have drawn attention to the fact that, while both visual structures and verbal structures can be used to express meanings drawn from a common cultural source, the two modes are not simply alternative means of representing ‘the same thing’. It is easy to overemphasize either the similarity or the difference between the two modes. Only a detailed comparison can bring out how in some respects they realize similar types of meaning, though in different ways, while in other, perhaps most respects they represent the world quite differently, allowing the development of the different epistemologies we discussed in the previous chapter. In this brief final section we wish to explore this in some detail with respect to narrative visual structures.
By comparison to the structures we will discuss in chapter 3, narrative visual structures are comparatively easy to ‘translate’; though, as we will see, there certainly is no one-to- one correspondence. Like ‘non-transactional actions’, ‘one-participant material processes’ (Halliday, 1985: 103ff.) represent events as though they bear no relation to, and have no consequences for, participants other than the Actor. In Many people migrated one cannot add a second participant to this clause and say, for instance, Many people migrated their relatives, although one can of course add circumstances: Many people migrated to Aus- tralia. And, like ‘transactional actions’, ‘two-participant material processes’ (Halliday, 1985: 103ff.) involve two participants (e.g. Migrants invaded Australia). But linguistic ‘Events’ and visual ‘Events’ are quite different. Linguistic Events have processes that are ‘happenings’ which cannot have an Actor, as in Many of my relatives died. In the case of visual Events, the Actor is left out, but could have been used. They are the equivalent of passives with agent deletion, of clauses like Many of my relatives were killed, rather than of clauses like Many of my relatives died. To show someone dying it is necessary to show someone being killed, or to show someone performing an action that represents his or her death. Also, while in English many processes can take a third participant, the ‘Beneficiary’ (traditionally ‘indirect object’ in, e.g., Mary gave him the book), in images the possibility of such a third participant does not exist. What is a Beneficiary in English becomes a Goal in images (‘she message-sends him’ instead of ‘she sends him a message’).
On the other hand, English lacks the visual mode’s structural devices to represent events as cyclical (although linguistic participants can have a double role in English – for instance, in examples like He made them do it, where them is Goal of make do as well as Actor of do; cf. Halliday, 1985: 153). Nor is there an ‘interactional’ process. To realize what we have called ‘Interactors’, English would have to make use of reflexive pronouns. Consider, for instance, the problem of trying to ‘translate’ figure 2.18, de Saussure’s ‘speech circuit’ diagram, into English. A single visual process indicates something for which, in English, we need at least four clauses: ‘A speaks to B’, ‘B speaks to A’, ‘A listens to B’, ‘B listens to A’. How can one render this in one clause? ‘A and B communicate with each other’? But that causes ‘A’ and ‘B’ to lose their separate identity, transforming a reciprocal, bidirectional transaction into a jointly authored non-transactional action.
What we have called the ‘non-transactional reaction’ is in some ways akin to what Halliday calls the behavioural process (1985: 128), a process type which can take only one
participant (who must be human) and serves to realize a restricted field of action, the field of ‘physiological and psychological behaving’ (1985: 128). But the meanings of visual ‘non-transactional reactions’ form a more restricted field, tied up as they are with one kind of behaviour – looking.
Projective processes – that is, mental and verbal processes – play an important part in English, and it is possible to distinguish a number of different types of each on the basis of formal grammatical criteria (Halliday, 1985: 106ff., 129). Mental processes, for instance, include processes of perception (‘see’, ‘hear’, etc.), processes of affection (‘like’, ‘fear’, ‘wish’, etc.) and processes of cognition (‘know’, ‘think’, ‘believe’, etc.). Each has a Senser, the person who does the seeing, or liking, or knowing (this has to be a person, or a participant represented as human), and a Phenomenon, someone or something seen, or liked, or known, by the Senser. Phenomena may be realized by participants or whole structures, just as in the case of images. In Many people want to migrate to Australia, the clause to migrate to Australia is Phenomenon; this sets mental processes apart from actions and transactions, which cannot have a clause as Goal. But what we have called the ‘transactional reaction’ can, if one wishes or needs to, be related to only a subset of the perception process, because non-visual Phenomena cannot directly be realized in the visual semiotic. Mental processes form, as we have shown, only a minor category in the visual semiotic; as far as we can see, there are no structural visual devices for making the strong distinction between ‘cognition’ and ‘affection’ processes that has come to characterize the ideational resources of English. The cinema, however, has developed a fairly extensive set of projective conventions for realizing different kinds of mental processes such as memories, dreams, hallucinations, and so on.
In English, verbal processes differ from mental processes in that they do not need a human ‘Sayer’ (one can say The document said that . . ., but not The document thought that . . .). On the other hand, like mental processes, they can take whole clauses as their object, and this in two different ways – in the form of Reported Speech (as in He said [that] he had no idea) and in the form of Quoted Speech (as in He said ‘I have no idea’). There seems to be no direct, structural way of expressing this kind of difference visually. ‘Dia- logue balloons’ always quote.
We have identified only three different types of circumstance in images: location, means and accompaniment. All three exist in English (Halliday, 1985: 137ff.), but there they are by no means the only types. English allows all kinds of information to be added to the basic narrative proposition conveyed by the process (‘What happened?’) and the participants (‘Who or what was involved?’); information about time (‘When did it happen?’; ‘How long did it last?’); about purpose (‘What did it happen for?’); cause (‘Why did it happen?’) and so on.
The following table gives an overview of some of the correspondences between linguistic and visual narrative processes:
Comparisons such as these can highlight which ways of representing the world can be realized linguistically, which visually and which (more or less) in both ways. This, in turn, is useful as a background for analysing representation in multimodal texts: photographs and their captions, diagrams and their verbal glosses, stories and their illustrations. If, for instance, a diagram shows an arrow emanating from a participant labelled ‘environment’ and directed at another participant labelled ‘message’, then a ‘literal’ translation would be ‘the environment acts upon the message’. If the accompanying text says that the ‘com- munication process’ is ‘interacting with factors (or stimuli) from the environment’ (Watson and Hill, 1980: 14), it ‘mistranslates’, and the mistranslation is not due to the limitations of either English or visual communication. A literal ‘translation’ of ‘the source sends a message to the receiver’, on the other hand, is not possible: the spatial representa- tion of verbally conceived ideas changes the ideas themselves, and vice versa. In the next chapter we will show further examples of this problem.
Notes
1 We are grateful to the Danish art historian Lise Mark for pointing out that, in the previous edition of this book, this painting was reproduced ‘on its side’, i.e. tilted by 90 degrees.
2 We would like to thank Bente Foged Madsen for some useful comments that have led to improvements of this diagram.
Table 2.1 Narrative process in language and visual communication
Visual narrative processes Linguistic narrative clauses
Non-transactional action One-participant (Actor) material process (‘action’) Unidirectional transactional action Two-participant material process
Event Passive transactional clause with agent deletion Bidirectional transactional action –
Non-transactional reaction Behavioural process (field of looking) Transactional reaction Mental process: perception (visual only) Mental process Mental process (cognition and affection) Verbal process Verbal process (quotation)
– Verbal process (affection)