Close
window to return home or click here |
||
| IIID | Expert Forum for Knowledge Presentation | |
| Conference | Preparing for the Future of Knowledge Presentation | |
| |
||
| Gunther Kress |
Reading Images: Multimodality, Representation and New Media | |
© 2004
Gunther Kress |
|
|
| |
|
|
| Conference presentation Video | ||
| |
|
|
| Keywords |
Multimodality,
literacy, representation, design, media affordances |
|
| Abstract |
In
this paper I wish to point to what I see as the central issues in the
linked shifts in representation and dissemination: that is, from the
constellation of mode of writing and medium of book / page, to the constellation
of mode of image and medium of screen. In particular I will draw attention
to consequent shifts in authority, in changes in forms of reading, shifts
in shapes of knowledge and in forms of human engagement with the social
and natural world. |
|
| Paper |
Readers
of this journal are experts in design. What I can offer is a particular
take on certain issues in design from the perspective of (Social) Semiotics,
and more specifically, from the perspective of multimodality, which
deals with all the means we have for making meanings – the modes
of representation - and considers their specific way of configuring
the world. To make this concrete, here is a small example. Say I am
designing a biology text-book. The subject matter is ‘plant-cells’.
If I use words, I will have to say “Every cell has a nucleus”.
If I use an image, I will need to place a large dot somewhere in the
circle which indicates the cell to represent the nucleus (Figure 1). |
|
| Figure 1. Cell and nucleus
|
 |
|
| |
To
reflect on this: in writing or speaking I have to use a sentence in
which two entities – cell and nucleus – are related by a
verb, have, indicating a possessive relation: the cell has something
(much like: “I have a car, a house, two kids, etc”). I could
use a different verb: “In every cell there is a nucleus”.
The meaning is now quite different: about existence, there is and location,
in. If I draw, I have to place a large dot representing the nucleus
somewhere in a circle which represents the cell. Wherever I place it,
someone looking at the image is entitled to assume that the nucleus
actually is where I have placed it in the circle/cell – whether
I intended to or not, or whether it actually belongs there or not. Each
mode forces me into making certain kinds of commitments about meaning,
intended or not. The choice of mode has profound effects on meaning,
and textbook designers, for instance, need to be aware of such meaning
effects of different modes. |
|
| Modes and their affordances: the materiality of modes | ||
| The
kinds of meanings made by the letter “e”, by the word “Grille”,
and by the colour “red” are different kinds of meanings. Not
only do they mean different things, they mean differently. You can’t
look up the meaning of “e”, nor the meaning of “lurid
red”. What “e” does is not so much refer to some object,
such as a Grill, or a Bar, but rather to evoke by cultural associations.
It has a history of use in particular places (in ‘marketing speak’
for instance), and it is knowing its provenance that gives it its meaning.
“e” puts me in the world of ‘Olde England’ with
all its mythic associations. In one sense, colours work similarly: I have
encountered the colour ‘red’ in many instances, as in “red
light district”, as a colour of lipsticks: so in this context it
is eroticized. Words have their histories, but they also refer; they name
things (as nouns) or actions (as verbs) or attributes (as adjectives)
or as relations of location (as prepositions), and so on. One of the present tasks of a social semiotic approach to multimodality is to describe the potentials and limitations for meaning which inhere in different modes. For that, it is essential to consider the materiality of modes. Speech uses the material of (human) sound; writing uses the material of graphic substance. There are things you can do with sound that you cannot do with graphic substance, either easily or at all; not even imitate all that successfully graphically. The up and down of the voice, which produces the melody of (English) speech, makes many meanings, from straightforward questions to highly modulated ones: imagine saying, in a tone of incredulity, ”you did what?”; to many varying forms of emotion and affect. Even highly experienced writers find it impossible to reproduce these meanings in writing and need to take recourse to devices such as “… she said incredulously”. Maybe the major shift in the new landscape of communication in this respect lies in the increasing use of image, even in situations where previously writing would have been used. Consequently an urgent task is understanding the different affordances of writing and image. In alphabetic cultures writing tends to start, in words, grammar and syntax, as the transcription of speech. It quickly develops its own structures and forms (syntax, punctuation, layout, for instance), so that written English is now very different to spoken English; yet writing does ‘lean on’ speech. Speech happens in time: one sound, one word, one sentence follows another. The ‘logic’ of temporal sequence is the major principle of ordering of languages such as English. Speech and writing are organized by the logic and the ordering principle of sequence in time. This underlies the syntax of English, which is enormously more complex than mere sequence, but is there nonetheless. If I have two simple sentences, such as: The mists dissolved and the sun rose. It matters in what sequence I place them. The sun rose and the mists dissolved is very different in meaning from The mists dissolved and the sun rose. The one tells us how weather works; the other puts us in the magical, mysterious world of Lord of the Rings maybe. Sequence implies causality: the sentence which comes first seems to be causally prior to that which comes after. But notice that that is so whether I want that meaning or not: I cannot but order them in some way. If I have two friends, Amanda and Josh, and they have jus got married, I might want to say either Amanda married Josh or Josh married Amanda; the two are different in causal terms – who was responsible for what. They are also different in terms of affective ‘proximity’: I may be closer to Amanda than to Josh, and so I place that person’s name first. In speech as in writing we use words. Yet only that for which there is a word can be brought into communication: no word, no communication about it. In image, if there is something that we wish to depict, we can depict whatever we want. We don’t ask: ”Is there an appropriate image we can use?” Contrary to common sense assumptions about language, words are vague. You have no doubt fully understood the sentences about Josh and Amanda, yet you know very little about either of them: how tall Josh is, what age Amanda, what colour hair they have, and so on. If you saw a photo of them, or even a drawing, much of this would be clear. Words are (relatively) vague, often nearly empty of meanings; by contrast images are full, ‘plain’ with meaning. With image the placement of the depicted entities relative to one another in the image-space is the principle used for making meaning. Take the two images below, drawn by the then four year-old Georgia. |
||
Figure 2a, 2b. Georgia at the side of her mother, and Georgia between her parents |
||
|
|
|
|
| The
difference in meaning depends on the relation of the depicted entities
to each other in the frame of the picture-space: the resultant difference
in Georgia’s sense of herself and her family is an effect of these
spatial relations. In drawing the materiality of sound is not available
for making, to indicate just how ‘being’ Georgia’s parents
seem to her, instead the affordance of space is used – making things
taller or shorter, broader or thinner. In fact, Georgia was quite a bit
taller than she drew herself here; and her father was quite a bit shorter
than her mother. Size here shows the metaphoric use of vertical extension:
Georgia sees her parents as affectively /psychically much taller than
they actually were; and she makes her father seem as tall as her mother
by ‘lifting him off the ground’ somewhat. That leaves aside
the meanings of colour. One further point needs to be mentioned here; it follows from the distinct ordering principles of the two modes. The written text – as indeed the spoken – forces the reader (and the listener) to stick to its order: the elements have to be read in the sequence in which they occur. That is not the case, or far less so, with the image text. Yes, the elements are there in certain spatial relations, but how the reader reconstitutes them is largely up to the reader. The order of the written text is fixed; the order of the image text is (relatively) open. |
||
| Media and their interrelation with modes |
||
| Modes
and media exist in culturally and historically shaped ‘constellations’.
The one that has dominated the alphabetic cultures of the ‘West’
over the last 300 years or so is that of mode of writing with medium of
book and page. Writing as mode and book as medium have shaped western
imagination, forms of knowledge, practices of reading; the technology
of writing has shaped the book, and the technology of the book has shaped
how writing has developed. The traditional book represented the work of
the author, who had laboured to produce a text, which in its ordering
represented a ‘body of knowledge’ or the shape of the world
– whether fictional or actual. Chapters in the book were coherent
and complete in themselves; paragraphs had their logic; and sentences
derived their form and purpose from the organization of the paragraph
and the larger text. In that world the reader’s task was to attempt to follow the pre-given ordering of the written text, embodying the authority of the author, working assiduously to reproduce the meaning which the author had intended for the reader. In that world, authors could confidently speak and act on behalf of the reader, as did the author of the example in Figure 3, The Boy Electrician: “The prime instinct of almost any boy is to make and to create… At seven he will wire the whole house with his telephone system made from empty tins connected with varying lengths of string. His older brother will improve on that by purchasing a crystal, a telephone receiver, and a few pieces of insulated copper wire…” (p 5) |
||
| Figure 3. Spread from “The Boy Electrician” | ||
|
|
 |
|
| Certain
texts – novels for instance – encourage the reader to engage
in the semiotic work of imagination, following the given order of words
on the line but filling the relatively ‘empty’ words with
the reader’s meaning. Contemporary texts - whether information books
of all kinds, web-pages, the screens of CD ROMs, and so on - in their
increasingly often image-like textual organization, ask the reader to
perform different semiotic work, namely to design the order of the text
for themselves. Consequently two phenomena are now becoming noticeable,
as in Figure 4, which had been present but never noticed before: the entry
point of the ‘page’ and its reading path. |
||
| Figure 4. Home page of the University of London Institute of Education (www.ioe.ac.uk) | ||
|
||
| The
page of The Boy Electrician has one entry point, at the top left of the
page; it had long become naturalized and therefore was no longer visible.
Nor was the reading path: it asked the reader to follow the lines, in
the order in which the culture had determined. The page/screen in Figure
4 has, by contrast, about 13 entry points. The reader interest determines
where he or she wishes to enter the page. The same applies to the ‘reading
path’ which the reader (now usually called a ‘visitor’)
wishes to construct: it too is determined by the reader’s interest.
For design this is a crucial factor, and a profound change. The designer of such ‘pages’ / sites is no longer the ‘author’ of an authoritative text, but is a provider of material arranged in relation to the assumed characteristics of the imagined audience. The power of the designer is to assemble materials which can become ‘information’ for the visitor, in arrangements which might correspond to the interests of the visitor. For the visitor however “Information is material which is selected by individuals to be transformed by them into knowledge to solve a problem in their life world” (Boeck, 2002) |
||
| Making texts and reading texts |
||
| In the
conception outlined here, the processes of making texts and reading texts
are both are processes of design; and both are in important sense inversions
of the social and semiotic arrangements of the era of the dominance of
the constellation of writing and book. It has now been overtaken by the
new constellation of image and screen. The (at least mythically) dominant
media are now those of the screen - whether of the Gameboy, the mobile
telephone, the PC, or still the TV and video. The book and its page had
been the site of writing and the logic of writing had shaped the order
of the page and the book; the screen is the site of the image and the
logic of the image is shaping the order and the arrangements of the screen.
Writing can appear on the screen; but when it does it is subordinated to the logic of the image; just as image could appear on the page, though subordinated to the logic of writing. The logic image will more and more shape the appearance and the uses of writing, a process which is already apparent in many instances of public communication. In the former arrangement, the figure of the author and the mode of writing dominated; in the new arrangements the designer and the mode of image dominate; the story-board is an apt metaphor for this change - image led, and very often the product of a design-team. |
||
| Design
as choice in context |
||
| In the
multimodal landscape of communication, choice and therefore design become
central issues. If I have a number of ways of expressing and shaping my
message, then the questions that confront me are: which mode is best,
most apt, for the content / meaning I wish to communicate? Which mode
most appeals to the audience whom I intend to address? Which mode most
corresponds to my own interest at this point in shaping the message for
communication? Which medium is preferred by my audience? Or by me? How
am I positioning myself if I choose this medium or this mode rather than
those others? All of these call for choices to be made, resting on my
assessment of the environment in which communication takes place, in all
its complexity, in its widest sense, in which a commodity – the
smell of my shampoo, the packaging of the bag of flour, the shape of the
bottle of soft-drink – are all ‘messages’ to interpret.
The question of choice is illustrated by the contrast of say, Figure 3
with Figure 4, or of Figure 3 with Figure 5. |
||
| Figure 5. Visual geography-tectonics | ||
 |
||
| The
page in Figure 3 is the realization of choices – of stylistic choices
in relation to writing, choices of font (though for any one publishing
house there might not have been choice), the framings of the text through
syntax (marked by punctuation) and in text (marked by paragraphing, for
instance), and by layout in spacings, as well as the frame around the
‘densely printed page’ (Reading Images, 1996). However, these
choices had nearly faded into invisibility through the two aspects of
habituation and convention. By contrast, the page in Figure 5 shows a
plethora of choices made and realized through the modes of writing, layout,
colour, and image. Design is a prospective enterprise. The question it asks is: “what, in this environment, with this kind of audience, with these resources that are available for implementing my design, given these social, economic, ‘political’ constraints, and with my interests now at this moment, is the best way of shaping that which I wish to make, whether as ‘message’ or as any object (of design)?” Here, briefly, are two examples, showing choices made and interests expressed. Figure 6 is the result of the request of the teacher of a class of six year-olds to “make me a drawing and write me a story of our trip to the British museum.” |
||
| Figure 6. Notes from a trip to the British Museum | ||
 |
||
| The
different ‘take’ on the representation of the day in writing
and drawing is startling (all the images and stories showed this contrast):
salient object-entities in spatial relation in the visually represented
world, contrasted with salient events/actions in temporal relations. We
might dismiss this as childish representation. Or we might say that these
six-year olds are using the two modes of writing and image in line with
their inherent affordances – the (transformed) recollection of the
visually encountered world through the spatially organized mode; and the
(transformed) recollection of the actionally experienced world through
the temporally organized mode. If we take that approach we see that the
children have made apt use of the affordances of each mode. The facts
of the representational world are certainly moving in the same direction. In the next example two modes co-exist in one integrated textual object, the question is the same one: what are the principles for the use the modes (the question of “principled use” can and needs to be asked of all my examples). At the end of four lessons on ‘plant cells’, the teacher had asked the 14 year old students, working in groups of four, to prepare a slide of the epidermis of an onion, look at it through a microscope, and then “write what you did” and “draw what you saw”. |
||
| Figure 7a, 7b. Eye-piece of the microscope, and Cells as a “brick wall” | ||
 |
||
| The
teacher had given two additional instructions: ”put your writing
at the top of the page, and the drawing at the bottom”, and “use
only black pencils in your drawing”. Apart from the different responses
to this instruction (7 b used colour pencils) there is the startling difference
in what each “saw” and what each wrote. One written text is
a recount, the other a procedure. The recount, generically speaking, says:
“this is what happened”; the procedure, generically speaking,
says: “this is what ought to happen”. The drawings differ
equally profoundly. One declares “this is what theory tells us is
the case” (on a worksheet there had been a comment “what you
should see is something like a brick wall; each cell is a brick”);
the other declares “this is what I actually observed and recorded”.
The first is the "truth" of theory; the second is the "truth"
of the empirical, reliably recorded. The question of design is in the center here. The matter at issue is of course ‘plant cells’; but maybe even more than that it is: ”what is it to be scientific?” In each case the answer is broadly the same (though differently realized modally): “to be scientific is to adhere to the "truth" of theory”. In Figure 7 a, the student lodges the "truth" of the facts of the empirical world in the drawing, and the "truth" of theory through the replicability of scientific practice in the written text. In Figure 7 b, the student lodges the "truth" of scientific theory in the drawing, and the "truth" of actual practice in the written. In each case event-like representation uses the mode of writing; and the representation of object-entities is lodged in drawing. |
||
| Design as a part of rhetoric of communication | ||
| The
contemporary social world is marked by increasing fragmentation and individuation
(Beck, 1986); in stark contrast to the world of the 19th and early 20th
century, the world of stable structures and of individual integration
and definition in those structures. Strong frames, and integration into
strong frames had their analogues in communication through stable genres,
and through stable modal ‘choices’. In periods of stability
the question of effective communication is answered by the idea of convention
and of competent action in relation to those conventions. In periods of
fragmentation and individuation communication is fraught: each environment
of communication asks that social and ‘political’ relations,
tastes, needs and desires be newly assessed. The question of rhetoric
– how to make my communication most effective in relation to this
audience, here and now - has moved newly, urgently into the center. Rhetoric
has become a major issue for design. |
||
| |
||
| Bibliography | Kress,
G.R. (2003). Literacy in the new media age. London: RoutledgeFalmer Kress, G.R. and van Leeuwen, T. (1996). Reading Images: the grammar of graphic design. London: Routledge Kress, G.R. and Van Leeuwen, T. (2002). Multimodal Discourse: the modes and media of contemporary communication. London: Edward Arnold Jewitt, C. and G.R. Kress (eds) (2003). Multimodal Literacie. New York: Peter Lang Burn, A. and Parker, D. (2003). Analysing media texts. London: Continuum |
|
| Gunther Kress | Gunther
Kress is professor of Education/English at the Institute of Education,
University of London. He has a specific interest in the interrelations
in contemporary texts of different modes of communication - writing,
image, speech, music - and their effects on forms of learning and knowing.
He is interested in the changes - and their effects and consequences
- brought by the shift in the major media of communication from the
page to the screen. Some of his recent publications are: Reading Images:
the grammar of graphic design; Before Writing: rethinking the paths
to literacy; (both published by Routledge); Multimodal teaching and
learning: the rhetorics of the Science Classroom (Continuum); Multimodal
Discourse: the modes and media of contemporary communication (Edward
Arnold); and Literacy in the New Media Age (Routledge). |
|
| Contact | Links | |
| |
||
Close
window to return home or click here |
||
