E300F Term 2 Week 10-1

Visual Communication for Making Meaning (I):

Word and Image in Advertisement and Poetry
Visual communication and
literary creativity
• These lectures will focus on a range of
printed literary texts (e.g.
advertisements, poems, narratives)
which use visual communication (e.g.
illustrations, images) as a meaning-
making resource.

• The way in which different aspects of

texts and images combine will be
considered with a view to
understanding how their analyses can
illuminate aspects of literary creativity.
• In order to look more closely at how
combinations of words and images
communicate to us as readers and
make meaning (i.e. produce literary
creativity), two approaches will be
adopted; they are:

– 1) semiotics; and
– 2) a ‘literary studies’ approach.

• Both approaches enable us to

investigate the use of different
forms of communication (e.g. word,
image) in a single text to make
meaning and communicate 
multimodality.
Multimodality

• Multimodality refers to the use of different forms of

communication in a single text to make meaning and
communicate.

• In these lectures, we are primarily concerned with

texts of the genres of poetry and fiction and with how
they utilize images and words to make meaning.

• Multimodality is researched in a range of academic

disciplines such as linguistics, art history, information
technology, sociology, cultural studies and media
studies.
• In these disciplines, multimodal
texts are suggested to tell us the
following HOW aspects:

– How individual modes can

communicate meaning.

– How the interaction of different

modes can communicate meaning
which is different from the meaning
of the constituent individual modes
and how the interaction opens up
new possibilities for interpretation.

– How such combinations and

juxtapositions of different modes can
be seen to enhance, reinforce or
contradict each other, making
meaning unstable and challenging.
• Based on what can be suggested by
multimodality, multimodality is in place to tell
us something about:

– linguistic and artistic creativity;

– the social and cultural spheres in which we live;

and

– ourselves as producers and consumers of texts.

A note about ‘text’

• Since multimodality is concerned with the

interaction of different modes in a single ‘text’,
it is important to set out a definition of ‘text’
that is broad enough for the purposes of
studying multimodality in ‘texts’.
• According to Rubrecht (2001), a text is:

– any artifact;
– produced or modified;
– to communicate meaning

• This definition allows us to be acquainted with

a broader sense of ‘text’ and to consider how
meaning can be conveyed via a range of
textual elements (e.g. words in their
upper/lower case, monochrome or colourful
images, punctuations, typographic
conventions).
• Rubrecht’s understanding of ‘text’ is that
something which is able to make
meaning is called a ‘text’. In this way:

“A piece of driftwood on the beach is

not an artifact, just a random object
shaped and placed by natural forces. If a
beachcomber takes it home, paints a
face on it, and hangs it on a wall, it turns
into a ‘text’ communicating the
beachcomber’s ideas about what is
interesting and beautiful. […] A text is
[thus] purposeful” (Rubrecht, 2001).
Semiotics

• In order to analyse multimodal texts, we have to turn to

semiotics.

• Semiotics is a well-established approach to the study of

language and other forms of communication which are
socially and culturally meaningful.

• Its fundamental premise is that we use signs – words (both

spoken and written), images, clothing, gesture – to
communicate meaning.

• In this way, a semiotic framework is applicable to language,

images, photographs, diagrams – any aspect of the text
which can be seen to carry meaning.
Theoretical underpinnings of semiotics:

• In semiotics, the basic unit of communication

is the ‘sign’.

• Each ‘sign’ is made up of two parts:

– Signifier  the linguistic (written, spoken) or

visual representation

– Signified  the concept the signifier represents.

• Example:

– ‘Apple’  signifier  the written

representation

–[ ]  signifier  the spoken

representation

– ‘A hard round fruit that has red skin

and is white inside’  signified
• There are three types of sign:

1) Symbolic signs  the signifier does not

resemble the signified

• Example:

– Neither the written representation ‘apple’ nor the

spoken representation [ ] resembles the
concept the representation refers to.

– In this way, the relationship between the ‘signifier’

and the ‘signified’ is ‘arbitrary’ and the relationship
(i.e. meaning) has to be learnt and understood.
 2) Iconic signs  the signifier
resembles the signified

• Examples:

– When you look at the portrait of

someone, you readily recognize who
s/he is.

– Chinese is a logographic language. Most

Chinese characters are iconic in their
origin, though almost all of them have
become more and more arbitrary.
 3) Indexical signs  the signifier
has some kind of causal relationship
with the signified

• Example:

– Smoke is an index of fire.

– Chicken crow is an index of

sunrise/morning.
A semiotic approach to advertisement
• Task: Take a look at
the image ‘Baby
McFry’. What signs
seem meaningful to
you and how do you
interpret the image?
• Analysis:

• There are two levels of analysis of the

signs of the image on the levels of
denotation and connotation.

• On the level of denotation:

– The image is iconic – a photograph of a

toddler.

– The image denotes a particular child,

wearing those particular clothes, at
the particular time the photograph
was taken.
• On the level of connotation:

– The image is complex.

– It is made up of component
signs. These signs include:

• The McDonald’s corporate logo

• The McDonald’s logo is one of
the best known logos in the
world
• The bib bearing the logo is
significant
• The hat bearing the McDonald’s
French fries is also significant
• Connotations evoked by
signs, unlike denotations,
are not universal.

• In other words, different

people may read the same
image in different ways.

• In this way, images require

some interpretative effort
on the part of the reader:
the more time you spend
looking at them, the more
you will probably see.
Questions:

• What does the ‘Baby McFry’ connote? What

are the connotations evoked to you by the
‘Baby McFry’ image?
Answers:

• On the one hand, the

image may connote
pleasure, attracting you to
McDonald’s to have a meal.

• On the other, the image is

indeed an ‘anti-
advertisement’ or
‘subvertisement’
produced by Adbusters.
• More about Adbusters:

– Adbusters is a network of artists and activists

concerned about ecological and commercial issues.

– They are known for their anti-consumption campaigns

such as ‘Buy Nothing Day’ and ‘TV Turnoff Day’.
• McDonald’s is a global brand and it has been targeted
by anti-capitalism activists (e.g. Adbusters), who raise
concerns about the environmental damage stemming
from the production of ‘fast food’ (e.g. McDonald’s).
• When you are viewing the advertisement, you
may or may not have access to all of this
background information (i.e. McDonald’s being
targeted by anti-capitalism activists).

• Your interpretation of such a spoof advertisement

therefore depends on your recognition (or not) of
at least some of the current controversies at the
time when the advertisement was shown.

• To conclude:

One’s interpretation of the possible connotations of

such ‘texts’ is dependent on one’s experience in a
socio-cultural context.
A semiotic approach to poetry
• This section will focus on the
semiotic analysis of images and
words in poetry, by means of
which the literary creativity in such
multimodal texts is explored.

• Two visual aspects will be

investigated:

– paralanguage such as letterforms,

punctuation and layout; and

– concrete poetry.
Paralanguage

• Definition of paralanguage:

– Linguists generally define paralanguage as features of

language which are combined with words to create
meaning and to produce additional meaning.

– In spoken texts, paralanguage refers to features such as

intonation, pitch, tempo, tone, gesture, facial expression
and movement.

– In written language, the equivalent refers to the way

certain writers play creatively with letterforms, layout of
words on the page and different typefaces, all of which can
be highly motivated, creative and meaningful.
• Task: Read the poem below, by e e
cummings, about the experience of
driving a new car. Try to work out
the following questions about the
relationship of paralanguage and
meaning making:

– How is the spatial layout used as a

semiotic device?

– How are punctuation and case, being

signs, used to create meaning?

– How are deviations from convention

used creatively?
she being Brand

-new;and you
know consequently a
little stiff i was
careful of her and(having

thoroughly oiled the universal

joint tested my gas felt of
her radiator made sure her springs were O.

K.)i went right to it flooded-the-carburetor cranked her

up,slipped the
clutch(and then somehow got into reverse she
kicked what
the hell)next
minute i was back in neutral tried and

again slo-wly;bare,ly nudg. ing (my

lev-er Right-
oh and her gears being in
A 1 shape passed
from low through
second-in-to-high like
greasedlightning)just as we turned the corner of Divinity

avenue i touched the accelerator and give

her the juice,good

(it
was the first ride and believe i we was
happy to see how nice she acted right up to
the last minute coming back down by the Public
Gardens i slammed on
the
internalexpanding
&
externalcontracting
brakes Bothatonce and

brought allof her tremB

-ling
to a:dead.

stand;-
;Still)

(cummings, 1960, p.15-16)

Analysis:

1) Spatial layout

• The overall spatial layout of the poem is highly connotative.

• The layout of the poem conveys through its iconic shape

the juddering first minutes of a man trying out his new car.

• The iconic shape includes:

– The lines are arranged in short stanzas.

– Words are split across both lines and stanzas.

• Such a layout of the poem is iconic because the way the

reader reads the poem in a disrupted manner resembles
the jerkiness of the driving experience.
2) Deviation (punctuation, words, grammar)

• Punctuation

– The punctuation is highly non-standard and

forces us to notice it.

– Punctuation in the poem functions to slow

us down and speed us up, which resembles
the driver’s jerky experience of his driving
the car.
• Words

– Words in the poem are also compressed into single

units by the removal of spaces (e.g. Bothatonce),
which conveys speed and abruptness in the driving
experience.

• Grammar

– There are instances of creative rule-breaking on the

grammatical level and capitalization

– Examples:

• believe i we was
• ‘i’ for the driver; ‘she’ for the car
Concrete poetry
• Definition of ‘concrete
poetry’:

– Concrete poetry, also called

‘pattern poetry’, is where lines
of poems are arranged in a
specific shape on the page in
a meaningful way. Concrete
poetry is used for poems
where the visual shape is
paramount so that the visual
shape reinforces or acts as a
counterpoint to the verbal
meaning.
• Task: The following is an example of concrete
poetry -- ‘The Mouse’s Tale’ from Lewis
Carroll’s Alice’s Adventures in Wonderland
(1865). When you read the poem, think of
how the physical shape of the poem
influences your interpretation.
Analysis:

• When read aloud, this poem is similar to others in that its rhyme
and metre are easily felt.

• However, the words are arranged on the page in a significantly

deviant way.

• The significantly-deviant layout of the poem is eye-catching (i.e.

made strange) due to its usual shape and thus can be seen as
deviating from the ‘normal’ layout convention of poetry 
deviation/defamiliarization

• The poem is also stylistically poetic because it contains an obvious

pun.

• The pun operates between two semiotic modes:

– Verbal semiotic mode  tale

– Visual semiotic mode  tail