International Journal of Multilingualism

ISSN: 1479-0718 (Print) 1747-7530 (Online) Journal homepage: http://www.tandfonline.com/loi/rmjm20

Movie got your tongue? Effects of language
switching on film reception
Xavier Aparicio & Dominique Bairstow
To cite this article: Xavier Aparicio & Dominique Bairstow (2016) Movie got your tongue?
Effects of language switching on film reception, International Journal of Multilingualism, 13:1,
106-119, DOI: 10.1080/14790718.2015.1037306
To link to this article: http://dx.doi.org/10.1080/14790718.2015.1037306

Published online: 08 May 2015.

Submit your article to this journal

Article views: 52

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=rmjm20
Download by: [University of California, San Diego]

Date: 26 February 2016, At: 08:18

International Journal of Multilingualism, 2016
Vol. 13, No. 1, 106–119, http://dx.doi.org/10.1080/14790718.2015.1037306

Movie got your tongue? Effects of language switching on film
reception
Xavier Aparicioa,b* and Dominique Bairstowa,b

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

a

Laboratory Epsylon EA 4556, University of Montpellier 3, Place Albert 1er, 34000 Montpellier,
France; bDepartment of Psychology, University of Paul Valery Montpellier, Route de Mende,
34090 Montpellier, France
(Received 17 December 2014; accepted 23 March 2015)
Cinema is in part a reflection of our society and, in these times of cultural mix, it is
more and more common to find different language communities appearing on-screen
together. Thus, it is not unusual to have to process (voluntarily or not) more than one
language throughout the day. From a cognitive point of view, language switching is
widely studied as it enables to observe interferences or facilitations across languages,
modulated by an individual’s knowledge of those languages as well as by the
orthographic proximity between them. F; ilm comprehension is also studied, particularly depending on the languages used on-screen and those known by the viewer. This
study proposes to unite both paradigms by using a multilingual film (English and
Spanish dialogues), either with subtitles (English or French) or without, to analyse the
effects of language switching on visual and dialogue processing, depending on the
viewers’ knowledge of the languages provided. The results obtained in this study
should increase our understanding of languages through an original approach.
Keywords: foreign language acquisition; bilingualism; code switching; multilingualism;
speech perception; reading comprehension

1. Introduction
1.1. Films and subtitles
Films provide a very interesting and ecological context for studying languages. Indeed,
there is often a close parallel between cinema and everyday situations, which are more
and more multicultural, and therefore increasingly multilingual. As has often been noted,
movies were silent to begin with and, with the arrival of sound and dialogues, their
dissemination was made difficult by language barriers. In order to overcome this issue,
various audiovisual translation techniques were established, either by replacing the
original soundtrack with a new one in the targeted audience’s language (dubbing) or by
adding a translation of the dialogues, orally (voice-over) or in a written form (subtitles).
This additional linguistic information can have consequences on the viewer’s
experience of the film (Lavaur & Bairstow, 2011). Indeed, a movie is made up of
many different sources of information, audio and visual, that can convey either linguistic
(respectively, dialogues and written information) or non-linguistic information (respectively, music or noises and images). Furthermore, this data will be in constant interaction
*Corresponding author. Email: xavier.aparicio@univ-montp3.fr
© 2015 Taylor & Francis

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

International Journal of Multilingualism

107

with the viewer’s own knowledge concerning films in general and, more particularly, the
specific situation shown on-screen. Therefore, the viewing of a movie involving different
language communities should impact its comprehension, with regards to the language
knowledge/experience of the viewer. The assumption is that comprehension difficulty
originates in the presence of language switching within the dialogues of the movie. In
many studies, language switching issues are investigated in an artificial way, with single
word recognition tasks in different languages (see Aparicio & Lavaur, 2013) or within
sentence reading (Bultena, Dijkstra, & Van Hell, in press) or hearing (Costa &
Santesteban, 2004). Here, the use of a multilingual movie provides a more ecological
approach for understanding language switching mechanisms.
In the case of a multilingual film, the presence of more than one language in the
dialogues can give indications as to the situation in which the movie is set or the ethnic/
social position of the characters, amongst other things. When this type of film is
presented to another language community to the ones on-screen, comprehension may be
affected and different subtitling solutions are possible. On the one hand, if the viewers do
not understand any of the languages used in the dialogues, the whole movie
(independently from the language spoken) can be subtitled in the audience’s own
language, implying the presence of three different languages on the screen (two spoken,
one written). On the other hand, if one of the spoken languages is known to the viewers,
then only the other will need subtitling. In this case, two languages will be presented onscreen, one spoken, the other alternatively spoken (dialogues) or written (subtitles). This
can make for very interesting language switching situations and provide important data
concerning the effects of these switches in non-laboratory contexts.
1.2. Multilingual films
The complexity of the situation in which a viewer is placed when seeing a multilingual
film is self-evident when all the different types of information provided by such a movie
are taken into consideration. Indeed, not only does visual and linguistic data need to be
simultaneously processed and linked to previous knowledge, but different languages must
also be understood. When studying multilingualism and multilingual information
processing, this is very interesting as it offers a fairly ecological setting, resembling
real-life situations (Brunick, Cutting, & Delong, 2013; Carroll, 1996; Merleau-Ponty,
1947; Mitry, 2001; Zacks & Magliano, 2011), in which various languages are used.
What is more, the number of films requiring the viewer to process more than one
language has been steadily increasing since the 1980s (Heiss, 2004, with examples for
Germany). This can easily be explained by the fact that movies have a long history of
representing the way in which our societies evolve, in this case in relation to mass
immigration, cultural mix and the side-by-side coexistence of various language
communities in one same country (Baldo, 2009).
These factors, linked to the strong increase in people speaking more than one
language (Lemhöfer, Dijkstra, & Michel, 2004), imply that this line of research is
particularly relevant, especially in the field of cognitive psychology.
1.3. Subtitled film comprehension and language switching
Understanding a film is an amazing feat of cognitive processing (Zacks & Magliano,
2011), implying that the additional processing of different languages and linguistic
sources (spoken and written, in the case of subtitling) is even more exceptional.

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

108

X. Aparicio and D. Bairstow

Indeed, studies have shown that the very presence of subtitles can lead to an automatic
reading behaviour, even when they are unnecessary to comprehension (d’Ydewalle, Praet,
Verfaillie, & Van Rensbergen, 1991), which can induce, in certain cases, a partial loss of
information (Lavaur & Nava, 2008). On the other hand, this visual input of the linguistic
information (subtitles) can also prove to be a benefit as it can add sense to the spoken
input (Marian, 2009). Moreover, these contrasting effects of subtitles on comprehension
have been shown to be strongly dependent on the viewers’ fluency in the languages onscreen, both spoken and written (Lavaur & Bairstow, 2011).
Another line of research in cognitive psychology investigates language switching
which can happen in many different everyday situations. Language switching is a very
common phenomenon in populations speaking different languages, as is the case in
Luxembourg for instance, where at least three different languages are used (Luxembourgish, French and German). From a psycholinguistics point of view, it is widely admitted
that reading or hearing a word initiates a matching with the abstract representations of the
words known by a person. These representations are stored inside the brain in a vast
database called the mental lexicon (van Heuven, Dijkstra, & Grainger, 1998). There, we
store not only the words, but also all the information we possess about their
characteristics (orthographic, phonologic, semantic, syntax), as well as the language
they belong to in the case of multilinguals (Dijkstra, 2005; Grainger & Beauvillain,
1987). The most widely accepted hypothesis in psycholinguistics is that the access to the
multilingual lexicon is non-selective, or language independent, i.e. an incoming word will
first activate all the lexical candidates from the different languages before being identified
(Chauncey, Grainger, & Holcomb, 2008; Christoffels, Firk, & Schiller, 2007; Dijkstra &
Van Hell, 2003; Lemhöfer et al., 2004; Thomas & Allport, 2000; Van Heuven et al.,
1998). According to the Bilingual Interactive Activation model (Grainger & Dijkstra,
1992; van Heuven et al., 1998), this implies that two or more languages could be active at
the same time, leading to possible interferences between them (see Figure 1).
These interferences could induce the intrusion of a word in a different language from
the one spoken (unexpected switches), during language production for instance. This
raises the question of language control in multilinguals, which has a direct impact on
language switching. The term code switching is intended to designate the process
whereby speakers move from one language to another either within a single utterance or
between one utterance and the next in the same interaction (Bentahila & Davies, 1995;
Cantone, 2007). It is different to code-mixing, because it refers to all cases where
lexical items and grammatical features from two languages appear in one sentence
(Muysken, 2000).
To be efficient in the relevant language, it is necessary for multilingual people to
reduce the interferences from the words of the other languages (Dijkstra & Van Hell,
2003). Multilinguals mostly succeed in this task with a relative ease. As mentioned
before, it is not unusual to notice the intrusion of words from a different language to the
one currently spoken within a conversation, intentionally or not (Meuter, 2005; Moreno,
Federmeier, & Kutas, 2002). It is important to notice that the intrusion of words could
occur in a totally unexpected way, eliciting a greater difficulty of comprehension for the
speaker that have to process the switch utterance. According to Moreno et al. (2002), the
intrusion of a word from a different language originates in a retrieval problem of a word
in the language currently spoken, leading to a mix-up with a translation equivalent, used
in order to preserve the meaning of what is being said. For the speaker, language
switching seems easy to operate so as to maintain the intended meaning. For the person at
the receiving end of this switch, however, it is more difficult to process the word in the

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

International Journal of Multilingualism

109

Figure 1. The Bilingual Interactive Activation Model, (BIA, adapted from Grainger, Midgley, &
Holcomb, 2010). For languages sharing the same alphabet (here, English and French), a printed
word stimulus can activate whole-word orthographic representations from both languages as a
function of their orthographic overlap with the stimulus. All activated representations enter in the
competition for word identification, but the probability that a given word is in fact the stimulus is
regulated by the probability that the stimulus is a word in one or the other language (as indexed by
the activation of language nodes).

new language than the word expected in the original language. Indeed, language
switching is often unexpected or unintentional and not necessarily relevant in the context
of the conversation. Nevertheless, it does frequently occur, and not only during verbal
exchanges, but also in reading (Altarriba & Basnight-Brown, 2007), making multilingual
films with subtitles a very interesting and ecological tool for studying language switching
aspects.
1.4. Aims
The main aim of this work was to investigate film comprehension in a multilingual
context, while evaluating the effects of language switching on the processing of movie
information. Indeed, multilingualism is a very up-to-date issue as an increasing number of
people speak, or at least have some understanding, of more than one language, which
increases the occurrence of language switching situations. In addition to this, studies
looking into the mechanisms of language switching have generally measured effects in
highly specific situations using visual presentation of single word lists and rarely with
auditory presentation. Although the data obtained in this way is very interesting since it
gives insight into very precise cognitive processes, whether or not these effects can be
generalised to everyday situations remains an issue.
Bearing this in mind, we carried out an exploratory study in order to examine the
effects of language switching on movie comprehension, depending on different factors.
To begin, the number of languages on the screen was manipulated by adding (or not,
depending on the condition) subtitles to a film extract in which two languages were used
in the spoken dialogues (US-English and Mexican Spanish). In this way, either the film
was presented without subtitles or with either French or English subtitles. Concerning the

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

110

X. Aparicio and D. Bairstow

participants, different profiles were taken into account, depending on their knowledge of
the three languages (French–English–Spanish trilinguals, English monolinguals and
French monolinguals).
When the non-subtitled version of the film is shown to trilingual participants, their
knowledge of both languages used in the dialogues should enable a good global
comprehension of the movie. However, this comprehension may be somewhat affected
when language switches occur, since these switches will take place between their two
non-dominant languages (L2 and L3, see Aparicio & Lavaur, 2013). On the other hand,
when this trilingual population is shown the French-subtitled version, all three of their
languages will be on-screen, presented two at a time (English dialogues with French
subtitles, alternated with Spanish dialogues with French subtitles). In this case, the
viewers can use different viewing strategies: these strategies can include either choosing
to ignore the subtitles as they understand the languages used in the dialogues (although a
certain amount of automatic reading is unavoidable); ignoring the dialogues in order to
concentrate only on the subtitles which are always in the same language; or use the
subtitles selectively when they come across spoken words they do not understand. Either
way, it is in this situation that language switches can be expected to be the most costly for
comprehension since three different languages will be presented on-screen, requiring the
use of successive inhibition/activation mechanisms.
When examining monolinguals’1 French speakers watching the French-subtitled
movie should achieve a good level of comprehension by reading the subtitles and
watching the images. Therefore, language switching should have little effect on their
understanding of the sequence, even though a certain amount of audio information
processing implies that these participants should notice when these language switches
occur. Given that they are reading the subtitles in order to understand the dialogues, we
can also make the hypothesis that their reading activity could lead them to pay less
attention to visual information (especially background visual data).
Finally, with regards to English monolinguals seeing the English-subtitled version of
the movie, language switches in the dialogues (from English over to Spanish, and vice
versa) should compel the viewers to refer to the subtitles when the spoken language is
Spanish. It can therefore be predicted that this condition will provide the biggest effect on
overall film comprehension since the viewers must continuously adapt their viewing
strategy throughout the movie (alternatively watching and listening or watching and
reading).

2. Experimental study
2.2. Method
2.2.1. Materials
An extract from the movie ‘Real women have curves’ (P. Cardoso, 2002) lasting 5
minutes and 10 seconds was selected for this experiment on the basis of strict criteria
(such as languages used in the dialogues, speech intelligibility, time spent speaking each
language, number of switches, number of characters on-screen and action/dialogue ratio,
amongst others). Three versions of the extract were prepared, the non-subtitled version
with English (US) and Spanish (Mexican) dialogues, the French-subtitled version and the
English-subtitled version. In the case of these last two versions, it was made sure that the
subtitles were an exact translation of the spoken dialogues.

International Journal of Multilingualism

111

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

A multiple-choice questionnaire concerning the film was created, including 32 items
referring to the images (16 visual questions) and to the dialogues (16 questions).
Each question was provided with one correct answer, three distractors and one possibility
of answering ‘I don’t know’. A second questionnaire concerning personal data was also
presented to the participants, including questions about language use and proficiency, as
well as auto-evaluation on a five points Likert scale (self-rating). The self-rating includes
questions about reading comprehension, reading frequency, oral production and oral
comprehension for the three languages (French, English and Spanish).
2.2.2. Participants
About 48 participants took part in this study, divided into groups depending on their
language knowledge and the version of the film they were shown (see Table 1). In this
way, two groups of 12 trilinguals (L1 French, L2 English and L3 Spanish; seeing either
the non-subtitled movie or the French-subtitled version) and two groups of monolinguals
(12 French monolinguals seeing the French-subtitled version and 12 English monolinguals seeing the English-subtitled version) were used in this study.
2.2.3. Procedure
To begin, participants were informed that they would see a short film extract (with
subtitles, when it was the case). They were asked to watch the film attentively as they
would have to answer questions about it when it was finished. It was also demanded that
they do not pause the film or go back to re-watch. Following the viewing, the participants
had to fill in the comprehension questionnaire, presented on the same computer. To finish,
they were asked to complete the personal data and the self-rating questionnaires.
All together, the experiment lasted approximately 20 minutes.

Table 1. Main characteristics of all participants depending on the group they were assigned to,
including mean age, mean percentage of time spent using each of the three languages weekly, and
mean self-ratings (on a 5-point Likert scale, standard deviation in parentheses).
Group/
Version

Mean% of language use

Language self-rating

Participants

Mean age

French

English

Spanish

French

English

Spanish

a

TriOV

12

TriFRa

12

MonoFRa

12

MonoENa

12

27.3
(4.1)
33.3
(10.4)
32.1
(7.7)
35
(7.3)

76.7
(23.9)
75.2
(20.5)
86
(14.6)
7.1
(4.1)

12.3
(6.4)
13.1
(7.4)
7
(3.6)
88
(15.6)

11.0
(7.8)
11.7
(5.9)
7
(3.2)
4.9
(3.6)

4.9
(0.2)
5
(0)
4.9
(0.2)
2.7
(1.3)

3.7
(0.6)
3.6
(0.6)
1.5
(1.1)
4.8
(0.4)

3.5
(0.9)
2.4
(1.1)
1.6
(0.6)
1.4
(0.5)

a

Tri refers to trilingual participants, mono refers to monolinguals. OV represents the original version (no
subtitles), FR refers to the version with French subtitles, and GB to the version with English subtitles. Data
presented for the language self-rating are the mean of the answer given by the participants on four criteria
(reading comprehension, reading frequency, oral production and oral comprehension).

112

X. Aparicio and D. Bairstow

This being a preliminary study, full statistical analysis was not carried out from lack of
enough participants for it to be relevant. A descriptive analysis is provided in order to
point out the main tendency followed by our results, encouraging further data collection.
To begin, the results of the personal data questionnaires were analysed in order to sort
the participants into groups depending on their proficiency in each of the languages of
this study. The experimental results of seven participants, that did not meet the language
requirements, were excluded from the study report.
Next, the mean scores for each item of the comprehension questionnaire were
represented graphically in order to examine the way in which the movie data were
processed by each group. The items were coded depending on the type of information
they question (visual- or dialogue-related), but also their level of importance within the
story (level 1 being the most central data and level 2, the more secondary information).
Indeed, concerning visual data, centrality was defined as elements presented in the middle
of the screen, that are integral part of the story (the cake presented to the main character
for example), whereas secondary information relates to elements peripheral to the story
and screen, presented in the background and not particularly relevant to the storyline (the
clothes worn by secondary characters for instance). In the same way, central dialogue
information is part of the exchanges between main characters and that has an incidence
on the story, while secondary information is just passing comments that are not connected
to the main plot. Thus, four types of items were obtained: 8 central visual items (V1-1
through 8); 8 secondary visual items (V2-1 through 8); 8 central dialogue items (D1-1
through 8); and 8 secondary dialogue items (D2-1 through 8), dispatched chronologically
throughout the film (see Appendix for an example of each type of item; the evolution of
comprehension according to visual (V) and dialogue (D) items are presented in Figure 2).
This first graph shows the overall pattern of film comprehension as the film extract
unfolds. Indeed, the bold black line represents the mean comprehension score of all
participants, independently of the version of the movie they were shown. The upper and
lower dotted curves represent the standard deviations at each point in time and provide a
good picture of the variations in comprehension scores due to the different versions of the
movie presented to the participants. Looking at this representation, we can see that film
comprehension is far from being a linear phenomenon and that processing the different
elements that make up a movie can be either positively or negatively influenced. Many
factors can certainly have such an influence, although the experimental setting was
designed to minimise the effect of such factors as movie length, number of characters,
14
Mean comprehension scores

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

3. Results

12
Mean
comprehension

10
8

Upper error

6
4

Lower error

2
0
-2 V1-1 D1-1 V2-1 V2-2 V2-3 V1-5 D1-2D2-4D1-3 V2-5 V2-6 D2-5D1-6D1-8 V1-8 V2-8
Items

Figure 2.

Evolution of comprehension according to visual (V) and dialogue (D) mean scores.

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

International Journal of Multilingualism

113

plot complexity, dialogue complexity and speech intelligibility (speed and accents); thus
we can see that the mean comprehension pattern follows a series of drops that do not take
place at random times. Indeed, we can see that the lowest comprehension scores are
achieved for items referring to information provided around the time of language switches
(illustrated by the vertical dashed lines).
Furthermore, Table 2 also shows that the English monolinguals (MonoEN) seem to
achieve the lowest comprehension level throughout the movie in comparison to all three
other groups, which could indicate that processing linguistic information alternately via
an audio or a visual media has a negative impact on general comprehension. When
comparing the results of the three other groups (TRIOV, TRIFR and MONOFR), the
mean comprehension levels appear near-equivalent. This could imply that the presence of
subtitles in the viewers’ first language can help to overcome language barriers (French
subtitles in the MONOFR condition), provided this language is not also partially present
in the dialogues (English used either in the dialogues or in the subtitles in the MONOEN
condition). Thus, the fact that the French monolinguals (MonoFR) achieve an overall
higher level of comprehension than the English monolinguals (MonoEN) seems to imply
that it is better for comprehension that subtitles in the viewers’ mother tongue be
presented throughout the sequence (French condition), rather than having subtitles during
only those parts in which a different language is spoken (English condition). This will of
course have to be tested with further research.
In this table, the mean number of correct answers for each type of item shows that
visual information is globally processed in a better way when no subtitles are on-screen
(TRIOV) than when they are present. This result is in line with most findings in the field
(Bairstow & Lavaur, 2012; Lavaur & Nava, 2008) evidencing that the presence of
subtitles can distract the viewer from fully processing the visual information provided.
However, concerning dialogue information, the mean scores are generally higher when
subtitles in the viewer’s language are continuously shown (TRIFR and MONOFR) than
when they are absent (TRIOV) or only shown when Spanish is spoken (MONOEN).
When looking at the detailed scores, we can see that the negative effect of the
presence of subtitles on visual information processing seems to be stronger on secondary
elements (V2) rather than on central information (V1). This could be explained by the
central position of the subtitles on the screen which is also where most of the relevant
visual action will take place, making it easier to switch from the subtitles to that zone of
the picture, rather than to a more peripheral part of the screen.
Concerning dialogue information, the results in Table 3 show that both central (D1)
and secondary (D2) information is better processed by the French monolinguals seeing
the French-subtitled version of the film (MONOFR). This indicates that seeing a movie
Table 2. Mean comprehension scores (and standard deviations) for each language level (trilinguals
and monolinguals) and each version of the films (original version – OV, version with French
subtitles – FR, version with English subtitles – EN, according to the spoken language).
English dialogues

TRIOV
TRIFR
MonoFR
MonoEN

Spanish dialogues

Mean

SD

Mean

SD

6.7
7.5
9.2
5.0

4.2
3.4
3.3
2.7

9.5
9.8
10.2
6.6

2.7
2.5
2.2
1.8

114

X. Aparicio and D. Bairstow

Table 3. Mean scores (and standard deviations) to each type of item (visual or dialogues, central or
secondary) by each language level (trilingual or monolingual) and for each version of the film
(original, subtitled in French or subtitled in English).

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

Visual
Group/version

Central (V1)

TRIOV
TRIFR
MONOFR
MONOEN

10.25
9.25
10.50
6.00

(1.98)
(2.05)
(1.41)
(2.07)

Dialogue

Secondary (V2)
9.00
6.50
7.25
5.13

(3.07)
(3.82)
(3.81)
(3.39)

Central (D1)
7.25
8.75
11.00
6.00

(4.23)
(2.92)
(0.53)
(2.00)

Secondary (D2)
7.13
11.25
10.50
6.88

(4.05)
(1.04)
(2.00)
(1.56)

with dialogues in foreign languages and subtitles in the mother tongue is effective for a
good comprehension of linguistic information. When the viewer has some knowledge of
the languages used in the dialogues however, the presence of subtitles seems to have a
positive effect primarily on the processing of secondary elements (D2). The subtitles
therefore appear to enable the viewer to process details of linguistic information that were
not readily available through listening to the dialogues presented in the trilinguals’ L2
(English) and L3 (Spanish).
Finally, both central and secondary dialogue information seem to be more difficult to
process when the linguistic information is given in turn in either the dialogues or in the
subtitles (MONOGB). This alteration of dialogue comprehension could be linked to the
discontinuous transmission mode of linguistic information, switching from an oral mode
(dialogues) to a written on (subtitles) throughout the film and shall have to be the subject
of more extensive research in the future.

4. Discussion
Four groups differing in terms of language knowledge and use (trilinguals and
monolinguals) were asked to watch an excerpt of a movie in its original version or
with subtitles in their native language. The experimental conditions involved respectively
two (original version with English and Spanish dialogues, either with or without English
subtitles) or three languages (additional French subtitles), and the performance of the
participants was evaluated according to their good processing of visual aspects, dialogues
and languages involved in the film. Our explorative study was designed to examine the
influence of the knowledge of the participants concerning the languages used on-screen
on the dimensions associated with film comprehension (dialogues and visual components). The choice of these different populations of participants was made in order to
evaluate different combinations of languages with regards to the version of the movie
visualised. The study was carried out keeping in mind that the trilingual participants are
used to dealing with language switching aspects, as opposed to the monolingual
participants, who only have a very low knowledge of any other languages apart from
their mother tongue.
4.1. General comprehension
According to our findings, the comprehension of a movie is far from being frozen during
the viewing, as the viewers tend to lean more on one dimension or the other (visual or
dialogue) as the story unfolds. Of course, the results obtained here are modulated by the

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

International Journal of Multilingualism

115

version of the movie and the participants’ skills in terms of language proficiency, but the
evolution of comprehension is verified across our four groups. The mean scores show that
the general performance of the different groups concerning both dialogue and visual
aspects are very close, except for the group of monolingual English speakers, which is on
average below the three other groups. This difference could originate in the fact that,
when monolingual English speakers process the original version of the film with English
subtitles, they must process linguistic information presented alternately in both spoken
dialogues and on a visual modality (subtitles). These participants therefore have to change
viewing strategies at each language switch as it is very different to watch a film that is not
subtitled and one in which the viewer must read the subtitles to achieve satisfactory
comprehension. This constant switching of viewing strategies may be the cause of the
lower overall comprehension scores for this group and may be linked to an inability to
inhibit (or not properly) one modality or the other (or to handle both modalities; for
cognitive control, see Bialystok, Craik, & Luk, 2008; Aparicio, Heidlmayr, & Isel, 2015).
This hypothesis shall have to be tested in future research with an appropriated paradigm
and measuring tool. Concerning the performances of the other groups, clear differences
seem to arise depending on the version watched and their language abilities (Bairstow,
2011; Lavaur & Bairstow, 2011; Lavaur & Nava, 2008).
The comparison between French and English monolinguals clearly highlights better
performance for French participants. An attempt to explain the monolingual English
group’s lower results has been provided in the previous paragraph. For the second group,
and given that French monolinguals are not familiar with the two languages spoken, it is
clear that they can freely concentrate on the visual modality (images and subtitles),
although the auditory channel is never totally ignored (as proved by the high scores to the
language recognition questions). Moreover, when comparing the French monolinguals,
seeing the film with subtitles in their mother tongue, with the trilinguals seeing the
original version, we can see that similar comprehension levels are achieved. This result
indicates that subtitles do enable satisfactory comprehension of a film and are thus an
effective method to overcome language barriers, confirming the results of previous
studies (Bairstow, 2011; Bairstow & Lavaur, 2012; Grignon, Lavaur, & Blanc, 2007;
Lavaur & Bairstow, 2011).
However, although visual elements are mainly correctly processed by the viewers
seeing the subtitled film, we can see that the processing of secondary visual elements is
affected, since lower scores are obtained. These results confirm data collected in previous
experiments run by the authors that seem to point to the fact that while the presence of
subtitles do not necessarily impair visual processing as such, it may however impede in
the perception or processing of more secondary information. This trend needs to be
explored, however, notably to determine whether this effect is linked to the more
peripheral location of this type of visual information on the screen, or whether it is a
strategy to ignore information that seems less important in order to concentrate limited
cognitive resources on more important data processing.
In the present study, the two groups of trilinguals represent the central issue, because
they have knowledge of each of the three languages presented within the movie. The
comparison of these two groups confirms our hypothesis concerning the different effects
of the presence of subtitles on both dimension of film comprehension (visual and
dialogue). According to our results, the presence of subtitles leads to lower performances
on the visual dimension, while better results for dialogue comprehension are obtained.
This seems to indicate that more attention is focused on the subtitles presented in their
mother tongue than to the spoken dialogues (in their second and third languages). The

116

X. Aparicio and D. Bairstow

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

higher dialogue-related information processing can therefore be linked to the fact that the
first language has a higher level of activation inside the mental lexicon and is basically
automatically processed (because of the well-demonstrated automaticity of reading).
4.2. Language switching and language effects
As was mentioned in the theoretical framework presented at the beginning of this chapter,
language switching is often associated with a cost, which originates in the difficulty of
inhibiting one language in order to activate the other one, even if all languages possess a
residual level of activation (Aparicio & Lavaur, 2013; Dijkstra, 2005). Here, the two
groups of monolingual participants are considered as control groups, because they have a
very limited knowledge of the other languages presented on the screen. In the absence of
representations of words from an unknown language in the mental lexicon, the
interferences are of course non-existent.
Nevertheless, very interesting findings arise when looking at the performance of both
groups of trilinguals. For both groups, there is a disruption consecutive to a language
switch leading to a drop in the mean number of correct answers. Interestingly, this cost
associated with language switching seems to affect more strongly the visual components
of comprehension. This could be due to the fact that the activity of switching from one
language to another strongly requires linguistic components in order to successfully
process this unexpected language switch. As this activity requires attentional processes
and the intervention of executive functions (Aparicio & Lavaur, 2013; Aparicio et al.,
2015; Bialystok et al., 2008), this could explain the lower score obtained for visual
information. In other words, it is possible that during and straight after a language switch,
the participants’ cognitive resources are temporarily focused in the act of re-establishing
linguistic coherence, inducing an instantaneous drop in movie information processing.
Once again this hypothesis will need to be tested in an appropriate paradigm enabling
online measures of cognitive effort.
Moreover, trilingual participants are clearly more accurate at determining which
language was spoken at a key moment of the film (79.6% and 79.2% of accuracy)
compared to monolingual participants (61% for French monolinguals, 69.4% for English
monolinguals). These findings show that the knowledge of the languages, associated with
a greater ability to deal with language switching, permits a better ability to identify
languages. For monolinguals, although they are able to identify that the spoken language
has changed, the identification of the language spoken at a key moment is less efficient.
Nevertheless, the better results obtained by English monolinguals in comparison with the
French ones is probably due to a strategy bias, given that if it was not their mother tongue
which was spoken, it was necessarily Spanish.
To finish, it is also important to notice that in the trilinguals’ performances after
viewing the original version, there is a lower level of general comprehension when Spanish
is spoken on the screen. This effect could be due to the fact that Spanish is the latest
acquired language of our participants and, as demonstrated by Aparicio et al. (2012), age of
acquisition as well as proficiency could account for semantic processing in multilinguals.
4.3. Conclusions
This preliminary study has provided some interesting data concerning multilingualism
and multilingual situations, in addition to effects of language switching. Many differences
in processing have been found depending on various factors, such as language

International Journal of Multilingualism

117

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

knowledge, number of languages on the screen, presentation mode (spoken dialogues
and/or written subtitles) and components of comprehension (visual or linguistic
information). Some hypotheses concerning the reasons behind these differences have
been provided, notably in relation to cognitive load and working memory limitation,
which constitute good leads for future research. In this view, this line of research appears
as very interesting for investigating many different aspects of multilingual situations in a
wide range of experimental studies. Considering the large number of dimensions that can
be examined using multilingual films as a basis, a rich, long and interesting future can be
predicted for these studies.
Disclosure statement
No potential conflict of interest was reported by the authors.

Note
1.

As it is more and more difficult nowadays to find monolingual people, in the present study the
term monolingual is associated with a very scarce use and knowledge of a second language
(below 10%).

References
Altarriba, J., & Basnight-Brown, D. M. (2007). Methodological considerations in performing
semantic- and translation-priming experiments across languages. Behavior Research Methods,
39(1), 1–18. doi:10.3758/BF03192839
Aparicio, X., Heidlmayr, K., & Isel, F. (2015). Effect of inhibition training on language switching
and Stroop tasks: Evidence from late bilinguals and simultaneous interpreters. Language and
Cognitive Processes. Manuscript submitted for publication.
Aparicio, X., & Lavaur, J.-M. (2013). Recognising words in three languages: Effects of language
dominance and language switching. International Journal of Multilingualism, 11, 164–181.
doi:10.1080/14790718.2013.783583
Aparicio, X., Midgley, K. J., Holcomb, P. J., Pu, H., Lavaur, J.-M., & Grainger, J. (2012). Language
effects in trilinguals: An ERP Study. Frontiers Psychology, 3(402), 1–10.
Bairstow, D. (2011). Audiovisual processing while watching subtitled films: A cognitive approach.
In A. Şerban, A. Matamala, & J. M. Lavaur (Eds.), Audiovisual translation in close-up:
Practical and theoretical approaches (pp. 205–219). Bern: P. Lang.
Bairstow, D., & Lavaur, J.-M. (2012). Audiovisual information processing by monolinguals and
bilinguals: Effects of intralingual and interlingual subtitles. In A. Remael, P. Orero, & M.
Carroll (Eds.), Audiovisual translation and media accessibility at the crossroads (pp. 273–294).
Amsterdam/New York, NY: Rodopi.
Baldo, M. (2009). Subtitling multilingual films. The case of Lives of the Saints, an ItalianCanadian TV screenplay. In F. Federici (Ed.), Translating voices for audiovisuals (pp. 117–
135). Roma: Aracne, chapter 6.
Bentahila, A., & Davies, E. E. (1995). Patterns of code-switching and patterns of language contact.
Lingua, 96, 75–93. doi:10.1016/0024-3841(94)00035-K
Bialystok, E., Craik, F., & Luk, G. (2008). Cognitive control and lexical access in younger and
older bilinguals. Journal f Experimental Psychology: Learning, Memory, and Cognition, 34,
859–873.
Brunick, K. L., Cutting, J. E., & Delong, J. E. (2013). Low-level features of film: What they are and
why we would be lost without them. In A. Shimamura (Ed.), Psychocinematics: The aesthetic
science of movies (pp. 133–148). New York, NY: Oxford.
Bultena, S., Dijkstra, T., & Van Hell, J. G. (in press). Processing language switches in sentence
context: Evidence from self-paced reading. Bilingualism: Language and Cognition, 1–17.
doi:10.1017/S1366728914000145
Cantone, K. F. (2007). Code-switching in bilingual children. Studies in Theoretical Psycholinguistics, 37. Springer Netherlands.

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

118

X. Aparicio and D. Bairstow

Carroll, N. (1996). Theorizing the moving image. Cambridge: Cambridge University Press.
Chauncey, K., Grainger, J., & Holcomb, P. J. (2008). Code-switching effects in bilingual word
recognition: A masked priming studies with ERPs. Brain and Language, 105, 161–174.
doi:10.1016/j.bandl.2007.11.006
Christoffels, I. K., Firk, C., & Schiller, N. O. (2007). Bilingual language control: An event-related
brain potential study. Brain Research, 1147, 192–208. doi:10.1016/j.brainres.2007.01.137
Costa, A., & Santesteban, M. (2004). Lexical access in bilingual speech production: Evidence from
language switching in highly proficient bilinguals and L2 learners. Journal of Memory and
Language, 50, 491–511. doi:10.1016/j.jml.2004.02.002
Dijkstra, T. (2005). Bilingual visual word recognition and lexical access. In J. F. Kroll & A. de
Groot (Eds.), Handbook of bilingualism: Psycholinguistic approaches (pp. 179–200).
Cambridge: Cambridge University Press.
Dijkstra, T., & Van Hell, J.G. (2003). Testing the language mode hypothesis using trilinguals.
International Journal of Bilingual Education and Bilingualism, 6(1), 2–16. doi:10.1080/
13670050308667769
D’Ydewalle, G., Praet, C., Verfaillie, K., & Van Rensbergen, J. (1991). Watching subtitled
television: Automatic reading behaviour. Communication Research, 18, 650–666.
Grainger, J., & Beauvillain, C. (1987). Language blocking and lexical access in bilinguals. The
Quarterly Journal of Experimental Psychology, 39, 295–319. doi:10.1080/14640748708401788
Grainger, J., & Dijkstra, T. (1992). On the representation and use of language information in
bilinguals. In R. J. Harris (Ed.), Cognitive processing in bilinguals. Amsterdam: North Holland.
Grainger, J., Midgley, K. J., & Holcomb, P. J. (2010). Re-thinking the bilingual interactiveactivation model from a developmental perspective (BIA-d). In M. Kail & M. Hickmann (Eds.),
Language acquisition across linguistic and cognitive systems (pp. 267–284). New York, NY:
John Benjamins.
Grignon, P., Lavaur, J.-M., & Blanc, N. (2007, July 8–11). The effect of subtitles on film
understanding. Presented at the Seventeenth annual meeting of the society for Text and
Discourse. Glasgow, Scotland.
Heiss, C. (2004). Dubbing multilingual films: A new challenge? Meta, 49, 208–220.
Lavaur, J.-M., & Bairstow, D. (2011). Languages on the screen: Is film comprehension related to
the viewer's fluency level and to the languages in the subtitles? International Journal of
Psychology, 46, 455–462. doi:10.1080/00207594.2011.565343
Lavaur, J.-M., & Nava, S. (2008). Interferences related to intra-lingual subtitling on pictures
processing of a movie. In J.-M. Hoc & Y. Corson (Eds.), Published acts of the 2007 conference
of the French Society of Psychology (pp. 59–64). Nantes, France.
Lemhöfer, K., Dijkstra, A., & Michel, M. C. (2004). Three languages, one ECHO: Cognate effects
in Trilingual word recognition. Language and Cognitive Processes, 19, 585–611.
Marian, V. (2009). Audio-visual integration during bilingual language processing. In A. Pavlenko
(Ed.), The bilingual mental lexicon: Interdisciplinary approaches (pp. 52–78). Bristol:
Multilingual Matters.
Merleau-Ponty, M. (1947). Cinema and the new psychology. Modern Times, 3, 930–947.
Meuter, R. F. I. (2005). Language selection in bilinguals: mechanisms and processes. In J. F. Kroll
& A. de Groot (Eds.), Handbook of bilingualism: Psycholinguistic approaches (pp. 349–368).
Cambridge: Cambridge University Press.
Mitry, J. (2001). Esthetics and psychology of cinema. Paris: Cerf.
Moreno, E. M., Federmeier, K. D., & Kutas, M. (2002). Switching languages, switching palabras
(words): An electrophysiological study of code switching. Brain and Language, 80, 188–207.
doi:10.1006/brln.2001.2588
Muysken, P. (2000). Bilingual speech: A typology of code mixing. Cambridge: Cambridge
University Press.
Thomas, M. S. C., & Allport, A. (2000). Language switching costs in bilingual visual word
recognition. Journal of Memory and Language, 43(1), 44–66. doi:10.1006/jmla.1999.2700
Van Heuven, W. J. B., Dijkstra, T., & Grainger, J. (1998). Orthographic neighborhood effect in
bilingual word recognition. Journal of Memory and Language, 39, 458–483. doi:10.1006/
jmla.1998.2584
Zacks, J. M., & Magliano, J. P. (2011). Film, narrative, and cognitive neuroscience. In F. Bacci &
D. Melcher (Eds.), Art and the senses (pp. 435–454). New York, NY: Oxford University Press.

International Journal of Multilingualism

119

Downloaded by [University of California, San Diego] at 08:18 26 February 2016

Appendix. Examples of each type of item from the comprehension questionnaire
(visual central and secondary; dialogue central and secondary), with the multiplechoice answers (correct answer in bold characters).

Type of item

Item

Visual
central (V1)
Visual
secondary
(V2)
Dialogue
central (D1)

The young lady enters a house
and speaks with:
When Anita arrives in the garden,
her aunt:

Dialogue
secondary
(D2)

A young man; A woman; A man; A young
lady; I don’t know
Gives her a drink; Gives her a hug; Gives her
a slice of cake; Gives her a seat; I don’t know

Her mother reproaches her for
quitting her job. Anita
explains that:

She couldn’t stand the long hours; She wasn’t
getting along with the manager; She wasn’t
being paid enough; She wasn’t getting along
with her colleagues; I don’t know
The mother finishes by saying that Sixteen; Fifteen; Fourteen; Thirteen; I
she’s been working since the
don’t know
age of:

Note: The items shown in this table are examples and are not necessarily consecutive within the experimental
questionnaire.