RELC Journal http://rel.sagepub.

com/

Second Language Listening and Unfamiliar Proper Names: Comprehension Barrier?

Polina P. Kobeleva RELC Journal 2012 43: 83 DOI: 10.1177/0033688212440637 The online version of this article can be found at: http://rel.sagepub.com/content/43/1/83

Published by:
http://www.sagepublications.com

Additional services and information for RELC Journal can be found at: Email Alerts: http://rel.sagepub.com/cgi/alerts Subscriptions: http://rel.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://rel.sagepub.com/content/43/1/83.refs.html

>> Version of Record - May 1, 2012 What is This?

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

440637
2012

REL43110.1177/0033688212440637KobelevaRELC Journal

Article

Second Language Listening and Unfamiliar Proper Names: Comprehension Barrier?

Polina P. Kobeleva
School of Information and Social Sciences, Open Polytechnic, New Zealand

RELC Journal 43(1) 8398 The Author(s) 2012 Reprints and permission: sagepub. co.uk/journalsPermissions.nav DOI: 10.1177/0033688212440637 rel.sagepub.com

Abstract
This study examines whether unfamiliar proper names affect English as a second language (ESL) learners listening comprehension. A total of 110 intermediate to advanced ESL learners participated; comprehension of a short news text was tested under two conditions, Names Known (all proper names pre-taught in advance) and Names Unknown (all proper names unfamiliar prior to listening). The Names Known group performed significantly better than the Names Unknown group on two measures, true-false-dont know statements and open-ended questions. The effect of unfamiliar proper names on comprehension remained statistically significant after the listening proficiency variable was controlled. Listeners in the Names Unknown treatment were able to derive 44.8% of information about the referents of proper names from context but they could not identify all the referents successfully. Unfamiliar proper names evidently hinder comprehension, which was also supported by the participants task difficulty ratings. When proper names were familiar, participants rated all comprehension tasks as easier to do and self-reported higher comprehension success. In contrast, when proper names were unfamiliar, participants rated all comprehension tasks as harder to do and self-reported a lower amount of comprehension. It has long been established that vocabulary knowledge is a prerequisite to successful comprehension. This study confirms that, similarly to common vocabulary, the presence of unfamiliar proper names decreases the ESL learners chances of understanding spoken English. Implications of the findings for second language learning and testing are discussed. It is recommended that teachers be flexible when setting up listening exercises and use different techniques depending on the depth of understanding the learners are supposed to achieve. If the instructional goal is listening for gist, proper names will unlikely affect the outcomes of the activity. However, if detailed and precise understanding is required, then it is unrealistic to expect ESL listeners to cope with unfamiliar proper names on the spot without assistance or preparation. It is also advisable to avoid unfamiliar proper names in a testing situation because they may cause unwarranted anxiety and reflect negatively on the ESL listeners performance.
Corresponding author: Polina P. Kobeleva, School of Information and Social Sciences, Open Polytechnic, 3 Cleary Street, Lower Hutt 5011, New Zealand. Email: Polly.Kobeleva@openpolytechnic.ac.nz

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

84 Keywords

RELC Journal 43(1)

proper names, proper nouns, listening comprehension, adequate comprehension, comprehension testing, English as a second language, vocabulary learning, vocabulary threshold

Introduction Comprehension and Vocabulary Knowledge

Comprehension is vitally important for foreign language and second language learners. A lot of effort in applied linguistics has been dedicated to examining various aspects of comprehension as a process as well as ways to facilitate the development of comprehension skills via effective teaching. In recent years excellent progress has been made with investigating how listening comprehension differs from reading comprehension and what factors influence English as a second language (ESL) learners ability to understand connected speech. Listening comprehension is affected by a large number of factors which can be divided into text-based, context-based and listener-based. Text-based factors include text genre and length (Freedle and Kostin, 1999; Shohamy and Inbar, 1991), complexity of syntactic structure (Blau, 1982; Buck, 2001: 150), availability of discourse organization clues (Jung, 2003), and acoustic qualities of the recording or the speakers voice (Anderson-Hsieh and Koehler, 1988; Blau, 1990; Griffiths, 1992; Whitson, 1972, cited in Kelly, 1991). The position of comprehension questions (whether the questions precede, follow, or are integrated into the text) (Hung, 2007; Sherman, 1997; Wu, 1998) and the presence or absence of visual support (Arcario, 1990: 113-17) are examples of contextual factors. Listener-based factors include background knowledge of the subject (Long, 1990; Markham and Latham, 1987; Schmitt-Rinehart, 1994), affective state (Elkhafaifi, 2005; Matsuura, 2007), general language proficiency level (Hirai, 1999), and, in particular, listening proficiency (Chiang and Dunkel, 1992). However, it has long been established that both for native speakers and second language users alike, vocabulary knowledge is the main variable affecting comprehension and a prerequisite for success (Laufer and Sim, 1985a, 1985b; Kelly, 1991; Nuttall and Alderson, 1996: 62-77; Buck, 2001: 54, 58). To put it simply, the more words you know, the more you are likely to understand. The link between comprehension and vocabulary knowledge is well documented but the vast majority of studies in this area have dealt with written as opposed to spoken texts. This study aims to fill the gap by building on the few available studies which investigate how vocabulary knowledge impacts on ESL learners ability to understand spoken English.

Comprehension and Proper Names

A proper name is a word or group of words whose primary function is referring to a particular fixed entity and differentiating this entity from others. Proper names include names of persons, geographical locations, organizations, events, works of art etc. Proper names have been scrutinized in philosophy and linguistics but to date largely overlooked in second language acquisition and vocabulary research. Unlike general high frequency words, proper names do not provide significant text coverage therefore, it is concluded,

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva

85

they are not important for ESL learners to learn: whenever somebody learns a proper name of an entity his or her gain is typically limited to mastering a foolproof ability to refer to only one person or object (i.e. the one he or she has been introduced to), whereas learning a new non-name gives the speaker a tool to refer to any number of referents (Berezowski, 2001: 66). Some authors also hold that knowledge of proper names is encyclopaedic rather than linguistic (Napoli, 1997). No ESL learner can claim that they have mastered English if they do not understand common words, for instance person or town. In contrast, such expressions as Polly Kobeleva or Hobsonville are optional in the sense that one can be a competent English user without knowing them. For these reasons, applied linguists did not accord proper names the attention they deserve. For example, Wilbur Ames was interested in classifying various clues that help readers to guess the meanings of unknown words from context; in his carefully chosen sample of experimental texts proper nouns (perhaps excusably) were eliminated from consideration (1966: 61). To cite a more recent example, Laufer and Nation (1995) proposed an improved method to measure lexical richness, i.e. the extent to which ESL writers are using a varied vocabulary. When the new instrument was tested on student compositions, proper nouns were deleted from the samples (Laufer and Nation, 1995: 315). Although this decision might have been licensed by the logic of the research goals, it certainly seems to imply that productive knowledge of proper names does not make ESL learners vocabulary any richer. In vocabulary threshold studies set up to calculate the minimum percentage of familiar words in a text that still allows learners to achieve reasonable comprehension proper names have sometimes been counted as automatically known items (Hirsh and Nation, 1992; Hu and Nation, 2000). There is an implicit assumption that in the course of ESL learners vocabulary development proper names will somehow take care of themselves but is it justified? Can the presence of unfamiliar names affect comprehension detrimentally? Until fairly recently it was believed that 95% of the running words in a text must be known for ESL readers to be able to guess the rest from context (Liu and Nation, 1985) and achieve reasonable comprehension (Laufer, 1989). However, Hu and Nation (2000) demonstrated that even 95% coverage is usually not enough. They replaced various proportions of words in a short story with nonsense words. Each group of advanced ESL learners read one of the four versions of the story where 80%, 90%, 95% or 100% of the running words were familiar. None of the participants understood the story at 80% coverage. Although some readers at 90% and 95% levels were able to gain adequate or close to adequate comprehension, they were still a small minority. At 100% coverage, most learners gained adequate comprehension. Applying a regression model to the data allowed Hu and Nation to calculate the effect of coverages other than those directly measured in the experiment:
The results suggest that for this kind of reading [i.e. unassisted reading of a short fiction text addition mine] learners need to know at least 98% of the running words in the text. That is, the density of unknown words should be around one in fifty (2000: 423).

Thus, the currently accepted view is that 98% of the words in a text ought to be known for an ESL reader to achieve good comprehension (Nation, 2006).

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

86

RELC Journal 43(1)

Some studies assumed that proper names are easily picked up on the spot, and therefore can be equated with known words:
There are strong reasons for considering proper nouns as words that do not require previous learning. First, the text reveals what we need to know about them as the story progresses. [] Second, their form (an initial capital letter) and their function clearly signal they are proper nouns (Hirsh and Nation, 1992: 691).

True, proper names are unlikely to disrupt comprehension in reading, particularly in short novels that were the subject of Hirsh and Nations study. However, this does not guarantee that the same should hold true for listening. Listening and reading modalities, while sharing some cognitive mechanisms, nevertheless have a number of distinct differences. First language research shows that for adults, reading usually results in better comprehension than listening (Kintsch and Kozminsky, 1977; Sanders, 1973; Walker, 1976). In some second language studies, reading was also found to be more efficient (i.e. allowed participants to understand more than listening) (Hirai, 1999; Lund, 1991). Very few studies have examined how vocabulary knowledge and second language listening comprehension are related. Nevertheless, it has been shown that acceptable comprehension requires substantial familiarity with text vocabulary. Kelly (1991) analyzed a large sample of errors made by 38 advanced French learners of English and a French-speaking English teacher. Participants were required to transcribe and translate extracts from the BBC radio news broadcasts. Lexical errors associated with unfamiliar words/collocations or ignorance of a particular meaning in familiar lexical items turned out to be more frequent than perceptual and syntactical errors. Then Kelly divided all errors into two categories: (i) errors that affected comprehension only minimally; and (ii) errors that obscured the meaning considerably. Lexical errors accounted for 65.5% of cases where comprehension was severely impaired. Kelly concluded that insufficient vocabulary knowledge is the main obstacle to listening comprehension with advanced foreign language learners (1991: 147). Bonk (2000) went on to show that at other proficiency levels reasonable comprehension is likely to occur only if over 90% of the words in the text are familiar. Four listening passages with increasing amounts of low-frequency vocabulary were developed and played to 59 Japanese learners of English. Comprehension was measured by first language recall protocols. Each protocol was assigned a holistic rating on the scale from 1 to 4. Next, protocols were divided into two groups, namely the inferior comprehension group (ratings 1-2) and the good comprehension group (ratings 3-4). Familiarity with text vocabulary was measured by a dictation. A significant correlation was found between comprehension (quality of recall) and familiarity with vocabulary (dictation scores): . . .Good comprehension seldom occurred with text-lexis familiarity levels lower than 75 percent, but occurred frequently at 90+ percent levels. This pattern was observed equally for learners of high, middle, or low second-language listening proficiency (Bonk, 2000: 14). It is likely that the relationship between the amount of un/familiar lexis and comprehension is more complicated in listening compared to reading. Sometimes ESL listeners do not seem to be able to achieve adequate comprehension even while reporting familiarity with 100% of words in the text (Bonk, 2000; Olsen and Huckin, 1990).

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva

87

So we know that to be able to understand spoken English an ESL listener must know at least 90% of the words and likely more. According to the Brown Corpus, depending on the text genre, proper names can make up from 2.3% to over 9% of the running words (Francis and Kuera, 1982: 539, 543). ESL listeners usually face considerable linguistic deficits, i.e. their vocabulary knowledge is far lower than the necessary 90% threshold to begin with. If the assumption that in connected speech proper names are automatically understood on the spot is incorrect, then a further 2.3-9% drop of the number of known words could reduce ESL listeners chances of adequate comprehension. This study was designed to answer two research questions: 1. Does the presence of unfamiliar proper names influence ESL learners comprehension of a spoken text? 2. While listening to connected speech, how well are ESL learners able to determine what proper names refer to (how much information about the referents of proper names can ESL listeners get from context)?

Method Participants
For this study 110 intermediate to advanced ESL learners took part in the experiment, 55 in the Names Known treatment group and 55 in the Names Unknown group. Participants came from 10 different classes in four institutions two New Zealand universities and two private language schools. Participants were aged from 17 to 46 years old, with a mean age of 26.4. Most students were in their early twenties with 72.7% below age 30. Prior to the experiment they spent 1-520 weeks in an English speaking environment. The median was 20 weeks (5 months); 76.4% of participants arrived in New Zealand less than 32 weeks (8 months) before the experiment. Exactly half, 55 learners were male and 55 female. Participants came from 20 different first language backgrounds. Ethics agreement for the study was obtained from the Human Ethics Committee, Victoria University of Wellington.

Materials
Experimental Text. A news story was chosen to serve as listening input. One of the challenges in testing listening is there is no strict criterion for measuring text interpretation since interpretations may differ between individuals according to a multiplicity of cognitive and affective factors (Brindley, 1998: 173). Various legitimate understandings of the same input may be possible, none of which is necessarily less correct (Sarig, 1989). Language testing experts recommend that listening tests should primarily target explicit propositional content (literal meanings) as opposed to implied meanings that require extensive inferencing and are thus open to individual variation: the default construct should test only those things for which there is clear and unequivocal linguistic evidence in the text (Buck, 2001: 121). The news story was factual and satisfied these requirements.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

88

RELC Journal 43(1)

The text was sourced from the Wellington Corpus of Spoken New Zealand English and modified. First, the original news report was cut down to 149 running words. It was hoped that with a short passage participants would be able to maintain close attention throughout the listening task. Second, all low-frequency vocabulary except proper names were taken out or replaced by the first 2,000 high-frequency words (West, 1953) where appropriate. Some long sentences were also broken up into parts or substituted with easier grammar constructions. The aim was to make the amount of difficult non-names vocabulary as low as possible. Lastly, all proper names were replaced by artificial proper names that the researcher invented. This was necessary to ensure that all proper names were unfamiliar to participants prior to the experiment. The resulting text contained 10 proper name types (17 tokens). A one minute long sound file was recorded by a male New Zealand English speaker. Comprehension Measures. Most researchers take a multi-measure approach to assessing listening comprehension as it is a complex and invisible process. A large body of literature describes various test item types; however, each type has its advantages and disadvantages (Brindley, 1998: 177-79; Buck, 2001: 134-48; Weir, 1993). Any test of listening comprehension involves language skills other than listening comprehension as such; consequently, combining different response formats helps to obtain more reliable results. Three sets of items were developed for the experiment: (i) 14 true-false-dont know statements; (ii) 11 open-ended questions; and (iii) 15 multiple choice questions. The true-false-dont know items measured proper names related comprehension comprehension of those ideas in the story that were dependent on knowing the referents of the proper names. Open-ended and multiple choice items were a measure of general comprehension; they did not require the knowledge of proper names to be answered correctly. Proper Names Test. The proper names test was designed to measure how well listeners could understand the proper names themselves as opposed to the content of the story. The test consisted of 10 tick-all-options-that-apply questions (corresponding to the 10 proper names in the text). For each name there were six answer options and the participant needed to tick as many as they considered correct. The test included 20 correct options and 40 distractors. Task Difficulty Ratings. The presence of unfamiliar proper names was not anticipated to have a strong effect on comprehension as proper names make up only a small proportion of the text. Difficulty ratings were used to determine if unfamiliar proper names would make the text seem harder for the listeners to understand. Participants were asked to indicate on a 7-point scale (i) how much of the story content they thought they understood; and (ii) how difficult it was for them to complete the experimental tasks.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva

89

Listening Proficiency Measure. In between-groups designs, it is important to ensure that treatment groups are comparable, and that participants performance on the dependent variable is not contaminated by extraneous factors. The International English Language Testing System (IELTS) listening module was used to benchmark the participants listening proficiency (IELTS Specimen Materials April 1995).

Procedures
Ten 1.5-hour long experiment sessions were conducted in a group setting within the participants regular class time in their usual classrooms. The sessions were advertised as voluntary but all students who attended their English class on the day opted to take part. Participants were randomly assigned to one of the treatment groups: Names Known or Names Unknown. After filling in consent forms, the Names Known group stayed with the researcher and received 30 minutes of instruction pre-teaching of the 10 proper names from the listening passage. Meanwhile, the Names Unknown group was taken by an assistant to a different classroom to have afternoon tea. Pre-teaching was thorough. Its aim was to make sure the students in the Names Known group gained both receptive and productive knowledge of the form and meaning of each proper name. The Names Known group learnt the referent of each proper name, worked with word cards and pictures of the referents, and did several practice exercises both individually and in pairs. After pre-teaching the treatment groups came together (the assistant brought the Names Unknown group back) and everyone listened to the experimental text two times. Next participants filled in the true-false-dont know answer sheet. The researcher pronounced sentences clearly and slowly, and repeated each item twice. Open-ended questions and multiple choice questions were answered in writing. The proper names test was administered as a whole class activity. The researcher pronounced each proper name twice and gave participants half a minute to a minute to make their choice and fill in the answer sheet. In the last half an hour participants completed the task difficulty ratings and the IELTS listening module.

Results
True-False-Dont Know Statements (TFDN). Mean scores on the measure of proper names related comprehension are presented in Table 1. The Names Known group performed significantly better than the Names Unknown group (equal variances independent samples t-test, 2-tailed, p<0.01). This means that the participants in the Names Unknown treatment could not understand idea units related to the proper names as well as the participants in the Names Known group. The strength of association test showed that 36% of the variability in participants scores in this sample was accounted for by group, i.e. familiarity or lack of familiarity with proper names (2 =0.3616). Open-Ended Questions (OE). To analyse the data from open-ended questions, all participants responses to the first question were typed up. Then the researcher compared these responses and gave each a rating from 0 to 3:

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

90

RELC Journal 43(1)

Table 1. Mean Scores on True-False-Dont Know Statements (out of 14) by Treatment Group Group Names Known Names Unknown N 55 55 Median 9 6 Minimum 4 2 Maximum 13 11 Mean 8.95 5.91 Std. Deviation 1.995 2.075

Table 2. Mean Scores on Open-Ended Questions (out of 33) by Treatment Group Group Names Known Names Unknown N 55 55 Median 13 8 Minimum 1 0 Maximum 30 26 Mean 13.84 8.64 Std. Deviation 7.241 5.310

incorrect answer/no answer = 0 correct imprecise answer = 1 more precise partial answer = 2 precise full answer = 3

The same procedure was repeated with the rest of the questions. Two weeks later the researcher rerated all responses without looking at her previous work and compared the two ratings. There were few discrepancies and these were resolved by consulting with a colleague. After that, the researcher entered the ratings for all responses for each participant and tallied them up to obtain the participants overall scores. Mean scores on the first measure of general comprehension (open-ended questions, OE) are presented in Table 2. Participants performed considerably better in the Names Known condition compared to the Names Unknown condition. A significant difference between the treatment groups was found (independent samples unequal variances t-test, 2-tailed, p<0.01). This means that the listeners in the Names Known group were able to form a more precise understanding of the passage content than the Names Unknown group. The strength of association test showed that 16% of the variance in participants scores in this sample was accounted for by group, i.e. familiarity or lack of familiarity with proper names (2 = 0.1569). Qualitative analysis also demonstrated that a considerable proportion of incorrect responses conformed to a pattern that could be attributed to the presence of unfamiliar names. Many participants in the Names Unknown condition appeared to have difficulty with establishing a framework for the story who is who and what roles the characters played in relation to each other. Multiple Choice Questions (MC). Mean scores on the second measure of general comprehension (multiple choice questions, MC) are presented in Table 3. Although the Names Known group performed slightly better, no statistically significant difference between the treatment groups was found (equal variances independent samples t-test, p=0.3). This means that both groups successfully understood the gist of the text regardless of whether proper names were familiar or unfamiliar.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva
Table 3. Mean Scores on Multiple Choice Questions (out of 15) by Treatment Group Group Names Known Names Unknown N 55 55 Median 10 10 Minimum 4 3 Maximum 15 14 Mean 10.05 9.56

91

Std. Deviation 2.606 2.379

Table 4. Mean IELTS Listening Module Scores (out of 40) by Treatment Group Group Names Known Names Unknown N 55 55 Median 23 21 Minimum 14 4 Maximum 37 34 Mean 23.22 20.65 Std. Deviation 5.792 7.176

Table 5. Mean Overall Comprehension Scores (out of 62) by Treatment Group Group Names Known Names Unknown N 55 55 Median 32 24 Minimum 9 7 Maximum 57 45 Mean 32.8364 24.1091 Std. Deviation 10.07521 7.71884

Listening Proficiency and ANCOVA (TFDN+OE+MC).Comparison of the mean IELTS module scores between the treatment groups revealed that the listening proficiency of the Names Known group was slightly higher than that of the Names Unknown group (see Table 4). The difference appeared to be marginally significant (equal variances independent samples t-test, 2-tailed, p =0.042), so the treatment groups could not be considered entirely equal in terms of listening proficiency. To determine whether the presence of unfamiliar proper names has an effect on comprehension over and above that of listening proficiency, an overall comprehension score was calculated for each participant by summing up their scores on all comprehension measures (true-false-dont know statements, open-ended questions and multiple choice questions, see Table 5). Then ANCOVA was used to isolate the effect of unfamiliar proper names on comprehension from proficiency effects. Table 6 shows ANCOVA results where the dependent variable is the participants overall comprehension scores (TFDN+OE+MC), the independent variable is the treatment group (Names Known versus Names Unknown) and the covariate is listening proficiency (IELTS listening module scores). After the listening proficiency variable is controlled, the effect of the treatment group on overall comprehension scores is still highly significant (p<0.01). Around 17% of remaining variance in the scores can be attributed to the treatment group, i.e. to whether the proper names in the text were familiar or unfamiliar. Proper Names Test. Two scores were recorded for each participant: PNgood (the number of correct options ticked) and PNbad (the number of incorrect options ticked). The mean PNgood and PNbad scores are presented in Table 7 and Table 8. The PNgood scores appeared normally distributed, so the t-test was used to compare the performance of the

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

92
Table 6. ANCOVA Results Source Corrected Model Intercept IELTS Group Error Total Corrected Total
aR

RELC Journal 43(1)

Type III Sum of Squares 4777.656a 1202.872 2683.110 1211.814 6015.762 99970.000 10793.418

df 2 1 1 1 107 110 109

Mean Square 2388.828 1202.872 2683.110 1211.814 56.222

F 42.489 21.395 47.723 21.554

Sig. .000 .000 .000 .000

Partial Eta Squared .443 .167 .308 .168

Squared = .443 (Adjusted R Squared = .432)

two treatment groups. The Names Known group performed much better and ticked significantly more correct options than the Names Unknown group (equal variances independent samples t-test, 2-tailed, p<0.01). The distribution of PNbad scores appeared to deviate from normal. The Mann Whitney U test was used to compare the performance of the Names Known and the Names Unknown group. The Names Known group performed much better and ticked significantly fewer incorrect options than the Name Unknown group (Mann Whitney U, p<0.01). Data from the proper names test confirms that ESL listeners are not always able to guess the referents of unfamiliar proper names from context. The maximum number of correct options was 20; in the Names Unknown condition the mean number of correct options ticked was 8.96 while in the Names Known condition the mean number of correct options ticked was 13.96. This means that participants in the Names Unknown group were able to derive 44.8% of information about the referents of unfamiliar proper names from context; however, they were not able to identify all the referents successfully. Participants in the Names Known group managed to make use of 69.2% of information about the referents of the proper names that were pretaught to them prior to listening. Task Difficulty Ratings. Participants indicated how well they believed they understood the listening passage. Perceptions of comprehension success ranged from 1 (I understood nothing) to 7 (I understood everything). The higher ratings correspond to the higher fraction of the experimental text that participants subjectively judged as understood. Frequencies of particular responses, from 1 to 7, were tabulated across treatment groups (see Table 9). For the Names Known group, there were more responses towards the higher end of the scale (i.e. participants reported understanding more of the story) while for the Names Unknown group, the majority of responses lay at the lower end of the scale (i.e. participants reported understanding less of the story). The chi-square test showed that there was a highly significant association between the frequency of particular responses and treatment group (Fishers exact test, 2-sided, p=0.001). This means that when unfamiliar proper names are present, listeners report achieving a lower amount of comprehension.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva
Table 7. Mean PNgood Scores (out of 20) by Treatment Group Group Names Known Names Unknown N 55 55 Median 15 9 Minimum 6 3 Maximum 20 16 Mean 13.84 8.96

93

Std. Deviation 3.799 3.350

Table 8. Mean PNbad Scores (out of 40) by Treatment Group Group Names Known Names Unknown N 55 55 Median 2 7 Minimum 0 1 Maximum 12 22 Mean 2.91 8.05 Std. Deviation 2.222 4.474

Table 9. Cross Tabulation: Group * Perceived Comprehension Success Group Perceived comprehension success 1 Names Known Count % within group Names Unknown Count % within group 0 .0% 2 3.6% 2 7 12.7% 15 27.3% 3 8 14.5% 13 23.6% 4 20 36.4% 20 36.4% 5 12 21.8% 4 7.3% 6 8 14.5% 0 .0% 7 0 .0% 1 1.8%

Table 10. Mean Difficulty Ratings of Experimental Tasks by Treatment Group Task True-false-dont know Open-ended questions Multiple choice questions Proper names test Mean rating, Names Known (N=55) 3.73 3.53 4.75 4.91 Mean rating, Names Unknown (N=55) 3.18 2.82 3.76 3.00

Participants also indicated how difficult they found the experiment tasks (1 = very difficult, 7 = very easy). The higher the rating a participant assigned to a task, the easier they perceived that task to be and vice versa. Mean ratings by group are presented in Table 10. All tasks were rated as easier to do by the Names Known group and as harder to do by the Names Unknown group. Although difficulty ratings likely reflect the students perceptions of their own English proficiency, their past personal experiences with various task formats etc., these findings clearly indicate that the presence of unfamiliar proper names made the experimental text and associated comprehension tasks seem more difficult to the participants in the Names Unknown group. This is indirect evidence that the presence of unfamiliar proper names indeed makes spoken input more challenging for ESL learners to handle.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

94

RELC Journal 43(1)

Summary and Discussion

This study found that for low intermediate to advanced ESL listeners the presence of unfamiliar proper names hinders comprehension of a short spoken text. The Names Known group performed significantly better than the Names Unknown group on two out of three comprehension measures, true-false-dont know statements (proper names related comprehension) and open-ended questions (general comprehension). No statistically significant difference was observed on multiple choice questions. This suggests that at least at the intermediate to advanced levels of proficiency, the effect of unfamiliar proper names on second language listening comprehension is local, i.e. concerns only the ability to understand details, especially idea units directly associated with processing proper names themselves. The ability to comprehend main ideas appears unaffected; both treatment groups successfully grasped the gist of the story. Although the treatment groups were not quite equal in terms of listening proficiency at the time of testing, the effect of unfamiliar proper names on comprehension was highly significant even after the proficiency variable was controlled. The findings also demonstrate that after two attempts at listening ESL learners are not always able to determine what unfamiliar proper names refer to. Listeners in the Names Unknown treatment derived 44.8% of information about the referents of unfamiliar proper names from context. The conclusion that unfamiliar proper names hinder listening comprehension was also supported by the participants difficulty ratings of experimental tasks. When proper names were familiar, participants rated all comprehension tasks as easier to do, and their perceived amount of comprehension ratings in particular were higher. On the other hand, when proper names were unfamiliar, participants rated all comprehension tasks as harder to do and self-reported a lower amount of comprehension. It has been firmly established in the literature that vocabulary knowledge is a crucial factor affecting both reading and listening comprehension (Bonk, 2000; Burns, Dean and Foley, 2004; Kelly, 1991; Stahl and Fairbanks, 1986). This study confirms that, similarly to common vocabulary, unfamiliar proper names influence the ESL learners ability to understand connected speech. One objection to the method used in this study can be that in real life listening, it would be rare for all proper names in a text to be unfamiliar; therefore, their overall impact on comprehension could be smaller. However, it is also possible that the effect might not have fully manifested itself in our experiment. First, pretaught proper names cannot be equated to known. Although the pre-teaching was thorough, this one-off intervention could not have substituted for a gradual learning process whereby a new expression becomes solidly engrained in the learners mental lexicon. In a meta-analysis of 52 studies, Stahl and Fairbanks (1986) found a strong correlation between the time allocated for new vocabulary instruction and the subsequent effect of this pre-taught vocabulary on reading comprehension. They caution that words need to be learned thoroughly in order to contribute to passage comprehension (Stahl and Fairbanks, 1986: 101). In this study, most participants in the Names Known treatment did very well during pre-teaching. However, they still made mistakes in the proper names test, which indicates that their recently acquired knowledge of the new names was still unstable. Second, what could also have masked the effect of unfamiliar names is the fact that in our experiment comprehension was tested under rather favourable conditions:

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva

95

(i) participants had two attempts at listening; (ii) the text was very short; (iii) the text was narrative rather than expository; and (iv) apart from proper names, the text did not contain any low-frequency vocabulary. Considering the participants fairly high English proficiency, there are grounds to speculate that if the effect of unfamiliar proper names is present under such supportive conditions, it may be stronger in real life listening which is more demanding. Another limitation is that in this study only a single news text was used. An important avenue for future research is to assess the impact of proper names on comprehension with various kinds of listening input. This can be done by manipulating the text length, genre and structure as well as the percentage of unfamiliar proper names and low-frequency common vocabulary. The burden of unfamiliar proper names could decrease in a longer text in which the core set of names would be encountered many times. Proper names cover only a small percentage of a given text so their impact on second language listening comprehension is necessarily smaller than that of common expressions. Thus it should come as no surprise that the effect emerged only on some measures and not on others. Overall, the results of the present experiment parallel some earlier findings regarding common vocabulary. For example, Wixson (1986) investigated the effect of preteaching common words of central and non-central importance within a text on first language reading comprehension. She found that preteaching unfamiliar vocabulary can have a direct effect on students understanding of specific ideas within a text (Wixson, 1986: 327). When the central words were pretaught, the children in her experiment understood more ideas related to the central words. When the non-central words were pretaught, the children understood more ideas related to the non-central words. The effect reported in this study is similarly local, i.e. concentrates around the idea units associated with proper names. Wixson (1986) and Stahl and Fairbanks (1986) demonstrated that word-specific measures designed to evaluate understanding of parts of the text directly related to the instructed vocabulary are more sensitive and produce larger effects than general comprehension measures that do not involve the pretaught words. Likewise, the proper names related comprehension measure in this study produced a highly significant difference between the treatment groups. As for the general comprehension measures, they can be ranked from harder (open-ended questions) to easier (multiple choice questions). A significant difference was only observed on the more conservative measure (openended questions). Kameenui, Carnine and Freschi found that preteaching unfamiliar common vocabulary enhanced comprehension but only as measured by inferential questions; in their study, the effect did not show on literal questions (1982: 379). In this study, due to the highly factual nature of the text, the difference between inferential and literal questions was negligible. However, assuming that inferential questions are more difficult than the literal ones, it can be argued that our results conform to a pattern similar to Kameenui, Carnine and Freschis (1982): the effect of unfamiliar proper names on comprehension emerged only on the more conservative measure.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

96

RELC Journal 43(1)

As task difficulty ratings demonstrated, unknown proper names make the listening input seem more confusing to second language listeners, which has clear implications for ESL teaching and testing. In an ESL classroom, if the aim is listening for gist, proper names should not be a concern. However, if the aim is detailed, close to 100% understanding, then proper names will be a burden; they should either be known or at least pretaught. Teachers, textbook developers and particularly test developers need to bear in mind that proper names increase the difficulty level of listening materials, especially if the percentage of names in the text is above 4-5%. ESL listeners cannot be expected to guess the referents of unfamiliar proper names from context. This study shows that in the best of circumstances their guesses are successful only in 44.8% of cases.

Conclusion
This study found that proper names raise the vocabulary threshold in second language listening. They place high demands on the ESL learners cognitive resources and cannot be considered automatically known items without a learning burden. Depending on the learners proficiency and the characteristics of the text, proper names can significantly disrupt comprehension. According to Buck texts with fewer things or people to be distinguished tend to be easier than texts with more things to be distinguished (2001: 150). Unfamiliar proper names do precisely this - add extra things to be distinguished. Additional unknown elements may simply make the ESL listener panic; participants in this study invariably judged all comprehension tasks as harder to do when unfamiliar names were present. It is recommended that teachers be flexible when setting up listening tasks and use different techniques depending on what depth of understanding the learners are supposed to achieve. If the instructional goal is listening for the main points, most likely proper names will not affect the outcomes of the learning activity. It would be desirable to provide plenty of practice on texts that contain unfamiliar names so that learners develop strategies for dealing with input that is only partially comprehensible. However, if detailed and precise understanding is required, then it is unrealistic to expect ESL listeners to cope with unfamiliar proper names on the spot without assistance or preparation. The presence of unfamiliar proper names evidently inhibits comprehension for intermediate to advanced ESL learners. Therefore, it is best to minimise exposure to unfamiliar proper names in a testing situation because they may cause unwarranted anxiety and reflect negatively on the learners performance. Especially in high stakes placement tests the impact of unfamiliar names must be estimated thoroughly. Unless the context and the form of the names absolutely guarantee easy referent identification, it is recommended to avoid proper names where possible. References
Ames WS (1966) The development of a classification scheme of contextual aids. Reading Research Quarterly 2(1): 57-82. Anderson-Hsieh J, Koehler K (1988) The effects of foreign accent and speaking rate on native speaker comprehension. Language Learning 38(4): 561-613.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

Kobeleva

97

Arcario P (1990) Criteria for selecting video materials. In: Stempleski S, Arcario P (eds) Video in Second Language Teaching: Using, Selecting and Producing Video for the Classroom. Teachers of English to Speakers of Other Languages Inc, 109-21. Berezowski L (2001) Articles and Proper Names. Wrocaw: Wydawnictwo Uniwersytetu Wrocawskiego. Blau EK (1982) The effect of syntax on readability for ESL students in Puerto Rico. TESOL Quarterly 16(4): 517-28. Bonk WJ (2000) Second language lexical knowledge and listening comprehension. International Journal of Listening 14: 14-31. Brindley G (1998) Assessing listening abilities. Annual Review of Applied Linguistics 18: 171-91. Buck G (2001) Assessing Listening. Cambridge: Cambridge University Press. Burns MK, Dean VJ, Foley S (2004) Preteaching unknown key words with incremental rehearsal to improve reading fluency and comprehension with children identified as reading disabled. Journal of School Psychology 42(4): 303-14. Chiang CS, Dunkel P (1992) The effect of speech modification, prior knowledge, and listening proficiency on EFL lecture learning. TESOL Quarterly 26(2): 34574. Elkhafaifi H (2005) Listening comprehension and anxiety in the Arabic language classroom. The Modern Language Journal 89(2): 206-20. Francis WN and Kuera H (1982) Frequency Analysis of English Usage: Lexicon and Grammar. Boston, MA: Houghton Mifflin Company. Freedle R and Kostin I (1999) Does text matter in a multiple-choice test of comprehension? The case for the construct validity of TOEFLs minitalks. Language Testing 16(1): 2-32. Griffiths R (1992) Speech rate and listening comprehension: further evidence of the relationship. TESOL Quarterly 26(2): 385-91. Hirai A (1999) The relationship between listening and reading rates of Japanese EFL learners. The Modern Language Journal 83(3): 367-84. Hirsh D, Nation ISP (1992) What vocabulary size is needed to read unsimplified texts for pleasure? Reading in a Foreign Language 8(2): 689-96. Hu MH and Nation ISP (2000) Unknown vocabulary density and reading comprehension. Reading in a Foreign Language 13(1): 403-30. Hung HCM (2007) Split attention in reading comprehension: a case of English as a second/foreign language. Masters thesis. The University of New South Wales, Sydney. IELTS Specimen Materials April (1995) (updated November 1997). UCLES [University of Cambridge Local Examinations Syndicate], the British Council, IDP Education Australia. Jung EH (2003) The role of discourse signalling cues in second language listening comprehension. The Modern Language Journal 87(4): 562-77. Kameenui EJ, Carnine DW, Freschi R (1982) Effects of text construction and instructional procedures for teaching word meanings on comprehension and recall. Reading Research Quarterly 27(3): 367-88. Kelly P (1991) Lexical ignorance: the main obstacle to listening comprehension with advanced foreign language learners. IRAL 29(2): 135-49. Kintsch W, Kozminsky E (1977) Summarizing stories after reading and listening. Journal of Educational Psychology 69(5): 491-99. Laufer B (1989) What percentage of text-lexis is essential for comprehension? In: Laurn C, Nordman M (eds) Special Language: from Humans Thinking to Thinking Machines. Clevedon: Multilingual Matters, 316-23. Laufer B, Nation ISP (1995) Vocabulary size and use: lexical richness in L2 written production. Applied Linguistics 16(3): 30722.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013

98

RELC Journal 43(1)

Laufer B, Sim D (1985a) Taking the easy way out: non-use and misuse of clues in EFL reading. English Teaching Forum 23(2): 7-10, 20. Laufer B, Sim D (1985b) Measuring and explaining the reading threshold needed for English for Academic Purposes texts. Foreign Language Annals 18(5): 405-11. Liu N, Nation ISP (1985) Factors affecting guessing vocabulary in context. RELC Journal 16(1): 33-42. Long DR (1990) What you dont know cant help you: an exploratory study of background knowledge and second language listening comprehension. Studies in Second Language Acquisition 12: 65-80. Lund RJ (1991) A comparison of second language listening and reading comprehension. The Modern Language Journal 75(2): 196-204. Markham P, Latham M (1987) The influence of religion-specific background knowledge on the listening comprehension of adult second-language students. Language Learning 37(2): 157-70. Matsuura H (2007) Intelligibility and individual learner differences in the EIL context. System 35(3): 293-304. Napoli E (1997) Names, indexicals, and identity statements. In: Knne W, Newen A, and Anduschus M (eds) Direct Reference, Indexicality, and Propositional Attitudes. Stanford, CA: Center for the Study of Language and Information, 185-211. Nation ISP (2006) How large a vocabulary is needed for reading and listening? The Canadian Modern Language Review 63(1): 59-82. Nuttall CE, Alderson CJ (1996) Teaching Reading Skills in a Foreign Language. Oxford: Heinemann English Language Teaching. Olsen LA, Huckin TN (1990) Point-driven understanding in engineering lecture comprehension. English for Specific Purposes 9(1): 33-47. Sanders JR (1973) Retention effects of adjunct questions in written and aural discourse. Journal of Educational Psychology 65(2): 181-86. Sarig G (1989) Testing meaning construction: can we do it fairly? Language Testing 6(1): 77-94. Schmidt-Rinehart BC (1994) The effects of topic familiarity on second language listening comprehension. The Modern Language Journal 78(2): 179-89. Sherman J (1997) The effect of question preview in listening comprehension tests. Language Testing 14(2): 185-213. Shohamy E, Inbar O (1991) Validation of listening comprehension tests: the effect of text and question type. Language Testing 8(1): 23-40. Stahl SA, Fairbanks MM (1986) The effects of vocabulary instruction: a model-based meta-analysis. Review of Educational Research 56(1): 72-110. Walker L (1976) Comprehending writing and spontaneous speech. Reading Research Quarterly 11(2): 144-67. Weir CJ (1993) Understanding and Developing Language Tests. London: Prentice Hall International. West M (1953) A General Service List of English Words. London: Longman, Green & Co. Wixson KK (1986) Vocabulary instruction and childrens comprehension of basal stories. Reading Research Quarterly 21(3): 317-29. Wu Y (1998) What do tests of listening comprehension test? A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing 15(1): 21-44.

Downloaded from rel.sagepub.com at Universitas Pendidikan Indone on September 3, 2013