Language Learning & Technology http://llt.msu.edu/vol8num3/jones/
September 2004, Volume 8, Number 3
TESTING L2 VOCABULARY RECOGNITION AND RECALL USING PICTORIAL AND WRITTEN TEST ITEMS
University of Arkansas
This article describes two studies that examined the effects of pictorial and written annotations on second language (L2) vocabulary learning from a multimedia environment. In both studies, students were randomly assigned to one of four aural multimedia groups: a control group that received no annotations, and three treatment groups that provided written, pictorial, or both written and pictorial annotations while listening. In the first study, students in the three treatment groups recognized English translations or pictorial representations of French keywords better than the control group that received no annotations during listening. In the second study, students produced English translations of French keywords best when the mode of testing matched the treatment mode. These results add to the growing body of literature on the beneficial effects of annotations on L2 vocabulary recognition and recall.
In recent years, media-based listening comprehension activities have evolved from a purely audio-only approach to one that is more holistic and multi-sensory. No longer are materials focused on nonsensical sentence structures. Instead, students now experience lengthier, authentic audio passages embedded in video, interactive CD-ROMs, or Web sites. Numerous computer-based and online packages have been developed by researchers, faculty, and publishing companies (Amon, Muyskens, & Omaggio Hadley, 2000; Chun & Plass, 1997; Larson & Bush, 1992; Otto & Pusack, 1992; Sabo, Restrepo, & Jones, 2000; University of Texas, 2001, 2004) to assist students' L2 aural and written development. Français Interactif (University of Texas, 2001, 2004) is one of the more innovative on-line French language learning programs produced in recent years. It provides multiple levels of semester-long instruction with a mixture of multi-sensory materials such as aural, pictorial, video, and written presentations that help students to learn the target language. Interactive self-checking exercises provide them with an opportunity to examine their output in terms of recall of the target language material. Cyberbuch (Chun & Plass, 1997), another innovative program on CD-ROM, provides advance organizer videos and annotated information to support students' reading of a German text. This highly focused program promotes interaction with multi-sensory annotated information in the form of text translations, images, short video clips, and audio to facilitate students' understanding of keywords and the literary texts.
As the development of L2 multimedia packages such as Français Interactif and Cyberbuchincreases, researchers strive to better understand how the attributes of multimedia can enhance listening and reading comprehension and vocabulary acquisition (Brett, 1995, 1997; Hoven, 1999; Jones & Plass, 2002; Lynch 1998; Salaberry, 2001). For example, Chun and Plass (1996) examined the influence of written and pictorial annotations on students' vocabulary learning from a written text whereas Jones and Plass (2002) examined their influence on vocabulary learning from an aural text. Other researchers, for example Doughty (1991) and Pica, Doughty, and Young (1986) studied the effects of student control over the L2 multimedia environment, while researchers such as Herron (1994) and Herron, York, Cole, and Linden (1998) have closely examined the influence of advance organizers on L2 learning. Despite these advances, many questions remain concerning the effects of multimedia components on students' L2 development.
This article expands upon previous research (Jones, 2003; Jones & Plass, 2002) by describing two studies that used three separate vocabulary tests (written recognition, pictorial recognition, and written recall) to examine how pictorial and written test items might demonstrate how well written and pictorial annotations assist in the learning of new vocabulary when students listen to an aural L2 passage in a multimedia environment. The present article examines this topic. It begins with a review of the role of written and pictorial annotations in L2 reading and listening comprehension, of the process of incidental vocabulary learning, and of the different methods and tests used to examine students' vocabulary recognition and recall. The article concludes with a discussion of the findings, the implications and limitations of this study, and suggestions for future research.
VOCABULARY LEARNING FROM ANNOTATIONS
Information is cognitively processed through visual or verbal channels (Mayer, 1997, 2001, 2002; Paivio, 1986). A dual processing strategy assumes that individuals develop mental pictorial representations of graphic input and mental verbal representations of linguistic input. The presence of both pictorial and verbal cues can facilitate learning, in particular when the corresponding visual and verbal representations are contiguously present in working memory (Mayer, 1997, 2001, 2002; Wittrock, 1989). Mayer's generative theory of multimedia learning (1997, 2001) states that in order to meaningfully comprehend a text in a multimedia format, learners must select relevant pictorial and/or linguistic information from it, organize the input into coherent visual and verbal mental representations, and then integrate the latter by constructing referential connections between the two.
Researchers have long been interested in examining the effects of pictorial and verbal cues on L2 vocabulary learning, and have found that processing supportive information such as pictures or translations enhances language learning. With regard to high-imagery concrete vocabulary learning, Kellogg and Howe (1971) found that foreign words associated with images or actual objects are learned more easily than those without such additional information. Terrell (1986) found that combining an unknown L2 word with a visual representation bypasses a direct translation and facilitates vocabulary learning. Underwood (1989) suggested that we "remember images better than words, hence we remember words better if they are strongly associated with images" (p. 19). Other research suggests that foreign words associated with aural or written translations and images are learned more easily than are those accompanied by pictures or text alone (Baltova, 1999; Guillory, 1998; Jones & Plass, 2002; Oxford & Crookall, 1990; Plass, Chun, Mayer, & Leutner, 1998). For example, Oxford and Crookall (1990) suggest that the combination of pictures and text accesses more parts of the brain, thereby leading to greater depth of processing than when text is processed alone. Baltova (1999) examined the effects of viewing a French video with either French audio and French subtitles (bimodal format), or English audio and French subtitles (reversed format) on students' vocabulary learning. She found that students learned significantly more vocabulary when they viewed the audio-visual material with both French subtitles and French audio present than in the reversed format where they viewed the video with English audio and French subtitles.
Incidental vocabulary learning is the process of acquiring vocabulary while reading or listening for comprehension rather than focusing solely on memorizing lists of words (Hulstijn, 1989; Hulstijn, Hollander, & Greidanus, 1996; Yoshii & Flaitz, 2002). Any incidental vocabulary learning that occurs in a multimedia environment may depend upon the type of annotations processed, and the depth of experience with them. For example, Hulstijn, Hollander, and Greidanus (1996) examined how the presence of glosses for targeted words, or dictionary lookup of words in a written text, might affect incidental vocabulary learning. They found that incidental learning of words frequently occurring in the text was more likely when learners were provided with access to word meanings through marginal glosses or dictionaries than when no helpful information was made available to them. Hulstijn (1992) determined that deep elaboration of the meaning of an unknown word also led to incidental vocabulary learning. Jones (2003) found that students believed that pictures demanded deeper processing than did verbal translations because they had to "figure out" the meaning which they did not have to do if they saw the translation immediately. Quantitative results confirmed her beliefs: Students who accessed pictorial annotations demonstrated greater incidental vocabulary learning than those who did not access this annotation type. Other researchers found that if the context of a written or spoken passage was not clear from the onset, deeper processing might fail to support incidental vocabulary learning, and students who do not have access to annotated information would run the risk of learning words incorrectly (Chun & Plass, 1996; Hulstijn, 1992; Jones, 2003; Jones & Plass, 2002).
Recognition and recall tests are often used to examine students' vocabulary knowledge. However, test and measurement studies indicate that these two forms of testing are quite different and demand separate processing strategies (Cariana & Lee, 2001; Jonassen & Tessmer, 1996). For example, recognition tests usually involve multiple choice activities whereby learners select or guess the correct response from the alternatives given. Such tests may strengthen any existing memory traces (McDaniel & Mason, 1985). Recall, on the other hand, demands the production of responses from memory. It is more difficult than recognition because learners must search for the correct response within their mental representation of the newly experienced information (Cariana & Lee, 2001; Glover, 1989; McDaniel & Mason, 1985).
Several studies have investigated the use of pictorial and written annotations in L2 multimedia reading and listening comprehension using different testing formats (Chun & Plass, 1996; Jones & Plass, 2002; Plass et al., 1998). Plass et al. found that when students accessed both pictorial and written annotations as they read a multimedia-based German text, they scored higher on a written vocabulary production test than when only one annotation type was accessed. The combination of both annotation types allowed for more than one retrieval route to the information in long term memory. These researchers also found that written annotations had a stronger impact on vocabulary production than did pictorial annotations. Jones and Plass (2002) reported that those students who accessed both pictorial and written annotations as they listened to a multimedia-based aural French text performed better on a written vocabulary recognition test than those who accessed single annotations, or no annotations at all. However, unlike subjects in the Plass et al. study, those who accessed pictorial annotations alone or combined with written annotations outperformed those who did not access pictorial annotations on a written vocabulary recognition test. Chun and Plass (1996) further examined the effects of multimedia annotations on L2 vocabulary learning from a reading passage using a written production and a recognition test with a balance of pictorial and written test items that paralleled the modality in which the information was presented. They, too, found that students performed best on both types of tests when both pictorial and written annotations were viewed than when single or no annotations were accessed during reading. They also observed that when the method of testing more closely paralleled the way in which information was presented, student performance improved considerably, resulting in 77% of correct responses on immediate and delayed vocabulary tests, a percentage much higher than the 23%-55% typically expected in select-definition tests (Knight, 1994).
All of the above studies suggest incidental vocabulary learning can be increased if learners are given opportunities to look up word meanings, visually or verbally, while listening or reading. However, none of these studies specifically examined students' incidental vocabulary learning from a listening comprehension activity using vocabulary tests that complemented or ignored the annotation type accessed.
The following two studies, therefore, investigated how pictorial and/or written annotations affect students' performance on incidental vocabulary learning tests that required them to either recognize or recall vocabulary incidentally learned from an aural text, using pictorial or written test items. Three hypotheses are thus proposed that coincide with the three dependent measures used in this study and introduced in the next section:
1. Students with access to pictorial and written annotations during a L2 listening comprehension activity will recognize more written translations of keywords on a written vocabulary recognition posttest than those with access to one type of annotation, or no annotations at all. In addition, students who access written annotations will outperform those without such access on a written vocabulary recognition posttest.
2. Students with access to pictorial and written annotations during a L2 listening comprehension activity will recognize more pictorial representations of keywords on a pictorial vocabulary recognition posttest than those with access to one type of annotation, or no annotations. In addition, students who access pictorial annotations will outperform those without such access on a pictorial vocabulary recognition posttest.
3. Students with access to pictorial and written annotations during a L2 listening comprehension activity will recall more keyword translations on a written vocabulary production posttest than students with access to only one type of annotation or no annotations. In addition, students who access written annotations will outperform those without access to such annotations on a written vocabulary production posttest.
THE PRESENT STUDIES
Participants, Study 1
Eighty second-semester English-speaking beginning students of French, enrolled at the University of Arkansas in the fall of 2001, voluntarily participated in the study during their regular class time. The students completed a 25-item vocabulary recognition pre-test to determine their prior knowledge of the vocabulary in this study. All students demonstrated low prior knowledge of the vocabulary with an average score of 4 out of a maximum score of 25 (M = 4.04, SD = 3.60; Table 1). A Tukey HSD (honestly significant difference) multiple comparison test showed no significant differences among the four groups.
Table 1. Vocabulary Pretest Results Based on Random Assignments to Four Treatments, Study 1
Sixty seven second-semester English-speaking beginning students of French, enrolled at the University of Arkansas in the spring of 2002, voluntarily participated in the study during their regular class time. They completed a 25-item pre-treatment vocabulary recognition test based on the words used in this study and demonstrated low prior knowledge of the vocabulary with an average score of 1.5 out of 25 (M = 1.57, SD = 1.23; Table 2). A Tukey HSD multiple comparison test showed no significant differences among the four groups.
Table 2. Vocabulary pretest results based on random assignments to four treatments, Study 2
Pictorial and Written Annotations
Materials and Apparatus for Studies 1 and 2
Four aural multimedia treatments, developed using Adobe Premiere 4.2 (Adobe, 1994) and Authorware 4.0 (Macromedia, 1997), were presented to students using a 24-station Macintosh computer lab, arranged so that the students could view only their own computer screens.
All groups first saw an opening screen that instructed them how to use the program and provided an advance organizer in the form of a brief written paragraph that placed the aural passage about an important event in its historical context (Figure 1). This screen provided additional instructions to assist students with the annotations available in their respective treatments.
Figure 1. Example of the opening screen which provides instructions and advance organizer information, based on the treatment, prior to listening to the aural passage
The opening screen was followed by five separate listening comprehension screens tailored to each treatment. Within each screen, students could click on audio buttons to listen to a 2 minute and 20 second aural passage (Buzhardt & Hawthorne, 1993; see Appendix A). Twenty-seven French keywords, including nouns, verbs, adjectives, and adverbial phrases, were chosen by two experienced French professors for their importance to understanding the story. They were placed in order of appearance on the left side of each screen. To hear their pronunciation, listeners dragged the keywords to a speaker icon in the upper right section of the screen.
In the control group, students could only listen to the pronunciation of French keywords (Figure 2).
Figure 2. Example of control group treatment (no annotations available)
In the pictorial annotations group, students could drag the keywords to a camera icon to view their pictorial representations (Figure 3).
Figure 3. Example of treatment for the pictorial annotations group (only pictorial annotations available)
In the written annotations group, students could drag the keywords to a book icon to view their English translations (Figure 4).
Figure 4. Example of treatment for the written annotations group (only written annotations available)
In the pictorial and written annotations group, students could drag the keywords to the camera and/or book icon to view the picture and/or an English translation (Figure 5).
Figure 5. Example of treatment for the written and pictorial annotations group
Students could select any annotation available in their treatment at any time before, during, or after each aural segment. A login script tracked the number of annotations accessed and the amount of time spent on each.
The written English translations of the French keywords were presented in a bold, 14-point Helvetica font. The color drawings and photos were pictorial representations of the same French keywords (Appendix B). The pictures were selected based on a pilot study conducted in the summer of 1999 and were used in two subsequent studies (Jones, 2003; Jones & Plass, 2002). While all written annotations provided exact English translations of the French keywords, pictorial annotations may not have precisely represented the meaning of some low-imagery French keywords such as étouffé (smother).
Dependent Measures and Scoring
Not all posttests were conducted in both studies since they were originally meant to study unrelated issues. However, the results of these two studies warranted a combined analysis, albeit not a full one.
In Study 1, two dependent variables examined the effects of the four treatments on students' L2 incidental vocabulary learning. The multiple choice written vocabulary recognition and pictorial recognition tests were administered immediately after the treatment and again three weeks later (Appendix C). They consisted of 25 of the 27 French keywords described above. The maximum score for each test was 25 with each correct response worth one point. The written vocabulary recognition pre- and posttests were identical. In these multiple choice tests, students had six English translations to select from for each test item. The pictorial vocabulary recognition posttest provided five pictorial representations to select from for each test item.
In Study 2, a written vocabulary production posttest (Appendix D) was used to examine the effects of the four treatments on students' vocabulary learning both immediately after the treatment and 3 weeks later. It consisted of 25 of the 27 French keywords used in each treatment, and required students to provide their English translations from memory. The maximum score for this test was 25, with each correct response worth one point.
Procedures for Studies 1 and 2
A pre/posttest control group design was used to observe the effects of the absence or presence of written and pictorial annotations on students' L2 vocabulary learning from the aural passage. All activities took place during three separate class periods of a normally scheduled French class. During the first class period, students had 8 minutes to complete the written vocabulary recognition pretest. Two days later, each participant was randomly assigned to one of four treatments: (1) no annotations (2) pictorial annotations, (3) written annotations, and (4) pictorial and written annotations. Students were given 14 minutes to listen to the passage and to access the annotations. Students in Study 1 then had 8 minutes each to complete the written and the pictorial recognition posttests. Students in Study 2 had 8 minutes to complete the immediate written vocabulary production posttest. Three weeks later, without any additional experience with the aural passage and without prior warning, students in both studies completed the delayed vocabulary tests that were identical to the tests given immediately after treatment.