Assessing User Needs in Early Stages of Program Development:
The Case of Foreign Language Reading
James N. Davis, Mary Ann Lyman-Hager, Susan B. Hayden
ABSTRACT: In this article, we argue that software developers should use a combination of methodologies to determine the reading difficulties of lower-level
foreign language students. During the pre-production phases of our program, ClearText, we asked students of third-semester French at Pennsylvania State University to read a passage in French. One group circled all the unknown words and expressions in the passage; other students wrote the definitions of words we had underlined in the text; the last group wrote recall protocols based upon what they remembered from their reading. We demonstrate that each of these assessment instruments was necessary in product development in order for us to identify the different types of foreign language reading problems encountered by students at this instructional level.
KEYWORDS: foreign language reading, French language, literature in a foreign language; software for reading comprehension; research in software development.
Software developers appear to be in general agreement as to the importance of user input during the planning phase of a new project. In conjunction with educational research on general characteristics of the target population, close scrutiny of the needs of potential users of a product can provide developers with a more solid empirical and conceptual foundation upon which to build. The means of obtaining learner input, as noted by Flagg (1990), include written questionnaires, small-group observations, and interviews. In this article we discuss the utilization of assessment instruments to identify user difficulties in the early stages of the development of software for undergraduate readers of literary texts in a foreign language. We will, first, briefly review several trends in foreign language reading research that lent theoretical underpinnings to the present project. Second, we will describe ClearText, a software project we are developing at the Pennsylvania State University. Third, we will explain the data collection procedures employed to determine user needs and will discuss how the results of our study were applied to program development.
Until fairly recently, foreign language researchers and practitioners focused little attention on reading. The assumption in the field was that "[foreign language] reading will follow from knowing the structure of the [foreign] language and knowing how to read in the first language" (Weber 1991, 45). Beginning in the early 1980s, investigators recognized that, in addition to knowledge of the foreign language in which a passage was written, awareness of such factors as the content and structure of a text was also necessary for understanding (Bernhardt 1983; Carrell 1983; Davis, Lange & Samuels 1988). Perhaps the most comprehensive view of the types of knowledge essential in foreign language reading was proposed by Bernhardt (1990). In her Model of Second Language Reading, based upon data collected from undergraduate foreign language students, Bernhardt proposed six components essential for effective comprehension: word recognition (i.e., understanding the meaning of individual words),
phonemic/graphemic decoding (i.e., recognizing words through perceptions of their pronunciation or spelling), syntactic feature recognition (i.e., understanding the
relationship among words), intratextual perception (i.e., reconciling each segment of a passage with segments that come before and after it), prior knowledge (i.e., the
awareness of topic and structure a reader brings to a text), and metacognition (i.e., a reader's awareness of the cognitive processes used during reading). An important implication of Bernhardt's model is that, in order to understand what they are reading, foreign language students need much more than a dictionary in which to look up word meanings.
Recently, Computer-Aided Instruction (CAI) specialists have turned their attention to using the capabilities of the computer to enhance the comprehension of foreign
language readers. Computer-Assisted Language Learning (CALL) programs such as Foelsch's Annotext (1992), now in Beta testing, are demonstrating what computer mediation can contribute to the novice foreign language reader's experience of literary texts. ClearText, a program in development at the Pennsylvania State University (Davis, Lyman-Hager, & Hayden in progress), provides foreign language readers with five types of information on unfamiliar words and expressions. Inspired by the Bernhardt model, these informational categories are: Word Definitions (i.e., two types of word meanings, in English and in French), Grammar (i.e., brief notes on, e.g., how verb tenses are used in a text), Culture (i.e., background material on the cultural references in the text), and Relations (e.g., details on connections between characters in a passage). In order for students to be able to read without interruption, the ClearText glosses are unobtrusive. Users need only to click on unfamiliar words or expressions for help. As appropriate, all five types of information are available for consultation.
Programmers at Penn State's Computer-Based Educational Laboratory are designing a ClearText prototype that can be applied to virtually any foreign language text in any language. An important component of ClearText is a tracking device that will enable researchers to determine how often users consult glosses, which categories of glosses are consulted and how long users spend actually examining the glosses. The program will also permit the disabling of certain gloss components as a way to determine which adjunct aids are optimally useful for foreign language reading comprehension. The ClearText tracking device will be used in conjunction with other instruments such as recall protocols and individual learner strategy questionnaires in order to measure the effects of various categories of information on different types of readers.
The passage we are using in the development of ClearText is from the first chapter (the first approximately 1600 words) of Ferdinand Oyono's Une vie de boy (1956). Une vie de boy is a novel about Africa during the period of French colonization. In addition to problems understanding the language in which the novel is written, many foreign language readers also experience difficulties with unfamiliar cultural references. Most of the intended users of this software are students at the third-semester level of
undergraduate French. Students at this instructional level are just beginning to read literature in the foreign language.
During the pre-production phase of the ClearText prototype, an important issue was determining which words and expressions should be glossed for users. To identify difficulties with the passage, we assessed the comprehension of third-semester students of French (n=101) using three different procedures, given to three main treatment
groups. First, we passed out copies of the text to one group of students and asked them to circle all the words of which they did not know the meaning. Second, we identified 140 words that we assumed would be unknown to readers at this instructional level. We underlined the words we had identified and asked another group of students to write their definitions in English in spaces provided at the end of the passage. There were seven different versions of the second task (twenty different words per version), which was completed by seven subgroups. Third, we asked the last group of students to read the passage and then (without referring to the text) to write down everything they recalled from it. This procedure, known as the written recall protocol, is probably the most commonly used research tool in reading. Recall protocols provide
investigators with information about words that readers do not understand as well as about more general comprehension problems involving an entire passage.
On the first measure (students' circling unknown words and expressions), if we found that more than one reader had circled a word or expression, we decided to gloss that word or expression in our program. On the second task (writing the definitions of
words and expressions we had circled), if more than one reader had written an incorrect definition, we, likewise, selected the word or expression to be glossed. It was fairly common to find words not circled by the first group that were incorrectly defined by students in the definition-task groups. We assumed that many of the students who did not circle these words did not really know their correct meaning. They were simply unaware of their misunderstanding. As several studies of foreign language reading have shown, beginners are often simply not conscious of comprehension problems (e.g., Carrell 1984; Davis 1987) Our findings and those of other reading researchers
underscore the desirability of using two different vocabulary knowledge measures, one based partly upon our own intuitions and the other based entirely upon the users' perceptions.
While the vocabulary knowledge measures, discussed above, provided important data on specific words and expressions readers did not recognize, the recall protocols yielded information not available from the first two instruments. Input from the recall protocols was useful in three ways. First, the recall protocols helped us to identify unknown words and expressions that had not been revealed by the two vocabulary knowledge measures. For instance, several readers, seeing the French expression "en train de," rendered in English by "in the process of," thought that a train had been mentioned in the passage and wrote sentences such as: "He finds a train and ends up going to live with Father Gilbert whom he meets." Although our intuitions (and the students' perceptions) would have led us to assume that students at this level would not misinterpret the rather common expression "en train de," this measure showed us that it was, in fact, unfamiliar to a number of readers.
Second, data from the recall protocols led us to include pronouns as part of the glossing system. In a number of instances, students seem to have misunderstood which character was being referred to by a particular pronoun. For example, in the sentence "Il nous lançait ses petits cubes sucrés comme on jette du grain aux poules." [He would throw us his little sugar cubes as you might throw grain to chickens.], the pronoun "il" [he] refers to the priest in the novel. Students wrote sentences such as "Whites offered Toundi, the boy, sugar cubes." and "He describes how he took sugar cubes from the white people..." Knowing which character performed a given action is, of course, important to a
Third, what students wrote in their recall protocols also led us to decide to do much of the glossing in groups of words rather than as individual words. The sentences of the novel were somewhat longer and more complex than the "textbook" language most students were accustomed to. Probably because of this factor, many students combined individual words in ways not intended by the author. For instance, the extract
"...l'homme blanc aux cheveux semblables à la barbe de maïs, habillé d'une robe de femme, qui donnait de bons petits cubes sucrés aux petits Noirs." [the white man with hair like a corn-colored beard, dressed in a woman's dress, who gave good little sugar cubes to the black children] was misunderstood as "There was a woman who gave all the good little black children little cubes of sugar." and "The missionary's wife gave the little black children cubes of sugar." Because the priest is mistakenly described by the African narrator as wearing a woman's dress (actually a priest's robe), the person performing the action was incorrectly thought to be a woman. It is likely that most of the words in this sentence, tested individually, would have been familiar to students. Probably because they appeared in such a long and complex sentence, their
combination in this manner caused comprehension problems for this group of novice foreign language readers. For this reason, we decided that glossing many of the words in the text in larger groups would facilitate understanding.
In this article, we have shown that input from potential users is a key component in pre-production stages of educational software development. We have, furthermore,
emphasized the necessity of a combination of assessment instruments for more precise evaluation of learner needs, particularly in the case of foreign language readers.
Our aim is to incorporate data about users during all phases of the development of ClearText, including, as we noted above, post-production. As Foelsch (personal
communication) has remarked, software developers do not build sufficiently upon the successes—and failures—of existing programs. In the future, an essential component of all educational software should be the means to measure both the enjoyment and actual learning that users experience. As Garrett (1991) has asserted, software should be both a teaching and a research too]. Judicious applications of research findings from CAI investigations as well as from appropriate source fields will not only add to our small knowledge base but also result in more successful courseware.
REFERENCES
Bernhardt, E.B. (1983). Three approaches to reading comprehension in intermediate German. Modern Language Journal 67, 2, 111-15.
________. (1990). A model of L2 text construction: The recall of literary texts by learners of German. In A. Labarca (Ed.) Issues in L2: Theory as practice/Practice as theory. [21-43] Norwood, NJ: Ablex.
Carrell, P.L. (1983). Three components of background knowledge in reading comprehension. Language Learning 33, 2, 183-207.
__________.(1984). Evidence of a formal schema in second language comprehension. Language Learning 34, 2, 87-112.
Davis, J.N. (1987). Effects of knowledge of text structure on comprehension of expository prose read in a foreign language. Unpublished Dissertation, Univ. of Minnesota, Minneapolis.
_________, & Samuels, S.J. (1988). Effects of Text Structure Instruction on Foreign Language Readers' Recall of a Scientific Journal Article. Journal of Reading Behavior, 20, 3, 203-14.
__________, Lyman-Hager, M., & Hayden, S.B. (In Progress). ClearText. University Park, PA: Computer-Based Educational Laboratory.
Flagg, B.N. (1990). Formative Evaluation for Educational Technologies. Hillsdale, N.J.: Erlbaum.
Foelsch, O. (1992). Annotext. Hanover, NH: PandaPress.
Garrett, N. (1991). Technology in the service of language learning: Trends and issues. Modern Language Journal 75, 1, 74-101.
Oyono, F. (1956). Une vie de boy. Paris: René Julliard.
review editor for Language Learning and is currently developing a French reader for Heinle & Heinle Publishers.
Mary Ann Lyman-Hager (M.A. French, University of Arizona, Tucson, M.Ed. and Ph.D., University of Idaho, Moscow) is a researcher in second language acquisition and technology. She publishes in the CALICO Journal, the IALL Journal, and the Modern Language Journal and is Project Director of the Language 3 Initiative, sponsored by I.B.M. She is president of the North East Association of Learning Laboratories and Vice Chairperson of the Hypermedia Special Interest Group of CALICO.
Susan Hayden (M.A.T. in French, Portland State University) is currently in a doctoral program in Second Language Acquisition in French at the Pennsylvania State
University. She teaches high school French, trains secondary foreign language teachers at Portland State, and is an O.P.I. tester in French.
AUTHORS'ADDRESSES
James N. Davis
Department of French 316-S-Burrowes
Pennsylvania State University University Park, PA 16802 Tel.:(814) 863-4388
E-Mail: jndl@psuvm Mary Ann Lyman-Hager Department of French 316-S-Burrowes
Pennsylvania State University University Park, PA 16802 Tel.:(814) 865-1492 E-Mail: mal1@psuvm Susan B. Hayden Department of French 316-S-Burrowes
