Is Computer-Based Grammar Instruction as Effective as Teacher- Directed Grammar Instruction for Teaching L2 Structures?

Is Computer-Based Grammar

Instruction as Effective as

Teacher-Directed Grammar Instruction

for Teaching L2 Structures?

Joyce Nutta

ABSTRACT

The study described here compared postsecondary English as a Second Language (ESL) students’ acquisition of selected English structures based on the method of instruction—computer-based instruction versus teacher-directed instruction.1 _{The results showed that for all levels of English}

pro-ficiency, the computer-based students scored significantly higher on open-ended tests covering the structures in question than the teacher-directed students. No significant differences were found between the computer-based and teacher-directed students’ scores on multiple choice or fill-in-the-blank tests. The results indicate that computer-based instruction can be an effective method of teaching L2 grammar.

KEYWORDS

Computer Assisted Language Learning (CALL), Grammar Instruction, Focus on Form, Structural Syllabus, English as a Second Language (ESL), Computer-Based Instruction

INTRODUCTION

As the use of computers in language teaching increases, it is essential to establish research-based indications of the appropriate roles of Computer Assisted Language Learning (CALL) in curriculum and instruction. One of the most important questions focuses on whether technology should attempt to emulate the characteristics of a communicative classroom, en-gaging students in real and meaningful communication, or provide the types of tutorials and drills that tend to be de-emphasized in current teach-ing practice. Numerous Second Language Acquisition researchers have

asserted that the computer should be used to replicate what they believe ought to occur in the classroom. Many proponents of CALL have advo-cated the development of communicative computer programs that pro-vide opportunities for meaningful communication (Garrett, 1991; Underwood, 1993; Quinn, 1990; Lavine, 1992). Although some educa-tors have decried the use of computers as electronic workbooks for drill-and-practice exercises (Chun & Brandl, 1992; Underwood, 1993), others have advocated their use for tutorials and drills to free up more classroom time for real communication (Gilby, 1996; Hoffman, 1996).

The teaching context often determines the role of CALL. In an English as a Second Language (ESL) environment, communicative CALL pro-grams often supplement and augment classroom activities by providing games for practice or word processing applications for compositions. In some English as a Foreign Language (EFL) programs, computer programs may complement or completely supplant classroom instruction by provid-ing instruction in a subject or skill not taught in the classroom (Soo & Ngeow, 1996). In the latter case, computers and other technologies are relied upon to provide a model of native speech that the instructors, many of whom are nonnative speakers of English, cannot offer.

During the past decade, numerous studies have examined different ap-proaches to teaching grammar (Doughty, 1991; Ellis,1985, 1993; Fotos, 1993; Green & Hecht, 1992; Tomasello & Herron, 1988). These studies have shown that many effective means of teaching L2 grammar are avail-able—from teacher-directed Constructivist tutorials (Adair-Hauck & Donato, 1994), to cooperative group work (Fotos, 1994) and individual study with textbooks (Scott & Randall, 1992). Because these studies indi-cate that various effective ways to learn grammar exist, teachers have an ever increasing array of options with which to meet the needs of students. The use of computers to teach grammar has not received the same amount of attention as communicative CALL, but computer-based grammar in-struction offers many potential benefits. Although it is currently impos-sible for the computer to engage learners in authentic two-way communi-cation, it is, in fact, possible for CALL to provide rich input in the form of integrated multimedia programs and to provide explicit grammar expla-nations that can be viewed and reviewed at the learner’s own pace. In a metaanalysis of research on the use of multimedia to teach a variety of subjects, Ragan, Boyce, Redwine, Savenye, and McMichael (1993) found that, in general, multimedia instruction reduces learning time by 30% compared to traditional instruction. They further demonstrated that fea-tures of multimedia instruction such as learner interactivity and learner control over programs produce improved outcomes in achievement.

Results of studies on computer assisted instruction have been generally positive. For over two decades, researchers have examined the effective-ness of computer assisted instruction to teach many different subjects. In

1987, Kulik and Kulik published a ground-breaking metaanalysis of com-parative studies on computer assisted and classroom instruction in which they calculated an overall effect size of .31. Other studies by Niemiec and Walberg (1987) and McNeil and Nelson (1991) uncovered similar results. In spite of the abundance of comparative research on computer assisted instruction in other academic fields such as reading (Rachal, 1995) and the growing body of research on methods of teaching grammar, only re-cently have researchers investigated the use of computer-based L2 gram-mar instruction (McEnery, Baker & Wilson, 1995; Nagata, 1996). The results of these studies seem to indicate that computer-based grammar instruction can be as effective or more effective than more traditional in-struction (e.g., workbooks and lectures).

If subsequent research bears out this initial indication of the benefits of computer-based grammar instruction, this kind of instruction could be-come an important complement to communicatively oriented language classes. By using the computer for the presentation, explanation, and ap-plication of grammatical structures, more classroom time could be dedi-cated to real communication that focuses on expressing meaning and us-ing appropriate grammatical structures to express that meanus-ing. Shiftus-ing the systematic study of grammar points from the realm of the classroom to the domain of the computer laboratory would enable instructors to take advantage of classroom interpersonal dynamics and allow them to take into account differences in background knowledge and learning styles. The use of computer-based grammar instruction would also support more individualized instruction in programs that have open-entry/open-exit enrollment such as adult and vocational education courses. Students would not enter the class in the middle of a linear instructional sequence based on a grammatical syllabus. Instead, the syllabus could be based on themes of interest and relevance to students, and individual students could follow a grammatical syllabus in computer-based instruction outside the classroom.

PURPOSE OF THIS STUDY

This study examined whether computer-based grammar instruction is as effective as teacher-directed grammar instruction for postsecondary stu-dents at multiple levels of proficiency in an intensive ESL program. Two experiments were conducted at different levels of English language profi-ciency, one at the first and second level of instruction and the other at the third and fourth level. The primary research question posed in the study was whether there was a difference in the acquisition of a specific gram-mar point for students taught in a teacher-directed class versus those taught in computer-based instruction.

Method

The first experiment compared the performance of one and level-two students in a computer-based group versus those in a teacher-directed group. The second experiment compared the performance of level-three and level-four students in a computer-based group versus those in a teacher-directed group. The purpose of conducting the two experiments was to examine the acquisition of discrete structures at different levels of profi-ciency in order to increase the generalizability of the results.

Sample

The population of the study consisted of 53 students (24 females and 29 males) enrolled in an intensive academic ESL institute at a major univer-sity in Florida. The ESL institute offers four levels of instruction and uses the Comprehensive English Language Test (CELT) to place in-coming stu-dents. Ten students were enrolled in level one, 9 in level two, 20 in level three (in two sections), and 14 in level four. Students were matched for native language (Japanese, Thai, Korean, Arabic, Spanish, Italian, French, and Portuguese) and pretest scores on the structures in question and then randomly assigned to computer-based or teacher-directed sections. Al-though the groups were not matched for gender, the breakdown was fairly even, each group differing in gender makeup by no more than two stu-dents.

Independent Variable

The independent variable was the method of grammar instruction, ei-ther teacher-directed or computer-based. Because the participants in each experiment consisted of two levels of English proficiency (first and sec-ond levels combined, third and fourth levels combined), grammar points were introduced at the lower level of the combined groups and then re-viewed and expanded at the higher level. This cycle held true for the teacher-directed as well as computer-based groups. The treatment consisted of one hour of instruction per day for seven days. Because studies have indi-cated that certain grammar points are more easily acquired than others (Green & Hecht, 1992; Scott & Randall, 1992; Krashen, 1981), this study examined the acquisition of verb tenses, which included elements of as-pect, one of the more difficult forms for ESL students to master (Green & Hecht, 1992).

Teacher-Directed Group

Students in the teacher-directed groups were taught by five different teachers of varying degrees of experience. Classes were held in rooms with-out computer equipment or other instructional technology. Class size ranged from four to seven students. Students at all levels used the Focus on

Gram-mar (1994) textbook series and engaged in a variety of types of activities

emphasizing interactive, meaningful, and creative expression.

Computer-Based Groups

Students in the computer-based groups received computer-based instruc-tion outside the classroom. The students who participated in Experiment One used ELLIS Middle Mastery (1996), and the students who partici-pated in Experiment Two used ELLIS Senior Mastery (1996). ELLIS was selected because of its multimedia delivery (audio and video, recording capability, etc.), its modeling of natural and contextualized language, its interactivity, and its clear grammar explanations and practice activities. Although ELLIS lacks a mechanism for sophisticated learner feedback (see Nagata & Swisher, 1995), its exercises do provide simple corrections. Because the ELLIS program is organized around communicative needs rather than grammatical structures, students followed a sequential check-list developed by the researcher to direct their navigation through the pro-gram. (See the sample navigational guide in Appendix A.)

Dependent Variables

In Experiment One, the grammar structure of interest was the past tense, and in Experiment Two, the conditional tense. The dependent variables were students’ achievement scores on three separate criterion-referenced tests over the selected structures. The tests on the past tense included items covering the simple past in regular and irregular forms, the past continuous, and the present perfect (regular form). The tests on the con-ditional tense included items covering the factual, unreal, and unlikely conditional.

The three tests consisted of (1) a discrete-point multiple-choice test, (2) a fill-in-the-blank test, and (3) an open-ended test. The researcher, to-gether with the ESL institute’s curriculum specialist and the grammar teach-ers, decided which grammar points were appropriate to teach and assess. The researcher then developed test instruments based on the content of the instruction in the teacher-directed and computer-based groups.

mea-surements, not combined into a single dependent variable, since there is a theoretical basis for considering them as measures of different constructs. For example, multiple choice tests do not measure linguistic production (Hughes, 1989) and therefore can be appropriately used to assess stu-dents’ knowledge of grammar within the framework of grammatical con-sciousness-raising (Sharwood-Smith, 1981). Since open-ended tests mea-sure students’ linguistic production, they represent more demanding tasks of application of knowledge than do multiple choice tests. The same bat-tery of tests were used for pretests, immediate posttests, and delayed posttests.

A pool of test items was field tested at two adult education schools. From the initial item pool, ten items were selected for the fill-in-the-blank tests (scale of zero to five), ten items for the multiple choice tests, and five items for the open-ended tests (scale of zero to one). Teachers at both adult education schools, as well as the grammar teachers at the institute, examined the tests for content validity and made suggestions for changes that were eventually incorporated into the final version of the tests.

In addition to analyzing students’ performance as represented by their values for the three dependent variables, the researcher conducted native language interviews with students in the computer-based group of Experi-ment One and administered questionnaires to students in the computer-based group of Experiment Two. The purpose of these interviews and questionnaires was to solicit students’ views of the computer program and computer-based learning.

Data Collection and Analysis

The pretests were administered three days prior to the beginning of the treatment, the immediate posttests were administered on the last day of the treatment, and the delayed posttests were administered two weeks after the posttests. After having been trained, the classroom teachers ad-ministered the pretests and delayed posttests to all students and the im-mediate posttests to the students in the teacher-directed groups. The re-searcher administered the immediate posttests to the students in the com-puter-based group. The researcher scored all the tests blindly with no knowledge of students’ identity or group status. After the researcher scored the tests, a certified ESL teacher reviewed students’ answers. For all po-tentially problematic answers and scores, the researcher and the teacher reached consensus on the basis of the scoring sheet. (See sample ques-tions and scoring criteria in Appendix B.)

For the pretests, immediate posttests, and delayed posttests in Experi-ment One, the calculation of internal consistency was .90 for the fill-in-the-blank test, .58 for the multiple choice test, and .63 for the open-ended

test. For the pretests, immediate posttests, and delayed posttests in Ex-periment Two, the calculation of internal consistency was .67 for the fill-in-the-blank test, .40 for the multiple choice test, and .43 for the open-ended test. Analysis of Covariance (ANCOVA) was used to analyze the data. Because of the small sample size, the alpha level was set at .10 for tests of significance.

Results

As in most comparative CALL studies, it was difficult to distinguish which features of the computer-based instruction led to the outcomes found in the study and whether these factors were inherent to computer-based instruction or simply an instructional strategy that could have been em-ployed with a variety of media under different circumstances (see Will-iams & Brown, 1991). Nonetheless, the results showed that computer-based grammar instruction is at least as effective as, and in some cases more effective than, teacher-directed grammar instruction.

Experiment One

Table 1 summarizes the results of the descriptive analysis of students’ test scores in Experiment One.

Table 1

Mean Test Scores by Group in Experiment One

Type Time Computer-Based Group Teacher-Directed Group of Test of Test (N = 9) (N = 10)

Mean Standard Mean Standard Deviation Deviation Open- Pretest 0.44 0.53 0.30 0.68 Ended Immediate 3.10 1.76 1.20 1.62 Posttest Delayed 3.44 1.60 1.80 1.75 Posttest Fill-in- Pretest 13.67 11.14 8.20 9.27 the-blank Immediate 31.11 14.62 14.90 14.01 Posttest Delayed 22.67 17.95 16.80 12.04 Posttest Multiple Pretest 5.67 2.45 5.00 1.89 Choice Immediate 6.22 2.17 6.80 1.87 Posttest Delayed 6.56 2.46 5.70 1.77 Posttest

Analysis revealed a significant difference between the test scores of the students in the computer-based and those in the teacher-directed groups on the open-ended immediate posttest (p < .10). The computer-based group’s mean score on this test was 3.10, and the teacher-directed group’s, 1.20. Using adjusted means, the effect size for the difference was 1.56. Analysis revealed a similar significant difference between the computer-based and the teacher-directed groups on the open-ended delayed posttest (p < .10). The computer-based group’s mean score was 3.44, and the teacher-directed group’s, 1.80. Again using adjusted means, the effect size for this difference was 1.73. Finally, analysis also showed a significant difference between the computer-based and teacher-directed groups’ mean scores on the fill-in-the-blank immediate posttest, 31.11 versus 14.90 (p < .05). This difference disappeared, however, by the time of the delayed posttest. No significant differences were found between the groups for the multiple choice immediate posttest or delayed posttest.

Experiment Two

Table 2 summarizes the results of the descriptive analysis of students’ test scores in Experiment Two

Table 2

Mean Test Scores by Group in Experiment Two

Type Time Computer-Based Group Teacher-Directed Group of Test of Test (N = 14) (N = 10)

Mean Standard Mean Standard Deviation Deviation Open- Pretest 3.14 1.56 2.80 0.92 Ended Immediate 4.50 1.02 4.20 0.79 Posttest Delayed 4.79 0.43 4.00 0.67 Posttest Fill-in- Pretest 37.29 2.89 38.90 5.02 the-blank Immediate 45.64 5.18 43.90 4.11 Posttest Delayed 46.29 3.73 45.90 4.61 Posttest Multiple Pretest 7.00 2.08 6.40 2.32 Choice Immediate 8.43 1.28 8.10 1.52 Posttest Delayed 8.71 1.33 8.90 1.10 Posttest

Analysis showed a significant difference between the computer-based and the teacher-directed groups’ mean scores on the open-ended delayed posttest. The computer-based group’s mean score was 4.79, and the teacher-directed group’s, 4.00. The effect size, once more using adjusted means, was 1.19. No significant differences were found on any of the other posttests.

Interviews and Questionnaires

Insights into students’ experience with computer-based instruction emerged from the student interviews and questionnaires. By and large, students were satisfied with the computer-based instruction and expressed a desire to spend more time per day using it. Students indicated that the features of computer-based instruction that were most useful were the computer’s capacity that allowed them to review the tutorial as many times as they wished, to proceed at their own learning pace, to record their voices and compare them against the model, and to get immediate feed-back on the exercises. The Asian students especially appreciated not being “singled out” to speak in class, while some of the Latin students indicated that they would have preferred more human interaction.

Discussion

Although the sample size of this study was too small to draw definitive conclusions, the study does present evidence of meaningful differences in the computer-based and teacher-directed groups’ achievement scores on the open-ended tests. If open-ended tests measure students’ ability to use grammatical structures creatively, it would seem that some elements of computer-based study support the development of this skill more effec-tively than traditional classroom instruction.

Surprisingly, the scores of the students in the computer-based groups rose on the open-ended delayed posttests. This result contrasts with those of other studies of classroom grammar instruction in which students’ pro-duction of grammatical structures was accurate on immediate posttests but fell on the delayed posttests. Perhaps the interim period between the immediate posttest and the delayed posttest described here allowed the students in the computer-based group time to apply the structure in ques-tion in real communicative situaques-tions, providing them with an addiques-tional opportunity to negotiate meaning and monitor their own output. This find-ing may indicate, as Ellis (1993) has suggested, that the use of computer-based grammar instruction can complement individualized structural syl-labi in communicative classrooms and more effectively enable students to

use the newly acquired structures to negotiate meaning.

The potential of the computer laboratory to go beyond providing simple practice and reinforcement of grammar points taught in the classroom is only beginning to be explored. Research should be conducted with differ-ent populations (e.g., elemdiffer-entary and secondary studdiffer-ents) and differdiffer-ent types of courses (e.g., Vocational ESL and English for Specific Purposes). Moreover, additional research is needed to ascertain which features of computer-based grammar programs promote the acquisition of L2 struc-tures (e.g., degree of learner control, feedback strategies, etc.). As the body of knowledge on the application of computer-based grammar in-struction increases, so will the flexibility and number of options available to teachers and students.

Appendix A

Sample ELLIS Navigational Guide

Conversations by Grammar Topic Date

Calling to set up a job interview—See your selection now 1. Could I make an appointment for an interview?

Click on each paragraph Culture

Vocabulary Grammar

Video (Script, Keywords, Role-Play—Record) Phrases

Exit—Return to Conversations by Grammar Topic

Conversations by Grammar Topic Date

Calling for information about a job—See your selection now 1. Could I make an appointment for an interview?

Grammar Guide Conditional Sentences Easy—page 1 view/hear Easy—page 2 view/hear Easy—page 3 view/hear Medium—page 1 view/hear Difficult—page 1 view/hear Review—page 1 hear

Main Activity Menu

Practice Activities Date Practice Items Easy Grammar Practice Activities Practice Items Medium Grammar Practice Activities Practice Items Difficult Grammar

Main Activity Menu

Listening Activities Date Listening Comprehension

Easy Medium Difficult Skills Check

Appendix B

Sample Test Items and Scoring Criteria Open-Ended

If I didn’t have to study to get good grades, ___________________________ Scored as correct (1 point) or incorrect (0 points). Answers scored as correct must make sense and be correctly formed.

Multiple Choice

___________ the movie good last night? A. Is B. Did C. Was D. Were One correct answer per item.

Fill-in-the-Blank

About two hours ago, a very unusual thing ______________. A student from the English Language Institute __________________ a large, round UFO…

5 points—correct formal use of grammar and semantically correct 4 points—correct use of formal grammar and semantically incorrect 3 points—correct use of informal grammar and semantically correct 2 points—correct use of informal grammar and semantically incorrect 1 point—incorrect use of grammar and semantically correct

Answers were judged as informal grammar use if they were variations of the structure that were not taught and that are not part of standard, writ-ten usage (e.g., Did you ever go to Europe?)

NOTE

1 _{This study was funded, in part, by a Florida Department of Education Section}

353 Grant.

REFERENCES

Adair-Hauck, B. & Donato, R. (1994). Foreign language explanations within the Zone of Proximal Development. The Canadian Modern Language Re-view, 50, 532-557.

Chun, D. M., & Brandl, K. K. (1992). Beyond form-based drill and practice: Mean-ing-enhanced CALL on the Macintosh. Foreign Language Annals, 25, 255-267.

Doughty, C. (1991). Second language instruction does make a difference. Studies in Second Language Acquisition, 13, 431-469.

ELLIS [Computer software]. (1996). Salt Lake City, UT: CALI.

Ellis, R. (1985). Understanding second language acquisition. Oxford: Oxford Uni-versity Press.

Ellis, R. (1993). The structural syllabus and second language acquisition. TESOL Quarterly, 27, 91-113.

Focus on Grammar: Longman Grammar Series. (1994). New York: Addison Wesley. Fotos, S. S. (1993). Consciousness raising and noticing through focus on form: Grammar task performance versus formal instruction. Applied Linguis-tics, 14, 385-407.

Fotos, S. S. (1994). Integrating grammar instruction and communicative language use through grammar consciousness-raising tasks. TESOL Quarterly, 28, 323-351.

Gilby, W. (1996). Irrwege des Zweitsprachenerwerbs: Gehort auch das Computerlabor dazu? [False directions in second language acquisition: Does the computer laboratory also count as one?]. Unterrichtspraxis/ Teaching German, 29, 87-91.

Garrett, N. (1991). Technology in the service of language learning: Trends and issues. Modern Language Journal, 75, 74-101.

Green, P. & Hecht, K. (1992). Implicit and explicit grammar: An empirical study. Applied Linguistics, 13, 168-84.

Hoffman, S. (1996). Computers and instructional design in foreign language/ESL instruction. TESOL Journal, 5 (2), 24-29.

Hughes, A. (1989). Testing for language teachers. New York: Cambridge. Krashen, S. (1981). Second language learning and second language acquisition.

Oxford: Pergamon.

Kulik, J. & Kulik, C. (1987). Review of recent research literature on computer-based instruction. Contemporary Educational Psychology, 12, 222-30. Lavine, R. Z. (1992). Rediscovering the audio language laboratory: Learning through

communicative tasks. Hispania, 75, 1360-1367.

McEnery, T., Baker, J. P., & Wilson, A. (1995). A statistical analysis of corpus based computer versus traditional human teaching methods of part of speech analysis. Computer Assisted Language Learning, 8, 259-274.

McNeil, B. J. & Nelson, K. R. (1991). Metaanalysis of interactive video instruction: A 10 year review of achievement effects. Journal of Computer-Based In-struction, 18 (1), 1-6.

Nagata, N. (1996). Computer vs. workbook instruction in second language acqui-sition. CALICO Journal, 14, 53-75.

Nagata, N. & Swisher, M. V. (1995). A study of consciousness-raising by computer: The effect of metalinguistic feedback on second language learning. For-eign Language Annals, 28, 337-347.

Niemiec, R. & Walberg, H. (1987). Comparative effects of computer-assisted in-struction: A synthesis of reviews. Journal of Educational Computing Re-search, 3, 19-37.

Quinn, R. A. (1990). Our progress in integrating modern methods and computer-controlled learning for successful language study. Hispania, 73, 297-311. Rachal, J. R. (1995). Adult reading achievement comparing computer-assisted and traditional approaches: A comprehensive review of the experimental lit-erature. Reading Research and Instruction, 34, 239-258.

Ragan, T., Boyce, M., Redwine, D., Savenye, W. C., & McMichael, J. (1993, Janu-ary). Is multimedia worth it?: A review of the effectiveness of individual-ized multimedia instruction. Paper presented at the Association for Edu-cational Communications and Technology Convention, New Orleans, LA. Scott, V. M. & Randall, S. A. (1992). Can students apply grammar rules after reading textbook explanations? Foreign Language Annals, 25, 357-367. Sharwood-Smith, M. (1981). Consciousness-raising and the second language learner.

Applied Linguistics, 11, 159-168.

Soo, K. S. & Ngeow, Y. H. (1996). The English teacher vs. the multimedia com-puter: The UNIMAS experience. Manuscript submitted for publication. Tomasello, M. & Herron, C. (1988). Down the garden path: Inducing and

correct-ing overgeneralization errors in the foreign language classroom. Applied Psycholinguistics, 9, 237-246.

Underwood, J. H. (1993, August/September). The lab of the future: Using technol-ogy to teach foreign language. American Association of Community Col-leges Journal, 33-39.

Williams, C. & Brown, S. (1991). A review of the research issues in the use of computer-related technologies for instruction: An agenda for research. Educational Media and Technology Yearbook, 17, 26-46.

AUTHOR’S BIODATA

Joyce W. Nutta is an assistant professor of Foreign Language Education/ TESOL at the University of South Florida. She holds a Master’s Degree in Applied Linguistics and a Ph.D. in Second Language Acquisition and In-structional Technology. Her research interests include CALL and distance learning. She has co-authored Virtual Instruction: Issues and Insights from

an International Perspective, Libraries Unlimited (in press) and is

cur-rently completing The Teacher’s Guide to Dynamic Distance Learning.

AUTHOR’S ADDRESS

Joyce W. Nutta, Ph.D.

Assistant Professor of Foreign/Second Language Education Department of Secondary Education—EDU 208B

University of South Florida 4202 East Fowler Avenue Tampa, FL 33620-5650

Phone: 813/974-3759; 813/974-3533 Fax: 813/974-3837