Research and Practice

Measuring Assistive Technology Outcomes in Writing Given the importance of the topic of assistive technology outcomes to both researchers and practioners, this column is devoted to a series of articles on measuring assistive technology outcomes in specific academic domains. In the previous column I provided an overview of key concepts associated with measuring assistive technology outcomes. In subsequent columns I will examine issues associated with measuring assistive technology outcomes in math and reading. In this column, I focus on issues of assessing assistive technology outcomes in writing.

W

T

The importance of learning to express one’s self through writing has long been considered a hallmark of the educated citizen. Beginning with the physical process of handwriting; mastering the conventions of writing such as spelling, capitalization, and grammar; developing the habits of mind necessary to think deeply, capture ideas on paper, and revise; and acquiring tools of style, voice, and rhetoric; the successful writer ultimately integrates all of these components into an executive cognitive function. Numerous authors have highlighted the importance of teaching children to write effectively and have described various instructional interventions for enhancing student achievement in writing (Graham & Harris, 1988; Graves, 1985; Newcomer & Barenbaum, 1991; Scardamalia & Bereiter, 1986).

How Does Technology Enhance Writing?

Few would argue with the fact that the word processor provides a quantum leap over traditional tools, like the typewriter, for tasks associated with generating text. Most writers point to the ease of making corrections, the value of copy and paste, and ready access to writing aids like spelling and grammar checkers as critical features that enhance their performance as a writer. An extensive literature describes the potential and promise technology holds for writers of all ages and abilities (Cochran-Smith, 1991; Daiute, 1985; Montague, 1990). Arguably, the most significant finding suggests that electronic tools hold special promise for engaging individuals in the process of writing, in ways, that over time, lead to sustained efforts to express one’s self through writing, increased personal satisfaction with both the process and products of writing, and in some cases, quantifiable improvements in writing.

Assistive Technology and Writing

The field of special education technology has long recognized the area of writing as one in which technology

holds considerable promise for individuals with disabilities. This long-standing interest can be characterized as having three focal points: (a) alternatives for students who’s access to the development of written expression skills is impaired by handwriting difficulties, (b) interventions for engaging reluctant writers (i.e., I don’t want to... I can’t...), and (c) specific strategies that enhance written expression performance (i.e., planning, drafting, spelling, peer editing).

Charts like the one shown in Table 1 are commonly used to make educators aware of the application of specific assistive technology tools for students with disabilities. In addition, teachers, administrators, technology specialists, and related services personnel have found decision-assist frameworks (Wisconsin Assistive Technology Assessment Forms, http://www.wati.org; WATI Student Information Guide http://webschoolsolutions.com/wati/wati-stuguide.htm; Georgia Tools for Life, http://www.gatfl.org) useful to guide the process of identifying and selecting appropriate assistive technology for students with writing difficulties. In addition, the literature continues to showcase systemic approaches to assistive technology service delivery in the area of writing (Bierly & McCloskey-Dale, 1999; Hardin & Miracola, 2003; Laine & Sitko, 2001).

Research has explored and documented the value of various forms of assistive technology for student writers with disabilities. The research on effectiveness (What works?) offers a growing knowledge base on which to guide decision making concerning the selection and use of: spelling checkers (Dalton, Winbury, & Morocco, 1990; MacArthur, Graham, Haynes, & De La Paz, 1996; Montgomery, Karlan, & Couthino, 2001); speech synthesis (Borgh & Dickson, 1992; MacArthur, 1996); voice recognition (De La Paz, 1999; Icke, Perella, & Temple, 2001); word prediction (Koester & Levine, 1994; MacArthur, 1996); and word processors (Lewis, Ashton, Haapa, Kieley, & Fieldon, 1999; MacArthur, 1996). Recent work in the area of bilingual special education offers significant promise for understanding the unique assistive technology needs of English language learners with disabilities (Graves, Valles, & Rueda, 2000).

M

O

W

T

U

E

W

Interest in the measurement of assistive technology outcomes is a relatively recent phenomena. As a result, researchers often find themselves drawing upon the literature associated with measurement of writing improvement. Notable works include grading student writing (Hall, Salas, & Grimes, 1999; Speck, 1998; Tchudi, 1997), curriculum-based assessment (Espin, Shin, Deno, Skare, Robinson, & Brenner,

Research and Practice

Associate Editor’s Column

2000; Parker, Tindal, & Hasbrouck, 1991; Watkinson, & Lee, 1992), and measurement of semantic content (Lemaire & Dessus, 2001).

As a member of the Assistive Technology Outcomes Measurement System (ATOMS) Project, we have identified the following design, measurement, analysis, and decision-making factors that will need to be addressed in the process of creating an outcomes systems for measuring the impact of assistive technology:

Repeated measures research design Standardize the performance task

Standardize the data collection and coding process Analyze results using standardized metrics and benchmarks

Decision-making

In the sections that follow, I will describe each component in the context of trying to answer the question: How do you measure the outcomes of assistive technology for writing? Repeated Measures Research Design

Central to the definition of assistive technology is the expectation of enhanced performance. Smith (2000) outlines a theoretical view known as Time Series Concurrent and Differential (TSCD) Approach which involves a series of performance measures of an individual when s/he is completing a specific task, with AT, and without AT. Ideally, the results reflect a pattern that shows growth in improved performance in both conditions, however, the performance with AT is significantly greater than the performance without AT. The differences between the two measurements isolates the specific impact of AT and provides evidence of the impact and outcome over time.

The general utility of this approach to research seeking to measure the outcomes of assistive technology for writing appears promising. At this time, it is not known what is the optimal time period for data collection (daily, weekly, monthly) that would provide measures sensitive enough to demonstrate change. Nonetheless, this research design

appears highly compatible with existing instructional practices to warrant consideration. In addition, the research design will permit the collection of data from students without disabilities which will contribute significantly to subsequent performance benchmarking efforts.

Standardize the Performance Task

While high-stakes testing has introduced a number of activities that are used to assess writing performance, the assistive technology outcomes community needs to develop a series of tasks that they believe accurate reflect desired writing outcomes. It is important to note here, that just because something is easy to measure (i.e., number of words typed per minute) does not mean that it properly reflects the intended outcome (i.e., improvement in the quality of writing). As a result, it is necessary to identify a series of performance tasks (i.e., preparing a report after locating, evaluating, summarizing, analyzing, and synthesizing appropriate sources) that will standardize the data collection process of key outcomes. Standardize the Data Collection and Coding Process

Researchers often use commercial instruments (e.g., Test

of Written Language, Test of Written Spelling) in the

assessment of writing as well as holistic scoring tools, and rubrics. Unfortunately, there is a considerable void in the literature regarding the standardization of data collection procedures, timelines, and instruments for measuring assistive technology writing outcomes.

Analyze Results using Standardized Metrics and Benchmarks

Despite the lack of standardized data collection protocols, Fennema-Jansen (2001) mined the research literature and discovered an extensive array of measurement procedures that have been utilized to document progress in writing when students with disabilities use assistive technology. While these findings offer researchers an interesting menu of possibilities, additional work is needed to determine which metrics will be used with which specific data collection protocols.

An additional challenge in analyzing the results involves interpreting the data regarding the performance. That is, what benchmarks will be used to classify levels of performance? Little work has been devoted to understanding how to interpret assistive technology performance. The TSCD Approach advanced by Smith (2000) is based on the supposition that gain in technology-enhanced performance will be clearly visualizable in contrast to unaided performance. While this may be true, other factors such as slope (rate of increase) and performance over time may influence judgments about the performance. As we have learned from training raters in holistic scoring of writing, this Table 1.

Types of assistive technology interventions aligned with instructional challenges that confront writers.

challenge assistive technology interventions handwriting pencil grips, slantboards, dictation keyboarding keyboarding software, portable keyboards brainstorming and planning Kidspiration, Inspiration

word prediction Co:Writer, WordQ spelling hand-held spelling checkers text-to-speech TextHelp, Write:OutLoud

Table 2.

process of interpreting performance is likely to take considerable effort, over time, to properly calibrate the interpretive functions of the professionals involved in this process. Thus, there is little evidence to suggest that professionals currently have the ability to render the same interpretation of a given graph of performance.

Decision-making

Finally, the ultimate question involved in the assessment of assistive technology outcomes in writing relates to the following question: How do we understand/recognize technology-enhanced writing? That is, given a claim that a

specific assistive technology device enhances the writing performance of a specific student with a disability, can we support that claim? Or, do the data suggest the need to make a change in the instructional/remedial process? Or, change devices? Or, continue on the path we have set? Clearly, more work is needed regarding the use of assistive technology outcome data for decision-making.

S

The purpose of this article was to provide an introduction to the measurement issues associated with measuring assistive technology outcomes in writing. While the research Table 2.

Continued

Source: Fennema-Jansen, S. (2001). Measuring effectiveness: Technology to support writing.Special Education Technology Practice, 3(1), 16-22. Reprinted with permission.

References

Fifield, B. (1998). Evaluation definitions for the literacy technology project. [On-line} Available: http://www.ndcpd.org/ndcpd/people/staff/fifield/littech/contents.html. Hasbrouck, J.E., Tindal, G., & Parker, R.I. (1994). Objective procedures for scoring students’ writing.Teaching Exceptional Children, 26(2), 18-22.

MacArthur, C.A. (1998). From illegible to understandable: How word recognition and speech synthesis can help.Teaching Exceptional Children, 30(6), 66-71. MacArthur, C.A. (1998). Word processing with speech synthesis and word prediction: Effects on the dialogue journal writing of students with learning disabilities.

Learning Disability Quarterly, 22(3), 151-164.

Rousseau, M.K. (1990). Errors in written language. In R.A. Gable & J.M. Henderickson (Eds.),Assessing students with special needs: A sourcebook for analyzing and correcting errors in academics(pp. 89-101). NY: Longman.

Tindal, G., & Parker, R. (1989). Assessment of written expression for students in compensatory and special education programs.The Journal of Special Education, 23(2), 169-183.

and pedagogical knowledge base which informs current practice concerning students with disabilities, assistive technology, and writing is considerable, much more work needs to be done to create measurement tools and procedures that will enable the profession to make definitive statements about assistive technology outcomes.

R

Bierly, D.B., & McCloskey-Dale, R. (1999). The tasks, the tools: Needs assessment for meeting writing demands in the school curriculum. Closing the Gap, 18(3), 1, 24-25. Borgh, K., & Dickson, W.P. (1992). The effects of children’s writing

on adding speech synthesis to a word processor. Journal of

Research on Computing in Education, 24(4), 533-544.

Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of the literature.

Review of Educational Research, 61(1), 107-155.

Daiute, C.A. (1985). Writing and computers. Reading, MA: Addison-Wesley.

Dalton, B., Winbury, N. E., & Morocco, C. C. (1990). “If you could just push a button”: Two fourth grade boys with learning disabilities learn to use a computer spelling checker.

Journal of Special Education Technology, 10, 177-191.

De La Paz, S. (1999). Composing via dictation and speech recognition systems: Compensatory technology for students with learning disabilities. Learning Disabilities Quarterly,

22, 173-182.

Espin, C., Shin, J., Deno, S.L., Skare, S., Robinson, S., & Brenner, B. (2000). Identifying indicators of written expression proficiency for middle school students. Journal of Special

Education, 34(3), 140-153.

Fennema-Jansen, S. (2001). Measuring effectiveness: Technology to support writing. Special Education Technology Practice,

3(1), 16-22.

Graham, S., & Harris, K.R. (1988). Instructional recommendations for teaching writing to exceptional students. Exceptional Children, 54(6), 506-512.

Graves, A.W., Valles, E.C., & Rueda, R. (2000). Variations in interactive writing instruction: A study in four bilingual special education settings. Learning Disabilities Research

and Practice, 15(1), 1-9.

Graves, D.H. (1985). All children can write. Learning Disabilities

Focus, 1(1), 36-43.

Hall, J.K., Salas, B., & Grimes, A.E. (1999). Evaluating and

improving written expression: A practical guide for teachers

(3rd ed.). Austin, TX: ProEd.

Hardin, S., & Miracola, F. (2003). Write more, more often and with better results. Closing the Gap, 21(5), 1, 10-11. Icke, N., Perella, R., & Temple, C. (2001). You can talk to your

computer, but will it listen? Closing the Gap, 20(3), 1, 20-21. Koester, H.H., & Levine, S.P. (1994). Learning and performance of able-bodied individuals using scanning systems with and without word prediction. Assistive Technology, 6, 42-53.

Laine, C., & Sitko, M. (2001). Using computer assisted writing in inclusive settings. Closing the Gap, 20(1), 1, 12, 31. Lemaire, B., & Dessus, P. (2001). A system to assess the semantic

content of student essays. Journal of Educational Computing

Research, 24, 305-320.

Lewis, R.B., Ashton, T.M., Haapa, B., Kieley, C.L., & Fieldon, C. (1999). Improving the writing skills of students with learning disabilities: Are word processors with spelling and grammar checkers useful? Learning Disabilities: A

Multidisciplinary Journal, 9(3), 87-98.

MacArthur, C.A. (1996). Using technology to enhance the writing processes of students with learning disabilities.

Journal of Learning Disabilities, 29(4), 344-354.

MacArthur, C.A. (1996). From illegible to understandable: How word prediction and speech synthesis can help. Teaching

Exceptional Children, 30(6), 66-71.

MacArthur, C.A. (2000). New tools for writing: Assistive technology for students with writing difficulties. Topics in

Language Disorders, 20(4), 85-100.

MacArthur, C.A., Graham, S., Haynes, J.A., & De La Paz, S. (1996). Spelling checkers and students with learning disabilities: Performance comparisons and impact on spelling. Journal of Special Education, 30(1), 35-57. Montague, M. (1990). Computers, cognition, and writing

instruction. NY: State University of New York Press.

Montgomery, D.J., Karlan, G.R., & Couthino. M. (2001). The effectiveness of word processor spell checker programs to produce target words for misspellings generated by students with learning disabilities. Journal of Special Education

Technology, 16(2), 27-42.

Moroco, C., & Neuman, S. (1986). Word processors and the acquisition of writing strategies. Journal of Learning

Disabilities, 19, 243-247.

Newcomer, P.L., & Barenbaum, E.M. (1991). The written composing ability of children with learning disabilities: A review of the literature from 1980 to 1990. Journal of

Learning Disabilities, 24(10), 578-593.

Parker, R.I., Tindal, G., & Hasbrouck, J. (1991). Progress monitoring with objective measures of writing performance for students with mild disabilities. Exceptional Children,

58(1), 61-73.

Scardamalia, M., & Bereiter, C. (1986). Research on written expression. In M.C. Wittrock (Ed.), Handbook of Research

on Teaching (3rd ed.). NY: Macmillan.

Smith, R.O. (2000). Measuring assistive technology outcomes in education. Diagnostique, 25, 273-290.

Speck, B.W. (1998). Grading student writing: An annotated

bibliography. Westport, CT: Greenwood Press.

Tchudi, S. (1997). Alternatives to grading student writing. Urbana, IL: National Council of Teachers of English. Watkinson, J.T., & Lee, S.W. (1992). Curriculum-based measures

of written expression for learning-disabled and nondisabled students. Psychology in the Schools, 29, 184-191.