Testing the checker’s effectiveness

Empirical examination of the black box that is the Word 2013 grammar checker begins by considering its function: what goes into this error-detecting machine and what comes out. The two most through research models for discovering what comes out of the box – that is,

finding out how well the checker works – appear in the studies by Caroline Haist (on Word 97) and by David Major (on Word 2007). A look at what went into the box – the data sets they used to test its effectiveness – reveals both the importance and the limitations of such research. The content of their data sets, especially Major’s, has also provoked some of the broader theoretical questions about error and implications about register that shaped the conclusions of this dissertation. Both Haist and Major began their tests of the checker from pedagogical questions about its effectiveness, wanting to know how effectively and reliably it performed its error- detecting tasks, so that they would know whether and how to recommend its use to their students.

Models of testing methods

Because the checker cannot detect error across sentence boundaries, the test data that both Haist and Major used were banks of individual sentences. Not only were these sentences like those a traditional grammar book would use to test a student’s ability to find and edit errors: they were, in fact, just such sentences. Both researchers drew primarily from exercises designed to test for certain error types, such as agreement, fragments, or case. The data was therefore effectively “tagged,” analogous to the corpus linguistics sense of having descriptive information already attached to the data – labeling which any researcher would need to perform on any test sentences in order to compare the checker’s results against known error types, if it were not already thus labeled. Fortunately, the criticism that such drill exercises are mind-numbing to students is not a concern with a computer. Indeed, one might argue that such drills attempt to turn students into mindless error-detecting machines. Since the grammar

checker is already a mindless error-detecting machine, sentence sets from grammar book exercises, grouped by error type, are perfectly well-suited to the task of testing the checker’s abilities.

There are other ways to test the checker, and Major, like Vernon, also evaluates the checker with a small corpus of student essays. But it is difficult to control for the multitude of factors that shape error occurrence in such naturally occurring texts, which usually cannot control for the multitude of relevant variables in a piece of writing and which require extensive tagging, if one is to determine how successfully the checker detects specific error types, as was the goal for Haist, for Major, and here in this study. I have replicated only the sentence-

delineated testing in my study.

Both Haist and Major ran sets of sentences through the checker and counted the frequency with which it returned accurate and inaccurate flags of errors. Running the data involves pasting the sentences into a Word document, making sure that the checker is set to start checking in all or certain of the categories it offers, and then collecting the quantitative and qualitative information on what the checker marks as an error and what feedback it offers on those errors. (The technical steps for reaching the checker menu are presented in Chapter 6 which analyzes its text.)

The data banks

Both Haist and Major have been kind enough to share the data sets of sentences they collected to test the checker – Haist’s with the permission of Canadore College, which houses the data bank used in her study. This sharing has created a unique opportunity for this study

to compare results across several iterations of Microsoft Word, current and historical. These data sets also provide a continuity linking the results of composition research on grammar checkers across decades and set a precedent for future research, of sharing data used to test technology, to see if other scholars can replicate results. (Ideally, we will develop shared data banks of sentences or tagged corpora – sturdy, reusable, customizable test instruments – that can evaluate new software in comparison with existing technology, at any point going forward.)

The ability to see these two sets of data allows for an evaluation of their similarities and differences, which determines how well the statistics from Haist and Major present an apples- to-apples comparison between Word 97 and Word 2007 with my new results, on Word 2013. I was also able to reuse their data for my tests, further strengthening the comparison. A look at the nature of the errors in the sentences reveals apples-to-oranges differences between their data banks that may explain some of the differences in their quantitative findings.

Limitations and differences in the test banks, testing categories, and methods

Major’s data is stored in plain-text documents that can be opened as Word documents. Unfortunately, because Haist used her more than 2000 data sentences for other research purposes as well as for testing the checker, they were last formatted for use in databases, and this only available version of the sentences buried them in a great deal of unrelated text and formatting intended for the machine. To be usable, the data would have required a copious amount of cleaning and processing that were beyond the scope of this research. I therefore chose to use Major’s sentence banks for testing Word 2013 for this round of research, and fuller use of Haist’s data must await future study. I was able to make use of a subset of her sentences

(which included many that Microsoft itself had used as examples in the program) – those that she included as illustrations in her final report – and they led to some curious discoveries.

Both Haist and Major established their own categories of errors to test, rather than simply using categories delineated by the item names on the checker menu. This choice makes sense, not least because it is not clear from some items names such as Noun Forms or Punctuation

– style suggestions what errors they check for (as explained in detail in Chapters 6 and 7). The

categories in the checker have also changed slightly over the versions. Due to the vagueness of some item labels, it is unclear whether the added categories have represented new checking abilities or simply subdivided existing abilities formerly grouped together on the menu. In the spirit of critical engagement rather than passive acceptance of the software’s choices, a writing teacher is likely to have her own areas of concern about student error and be curious to know whether the checker can be helpful in those categories, rather than simply allowing the checker to set the agenda on errors. Some of the test-bank categories align with item names in the checker (Subject-verb agreement) while others do not (e.g., Major tested for “Apostrophes” as a category, for which there is no specific item by that name on the checker menu). While Haist’s and Major’s lists of categories were not identical, some were similar enough to allow for nearly direct comparison, between the two of them as well as with my new results, so these were the categories chosen for this study.

For the straightforward quantifications required in this type of study, Major makes his method clear about one quantity that can skew results: what to do with false positives. In presenting his findings, he creates a separate table column for offering the percentage that includes these data, by subtracting it from the total score (leaving the checker with a negative

success score at accurately detecting some error types). Haist does not include a description of how she treated false positives in her results, and her research was long ago, such that she cannot recall how she incorporated this aspect of the data. Because false positives are one of the most-noted annoyances of the checker, the absence of this information is a loss. Also, Haist includes a comprehensive percentage total for many but not all categories of error types, and her methodology for choosing which to include is not defined. I was able to include data on false positives in the subset of sentences that I collected directly from the examples sentences in Haist’s report.

Finally, one significant difference between the two earlier studies does create some unmeasurable limitations on the apples-to-apples comparability of their results: Major chose to test only the items listed in the Grammar section of the checker menu, unchecking all the items on the Style menu. Haist tested both, and she discusses in her findings how she experimented with checking and unchecking various items on the menu, from both sections, noting how the change affected the checker’s ability to catch certain types of error (2). Major was seeking to determine how effectively the checker detected true, severe, alarm-bell errors, not ostensibly optional style choices. For users, this distinction can quickly become caught in the bog of vocabulary discussed in Chapter 3, on grammar and style, regarding what counts as an error and how we use these labels. And as Chapter 6 and 7 will discuss, some of the items listed under Style, some readers (and writers) would consider true errors. For Haist, the items still checked in the menu when using the “Casual” style option in Word 97 failed to catch significant errors such as “fragments, run-ons, and errors in possessives and plurals” (2). But Major’s choice not to leave any Style items selected in Word 2007 disallows the possibility of comparing

with Haist or with new data in certain categories, which I have thus chosen not to test. Following from Haist’s discoveries, it is also possible that activating the items in the Style section might have improved the checker’s function in the categories of error, or “grammar,” that Major did test using only the Grammar section items, but there would be no way to know without re-running the data in Word 2007 with those items checked – not a worthwhile

endeavor for the purposes of this study. Notwithstanding Major’s choice to deactivate the Style items and with only one exception, the overall arc in the equivalent categories from Haist’s to Major’s results and then to mine did not show a degradation of measurable effectiveness from Word 97 to Word 2007 to Word 2013,

One inevitable dimension of difference between the two data sets appears at the level of individual errors and will be discussed as part of the findings and conclusions in this chapter, as it interweaves with this study’s broader conclusions about error pedagogy, register, and research.

Given how little the Microsoft grammar checker has changed from Word 97 through Word 2013, compositionists desiring further detail on the program’s functions and effectiveness would do well to read both Haist’s and Major’s studies. Each provides not only quantitative data on the checker but detailed insight into the checker’s functions and recommendations relevant to pedagogy.

Methods

For this testing of the checker’s effectiveness, I aligned my methods as closely as possible with the earlier researchers’, for consistency and thus comparability of results. I chose to test in

the seven categories which Major tested and for most of which Haist also offered summary percentages in the same categories. (In one group, Run-ons and Comma Splices, she combines two categories that Major keeps separate, so I have combined both into one category, which I label Comma Splices in my graph, but also considered the subsets. And though she ran many tests on various pronoun types, she does not offer a summary percentage for Pronoun Case. Also, her reported figure on apostrophes included all possessives but not other apostrophe use, while Major tested apostrophes but focused on their use in possessives, so this error type is nearly but not exactly identical across the two.) I used Major’s test data, grouped in individual documents under the labels he uses in his report, each of the seven documents holding between 100 and 116 errors, some sentences containing more than one error. I selected for the checker to activate all 35 items on its menu while checking.

To code the data, I counted the number of times that the checker accurately flagged the mistakes in question and the number of missed mistakes. Notably, as discussed in my findings, there were few false flags. There were numerous instances in which the checker marked an error accurately but labeled it inaccurately: I did not distinguish between these two checker response types in quantifying errors marked, for reasons I discuss in my conclusions to this chapter. I calculated the number of detected errors in each of the tested categories as a

percentage of the total number of errors of that error type; these percentages are listed in the bar graph in Figure 2, compared with Haist’s and Major’s percentages.

I also conducted a study of the 123 test sentences that Haist included as examples in her report on the checker, which she had identified by error type and as to whether the checker had

succeeded or failed in detecting them. I tested the sentences using the same method as used for Major’s data, and the graph of findings in Figure 3 compares my results to Haist’s.