This section provides an overview of the three genres which are the focus of this thesis. These genres vary in terms of writer competency. Automatic summaries can be expected to contain the most errors with regard to writing quality. Academic writers comprise a mix, novice, expert researchers and non-native speakers of the language. In the case of science journalism, the authors are professional and trained writers and so the average writing quality is very high. Therefore these texts are good and deficient in different matters of quality. Below we detail which differences (related to quality) in writing are most noticeable in these genres and how predicting text quality in these genres is useful for applications.
2.3.1 System generated summaries
The desired qualities of a summary are that it should contain important content from the source text, be concise and well-written. Automatic generation of coherent text is a hard problem. Therefore, most automatic summarization systems do not generate completely new sentences based on the source content. Rather they extract full sentences from the source document and use them to compose summaries. This approach leads to several problems in the quality of the output text.
Firstly, since extraction of full sentences is done, individual sentences can have con- tent that is unnecessary given the context of the summary. Such unnecessary content could lead to verbose summaries. Sentence compression techniques [21,52, 76,103] aim to tackle this issue by shortening sentences while at the same time retaining important content and keeping the sentences grammatical. Further since the sentences come from different parts of a source document or from different documents (in the case of multi- document summarization), the ordering of sentences in a summary could also be prob- lematic. Adjacent sentences in extractive summaries often lack continuity and to this end, a number of techniques have been developed to help re-organize summary content or score its organization quality [6, 80, 123]. In addition, referring expressions in the ex- tracted sentences also need to be updated given the new context of these sentences in the summary. Some solutions proposed for this problem are automatic methods for revising
references to people [115] and also generic noun phrases [112] in summaries.
Some of these issues can be noticed in the example automatic summary (taken from the Document Understanding Conference datasets) below. The summary was produced from multiple source documents on the topic of a tunnel.
Another is digging from the French coast in what the tunnel builders call the largest civil engineering project now under way in the world. LONDON – A shadow is falling across the light at the British end of the Channel Tunnel. While the French are forging ahead with a high-speed rail link to their end of the tunnel, for example, state- owned British Rail is dragging. Britain and France were linked beneath the English Channel on Tuesday when workers used a two-inch probe to connect two halves of a
31-mile undersea rail tunnel, officials reported. On May6 the Queen and President Mitterrand will declare the Channel Tunnel open.
Therefore measures to evaluate the content and writing quality of summaries can have great use in development of summarization systems and their evaluation.
2.3.2 Academic writing
Clear writing is of great importance in academic publications. Consider the following abstract taken from a paper [60] which received the Best Paper Award at the HLT-NAACL conference in2010.
Coreference resolution is governed by syntactic, semantic, and discourse constraints. We present a generative, model-based approach in which each of these factors is mod- ularly encapsulated and learned in a primarily unsupervised manner. Our semantic representation first hypothesizes an underlying set of latent entity types, which gen- erate specific entities that in turn render individual mentions. By sharing lexical statistics at the level of abstract entity types, our model is able to substantially reduce semantic compatibility errors, resulting in the best results to date on the complete end-to-end coreference task.
The abstract has a clear structure going from introduction of the task and issues to the authors’ ideas and results. Understanding what factors contribute to a good structure for
academic writing can help in at least two tasks.
There is little training for novice researchers, students and non-native speakers on how to write research papers and how to convey technical content in a clear manner. Au- tomatic metrics to judge the quality of writing can provide feedback to such an audience. In addition, for novice students, it may also be helpful to suggest reading material such that the recommended papers not only convey the best ideas in the field but are also written well.
In addition to assessment, metrics for quality of academic writing would be quite use- ful for generation systems. Particularly there is interest in recent years on summarization of scientific articles [105, 130, 131]. So far, these studies have only focused on content selection from the papers. To create coherent and well-formed summaries text quality measures, also specific to this genre, are necessary.
There is fairly good spelling and grammar in most articles in this genre since some editing and review is done before publication. We expect that problems in organization and other discourse aspects and clarity of writing are the most relevant aspects to explore for quality prediction in this genre.
2.3.3 Science journalism
Science journalism has the most advanced writers. Here the authors are professional jour- nalists and in our corpus which we describe later in this thesis, the articles for assessment are taken from the New York Times newspaper. So these articles have high quality in general. Moreover, while most news related to events focus on presenting facts, science journalism is meant to explain and also entertain. Consider the following snippet taken from an article by David Quammen and which appeared in the Harper’s magazine.
One morning early last winter a small item appeared in my local newspaper announcing the birth of an extraordinary animal. A team of researchers at Texas A&M University had suc- ceeded in cloning a whitetail deer. Never done before. The fawn, known as Dewey, was developing normally and seemed to be healthy. He had no mother, just a surrogate who had carried his fetus to term. He had no father, just a “donor” of all his chromosomes. He was the genetic duplicate of a certain trophy buck out of south Texas whose skin cells had been
cultured in a laboratory. One of those cells furnished a nucleus that, transplanted and rejig- gered, became the DNA core of an egg cell, which became an embryo, which in time became Dewey. So he was wildlife, in a sense, and in another sense elaborately synthetic. This is the sort of news, quirky but epochal, that can cause a person with a mouthful of toast to pause and marvel. What a dumb idea, I marveled.
The passage provides much detail about the research. But it is also written in a clever story-like manner. Since writers in this genre employ many different techniques to create engaging articles, this genre presents the opportunity to examine which properties of writing are better at captivating reader interest. Among the six traits, we expect that there is most variation in the aspects related to creative writing style—the voice, sentence fluency and word choice traits.
Automatic measures to identify interesting articles can be useful in search and rec- ommendation applications mainly. In addition, educational settings can also benefit from such metrics. For example, a high school teacher can use such metrics to select well- written science journalism articles to supplement readings from text books.