Underestimated flaws and recent advances in discovering them

2.2.1.1 Introduction to text quality issues

Text quality variation throughout the corpus can be a consequence of corpus design, of doc-ument collection processes, and/or preprocessing. For a number of reasons described in the paragraphs below, and despite ongoing work on the issue, there are many different subsequent questions to address. They concern text quality, entailing much work, and more precisely further interdisciplinary work.

In fact, quality does not only fall within the field of computational linguistics. Other disci-plines such as information retrieval have seen a similar evolution, where problems afferent to text quantity such as document gathering on a large scale as well as scalability of procedures can be considered (at least) as partly solved. Attention is now turned towards text quality.

In information retrieval particularly, there seems to be a void concerning web document quality analysis:

“Many existing retrieval approaches do not take into account the content quality of the retrieved documents, although link-based measures such as PageRank are commonly used as a form of document prior." (Bendersky, Croft, & Diao, 2011)

Because of particular affinities for graph-based approaches in computer science, link-based indicators have been favored as a matter of fact. But even in the case of documents returned as they are to the end user, i.e. without annotation or linguistic processing, discrimination based on quality metrics are gaining in popularity.

The question whether bigger data is better data where it concerns corpus linguistics is discussed in the previous chapter (see p. 35). The remainder of this section turns to specific examples of text quality issues in natural language processing and beyond.

2.2.1.2 Examples

In order to get a glimpse of the phenomena involved in text quality assessment, a few ex-amples of the difficulties are listed below. They are grouped into four different kind of issues:

automatically generated text, machine-translated text, human-induced spam, and multiple lan-guages. The list is not supposed to be exhaustive, it rather summarizes a few salient problems for which studies have been undertaken. The tasks are described in section 2.2.1.3, while the current approaches in several research fields are evoked in section 2.2.2.

Automatically generated text

The following kind of repeated text¹is probably induced by the crawler, which because of its technical specifications hinders a completely dynamic rendering of a given web page. A human user using a state-of-the-art browser would possibly see normal text injected from another source at this particular point of the page:

❚❤❡r❡ ✇❛s ❛♥ ❡rr♦r s❡♥❞✐♥❣ t❤❡ ♠❡ss❛❣❡✱ ♣❧❡❛s❡ tr② ❛❣❛✐♥ ❧❛t❡r✳

❞❡s❝r✐♣t✐♦♥ ♥♦t ❛✈❛✐❧❛❜❧❡ ✳✳✳

❚❤❡r❡ ✇❛s ❛♥ ❡rr♦r s❡♥❞✐♥❣ t❤❡ ♠❡ss❛❣❡✱ ♣❧❡❛s❡ tr② ❛❣❛✐♥ ❧❛t❡r✳

It is a basic but frequent kind of machine-generated text. Such sentences or paragraphs very often contain no useful information, and because of their repetitive nature, all the more on a scale of a whole website, there are clearly unwanted in the case of a corpus for linguistic studies.

Machine-translated text

Machine-translated text is sometimes hard to distinguish from a low-level of proficiency

1Extracted from test data analyzed in (Schäfer, Barbaresi, & Bildhauer, 2013)

❤tt♣✿✴✴✇✇✇✳❝❛r♦❝❡❛♥✳❝♦✳✉❦✴❢♦r✲s❛❧❡✲❘❡♥❛✉❧t✰❈♦❧❝❤❡st❡r✳❤t♠❧

by a speaker or even from automatic content templates. However, the example below makes clear how different it is from normal utterances in a target language. On one hand, here is an occurrence of what is called by Arase and Zhou (2013) the phrase salad phenomenon:

❖❢ s✉r♣r✐s❡ ✇❛s ✉♣ ❢♦r❡✐❣♥❡rs ✢♦❝❦❡❞ ♦✈❡rs❡❛s ❛s ✇❡❧❧✱ t❤❡②

♣✉❜❧✐❝✐③❡❞ ♥♦t ♦♥❧② ❏❛♣❛♥✱ s❛✇ ❛♥ ❛rt✐❝❧❡ ❢r♦♠ t❤❡ ♥❡✇s✳

On the other hand, here is how the authors translate it into natural English:

“The news was broadcasted not only in Japan but also overseas, and it sur-prised foreigners who read the article."²

Since the example concerns the pair Japanese-English, which is notably more difficult to translate than translations within the same language family for instance, the “falseness"

of the result is particularly striking. It is no rare case however, since for various reasons machine-translated content now belongs to the very essence of a user experience (see p. 76 for a discussion).

Human-induced spam

Real spam is sometimes more tricky to identify than generated or machine-translated content, of which it can be a subcategory, as it is mostly the result of a human intervention. One of the most salient cases of spam is found in vague and elusive blog comments, which fit all cases and thus can be posted nearly everywhere, in order to advertise a product or simply to point links to a certain website, as was probably the case in these two examples³:

• ❚❤✐s ✐s ❝❡rt❛✐♥❧② ❛ ❛♠❛③✐♥❣ ❛rt✐❝❧❡✳ ❚❤❛♥❦s ❛ ❧♦t ❢♦r ♠❛❦✐♥❣ t❤❡

❡✛♦rt t♦ ❡①♣❧❛✐♥ t❤✐s ❛❧❧ ♦✉t ❢♦r ✉s✳ ■t ✐s ❛ ❣r❡❛t ❤❡❧♣✦

• ■t ✐s ❛ ❢❛♥t❛st✐❝ ♣♦st✳ ■ ✇✐❧❧ ❜❡ s♦ t❤r✐❧❧❡❞ t❤❡ ✇❡❜ ✐s st✐❧❧

❡q✉✐♣♣❡❞ ✇✐t❤ ✇♦♥❞❡r❢✉❧ ❝♦♥t❡♥t ♠❛t❡r✐❛❧✳

The potential damage in terms of corpus occurrences is lower than in the other types of issues. Nonetheless, a web-scale corpus is bound to gather numerous examples of such sentences, which efficiently distort the view on language offered by the corpus.

Multiple languages

The following three comments⁴were found just one after the other on a single web page.

They are most probably blog comments following the same principles as human-induced spam mentioned just above.

❉❛s ●♠❜❍✲❍❛✉s st❡❤t ■❤♥❡♥ s♦✇♦❤❧ ❢ür ❞✐❡ ❜ür♦❦r❛t✐s❝❤❡ ❆❜✇✐❝❦❧✉♥❣ ✈♦♥

■❤r❡♠ ●❡s❡❧❧s❝❤❛❢ts❦❛✉❢ ♦❞❡r ❯♥t❡r♥❡❤♠❡♥s❦❛✉❢ ③✉r ❱❡r❢ü❣✉♥❣ ❛❧s ❛✉❝❤

❜❡✐ ❞❡r ❊t❛❜❧✐❡r✉♥❣ ✈❡rs❝❤✐❡❞❡♥❡r ❋✐r♠❡♥♠ä♥t❡❧✳ ❏❡❞❡r ❋✐r♠❡♥♠❛♥t❡❧

2Source: (Arase & Zhou, 2013, p. 1599), see below for a more detailed analysis

3Both comments extracted from test data analyzed in (Schäfer et al., 2013)

4Extracted from test data analyzed in (Schäfer et al., 2013)

❤tt♣✿✴✴❧❡❛r♥❛♥❛t♦♠②❛♥❞♣❤②s✐♦❧♦❣②✳❝♦✳✉❦✴❛❜♦✉t✲✷❄r❡♣❧②t♦❝♦♠❂✷✸✺✸✸

✭③✳ ❇✳ ❞❡r ●♠❜❍ ▼❛♥t❡❧ ♦❞❡r ❞❡r ❆● ▼❛♥t❡❧✮ ❜✐r❣t ✉♥t❡rs❝❤✐❡❞❧✐❝❤❡

st❡✉❡r❧✐❝❤❡ ❱♦rt❡✐❧❡✳ ●❡r♥❡ ❡r❧ä✉t❡r♥ ✇✐r ■❤♥❡♥ ❞✐❡ st❡✉❡r❧✐❝❤❡♥

▼ö❣❧✐❝❤❦❡✐t❡♥ ❜❡✐ ❡✐♥❡♠ ▼❛♥t❡❧❦❛✉❢✳

■✬♠ t❤✉s s✉❝❝❡ss❢✉❧ t♦ ♦✇♥ t❤✐♥❦ ♦❢ t❤✐s s✐t❡✳ ❆♥ ✐♥❞✐✈✐❞✉❛❧

♣r❛❝t✐❝❛❧❧② ❞❡❝❧❛r❡❞ ♠❡ ❥✉st ✇❤❛t ▼② ♣❛rt♥❡r ❛♥❞ ✐ ♦♣t❡❞ t♦ ❜❡ ❛❜❧❡

t♦ t❛❦❡ ♥♦t❡ t♦ ❜❡ ❛❜❧❡ t♦ ❛♥❞ ❛❧s♦ ❛❢t❡r✇❛r❞ ❛ ❧♦t ♦❢✳ ❆♠❛③✐♥❣

♣✉❜❧✐s❤✐♥❣ ❛♥❞ ❛❧s♦ ❛❧❧ t❤❡ ❜❡st ❛❣❛✐♥ r❡❣❛r❞✐♥❣ ❛❝❝♦♠♣❧✐s❤✐♥❣

t❤❡ ❢♦❧❧♦✇✐♥❣ s✐♠♣❧② ♥♦ ❢❡❡✦

▲❛✈♦r♦ ♣❡r ✐❧ ❞♦♠❛♥✐ ✲ ✉♥♦ s❣✉❛r❞♦ ❞✐ ❨❡st✉r❞❛②s ❛❞ ❛❧❝✉♥✐ ❡s❡♠♣✐

Obviously, they are written in three different languages (respectively German, English and Italian). In fact, each paragraph taken apart is perfectly sound, although the English and Italian examples are not as structured as the German one, but the presence of all three on the same web page typically lowers one’s expectations regarding content quality.

Even without taking a closer look at them one may think they are most probably spam, and as such text parts that ought to be deleted.

However, this last type of difficulty is trickier, since its perception depends on the filtering level. On sentence or even paragraph level there is no problem at all, parts which do not correspond to the target language can be left out quite easily. On the web page level one does have a problem, since a series of heuristics have to be applied in order to systematically decide whether to keep the page or not. In such cases, the amount of text in the target language would probably be a productive criterium.

The issue of mixed-language documents is discussed in the next section, while experi-mental results on web document selection are treated further below (see chapter 4).

2.2.1.3 Discussion: Machine-generated/translated content, traps, and spam

Machine-generated content and traps Apart from the cases exemplified just above, machine-generated content may also serve the purpose of tricking other machines, especially crawlers, into falsely assessing a web page’s content or content quality, particularly for so-called “black hat" -i.e. potentially malicious- search engine optimization techniques. Deception mechanisms targeted at machines are called crawler traps. A frequent goal is to trick search engines into assigning a web page a higher rank than would otherwise have been the case, thus generating more clicks and more revenue.

“Another phenomenon that inflates the corpus without adding utility is crawler traps: Web sites that populate a large, possibly infinite URL space on that site with mechanically generated content. [...] Not much research has been published on algorithms or heuristics for detecting crawler traps directly." (Olston & Najork, 2010, p. 226-227)

Since web data is harvested automatically, one’s software can be expected to fall for crawler traps, which means that text filtering has to take it into account. Since crawler traps cannot be actively detected, like Olston and Najork (2010) state, their precise impact on the course of corpus construction and/or on the final content of a document collection is unknown. Together

with machine-generated content, crawler traps call for exhaustive filtering steps aiming at detection of duplicate content.

Machine-translated content The amount of machine-translated content on the Web varies by language. For high-density languages such as English, Japanese, and German, only a small percentage of web pages are generated by machine-translation systems.

According to Rarrick, Quirk, and Lewis (2011), among pages for which they identified a parallel document, at least 15% of the sentence pairs annotated for both English-German and English-Japanese appear to contain disfluent or inadequate translations. Still according to Rarrick et al. (2011), the amount of machine-translated content on the Web rises sharply for lower density languages such as Latvian, Lithuanian and Romanian. Latvian and Lithuanian had the highest percentages, with each over 50%.

The problem with this proportion of machine-translated content is twofold. On the one hand, it affects corpus construction directly because what linguists are after is content produced by real human speakers and not by machines. On the other hand, machine-generated texts do not appear to be flawless, which can impede corpus research at any level, thus requiring detection and filtering of such content:

“The quality of these machine-translated sentences is generally much lower than sentences generated by native speakers and professional translators. Therefore, a method to detect and filter such SMT results is desired to best make use of Web-mined data." (Arase & Zhou, 2013, p. 1597)

This makes the case of low-density languages even more complicated, in addition to these languages already needing special procedures (see p. 54).

All in all, machine-translated content is a major issue, as is text quality in general, especially when it comes to web texts (Arase & Zhou, 2013). Detection of machine-translated content has been proven to be efficient when developing a machine-translation system, so that it cannot be said in this particular case that “more data is better data" (see p. 35 for a discussion of this axiom):

“Trained on our filtered corpus, our most successful MT system outperformed one trained on the full, unfiltered corpus, thus challenging the conventional wisdom in natural language processing that ‘more data is better data’" (Rarrick et al., 2011, p. 1)

Mixed-language documents First of all, one may want to be sure that the text is mostly written in a given language. There are many Web documents that are mixed, first because the content comes from multiple sources, second because web servers adapt depending on geo-graphic information, and last because there is globally a majority of speakers who use several languages on a regular basis and who may switch between them. In this respect, European countries where speakers focus on a single language are usually an exception.

Mixed-language documents slow down text gathering processes (King & Abney, 2013), which is another thing particularly true for lower-density languages:

“We found that the majority of webpages that contain text in a minority language also contain text in other languages." (King & Abney, 2013, p. 1110)

The characteristic the authors describe seems to be inversely proportional to the popular-ity of a language, with a very high probabilpopular-ity to find monolingual documents for the most frequently spoken languages:

“If a language is not spoken widely enough, then there is little chance of finding any text in that language on the Web. Conversely if a language is too widely spoken, then it is difficult to find mixed-language pages for it." (King & Abney, 2013, p. 1112)

These findings give interesting insights on actual language use in everyday web experience, which is a potential interest of web corpora, and more generally data harvesting on a web scale, since it makes it possible to capture global trends which could before only be extrapolated from sample studies. The variations between widely spoken languages and others make the case of less-resourced languages (see p. 54) even more complicated.

Several forms of spam The main cause for spam are business models grounding on imi-tation, falsification, or generation of content. While scams and phishing are probably spam forms that are better known to most Internet users, from the point of a web crawler the most frequent form is related to search engine optimization techniques. In that case, web pages act as empty shells which are designed to generate “link juice" for others by building up a net of supposedly valid and influential websites. As long as the sites appear to be legitimate, their

“good reputation" in terms of link-based algorithms can be monetized.

“Web spam is motivated by the monetary value of achieving a prominent position in search-engine result pages." (Olston & Najork, 2010, p. 227)

Thus, the main problem with page ranking spam is not that these sites are numerous, but rather the fact that their content is just a mere addition of templates with little or no “real" text, i.e. no text which results from a natural utterance by a speaker.

The great majority of texts available on the web are natively digitized. As such they do not include potential digitalization flaws such as optical character recognition mistakes. Neverthe-less, idiosyncratic biases such as spam and diverse optimization techniques mentioned in this section show that general web content cannot be expected to be flawless. Therefore, web texts need to be scrutinized and carefully filtered and cleaned.