Restricted retrieval

4.2.3.1 Processing chain

URL-based (restricted) retrieval Although there are distinctions to be made between po-tential crawling strategies (see above), what I have called restricted retrieval here is roughly comparable to focused crawling.

In fact, it is a hybrid approach which benefits from manual intervention. The crawl does not merely follow the instructions of the robots.txt files (see above), it also uses a blacklist which can be defined manually, in order to avoid particular (unwanted) types of content, such as cooking recipes on a newspaper website or videos.

Example: extracting list of links If the website has an archive, a sitemap or a general list of its contents one can save time by picking the interesting links once and for all. Algorithm 4 describes a basic way to crawl a particular website.

Download en masse (archiving) or on-the-fly extraction (scraping) Once the list of target URLs is set, the download can begin. At that point, one may distinguish two main types of retrieval.

In the first, an en masse download of the target URLs and everything they link to within the same domain name is performed. This approach is very close to the one adopted by web archives, as so-called dumps or mirrors of a whole web site are made.

Crawldocumentquality

0.65 0.70 0.75 0.80 0.85 0.90

0.800.850.900.95

fr − dmoz

id − dmoz nl − dmoz

sv − dmoz

fr − etools

id − etools

nl − etools sv − etools

fr − friend

id − friend

nl − friend

sv − friend

fr − identica

id − identica nl − identica

sv − identica

fr − wiki

id − wiki

nl − wiki sv − wiki

Seed quality

Figure 4.4: LM for crawl document quality modeled by seed quality on short breadth-first crawls (R² = 0.79). Four different languages: Dutch (nl), French (fr), Indonesian (id), and Swedish (sv). Four different sources: DMOZ, etools (meta search engine), Friendfeed, and identi.ca. Source: joint work with F. Bildhauer and R. Schäfer.

There are several caveats to such an approach. Contrary to the procedure for web archives, not all the content is to be stored, but only relevant text, meaning a fraction. The download operation requires storage space, sometimes quite a lot, even though part of the downloaded material is not strictly necessary. Additionally, as duplicate content is frequent on the Web, it may be that documents have to be filtered in this respect, thus making the use of storage space even more questionable.

However, as the different filtering operations leading to the final text collection may change in the course of time, it could be useful to have a full archive to start from, in order to ensure reproducibility of filtering and make potential comparisons possible.

In the second case, content is extracted on the fly, which is sometimes named web scraping.

It is then impossible to come back to a “raw" version of the sourced web pages. That is why the scraping process has to be optimized before being used on a large scale, as results will not be comparable otherwise.

On-the-fly extraction: example of a page explorer The main scraping component is the module that indexes or in this particular case selects the desired content and stores it in a file.

The algorithm 5 describes a potential way to write an program to perform scraping.

The scraping approach is also feasible regarding web crawling by itself, meaning that one

Algorithm 4 Example of algorithm for restricted crawling while going through a shortlist of archives do

fetch page extract the links for each link do

if the link matches a given regular expression then store in a list

end if end for

if there are other results or archives pages then for all the available items do

add ?p=... or ?page=... or anything suitable to the last seen page fetch page, find the links, process each link

end for end if end while

remove the duplicate items from the list write to file (after a last control)

Algorithm 5 Example of algorithm for web scraping while going through the list of pages do

fetch page

cut the top and the bottom (e.g. HTML HEAD and unwanted information) if there is something like a <div class="title"> or <h1> then

extract it clean it store it end if

look for the paragraphs of the text, clean and store them end while

write the text with desired meta-information to file

may also scrape the links on the fly. It lacks the systematic approach and the clear overview which is provided by a precomputed URL list, but it is potentially efficient on every website.

Moreover, it is interesting to see how far one may go starting from one single page for each given website.

If there are several pages, one can either change the URL before the page is fetched (if there is a “text on one page" option) or one can proceed as described by the links.

Metadata extraction and completion Metadata extraction is part of boilerplate removal, since meta-information contained for instance in the header or in particular HTML tags should be separated from the actual content. At the same time, relevant information can be extracted inside and outside of the document, external sources can help to complete metadata extraction.

The URL itself is relevant, as well as download information such as time indications and details

in server communication.

Especially concerning news articles, the contents of a given website are prone to change daily if not hourly. For this reason it is advisable to explore the website bit by bit (or more precisely rubric by rubric) and/or on a regular basis, rather than to perform a full-depth crawl once only.

One might want to write a shell script to fire the two main phases automatically and/or to save the results on a regular basis. That way, if something goes wrong after a reasonable amount of explored pages not all the work is lost.

4.2.3.2 Examples of several approaches to retrieval

Example 1: Sources of political speeches The speeches were crawled from the online archive of the German Presidency (bundespraesident.de) and from the official website of the German Chancellery (bundesregierung.de). Strange as it seems, the resource is not stable, with texts ap-pearing or disapap-pearing for no obvious reason. Additionally, not all the speeches in the corpus can be found on this website anymore due to a change of design. Further details are mentioned in Barbaresi (n.d.).

This ordering was made using regular expressions in both titles and excerpts. It seemed to work properly but does not guarantee a perfect classification.

An automaton stripped off the salutatory addresses of the speeches using regular expres-sions, with good accuracy, although not perfect due to the extreme variation among speakers.

In the texts from before 2005 the encoding is deficient, mostly affecting the punctuation marks and the spaces. This is a typical issue for corpora from the web, which is still easier to deal with than OCR problems.

Example 2: Crawling of newspaper websites Starting from the front page or from a given list of links, the crawlers retrieve newspaper articles and gather new links to explore them as they go. The HTML code as well as superfluous text are stripped in order to save disk space, the remaining text (which is possibly the exact content of the article) is saved as a raw text file with relevant metadata (such as title, subtitle, excerpt, author, date and URL). Details are described in ? (?).

The crawlers also detect and filter undesirable content based on a URL analysis, as the URLs give precious hints about the article column or about website specifics and internal documents.

In fact, a few columns were discarded because most of the text they contained was of a different nature or genre, such as the cooking recipes of Die Zeit or the pictures of topless women on the website of Bild.

As the crawling process took place at a time when online versions of the newspapers were emerging, it was impacted by the editorial changes related to this evolution. The duplication of the articles (see below) as well as the existence of near duplicate documents are symptomatic for the erratic decisions taken by the editorial staff, leading for example to the parallel existence of online and print versions on the website of Die Zeit in 2008.

Example 3: OpenSubtitles The subtitles were retrieved from the OpenSubtitles project³⁰, a community-based web platform for the distribution of movie and video game subtitles. Further details are given in Barbaresi (2014b).

30❤tt♣✿✴✴♦♣❡♥s✉❜t✐t❧❡s✳♦r❣

The subtitle files were searched for using two different sources: sifting through the dumps provided by OpenSubtitles as well as querying the XMLRPC API systematically. The full meta-data were also retrieved using the XMLRPC interface for the texts classified as being in Ger-man. More details are given p. 204.

4.2.3.3 Between general web crawling and focused retrieval: content discovery of blogs in German under CC license

The problems to be solved in order to be able to build reliable CMC (computer-mediated com-munication) corpora are closely related to the ones encountered when dealing with general web corpora, described above. The purpose was to design an intelligent crawler targeting specific content types and platforms in order to allow for a fruitful website discovery.

Website discovery First of all, where does one find "German as spoken/written on the web"?

Does it even concretely exist or is it rather a continuum? Considering the ongoing shift from web as corpus to web for corpus, mostly due to an expanding web universe and the potential need for a better text quality, it is obvious that only a small portion of the German web space is to be explored.

Now, it is believed that the plausible distribution of links between hosts follows a power law (Biemann et al., 2013). By way of consequence, one may think of the web graph as a polynu-clear structure where the nuclei are quite dense and well-interlinked, with a vast, scattered periphery and probably not so many intermediate pages somewhere in-between. This struc-ture has a tremendous impact on certain crawling strategies. There are ways to analyze these phenomena and to cope with them (Barbaresi, 2014a), the problem is that there are probably different linguistic realities behind link distribution phenomena.³¹

Blog discovery on wordpress.com I chose a specific blogging software, WordPress, and tar-geted mostly its platform, because this solution compared favorably to other platforms and software in terms of blog number and interoperability. First,✇♦r❞♣r❡ss✳❝♦♠ contains poten-tially more than 1,350,000 blogs in German³². Second, extraction procedures on this website are transferable to a whole range of self-hosted websites using WordPress, allowing to reach various blogger profiles thanks to a comparable if not identical content structure.

The crawl of the wordpress.com website was prepared by regular visits of a tags homepage (❞❡✳✇♦r❞♣r❡ss✳❝♦♠✴t❛❣s✴) listing tags frequently used in German posts. Then, a crawl of the tag pages (such as❞❡✳✇♦r❞♣r❡ss✳❝♦♠✴t❛❣✴❣❡s❡❧❧s❝❤❛❢t✴) enabled us to collect blog URLs as well as further tags. The whole process was used repeatedly to find a total of 158,719 blogs.

The main advantage of this methodology is that it benefits from the robust architecture of wordpress.com, a leading blog platform, as content- and language-filtering are outsourced, which appears efficient.

The discrepancy between the advertised and actual number of blogs can be explained by the lack of incoming links or tags, by a substantial proportion of closed or restricted access blogs, and finally by the relatively short crawl of✇♦r❞♣r❡ss✳❝♦♠ due to politeness rules used.

31While these notions of web science may seem abstract, the centrality and weight of a website could be compared to the difference between the language variant of the public speaker of an organization, and the variants among its basis.

32❤tt♣✿✴✴✇♦r❞♣r❡ss✳❝♦♠✴st❛ts