Structure and Methodology

We surveyed fifty Open Data repositories operated by national, regional and municipal gov- ernments, as well as international organizations. We based our list of repositories on a catalog published by the Open Knowledge Foundation1_{. Then, we expanded and refined this list using} the following steps:

1. We diversified the list by adding more repositories from all parts of the world, including repositories from all five continents.

2. We took care to add regional and municipal repositories in addition to national initiatives. 3. We included both official, i.e., state- or agency sponsored platforms, as well as community-

driven efforts.

4. We removed very small or inactive repositories, except for countries for which we could not find other repositories.

The full list can be found in the appendix and on the survey website2_{. It forms the basis of our} global assessment of open data repositories, which is presented in Section 4.1.1. In addition to that, a subset of five repositories was examined in more detail, which means for those we include further statistics that require more manual effort to collect. The results of this detailed analysis are presented in Section 4.1.1.

Global View

For the first part of the survey, we studied features of open data platforms that can be measured by just browsing the page or by writing a simple crawler for platforms that do not provide statistics on their content. More complex features that require downloading and analyzing the available datasets were studied for a limited number of platforms in the second part of our survey (see Section 4.1.1).

The goal of the global survey is to assess each repository’s suitability for automatic reuse of the data in general. The specific features that we surveyed will be described below. If not stated otherwise, the feature values were measured manually.

Number of published datasets Almost all platforms have the notion of a dataset: an inde-

pendent unit of data, describing one special topic or aspect of the world. One of the basic questions for an Open Data repository is: does it provide any interesting datasets? Although

1_{http://lod2.okfn.org/eu-data-catalogues/}

the absolute number of datasets can easily be skewed, a repository with more datasets is still more likely to useful data. Many sites provide the number of published datasets, and for all others, we measured by writing a simple Web crawlers to count them.

Standardized file formats Offering data in standardized file formats makes reuse much eas-

ier because it is immediately apparent from the metadata how to process the published files. In contrast, platforms allowing upload of any file format will always require manual processing to enable reuse of the data sets. We measured whether the platform regulates file formats.

Standardized metadata attributes Many platforms provide the option of adding metadata to

datasets. However, for automatic processing of the datasets, they should have a set of standard- ized attributes, which can be looked up for any dataset. Special cases of metadata attributes that we survey separately are the existence of domains, i.e., some form of dataset grouping, which is help for retrieval and discovery of related datasets. Furthermore, we examined whether the platforms offer standardized temporal and spatial metadata attributes. These are especially useful, as the majority of datasets only describe specific geographic entities and/or time inter- vals. Reuse of these datasets is facilitated when the respective spatial or temporal information is provided as metadata.

API This should be an obvious feature of every Open Data platform. Without an API, the

platform can only be accessed manually through its Web interface, preventing automated dis- covery and reuse. Furthermore, we differentiated between APIs that offers access to metadata, while keeping the data in downloadable files, and those providing direct access to the data through the API. A uniform data access saves users the effort of building parsers for individual datasets. This usually requires that the raw data is stored in a database management system.

Curation A curated platform is not necessarily better than a public editing platform. We

included this feature in the study to differentiate between platforms run by a governmental agency, which are usually curated, and platforms driven by an interest group, which usually allow unlimited dataset upload.

For every platform, we also recorded the country of origin, the administrative level it rep- resents (country, region, city, etc.), as well as policies regarding licensing, up-to-dateness and provenance.

Detailed Analysis

In addition to the global analysis, five repositories where surveyed in more detail. This de- tailed analysis was performed for the data.gov (US), opendata.ke.gov (Kenya) and data.gov.uk (UK) repositories as well as the global repositories data.worldbank.org of the Worldbank and data.un.orgof the United Nations. For this analysis, we surveyed additional features that re- quire more manual effort to gather.

Downloadable Datasets The fraction of the available datasets where we succeeded in down-

Country Repository Number Of Datasets Last Visited Date Down- loadable datasets Machine- readable datasets Avg. numbe of tags Tagged datasets Length of description Datasets with description

United Kingdom data.gov.uk 7.439 07.09.11 79.9% 42.20% 5.9 93.1% 349 92.4% United States data.gov 4.941 05.10.11 78.6% 77% 22.2 99.6% 381 99.8% Kenya opendata.go.ke 4.53 06.10.11 100.0% 100% 1.8 70.9% 67 80.8% Worldbank data.worldbank.org 5.500 04.11.11 100.0% 100% 0.0 0.0% 145 57.5% UN data.un.org 5.413 04.11.11 100.0% 100% 0.0 0.0% 0 0.0%

Table 4.1: Results of the detailed repository analysis

not include working downloads: many of the provided links were dead, returning HTTP error codes or timing out on request. Other links led not to the data, but to HTML documents con- taining links to the data, or sometimes just the home page of the organization distributing the data. In both cases, manual browsing is necessary to obtain the actual data. Of course, this prevents automatic retrieval and thus automatic search and analysis on the data. We measured this feature by obtaining all available download links from the respective platform’s API and following them. If there was a valid HTTP response, and the content was not in HTML format, we counted the dataset as downloadable.

Machine-readable Datasets The fraction of the downloadable datasets where we succeeded

in opening the downloaded files using standard, of-the-shelf parsers for the respective file for- mats. In our research on Open Data, we quickly noticed that a great share of the datasets are in obscure or proprietary formats, or not in the format given in the metadata. Obviously, this pre- vents automatic searching and processing of the data. We measured this feature by looking at the file type specified in the metadata and applying a standard parser for the respective format. If the file could not be parsed using this method, we counted it as not machine-readable.

Tags and Description Almost all surveyed platforms use tags to make datasets discoverable

and searchable. However, tags are only a useful instrument of navigation if they are applied to all datasets, in an adequate number and without repeating generic tags for many datasets. We counted the average number of individual tags per dataset. Furthermore, all surveyed plat- forms offer some textual information describing the dataset. We counted the average number of characters in the description of a dataset.

An overview of the results of the detailed survey are shown in Table 4.1, while the complete results, global and detailed, can be found at the survey website3. The next section will interpret and discuss the results, and derive a classification of Open Data Platforms. We will discuss the properties of these classes and support them using our data.