Document Release

After the SAC of historic photographs had been used for three years, theIAPR TC- 12 image collection provided a novel database for evaluation in 2006. Unlike many existing photographic collections used to evaluate VIR systems, this collection is very generic in content, with many different images of similar visual content but varying illumination, viewing angle and background. This makes it a challenge for the successful application of techniques involving visual analysis (see Chapter 4).

Document Access and Distribution

Only registered participants were granted access to the entire document collection of 20,000 photographs (and 20,000 corresponding thumbnails) on 15 March 2006. In addition, using the image collection management system presented in Section 6.3, we exported the semantic descriptions from the database and generated text files in English and German. The resulting corpus of 80,000 files was organised into one single archive as described in Section 4.2.6 and was subsequently made available for download to the registered participants.

Figure 7.2 provides an example of the English representation for image 16019.jpg. We used the following parameters to generate the text files for ImageCLEFphoto 2006: representation type, format, language, completeness and the level of orthog- raphy (these setting correspond to the ones shown in Figure 6.11).

Representation Type and Format

As far as the type and format of the logical image representations are concerned, we decided to create semantic descriptions using a similar format to that of the SAC (see Section 3.2.2) used in previous years: multilingual text representations in a semi-structured format.

Thus, similar to the SAC, the entire representation was nested between the <DOC> and </DOC> tags, and the <DOCNO> tag contained the pathname of the text file as a unique document identifier, while the title, description, notes, location and date fields were represented by the<TITLE>,<DESCRIPTION>,<NOTES>,<LOCATION> and <DATE> tags. In addition, the <IMAGE> and <THUMBNAIL> tags contained the path of the actual image file and its corresponding thumbnail respectively (see Figure 7.2).

The choice for this format had been based on the following two reasons:

• using an SGML format to encapsulate the individual fields would guarantee a high level of compatibility with existing TREC collections, and

• a similar representation format as used with SAC would offer a smooth tran- sition for our participants from the previously used SAC to the IAPR TC-12 Image Benchmark (e.g.to keep changes in existing retrieval scripts to a min- imum).

Representation Language

Unlike in previous years where only English representations were offered to partici- pants, we provided an additional language to the participants ofImageCLEFphoto 2006 by offering a German version of the representations as well. Figure 7.3 displays

Figure 7.3: The generated German caption for image 16019.jpg.

an example of the German representation for the image16019.jpg, whereby only the content of the tags is translated, while the tags themselves remain in their orig- inal English versions. Having two sets of representations in different languages is beneficial for a multilingual evaluation environment such asImageCLEF as it allows for the creation of many interesting retrieval and evaluation scenarios, including:

• the comparison of English and German monolingual retrieval;

• the comparison of translation directions (i.e. does English retrieval from Ger- man documents perform better than German retrieval from English docu- ments?);

• the evaluation of translation resources for third languages (e.g. the compari- son of retrieval performance based on Spanish-to-English against Spanish-to- German translations);

• the investigation whether combined retrieval from both collections would out- perform the results based on monolingual retrieval.

We did not offer the Spanish versions of the logical image representations as they were still being verified and were not in a release status yet.

Representation Completeness

Since consistent and careful semantic descriptions of images are typically not found in practice, we decided to create a more realistic scenario for participants by releas- ing a subset of the collection with a varying degree of representation “completeness” (i.e. with different representation fields available for indexing and retrieval). Thus forImageCLEFphoto 2006, we generated a subset that covered the following levels of completeness:

• 70% of the semantic descriptions contained title, description, notes, location and date;

• 10% of the semantic descriptions contained title, location and date;

• 10% of the semantic descriptions contained location and date; and

• 10% of the images were not annotated (or had empty tags respectively).

This distribution of representation completeness would allow for the subsequent analysis of whether more visual approaches would improve the retrieval results for topics that predominately target images with incomplete textual representations.

Orthography

We did not make use of the possibility of an additional orthographic challenge by further injecting spelling mistakes or typographical errors into the logical image representations. Although one might argue that this would reflect realistic data found in generic photographic collections (especially in private ones), the main goal of ImageCLEFphoto 2006 was to evaluate systems by their ability to retrieve relevant images, and not by the ability of detecting and correcting misspelled words. We therefore set the level of orthography to 100% (i.e. no typographical errors introduced) during the generation of the text files.