Assessing Documents

The general principles involved in handling documents as research data are not particularly different from other approaches in social research. However, the specific features of documentary sources require the consideration of their distinctive characteristics and particular techniques that are needed to handle them (Scott 1990). An important issue in documentary research is the interpretation and the evaluation of a document. Assessing a document and evaluating its usefulness for the aims of the research is one of the major tasks of the documentary researcher

(Macdonald 2001). Scott (1990) has identified four key criteria that a documentary researcher should use when selecting material. These four criteria might be regarded as the most commonly used ‘quality control criteria’ of a document (Platt 1981;

Wellington 2000; Macdonald 2001; Bryman 2001). They are: a) authenticity, b) credibility, c) representativeness, and d) meaning.

Authenticity is a fundamental criterion in documentary analysis and concerns its genuineness: whether it is actually what it claims to be. This should constitute a question in documentary research that the researcher should routinely address. An initial issue for the researcher is to examine whether the document is an original or a copy, and if a copy, whether it is a copy of the original or a copy of a copy.

Authentication of authorship, and assessment of the soundness of a document involves the use of both internal and external evidence. Internal evidence of vocabulary and literary style can be used to assess the coherence and consistency of a document, but this should also be complemented by such external evidence as the relation of factual claims to established facts, and the plausibility of the content as a reflection of the author’s known views. Coubertin’s and Diem’s writings are not

‘authentic’ in terms of vocabulary and literary style, as they constitute translated copies of the original documents. However, the external evidence provided in the documents is regarded as adequate for the purpose of the study, which assesses authors’ views in the broader political, socio-cultural and economic context.

Credibility refers to the extent to which the evidence is undistorted and sincere, free from error and evasion. Assessing the credibility of a document involves examination of the sincerity of the author’s view. The question of ‘sincerity’, therefore, is the question of whether the author actually believed what he or she recorded, and involves an assessment of the reasons why the author chose to produce this document. Langlois and Seignobos (1908; cited in Scott 1990: 22) have advocated a stance of ‘methodical distrust’, which means that “the researcher should distrust everything that is found in the documents unless there are good reasons to believe them”. For example, many official documents are based on a political interest in presenting one view rather than another, in transforming a kind of propaganda into

‘sincere’ information or in justifying a particular choice of action. In the case of personal documents the researcher must try to uncover any biases and prejudices that may have led the author to adopt a sympathetic or unsympathetic view in

relation to the people and events reported. As the aim of the research is to identify the process of the expression of values associated with Olympism, examining motives, interests and intentions in relation to the promotion of this ideology, assessing the credibility of the documents will provide further evidence of the analysis.

Representativeness involves a judgement as to whether the documents used are representative of the totality of relevant documents. The question of representativeness involves the two aspects of ‘survival’ and ‘availability’. Not all documents are deposited in places where they are likely to survive. Moreover, a great many documents are destroyed specifically in order to prevent their survival and availability to the public sphere. In order to decide whether a particular collection is representative of the totality of the documents, the researcher must have an idea of the number and type of relevant documents that might have been produced in the first place. Coubertin’s work comprises 30 books, 50 pamphlets, 1,300 articles and around thirty leaflets and posters, which together amount to around 15,000 printed pages. Previous estimates of 60,000 printed pages have been assessed as inaccurate, as many texts had been published several times, often under different titles (Müller 2000). Diem’s work comprises four editions about sport, and in particular about German sport, (Handbuch der Leibesübungen, 14 volumes, 1923-1930; Beiträtge zur Rurn-und Sportwissenschaft 1922-1929; Taschenbuch der Leibesübungen 1925-1930; and Jahrbuch der Leibesübungen, 30 volumes 1924-1932), and more than 2000 works on Olympism. Finally, 630 speeches were delivered at the IOA Sessions for young participants from 1961 to 1998.

For the selection of the IOA sample, theoretical sampling, a technique based primarily on the construction of a research protocol was used, which is described later in detail. The use of this technique facilitated the researcher’s selection of a representative sample of the total number of speeches in response to specific theoretical categories of the research protocol. However, in relation to Coubertin’s writings it was difficult to access the total number of documents, as they are stored in many different places, such as the archives of the Olympic Study Centre in Lausanne, the German Carl-Diem-Institut and different libraries in France as individual documents. Moreover, most of Diem’s documents are available only in German, which made it difficult for the researcher to access the total number of his

documents. In both cases, the use of published collections resolved the issue of accessing Coubertin’s and Diem’s documents, but it made it difficult for the researcher to claim ‘typicality’ of the sample, as the procedure of selection was undertaken by organisations (IOC and Carl-Diem-Institut respectively) which might have had a vested interest in producing this collection. However, Scott (1990) argues that good research can also be carried out with an unrepresentative selection, but the user must know to what extent and in what respects those documents are unrepresentative. Moreover, according to May (1997) the question of typicality depends on the aims of the research and for instance ‘untypicality’ may also be of interest. Absence of documents, which refer to authors’ actions and views that were later broadly criticised, may provide useful information about Olympic institutes’

motives and interests in relation to Olympic ideology.

Meaning concerns the understanding of the meaning and significance of what the document contains. The problem of meaning arises at two levels, the literal and the interpretative. The problem of the literal understanding of a text refers mostly to the texts from the remote past and the necessity to decipher the script and translate the language into the linguistic forms current in contemporary community. There may be problems of literal meaning in the translations of the works of Coubertin and Diem but these are only available to the author where controversies of translation are highlighted by other authors. Interpretative understanding, however, refers to all texts used for analysis, and is considered to be the ultimate purpose of examining documents. Interpretative understanding is the end-product of a hermeneutic process in which “the researcher relates the literal meanings to the contexts in which they were produced in order to assess the meaning of the text as a whole” (Langlois and Seignobos 1908; cited in Scott 1990: 30). The social context surrounding the document in question must be examined if the researcher wants to grasp the significance of the document itself even independently from the content (Wellington 1996). Similarly, Hodder (1994) argues that the context of the text is crucial in understanding its meaning, as a text can say many different things in different contexts. Moreover, past and present meanings of the same text should be questioned, as values and ideas are contested over time and anachronistic statements may distort the meaning of the text in its specific historical, political, and socio-cultural context.

In the process of searching for meaning, the study seeks what Altheide (1996) calls the ‘emergence of meaning’, which refers to the gradual shaping of meanings through understanding and interpretation of the documents themselves through the use of Ethnographic Content Analysis (ECA). This involves a constant comparison and investigation of documents over a period of time following a reflexive movement between the content of the document, the theoretical assumptions underpinning the study, and the broader historical context.