Existing Documents and Artifacts as Data Barbara M Wildemuth
ANALYZING DATA OBTAINED FROM DOCUMENTS AND ARTIFACTS In speaking about using documents as a data source, Platt (1981) rightly comments
that documentary research “can hardly be regarded as constituting a method, since to say that one will use documents is to say nothing about how one will use them” (p. 31). Documents and physical trace data are so varied in their forms and meanings that methods for analyzing the data gathered will need to be designed for each study. Most often, these methods will include some form of content analysis (see Chapters 29 and 30), but in some cases, other types of statistical or qualitative analysis methods are more appropriate. In essentially every study, data obtained from documents or artifacts will need to be analyzed in combination with data obtained using other methods. This approach is often called triangulation, as data from multiple sources are integrated to draw more valid conclusions. For example, memos describing the development of a new information system might be triangulated with interviews conducted with system developers to draw more valid conclusions about the process. In another study, data drawn from library circulation records about use of the science materials in the university library might be triangulated with a survey of science faculty and graduate students to draw more valid conclusions about the use of that portion of the library collection. Thus other data sources can be used to validate and cross-check the findings drawn from use of documents and artifacts (Patton, 2002). In this way, some of the weaknesses of data drawn from documents and artifacts can be minimized.
EXAMPLES
Three examples will be discussed here. The first (Maxwell, 2005) is a typical example of the use of publicly available documents to understand, in this case, the construction of the meaning of the term homeland security. The second (Buchwald, 2000) illustrates the way in which documents created within an organization can be used to study that organization, particularly if the researcher is a member of that organization. The third
162 APPLICATIONS OF SOCIAL RESEARCH METHODS
(Dube, 2005) combines the use of documents and other artifacts as evidence of the way in which a university communicates to its students and faculty about health issues.
Example 1: Public Documents as Data
Relying on the texts in publicly available documents, Maxwell (2005) traced the development of the term homeland security (first used in a 2000 report of the U.S. Commission on National Security) and how its meaning changed over time and across different groups. Maxwell based his analysis on published and publicly available doc- uments. His sample was a set of 286 documents that were related to this concept. The sample included reports, recommendations, journal articles, and press releases. It in- cluded 96 documents published by U.S. government agencies, identified on the Web sites of the Department of Homeland Security and the White House as well as through Lexis-Nexis searches on those agencies; 112 documents from the General Accounting Office (GAO) collection on homeland security, retrieved from the GAO Web site; 68 doc- uments from academic and nonprofit organizations, “retrieved from the Social Sciences Abstracts database, the Public Administration Review and searches on organizational Web sites mentioned in Congressional hearings, news reports, Web searches, and gov- ernment and organizational studies” (Maxwell, 2005, p. 159); and 10 local government reports, retrieved from local government association Web sites and Web searches. This procedure for identifying relevant documents is appropriate. It is unclear how documents were selected from those identified as potentially useful; presumably all the documents in the data set, at a minimum, included the phrase homeland security and provided some discussion of it.
Each text was assumed to take a particular perspective on homeland security. They were analyzed in multiple phases. First, relevant concepts and the terms used to represent them were identified by analyzing the texts with concordance software. This process resulted in the identification of 73 coding terms, such as the term oversight, which was coded as representing an instance of control in interorganizational behavior, or the term
constructive conflict, which was coded as representing an instance of responsive change
in change orientation. Using this coding scheme, the texts were then fully coded (using AtlasTi software). From this point, factor analysis based on the use of codes within individual documents was used to reduce the number of terms to be examined in further analysis. For example, the terms mutual aid, coordination, and facilitation, among others, were combined into one concept, cooperative relationships. Finally, an analysis of variance compared the codes associated with different agencies and/or authors and found that different agencies did have different understandings of the concept homeland
security.
In summary, “this study has explored the homeland security symbol from two per- spectives: First, as the structure and outcome of a debate in the political sphere that institutionalized the symbol in legislation and administrative mandates; and secondly, as an organizing theme through which stakeholders in the administrative knowledge cre- ation process debated the requirements of cooperation, coordination, and action within inter-governmental networks” (Maxwell, 2005, p. 166). It used a computer-supported type of content analysis to understand the many concepts related to the term homeland
security in a variety of types of documents created by both government agencies and
academic researchers. Thus it is an example of the collection and analysis of publicly available documents.
Existing Documents and Artifacts as Data 163
Example 2: Use of Organizational Documents by a Member of the Organization
This second example1is also set within the context of the public sector and focuses
on “the role of a public interest group, Canada’s Coalition for Public Information, in the federal information policymaking process for the information highway” (Buchwald, 2000, p. 123), with particular emphasis on universal access policies. In this study, the use of data from documents was incorporated within a naturalistic research design that also incorporated participant observation and interviews. Participant observation occurred primarily at the monthly Coalition for Public Information (CPI) steering committee meetings, but also at other meetings and through observation of the messages on the CPI Listserv. Initial interviews were conducted with three members of the steering committee executive group, followed by 22 additional interviews with participants in the policy-making process.
The documents included in the data set were primarily those created by CPI and were collected as they were created. As a participant, Buchwald (2000) was able to collect many of these documents from their creators and so was able to include many documents that would not have been available to an outside researcher. The final set of documents in the data set included “submissions to government hearings, correspondence among members via the internet, internet correspondence among public interest groups, and correspondence with federal government ministers . . . [CPI] meeting minutes, flyers, position papers, responses to reports, interventions to the CRTC, and its seminal policy document Future-Knowledge” (p. 130). In this study, it is likely that the author was able to collect all or almost all the relevant documents. In addition, this set of data was not too big to include in its entirety. Thus issues related to sampling and representativeness were resolved because the entire population of relevant documents was included in the analysis.
An inductive qualitative approach to analyzing the content of these documents was taken, incorporating them into the full data set, which included field memos (docu- menting the observations) and interview transcripts. In addition, the author was explicit in triangulating her findings, across sources (observations, interviews, and documents) and across creators (CPI, government agencies, industry participants, etc.). In addition, Buchwald (2000) followed Hodder’s (2000) recommendation that the conclusions drawn from a study must be internally coherent and must also correspond to the available theoretical frameworks (of which two—policy community theory and policy process theory—were used by Buchwald).
In summary, this example demonstrates how analysis of documents can be integrated into a study that also draws data from other sources such as observation and interviews. It is also an example of a very naturalistic research design, incorporating a thematic analysis of the document content.
Example 3: Artifacts as Evidence of Information Communication Behaviors
The purpose of Dube’s (2005) study was to investigate “the framework, nature and scope of HIV/AIDS information communication strategies employed by higher educa- tion institutions in South Africa” (p. 315). Thirty-six institutions of higher education in South Africa were surveyed about the ways in which they communicated information
164 APPLICATIONS OF SOCIAL RESEARCH METHODS
about HIV/AIDS to their students; 33 responded. The respondents in each institution “were the HIV/AIDS service providers/intermediaries, institutional libraries and health centres” (p. 318), though it was not clear whether multiple people at each institution re- sponded to similar questionnaires, or one person at each institution responded on behalf of that institution.
In addition to the survey data, Dube (2005) collected data through campus site visits, directly observing the “billboards, signs, posters, pamphlets, brochures, slogans, etc.” (p. 319) that were visible on each campus. An observation guide was used to capture data about these artifacts, making the findings comparable across institutions. While I am depicting these objects as artifacts, they might more accurately be described as documents because, in Scott’s (1990) words, their “central feature [is] an inscribed text” (p. 5). From another perspective, they (in particular, the billboards, signs, and posters) can be seen as a measure of accretion, with their visible presence on campus being treated as an indicator of the institution’s communication behaviors.
Since two visits were made to each campus to observe the communication media in use, we can assume that the sample of artifacts observed was relatively complete and representative. It is not as clear how they were interpreted. Dube (2005) does not clearly differentiate the findings based on these observations and the findings based on the survey results. While the concept of triangulation is mentioned, it is not clear how the observation data were juxtaposed against the survey data to provide a more accurate picture of each campus’s communication behaviors.
In summary, this study is a good example of the way that artifacts can be directly observed and incorporated into a study that also uses other data collection methods (i.e., a survey). The use of an observation guide to ensure the reliability of the data gathered across campuses is a strength of this study’s methods. It would also have been useful to include a more explicit discussion of the ways in which the observation data and the survey data supported or conflicted with each other.
CONCLUSION
The use of evidence gathered directly from preexisting documents or artifacts can greatly strengthen a study. If an appropriate sample can be gathered, this nonreactive approach to data collection can allow the researcher to see some aspects of a situation that could not be detected through more intrusive data collection methods such as interviews or questionnaires. While it has some advantages, this data collection approach is rarely used alone; it is more often used to provide a different perspective on a particular phenomenon. In this way, findings from multiple data sources can be used to draw more valid conclusions about the phenomenon of interest.
NOTE
1. This work was the recipient of the 2000 Eugene Garfield Doctoral Dissertation Award from the Association for Library and Information Science Education.
WORKS CITED
Bernard, H. R. (2000). Social Research Methods: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage.
Existing Documents and Artifacts as Data 165
Buchwald, C. C. (2000). A case study of Canada’s Coalition for Public Information in the information highway policy-making process. Library and Information Science Research,
22(2), 123–144.
Cooley, M. (1989). City Aphorisms: Sixth Selection. New York: Pascal Press.
Denscombe, M. (2003). The Good Research Guide for Small-Scale Social Research Projects (2nd ed.). Maidenhead, England: Open University Press.
Dube, L. (2005). Insights into the diffusion of HIV/AIDS information in higher education institu- tions in South Africa. International Information and Library Review, 37(4), 315–327. Gray, D. E. (2004). Doing Research in the Real World. London: Sage.
Hodder, I. (2000). The interpretation of documents and material culture. In N. K. Denzin & Y. S. Lincoln (eds.), Handbook of Qualitative Research (2nd ed., pp. 703–716). Thousand Oaks, CA: Sage.
Maxwell, T. A. (2005). Constructing consensus: Homeland security as a symbol of government politics and administration. Government Information Quarterly, 22(2), 152–169. Miller, G. (1997). Contextualizing texts: Studying organizational texts. In G. Miller & R. Dingwall
(eds.), Context and Method in Qualitative Research (pp. 77–91). Thousand Oaks, CA: Sage.
Neuman, W. L. (2006). Social Research Methods: Qualitative and Quantitative Approaches (6th ed.). Boston: Pearson.
Nicholson, S. (2005). Digital library archaeology: A conceptual framework for understanding library use through artifact-based evaluation. Library Quarterly, 75(4), 496–520. Patton, M. Q. (2002). Qualitative Research and Evaluation Methods (3rd ed.). Thousand Oaks,
CA: Sage.
Platt, J. (1981). Evidence and proof in documentary research: I. Some specific problems of documentary research. The Sociological Review, 29(1), 31–52.
Quine, W.V.O. (1960). Word and Object. Cambridge: Technology Press of the Massachusetts Institute of Technology.
Scott, J. (1990). A Matter of Record. Cambridge, MA: Polity Press.
Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (2000). Unobtrusive Measures (Rev. ed.). Thousand Oaks, CA: Sage.