Validity and Credibility

The previous two sections discussed adopting a research perspective to frame the research strategy and goals, and choosing specific research methods to meet the research goals. A third and critical component of a research plan is to produce valid and credible results—conducting research using an appropriate perspective and methods will be of little value if the results from the study are not convincing. As Hignett (2005) notes, “At a fundamental level one aim of all research should be to convince the reader” (p. 123).

What is required to “convince the reader”—to provide credible research results from qualitative field research? In design research, the ultimate measure of success is the quality of designs produced—even though this outcome cannot be fully attributed to the quality of the user research. Moreover, the critical design ethnography and action research perspectives both caution against

measuring success in terms of “solutions”—one must remain concerned with how people themselves are empowered. Techniques for developing valid and credible qualitative results, developed in the social sciences, can inform both of these perspectives. Researchers generally agree on three primary approaches for ensuring credible results: sampling, research technique, and verification

(Glesne, 2005; Hignett, 2005; Maxwell, 2005).

Sampling involves the selection of situations, times, people, and artifacts to observe, interview and analyze (Hignett, 2005; Maxwell, 2005). Positivist, quantitative research typically relies on equal-probability (also known as

“random”) sampling. By contrast, interpretive and qualitative research typically employs purposeful sampling, “in which particular settings, persons, or activities are selected deliberately in order to provide information that can’t be gotten as well from other choices” (Maxwell, 2005) (p. 88). Non-random sampling is often characterized as “convenience” sampling, but truly purposeful sampling is

in mind, Maxwell (Maxwell, 2005) identifies four factors that should be considered:

1) the representativeness or typicality of the sample

2) the extent to which the sample captures the diversity or heterogeneity of the population

3) the extent to which the sample contains cases (individuals, artifacts) critical to the research questions, issues or theories

4) the extent to which the sample allows comparisons between specific differences of interest

The researcher must weigh these different factors when seeking the overall objective of an effective sampling plan. In addition, these factors cannot be considered in isolation—the feasibility of the sampling plan and the researcher’s relationships with participants will influence many studies (Maxwell, 2005).

Another important step in ensuring credibility is choosing appropriate research techniques. In field work, it can be difficult to see people’s natural behavior—they may be nervous in the presence of a researcher, or seek to please the researcher by providing “desired” information. Researchers have suggested a combination of three techniques to combat this tendency (Bisantz & Drury, 2005):

1) explain the observer’s role, so participants become understand why the researcher is conducting the study, and that they are not being evaluated or judged

2) remain in the setting for an extended period of time, so participants can be become comfortable

3) focus on situations where participants are directly engaged in tasks, which is when people lose self-consciousness and anxiety

Finally, researchers can seek to verify their research results. Three primary methods for verification have been proposed (Bisantz & Drury, 2005; Maxwell, 2005):

1) conduct “member checks”—that is, test ideas developed during the research by speaking directly with participants

2) triangulate results with “parallel measures,” such as log or archival data, questionnaires, and focus groups

3) compare results with previous research in the field, or analogous studies in other fields

Each of these approaches can support research that “convinces the reader.” For example Bellotti et al. (2004) used a diverse sample of participants, met with participants multiple times over an extended period, combined interviews with

document analysis, and conducted member checks of their findings. These efforts led to credible conclusions about task management practices in a research organization. These approaches can also be used to identify discrepancies in the research results. For example, participants might challenge the plausibility of ideas proposed by the researcher, which could then be refined into stronger propositions. Once verification no longer yields significant discrepancies, the research results will have gained coherence and credibility.

2.5.4

Conclusion

The process of designing an effective qualitative field study involves understanding and applying an appropriate research perspective, choosing methods to gather data, and ensuring the credibility of these methods through appropriate sampling, research techniques, and verification.

These issues are framed by research perspectives, including design research, humanistic research, and institutional/infrastructural research. The design research perspective, which is central to this dissertation research, emphasizes the importance of understanding user needs through user research. User research seeks to understand behavior, particularly through field research, in order to support design. The findings from user research can then support an iterative design process, in which design concepts and prototypes are created, and then evaluated by both designers and representative users. The evaluation

identifies problems with and opportunities for improving the design. Ultimately, designs are refined into forms (artifacts) ready to be regularly used, such as a deployed research prototype, or a commercial or open-source product. As people use the artifacts, further opportunities for design and research become apparent.

Within the design research perspective, a range of methods and techniques can be used to gather data. Since every method has limitations, most studies combine (or “triangulate”) different forms of data collection. A key challenge is ensuring that the data (and subsequent) analysis are credible and trustworthy. Qualitative data are sometimes naively equated with anecdotes. But when qualitative researchers incorporate extended observation, rich description, and verification of findings and themes with participants and other sources of

evidence, the results can be highly credible. Therefore, qualitative field research, conducted from a design research perspective, can fruitfully inform the design of student-centered PIM systems.