Threats to the Validity of Inferences

In any study, it is important to anticipate and, to the extent possible, address potential threats to the validity of causal inferences drawn from its results. Shadish, Cook, and Campbell (2002) sort these threats into four categories: statistical conclusion validity, internal validity, construct validity, and external validity. The following

discussion highlights the validity threats believed to be most plausible in this dissertation, describes steps taken to address these threats, and, where possible, identifies the likely impact of the threats to the study’s conclusions.

Statistical conclusion validity concerns inferences about whether study treatments and outcomes covary and the strength of their relationship (Shadish et al., 2002). Despite the dissertation’s sample size of 415, statistical power may be an issue in testing some models. Three steps are taken to help increase power. First, survey respondents were drawn from five counties that share many features that may be relevant to adoption of farming practices including climate, physiographic region, farming economy, and crop types. By limiting the survey to these areas, these features do not need to be controlled in the statistical models. Second, the models include covariates that could influence

94

practice adoption, such as farm and producer characteristics, which should increase power (Shadish et al., 2002). Finally, the survey data were checked for outliers and the impacts of the identified outliers on statistical conclusions were assessed. If any models in the dissertation prove to be underpowered despite these measures, the likely effect will be that they will incorrectly conclude that the relationship between treatment and

outcome is insignificant (Shadish et al., 2002).

Internal validity concerns whether any identified covariance between treatments and outcomes reflects a causal relationship (Shadish et al., 2002). Two internal validity threats are potentially important, temporal precedence and selection. Establishing temporal precedence can be difficult in cross-sectional studies where all study variables are measured simultaneously. However, in this study, both the timing of the survey and theory help to diminish the plausibility of this threat. Nutrient management training was offered in the Neuse Basin counties between 2000 and 2002 and producers in the Basin were required to develop their nutrient management plans by December 2002. The survey was conducted in December, 2005, several years after the completion of these activities. As such, the treatments (training and planning) clearly took place prior to the measurement of the attitudes and use of nutrient BMPs. Additionally, as outlined in the literature review, there are strong theories and empirical data supporting the argument that the motivations under study in this dissertation influence environmentally-

responsible behaviors, such as adoption of BMPs. Though it is possible that there is some feedback from adoption to attitudes based on producers’ experiences with the practices, the predominant influence should be from attitudes to adoption.

95

Selection is a concern in this dissertation because producers chose the nutrient management activities in which they participated. It is probable that the producers who chose to participate in training were different from those who either chose to develop nutrient management plans, chose to do both, or chose to do nothing. Two approaches will be used to help address this threat. First, the statistical models control for covariates that could be related to selection into the different activities, primarily characteristics of the farm and producer. Second, the study uses control groups (i.e., the intend to train group and the intend to do both group) that should be very similar to the treatment groups on any unknown factors leading to selection into different treatments. The relatively high response rate in the survey of 74 percent diminishes the potential validity threat that those who chose to participate in the survey could be systematically different from those who did not.

Construct validity refers to how higher order constructs related to people, settings, treatments, and observations in a study are measured and how well the measures match the actual constructs (Shadish et al., 2002). Three potential construct validity threats are significant in this study: mono-operation bias, mono-method bias, and treatment

diffusion. Mono-operation bias stems from using only one measure, or

“operationalization” of a construct. Using only one measure can simultaneously fail to capture all aspects of the construct and include irrelevant constructs (Shadish et al., 2002). For most constructs in the study, such as age or participation in training, one measure is appropriate. For the motivation constructs, multiple measures would have been ideal, but were not possible in all cases due to strict survey length limitations. In mediation analysis, the impact of measurement error is to attenuate the mediated effect

96

estimates (MacKinnon, 2008). Thus, any bias from mono-operation bias should be to underestimate the role of the mediating variables.

Mono-method bias may exist in this study because all of the data come from the survey. Accordingly, what is actually studied in this dissertation is “self-reported” activity participation, attitudes, and adoption behavior, which could differ from more objective measures. Treatment diffusion may also be a factor in this study. Even though producers in the Tar-Pamlico River Basin did not have access to nutrient management training prior to implementation of the survey, it is possible that they were exposed to information from the training informally through contacts with Extension agents and other producers who had participated in the Neuse Basin. Exposure to this information by participants in the study’s control groups could have the effect of reducing the size and significance of any relationship found between participation in training and adoption of nutrient BMPs.

External validity concerns inferences about the extent to which the size and direction of causal relationships between treatments and outcomes are consistent over different people, settings, treatments, and outcomes (Shadish et al., 2002). The goal of the dissertation is to evaluate the impacts of one particular type of agricultural NPS policy on producers’ motivations and adoption of three specific practices. It does not attempt to generalize these results to other types of policies that may be very different in nature or contain different incentives and disincentives for adoption. Findings from this study will directly pertain only to the particular policies and training and planning activities that occurred in the Neuse Basin and to the particular nutrient BMPs investigated. However, investigating three different BMPs that are expected to be

97

influenced to varying degrees by training and planning provides much more information about Neuse Basin strategy’s impacts than investigating just one. It is also important to note that the study sample was not a random sample of all producers in these counties, but rather those who had signed up under the Neuse Basin and Tar-Pamlico Basin rules. The average farm size in this sample is larger than that found in the agricultural census. While use of this sampling frame precludes drawing inferences about all producers in the counties, it facilitates a focus on those farm operations most targeted by the agricultural rules in the two basins.