Research Methodologies for Macro and Micro-Level Analysis

Ostrom (1995) comments on the distinction between macro- and micro-level analytical techniques. She observes that “[s]tudying micro-level phenomena requires micro-level theories and empirical methods appropriate to testing these theories” while “[m]acro-level phenomena…require their own theories and methods of empirical analysis,” reasoning that researchers are “viewing a complex mosaic of recursive processes” and “there is no single level that provides the best answer to all questions” (Ostrom 1995, 174-8).

In policy research, as in other social science disciplines, scholars seek “to establish a balance between the competing claims of complexity and generality” (Ragin

3 _{I adopt the ‘case’-driven and ‘variable’-driven labels proposed in Approaches and Methodologies in the} Social Sciences (della Porta and Keating 2008).

and Zaret 1983, 731). The research question, in focusing on macro- or micro- phenomena, determines the appropriate methodologies.

On one end of the spectrum, macro-analysts consider a case in rich detail,

exploring a range of explanatory factors that, in combination, produce certain outcomes. On the other, researchers interested in micro-level processes quantify variables, gathering large data sets and employing sophisticated statistical techniques to measure the effects of an independent (or explanatory) variable on a dependent variable (or outcome of

interest). This distinction can be described as the difference between case and variable- driven research (della Porta and Keating 2008).

In policy circles, these approaches have been embraced by different research communities. A large contingent of the academic community has gravitated towards micro-level inquiries, employing variable-driven research methodologies. This research focus dominates peer-reviewed journals in the social sciences.

A smaller academic contingent and research institutes (including government entities and independent think tanks)4 tend to take a macro-policy perspective, adopting case-driven approaches to produce descriptive narratives on policy developments.

There is certainly overlap and a number of well-known scholars continue to employ the case-driven, analytic narrative in academic policy research.5 However, I suggest that the two groups tend to favor different research methodologies, and the distinction is sufficient to warrant some justification for taking a case-driven approach in an academic research project.

4 _{For example, the Government Accountability Office (GAO), the Congressional Research Service (CRS),}

the Center for Budget and Policy Priorities (CBPP), the Urban Institute, the Center for Law and Social Policy (CLASP), the Brookings Institution and others.

20

In academic circles, the discretion-capacity could be criticized for its failure to resolve policy endogeneity, a methodological issue specific to micro-level, variable- driven research that is, in certain peer-reviewed journals, central to the debate on federalism and social policy. The proposed framework neither seeks nor claims to resolve these issues. However, in light of that potential criticism, it seems important to distinguish between variable and case-driven research and demonstrate why the

methodological constraints of former should not be imposed on the latter.

In the next section, I justify the use of a case-driven approach and demonstrate how variable-driven techniques are incompatible with macro-level analysis.

Consequently, I claim that the assumptions limiting the specification of econometric models are not applicable to the discretion-capacity framework.

Tension between a macro-level view of federalism and variable-driven analysis

The objective of variable-driven research is to test a falsifiable hypothesis (Steinmo 2008; Ragin 2004), drawing evidence from a large number of cases to determine the causal relationship among relevant variables (della Porta and Keating 2008). The methodology rests on a number of assumptions that are problematic for macro-policy research in which state policy choices are the units of analysis. Before addressing those limitations, I offer a brief overview of the theoretical justification for using statistical methods to estimate the effect of causal conditions on outcomes of interest.

Given the challenges of conducting state-level randomized experiments, variable- driven analyses of state-level outcomes are, at best, quasi-experimental. These

Inference, which notes that units cannot be observed under both treatment and control conditions at the same time (see discussion in Holland 1986). Consequently, the

researcher is forced to make assumptions in order to estimate the treatment effect, or the effect of a causal condition on the outcome of interest.

Rubin’s model asserts that if a probability sample of sufficient size is drawn from the population and exposed to treatment then observed outcomes for the sample

approximate the outcomes for the population. Likewise, if another sample of different units is drawn and exposed to the control conditions then those outcomes approximate the outcome under control conditions for the entire population. The difference between these two yields the average treatment effect for the population. Several problematic issues arise in extending this logic to analyses of state policy choices.

First, this approach implies that universal laws exist (Ragin 2004, 127). Properly specified models ascertain whether causal conditions produce measurable effects on state policy choices. This assumes homogeneity of units and constant effects—that a causal condition in one state is also a causal condition in another state. In reality, it is uncertain that causal conditions hold across states, policies or time, even when controlling for observable relevant differences in state characteristics. If causal conditions do not hold then it is untrue that “permanent causes are systemic attributes of sampled units that characterize all units of the population” (Ragin and Zaret 1983, 743). In such a scenario, the theoretical justification for using variable-driven analysis is flawed.

States display considerable diversity across as many variables as a researcher can collect yet it is essential to variable-driven analysis that they are “homogeneous

22

used to make generalizations about other states, policies or time periods. It is difficult to construct a scenario in which causal conditions could be argued to have constant effects when the unit of analysis is the state. In fact, if the researcher is interested in the effect of federal legislation on state policy choices, the causal condition itself is often variable by definition. Federal social programs routinely include numerous state-specific provisions, grandfather clauses, waivers, and opt-out allowances that complicate clean definition of a causal condition.

Second, the methodology relies on statistical techniques to make causal inferences. Those techniques require the quantification of relevant variables, but consider the difficulty of quantifying concepts such as political ideology or past policy choices. Researchers have attempted to control for qualitative differences in state values by creating indices to quantify factors such as ideology (Berry et al 1998; Holbrook and Bibby 1999; Piven and Cloward 1988), policy entrepreneurialism (Weissert 1991;

Mintrom 1997), administrative responsiveness (Fossett and Thompson 2005) and interest group activity (Nownes and Neeley 1996; Nownes and Freeman 1998; Wolak et al 2002). Such metrics are difficult for researchers to construct and easy for critics to deconstruct. Their inclusion raises serious questions about the rigor of variable definition but their exclusion certainly leaves substantial room for unobserved heterogeneity in the error term.

Each state constitutes a unique environment that has been shaped over time by a combination of political actors, partisan ideologies, fiscal resources, demographic profiles and population characteristics, considerations that are difficult to model

effects of state political and economic variables on workers compensation policies. They first omit and then incrementally introduce political and economic variables to

demonstrate a significant impact on both the magnitude and direction of estimated effect sizes (Besley and Case 2000).

Third, Rubin’s model requires a “sample of sufficient size.” Considering the number of control variables required to minimize unobserved heterogeneity, the number of observations is often insufficient. Scholars of federalism who are interested in outcomes at the state level are faced with the reality that N=50 for each time period. Since panel data on states has not been consistently collected on many variables over a long period of time, it is often difficult to collect a sample size large enough to produce the necessary degrees of freedom to include a complete vector of control variables. Where panel data do exist on state policies, researchers are often faced with issues concerning consistent measurement and definition across states and even within states over time. This issue has been frequently cited in GAO reports on SCHIP and TANF in particular (Greenberg and Rahmanou 2005).

Fourth, Rubin’s model assumes that exposure to treatment is determined independently of outcome, treatment and control variables. Without the independence assumption, it is difficult to claim that the observed outcomes are attributable to the causal condition. As mentioned above, federal legislation often includes differential policy provisions for states based on socio-economic or fiscal indicators, which could present challenges to disentangling the effects from population characteristics.

This segues into a major issue for variable-driven methodologies in the field of federalism. Statistical models are often challenged on the grounds that variables used to

24

control for state variations in an expenditure function are endogenous. For example, if state social policy choices are described as a function of federal policy parameters, state fiscal conditions (e.g. per capita income), state political characteristics (e.g. number of Democrats in state legislature) and a vector of time-variant state-level demographic variables, the following problematic arguments could be made: 1) that fiscal capacity is either caused by or causes certain political characteristics; 2) that demographic variables influence political composition or even; 3) that the specified causal direction could be reversed and that state policy choices dictate certain outcomes in terms of federal policy and its own fiscal or political characteristics. Any of the arguments above would violate important assumptions that underpin statistical models.

Yet federalism is inherently and simultaneously a political and fiscal proposition (Pierson 1995). Though endogeneity threatens statistical validity (Shadish, Cook and Campbell 2004), it is difficult to resolve methodologically when it persists in the actual policy environment (Besley and Case 2000).

Disentangling the institutional effects from the impact of state level characteristics on social policy presents a methodological quandary. Since quantitative or variable- driven empirical analysis of the effects of federalism on state policy is difficult to model econometrically, research questions consequently tend to cluster around the effects of state policy on enrollment (e.g. Kronebusch and Elbel 2004; Nicholson-Crotty 2007), health and welfare outcomes (e.g. Card and Shore-Shepard 2004), or expenditures (e.g. Barilleaux and Miller 1988; Miller 1991; Poterba 1994; Endersby and Towle 1997; Kousser 2002; Hoover and Pecorina 2005). The findings, though interesting in their

particulars and breathtaking in their collective scope, tend to sidestep positioning federalism as an explanatory variable.

There are, of course, exceptions to this generalization and a handful of scholars identify institutional constraints as key determinants of state policy decisions. Their results underscore the difficulty of imposing micro-level, variable-driven methods on essentially macro-level research questions.

For example, Grogan (1999) examines the impact of federal mandates on state policy choices concerning eligibility levels and benefit coverage in Medicaid and Assistance for Families with Dependent Children (AFDC, the pre-cursor to TANF) and finds that federally mandated expansions in one policy area tended to produce benefit reductions in areas where states retained discretionary power. Using state level political variables to control for ideological differences, a significant positive relationship is shown to exist between both Democratic and Republican control of the legislature and AFDC financial eligibility levels (Grogan 1999). This result illustrates possible

shortcomings of a necessarily simplified measure of political ideology and opens the door for criticisms of omitted and unobservable variables. The study also reports a puzzling negative relationship between benefit coverage and state tax-capacity and effort, which the author concedes “raises some concerns about model specification” (Grogan 1999, 27).

Nicholson-Crotty et al. (2006) analyze the relative amounts of federal grant funding, controlling for variation in state needs and political ideology, to demonstrate how federal funding changes result in state-level budgeting tradeoffs. However, when counter-intuitive relationships are found between state needs and state expenditures, the

26

authors suggest they could be attributed to unobserved and omitted variables, including relative political power of beneficiary groups, institutional constraints, and increased state discretion over healthcare expenditures (Nicholson-Crotty et al. 2006).

While the Grogan (1999) and Nicholson et al. (2006) articles consider the effect of federal policies on state policy choices, as variable-driven analyses they encounter methodological hurdles related to policy considerations that are intrinsic to the federalist policy environment. In maintaining a macro-perspective, the discretion-capacity

framework acknowledges, but does not seek to resolve methodologically, the deeply inter-connected set of explanatory variables that influence state policy choices in the American federal system.

Case-driven research as a methodology for macro-level analysis

In case-driven research, cases are treated as interdependent wholes and the search for a simple cause and effect relationship is abandoned to allow for “mutual influence among many factors” (della Porta and Keating 2008, 27-30). A case study has been defined as “…a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context” (GAO/PEMD-91-10.1.9).

Analyzing a state or policy as a “complex unit” permits the consideration of historical context and the impact of institutional arrangements (Steinmo 2008, 136) that are difficult to quantify for variable-driven analysis. Furthermore, case-driven research allows for “temporally discreet causes” (Ragin and Zaret 1983, 743) and enables the researcher “to understand the principles by which the parts consistently fit together” (Smelser 1976, 204).

Ragin (2004, 132) emphasizes the importance of distinguishing between population characteristics and causal conditions in variable-driven research. This is challenging in the federalist context since causal conditions are likely combinatorial and temporally discreet. There might not exist generally applicable propositions about which variables are descriptive of the population and those which are causal.

Case-oriented researchers “anticipate finding several major causal pathways in a given body of cross-state evidence. A typical finding is that different causes combine in different and sometimes contradictory ways to produce roughly similar outcomes in different settings” (Ragin 2004, 134). A case-driven approach may be appropriate for researchers interested in identifying “patterns of multiple conjunctural causation” (Ragin 2004, 134).

Case-driven research promotes internal explanation that Ferejohn (2004, 150) defines as focusing on the “reasons for an action”. While the motivations for state policy choices are varied and context specific, patterns may exist that yield insights about how states respond to shifts in federal parameters.