FJC Study #3

The main study often cited by researchers as demonstrating that the summary judgment Trilogy did not have an effect on the number of federal civil trials is the third study by the FJC (Cecil et al. 2007)23. The structure of the study is similar to the 2001 study. Here, the same six district courts were reviewed with essentially the same dataset for the same years.24 Docket sheets were reviewed (or, had al- ready been reviewed), and for the last two years in the data PACER was utilized to gather the docket sheets through electronic collection.

Given the data is largely the same, the results proffered by the authors at least with respect to summary judgment filing rates and summary judgment grant rates is the same:

[A]fter we controlled for differences across courts and the changing mix of cases, we found few changes in summary judgment activity after the Supreme Court trilogy. The appearance of higher rates of summary judgment in general may be due to increased filings of civil rights cases, which have always had a higher-than-average rate of summary judg- ment motions and dispositions. Although increases in summary judg- ment may be part of the reason for the decrease in trial rates, the decline in trials reflects far broader changes in litigation practice than simply a response to the Supreme Court’s affirmation of summary judgment practice.

(Cecil et al., 2007, 906)

23_{There is a related study also issued in 2007 that confines itself to the year 2006; I will discuss}

that study last.

24_{In actuality, it appears the year in FJC Study #2 1985 has been converted to 1986 and the year}

1990 in FJC Study #2 has been converted to 1989. It is unclear why this change was made, although it is reasonable to postulate that the change in title of the years is simply a modification from consid- ering when the data were gathered versus when the data were presented in the studies. Whatever the reason, the numbers for the raw data match suggesting this is only a nomenclature issue, and not a substantive concern.

Given the data are the same, the filing and grant rates are the same for FJC Study #3 as FJC Study #2. But, the authors do more with the data this time around and produce logistic regressions for the contributing causes for whether a motion for summary judgment is filed and whether a motion for summary judgment is granted (again, in whole or in part). It is here where there is cause for concern with this study.

The logit estimations utilized by the authors (for both the filing rates and the grant rates) include as independent variables dummied categories for each district (with the Southern District of New York as the omitted category) and the year (1986 is the omitted category). These are all the independent variables utilized. As with Gelbach (2014), if we are to properly test a social scientific theory, failure to include other variables will bias the estimates. In short, the models are straw men: if, in particular, the time (dummy) variables show no statistical significance, then it is taken that with regard to the reference category (1986, the year of the Trilogy), no effect can be discerned. But of course, failing to include omitted variables in the logit estimation will cause biased coefficients that are overly deflated (Mood 2009), thereby suggesting no effect post-Trilogy. So, precisely because these are the only independent variables included in the model is the very reason the authors can claim to draw the conclusion that there is no post-Trilogy effect. It is a cyclical statistical problem wrought from model misspecification at the start.25

The authors also make a questionable call in determining whether a motion for summary judgment is granted or not. The process by which such a motion is granted is rather simple to understand, but I will elucidate it here for further clar- ity. When a party (read: defendant) wishes to have a claim (or claims) kicked out of 25_{As with Gelbach (2014), we have no model diagnostics and no notion as to whether these vari-}

federal court during the pre-trial stage, they will file a motion for summary judg- ment, the other side will respond, the district court will review the paper record, and a decision is then rendered by the district court judge as to whether the mo- tion will be granted (either in whole, or in part).26 Keep in mind the processed ordering of the motion: a summary judgment motion cannot be granted unless it is filed. In other words, a federal district court judge cannot grant a motion for summary

judgment unless such a motion is made by a party.27 The decision to grant the motion for summary judgment is conditional on the motion for summary judgment being filed. Cecil et al. (2007) do not model the grant rates in this fashion, however.

While a bit lengthy, the authors (almost) complete statement for this modeling strategy is worth evaluating:

This analysis of grant likelihood is not conditional on the presence of a motion for summary judgment. . . We conducted the analyses in this way for at least two reasons. First, policy discussions about changing litigation trends in general, and summary judgment activity specifi- cally, tend not to be couched in conditionals; rather, changes (such as increasing or decreasing termination rates) are spoken of as individual effects in what is understood to be a complex system. Second, we knew that reducing the sample size for each analysis through conditionals would limit the power of the analyses and increase the chances that meaningful results would be masked—although we did not anticipate substantial differences. In fact, when we did run a logistic regression estimating the likelihood of a summary judgment grant conditional on there being a motion for summary judgment, the results were generally

the same. As expected some districts’ previously significant coefficients became marginally significant or nonsignificant. . . However, the sam- ple years that possessed significant coefficients in the nonconditional analysis also did so in the conditional analysis. . . For a discussion of the benefits of a nonconditional analysis over a conditional analysis, see 26_{Recall Cecil et al. measure this “grant” as a grant in whole or in part. I have no qualms with}

this measurement of a “grant,” and indeed it is the same measure I employ in the micro-analysis portion of my research design discussed in Chapter 5.

27_{A district court judge could dismiss a case on their own, for intance, if a party fails to actively}

engage the litigation and lets it go dormant. This situation, however, is not what is discussed in this section.

A.N. Pettitt & S. Low Choy, Bivariate Binary Data with Missing Values: Analysis of a Field Experiment to Investigate Chemical Attractants of Wild Dogs, 4 J. Agric. Biological & Envtl. Stat. 57 (1999).

(Cecil et al., 2007, 893-94 n.75 (emphasis added))

There are several problems with this econometric rationale. As to their first contention, if the authors are truly concerned with “individual effects,” then re- taining the conditional impact of deciding only what has been filed is the individual

effect. To put it another way, to treat, for instance, whether a motion is granted or not as the only effect of what has happened in that particular case, but to also create a sense of semi-aggregation to all other cases, is to diminish the individual effects and obtain some estimate for the sample based on what other observations have done.28

Additionally, the authors’ reliance on the work of Pettitt & Choy (1999) (eval- uating the efficacy of certain chemical attractants for Australian dingos) to vouch for their modeling strategy misapprehends that work. In short, Pettitt & Choy

do not recommend that one models “missing” data by simply employing a non- conditional maximum likelihood estimation. Instead, what those authors propose is “a model conditioning on dingo presence/absence and hypothesizing a distribu- tion for dingo presence/absence [through the use of] an EM29 algorithm” (Pettitt 28_{It is difficult to tell how the authors have even coded this scheme. It is perhaps the case that}

they employed the following scheme: if a case did not have a motion for summary judgment filed, it was coded as a zero. Additionally, by definition, that observation would receive a zero for “grant” (because, obviously, no motion was filed and thus could not be granted). But, a case could also have a zero for the “grant” column if a motion was filed, but the court simply did not grant the motion. As a result, the zeros in effect have multiple meanings. Thus, rather than coding the zeros where no motion had even been filed as “missing,” the value of zero has been conflated with its true meaning: the motion was not granted. Without the data, it is difficult to determine if this coding scheme is what the authors are referencing with their statement, but it seems plausible that this approach is what they have employed when one backs out their “conditionals” discussion to its logical conclusion.

& Choy, 1999, 57). In other words, what the authors propose is a way to model the probability distribution for whether a dingo even went to a site or not, and then incorporate that distribution into the ultimate estimation for the effects of various chemical attractants. Cecil et al.’s (2007) work ignores the hypothetical distribution of non-filed motions that are not-granted, and condenses the data into an uncon- ditional estimation. This is an incorrect estimation strategy.

A simple manner for dealing with this problem is to consider the non-grants in non-filing cases as just that: missing data. The exclusion of these observations will necessarily diminish Cecil et al.’s N (as they correctly note), but with thou- sands of observations it would be surprising to see this N drop to levels where the asymptotic properties of the estimator can be called into question. Another method for dealing with this problem is to do a variant of what Pettitt & Choy recommend which is to model a probability distribution for the choice to “opt-in” to the system (whether a motion is filed), and then model what happens once in that system (grant/deny of a pending motion for summary judgment). This strat- egy could be accomplished through use of a two-stage procedure. In the Ordinary Least Squares (OLS) realm, one would use a Heckman estimation30; in a binary choice to file, and binary choice to grant situation, an extension of the two-stage rationale could be applied (see Greene, 2012, 880-83 (discussing sample selection is- sues and two-stage estimations for non-linear models)). Ultimately, while the data gathering efforts are substantial, and Cecil et al. employ a research design more carefully crafted than others, it still falls short of answering the ultimate question, which is whether the Trilogy impacted summary judgment motions practice.