Quantitative content analysis

In order to explore the Court’s decision-making, the information in Court of Appeal judgments is converted into numbers (i.e., ‘coded’) for analysis. This is done following a process of a quantitative content analysis. Riffe, Lacy and Fico define quantitative content analysis as being the:

133

‘systematic and replicable examination of symbols of communication, which have been assigned numeric values according to valid measurement rules and the analysis of relationships involving those values using statistical methods … to draw inferences’.628

Quantitative content analysis ‘[reduces] communication phenomena into manageable data (e.g. numbers)’ which can then be examined statistically.629

Three terms from the above definition are crucial to quantitative research: ‘systematic’, ‘replicable’, and ‘valid measurement’. This study has been designed to conform to these standards to allow for the analysis of the Court of Appeal’s decision-making. The extent to which this has been achieved is a key component of this thesis.

‘Systematic’ quantitative content analysis ‘requires identification of key terms or concepts involved in a phenomenon, specification of possible relationships amongst concepts, and generation of testable hypotheses regarding the potential relationships.630 _{The key concept under analysis in this thesis is impartiality,}

which was defined and explained in Chapter 2. The ‘phenomenon’ under analysis in this thesis is that some appeals are allowed and some are dismissed. This thesis seeks to explore the relationship between independent variables as a measurement of whether the Court appeared to have decided appeals impartially. The data collection is systematic because a set of hypotheses have been developed in relation to the possible relationship between independent predictor variables and the outcome of appeals against conviction. The hypotheses and variables used in this study are fully explained in Chapter 6.

‘Replicability’ is an essential component of quantitative analysis. It requires an ‘exactness’ to the research definitions and operations so that later readers can fully understand what was done.631 _{In relation to Empirical Legal Studies (ELS)}

628 _{D Riffe, S Lacy and FG. Fico, Analyzing Media Messages: Using Quantitative Content Analysis}

in Research (2nd _{ed, Lawrence Erlbaum Associates 2005) 25.} 629 _{ibid, 23.}

630 _{ibid, 25.} 631 _{ibid, 26.}

134

concerning judges, Epstein and King argued that ‘good empirical work adheres to the replication standard’, in that another researcher should be able to ‘understand, evaluate, build on, and reproduce the research without any additional information’.632 _{It is only by following this standard, Epstein and King}

argue, that it is possible to know that the research is not biased and so can present knowledge about the court under observation.633

‘Valid measurement’ (or ‘validity’) in quantitative content analysis means that the data collected must accurately represent what is being measured.634 _{In relation}

to this study, this means that the measures used (the independent variables) must accurately capture the underlying concept of impartiality.635 _{Epstein and}

King argued that to produce reliable and valid inferences, researchers should, 1) invoke theories that produce observable implications, 2) extract as many implications as possible, and 3) delineate how they plan to observe those implications.636 _{As discussed in Chapter 1, the variables collected in this study do}

not completely capture the principle of impartiality; the measures do not, therefore, have full validity. This limits the strength of conclusions which can be drawn regarding the Court’s impartiality.

As Hall and Wright argued, content analysis appears particularly appropriate as an ELS methodology, because it resembles what lawyers and legal scholars already do.637 _{‘Black-letter’ legal scholars frequently read a series of cases,}

collect information, and discuss their significance. Content analysis can bring a systematic rigour to the analysis of cases, which provides ‘a way of generating objective, falsifiable, and reproducible knowledge about what courts do and why they do it’.638 _{They argued that content analysis is more useful for some kinds of}

632 L Epstein and G King, ‘The Rules of Inference’ (2002) 69 U Chic L Rev 1, 27. 633 _{ibid, 31.}

634 _{Riffe, Lacy, and Fico (n 628) 31.} 635 _{Epstein and King (n 632) 62.} 636 _{ibid, 47. (Emphasis added).}

637 _{MA Hall and RF Wright, ‘Systematic Content Analysis of Judicial Opinions’ (2008) 96 Cali L}

Rev 63, 64.

135

legal analyses than for others. They note four different uses of content analysis in empirical legal research:

1) projects investigating the bare outcomes of legal disputes, 2) projects investigating the legal principles of case outcomes,

3) projects investigating the facts and reasons that contribute to case outcomes, and

4) ‘jurimetrics’ that attempt to predict the impact of facts on litigation.639

They argue that content analysis can work well for the first three, but that jurimetrics overreaches the epistemological aims of content analysis.640

This study could be considered ‘jurimetric’, but it is important to note that there is no attempt to predict future decisions. It would best be considered research of category 3. As Hall and Wright say, their third category is suggestive of research which seeks to ‘document trends in case law and the factors that appear important to case outcomes’.641 _{Category 3 research can be contrasted with}

‘jurimetrics’, which seeks to ‘predict the likely outcome of litigation or appeals based on real-world or trial-record views of the facts’.642 _{This thesis does not seek}

to predict the outcome of future litigation but seeks to discover which variables are ‘predictors’ of the outcomes of appeals which have already been decided. As Hall and Wright note, to predict future cases it would need to be assumed that the information provided in judgments is a complete reflection of everything which contributed to the decision. This is an assumption which is unlikely to hold.643

639 _{ibid, 85.} 640 _ibid. 641 _{ibid, 91.} 642 _{ibid, 99.} 643 _ibid.

136

The quantitative content analysis of Court of Appeal judgments was the method of collecting the data. These data are designed to offer some measurement of the impartiality of the Court. The principles behind the measurement of is now considered. The particular variables designed to provide a measurement of impartiality are discussed in Chapter 6. The variables are also listed in Appendix A, readers who wish to review the variables earlier may want to turn to Appendix A.