Challenging scientific evidence

Professor Gary Edmond

Director, Program in

Expertise, Evidence & Law

Chair, Evidence-based forensics

initiative School of Law

Challenging scientific

evidence

Overview

1.Background – how we’re travelling 2.A ‘little’ science

3.Admissibility (s79) and the form of the evidence 4.Mandatory and exclusionary discretions (after XY) 5.Prosecutorial obligations

6.Codes of conduct for expert witnesses

7.Other trial safeguards: Cross-examination, rebuttal expert, directions and warnings

8.System issues and principle 9.Strategic action

Kaye Ballantyne, Senior Research & Development Officer Victoria Police Emma Cunliffe Associate Professor, Allard School of Law, UBC

Gary Edmond, Professor & ARC Future Fellow, Faculty of Law UNSW Bryan Found, Professor & Chief Forensic Scientist, Victoria Police David Hamer, Associate Professor, Law School, University of Sydney Brynn Hibbert, Emeritus Professor, School of Chemistry, UNSW Richard Kemp, Associate Professor, School of Psychology, UNSW

Andrew Ligertwood, Emeritus Fellow, Adelaide Law School Kristy Martire, ARC DECRA Fellow, Science, UNSW

Gianni Ribeiro, PhD Candidate, Science, UQ

Andrew Robert, Senior Lecturer, Law School, University of Melbourne Mehera San Roque, Senior Lecturer Faculty of Law, UNSW

Rachel Searston, PhD Candidate, Science, UQ

Jason Tangen, Senior Lecturer School of Psychology, UQ Matthew Thompson, Post-Doctoral Fellow, Science, UQ David White, Post-Doctoral Fellow, Science, UNSW

Rachel Dioso-Villa, Lecturer, School of Criminology & Criminal Justice, Griffith University

Evidence-Based Forensics

Trial safeguards and legal awareness?

‘The report finds that the existing legal regime—including the rules governing the admissibility of forensic evidence, the

applicable standards governing appellate review of trial court decisions, the limitations of the adversary process, and judges and lawyers who often lack the scientific expertise necessary to comprehend and evaluate forensic evidence—is inadequate to the task of curing the documented ills of the forensic science

disciplines. This matters a great deal, because “forensic science is but the handmaiden of the legal system.” … there are serious issues regarding the capacity and quality of the current forensic science system; yet, the courts continue to rely on forensic

evidence without fully understanding and addressing the limitations of different forensic science disciplines.

National Academy of Sciences, Forensic Science in the United States (2009) 85, 12, 53, 96, 109, 110.

2.

A ‘little’ science

What do independent scientists, engineers and biomedical researchers have to say about latent fingerprint, ballistic, tool mark, bite mark, hair,

document, voice and image comparisons and so on?

‘What lawyers should know about the forensic “sciences”’ (2015) 36 Adelaide Law Review 33.

NAS, Strengthening forensic science in the

United States: A path forward (2009)

• Background: fingerprint errors and DNA exonerations (US Innocence Projects). • National Academy of Sciences (NAS) –

National Research Council. • Congressionally funded.

• Multidisciplinary committee:

statisticians, a senior federal judge (Edwards), a chemist, senior forensic scientists, an engineer, biologists,

computer scientists, a medical examiner and law professors. • Two year inquiry (2007-2009):

submissions and hearings. • 13 Recommendations.

Using forensic science: Guiding principles

‘Two very important questions should underlie the law’s admission of and reliance upon forensic evidence in

criminal trials: (1) the extent to which a particular

forensic discipline is founded on a reliable scientific methodology that gives it the capacity to accurately

analyze evidence and report findings and (2) the extent to which practitioners in a particular forensic discipline rely on human interpretation that could be tainted by

error, the threat of bias, or the absence of sound operational procedures and robust performance standards. These questions are significant.’

National Academy of Sciences, Strengthening Forensic

Validation of techniques

‘The simple reality is that the interpretation of

forensic evidence is not always based on

scientific studies to determine its validity. This

is a serious problem. … there is a notable

dearth of peer-reviewed, published studies

establishing the scientific bases and validity of

many forensic methods.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009) 8.

Absence of validation – implications

‘With the exception of nuclear DNA analysis …

no forensic method has been rigorously shown

to have the capacity to consistently, and with a

high degree of certainty, demonstrate a

connection between evidence and a specific

individual or source.’

(i.e. latent fingerprints*, shoe, foot, ear, tyre, bite mark, hair, fibre, soil, ballistics, handwriting, voice and image comparisons and so on).

National Academy of Sciences, Strengthening Forensic Science in the United States (2009) 7.

Accuracy, uncertainty and errors

‘Few forensic science methods have developed

adequate measures of the accuracy of

inferences made by forensic scientists. All

results for every forensic science method

should indicate the uncertainty in the

measurements that are made, and studies

must be conducted that enable the estimation

of those values.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).

Terminology and expressions

‘many terms are used by forensic examiners in

reports and in court testimony … Such terms include … “match,” “consistent with,” “identical,” “similar in all respects tested,” and “cannot be excluded as the

source of.” … the forensic science disciplines have not reached agreement or consensus on the precise meaning of any of these terms. This imprecision in vocabulary stems in part from the paucity of

research’.

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).

Human factors

‘Some initial and striking research has uncovered the effects of some biases in forensic science procedures ... The forensic science disciplines are just beginning to become aware of contextual bias and the dangers it poses. The traps created by such biases can be very subtle, and typically one is not aware that his or her judgment is being affected.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).

See also Expert Working Group, Latent Print Examination and Human

Factors: Improving the Practice through a Systems Approach (US National Institute of Standards and Technology & National Institute of Justice, 2012).

Standards

‘Often there are no standard protocols

governing forensic practice in a given

discipline. And, even when protocols are in

place, they often are vague and not enforced in

any meaningful way. … These shortcomings

obviously pose a continuing and serious threat

to the quality and credibility of forensic science

practice.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009)

3.

Admissibility (s79) and the form of the

evidence

‘The admissibility of forensic science and medicine evidence

under the Uniform Evidence Law’ (2014) 38 Criminal Law Journal

Admissibility under the Evidence Act 1995 (NSW) 76 Opinion Rule: opinion evidence is presumptively inadmissible

79 Exception: opinions based on specialised knowledge

(1) If a person has specialised knowledge based on the person’s training, study or experience, the opinion rule does not apply to evidence of an opinion of that person that is wholly or substantially based on that knowledge.

Dasreef v Hawchar [2011] HCA 21

‘A failure to demonstrate that an opinion

expressed by a witness is based on the

witness’s specialised knowledge based on

training, study or experience is a matter

that goes to the admissibility of the

evidence, not its weight.’

High Court s79 jurisprudence: two ‘criteria’

‘To be admissible under s 79(1) the evidence that is tendered must satisfy two criteria. The first is that the witness who gives the evidence “has specialised

knowledge based on the person’s training, study or

experience”; the second is that the opinion expressed in evidence by the witness “is wholly or substantially

based on that knowledge”.’

Dasreef Pty Ltd v Hawchar (2011) 243 CLR 588 [32]. See also

Section 79(1) requires the following:

1. specialised knowledge has to be based on

training, study or experience

2. opinion has to be wholly or substantially based on specialised knowledge

Opinion  specialised knowledge  training study or experience

NOT opinion based on experience

R v Tang [2006] NSWCCA 167 at [137]: ‘The focus must be on the words “specialised knowledge”, not on the introduction

of an extraneous idea such as “reliability”.’

‘one and the same’ and

‘unique identifiers’

Honeysett v The Queen [2014] HCA 29

Honeysett v The Queen [2014] HCA 29, [23]

‘“Specialised knowledge” is to be distinguished from matters of "common knowledge”. … It may be of matters that are not of a scientific or technical kind and a person without any formal

qualifications may acquire specialised knowledge by experience. However, the person’s training, study or experience must

result in the acquisition of knowledge. The Macquarie Dictionary defines “knowledge” as “acquaintance with facts, truths, or principles, as from study or investigation” (emphasis added) and it is in this sense that it is used in s 79(1). The

concept is captured in Blackmun J’s formulation in Daubert: “the word ‘knowledge’ connotes more than subjective belief or

unsupported speculation. … [It] applies to any body of

known facts or to any body of ideas inferred from such facts or accepted as truths on good grounds”.’

4.

Mandatory and exclusionary discretions

(after Shamouil and XY)

‘Christie, section 137 and forensic science evidence after Dupas v The Queen and R v XY’ (2014) 40 Monash Law Review 389.

Mandatory exclusion (UEL)

137 Exclusion of prejudicial evidence in

criminal proceedings

In a criminal proceeding, the court must refuse

to admit evidence adduced by the prosecutor if

its

probative value is outweighed by the danger

of unfair prejudice to the defendant

.

See also section 135

Tuite v The Queen [2015] VSCA 148 (Victoria)

A challenge to STRmix – used for mixed DNA profiles – via UEL s137.

Maxwell P, Redlich and Weinberg JJA:

‘The obvious risk in a criminal trial when expert evidence is led from a forensic scientist is that a jury will give the

evidence more weight than it deserves. To prevent unfair prejudice of that kind, it is essential that the reliability of expert evidence be established to the court’s satisfaction (under s 137) before it is led. We have concluded that the touchstone of reliability for this purpose is proof of

appropriate validation, both of the underlying science

(where necessary) and of the particular methodology being employed.’

danger of unfair prejudice to the defendant (NSW)

The danger of unfair prejudice is the risk that the fact-finder may use the evidence on an improper basis. It also includes “the danger that on hearing the evidence the fact-finder would be satisfied with a lower

degree of probability than would otherwise be required” (ALRC, 1985 at [644]; ALRC, 2006 at 16.23-16.26). The risk arises when:

•The fact-finder is not placed in a position to rationally evaluate the forensic science.

•Inappropriately deferring to the forensic practitioner.

•Misunderstanding or over-valuing forensic science evidence. •Failing to appreciate the significance of formal evaluation or its absence.

•Relying on general acceptance, longstanding use and previous admission.

•Relying on, or substituting, experience, confidence and demeanour. •Failing to appreciate the corrosive potential of contextual information. •Procedural disadvantage to the defendant.

5.

Prosecutorial obligations

‘(Ad)Ministering justice: Expert evidence and the

professional responsibilities of prosecutors’ (2013) 36 UNSW Law Journal 921

Prosecutors have serious obligations (that

apply to expert evidence)

‘It cannot be over-emphasised that the purpose of a

criminal prosecution is not to obtain a conviction, it is to lay before a jury what the Crown considers to be

credible evidence relevant to what is alleged to be a crime. Counsel have a duty to see that all available legal proof of the facts is presented: it should be done firmly and pressed to its legitimate strength but it must also be done fairly. The role of prosecutor excludes any notion of winning or losing …’

‘A prosecutor is a ‘minister of justice’. The

prosecutor’s principal role is to assist the court

to arrive at the truth and to do justice between

the community and the accused according to

law and the dictates of fairness.’

Office of the Director of Public Prosecutions (NSW),

‘Prosecution Guidelines of the Office of the Director of Public Prosecutions for New South Wales’ (1 June 2007) Guideline 2.

The New South Wales Barristers’ Rules include:

Rule 82. A prosecutor must fairly assist the court to arrive at the truth, must seek impartially to have the whole of the relevant

evidence placed intelligibly before the court, and must seek to assist the court with adequate submissions of law to enable the law properly to be applied to the facts.

Rule 83. A prosecutor must not press the prosecution's case for a conviction beyond a full and firm presentation of that case.

Rule 84. A prosecutor must not, by language or other conduct, seek to inflame or bias the court against the accused.

Rule 85. A prosecutor must not argue any proposition of fact or law which the prosecutor does not believe on reasonable

grounds to be capable of contributing to a finding of guilt and also to carry weight.

6.

Codes of conduct for expert witnesses

‘Model forensic science’ (2016) 48 Australian Journal of Forensic Sciences (forthcoming).

Practice Note: Expert Evidence in Criminal Trials

(Victoria)

Purpose

To enhance the quality and reliability of expert evidence relied on by the prosecution and the accused in criminal trials …

Expert’s duty to the Court

2.1 An expert has an overriding duty to assist the Court impartially, by giving objective, unbiased opinion on

matters within the expert’s specialised knowledge.

2.2 This duty overrides any obligation to the

commissioning party or to the person by whom the expert is paid.

Practice Directions: Content of all expert reports

4.1 All expert reports to which this Practice Direction applies

(including primary expert reports and responding expert reports) shall state the opinion or opinions of the expert and shall state, specify or provide— …

(c) whether and to what extent the opinion(s) in the report are based on the expert’s specialised knowledge, and the training, study experience on which that specialised knowledge is based;

(d) the material, observed facts, reported facts, assumed facts and other assumptions on which each opinion expressed in the report is based (a letter of instructions may be annexed); …

(f) (if applicable) that a particular question, issue or matter falls outside the expert's specialised knowledge;

(g) any examinations, tests or other investigations on which the

expert has relied, identifying the responsible laboratory by which, and the relevant accreditation standard under which, the

Practice Directions: Content of all expert reports

4.1 All expert reports to which this Practice Direction applies (including primary expert reports and responding expert reports) shall state the opinion or opinions of the expert and shall state, specify or provide— …

(i) any qualification of an opinion expressed in the report, without which the report would or might be incomplete or misleading;

(j) any limitation or uncertainty affecting the reliability of

(i) the methods or techniques used; or (ii) the data relied on, to arrive at the opinion(s) in the report; and

(k) any limitation or uncertainty affecting the reliability of the opinion(s) in the report as a result of—

Practice Directions: Content of all expert

reports

4.2 Where an expert is aware of any significant and recognised disagreement or controversy within the relevant field of specialised knowledge, which is directly relevant to the expert’s ability, technique or

opinion, the expert must disclose the existence of that disagreement or controversy.

7.

Other trial safeguards:

Cross-examination, expressions, rebuttal

experts, directions and warnings

‘How to cross-examine forensic scientists: A guide for lawyers’ (2014) 39 Australian Bar Review 174.

Other safeguards: Trials and appeals

•Admissibility standards and exclusionary rules •Prosecutorial restraint (Wood v R)

• Prosecutorial obligations (as a ‘minister of justice’) • Expert witness obligations and duties (and oath)

• Restrictions on expressions (e.g. similarity

evidence)

• Cross-examination

• Defence (‘rebuttal’) experts

• Judicial directions and warnings • Jury (and trial judge)

• Standard (i.e. BRD) and burden of proof • Appellate review

Trial safeguards?

‘_{Cross-examination, the adduction of contrary expert} evidence and judicial guidance at the end of the trial are currently assumed to provide sufficient safeguards in relation to expert evidence, by revealing to the jury factors adversely affecting reliability and weight.

However, … it is doubtful whether these are valid assumptions.’

Law Commission of England and Wales, Expert evidence

in criminal proceedings in England and Wales (2011)

[1.20]-[1.21], [3.4]. See also The Rt Hon Sir Brian Leveson,

Review of the Effectiveness of Criminal Justice (London,

Expression of opinion

(most reliable declining)

•DNA profiles: probabilistic (usually frequentist).

(Validated – derived from mainstream scientific research) •Latent fingerprints: positive ID (i.e. individualisation). •Ballistics: individualisation to a specific weapon. (NV) •Incriminating images (e.g. CCTV): No individualisation, just similarities (and, in theory, differences). (NV)

•Incriminating voice recordings: individualisation, even across languages. (NV)

•Microscopic hair comparison: historically, very similar

or the same and cannot exclude (Invalid)

•Bite marks: historically individualisation. (Invalid)

Recommendation 3.7: [E]mpirical evidence and statistical reasoning do not support a source attribution

… latent [finger]print

examiners should not report or testify, directly or by implication, to a source attribution to the exclusion of

all others.

McQuiston & Saks, (2008) 59 Hastings Law Journal 1159

Lay interpretations (of expressions)

8.

System issues – can the evidence be

rationally evaluated?

‘Forensic science evidence and the conditions for rational (jury) evaluation’ (2015) 39 Melbourne University Law Review 75-123.

Makita from Hillstead v R [2005] WASCA 116

‘It is a primary duty imposed on experts in

giving opinion evidence to furnish the trier of

fact with the criteria to enable the evaluation of

the expert conclusion: Makita (Australia) Pty

Ltd v Sprowles. … The ‘bare ipse dixit’ of a

scientist upon an issue in controversy should

carry little weight: Davie v Magistrates of

10. Legal versus non-legal approaches to

expertise

‘Legal versus non-legal approaches to forensic science evidence’ (2016) 20 International Journal of Evidence & Proof 3

Demonstrable reliability is the most important

issue for scientific (and non-scientific) expert

evidence, especially forensic science.

For forensic science evidence (esp. comparison

techniques), obtain clear answers to:

• Can the witness do the specific thing claimed and how do we know? How accurate are they and how do we know? What are the limitations with this

technique?

• In the absence of this information, it is far from obvious that opinions are actually expert or

(in)Appropriate heuristics?

According to courts:

‘field’, formal

qualifications, experience,

science/non-science, novelty, partisanship, previous

admission (in Australia or elsewhere), jury

can’t do it themselves, no identified error.

According to scientists:

validation and

reliability, uncertainty and error rates,

empirically-based standards, elimination of

contextual bias, publication, standardise

expressions, independence from law

enforcement.

GFMT scores for Australian passport officers

Normative (students):

M = 81.3%; SD = 9.7

Passport Officers:

M = 79.2%; SD = 10.4

Legal performance

•No court in Australia has produced anything remotely

resembling the kinds of systematic criticisms and concerns identified in the NAS and other reports. Trial safeguards

have not consistently identified, let alone conveyed, the most serious evidentiary limitations with forensic science evidence.

•Legal rules and concerns are frequently displaced from the best independent scientific advice.

•Trial safeguards cannot overcome evidentiary deficiencies – such as inadequate research.

•Awareness of serious evidentiary and legal system

limitations does not seem to be part of legal consciousness (or practice).

NSW is ‘out of step’ with comparator jurisdictions • US (federal and most states) – valid and

reliable

Daubert, Kumho, Federal Rules of Evidence r 702

• Canada – ‘threshold reliability’

Trochym, JLJ, Abbey, DD and the Goudge Report

• England – ‘sufficiently reliable’

Law Commission report (20110, Criminal

Procedural Rules (2014), Part 33; Leveson review (2015).

Issues from overarching trial principles

•What value does ‘expert’ opinion of unknown value have in a rational system of justice?

•How is the jury supposed to gauge the value of opinions in the absence of information about whether the technique works, its limitations, an indicative error rate and insight into the proficiency of the ‘expert’?

•Is the performance of an ‘expert’ under

cross-examination, or her credibility and demeanour, or

confidence, meant to operate as a surrogate to formal evaluation of techniques and procedures?

•Why should the accused by obliged to identify, unpack and explain limitations and oversights with untested

techniques (in routine use)?

•Should legal practice be so divergent from mainstream science? Is there a plausible rationale?