From fact to opinion 1. Managing the change from fact to opinion.

(1)

1

International Workshop on Biometrics and Forensics (IWBF)

Lisbon, April 4-5, 2013

Introducing a LR-based

identification system in

forensic practice:

opportunities and challenges

Prof. Christophe Champod

IC 1106

Objectives of this presentation

•  Expose my views on the opportunities and

challenges a probabilistic (LR-based) assessment of the weight of evidence may pose in traditional forensic identification areas.

•  Focus on one area: fingerprint comparison. But the considerations will apply to other areas.

•  More emphasis is put on the perspective of the

fingerprint examiner and organisations than on the perspective of the engineer / computer scientist.

2

A few themes

1. Managing the change from “

fact

” to

“

opinion

”.

2. Managing the arrival of

probabilistic

models, backing up expert’s opinion

and conflict resolution.

3. Q/A tools

and saying more from

“

inconclusive

” cases.

3

From

fact

to

opinion

•  Saks MJ, Koehler JJ. The individualisation Fallacy in Forensic Science Evidence. Vanderbilt Law Review. 2008;61:199-219.

•  Saks MJ, Faigman DL. Failed Forensics: How Forensic Science Lost Its Way and How It Might Yet Find It. Annual Review of Law and Social Science. 2008;4(1):149-71.

•  Cole SA. Forensics Without Uniqueness, Conclusions Without Individualization: the New Epistemology of Forensic Identification. Law Probability and Risk. 2009;8(3):233-255.

•  State of Maryland v. Bryan Rose (2007) Circuit Court for Baltimore County, Case No. K06-545. Judge Susan Souder: “The 100 percent certainty expressed by Mr. Meagher in this case, as well as in other forum, and others has been persuasively questioned by some academics and defense counsel. The absolute certainty has been proved to be wrong in the past.”

(2)

Individualization: NAS report

•  Difficult to report?

– p.112: Rather than claiming absolute truth, science approaches truth either through breakthrough discoveries or incrementally, by testing theories repeatedly.

– P.142: Although there is limited information about the accuracy and reliability of friction ridge analyses, claims that these analyses have zero error rates are not scientifically plausible.

– P.184:The concept of individualisation is that an object found at a crime scene can be uniquely associated with one particular source. By acknowledging that there can be uncertainties in this process, the concept of “uniquely associated with” must be replaced with a probabilistic association, and other sources of the crime scene evidence cannot be completely discounted.

5 National Research Council, Strengthening Forensic Science in the United States: A Path Forward. Washington, D.C.: The National Academies Press, 2009.

The challenge

“Radical skepticism of all possible assertions of uniqueness is not justified, and the optimal format for explaining the logical impact of a match is not self-evident. But it is clear that one

way or another, a less absolutist and more nuanced theory of

identification is essential if forensic scientists are to adhere to scientific precepts and to contribute to the just resolution of criminal cases.”

6 Kaye DHScience Evidence: Listening to the Academies. Brooklyn Law . Probability, Individualization, and Uniqueness in Forensic Review. 2010; 75: 1163-1185.

Recent changes in reporting

7

“Individualization of an impression to one

source is the

decision

that the likelihood the

impression was made by another (different)

source is

so remote

that it is considered as a

practical impossibility

”

SWGFAST Document #10 Standard for Examining Friction Ridge Impressions and Resulting Conclusions, Ver. 2.0,

http://www.swgfast.org/documents/examinations-conclusions/121124_Examinations-Conclusions_2.0.pdf

Just an Opinion ?

8 A. Campbell, The Fingerprint Inquiry Report.

Edinburgh: APS Group Scotland, 2011.

2.45 - The

decision

whether or not

a mark can be individualised is

potentially a complex one calling

for a series of subjective

judgments on the part of the

examiner. The

decision

is one of

(3)

3 Opinionization

We should require more than mere

ipse dixit

S.A. Cole, The 'Opinionization' of Fingerprint Evidence, BioSocieties. 3 (2008) 115-113. “To begin with, there is a sense in which ‘opinion’ can become an all-encompassing shield that deflects all accountability.”

9

Recent changes in reporting

10

“Individualization of an impression to one

source is the

decision

that the likelihood the

impression was made by another (different)

source is

so remote

that it is considered as a

practical impossibility

”

SWGFAST, Standards for Examining Friction Ridge Impressions and Resulting Conclusions, ver. 1.0.

That opens a brand new door on

…

11

http://leadershipfreak.files.wordpress.com/ 2010/03/open-door.jpg

probabilities

Greener grass, blue sky, really?

Your

statement

never mention that

adverse probability, why ?

I would

not

expect this

on another individual ?

What do you mean by

A Decision

?

A

likelihood

so remote ?

Practical impossibility

?

(4)

Just an Opinion ?

Edinburgh: APS Group Scotland, 2011.

38.24 - What matters

more than

the choice of language

(whether

the witness says that he is

‘confident’, ‘sure’, ‘certain’ or ‘in

no doubt’) is the

transparency

of

the opinion.

Transparent decision process

Weight of the forensic findings (MP < 10-15₎

ID decision – An adventitious match is a practical impossibility Utility All fingers on Earth “Leap of faith” (D. Stoney) Decision

1/all fingers LR for FP comparison 99.99…%

14 A. Biedermann, S. Bozza, and F. Taroni, "Decision theoretic properties of forensic identification: Underlying logic and argumentative implications," Forensic Science International, vol. 177, pp. 120-132, 2008.

How that decision is taken?

Expert Working Group on Human Factors in Latent Print Analysis, Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach. Washington DC: U.S. Department of Commerce, National Institute of Standards and Technology, 2012, p. 68 Framework (priors) Earth population paradigm Evidence Two generic questions forming a likelihood ratio Update (posteriors) Require both the priors and the evidence Decision on the ID or Exclusion

Based on the posterior probabilities and an utility function

15

What do you know about:

16

Relative frequencies

of level

1 features (general pattern)

and their use ?

Relative occurrence of

level 2 features

(

minutiae

) ?

Level 3 (

pores

) ?

And it goes beyond that…

(5)

5 Requirement for training

17

Regardless of the availability of statistical models to assess fingerprint comparison, there is an urgent need for fingerprint experts

to understand and to be able to articulate in court and in their statements the probabilistic nature of their decisions.

Equilibrium in the

spectrum

of knowledge

•  Years of experience •  Unstructured data collection from uncontrolled casework •  Relevant systematic studies (published or documented) •  Structured portfolio of cases •  Proficiency and collaborative testing

Probabilistic vs unfettered opinions

18

Arrival of probabilistic models

•  Help assign the weight of evidence to the whole

configuration without decomposing the contribution of its individual minutiae.

•  The more recent efforts have been successfully presented to the Royal Statistical Society: C. Neumann, I. W. Evett, and J. Skerrett, "Quantifying the weight of evidence from a forensic fingerprint comparison: a new paradigm," Journal of the Royal Statistical Society, vol. 175, pp. 371-415 (with discussion), 2012.

19

Neumann & al. (2012)

8 C. Neumann, I. W. Evett and J. Skerrett

(a)

(b)

Fig. 2. Illustrations of the radial triangulation used to orderminutiae(the variables listed in Table 1 are indicated in grey in (a);!, ‘directions’ of theminutiae): the type of theminutiais not indicated since it is a dis-crete variable; depending on distortion, radial triangulation ofminutiaeconfigurations may produce different ordered sets; the differences between (a) and (b) are presented by using thinner lines

unique, rotation invariant and robust to fingerprint distortion. For a given configuration the method requires the determination of a unique centre for the polygon, defined by the arithmetic

mean of the Cartesian co-ordinates of theminutiae. The polygon is then defined by connecting

theminutiae, which become the vertices, in clockwise order with respect to the centre of the

polygon (Fig. 2). Eachminutiais connected to two contiguousminutiaeand also to the centre

of the polygon, hence creating a series of contiguous triangles, which all share one vertex located

in the centre of the polygon. Providing that theminutiaeare always considered in the same order

with respect to the centre, the capture of information for a given configuration is invariant to the orientation of the fingerprint on the image. Furthermore, the circular acquisition of the minutiaerenders the acquisition process independent of the choice of startingminutia. Table 1

provides the notation for the data that are extracted from theminutiae. Note that the directions

of theminutiaeare recorded along the supporting ridge for ridge ending, and along the bisector

of the splitting ridges for a bifurcation (Fig. 2).

4.2. Organization of minutiae features

For a configuration ofk minutiaethere are 5kfeatures that are recorded in a feature array. A

fea-ture array that is recorded from a randomly selectedminutia, which is subsequently denoted 1,

may be written (following the notation in Table 1) w.k,1/₌_.

{δi,σi,θi,αi,τi,},i=1, 2, . . . ,k/:

We usewin this context because we may be considering a marky, printxdatabase member

zor a simulated mark (for the notation, see later). Note thatw.k,1/_{is an ordered set. The array}

that would be created by starting at the next (clockwise)minutia, which is denoted 2, would be

w.k,2/₌_._{_δ i,σi,θi,αi,τi,},i=2, 3, . . . ,k, 1/: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Quantifying Weight of Evidence from Fingerprint Comparison 21

3 4 5 6 7 8 9 10 11 12 −15 −10 −5 0 5 10 15 20 25 Log 10 LR (H d ) Number of minutiae 3 4 5 6 7 8 9 10 11 12 −15 −10 −5 0 5 10 15 20 25 Log 10 LR (H p ) Number of minutiae (b) (a)

Fig. 5. Distributions for LR.Hp/and LR.Hd/from the large validation study, for configurations of sizes kD3–12: (a) LRs whenHpis true; (b) LRs whenHdis true

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 20

To be part of the accreditation scope, regardless of its scientific merits, it will require operational validation

(6)

How LRs are assigned (generic)

21 J. Abraham, C. Champod, C. Lennard, and C. Roux, "Spatial Analysis of Corresponding Fingerprint Features from Match and Close Non-Match Populations," Forensic Science International, in press.

LR=P s | Genuine

(

)

P s | Impostor

₍

₎

The conditioning propositions are not so trivial as Genuine

and Impostor

Forensic Error Rates

RMED=3.4% RMEP=3.2% What is an acceptable error rate? We should confirm them through an operational validation Independent sets of marks and prints of known sources are required for an operational validation

Models can be used in two ways:

23

41.32 […] to provide background data

to assist fingerprint examiners with their evaluation of marks and to enable them to express the strength of their

conclusion in a transparent and verifiable manner.

A. Campbell, The Fingerprint Inquiry Report. Edinburgh: APS Group Scotland, 2011.

Backing up examiners ?

24

We need to precisely define how these “experts” will operate.

A set of SOPs need to be drafted before any operational implementation

To embrace it, I need to understand and

trust the model I need to be able to explain its meaning and limitations

I may want to identify regardless of the number given by the

model Models are

inevitable in the future

As in DNA, probabilities will be asked by both prosecution and defence

(7)

7 Pilot joint project

25

The

situation

we want to handle

26

My opinion is that the mark has been identified

to the right thumb of Mr X.

The probability for the mark to originate from someone else is so small

that I consider it to be a

practical impossibility.

We submitted your case a statistical analysis through the University XYZ, Prof S. Tat

The LR obtained is 1.8e+6, that amounts to a match probability of 5.6e-7

How do you get from a match probability of 5.7e-7 to an identification?

Expert 1 (LR = 5.8e10

5

₎

27

Expert 2 (LR = 9.6e10

10

₎

(8)

Conflict resolution procedures

29 Decision First examiner Statistical Model Second examiner (verifier)

A quick look at the task complexity

Understanding the concept of “sufficiency” in friction ridge examination

C. Neumann, C. Champod, M. Yoo, G. Langenburg, T. Genessay, NIJ award - 2010-DN-BX-K267

Results presented at the annual meeting of the American Academy of Forensic Science, Feb 2013.

30

Understanding “Sufficiency”

•  15 comparisons:

– 12 pairs latent/control prints from same source – 3 pairs latent/control prints from different sources

•  Information captured through a web-based software

designed to support the ACE process (Picture Annotation Software – PiAnoS –

https://ips-labs.unil.ch/pianos/index.html) •  Approximately 600 examiners contacted

– 145 completed first comparison – 123 completed all 15 comparisons

31

PiAnoS'

(9)

9 PiAnoS'

33 Minutiae annotated Minutiae annotated Change of size of displayed minutiae markers Change of size of displayed minutiae markers

PiAnoS'

34

PiAnoS'

35 120 4 21 127 11 120 17 7 3 126 34 7 93 29 6 98 73 13 45 43 19 66 2 106 19 100 4 21 13 9 103 11 73 40 76 8 38 72 18 33 28 5 90

Trial 01 same source Trial 02 different sources Trial 03 same source

Trial 04 same source Trial 05 same source Trial 06 same source

Trial 07 same source Trial 08 same source Trial 09 different sources

0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150

ID EXC INC ID EXC INC ID EXC INC

Decisions following comparison

N umb er of re sp on da nt s

(10)

04.04.13 Trial 8 (same source)

Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading

Categorical Towards Neutral Categorical Towards Neutral Categorical Towards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

Categorical Towards Neutral Categorical Towards Neutral CategoricalTowards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

Categorical Towards Neutral CategoricalTowards Neutral Categorical Towards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

Guiding correctly Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading

Categorical Towards Neutral CategoricalTowards Neutral Categorical Towards Neutral

Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

CategoricalTowards Neutral CategoricalTowards Neutral CategoricalTowards Neutral

Trial 12 (different sources)

Categorical Towards Neutral Categorical Towards Neutral Categorical Towards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

CategoricalTowards Neutral Categorical Towards Neutral Categorical Towards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

Categorical Towards Neutral CategoricalTowards Neutral Categorical Towards Neutral Level of confidence R el ia bi lit y of co ncl usi on Decision in analysis NV VID VEO Conclusion Exclusion Identification Inconclusive

Guiding correctly Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading Guiding correctly Neutral Misleading

Categorical Towards Neutral CategoricalTowards Neutral Categorical Towards Neutral

Conflict resolution procedures

39 Decision First examiner Statistical Model Second examiner (verifier)

Other usage: Q/A Tool ?

(11)

11 Other usage: Suitability tool?

41 Image from the FIGS group, Adrian Gardner

The expected LR associated with the 5-6 level 2 features is 12’540.

Is the mark suitable for further comparison,…, for identification? Is that a complex case?

Models can be used in two ways:

42

41.33 […] the possible application of probabilistic analysis to comparisons that examiners would otherwise consider to be inconclusive, the objective being to produce evidence that could be used in court of the probability of a match.

A. Campbell, The Fingerprint Inquiry Report. Edinburgh: APS Group Scotland, 2011.

“Unable to exclude” cases

Edinburgh: APS Group Scotland, 2011. 38.84 - Before a finding of ‘unable to exclude’ is led in evidence, careful consideration will require to be given to (a) the types of mark for which such a finding is meaningful and (b) the

proper interpretation of the finding. An examiner led in evidence to support such a finding will require to give a

careful explanation of its limitations.

Saying more than “inconclusive”

44

Again, we need to precisely define the scope of usage

The statistics may convey more weight than it deserves

Very useful source of additional information, either as evidence or for intelligence purposes

We don’t want to mislead anyone

Already a reality for the Dutch NFI

(12)

Value for comparison

Searchable on IAFIS

Value for identification

Diminishing match probabilities (or increasing LR) IDEN

T IF IC A T O N D EC ISI O N 45

Risks of misleading may outweigh benefits

Helps with the decision making Intelligence tool Allows transparency but marginal impact on decision making [1 – 1/10-3_] _[1/10-3_-1/10-9_] _[<10-9_]

Need more resources Need more resources Actual resources

Complex Non-complex

Managing the change

46

Educate Judiciary and field officers

Train fingerprint examiners Define the

exact scope

Define the SOPs

Conduct operational validation Agree on acceptable error rates Conflict resolution procedures

Contact details

Prof. Christophe Champod

Ecole des sciences criminelles / Institut de police scientifique Batochime / Quartier Sorge

CH-1015 Lausanne

Tel: +41 (0)21 692 46 29 Fax: +41 (0)21 692 46 05 E-mail: christophe.champod@unil.ch