THE PREDICTIVE CODING CASES A CASE LAW REVIEW

THE PREDICTIVE CODING CASES

WELCOME

 Thank you for joining

 Numerous diverse attendees

 Please feel free to submit questions

SPEAKERS

 Matthew Verga

AGENDA

 Introduction

 The Predictive Coding Cases:

 Key Takeaways

INTRODUCTION

 Since Monique da Silva Moore, et al., v. Publicis Groupe SA & MSL Group – No universal technical or legal standard has yet emerged,

– Each new order provides insights into what parties are attempting, how they’re defending it, and how courts and parties opponent are reacting

 The total number of such cases is still small, but grows each year. In Gibson Dunn’s “2014 Year-End E-Discovery Update” (January 20, 2015), they count:

– 6 such decisions in 2012, – 9 such decisions in 2013, and – 17 such decisions in 2014

 Many make only passing mention of predictive coding; some engage in extensive, educational discussion of issues

MOORE

 Monique da Silva Moore, et al., v. Publicis Groupe SA & MSL Group,

No. 11 Civ. 1279 (ALC)(AJP) (S.D.N.Y. Feb. 24, 2012)

– The first case in which the use of predictive coding was judicially approved – The primary discovery issue in question was the appropriate methodology

by which to evaluate approximately 3,000,000 emails the Defendants had gathered

 The Defendants proposed using a predictive coding tool and a

methodology that included transparency and disclosure of materials

 The Plaintiffs did not oppose the use of predictive coding, in theory,

but they opposed some specifics of the methodology proposed

MOORE

 The Plaintiffs’ also objected the “method lacks the necessary standards for assessing whether its results are accurate; in other words, there is no way to be certain if [the Defendants’] method is reliable”

– Accused him of “simply kicking the can down the road”

 Magistrate Judge Peck reminds everyone that proportionality is fact-based – Real information about prevalence, cost, and relevance must be considered:

o “In order to determine proportionality, it is necessary to have more information than the parties (or the Court) now has, including how many relevant documents will be produced and at what cost . . . In the final sample of documents deemed irrelevant, are any relevant documents found that are ‘hot,’ ‘smoking gun’ documents (i.e., highly relevant)? Or are the only relevant

documents more of the same thing? One hot document may require the software to be re-trained (or some other search method employed), while several documents that really do not add anything to the case might not matter. These types of questions are better decided ‘down the road,’ when real information is available to the parties and the Court.”

MOORE

 Ultimately, Magistrate Judge Peck approved

– And that approval was upheld on review by District Judge Carter

 In his order, Magistrate Judge Peck writes that his decision to

approve was based on his consideration of these factors:

– “. . . (1) the parties' agreement, (2) the vast amount of ESI to be reviewed (over three million documents), (3) the superiority of computer-assisted review to the available alternatives (i.e., linear

manual review or keyword searches), (4) the need for cost effectiveness and proportionality under Rule 26(b)(2)(C), and (5) the transparent

KLEEN PRODUCTS

 Kleen Products LLC, et al., v. Packaging Corporation of America, et al., No. 10 C 5711 (N.D. Ill. Aug. 21, 2012)

 The Defendants wanted to use a Boolean search method

– The Defendants had iteratively tested and refined search terms, used sampling, etc. – The Defendants contended that their search process would be tested and validated

to ensure accuracy as good as, or better than, predictive coding or any other option  The Plaintiffs criticized Boolean search methods as inherently inadequate

and lobbied the Court to require the use of predictive coding instead

– They argued the Defendants’ searches would find less than 25% of the responsive documents, while predictive coding would find more than 70%

KLEEN PRODUCTS

 Hearings were held regarding the proposed search methodologies  Rather than directing the parties to adopt a particular resolution, the

Magistrate Judge emphasized her support for Sedona Principle 6, which states: – “Responding parties are best situated to evaluate the procedures, methodologies, and

technologies appropriate for preserving and producing their own electronically stored information.”

 The Magistrate Judge urged the parties to consider whether the Defendants’ Boolean search methodology might be refined or supplemented in some way that would satisfy the Plaintiffs

 Eventually, the parties reached an agreement regarding search methodology for the first phase of discovery, with the Plaintiffs dropping their demand for the Defendants to employ predictive coding

KLEEN PRODUCTS

 Note 1: Educational hearing transcripts – February 21, 2012 Hearing Transcript

– March 28, 2012 Hearing Transcript (Ex. A to Joint Status Conference Report No. 3)  Note 2: Producing party gets deference about methodology

– “I am a believer of principle 6 of Sedona, and I’m not just because it’s Sedona, but I think the people who are producing the records, producing the documents, are in a better position to know [emphasis added], since they have to do the work, spend the money, spend the time, they know their people, they know their material, so as a basic premise, I think that’s a pretty fair premise here.”

 Note 3: Verifiably complete results are more important than which process – The Magistrate Judge would “almost call [predictive coding] a detour”

– “I assume . . . what you really are interested in is a search, regardless if it’s Boolean or computer-assisted [emphasis added], that is fair and statistically – and that can be validated statistically because that would be a good word search.”

GLOBAL AEROSPACE

 Global Aerospace Inc. v. Landow Aviation, L.P., No. 61040 (Loudoun

County, Va. Cir. Ct. Apr. 23, 2012)

– The first case in which the use of predictive coding was judicially permitted over the requesting parties’ objections

 The Defendants collected over 8TB of data and identified a pool of

approximately 250GB of reviewable data representing in excess of

2,000,000 unique documents requiring review

– Due to the size of this dataset, the Defendants proposed using predictive coding rather than either: a complete, linear first-pass review, or the use of keyword searching

– The Plaintiffs did not agree to the Defendants’ proposal, and so, the

Defendants moved the Judge for a protective order allowing them to use predictive coding

GLOBAL AEROSPACE

 Using the Moore case and published studies, they demonstrate superiority

– Additionally, they provide the results of preliminary testing done on proposed search terms to demonstrate concretely the extent of their imprecision and inconsistency, in this case, on this data  They offer a detailed protocol, including transparency and cooperation

– Unlike in Moore, they commit themselves to achieving at least 75% recall at the end of the process  They position their request well within the existing procedural rules:

– First, they make a proportionality argument based on the relative time and cost of manual review – Second, they explain why achieving 75% recall is adequate to satisfy the “reasonable inquiry”

requirement, and how this already exceeds what has been traditionally expected:

“Ironically, what is being proposed in this case to ensure ‘reasonable inquiry’ is far more than has ever been done with traditional discovery [emphasis added]. Never has it been suggested that a producing party would be obligated to sample the documents determined to be irrelevant by first-pass reviewers to demonstrate the adequacy of that first-pass review. Nor is it a typical practice to review the documents left behind by a keyword search, even though upwards of 80% of the relevant documents typically are missed. The ESI protocol proposed by [the Defendants] goes well beyond what would otherwise be available to opposing counsel to ensure that relevant documents are being reviewed for production.”

GLOBAL AEROSPACE

 The Plaintiffs principal objection was to the minimum recall of 75%

– The rules require the production of all responsive materials, not just ¾ of them – Unlike in Moore, the Plaintiffs did not just want the protocol changed, they wanted

traditional human review methods used instead of any predictive coding process  The Judge’s order allowed the Defendants to proceed, without discussion  However, this case does provide actual completed predictive coding results:

– The Defendants’ validation steps determined they had a recall rate of 81%

– They also estimated their elusion rate at 2.9%, which despite being low, meant that more than 32,000 responsive documents may have been missed by the process – No timely objections to the resulting production were made by the Plaintiffs so no

IN RE: ACTOS

 In Re: Actos (Pioglitazone) Products Liability Litigation, No.

6:11-md-2299 (W.D. La. July 27, 2012)

– The first example of Defendants and Plaintiffs cooperatively reaching agreement on a predictive coding protocol that was then judicially approved on the record

– Memorialized in a Case Management Order July 27, 2012

o Proof-of-concept nature

o Transparency and cooperation o Validation

IN RE: ACTOS

 Proof of Concept

– Designed as a “proof of concept” to see whether or not the predictive coding tool (Equivio Relevance, in this case) is viable for the case as a whole

– First applied to a “sample collection population” comprised of the e-mail of four mutually-chosen key custodians, plus selected documents – Allows both parties a lower-risk way to experiment without having to

IN RE: ACTOS

 Transparency and Cooperation

– Emphasis on transparency for the Plaintiffs and on cooperation and collaboration between the parties

– Control set, training set, and validation set review and coding would be shared with the Plaintiffs, and their agreement on coding would be sought:

o The Plaintiffs and the Defendants would each get to select three subject matter experts to collaborate on the relevance coding

o If the experts cannot agree on relevance, the parties must meet and confer; and, if the parties cannot agree on relevance, the Court can be involved

o The Plaintiffs and the Defendants also agreed to meet and confer, after the system is trained, to mutually determine the relevance score at which to cut off review

IN RE: ACTOS

 Validation

– Validation of the process and the final protocol results before acceptance:

o A 95% confidence level and a margin of error of +/-5%

o Collaboratively review a sample of the documents below the cutoff score (i.e., those deemed not relevant by the software and excluded from subsequent human review) o Collaboratively review a sample of documents above the cutoff score but coded as not

relevant during review (i.e., those for which human reviewers overruled the software)

 Good example of detailed discovery planning and of how to

approach discovery and the use of predictive coding cooperatively

IN RE: BIOMET

 In Re: Biomet M2a Magnum Hip Implant Products Liability

Litigation, No. 3:12-MD-2391 (N.D. Ind. Aug.21, 2013)

– Settled in February 2014 for approximately $56 million

o Before settlement two noteworthy orders were issued regarding predictive coding

– The Defendants collected 19.5 million documents

o They used keyword searches to cull that down to 3.9 million documents

IN RE: BIOMET

 The Defendants also used sampling to measure prevalence as they went: – Sampling of the total 19.5 million document pool: The Defendants estimated, with a

99% confidence level, that between 1.37 and 2.47% of this pool was relevant, i.e., that about 267,000 – 482,000 total relevant documents existed

– Sampling of the 15.6 million document pool, post-keywords: The Defendants

estimated, with a 99% confidence level, that between 0.55 and 1.33% of this pool was still relevant, i.e., that about 85,000 – 207,000 total relevant documents were missed by their keyword searches and omitted from review and production

– Sampling of the de-duplicated 2.5 million document pool: The Defendants

estimated, with a 95% confidence level, that between 14.41 and 17.91% of this pool was relevant, i.e., that about 360,000 – 448,000 total relevant documents were

successfully found by their keyword searches

 The Defendants then applied predictive coding (Recommind’s Axcelerate) – The Plaintiffs objected to this process, giving rise to the April 2013 Order

IN RE: BIOMET

 April 2013 Order

– The Plaintiffs did not object to the use of predictive coding itself, but

argued that the Defendants had tainted their predictive coding process by using keywords to cull the population before applying predictive coding

o The Defendants offered to let the Plaintiffs choose additional keywords

o The Plaintiffs wanted the Judge to order them to start over using all 19.5 million

– The Judge, however, did not accept the Plaintiffs framing of the issue:

o “The issue before me today isn’t whether predictive coding is a better way of doing things than keyword searching prior to predictive coding. I must decide whether [the

Defendants’] procedure satisfies [their] discovery obligations and, if so, whether [they] must also do what the [Plaintiffs] seek[].”

– The Judge goes on to reject the Plaintiffs’ request, both because the

IN RE: BIOMET

 August 2013 Order

– The Plaintiffs asked the Defendants to disclose the seed set documents

o The Defendants said they had been produced already but declined to identify them o The Plaintiffs then sought to have the Judge compel the identification

– The Judge, however, concluded that he had no authority to compel anything beyond the production of the discoverable materials:

o “The [Plaintiffs] want[] to know, not whether a document exists or where it is, but rather how [the Defendants] used certain documents before disclosing them. Rule 26(b)(1) doesn’t make such information disclosable.”

– A “troubling” refusal “below what the Sedona Conference endorses”

o “An unexplained lack of cooperation in discovery can lead a court to question why the uncooperative party is hiding something, and such questions can affect the exercise of discretion.”

PROGRESSIVE

 Progressive Casualty Insurance Company v. Delaney, No.

2:11-cv-00678 (D. Nev. July 18, 2014)

– The parties negotiated a “Joint Proposed ESI Protocol,” which the Magistrate Judge approved in an order issued October 24, 2013 – The Protocol specified:

o Agreed upon sources from which the Plaintiff would collect o Agreed upon search terms to run against the collected materials

o An agreement to either produce all the non-privileged results or manually review all the results for responsiveness, privilege, etc., and then produce

PROGRESSIVE

 The collection yielded approximately 1.8 million documents

– The search terms mandated by the protocol returned 565,000 documents – The Plaintiff began manual review but determined completion would take 6-8

months and “an unacceptably high cost”

– The Plaintiff began exploring other options instead, including predictive coding  On December 20, 2013, the Plaintiff notified the Defendants that the

original review plan was not feasible and that it planned to propose an alternative but would not disclose the details of its new plan

 Over the next three months, the parties engaged in a series of hearings with the Magistrate Judge and meetings with each other to negotiate

 Ultimately, the parties were not able to reach agreement, and on March 20, 2014, the parties submitted competing proposals to the Magistrate Judge

PROGRESSIVE

 The Plaintiff’s Proposal

– The Plaintiff had begun working with Equivio’s Relevance software

o After training, the software identified 90,000+ potentially responsive documents o Automated privilege screening suggested that 63,000 of these documents were not

likely to be privileged and 27,000 of them were likely to be privileged

– The Plaintiff proposed that the ESI Protocol be modified to cover the work already performed and allow it to continue with predictive coding

o Its next step would be to produce the 63,000 predictive coding results deemed not likely to be privileged, without further review (relying upon the clawback provisions) o After that, the Plaintiff would manually review the other 27,000 predictive coding

results for privilege and produce the non-privileged documents

o The Plaintiff estimated that this process would end up producing over 70% of the total number of responsive documents in the pool of 565,000 search results

PROGRESSIVE

 The Defendants’ Proposal

– The Defendants opposed the proposal for its departure from the original, negotiated agreement and for three additional reasons:

o Inevitable “satellite disagreements” o Lack of transparency and cooperation

o Failed to follow the software provider’s own best practices

– The Defendants proposed that the Plaintiff be directed to immediately turn over all 565,000 results of the negotiated search terms and to rely on the clawback provisions to get back privileged documents produced

o If predictive coding was permitted, the Defendants proposed that the Plaintiff be required to start over using predictive coding on all 1.8 million documents

PROGRESSIVE

 Predictive coding is good in theory but the terms of the negotiated ESI protocol and the Plaintiff’s attempt to deviate unilaterally govern:

– “In this case, the parties negotiated an ESI protocol which was adopted by the court . . . Had the parties worked with their e-discovery consultants and agreed at the onset of this case to a predictive coding-based ESI protocol, the court would not hesitate to approve a transparent, mutually agreed upon ESI protocol. However, this is not what happened.”

o “[The Plaintiff] proposes a ‘do-over’ of its own invention that lacks transparency and cooperation regarding the search methodologies applied.”

o “[The Plaintiff] is unwilling to engage in the type of cooperation and transparency that its own e-discovery consultant has so comprehensibly and persuasively explained is needed for a predictive coding protocol to be accepted by the court or opposing counsel as a reasonable method to search for and produce responsive ESI.”

 The Magistrate Judge concludes that the Defendants were correct in their prediction of complex, ongoing disputes and adopts the Defendants

proposal, ordering the Plaintiff to produce all 565,000 search results, excepting those identified by the automated privilege screening

PROGRESSIVE

 Although predictive coding was not permitted in this case, it is not really a blow to the further adoption and use of predictive coding

 The Magistrate Judge makes clear that she would have approved its use under different circumstances, particularly had there been more

transparency and cooperation about deviating from the agreed plan

 It is worth noting, however, one small problem with the Magistrate Judge’s commentary on transparency:

– In reviewing the Moore and Actos cases, the Magistrate Judge refers to them as requiring full transparency (e.g., seed set disclosure) for predictive coding’s use – While both cases discuss transparency favorably, the producing parties actually

BRIDGESTONE

 Bridgestone Americas, Inc. v. International Business Machines

Corp., No. 3:13-cv-01196 (M.D. Tenn. July 22, 2014)

– As in Progressive, negotiated, memorialized discovery plan

– Using search terms provided by the Defendant, the Plaintiff found 2,000,000 documents

o Plaintiff asked to be permitted to use predictive coding instead of manual review

– The Defendant objected

o “unwarranted change” in the original order

o “it is unfair to use predictive coding after an initial screening has been done with search terms”

BRIDGESTONE

 Extensive pleadings and a “lengthy telephone conference”

– It boils down to a “judgment call” about efficiency and cost-efficacy:

o “In the final analysis, the uses [sic] of predictive coding is a judgment call, hopefully

keeping in mind the exhortation of Rule 26 that discovery be tailored by the court to be as efficient and cost-effective as possible. In this case, we are talking about millions of

documents to be reviewed with costs likewise in the millions. There is no single, simple, correct solution possible under these circumstances [emphasis added].”

– Magistrate Judge allows the Plaintiff to “switch horses in midstream”

o Emphasizes the importance of transparency and openness (e.g., sharing seed set docs)

o Also extends the option to switch to predictive coding to the Defendant

– Case is ongoing: new order on February 5, 2015

o Seed set: “some of the documents listed as nonresponsive were, in fact, responsive.” o Emphasizes, again, that: “to the extent [the Parties] use predictive coding, he expects full

DYNAMO

 Dynamo Holdings Limited Partnership, et al., v. Commissioner

of Internal Revenue, 143 T.C. No. 9 (Sept. 17, 2014)

– Respondent sought to have the Petitioners produce the ESI stored on two specific back-up tapes or produce copies of the tapes themselves – Petitioners estimated that reviewing the ESI would “take many months

and cost at least $450,000”

o Petitioners asked the Judge to permit them to utilize predictive coding instead

– Respondent objected:

o Predictive coding is an “unproven technology”

DYNAMO

 Should parties seek prior judicial approval for predictive coding?

process and intervene when it is abused by the parties, the Court is not normally in the business of dictating to parties the process that they should use when responding to discovery. If our focus were on paper discovery, we would not (for example) be dictating to a party the manner in which it

should review documents for responsiveness or privilege [emphasis added], such as whether that review should be done by a paralegal, a junior

attorney, or a senior attorney. Yet that is, in essence, what the parties are asking the Court to consider – whether document review should be done by humans or with the assistance of computers. Respondent fears an incomplete response to his discovery. If respondent believes that the ultimate discovery response is incomplete and can support that belief, he can file another motion to compel at that time [emphasis added].”

DYNAMO

 He then addresses the use of computer-assisted review tools and reviews – The testimony of experts for both parties, relevant articles, and cases, including:

Moore, Progressive, and In Re: Actos

 He concludes that predictive coding is not “unproven technology” – But rather, a tool widely accepted in “the technology industry” and

– Accepted in past federal cases (particularly with transparency and cooperation)  He then grants the request to use predictive coding:

– “Where, as here, petitioners reasonably request to use predictive coding to conserve time and expense, and represent to the Court that they will retain electronic discovery experts to meet with respondent’s counsel or his experts to conduct a search acceptable to respondent, we see no reason petitioners should not be allowed to use predictive coding to respond to respondent’s discovery request.”

RIO TINTO

 Rio Tinto PLC v. Vale, S.A., et al., No. 1:14-cv-03042-RMB-AJP

(S.D.N.Y. Mar. 2, 2015)

– Magistrate Judge Peck – Acceptance of a protocol

o Including predictive coding

o To which both the requesting and producing parties stipulated

RIO TINTO

 Trend towards acceptance

– “. . . [i]n the three years since Da Silva Moore, the case law has

developed to the point that it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it [emphasis added]”

 Primary open issue

– “. . . TAR issue that remains open is how transparent and cooperative the parties need to be with respect to the seed or training set(s)”

– “where the parties do not agree to transparency, the decisions are split and the debate in the discovery literature is robust”

RIO TINTO

 Other validation options

– “In any event, while I generally believe in cooperation [footnote omitted], requesting parties can insure that training and review was done

appropriately by other means, such as statistical estimation of recall at the conclusion of the review as well as by whether there are gaps in the

production, and quality control review of samples from the documents categorized as non-responsive [emphasis added].”

 Time to move on

– “One point must be stressed - it is inappropriate to hold TAR to a higher standard than keywords or manual review [emphasis added]. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.”

KEY TAKEAWAYS

 Clear trend towards acceptance – If you seek prior judicial approval

o and you have the agreement of the requesting party, – then you should be able to get that approval

» Moore, In Re: Actos, Rio Tinto

– If you seek prior judicial approval

o but the requesting party opposes its use,

– then you may be able to get that approval anyway,

» if your proposal includes adequate transparency and cooperation » and does not run afoul of any previously approved agreements

• Global Aerospace, Progressive, Bridgestone, Dynamo

– If you seek prior judicial approval

o but the requesting party opposes its use

o and you wish to proceed without transparency or cooperation in the execution, – then you may not receive prior approval

KEY TAKEAWAYS

 More interesting question:

– Highlighted by the judges in Kleen Products, In Re: Biomet, Dynamo, and Rio Tinto

– Should producing parties seek prior judicial approval at all, when they would not do so for traditional methodologies?

– Review of the case law suggests that judges are eager for:

o Producing parties to get back to managing their own processes

o Requesting parties to get back to focusing on the completeness of the results o All parties to get back to working more of it out between themselves

ENGAGE WITH MODUS

 Modus email coming tomorrow with slides, recording, survey and

invite to next webinar

 Visit us online at

http://discovermodus.com/webinars/

for more

information on next webinar on May 27

_{at 1:00PM EDT entitled:}

– “Zeroing In: Data Targeting in eDiscovery to Reduce Volumes and Costs”

 Make sure to visit our website for valuable white papers, regular

blog posts, ongoing surveys and more at

www.discovermodus.com

– Including white paper “The Predictive Coding Cases”