2011 Winston & Strawn LLP

(1)

(2)

Today’s eLunch Presenters

Today s eLunch Presenters

John Rosenthal Litigation Washington, D.C. JRosenthal@winston com Scott Cohen Director of E‐Discovery Support Services New York SCohen@winston com [email protected] [email protected]

(3)

What Was Advertised

 Effective strategies for reducing the cost of an electronic ec e s a eg es o educ g e cos o a e ec o c

document review

 Use of second generation search technologies to both

filter document sets and more efficiently organize the materials for review, including concepting, clustering, threading and near duping technologies

threading and near‐duping technologies

 Implementing best practices in the review process to

ensure a high degree of precision and recall g g p

 Effective strategies for handling privilege documents  Use of lower‐cost reviewers

(4)

Why Should You Listen to Us?

 Spent the last two years studying the e‐discovery p y y g y marketplace and designing a new platform  Benchmarked the entire industry, including other l f d d f d law firms, vendors and software providers  Built an e‐discovery consulting shop behind Winston

& Strawn’s firewall to include: & Strawn s firewall to include:  Collection  Processing  Analytics  Review Platform Review Center

(5)

Electronic Document Review

 Excessive and unpredictable costs:p  58 % to 70 % of total litigation costs  Document review costs are rising due to the increasing amount of electronic information  Traditional document review is not accurate:  Evidence suggest there are high error rates in linear manual review review  Error rates lead to likelihood of inadvertent production of privileged or sensitive information

 Inability to defend the review process:

 Judges are increasingly focusing on the need for validation of

review processes

(6)

Goals of ESI Review

 Recall ‐ Identification and prioritization of relevant material  Recall Identification and prioritization of relevant material  Precision ‐ Elimination of irrelevant/non‐ responsive material  Identification of privileged material Relevant Data N l d i d Retrieved D

Relevant and not retrieved Non‐relevant and retrieved

Data

(7)

Accuracy of Human Review

y

 Recall  Recall Number of responsive documents retrieved Total number of responsive documents in the collection  Precision Number of responsive documents identified Total number of documents retrieved

(8)

Accuracy of Human Review

y

100% _Perfection 80% 90% 100% _Perfection Typical result in a Blair & Maron (1985) 70% 50% 60% cision Typical result in a manual responsiveness review 20% 30% 40% Pr e c 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 10% 0% TREC Best Benchmark (Best performance on Precision at a given Recall)

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

0%

M. Grossman Presentation - Technology-Assisted Review Can be More Effective and More Efficient Than Exhaustive Manual Review

(9)

Traditional Electronic Document

Review = Linear Review

 Over collectionOver collection Traditional Approach

 Little or no culling

 Ad hoc use of Boolean searches Manually Acquire Broad Amounts of Data  Linear review of the data set  Use of traditional associate work f t f i Data Process Data force to perform review Second Level Review First Level Review Produced Documents

(10)

Linear Review

(11)

Traditional Approach = $$$

pp

$$$

 Each custodian = 4 and 8 GB  1 GB of data = 60,000 pages  Average review rate = 350 to 500 pages per hour  1 GB = 120 hours of review time time  Law firm dynamic – 1 GB = $36,000 or $1.60 per page  Outsourced review 1 GB = $6,500 a GB to $11,000 (60 to 95 cents per page)

(12)

Why Has Legal Market Been So Slow to Adopt

Changes?

F il t d t d th t diti l d l i b k  Failure to understand the traditional model is broken  Does not understand how to use the new technologies technologies  Mistakenly interprets move toward outsourced

review as a result of the economic downturn review as a result of the economic downturn

 Continues to permit individual litigators to select

vendors review platforms and implement their own vendors, review platforms and implement their own processes

(13)

The Good News!

C t i t d ith i i l l f ESI

 Cost associated with reviewing large volumes of ESI

is forcing a change

Vendor entry into review space has accelerated

 Vendor entry into review space has accelerated

change

 Clients are demanding greater use of:

 Low‐cost domestic review attorneys

 Off‐shore review facilities

 Adoption of new technologies

(14)

Document Review Models

Outsourced Manual

Predictive Coding Process Oriented Review Review • Most prominent model used today Li i d lli d Predictive Coding • Great deal of confusion regarding what it means • Uses attorneys to Process Oriented Review • Development of broad and defensible relevance criteria • Limited culling and analysis • Heavy reliance on attorney review f • Uses attorneys to develop a seed set of data that can be fed into a black box to find similar documents relevance criteria through integrative ECA process • Process approach to review to increase • Use of sampling to ensure quality control similar documents • Emphasizes sampling of inclusion set and exclusion set • Never tested or review to increase efficiency, recall and precision , using legally accepted tool sets: • Threading accepted in any court • Near‐Duping • Advance search • Clustering

(15)

A Word On “Predictive Coding”

g

(16)

Technology‐Assisted Reviews

gy

= TREC 2009 Manual Review = TREC 2009

Technology-Assisted Review

M. Grossman Presentation - Technology-Assisted Review Can be More Effective and More Efficient Than Exhaustive Manual Review

(17)

Developing a Seed Set ‐ Precision

Document Set for Review

(18)

Developing a Seed Set: Recall

Documents Set Excluded From Review

(19)

Our Take on Predictive Codingg

 Not ready for prime time as a final determinant of o eady o p e e as a a de e a o relevance and privilege:  Never been accepted by a court – do you want to be the first?  “Predictive coding” technologies and processes vary wildly  As with any statistical model, caution should be exercised (“Torture numbers, and they'll confess to anything”)  Seed set methodology:  Garbage in = garbage out  Who is picking your seed set  Who is picking your seed set

 Cases change rapidly within the first few months  Is this scalable to all types of cases

Not able to address (images graphics excel files video voice etc )

(20)

Our Take on Predictive Coding (cont’d)

g (

)

 Over time, predictive coding and other technology

 Over time, predictive coding and other technology assisted methodologies will become both prevalent and accepted  Under the appropriate circumstances (e.g., agreement by the parties), it can be a valuable and cost effective tool  Until then ‐‐ predictive coding can be used to help d k ff organize and make a review more efficient

(21)

(22)

A Process Oriented Electronic Document Review

(23)

Old vs. New Models

Traditional Approach Process Oriented Approach

Old vs. New Models

Traditional Approach Process Oriented Approach

Manually Acquire Broad Amounts of Manually Acquire Broad Amounts of

Data

Process Data

Analytics/

Search, Acquire, and Process Narrower Amounts of Data

Review Second Level

Review First Level

Review

Filter and Cull

Produced Documents

P d d D t

Produced Documents Produced Documents

(24)

Phases to A Process Oriented Document

Review

• Working with client and data to develop a set of • Working with client and data to develop a set of defensible “relevance criteria” to select data subject to review Analytical • Use of search and retrieval at the front end can dramatically reduce the volume and cost • Risk consideration Collection • Employ more sophisticated processing tools to further reduce the volume set • Unilaterally vs. negotiate Processing– filtering and culling • Employ lower cost reviewers • Use technology and process to increase precision d ll th d t t Non‐Linear Review

(25)

Selecting the Review Set

g

 Recent focus by courts y

 Recognition that parties have failed to adequate

construct and test search terms to withstand judicial scrutiny (Victor Stanley)

scrutiny (Victor Stanley)

 Absent agreement, you will have to defend your

relevance criteria in terms of reasonableness and responsiveness:

 Selecting test data

 Quality controlling the results  Quality controlling the results

 Revising queries

 Transparent records from which you can defend your decision

(26)

Selecting the Review Set (cont’d.)

g

(

)

Filter Test Data Set Based

ESI Sources

QC via Select Test Data Set

Upon Queries Date Range Custodians File Types Archiver PCs/Laptops Network Stores Responsive Test Data Set QC via Sampling Archiver PCs/Laptops

Network Shares Process Dedupe, De-NIST, Extract Location SharePoint Loose Media SharePoint Loose Media Key Words

Filter Entire Data Set Based Upon Relevance

Queries Date Range C t di Revise Exceptions Process Custodians File Types Location Revise Queries as Appropriate Key Words Responsive Y/N?

Responsive Data Set

Dedupe, De-NIST, Extract

(27)

Rapid Filtering/Culling

(28)

Multi‐Faceted Display of Results

p y

(29)

Instant File Type Analysis

yp

y

(30)

Visualization of Email Traffic

(31)

Advanced Search

(32)

Review Phase

 70% of your production is e  70% of your production is e‐ mail  15% to 40% are duplicates or near duplicates near duplicates  Reviewers can review data at a higher rate with a greater d f i i d ll degree of precision and recall when looking at like information

(33)

Review Phase

Remove Store NR d t i from DB? data in QC via Sampling No Batch Documents p g Non-Responsive Cull via Metadata and

Smart Filters Privilege Filter Custodian

Non-Responsive QC via

Sampling

Initial Data Set Review

Smart Filters Date Range Custodians File Types To/From Location Key Words Key Word Metadata Responsive Privileged? E-mail Thread Separate Att Location Key Words S d t P i il Near-Dupes Cluster

Attorneys Send to Privilege Review

(34)

Organizing the Data For Review

g

 Batching will be the most significant decision in

 Batching will be the most significant decision in terms of expediting the review  Our approaches:pp  E‐mail threading  Near Duplicates  Review in the remainder  Clustering algorithm  Boolean queries  Boolean queries  Key custodian review is appropriate  Types of Documents M d ( d h i i )

(35)

E‐Mail Threads

• 70% of production is e-mail and of that nearly 65% or more are part of e-mail threads

Less Time

The Problem: N l th d t

eMail Threads – Step 1 Gro p into eMail sets

eMail Threads – Step 2

 Build tree structure Less Cost Less Errors

No clear method to identify eMail threads

eMails are reviewed multiple times and inconsistently

Group into eMail sets  Build tree structure

 Identify missing links

 Suppress duplicates

 Focus on inclusives

Less Cost

Extremely difficult to identify where missing eMails exist

(36)

Duplication and Near Duplication

• 15% to 40% of document population are duplicates or near d plicates

p

duplicates Less Time The Problem: No clear method to organize and allocate documents across

Near-Duping – Step 1

Group the near-duplicates

Identify the differences

Near-Duping – Step 2

 Assign near-dupe sets for coherent review to

reviewers _{Less Cost}

Less Errors

documents across reviewers

Documents are reviewed multiple times by different reviewers

among the near-duplicates

reviewers

 Reviewers prioritize and review only the

differences

 Apply coding to entire

Less Cost

High risk of different coding among similar documents

near-dupe sets where appropriate

(37)

Second Generation Search Technology

gy

 Concept search places a document or part of a document in this space.  Results are returned in order of relevance.  higher score = closer document document D t 1 98 • Document 1: 98 • Document 3: 92 • Document 4: 91

(38)

Advanced Features of Relativity Analytics

y

 Clustering – Automatic grouping (sorting) of a collection of

documents into subsets based on their conceptual content

 Categorization – Categorize and organize documents in the entire

database based on a small set of user example documents database based on a small set of user example documents  Keyword Expansion – Select keywords within the document viewer and Relativity will produce a list of conceptually related terms  Inline Concept Search – Utilize concept searching within Relativity’s innovative document viewer by right‐clicking on portions of a displayed document

(39)

Keyword Expansion

y

p

(40)

Best Practices ‐ Planning

g

 Plan the work and work the plan:

 Plan the work and work the plan:  Outside counsel  eDiscovery Team  Review team leader  Planning meetings:  Developing relevance criteria  Developing a review plan

 Deadlines must be established staffing needs must be defined

 Deadlines must be established, staffing needs must be defined,

and review speeds must be estimated to provide budget predictability to the client and avoid cost and time overruns

(41)

Best Practices ‐ Planning (cont’d)

g (

)

 Before commencing review, decide on format of

 Before commencing review, decide on format of production and privilege log  Rule 26(f) conference  Native vs. TIFF productions  Privilege log: metadata vs. manual Di t ti l l i h f t liti ti t id  Discuss potential exclusions, such as for post‐litigation outside counsel documents  Logging of email chains  Timing of privilege log production (with or after document productions)

(42)

Best Practices ‐ Roles

 eDiscovery Team  eDiscovery Team  Administration of the data set (e.g., loading of data, managing index, running clusters)  Assist Review Team Leader in developing relevance criteria  Creation of load file R i T L d  Review Team Leader  Coordinate with attorneys regarding development relevance criteria  Identify test data set  Validate test data set

(43)

Best Practices ‐ Training the Review Team

Best Practices Training the Review Team

 Substantial attention should be given to training

 Substantial attention should be given to training.  Briefing materials should be prepared:  List of counsel involved in the Legal Matter  List of key witnesses involved in the Legal Matter  Summary of the claims and issues  Summary of the request for production of documents and y q p responses  Instructions regarding how to use the review tool  Instructions regarding coding of the individual documents g g g  Procedure for resolving questions or ambiguities  Training on how to recognize and code privileged documents

(44)

Best Practices – Conducting the Review

g

 Uniformity in review team analysis should be the goal  Variations in review team analysis should be addressed each day and additional training provided as required  Daily feedback heightens quality and attention to detail by reviewers, and knowledge transfer to reviewers  Reduces possibility of accumulation of poorly reviewed documents in the population  Review team should be broken down into sub‐groups (e.g., separate privilege review group)  Computer equipment should foster speed and efficiencyp q p p y  Dual screens or wide screens, fast processing time  Review tool should be configured to reduce need for scrolling and key strokes  Procedures to address potential departure of review team members

(45)

Best Practices ‐ Quality Control

Q

y

 Review team leader is responsible for overall quality control f h d h ld h l of the review and should meet with review team extensively during review  Review questions Id tif t i f d t th t b d N/R  Identify categories of documents that can be removed as N/R  Prepare list of additional questions for litigation team  QC reviewers work by: E i i i di id l i d ti it d t  Examining individual reviewer productivity and error rates  Designating QC reviewers to re‐review established percentage of documents to verify accuracy of coding  Type of sampling (random, systematic, stratified)yp p g ( y )  Sample sizes  Review rate  Accuracy rate  Hours worked

(46)

Other Best Practices

Annotating and coding:

 Annotating and coding:

 Determination is required as to extent of annotation and coding

that would be beneficial

 Coding may include responsiveness, privileged, confidential,

sensitive and relevant to issue codes

 Review in native format

 Review metadata if necessary

 Review hidden data if necessary

(47)

Best Practices ‐ Privilege Review

g

 Create separate privilege review/privilege log preparation team  Create separate privilege review/privilege log preparation team  Segregate attorney ESI from the production

 Screen out potentially privileged documents:

 Run key word searches of privileged ESI (e.g., attorney, solicitor, privilege, work

 Run key word searches of privileged ESI (e.g., attorney, solicitor, privilege, work product)  URL analysis  Hashing comparisons Clustering  Clustering  Educate the review team on privilege coding and issues  Clarify handling of email chains and email attachments

 Include protocol for compliance with FRCP 26(b)(5)(B) and FRE  Include protocol for compliance with FRCP 26(b)(5)(B) and FRE

502(b) and (d)

 Additional pre‐production validation of privileged documents (on

sampling basis)

(48)

Review for Privilege

g

Review Strategies Return to Production Set Privilege Filter URL Search Terms Review Strategies Review all Docs Review Positive Hits Privilege (Y/N)? No Production Set Redaction Log QC via Sampling Clustering Positive Hits Sample Non-Privilege Hits Log g Yes

(49)

Best Practices – Privilege Log

g

 Automate log as much as possible  Increases speed and efficiency (saves time and money)  Decreases typos and inconsistencies  Goal is to avoid manual data entry wherever possible  Leverage metadata as much as possible  Be careful, though, of relying on metadata alone for email chains and

PDFs unless the parties have agreed PDFs, unless the parties have agreed

 Use drop‐down boxes where possible, having reviewers

choose the one or more descriptions that best fit the document

(50)

Washington, D.C. – Litigation Review Center

g

,

g

(51)

Staffing a Review

g

 Lower‐cost staff that specializes in review

 Lower cost staff that specializes in review  Properly recruited and screened  Proper supervised  Properly trained  Considerations:  Insource vs. outsource  Domestic vs. international JD or barred  JD or barred  Use of non‐lawyers  Use of technical experts

(52)

Ethical Issues

ABA 08 0451 (A 2008)

 ABA 08‐0451 (August 2008)

 U.S. lawyers are ethically permitted to outsource legal

work, including lawyers or non‐lawyers (both work, including lawyers or non lawyers (both domestically and internationally) if they adhere to ethics rules requiring: Competence  Competence  Supervision  Protection of confidential information  Reasonable fees  Not assisting unauthorized practice of law

(53)

Ethical Issues

ABA 08 0451 (A 2008)

 ABA 08‐0451 (August 2008)

 Minimal obligations:

 Conduct reference checks and background investigations of

lawyer or non‐lawyer service providers and any intermediaries

 Interview principal lawyers on a project, assessing their

educational background and evaluate the quality and educational background, and evaluate the quality and

character of any employees likely to access client information

 Review security systems

Visit the premises of the service provider

 Visit the premises of the service provider

(54)

Ethical Issues

If the provider is in a foreign country:

 If the provider is in a foreign country:

 Determine whether the legal education system in that country is similar

to that of the U.S., and whether professional regulatory systems

incorporate equivalent core ethics principles and effective disciplinary incorporate equivalent core ethics principles and effective disciplinary enforcement systems  Consider utilization of additional training, especially regarding privilege p g  Determine whether the foreign legal system protects client confidentiality and provides effective remedies to the lawyer’s client in case disputes arise  Some circumstances may require more rigorous supervision than others  May be necessary to obtain a client's consent before engaging outside assistance

(55)

Proper Supervision

p

Bray & Gillespie Mgmt LLC v Lexington Ins Co No 6:07

 Bray & Gillespie Mgmt. LLC v. Lexington Ins. Co., No. 6:07‐

cv‐222‐Orl‐35KRS, 2009 WL 546429 at * 21 (M.D. Fla. Mar. 4, 2009)

 Form of Production; Rule 37 Sanctions; Outside Counsel Changing  Form of Production; Rule 37 Sanctions; Outside Counsel Changing

Law Firms; Individual Attorney Sanctioned

 “While B & G, as the client, has the obligation to supervise its

lawyers, the evidence establishes that B & G's outside counsel made the decision how to produce ESI Additionally B & G has made the decision how to produce ESI. Additionally, B & G has already spent considerable time and effort to reproduce some ESI in native format, although problems remain with the form of that production. Under these circumstances, I find that it is not

appropriate to require B & G to pay the attorney's fees costs and appropriate to require B & G to pay the attorney s fees, costs, and expenses . . . . Should B & G fail to monitor its counsel's actions going forward, however, it will subject itself to all available

sanctions should additional problems occur.”

“ bli dl l i t id l f ll h t f th d t h h ffi f

 “ . . . blindly relying on outside counsel falls short of the duty he has as an officer of

(56)

Budgeting

g

 Now possible to model the entire review process to

 Now possible to model the entire review process to establish a reliable budget  Considerations:  E‐discovery costs  Size of review  Potential cull/filter rate  Percent of e‐mail/threads P t f d d d  Percent of dupes and near‐dupes  Complexity of the documents

(57)

(58)

Questions?

(59)

Contact Information

John Rosenthal Scott Cohen

John Rosenthal Chair, eDiscovery & Information Management P ti G Scott Cohen Director of eDiscovery Support Services N Y k Practice Group Washington, D.C. 1 (202) 282‐5785 New York 1 (212) 294‐3558 [email protected] [email protected]