Cathleen Peterson, Benita Sumey and Jennifer Fiorentino Reprinted with permission from Orrick eDiscovery Alert, May 2, 2011.
Maximizing the Value of Your Work Product
At one time, the goal of a document review was simply to code all of the documents and to furnish the production to the opposing party. Over time, document review management has grown more sophisticated and law firms are beginning to recognize the value that can be passed along to clients by running an efficient review. Litigants themselves are also becoming more involved in the management of a review and should demand that the information and knowledge gleaned through the often-expensive review process should not only result in a quality production but should also reduce the time investment required of the core litigation team.
This goal can be achieved through the use of some key “best practices,” such as:
x Maintain a list of the most important or “hot” documents that surface during review so the case team can readily identify, refer to, and incorporate them into deposition preparation, motion practice and trial.
x Maintain an evolving collection of daily observation logs and summaries of interesting documents.
x Review daily logs to identify important themes and flesh out key words or people not previously identified as relevant to the litigation.
x Leverage technologies, such as Case Map, to map out key themes, witness participation and timelines.
x In the typical law firm, relevant fact development is handled by associates charging rates of three to five times that of dedicated document review attorneys.
Assess whether your firm or staffing service can offer qualified, dedicated review teams at a fraction of typical associate rates, and explore how they plan to ensure the maximum use of the work product coming from their team.
A Better Approach to Data Reduction
Clearly, efforts to reduce data sets prior to review can generate substantial savings.
Two ways to reduce volume include: 1) effectively removing duplicate documents (a.k.a.
“deduplicating”) and 2) extracting clearly irrelevant documents through the use of analytics.
Global Deduplication
Efforts to reduce data sets prior to review have traditionally been limited to de-NISTing (a process that removes files with no evidentiary value, like system files) and
dedupli-cation within a single custodian. More recently, litigants have begun to embrace the efficiencies offered by global deduplication, a process that results in only one copy of a document in the review set.1 Some of the concerns with this approach harbored by the proponents of single custodian deduplication are addressed by the custodian tracking and reporting features that can be leveraged or designed in many review tools through close collaboration between discovery counsel and the vendor. These techniques can preserve the full range of custodian and source information for each document in a searchable format. The practice tips below offer suggestions for making the most of these features.
x In conjunction with global deduplication, ask your vendor to create a “duplicate custodian" field to list every custodian who had a copy of the document in their data set. This preserves the association of every custodian with the document and reduces the review set to a single instance of each duplicate document. If needed, you can produce this field in the metadata, which can help facilitate custodian-based productions.
x When rolling productions are necessary, consider producing a final overlay load file that updates the additional custodian field for data that is processed after initial productions have been made.
Analytics
Most collected data contain groups of documents that are clearly irrelevant to the discovery requests, or even the litigation generally. Isolating these documents and removing them from the review set will save significant time and money. Data analytics offers a variety of techniques to cull irrelevant material and streamline document review.2 In many instances, technologies with sophisticated analytics may charge a higher up-front price than competing technologies lacking these features; however, with strategic use of the technology, this cost is more than justified by the savings realized by avoiding the unnecessary review and hosting of irrelevant documents.
x Use analytics – including domain analysis (i.e., the part of the e-mail address following the “@”), content clustering, and strategic iterative searching – to identify and remove irrelevant spam and junk e-mails.
x Review vendor statistics on the cost savings that may be realized through use of a tool’s data analytics features.3
1 See generally, Anne Kershaw and Joseph Howie, EDD: "Highway Robbery? Are we cheating clients if we don’t purge data," Law Technology News, August 2009 (summarizing a study conducted by the Electronic Discovery Institute, which found a 21 percent reduction in volume when clients deduplicated within a single custodian, and a 38 percent reduction when deduplicating globally across the entire data population).
2 Data analytics is the process of defining and applying “criteria to eliminate unnecessary data as well as ensure that potentially relevant data is collected and/or preserved.” The Electronic Discovery Reference Model, Analysis Guide (2010), available at http://edrm.net/resources/guides/
edrm-framework-guides/analysis.
3 See The Sedona Conference, Commentary on Achieving Quality in the E-Discovery Process at 18 (2009), http://www.thesedonaconference.org/content/miscFiles/publications_html?grp=wgs110
x Ensure that you fully understand and that the vendor can articulate the processes involved with their data analytics capabilities. This may be important if the court or opposing party requires an explanation on the search methodologies employed.
A Better Approach to Search Term Development
Well-planned and well-executed searches result in major cost and time savings, and may also be necessary to ensure compliance with legal requirements and production deadlines.4 Lawyers are expected to develop reasonable strategies for searching, either through development of their own expertise or reliance on experts.5 Defensible searches often require not only knowledge of the subject matter of the case but also an understanding of how available technology can best be leveraged to ensure accuracy and efficiency in the discovery process.6
In most cases, search terms are developed by lawyers using their knowledge of the matter, and information gathered during client interviews. These terms are then run through the document set by a vendor who produces a “hit report,” reflecting the number of documents captured by each term. Based only on the vendor’s report, a case team may modify or exclude terms that appear to be resulting in too many or too few hits.
(discussing the merits of a “blended review” – when manual lawyer review is combined with automated technology to increase efficiency).
4 See, generally, Seven Seas Cruises v. Ships Leisure, No. 09-23411, 2011 Lexis 19465 (S.D.
Fla. Feb. 19, 2011) (imposing sanctions when the defendant repeatedly failed to produce relevant documents and attributing this in part to the defendant’s lack of proper search strategies). See also The Sedona Conference, The Sedona Conference Cooperation Proclamation (2008), http://www.thesedonaconference.org/content/miscFiles/publications_html?grp=wgs110 (promot-ing cooperation and early cross-party communication on electronic discovery topics), and The Sedona Conference, “Jumpstart Outline”: Questions to Ask Your Client and Your Adversary to Prepare for Preservation, Rule 26 Obligations, Court Conferences and Requests for Production (2008).
5 See e.g., Multiven, Inc. v. Cisco Sys., 2010 U.S. Dist. LEXIS 72449, *3-9 (N.D. Cal. June 2010) (ordering the plaintiff to use an e-discovery vendor after a three-month delay in production caused by plaintiff’s reliance on a manual document review and failure to use search terms); U.S. v.
O’Keefe, 537 F.Supp.2d 14, 24 (D.D.C. 2008) (opining that a lawyer must use care and rigor in the selection of search terms used in a keyword search); Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D.Md. 2008) (stating that care and planning must be exercised in the construction of a search and that the qualifications of a search developer should be considered by a court when evaluating the reasonableness of the search methodology used); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331 (D.D.C. 2008) (same).
6 See Patrick Oot, Anne Kershaw, & Herb Roitblat, "Mandating Reasonableness In a Reasonable Inquiry," 87 Denv. U.L. Rev. 533 at 551 (2010) (discussing the need for lawyers and judges to educate themselves about new and existing technologies and current trends in eDiscovery).
Currently, Orrick, Herrington & Sutcliffe LLP is participating in the Electronic Discovery Institute’s 2011 study to evaluate the efficiency of automated review; for more information visit:
http://www.prlog.org/11271387-ediscovery-institute-launching-technology-assisted-document-review-study.html. See also 2010 TREC Legal Track, http://plg.uwaterloo.ca/~gvcormac/trelegal/
(researching and analyzing the value of machine learning to enhance and expedite lawyer review in 2010, with the final results to be published in mid-2011).
This approach may suffice in matters that are particularly well-suited to search terms.
However, in larger and complicated cases, an iterative approach to search terms may produce better results.7 This strategy may result in a smaller, more relevance-rich data set to review.8
When developing search terms:
x Keep in mind that you may be called on to discuss and defend not only your search terms, but also that process by which you chose them.
x Leverage interviews with key document custodians to identify likely terms and acronyms.9
x Assess search term syntax to ensure that the vendor is employing the best search construction techniques.10
x Select review platforms that allow for iterative search term analysis, evaluating capabilities against cost.11
x Review sample sets to adjust search terms accurately. Grouping documents using near duplicate analysis,12 e-mail threading13 and concept clustering14 will make this process more efficient and effective.
7 See, generally, William A. Gross Const. Assoc. Inc. v. American Mfrs. Mut. Ins. Co., 256 F.R.D.
134, 135-36 (S.D.N.Y. 2009) (reasoning that in order to construct an adequate search, search term developers should test results for accuracy prior to finalizing the list). See also Eurand, Inc.v. Mylan Pharmaceuticals, Inc., 266 F.R.D. 79, 84-85 (D.Del. 2010) (articulating a reasonableness test in evaluating search methodology).
8 See, generally, Logan Herlinger and Jennifer Fiorentino, "Position: The discovery process should account for iterative search strategy," Submission to the ICAIL 2011: The Thirteenth International Conference on Artificial Intelligence and Law, available at http://www.orrick.com/
fileupload/3579.pdf (arguing that an iterative search strategy is often necessary to conduct a focused and accurate review).
9 See William A. Gross, 256 F.R.D. at 135 (addressing the need for attorneys to talk to document custodians in order to craft effective keywords, otherwise they are “designing keyword searches in the dark, by the seat of their pants”). This may be especially important in litigations involving documents in more than one language. See, e.g., Curtis Heckman, "Searches Without Borders,"
Submission to the ICAIL 2011: The Thirteenth International Conference on Artificial Intelligence and Law, available at http://www.orrick.com/fileupload/378.pdf (discussing the challenges of a multilingual document review set and suggesting that lawyers ask custodians up front to identify the language(s) used and any alternative alphabetic characters that may exist in the data.
10 Effective search syntax may differ among vendor review platforms, depending in large part on the programming language used to support the review platform.
11 For tips on finding the right vendor to assist with search and retrieval, see The Sedona Conference, Best Practices Commentary on the use of Search and Information Retrieval Methods in E-Discovery at 210 (2007), http://www.thesedonaconference.org/content/miscFiles/publications _html?grp=wgs110.
x Assess terms that result in excessive hits in order to tighten up searches and isolate pockets of irrelevant material.
A Better Approach to Review Batching
Review managers often create review batches of documents by custodian. However, this method of “batching” can be inefficient and may contribute to inconsistent review results. For example, a reviewer may code a portion of an e-mail chain in Custodian A’s batch one way, and another reviewer may view Custodian B’s portion of the same e-mail chain and code it another. Custodian-based review is also often a slower process, as two or more reviewers will need time to come up to speed on the same underlying subject matter.
x Consider grouping documents for review according to subject matter. Docu-ments can be grouped by subject line (for e-mails), by concept, by file name (or path), or by leveraging predictive coding technologies,15 when available, to sug-gest likely treatment of documents. Grouping of documents by subject matter helps to ensure that similar documents are treated consistently, and also speeds the review process by allowing a single team member (or designated members) to handle documents relating to a specific subject matter.
x If batching by e-mail subject line, confirm that attachments properly follow the parent e-mail. Some vendor technologies are challenged by alternative batching requests.
A Better Approach to Technology Selection and Vendor Management
A key to effective discovery is pairing the right discovery vendor and technology with the data set and litigation goals. Understanding the available financial and human resources, as well as the macro- and micro-goals of the review, is critical to making an informed decision.16
12 Near duplicate analysis refers to the identification and analysis of identical and nearly identical items within a set of documents. See The Sedona Conference Glossary: E-Discovery & Digital Information Management (3d ed.) at 36 (2010).
13 E-mail threading is a process by which all messages in an e-mail chain are grouped together for purposes of review. This allows the entire conversation to be read together and in context, as opposed to viewed in fragments scattered across the review population. See id. at 19, 50.
14 Concept clustering groups similar documents by concept or topic together using search and retrieval technology. See id. at 10, 12.
15 Predictive coding involves a lawyer examining and coding a subset of documents. The technology then propagates issue code indicators throughout the document population so that the larger set of documents is coded in a manner consistent with those reviewed manually. See, generally, Anne Kershaw and Joseph Howie, "Crash or Soar? Will the Legal Community Accept Predictive Coding," Law Technology News (Oct. 2010).
16 See The Sedona Conference, Best Practices for the Selection of Electronic Discovery Vendors:
Navigating the Vendor Proposal Process at 4 (2007), http://www.thesedonaconference.org/
content/miscFiles/publications_html?grp=wgs110 (providing a comprehensive overview of a
Follow these practice tips to avoid common pitfalls:
x When evaluating vendors, determine whether the data will be hosted domestically or in an offshore location, as there are data privacy, legal ethics and quality control issues to consider. If the data will be in the United States, ask for specifics about the storage environment (physical and virtual).17
x Review closely and negotiate, if necessary, the limited liability, system mainte-nance, and other provisions of the services agreement. Inquire specifically about the vendor’s policies on backing-up and storing the data, given the risks asso-ciated with maintaining data in multiple locations over which you have little control.
x Explore alternative fee arrangements or billing cycles that may result in discounts or cost savings. To provide more predictability in costs, vendors may offer a flat fee for a defined project or charge an all-in flat rate on a per-document basis (versus separate charges for processing and hosting on a per-gigabyte basis).
Hourly pricing for productions may also generate savings over a traditional per-page approach to tiffing, OCR, stamping and endorsements.
x When comparing costs in matters requiring significant lawyer review, be sure to factor in the cost of both the technology and the review. Very often, more expensive technologies pay for themselves by reducing the lawyer time spent on review. For example, using a “concept clustering” feature to group documents that relate to a similar topic can accelerate review and enhance consistency.
x Keep in mind the planned or agreed production specifications when selecting your vendor.18 For example, you may want to assess the vendor’s capabilities and pricing for a native format or repository productions.
x If potentially relevant data is housed in a structured database19 (e.g., an Oracle-based financial system), ask prospective vendors if they have the technical expertise required to query and potentially extract the data.20
scalable selection process and recommending that the first step in staging a review should involve defining the scope of the project in order to find the “best possible match of problem and solution.”).
17 This is certainly important from a data security standpoint, and may also have an impact on ethical requirements that must be met. See, e.g., State Bar of California, Standing Committee on Professional Responsibility and Conduct, Formal Op. 2010-179 (2010) (laying out six factors that lawyers should consider when using any legal technology, and discussing particularly some of the ethical implications of storing client data in a cloud environment).
18 See, generally, The Sedona Conference, Best Practices Recommendations & Principles for Addressing Electronic Document Production at 63 (2007), http://www.thesedonaconference.org/
content/miscFiles/publications_html?grp=wgs110 (listing different factors that the parties should consider when negotiating production specifications, including, but not limited to: the need for metadata, whether the information sought is reasonably accessible and, in general, the costs and burdens associated with the whole production process).
x Once a matter is fully resolved, it is important to instruct the vendor to dispose of the litigation data unless it is subject to preservation requirements or use in other litigation. Stale litigation data hosted by discovery vendors can generate both cost and risk in future litigations. On the other hand, if you are a repeat litigant, there may be great economies generated through strategically repurposing work product. Discovery counsel can be a great source of strategic advice on these topics.
A Better Approach to Review Staffing
One of the most important strategic decisions in the early stages of document review involves choosing a review team, and review team management, that is well-matched to the specific matter.21 Often, document review projects are staffed by contract attorney agencies on a project-by-project basis. The document review team is usually released at the end of the project, and may or may not work on another matter for that agency, firm or client. Great efficiencies may be lost through this approach if you are a repeat litigant. Speed and accuracy both increase with an informed review team; maintaining a team comfortable with your business, and perhaps even your frequent “key players,” can result in significant review cost savings.
x If you are a frequent litigant, consider identifying a preferred review provider, and negotiate with that provider for “bulk” pricing and a dedicated review team. Your team will soon become familiar with company document types, acronyms, business processes and your corporate structure. This institutional knowledge not only increases efficiency but also dramatically improves the accuracy of the review.
x When using advanced analytic tools, be sure to use personnel well-versed and experienced in the tool and its analytics. Unfortunately, companies may invest in an advanced technology but then fail to invest in personnel familiar with and capable of effectively deploying the advanced features of the tools.
19 Very generally, a “structured database” is a system that organizes and stores large amounts of related information by means of tables and keys that can be queried to retrieve or tie data elements to one another.
20 For a helpful discussion of the treatment of databases in discovery, see The Sedona Conference, The Sedona Conference Database Principles: Addressing the Preservation &
Production of Databases & Database Information in Civil Litigation at 9-10 (2011), http://www.thesedonaconference.org/content/miscFiles/publications_html?grp-wgs110 (“[N]ot all e-discovery consultants understand the technical aspects of database discovery and parties
Production of Databases & Database Information in Civil Litigation at 9-10 (2011), http://www.thesedonaconference.org/content/miscFiles/publications_html?grp-wgs110 (“[N]ot all e-discovery consultants understand the technical aspects of database discovery and parties