Patents
1. Systems and Methods for Conducting and Terminating Technology Assisted Review. United States. 15/186366. 2016/06/17.
Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process’s gain curve before an inflection point to the slope of the TAR process’ gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.
2. Systems and Methods for Conducting and Terminating a Technology Assisted Review. United States. 15/186377. 2016/06/17.
Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Patent
Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process’s gain curve before an inflection point to the slope of the TAR process’ gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.
Professor Maura Grossman 3. Systems and Methods for Conducting a Highly Autonomous Technology-Assisted Review Classification.
United States. 15/186382. 2016/06/17. Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information are provided by way of a Technology-Assisted Review (“TAR”) process, specifically an “Auto-TAR” process that limits discretionary choices in an
information classification effort, while still achieving superior results. Auto-TAR selects an initial relevant document from a document collection, selects a number of other documents from the document collection and assigns them a default classification, trains a classifier using a training set made up of the selected relevant document and the documents assigned a default classification, scores documents in the document collection and determines if a stopping criteria is met. If a stopping criteria has not been met, the process sorts the documents according to scores, selects a batch of documents from the collection for further review, receives user coding decisions for them, and re-trains a classifier using the received user coding decisions and an adjusted training set.
4. Systems and Methods for a Scalable Continuous Active Learning Approach to Information Classification. United States. 15/186387. 2016/06/17.
Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information are provided by way of a Technology-Assisted Review (“TAR”) process. In certain embodiments, the TAR process is a Scalable Continuous Active Learning (“S-CAL”) approach. In certain embodiments, S-CAL selects an initial sample from a document collection, trains a classifier by using a default classification for a portion of the initial sample, scores the initial sample, selects a sub-sample from the initial sample for review, removes the reviewed sub- sample from the initial sample, and repeats the process by re-training the classifier until the initial sample is exhausted. In certain embodiments, a classification threshold is determined using a calculated estimate of the prevalence of relevant information such that the threshold classifies the information in accordance with a determined target criteria. In certain embodiments, the estimate of prevalence is determined from the results of iterations of a TAR process such as S-CAL.
5. Systems and Methods for Conducting and Terminating Technology Assisted Review. United States. 15/186360. 2016/06/17.
Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods are provided for classifying electronic information and terminating a classification process which utilizes Technology-Assisted Review (“TAR”) techniques. In certain embodiments, the TAR process, which is an iterative process, is terminated based upon one more stopping criteria. In certain embodiments, use of the stopping criteria ensures that the TAR process will reliably achieve a level of quality (e.g., recall) with a certain probability. In certain embodiments, the TAR process is terminated when it independently identifies a target set of documents. In certain embodiments, the TAR process is terminated based upon whether the ratio of the slope of the TAR process’s gain curve before an inflection point to the slope of the TAR process’ gain curve after the inflection point exceeds a threshold. In certain embodiments, the TAR process is terminated when a review budget and slope ratio of the gain curve each exceed a respective threshold.
Professor Maura Grossman 6. Systems and Methods for Classifying Electronic Information Using Advanced Active Learning Techniques.
United States. 20150324451. 2015/07/22. Patent Status: Pending
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information or documents into a number of classes and subclasses are provided through an active learning algorithm. In certain embodiments, seed sets may be eliminated by merging relevance feedback and machine learning phases. In certain embodiments, the active learning algorithm forks a number of classification paths corresponding to predicted user coding decisions for a selected document. The active learning algorithm determines an order in which the documents of the collection may be processed and scored by the forked classification paths. Such document classification systems are easily scalable for large document collections, require less manpower and can be employed on a single computer, thus requiring fewer resources. Furthermore, the classification systems and methods described can be used for any pattern recognition or classification effort in a wide variety of fields.
7. Systems and Methods for Classifying Electronic Information Using Advanced Active Learning Techniques. United States. 9122681. 2013/03/15.
Patent Status: Granted/Issued Year Issued: 2015
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information or documents into a number of classes and subclasses are provided through an active learning algorithm. In certain embodiments, seed sets may be eliminated by merging relevance feedback and machine learning phases. In certain embodiments, the active learning algorithm forks a number of classification paths corresponding to predicted user coding decisions for a selected document.
8. Systems and Methods for Classifying Electronic Information Using Advanced Active Learning Techniques. United States. 8838606. 2013/06/18.
Patent Status: Granted/Issued Year Issued: 2014
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information or documents into a number of classes and subclasses are provided through an active learning algorithm. In certain embodiments, seed sets may be eliminated by merging relevance feedback and machine learning phases.
9. Systems and Methods for Classifying Electronic Information Using Advanced Active Learning Techniques. United States. 8713023. 2013/06/18.
Patent Status: Granted/Issued Year Issued: 2014
Inventors: Maura R. Grossman and Gordon V. Cormack
Systems and methods for classifying electronic information or documents into a number of classes and subclasses are provided through an active learning algorithm. Such document classification systems are easily scalable for large document collections, require less manpower and can be employed on a single computer, thus requiring fewer resources. Furthermore, the classification systems and methods described can be used for any pattern recognition or classification effort in a wide variety of fields, including electronic discovery in legal proceedings.