Minimum Information About a QSAR Model Representation

3.6 QSAR Model Representation

3.6.2 Minimum Information About a QSAR Model Representation

Minimum Information about a QSAR Model Representation (MIAQMR), proposed in this section, is a set of rules, that helps to define a model representation satisfy-

ing model governance framework requirements as discussed in Section 3.4. These

rules encode the minimum necessary information required to define model objects in the governance framework including: model provenance, model description, dataset provenance, model development, model reliability and predictivity. All these rules

comply with the current OECD requirements [77] for in-silico modelling and are in-

spired by these principles. They also form the basis for the XML model representation development called MIAQMR-ML. The general schema including the conceptual and

data layer is presented in Figure 3.6. The data layer is a simplified diagram of the

model representations. Each element corresponds to a rule that defines the minimum required information. There are in total six rules discussed below.

RULE I: Model Provenance

A model must comply with the following standards: authors must be clearly defined; timestamp of model creation, submission and updates must be provided; one or more referenced sources that describe the model development process should be included.

Model provenance defines in a very general way when and by whom the model

was created or developed. Listing 3.1 defines the XML schema for the provenance

MIAQMR MODEL PROVENANCE RULE I MODEL DESCRIPTION RULE II DATA REFERENCE RULE III MODEL DEVELOPMENT RULE IV MODEL RELIABILITY RULE V MODEL PREDICTIVITY RULE VI

CONCEPT

LAYER

DATA

LAYER

Origin Data Reference Model Metadata Training Set Metadata Validation Set Metadata Datasets Simulation

Results Result Files

Development Metadata Pre-processing Pre-processing Step Development Methodology

Descriptor List Descriptors Parameters Descriptor Providers Descriptor Result List Model Files’ URLs Applicability Domain Descriptor Result Descriptor Value Domain Bounds Performance Internal Validation Outlier List Classification Performance Regression Performance Parameters Outlier Performance Classification Performance Regression Performance Outlier List Outlier

and the submission date. This assists with model maintenance and validation, as other users may report errors or unexpected model behaviour directly to the authors. The model version and timestamp must be included to support users who wish to know about the model provenance. Model development and last modification timestamps help users to monitor any changes that may be made by authors. Too many changes in a short time may impact on the degree of trust in model predictivity. Additionally, a model should not be available to users before a stable version is released. The quality and version control system will monitor the updates and error reports and mark models for an external review. A model must include corresponding references for model quality evaluation. These references should include the data collections and the model development procedures written as an organisational documentation, white or scientific papers.

Listing 3.1: Model provenance element of the MIAQMR-ML.xsd

</complexType>

RULE II: Model Description

A model must comply with the 1st OECD QSAR model validation principle and have a clearly defined endpoint name and type.

According to Rule II, model representation must include at least minimal assay in-

formation such as: endpoint, units and species (see Listing 3.2). The endpoint and

species should be standardised to avoid inconsistency in data representation. The model description does not include the full experiment protocol description only the

exposure time should be recorded. Such information is required to report the model to the regulatory bodies but in the governance framework this information can be recov- ered from the dataset or assays descriptions. The model representation must include type and name. The name of the model should be standardised over the systems. Two models for the same endpoint should have the same endpoint name. This allows for unique models identification for a given endpoint. The model representation must include the dependent variable. This is because the dependent variable being modelled can be different from the measured endpoint. The model type annotation and a clear definition of the dependent variables will be used in the model selection and comparison process.

Listing 3.2: Model description element of MIAQMR-ML.xsd

RULE III: Data Reference

A model must have references provided for datasets used in model development and validation phases, supporting compliance with the 3rd OECD QSAR model validation principle.

According to Rule III, model representation must have associated training and val-

idation datasets (see Listing3.3). In the case when cross-validation methods such as

LOO, LMO, K-fold etc. are used, the validation dataset can be empty. This, however, can cause a reduction in model quality. Conversely, a model may have more than one validation dataset. Both datasets must include dataset provenance (including authors, timestamp and references) and number of chemicals. The dataset reference is necessary to recover information about the assay and the experimental protocol and the

number of chemicals used for statistical model comparison. Together with the information about applicability domain, this may provide information about how well the model can be applied across a wide chemical sub-space. These datasets should be sub- mitted in file format, together with the model, or be accessible via a provided link. The simulation results must be included for model submission. This is required in order to facilitate model identification and toxicity assessment without applying a model for new chemicals. To clarify, the predicted toxicity measurements are only stored in the files and used for the model performance analysis. The chemicals are related to the models via datasets.

Listing 3.3: Model datasets element of MIAQMR-ML.xsd

</complexType>

RULE IV: Model Development

A model must refer to the model development process to comply with the 2nd and 3rd OECD QSAR model validation principles, and to support compliance with the 5th OECD QSAR model validation principle.

Rule IV ensures transparency in the model algorithm within the model governance framework. This information has to be available for the quality control system to establish model performance according to the quality policies. This also allows the re- production of predictions as it includes information about data pre-processing, the algorithm, attributes/features used in the model development process and their provider, the access to the model executable file or to the tools that may be used for the further predictions and the applicability domain. Often, model supporting information pub- lished in the literature is not sufficient to reproduce model results. Thus the model must provide information about whether the dataset has been pre-processed or not. If the dataset has been pre-processed, the feature selection algorithm that was used

should be included (see Listing3.4). The method name must be defined. The name of

the software and its version must be specified. The method description must be provided including a list of parameters. Such information will help users to reproduce the model using the same or different modelling tools.

Listing 3.4: Model preprocessing element of MIAQMR-ML.xsd

</complexType>

The model representation must define model features (descriptors). This imposes a requirement to provide the descriptor list used in the model development process. Descriptors can be calculated from molecular structure, e.g. molecular weight, or measured such as logP. In this case, when descriptors values were calculated from a structure, the software (including version and parameters) from which the descrip-

tor values are derived is required (see Listing 3.5). Such captured information allows

users to reproduce results or to find a collection of models for a given endpoint. These models may be based on different feature sets and their combination can improve a future prediction for new chemical compounds.

Listing 3.5: Model descriptor type element of MIAQMR-ML.xsd

</complexType>

The model representation must have defined an applicability domain, where the model produces reliable predictions. This includes information about the applicability

domain estimation process and representation of applicability domain (see Listing3.6).

The model has to be represented in a machine readable format such as files for MS Excel, PMML, ARFF, R, Java, C etc. It is required to submit actual model files when submitting a model to the system.

Listing 3.6: Model applicability domain element of MIAQMR-ML.xsd

RULE V: Model Reliability

The model reliability must define the model ability to make predictions from the training dataset. Models must have a defined type, the model performance matrices, and a list of outlier chemicals clearly defined.

According to Rule V, the internal validation method must be specified, such as

cross validation, LOO, LMO, boosting etc (see Listing3.7). For classification models,

the following performance matrices are required: confusion matrix, precision/recall, overall accuracy, sensitivity, specificity and errors. For regression models, the fol-

lowing performance matrices are required: R-squared (r2_{), Q-squared (q}2_{) [}₁₁₄_{] and}

errors. The model must provide a list of outliers. While the simulation results can be stored, these values do not have to be provided by authors. Nevertheless, they can be calculated from the model and stored in the model representation format.

Listing 3.7: Model reliability element of MIAQMR-ML.xsd

</sequence> </complexType>

</simpleType> </attribute> </complexType>

RULE VI: Model Predictivity

The model predictivity must define the model ability to make predictions from outside the training data. Models must have a defined type, model performance statistics, and a list of the chemicals defined as outliers.

According to RULE VI and similar to the reliability rule, the model representation should include information about the model performance for the validation dataset

(see Listing3.8). For classification models, the following performance matrices are re-

quired: confusion matrix, precision/recall, overall accuracy, sensitivity, and specificity. For regression models, the following performance matrices are required: R-squared

(r2_{), Q-squared (q}2_{) and errors such as mean squared error, etc. The model must also}

provide a list of outliers. If a model has been validated on multiple validation datasets, each should have their own performance matrices. These values can be calculated from the simulation result, but it is important to encode it in a model representation to ensure the model representation consistency.

Listing 3.8: Model predictivity element of the MIAQMR-ML schema

In document Interpretation, Identification and Reuse of Models. Theory and algorithms with applications in predictive toxicology. (Page 82-91)