Microbiological Method Validation: How Do We Prove that a Method is Fit for Purpose? Thomas Hammack

(1)

Microbiological Method Validation: How Do

We Prove that a Method is Fit for Purpose?

Thomas Hammack

Chief

Microbial Methods Development Branch

Division of Microbiology

Office of Regulatory Science

Center for Food Safety and Applied Nutrition

U.S. Food and Drug Administration

(2)

Equivalence?

24 h, 35ºC

24 h, 35ºC and 42ºC

(3)

• Why validate methods?

Benefits public health and world trade

False negative results are unacceptable

ISO 17025 lab accreditation demands the use of

validated methods

(4)

Validation Programs

• AOAC Microbiological Guidelines

• ISO 16140:2003

• FDA’s Guidelines for the Validation of Analytical

Methods for the Detection of Microbial Pathogens in

Foods

(5)

AOAC International

• 1884—Association of Official and Analytical Chemists

– Methods Validation for Fertilizers

– Membership Restricted to Regulatory Chemists

• Supported by USDA

– Dr. Harvey Wiley -“Father of FDA” is also considered the “Father of AOAC”

• 1892—Secretary of AOAC

• 1898—Established AOAC’s Committee on Food Safety

– Housed within FDA

• 1939—Microbiological Sampling of Eggs and Egg products Official Method 939.14 still in use today

• Many AOAC methods Official Methods were developed and validated in FDA Labs

• 1972—FDA published acceptance of AOAC Official Methods in the Federal Register

– Allows FDA to use proprietary rapid methods for the analysis of regulatory samples

• 1979—became an independent non-profit organization no longer tied to FDA, but many FDA employees serve as volunteers

(6)

International Organization for

Standardization (ISO)

• International non-governmental organization

– Based in Geneva, Switzerland

– Comprised of 163 national standards institutes including ANSI in the US

• Started in 1926 as the International Federation of National Standardizing Associations (ISA)

– Focused on mechanical engineering

– Disbanded in 1942

– Reorganized as ISO in 1946

– Mission statement is to provide “International Standards for Business, Government and Society”

• 1991 ISO and European Committee for Standardization (CEN) signed the Vienna Agreement

– CEN is comprised of the 31 Member States of the European Union

– Vienna agreement stipulates that CEN and ISO Standards be identical to enable free trade within the EU

• ISO Technical Committee 34 (TC 34) Subcommittee 9 (SC 9)

– ISO exercises its role in the standardization of food microbiological testing methods through TC 34/SC 9

(7)

18

FDA’s Methods Validation Guidelines

The Science and Research Steering Committee (SRSC), of the Office of Foods and Veterinary Medicine (OFVM), approved guidance to be used for validation of microbiological and chemical methods.

Guidelines for the Validation of Analytical Methods for the

Detection of Microbial Pathogens in Foods

http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf

Guidelines for the Validation of Chemical Methods for the FDA

Foods Program

http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM298730.pdf

Scope

“These criteria apply to all FDA laboratories that develop and participate in the validation of analytical food methods for Agency-wide implementation in a regulatory capacity. This includes all research

laboratories, and field labs where analytical methods may be developed or expanded for potential regulatory use. These documents will supersede all other intra-agency documents pertaining to

food-related method validation criteria for microbial and chemical analytes. the SRSC will authorize the formation of a Methods Validation Subcommittee (MVS) to serve as the governing body for all method

(8)

FDA and Methods Validation

Method validation is a process by which a laboratory confirms by examination, and provides objective evidence, that the particular requirements for specific uses of a method are fulfilled. It serves to

demonstrate that the method can detect and identify an analyte or analytes: • In one or more matrices to be analyzed

• In one or more instruments or platforms

• With a demonstrated sensitivity, specificity, accuracy, trueness, reproducibility, ruggedness

and precision to ensure that results are meaningful and appropriate to make a decision.

• Reliably for its intended purpose. Intended purpose categories include, but may not be

limited to emergency/contingency operations; rapid screening and high throughput testing; and, confirmatory analyses.

• After the method developer has conducted experiments to determine or verify a number of

specific performance characteristics that serve to define and/or quantify method performance.

(9)

9

RESEARCH

VALIDATION

Micro

Method

Validation

Sub-group

The Office of Foods and Veterinary Medicine & the SRSC

“Roadmap

for Microbiological Method Development and Validation”

IMPLEMENTATION

Organizational

Partnerships

•BAM Council •FERN MCC •IFSH •ORA “Micronauts” VERIFICATION

(10)

10

ESTABLISHES

validation needs and priorities in consultation with the

SRSC-Micro Super-group, FDA Bacteriological Analytical Manual Council (BAM Council), FERN Method Coordinating Committee, ORA “micronauts” inter-center working groups and others as appropriate

ADOPTS

procedures to govern all administrative processes needed for emergency and non-emergency method validation proposals and studies.

PROVIDES

planning, guidance, oversight, and resources to participating laboratories during the method development and validation process; will be the responsible authority for recommendations, evaluations and final approval of all validation studies from planning through field implementation.

CONSULTS

with other governmental, and independent (commercial, and international) validation bodies to harmonize validation standards where possible and practices

(11)

11

BROAD REPRESENTATION

from CFSAN, ORA, CVM, and NCTR with additional

expertise from biostatisticians and QA/QC managers

CURRENT MICRO MVS COMPOSITION:

ORA

Palmer Orlandi (co-Chair), Cathy Burns

CFSAN Thomas Hammack (co-Chair), William Burkhardt, Darcy Hanes

CVM

Beilei Ge

NCTR

Steven Foley

FERN

Don Burr

NCFST

(CFSAN Moffett)

Ravinder Reddy

The Method Validation Sub-Group

(12)

 Qualitative assays

 Analyte

 Bacteriological, e.g.

 Salmonella spp.

 Pathogenic Escherichia coli

 Listeria monocytogenes

 Shigella spp

 Vibrio spp

 Campylobacter spp

 Microbial toxins

 Viral pathogens, e.g.

 Hepatitis A virus

 Norovirus

 Enterovirus

 Parasitic protozoan pathogens, e.g.

 Cryptosporidium

 Cyclospora cayetanensis

 Bioengineered analytes, e.g.

 Genetically-modified foods (GMOs)

 Applications

 Pre- and selective enrichment

 Microbial analyte recovery and concentration

 Screening, high-throughput, confirmation

 Procedures

 Phenotypic, e.g.

 Biochemical characterization

 Antibiotic resistance traits Antigenic characterization

Genetic, e.g.

 Nucleic acid isolation/concentration

 Polymerase Chain Reaction

 Conventional, Real-time

 Reverse transcription

 Sequencing, e.g.

 Whole genome

 Selective sequencing

 Single nucleotide polymorphism (SNP) analysis

 Strain-typing applications

METHOD VALIDATION SCOPE OF RESPONSIBILITY

Pathogens, Genetic Material, Toxins and Antigens

(13)

1. Submission of a new or original method, OR,

2. Any significant modification of a method that may alter its performance specifications or changes to the fundamental science of an existing method. Categories include:

• Substitutions of reagents/apparatus

• Expansion of the scope of an existing method to include additional analytes.

• Changes in intended use i.e. screening or confirmatory.

• Platform extensions or significant parameter changes e.g. adaptation to another real-time PCR thermal cycler.

• Matrix extensions.

• Changes to time/temperature incubation periods, or enrichment media.

• In cases where the sample preparation and/or the extraction procedure/analytical method is modified from the existing test procedure and protocol, i.e the new method should demonstrate that the modifications do not adversely affect the precision and accuracy or bias of the data obtained.

• Modification of a method’s performance range e.g. specificity, sensitivity beyond previously validated levels.

(14)

Levels of Validation

Four levels of performance are defined. The hierarchy of

scrutiny will provide general characteristics on the method’s

utility and insights for its intended use, the assessed risk, and

the food-borne illness potential for an analyte-matrix pairing.

Not all methods will or should be validated to meet the requirements of a Level 4: full collaborative study.

(15)

Method Validation Criteria

Level One

The lowest level of validation, with all the work done by one lab. Inclusivity and exclusivity testing has been tested, but by a limited number of strains. The analyte was tested at a level based on the intended use of the method, with just normal background flora. There is no aging of the artificially-inoculated samples and no comparison to an existing reference culture

method. The expectation would be for the originating lab to continue to conduct further testing to eventually elevate the method to a higher level of validation.

Intended Use: Emergency needs.

A method developed for the detection of an analyte, or a matrix not previously recognized or identified as a threat to food safety or public health. As the first level in the development of any method designed for regulatory use; performance of the method at this level of scrutiny will determine, in part, whether further validation is useful or warranted.

NOTE:

Under emergency situations where the rapid development and deployment of a method is needed to immediately address an outbreak event, Level 1 criteria should be followed as closely as the situation will allow. Representatives of the MVC and Agency subject matter experts (SMEs) should be in close consultation with the originating laboratory. Once the crisis has past and it has been determined that there is a need for further

(16)

Method Validation Criteria

Level Two

This is a more robust study, with the possibility of regulatory strength depending on the circumstances. The originating lab has done a more comprehensive initial study, with

inclusivity/exclusivity levels at the AOAC Collaborative Study level. If possible, a comparison has been done to an existing reference culture method. One other independent laboratory has participated in the collaborative study. Some of the criteria of the study are at a lower level than the full AOAC study, but still appropriate for the developing method at this stage.

Intended Use: Emergency needs.

Slightly higher false-positive rates are acceptable as all samples analyzed with methods validated to this level will require confirmatory testing.

(17)

Level Three

This is a validation level that should have full regulatory strength. Most of the criteria followed by the originating lab are at the AOAC level, including inclusivity/exclusivity, analyte levels, competitor strains, aging, and comparison to existing method when available. The additional collaborating labs follow many of the criteria of an AOAC collaborative study.

Intended Use: All methods validated to this level of scrutiny are acceptable for use in any and all circumstances e.g. confirmatory analyses; regulatory sampling, and

compliance support.

Level Four

This validation level has criteria equivalent to the AOAC Collaborative Study. Any

method reaching this level of validation should be able to be submitted for adoption by the AOAC as a fully collaborated method.

(18)

Originating Laboratory Criteria

Level One: Urgent usage

Level Two: Independent lab validation

Level Three: Multiple lab collaborative

Level Four: Full collaborative study †_AOAC Collaborative Study # of target organism (inclusivity) 10 (20 for Salmonella) 50 (unless 50 aren't available)a,b 50 (unless 50 aren't available)a,b 50 (unless 50 aren't available)a,b 50 a,b # of non-target organism

(exclusivity) 10 strains 30 strainsc 30 strainsc 30 strainsc 30 strainsc

# of foods 1 or mored 1 or mored 1 or mored 1 or mored Up to 20 foodse

# of analyte levels/food matrixf

set level based on intended use and

negative control

One inoculated levelfand

uninoculated level

One inoculated levelf

and uninoculated level

and uninoculated level Replicates per food at each

level tested 20 20 20 20 20

Aging of inoculated samples

prior to testing No Yes Yesg Yesg Yesg

Addition of competitor strainh Normal background

flora

In 1 food at +1 log>analyte at

fractional positivef analyte level

In 1 food at +1 log>analyte at fractional positivef analyte level In 1 food at +1 log>analyte at fractional positivef analyte level In 1 food at +1 log>analyte at fractional positivef analyte level Comparison to recognized

method No Yes, if available Yes, if available Yes, if available Yes, if available

Table 1. ORIGINATING Laboratory Requirements

I. General Qualitative Guidelines for Microbial Analytes

(19)

Collaborating Laboratory Criteria Level Two: Independent lab validation

Level Three: Multiple lab collaborative

Level Four: Full

collaborative study AOAC Collaborative Study

# of laboratories providing usable data 2 5 10 10

# of foods 1 or morea 1 or morea 1 or morea 1 to 6 foodsb

# of strains of organism 1 per food 1 per food 1 per food 1 per food

# of analyte levels/food matrixc 2 levels: One inoculated levelc

3 levels: One inoculated levelc

one at 1 log higher and uninoculated level

3 levels: One inoculated levelc_{, one at 1 log higher and}

uninoculated level

3 levels: One inoculated levelc

one at 1 log higher and uninoculated level

# of replicate samples/food 8 per analyte level 8 per analyte level 8 per analyte level 6 per analyte level †

Aging of inoculated samples prior to testing Noe _Yesd _Yesd _Yesd

Comparison to Recognized Method Yes, if available Yes, if available Yes, if available Yes, if available

Microbiology Validation Category Examples,

continued

Table 2 - General Qualitative Guidelines for Microbial Analytes-Collaborating Laboratory Requirements

(20)

Originating Laboratory Criteria Category One: Urgent usage Category Two: Independent lab validation Category Three: Multiple lab collaborativ e Category Four: Full collaborativ e study

Replicates per strain 3 3 8 8

Comparison to recognized methoda No Yes, if available Yes, if available Yes, if available Collaborating Laboratory Criteria Level One: Urgent usage Level Two: Independent lab validation Level Three: Multiple lab collaborative

Level Four: Full collaborative study

# of laboratories

providing usable datab n/a 2 5 10

Replicates per strain n/a 3 8 6

Comparison to

Recognized Methoda n/a Yes, if available

Yes, if

available Yes, if available

II. General Qualitative Guidelines for Food-borne Microbial Pathogens

That Present Unique Isolation and/or Enrichment Challenges

†

Microbiology Validation Category Examples,

continued

Table 1. ORIGINATING Laboratory Requirements

(21)

III. CRITERIA AND GUIDANCE FOR THE VALIDATION OF FDA-

DEVELOPED MOLECULAR-BASED ASSAYS

•Inclusivity and exclusivity

•Target gene(s) and controls (positive and negative).

•Comparison to the Reference Method

IV. FOOD MATRIX EXTENSION FOR VALIDATED MICROBIOLOGY

METHODS

The validation of a method for a food matrix not previously included in a validation study is necessary to assure that the new matrix will produce neither high false positive (matrix is free from cross reactive substances) no high false negative rates (matrix is free of inhibitory substances)

• Guidance to Support Field Laboratory Expedience

• Guidance for the Formal Expansion of Validated Food Matrices

Microbiology Validation Category Examples,

continued

(22)

VI. CRITERIA AND GUIDANCE FOR THE VERIFICATION AND VALIDATION

OF COMMERCIALLY- AVAILABLE MICROBIOLOGICAL DIAGNOSTIC KITS AND PLATFORMS

1. For commercially-available microbiological diagnostic kits whose performance parameters have been fully validated in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc. 2. For commercially-available microbiological diagnostic kits whose performance parameters are supported by data

obtained through an abbreviated validation protocol and evaluated by an independent accrediting body e.g. AOAC-RI.

Validation of an Alternative method: Demonstration that adequate confidence is provided when the results obtained by the alternative method i.e. the commercially-available kit, are comparable to or exceed those obtained using the reference method using the statistical criteria contained in the approved validation protocol.

Verification: The confirmation by examination and the provision of objective evidence that specified requirements have been fulfilled.

Criteria

For Kits Fully Validated in a Collaborative Study Monitored by an Independent Accrediting Body

(23)

23

Current Micro MVS Priorities

I. Hepatitis A Virus

a. Real-time RT-qPCR assay to detect Hepatitis A virus b. In a food matrix (green onions)

Status: Final report nearing completion

2. †_{Non-O157:H7 STEC}

Screening method to detect non-O157:H7 STECs using the BioPlex

Status: Multi-lab collaborative study has been completed..

3. ‡_{Salmonella Enteritidis}

Isolation and Detection of Salmonella Enteritidis (SE) from Whole Shell Eggs-Cultural and Molecular Applications

4.

Norovirus

a. Real-time RT-qPCR assay to detect Noroviruses

b. In a food matrix (molluscan shellfish)

(24)

24

Method Authors

Andrew Lin

Teresa Lee

Laurie Clotilde

Julie A. Kase

Insook Son

J. Mark Carter

Carol R Lauzon

Validation of an Identification Method for

Shiga-toxigenic

E. coli

Somatic (O) Groups using the

(25)

25

• STECs are a significant public health concern

–

Non-O157 STECs are responsible for over 60% of STEC

infections or an estimated 112,000 illnesses in the U.S. each year

– Over 74.2% of STEC infections in the U.S. are caused by serogroups O26

(23.9%), O45 (7.8%), O91 (2.3%), O103 (16.7%), O111 (12.6%), O121 (7.5%), and O145 (3.4%)

– O26, O103, O111, O121, and O145 are known to cause HC and HUS, and

O45 is associated with HC

– Other serogroups that may cause HC and HUS, but are less commonly

isolated, are O91, O113, and O128

(26)

26

BioPlex Assay Overview

• Must be performed on Pure Cultures

• Bead-based assay that can be multiplexed

– Conventional and real-time PCR assays for STEC O serogroup - limited by resolution of band sizes

on a gel, or the # of fluorescent channels

• 96-well plates

• Targets = nucleic acid, proteins – Ab, antigens, cytokines

• Each bead = a separate assay

• 100 different color-coded beads (magnetic

or polystyrene)

– Unique color comes from different ratios of two distinct flourophores

• Two lasers, fluidics, optics, detectors

• Automated, High-Throughput, Fast, Expandable

(27)

27

Method Status

• In-House Validation Study Completed

–

Paper

–

LIB

• 5 Lab Collaborative Study Successfully Completed

• Approved by the BAM Council

(28)

28

In-house Validation Results Inclusivity Panel

(29)

40

Exclusivity Panel

(30)

BioPlex Results

• Fluorescence signals are quantified for each micro

bead

• Signal to background ratios are calculated, where

background is measured using all no template

reactions

• Samples were considered to be positive when

signal to background ratio was greater than 5.0

(31)

31

BioPlex Results

(32)

32

Development and Validation of a Method for the

Detection and Isolation of Salmonella Enteritidis (SE)

from Whole Shell Eggs

Guodong Zhang

Eve Thau

Eric W. Brown

Thomas Hammack

Validation of a Method for

(33)

33

Background

• SE is the second most commonly isolated Salmonella serotype

in the United States

–

14.18% (1968-1998)

• SE is most commonly associated with whole shell eggs and

egg products

• FDA’s egg rule (74 FR 33030) recommends various preventive

control measures including sampling and sample analysis

• FDA’s BAM reference culture method for the detection of SE in

whole shell eggs takes 9 days to complete

• There are no AOAC Official Methods of Analysis methods for

the detection of SE in whole shell eggs

Validation of a Method for Salmonella in

Whole Shell Eggs

(34)

34

Clean surface Surface disinfection

20 eggs to a container Hand homogenize sample

Hold at room Temp (20-24o_C)

for 96 h 25 ml eggs to 225 ml TSB + ferrous sulfate; 24 h at 35o_C 1 ml to 10 ml TT 24 h at 35o_C Streak on XLD, BS, HE 24 h at 35o_C TSI, LIA 24 h at 35o_C Serological Test 0.1 ml to 10 ml RV 24 h at 42o_C Streak on XLD, BS, HE 24 h at 42o_C TSI, LIA 24 h at 35o_C Serological Test