Standard for Human Cytomegalovirus
R. T. Hayden,aJ. Preiksaitis,bY. Tong,bX. Pang,b,cY. Sun,dL. Tang,dL. Cook,eS. Pounds,dJ. Fryer,fA. M. Caliendog
Departments of Pathologyaand Biostatistics,dSt. Jude Children’s Research Hospital, Memphis, Tennessee, USA; University of Alberta, Edmonton, Alberta, Canadab; Provincial Laboratory for Public Health, Edmonton, Alberta, Canadac; Department of Laboratory Medicine, University of Washington, Seattle, Washington, USAe; National Institute for Biological Standards and Control, Potters Bar, United Kingdomf; Department of Medicine, Alpert Medical School of Brown University, Providence, Rhode Island, USAg
Quantitative detection of cytomegalovirus (CMV) DNA has become a standard part of care for many groups of immunocompro-mised patients; recent development of the first WHO international standard for human CMV DNA has raised hopes of reducing interlaboratory variability of results. Commutability of reference material has been shown to be necessary if such material is to reduce variability among laboratories. Here we evaluated the commutability of the WHO standard using 10 different real-time quantitative CMV PCR assays run by eight different laboratories. Test panels, including aliquots of 50 patient samples (40 posi-tive samples and 10 negaposi-tive samples) and lyophilized CMV standard, were run, with each testing center using its own quantita-tive calibrators, reagents, and nucleic acid extraction methods. Commutability was assessed both on a pairwise basis and over the entire group of assays, using linear regression and correspondence analyses. Commutability of the WHO material differed among the tests that were evaluated, and these differences appeared to vary depending on the method of statistical analysis used and the cohort of assays included in the analysis. Depending on the methodology used, the WHO material showed poor or ab-sent commutability with up to 50% of assays. Determination of commutability may require a multifaceted approach; the lack of commutability seen when using the WHO standard with several of the assays here suggests that further work is needed to bring us toward true consensus.
Q
uantitative detection of cytomegalovirus (CMV) DNA hasbecome a standard part of care for many groups of immuno-compromised patients, but particularly for transplant recipients (1–4). A host of laboratory-developed real-time PCR methods have been described for this purpose, and more recently, FDA-approved assays have become available. The benefits of such test-ing to trigger preemptive or therapeutic treatments or to monitor the efficacy of such treatments and determine therapeutic end-points are widely accepted. However, benchmark values for such interventions have yet to be established, and it is now decades after the introduction of molecular diagnostics. The reasons for this lack of consensus have become clear over a number of studies which have shown poor quantitative concordance between labo-ratories and between methods. Several investigators have docu-mented discordant results for human cytomegalovirus (HCMV) testing as well as for other viral load measures that depend largely upon methods developed in the laboratory (5–7). The reasons for such variability may be numerous (8); however, intuitively, differ-ent quantitative calibrators may directly relate to differdiffer-ent quan-titative results. This has been confirmed in the literature, and the importance of developing consensus standards has been well ac-cepted (9–11) and confirmed by past experience with other vi-ruses.
The advent of the first WHO International Standard for HCMV DNA was meant to address this need (12). This material consists of a known quantity of the CMV merlin strain, prepared in buffer with human serum albumin in lyophilized 1-ml aliquots. Initially made available in 2010, limited quantities are available for sale to individual laboratories and vendors intended for calibrat-ing secondary standards for wider dissemination to the diagnostic community. It has been hoped that implementation of calibrators traceable to a higher-order reference would rapidly result in
re-duced interlaboratory variability of results, allowing portability of clinical data, definition of consensus therapeutic breakpoints, and improved comparability of research data. However, challenges to the fulfillment of these promises remain. The utilization of sec-ondary standards is a logical means of broadening the reach of a limited quantity of WHO reference standards. This is a common-place practice, but one that is fraught with challenge. Recent work has examined whether such secondary materials are actually com-parable in value and provide an accurate representation of the WHO standards (13). Commutability should be assessed at all phases of the process of quantification, including calibrators, higher-order, traceable standards, and considering other aspects of reaction chemistry and performance that can affect quantitative results.
However, perhaps more important are the questions of whether the WHO standards themselves behave similarly in dif-ferent assay environments and whether this behavior is similar to
Received5 June 2015Returned for modification14 July 2015
Accepted6 August 2015
Accepted manuscript posted online12 August 2015
CitationHayden RT, Preiksaitis J, Tong Y, Pang X, Sun Y, Tang L, Cook L, Pounds S,
Fryer J, Caliendo AM. 2015. Commutability of the first World Health Organization international standard for human cytomegalovirus. J Clin Microbiol 53:3325–3333.
doi:10.1128/JCM.01495-15.
Editor:M. J. Loeffelholz
Address correspondence to R. T. Hayden, Randall.Hayden@stjude.org.
Supplemental material for this article may be found athttp://dx.doi.org/10.1128
/JCM.01495-15.
Copyright © 2015, American Society for Microbiology. All Rights Reserved.
doi:10.1128/JCM.01495-15
on May 16, 2020 by guest
http://jcm.asm.org/
that of clinical material. These questions embody the principles of commutability which have been well defined and accepted in the clinical chemistry literature and more recently in clinical molecu-lar virology (14–16). Commutability of reference material is crit-ical to producing quantitative values reflective of realityin vivo and is a necessary ingredient if such material is to reduce variabil-ity among laboratories. In fact, commutable quantitative stan-dards have been shown to improve interlaboratory agreement, while noncommutable standards have actually been shown to in-crease disparity among testing centers (17). Here we examine the commutability of the first WHO International Standard (WHO IS) for HCMV (09/162), among a wide variety of commonly used CMV quantitative assays.
MATERIALS AND METHODS
Patient samples and quantitative standards.Plasma samples from 50 patient samples, 40 previously determined by routine clinical testing to be positive for CMV and 10 negative for CMV, were included in the study. Each sample was from a different patient, except for two positive samples run in duplicate, which means that the 40 positive samples were from 38 patients. All samples were leftover material, previously collected for clin-ical testing from solid-organ transplant patients at the Provincial Labora-tory for Public Health (ProvLab) in Edmonton, Canada. Patients and samples were selected such that positive results spanned a dynamic range
from log102 to log10 6 IU/ml. Serial samples from each patient were
pooled prior to extraction to produce at least 3.5 ml (7 ml for samples tested in duplicate) sufficient for processing. Approval from the Univer-sity of Alberta ethics review board was obtained for use of the clinical samples for this study. All samples were anonymized subsequent to
pool-ing, then aliquoted, and frozen at⫺70°C. Aliquots were shipped to each
testing center on dry ice, and the volumes of the aliquots were sufficient for testing by the method(s) used by each given laboratory. Similarly, vials containing the first WHO international standard for HCMV (lyophilized)
(5⫻106IU/ml) were distributed to each testing site, together with CMV
antibody-negative and CMV DNA-negative plasma from a single donor. Once the pooled clinical sample aliquots were received, they were thawed only once by each site prior to use; nucleic acid extracts were stored less than 2 days at 4°C prior to use. The WHO standard was reconstituted (locally) in 1.0 ml of deionized, nuclease-free molecular biology-grade water, and four 10-fold serial dilutions were performed with the provided
CMV-negative plasma, resulting in five dilutions, from 2.7 to 6.7 log10
IU/ml (5⫻102to 5⫻106IU/ml). The diluted standards were stored at
4°C and used within 48 h of reconstitution.
Quantitative testing.Test panels, including aliquots of each patient sample, lyophilized CMV standard, and CMV-negative plasma, were dis-tributed to each of six testing centers. Each center used its routine sample preparation methods, including nucleic acid extraction, and performed between one and three different CMV real-time quantitative PCR as-says (in singlet) on the panel members. There were a total of 10
differ-ent assays run (Table 1), with two sites running the Qiagen (Artus)
method, two sites running Altona, and two sites using (different) lab-oratory-developed tests (LDTs), meaning that the panel was run a total of 10 times. Testing was performed per each laboratory’s standard operating procedures (including quantitative calibrators, reagents, cy-cling conditions, and data analysis), where applicable based on man-ufacturer’s recommendations. All quantitative results were reported in international units per milliliter. Methodologic details for each
labo-ratory were collected by questionnaire and are shown inTable 1. All 50
clinical samples and all concentrations of the WHO standard were run as unknowns, in an identical manner.
Statistical methods.We adopted the definition of “commutability” as described in Clinical and Laboratory Standards Institute (CLSI)
guide-lines and International Organization for Standards (ISO) documents (18,
19). Commutability describes the property of a reference material of
in-terest, which is the closeness of quantitative relationships among mea-sured results obtained from various assays or measurement procedures, compared to the relationship derived from human specimens.
Commutability of the WHO standard material was first examined with linear regression analyses between all pairs of measurement
proce-dures, as previously described (18,19). In brief, a regression line was fitted
with average values of replicated measurements of patient samples from two measurement procedures. A 95% prediction interval for patient sam-ple was then plotted and compared to points representing the average measures of the WHO standard material from these two procedures. The standard material was considered commutable between two measure-ment procedures if the measured values of all WHO standard dilutions fell within the prediction limits, noncommutable if more than one dilution fell out of the limits, and marginally commutable if exactly one dilution fell outside those limits. The measure of commutability in this context was specific to the two assays evaluated in any given regression.
The latter reflects a primary limitation of the linear regression ap-proach, in that only two assays can be simultaneously evaluated. Thus, we also assessed the commutability of the WHO standard material across all
10 measurement procedures with correspondence analysis (18,19).
Un-like linear regression, correspondence analysis presents the behavior pat-tern of human specimens and the WHO standard material multidimen-sionally, reflecting data from all 10 assays. Patient samples and assay profiles were first used to develop the factorial plane. The fundamental idea of a profile (patient sample profile or assay profile) was to transform each data point to a row- or column-specific proportion, where a patient sample or an assay represents a row or column in a data matrix. Similar to principal component analysis, correspondence analysis summarized the majority of information characterizing associations of behavior of patient specimens and assays in a space of significantly lower dimensions (usually a two-dimensional plane called the “factorial plane”), by replacing origi-nal data (measures of various assays with multiple patient specimens) with a new set of orthogonal linear coordinates derived from eigenvectors
of a covariance matrix of2distance. The major distinction from
princi-pal component is that the latter relies on eigenvectors derived from a covariance matrix of Euclidean distances. The usage of correspondence analysis allows a simple two-dimensional plot representation of the orig-inal data when the data are multidimensional (multiple patient specimens and multiple assays being studied). The axes of this plot are usually con-sidered the first two “extracted factors” from the data, which together explains the majority of information of the original data. The WHO stan-dard material measures were then projected on the factorial plane devel-oped from patient specimens and assay profiles, described as “inactive
elements” in reference20. They were considered “inactive” because they
did not participate in establishing the two-dimensional plane, which al-lowed one to examine whether they behave similarly to patient specimens in various assays. The WHO standard profile falling within the 95% con-fidence region on the plane was considered commutable in general on all assays, and the degree of commutability could be evaluated visually by the distance of the projected WHO standard material profile to the center of patient specimen profiles. Although not the focus of this paper, the cor-respondence plot could also inform how different the behavior of a par-ticular assay was from other assays on patient samples, by examining whether this assay’s profile fell within the prediction ellipse or not. The mathematical procedure of correspondence analysis is very complex, and
interested readers can refer to Bretaudiere et al. (23) for technical details.
All statistical analyses were performed with SAS 9.3 unless otherwise spec-ified.
RESULTS
A total of 50 patient samples and five concentrations of WHO standard material were assayed in singlet by six laboratories using eight different quantitative PCR reagent sets (two of which were run by two different laboratories), resulting in 10 data sets for inclusion in the analysis. It should be noted that that while two
on May 16, 2020 by guest
http://jcm.asm.org/
TABLE 1 CMV test assays and laboratories Test lab Assay/reagent a Manufacturer of assay/reagent b Registration status c Gene target(s) d Amplicon size(s) (bp) Calibrator/secondary standard e Probe chemistry f Extraction method Extraction kit g Detection system Lab A MultiCode-RTx CMV Luminex Corporation, Madison, WI, USA Commercial, ASRs DNA polymerase (UL54) 52 LD amplicon FRET-labeled primers, no probes Qiagen EZ1 Advanced XL Qiagen EZ1 Advanced XL DNA tissue card ABI Prism 7500 Lab A Cobas AmpliPrep/Cobas TaqMan CMV test (CAP/CTM) Roche Molecular System. Pleasanton, CA, USA Commercial, FDA approved CE marked DNA polymerase (UL54) 340 MP plasmid TaqMan Roche Cobas AmpliPrep Roche Cobas AmpliPrep Cobas TaqMan 48 analyzer Lab B RealStar CMV PCR kit 1.0 Altona Diagnostics, Hamburg, Germany Commercial, CE marked Confidential ⬍ 100 MP amplicon TaqMan King Fisher FLex King Fisher MagMAX viral isolation kit Rotor-Gene 3000/6000 Lab C RealStar CMV PCR kit 1.0 Altona Diagnostics, Hamburg, Germany Commercial, CE marked Confidential ⬍ 100 MP amplicon TaqMan Qiagen QIAcube Qiagen DNA minikit ABI 7500 Lab C CMV LC-PCR LDT, Edmonton, Canada LDT Glycoprotein B (UL55) 254 LD plasmid FRET Qiagen QIAcube Qiagen DNA minikit LightCycler 1.0 Lab C Artus CMV RG PCR kit Qiagen, Hilden, Germany Commercial, FDA approved CE marked MIE 105 MP amplicon TaqMan Qiagen EZ1 Advanced Qiagen EZ1 DSP virus kit Rotor-Gene Q Lab D Quantitative TaqMan PCR (UL55/UL123-exon 4) LDT, Seattle, WA, USA LDT Glycoprotein B (UL55) and IE (UL123) 64 (UL55), 76 (UL123) Plasmid calibrated to Acrometrix whole virus TaqMan Roche MagNa Pure 96 Roche MagNA Pure 96 DNA and viral NA small-volume kit ABI step one Lab E Abbott RealTime CMV Abbott Molecular, Des Plaines, IL, USA Commercial, CE marked UL34 and UL80.5 95 (UL80.5) 105 (UL34) MP plasmid TaqMan Abbott m2000 Abbott sample prep system DNA kit Abbott m2000 Lab E Artus CMV RG PCR kit Qiagen, Hilden, Germany Commercial, FDA approved CE marked MIE 105 MP amplicon TaqMan Qiagen EZ1 Advanced XL Qiagen EZQ virus minikit v2.0 Rotor-Gene Q Lab F Simplexa CMV kit Focus Diagnostics, Inc., Cypress, CA, USA Commercial, CE marked UL83 86 MP amplicon Scorpion Roche MagNa Pure LC Roche MagNa Pure LC total nucleic acid isolation kit 3M integrated cycler a LC-PCR, LightCycler PCR. b The manufacturer of the assay or reagent is given. If the assay/reagent was developed in the laboratory, LDT (laboratory-developed test) and the loca tion of the laboratory are given. c ASRs, analyte-specific reagents; CE, Conformité Européenne. d MIE, major immediate early gene. e LD, laboratory developed; MP, manufacturer provided (same manufacturer as assay). f FRET, fluorescence resonance energy transfer. g NA, nucleic acid.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:3.585.186.441.65.721.2]real-time reagent sets were each run in two sites, each had some differences in other aspects of methodology (such as extraction method [Table 1]) that might be expected to affect quantitative results. For the purposes of commutability analysis, all measure-ments below the assay limit of quantification for each assay were excluded. The reader is referred elsewhere for a discussion of vari-ability in the results and the value of the WHO standard in im-proving result harmonization (J. Preiksaitis, R. Hayden, Y. Tong, X. Pang, J. Fryer, A. Heath, and A. Caliendo, unpublished data). Analysis here was limited to the question of commutability. The latter analysis was conducted using both pairwise linear regression and correspondence analysis.
Linear regression was performed between results of clinical samples from two lab-assay combinations and the corresponding 95% prediction interval was calculated (see the supplemental ma-terial). Measured results for WHO IS dilutions were then com-pared against the 95% prediction intervals. This analysis included 45 assay pairs, of which 25 were commutable, 12 were not com-mutable, and 8 were marginally comcom-mutable, as defined above in “Statistical methods” (Fig. 1).
WHO standards were commutable for all assay pairs including the Lab B Altona assay and for most assay pairs including either the Lab E Qiagen, Lab A Roche, or Lab D LDT assay. All other data sets (representing individual assays run at a given site) included at least five assay pairs showing either commutability or marginal commutability. The Qiagen reagents run by Lab C had the fewest assay pairs (two) showing commutability; the Lab C Altona and Qiagen assays and the Focus assay both showed the most noncom-mutable assay pairs, with each including four noncomnoncom-mutable pairs. Abbott, Luminex, and the Lab C LDT each were included in three noncommutable assay pairs. Based on pairwise linear regres-sions, commutability of the WHO standard material appeared to be highly dependent on the assay. The WHO standard material was commutable or marginally commutable in more than 70% of the pairs under consideration; however, it was still not commut-able in more than 20% of the pairs, indicating much room for improvement in order to achieve a more universal commutability.
Correspondence analysis was performed to explore the com-mutability of the WHO standard material on all 10 assays (Fig. 2). The two factorial axes explained about 61% (axis 1, 32%; axis 2, 29%) of the information contained in the mathematically trans-formed data. The percentage is the proportion of the data infor-mation explained by the first two extracted factors. Existing liter-ature suggested that factorial of higher orders, in our case consisting of 39% of the total data information, may relate to random errors (20). The distance on the axis from the center of the plot indicates discrepancy in behavior pattern. For example, pa-tient samples projected on the factorial plane falling far from the center indicate that they have dissimilar interassay behaviors, while projected assays far from the center indicate that the assay yields disparate measure profiles with respect to all patient speci-mens under examination. Results suggested that Lab A Roche as-say, followed by the Lab C LDT and Lab B Altona assays, in general yielded results that differed the most from other assays. Indeed, Roche results were lowest on most patient specimens, while Lab C LDT reagents and the Lab B Altona assay tended to yield lower results from patient samples, compared to mean patient sample loads across all assays (results not shown). On the other hand, commutability by definition does not necessarily indicate the closeness in numerical quantitative results between different as-says, when measures are made on the same sample. Rather, a high degree of commutability suggests agreement in the mathematical relationship when comparing results between assays and whether results are derived from patient specimens or from reference ma-terial dilutions. Considering the results of the 10 assays together, the WHO material seemed overall marginally commutable, using the 95% prediction area as the determining criterion. Conclusions were the same if 90% or 99% prediction areas were adopted (re-sults not shown). This is in fact consistent with the overall conclu-sions from linear regression analysis, where we found that the WHO standard material was commutable on the majority of the paired assays. However, when evaluated in the context of all 10 assays, the WHO standard material measures fell away from the
FIG 1Assessment of commutability of WHO standard material using linear regression with prediction limits according to CLSI guidelines. The numbers in the rectangles are the number of tests that were commutable/total number of tests conducted. (See scatter and linear regression plots for each pair of methods in Fig. S1 in the supplemental material). LDT, laboratory-developed test.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:4.585.136.451.65.268.2]center of the clustered patient specimen results, suggesting that overall commutability was suboptimal.
Correspondence analysis has a potential advantage over linear regression in its ability to assess more than two measurement pro-cedures at the same time. However, this study indicates that nei-ther method should necessarily be viewed alone. For example, on the basis of linear regression results, we found that results ob-tained from Lab C LDT, Lab E Qiagen, Lab E Roche, Lab D LDT, and Lab B Altona assays were always fully commutable to each other (Fig. 1), while the other five assays seemed to possess non-commutable or marginally non-commutable results to each other. Us-ing this behavior as a guide, the 10 assays can be divided into two groups, with correspondence analysis performed on each sub-group. In the first five assays mentioned above, WHO material shows strong commutability (Fig. 3). In contrast, in the remaining five assays, WHO standard is not commutable (Fig. 4). This dem-onstrates that apparent commutability can vary based on the an-alytical methodology applied and on which assays are included in any given evaluation. Furthermore, it suggests that the ultimate determination of commutability may best be thought of as multi-phasic process, using a consensus of these different data analysis methods.
DISCUSSION
Previous work has shown that commutability is a critical property to evaluate when developing quantitative standards (14,16,20). Commutable standards can dramatically improve interlaboratory agreement of viral load measures, while noncommutable
stan-dards can have a detrimental impact and actually widen the dis-parity between results (17). This work extends our understanding of commutability with a look at a large number of assays per-formed at numerous clinical laboratories, critically examining the commutability of the first WHO international standard for HCMV by two statistical approaches. The findings here tell us that even though this biological standard has been thoroughly charac-terized by other measures, we can see a range of behaviors in relation to actual clinical samples. That is, the commutability of the WHO material differs among the methods that were evalu-ated, and these differences appeared more or less pronounced depending on the method of statistical analysis used and the co-hort of assays included in the analysis. These analyses showed that several of the assays (or assay pairs) showed either reduced com-mutability or noncomcom-mutability. It should be remembered that reduced result agreement and reduced commutability, while po-tentially interrelated, are still two different properties. However, it is also reasonable to suppose that any of the numerous aspects of assay design and performance that have been shown to affect re-sult agreement may also potentially affect commutability. Nota-bly, secondary standards may themselves suffer from reduced agreement and commutability. We can state from the present re-sults that variable commutability is seen among assays using the WHO standard. However, since these assays represent the sum total of many different reagents (including assay-specific second-ary standards) and methods, one cannot assess from the data pre-sented here what impact (detrimental or otherwise) these second-ary standards or other aspects of any of these assays contributed to
FIG 2Assessment of commutability of WHO standard material using correspondence analysis for all 10 methods. Projection on the first significant factorial plane is shown. The axes represent the first two dimensions of the correspondence analysis that account for most of the information in the clinical samples. The proportions of variance explained by axes are reported in parentheses. Clinical sample projections, WHO standard dilution projections, and method projections are indicated. The black ellipse defines the 95% confidence area describing the variability in the clinical samples with respect to all 10 methods.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:5.585.111.475.67.345.2]altered commutability. Irrespective of the root causes, the findings here present a fundamental challenge to the successful adaptation of this or any other truly universal quantitative standard, if such adaptation alone is to improve quantitative agreement among lab-oratories.
The value of improving interlaboratory agreement is well rec-ognized and includes allowing portability of results, development of standardized therapeutic breakpoints, and improved ability to interpret studies in the literature. To date, several authors have shown that various laboratories or methods can produce widely disparate results from the same samples (5–7,21). While these differences can be attributed to numerous factors, it is clear that the variety of quantitative standards used among molecular meth-ods and the previous absence of an international quantitative ref-erence standard (for CMV and for other viruses) has played a major role. Further, it has been widely agreed that the develop-ment of such a reference standard would be a crucial step in bring-ing results into closer agreement. That the development of a con-sensus reference standard in and of itself may not be sufficient is exemplified by studies showing the importance of commutability to the successful implementation of shared quantitative stan-dards. This has primarily been demonstrated on a smaller scale, with just a few assays. In the latter case, implementation of com-mon, commercially produced standards could only improve agreement among labs when commutable and actually could re-duce agreement when noncommutable (17). Other factors may come into play as well. Commercially produced standards may
not be equivalent in all aspects; they may not have equal trueness or comparability to WHO material, nor may they behave the same among assays or reagents (13).
Commmutability is an assay-specific characteristic and can be influenced by all steps in the measurement procedures including the standards used. Any statements about the commutability of a reference standard must include more information regarding the assays evaluated, as the reference standard may be commutable for some assays and noncommutable for others. Both the linear re-gression and correspondence analysis methods are limited in the sense that they can evaluate only relative commutability among the set of assays included in the analysis. Points falling outside the prediction region indicate only that the included assays do not have the same level of commutability; they do not directly quan-tify commutability of any individual assay. Thus, we cannot state which assays truly have the best intrinsic commutability but can only identify specific pairs or sets of assays that have similar levels of commutability with one another.
This creates a complex challenge for individual laboratories, for manufacturers, and for the international community as a whole. Commutability may need to be evaluated subsequent to development of reference material for most or all potential assays in which it may be used. This is a particular challenge currently, with a fragmented diagnostic field, including FDA-cleared prod-ucts, analyte-specific reagents (ASRs), and pure laboratory-devel-oped tests (LDTs). As shown here, it is not only the reagents that are important, but the method as a whole. Identical reagents run
FIG 3Assessment of commutability of WHO standard material using correspondence analysis for the Lab C LDT, Lab E Qiagen, Lab A Roche, Lab D LDT, and Lab B Altona methods. Projection on the first significant factorial plane is shown. The axes represent the first two dimensions of the correspondence analysis that account for most of the information in the clinical samples. The proportions of variance explained by axes are reported in parentheses. Clinical sample projections, WHO standard dilution projections, and method projections are indicated. The black ellipse defines the 95% confidence area describing the variability in the clinical samples with respect to all 10 methods.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:6.585.111.476.67.345.2]by different laboratories (varying primarily by extraction method) in the present study yielded different results with respect to com-mutability. Of course, it is not possible to evaluate all methods simultaneously, but we feel that this study does present a repre-sentative sampling for the purposes of demonstrating the princi-ples involved. Likely, in the current setting, large survey studies, such as this one, may be needed periodically, to assess previously unavailable methods. As the field migrates primarily toward the use of secondary standards, it will be critical that these standards are not only traceable but also normalized to the WHO material, using methods for which the latter has already shown commutable behavior. It will then be incumbent on the manufacturers of the secondary standards to demonstrate commutability for their ma-terials, using as many methods as possible. Likewise, laboratory directors should seek out assays and secondary standards that have demonstrated commutability for those particular assays. As the field consolidates to fewer, commercially produced methods, the importance of commutability remains, although the challenge of evaluating the entire universe of methods may diminish.
The complexity of the problem, however, is not limited to our ability to evaluate all assays. The best means of evaluation remains a question, to some extent both complicated and clarified here. The two primary means of evaluation utilized in this study, linear regression and correspondence analyses yielded similar, but not identical, results. Linear regression, by its nature, is limited to pairwise analyses. However, correspondence analysis can include any number of methods simultaneously, and the results are
de-pendent (as with linear regression) on the particular methods that are included in the evaluation. To some extent, this may make the entire process seem somewhat arbitrary, and in fact, other assays might have been included in this study that could change the results to some extent. Nonetheless, by including a large, presum-ably representative, group of assays, there may be some assurance that we have a picture of the variability of behavior of the WHO standards across available assays as a whole. Furthermore, by starting with a large number of methods and gradually looking at smaller groups, most potential combinations can be explored. While which assays are commutable does change, the relative be-havior of different methods remained fairly consistent. Here two groups of methods seemed to cluster in terms of behavior relative to commutability. Depending on whether they were evaluated to-gether or separately, all appeared commutable, or one of the two groups fell out as noncommutable. The optimal determination of commutability may then be somewhat of an iterative process. Pairwise evaluation by linear regression could be followed by in-clusion of all methods in correspondence analysis, first together and then in subsets. Commutability may best be thought of as a consensus of the two methods. It is important to realize as well that the absence of commutability must also be viewed critically. If the results of patient testing cluster very tightly, the 95% confi-dence interval around those results might be so small that even a slight (perhaps nonclinically significant) variation in behavior of the reference material would result in a noncommutable designa-tion. This is illustrated if one compares the linear regression plots
FIG 4Assessment of commutability of WHO standard material using correspondence analysis for the Lab A Luminex, Lab E Abbott, Lab C Altona, Lab C Qiagen, and Lab F Focus methods. Projection on the first significant factorial plane is shown. The axes represent the first two dimensions of the correspondence analysis that account for most of the information in the clinical samples. The proportions of variance explained by axes are reported in parentheses. Clinical sample projections, WHO standard dilution projections, and method projections are indicated. The black ellipse defines the 95% confidence area describing the variability in the clinical samples with respect to all 10 methods.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:7.585.112.475.67.344.2]shown in the supplemental material (see Fig. S1 in the supplemen-tal material;Fig. 5); these show a wide variation in the 95% con-fidence intervals among assay pairs. For example, the plot inFig. 5Ashows quite a wide interval, while the one inFig. 5Bshows a much narrower interval, the latter then requiring much less vari-ation in behavior to be classified as noncommutable (Fig. 5). Therefore, a degree of judgment might be indicated in deciding when noncommutability is significant.
The reasons for noncommutability have not previously been explored with respect to viral load testing and cannot be deter-mined based on the results here. However, some speculation can be made based on these results. As noted above, differing behavior between clinical samples and reference materials could have to do with any component of the real-time PCR process, in turn pro-ducing differences in relative results. One example might be of a given extraction process giving better purification from reconsti-tuted, purified virus than from actual patient material. Another might be differential primer or probe binding to the HCMV strain used in the WHO standard compared to the circulating strains. Others might involve variable cycling conditions or the use of different polymerases or of different secondary standards as cali-brators for individual assays. Finally, much may relate to the frag-mentation of circulating virus that has previously been shown, compared to intact virus used in the reference standard. Fragmen-tation may result in altered secondary and tertiary structures, po-tentially affecting amplification efficiency. Perhaps more impor-tantly, fragmentation may favor those primer/probe sets with smaller amplicons, while the primer/probe sets with larger targets may have differential activity with intact viral particles that com-prise a higher proportion of control material. A study including a small group of renal transplant patients showed that most circu-lating CMV DNA was highly fragmented and also demonstrated that assays with smaller amplicons generated markedly different quantitative data from those patients (22). The impact of ampli-con size on assay performance appears to be supported here by the fact that two of the three assays with the most outlier points by correspondence analysis (Fig. 2) had the two largest amplicon sizes of all methods (it should be noted that the Altona reagents were used in two methods, only one of which was an outlier, although different extraction methods were used). If it is true that smaller target and amplicons enhance commutability and harmo-nization of results, this may prompt manufacturers to move to-ward smaller amplicons, or it may suggest that fragmentation or
shearing of reference material be performed during the manufac-turing process to more closely mimic clinical samples and there-fore improve commutability.
Future work is planned to further explore both the causes of commutability and the impact of measures such as pretreatment of standards by fragmentation. Clearly, we are learning that the improvement of viral load testing is a tremendous challenge, par-ticularly in the context of a wide range of test manufacturers, reagents, reaction conditions, and sample preparatory techniques. Development of international quantitative reference standards has been a major accomplishment, but for those standards to have their desired impact on interlaboratory agreement, reference ma-terial must be commutable. The determination of commutability, in turn, may require a multifaceted approach, and the apparent lack of universal commutability suggests that further work is needed to bring us toward a true consensus standard.
ACKNOWLEDGMENTS
The assistance of Alan Heath, National Institute for Biological Standards and Control, in study design is gratefully acknowledged. Thanks also to Min Cao and The Provincial Laboratory for Public Health (ProvLab) for processing and providing samples, to the National Institute for Biological Standards and Control for providing the first WHO International Stan-dard for HCMV DNA, to Roche Diagnostics, Abbott Molecular, and Qia-gen for providing test kits, and to Focus Diagnostics for providing testing services.
This work was supported in part by the Emory Center for AIDS Re-search (P30 AI050409) and by ALSAC. Jacqueline Fryer is employed by the National Institute for Biological Standards and Control, which pro-duced and provided the WHO International Standard evaluated in this study.
REFERENCES
1.Atkinson C, Emery VC.2011. Cytomegalovirus quantification: where to
next in optimising patient management? J Clin Virol51:223–228.http:
//dx.doi.org/10.1016/j.jcv.2011.04.007.
2.Bieniek R, Kirby JE, Cheng A, Eichelberger K, Qian QF.2011. Effective use of PCR for the detection of cytomegalovirus viremia and monitoring
therapy in immunocompromised patients. LabMedicine42:339 –343.
3.Boeckh M, Ljungman P.2009. How we treat cytomegalovirus in
hema-topoietic cell transplant recipients. Blood113:5711–5719.http://dx.doi
.org/10.1182/blood-2008-10-143560.
4.Kraft CS, Armstrong WS, Caliendo AM.2012. Interpreting quantitative cytomegalovirus DNA testing: understanding the laboratory perspective.
Clin Infect Dis54:1793–1797.http://dx.doi.org/10.1093/cid/cis212.
5.Hayden RT, Hokanson KM, Pounds SB, Bankowski MJ, Belzer SW, FIG 5Representative linear regression analyses demonstrating a wide variation in the 95% confidence intervals among assay pairs.
on May 16, 2020 by guest
http://jcm.asm.org/
[image:8.585.135.452.65.212.2]Carr J, Diorio D, Forman MS, Joshi Y, Hillyard D, Hodinka RL, Nikiforova MN, Romain CA, Stevenson J, Valsamakis A, Balfour HH, Jr, US EBV Working Group.2008. Multicenter comparison of different real-time PCR assays for quantitative detection of Epstein-Barr virus. J
Clin Microbiol46:157–163.http://dx.doi.org/10.1128/JCM.01252-07.
6.Hoffman NG, Cook L, Atienza EE, Limaye AP, Jerome KR. 2008. Marked variability of BK virus load measurement using quantitative
real-time PCR among commonly used assays. J Clin Microbiol46:2671–2680.
http://dx.doi.org/10.1128/JCM.00258-08.
7.Pang XL, Fox JD, Fenton JM, Miller GG, Caliendo AM, Preiksaitis JK.
2009. Interlaboratory comparison of cytomegalovirus viral load assays.
Am J Transplant9:258 –268.http://dx.doi.org/10.1111/j.1600-6143.2008
.02513.x.
8.Hayden RT, Yan X, Wick MT, Rodriguez AB, Xiong X, Ginocchio CC, Mitchell MJ, Caliendo AM.2012. Factors contributing to variability of quantitative viral PCR results in proficiency testing samples: a
multivari-ate analysis. J Clin Microbiol50:337–345.http://dx.doi.org/10.1128/JCM
.01287-11.
9.Gantzer ML, Miller WG.2012. Harmonisation of measurement
proce-dures: how do we get it done? Clin Biochem Rev33:95–100.
10. Holden MJ, Madej RM, Minor P, Kalman LV.2011. Molecular diag-nostics: harmonization through reference materials, documentary
stan-dards and proficiency testing. Expert Rev Mol Diagn11:741–755.http:
//dx.doi.org/10.1586/erm.11.50.
11. Madej RM, Davis J, Holden MJ, Kwang S, Labourier E, Schneider GJ.
2010. International standards and reference materials for quantitative
mo-lecular infectious disease testing. J Mol Diagn12:133–143.http://dx.doi
.org/10.2353/jmoldx.2010.090067.
12. Fryer JF, Heath AB, Anderson R, Minor PD, Collaborative Study Group.2010. Collaborative study to evaluate the proposed 1st WHO International Standard for Human Cytomegalovirus (HCMV) for nucleic acid amplification (NAT)-based assays. WHO/BS/10/2138. World Health Organization, Geneva, Switzerland.
13. Hayden RT, Gu Z, Sam SS, Sun Y, Tang L, Pounds S, Caliendo AM.
2015. Comparative evaluation of three commercial quantitative cytomeg-alovirus standards by use of digital and real-time PCR. J Clin Microbiol
53:1500 –1505.http://dx.doi.org/10.1128/JCM.03375-14.
14. Caliendo AM, Shahbazian MD, Schaper C, Ingersoll J, Abdul-Ali D, Boonyaratanakornkit J, Pang XL, Fox J, Preiksaitis J, Schonbrunner ER.
2009. A commutable cytomegalovirus calibrator is required to improve
the agreement of viral load values between laboratories. Clin Chem55:
1701–1710.http://dx.doi.org/10.1373/clinchem.2009.124743.
15. Cattozzo G, Franzini C, d’Eril GV.2001. Myoglobin and creatine kinase isoenzyme MB mass assays: intermethod behaviour of patient sera and
commercially available control materials. Clin Chim Acta303:55– 60.
http://dx.doi.org/10.1016/S0009-8981(00)00370-3.
16. Vesper HW, Miller WG, Myers GL.2007. Reference materials and
com-mutability. Clin Biochem Rev28:139 –147.
17. Hayden RT, Shahbazian MD, Valsamakis A, Boonyaratanakornkit J, Cook L, Pang XL, Preiksaitis JK, Schonbrunner ER, Caliendo AM.2013. Multicenter evaluation of a commercial cytomegalovirus quantitative standard: effects of commutability on interlaboratory concordance. J Clin
Microbiol51:3811–3817.http://dx.doi.org/10.1128/JCM.02036-13.
18. International Organization for Standardization.2009. In vitro diagnos-tic medical devices. Measurement of quantities in samples of biological origin. Requirements for certified reference materials and the content of supporting documentation. ISO 15194:2009. International Organization for Standardization, Geneva, Switzerland.
19. Clinical and Laboratory Standards Institute.2010. Characterization and qualification of commutable reference materials for laboratory medicine; approved guideline. EP30-A. Clinical and Laboratory Standards Institute, Wayne, PA.
20. Cattozzo G, Franzini C.2013. Commutability: a peculiar property of calibration and control materials. Definition and evaluation. Clin Chem
Lab Med51:e167– e168.http://dx.doi.org/10.1515/cclm-2013-0065.
21. Wolff DJ, Heaney DL, Neuwald PD, Stellrecht KA, Press RD.2009. Multi-site PCR-based CMV viral load assessment-assays demonstrate lin-earity and precision, but lack numeric standardization: a report of the
Association for Molecular Pathology. J Mol Diagn11:87–92.http://dx.doi
.org/10.2353/jmoldx.2009.080097.
22. Boom R, Sol CJ, Schuurman T, Van Breda A, Weel JF, Beld M, Ten Berge IJ, Wertheim-Van Dillen PM, De Jong MD.2002. Human cyto-megalovirus DNA in plasma and serum specimens of renal transplant
recipients is highly fragmented. J Clin Microbiol40:4105– 4113.http://dx
.doi.org/10.1128/JCM.40.11.4105-4113.2002.
23. Bretaudiere J-P, Dumont G, Rej R, Bailly M.1981. Suitability of control materials: general principles and methods of investigation. Clin Chem
27:798 – 805.
