Multi Laboratory Evaluation of an Automated Microbial Detection/Identification System

(1)

MICROBIOLOGY, Dec. 1978, p. 0095-1137/78/0008-0657$02.00/0

Multi-Laboratory

Evaluation of

an

Automated

Microbial

Detection/Identification

System

P. B. SMITH,* T. L. GAVAN,2 H. D. ISENBERG,3A. SONNENWIRTH,4W. I. TAYLOR,5 J.A. WASHINGTON Il,6 ANDA. BALOWS'

CenterforDiseaseControl,PublicHealthService, Department of Health,Education,and Welfare,Atlanta,

Georgia 30333;' Cleveland ClinicFoundation, Cleveland,Ohio44106;2 LongIslandJewish-Hillside

MedicalCenter,NewHyde Park,NewYork11040;3JewishHospital ofSt.Louis,St.Louis,Missouri 6311i0; St.Mary ofNazarethHospital, Chicago,Illinois60622;5andMayoClinicandMayo Foundation,

Rochester,Minnesota 559016

Receivedfor publication2August 1978

An automated andcomputerizedsystem(Automicrobic System[AMS])forthe

detection of frequently encountered bacteria in clinical urine specimens was

tested inacollaborativestudyamongsix laboratories. Thesensitivity,specificity,

reliability, andreproducibility of the AMSweredetermined, and thesystemwas

compared with conventional detection and identification systems.Inthisstudy,

pure cultures and mixtures ofpurecultureswereusedtosimulateclinical urine specimens.Withpurecultures, the sensitivity of the AMS inidentifyingthe nine

groupsoforganismsmostcommonly found in urine averaged 92.8%.The

specific-ity averaged 99.4%, and the reliabilspecific-ity ofa positiveresult averaged 92.1%. The

latter value was strongly influenced by a relatively high occurrence of false

positive Escherichia coli results. The AMS wascapable ofdetecting growth of mostorganisms, including those which itwasnotdesignedto identify. However, itidentifiedsomeoftheseincorrectlyas commonurinarytractflora. Reproduci-bility of results, both within laboratories andamongdifferent laboratories, was

high. Fast-growing organisms, suchasE. coli and Klebsiella/Enterobacter

spe-cies,were detected often at cellpopulations well below the AMS enumeration threshold of 70,000/ml. In mixed culture studies, high levels of sensitivity and specificityweremaintained, but when Serratia specieswerepresentinmixtures

with otherorganisms, therewasoftenafalse positivereportof E.coli.Theoverall

performance of the AMS wasconsidered satisfactory under the testconditions used.

A novelsystem for the automated and

com-puterized detection,

enumeration, and identifi-cation of

organisms

inurine, the Automicrobic System (AMS), has recently been described by Aldridge et al. (1). This system represents a

drastic

departure

from conventional

methodol-ogy in microbiology, in that no prior isolation

and purification of

organisms

is required;

in-stead,theAMSisdesignedtodetectandidentify specific groups and/or species oforganisms in urine specimens representing either single or

mixedinfections.

During its developmental stages, the AMS wasevaluated with both seeded(simulated) and clinical specimens in a study reported by Son-nenwirth (2). He found about 90% agreement between AMSresults and conventional results withEscherichia coli, Klebsiella/Enterobacter species (the AMS does notdifferentiatethem),

Serratia species, Proteus species, Citrobacter freundii, group D enterococci, andyeasts

(pri-marily Candida and Torulopsis species), but

only about 75%agreementwithStaphylococcus

aureus andPseudomonasaeruginosa. Sonnen-wirth's study also indicated the need for im-provement of (i) reagents, (ii) sensitivity and

specificityfor someorganisms, and (iii) manual

methods used forcomparison andvalidationof results.

The present collaborative study was

under-taken after the manufacturer had instituted some of the recommended improvements and had modified the instruments. A two-stage,

multi-laboratory evaluation of this system was undertaken to (i) enlargeupon the data gener-ated by the twopreliminary reports (1, 2), (ii)

testurinarytractisolatesfromwidely dispersed

geographicareas,(iii) evaluatethe intra-labora-tory andinter-laboratory reproducibility of the system, and (iv) challenge the sensitivity and

specificity of theAMS. Seededspecimens were used in this evaluation. In another evaluation 657

on February 7, 2020 by guest

http://jcm.asm.org/

(2)

study, to be reported, clinical specimens from

various hospitals will beused.

MATERIALS AND METHODS Cultures. In the firstphaseofthis study, in which only pure cultures were used, bacterial and yeast cul-tures recently isolated in the various collaborating

laboratories werelyophiized, coded,and distributed

as unknowns to each laboratory by the Center for

DiseaseControl. The selectionofcultures used (see

Table 1) was notbased on the distributionnormally

encountered in clinical urinespecimens, but rather on

the idea of extensively testingeachidentification

ca-pabiity of the AMS system. Additional cultures,

which were not among those that the AMS was

de-signed to identify, were taken from the Center for

Disease Control stockculturecollections,coded, and

distributedasviablesubculturesonagar slants. In the

second phase of the study, in which mixed cultures

wereused, oneculture ofeach of thevarious groups or

species identifiable by the AMSwasselected, coded,

anddistributedas aviableagarslanttoeach

partici-pating laboratory. Ineach laboratory, these cultures

were mixedaccording to specific instructions. In all instances, thecodesforthe unknownswere notbroken untilresults had been obtained.

Equipment. The instrumentation and disposable

components of the AMS, including the Identi-Pak, have beenthoroughly describedbyAldridgeet al. (1). The system is based on theprinciple ofutilizing an array ofselectivemedia topermit significantgrowth

of only one organism, or a group ofclosely related

organisms,inspecific microchambers. Thesystem

per-mits desiredorganismstooutgrowcompetingones, if present, and automaticallycompares initialreadings

withsubsequentones.Aseparate enumeration system,

basedonmost-probable-numbertheory andutilizing

non-inhibitory media, givesapproximate totalcounts

ofall organismspresent. Both the identification and

enumeration systems have adjustable thresholds

which must be exceeded before a positive result is

recorded. In this study, the enumeration threshold

was set at7 x104colony-formingunits(CFU) perml, but lesser populations could be detected because of the separate enumeration and identification systems and because of the highly sensitive selective media employed.

TheIdenti-Paks used in thisstudywereessentially the same as those described by Aldridge et al. (1), except that some selective media formulations had

been changed slightly, following Sonnenwirth's

rec-ommendations.

The manufacturer provided each laboratory with

thecompletesystem,allnecessarymaterials,and

spe-cific instructions foruse.These instructions have also been described(1).The AMS modelemployedhada

capacityof120specimens, containeda tape deck for

recording time-history profiles, and automatically

printed all results after 13 h of incubation.Preliminary results could be obtainedatanytime,if desired.

Media and reagents. All bacteriological media and reagents required in the manual method (see

below)weresuppliedtothecollaboratinglaboratories

fromRegionalMediaLaboratories, Inc., Lenexa,Kans.

Insofar as possible, all laboratories used media and reagents from the same production batches. Media and reagentsused in the routinemethod (see below) were those normally used in the participating labora-toriesand came fromdifferent sources.

Methods. (i) Pureculture study. Each laboratory used three distinct identification schemes on each

unknown specimen: (i) the AMS procedure, as

di-rected by the manufacturer, (ii) a manual method

uponwhich allcollaborators agreed,and(iii) whatever

routine method each of the six laboratoriesnormally used. Theprocedures employedin theAMS and

man-ual methods are described further. In addition, the

Center forDiseaseControllaboratory alsoprovideda

separate "reference" identification for each culture employed.

For the AMS procedure, each coded lyophilized

bacterialculturewas rehydrated withsterile

Trypti-casesoy broth,streaked onto a 5% sheep blood agar

plate, and incubated for 18 to 24 h at 35 to 37°C.

Identical isolated colonies were picked into sterile,

poolednormalurine (suppliedlyophiizedtoall

collab-orators) to form a suspension of about 107 CFU/ml

(by comparison with a turbidity standard), and two

10-folddilutions were madeof this suspension, again

withsterile urine used asdiluent.This final suspension wascalled the "seeded specimen" and served as inoc-ulumfor the AMS and for otherprocedures. Every

fifthspecimen receivedtwoadditional 10-folddilutions

which were used to determine some of the lower

bacterial populations which the AMS could detect.

Culturesofyeasts werehandledsimilarlytothose of

bacteria,exceptthat theoriginal cellsuspensionsfrom

isolated colonieswereadjustedto aturbidityequal to

about 108 CFU/ml. Colony counts were made ofall

culturesfrom the seededspecimen by using eithera

model480 Artek-Fisher Automatic Bacterial Colony

Counterorconventionalcolonycountingmethods.All

Identi-Paks which (i) detected the presence of an

organismbutfailedtoidentifyitor(ii) failedtodetect

an organism when one wasknown to be present by

the manual method wereenteredaseptically through

the plastic membrane of the Identi-Pak by meansof

a sterileneedle andsyringe. Contents of thepositive

controlchamberwerewithdrawn and examinedbythe

manual method.

Themanual methodwasrecognizedtohave certain

deficiencies, but was considered to be a reasonably

adequate system for the identification oforganisms

commonly encountered in urine and to have a

per-formance levelcomparabletothat of the AMS. The

colony characteristics of each bacterial culturewere

observed on the blood agar plate inoculated in the

AMS procedure described above. Selected

well-iso-lated colonieswere then identified by theprocedure shown inFig. 1.It should be noted thatidentification

ofstreptococciwaspresumptive,and thatserological

confirmation was not done. Yeasts were identified from colonies on bloodagar plates by means of the

followingtests: colonial andmicroscopic morphology;

germ tubeformation;pellicleformation;capsule pro-duction (Indiainkstain); urease;dextrose, fermenta-tion; and assimilation of sucrose, maltose, raffinose,

galactose, trehalose, and melibiose. Colony counts

weremade on each culture from the seededspecimen

on February 7, 2020 by guest

http://jcm.asm.org/

(3)

Gram stain

(rods)

Oxidase

~1

Enterobacteriaceae Tests

i

Fluorescein, Pyocyanin,

42Cgrowth,

Cetrimide tests

FIG. 1. Manual methodidentificationprocedure. (a) Enterobacteriaceaetests:Triple sugariron,

Christen-sen ureaagar; lysineandornithinedecarboxylase; argininedihydrolase; indole;Simmoncitrate; motility

agar; deoxyribonuclease; phenylalanine deaminase; arabinose, inositol, and adonitolfermentation. (b) If

Acinetobacter,Flavobacterium,etc. aresuspected,useprocedureofindividual choice.

prepared in the AMS procedure. Both blood agar

plates and MacConkey agar plates were used for

countsofbacteria;only blood agarplateswereused for the yeasts.

(ii)Mixed culturestudy.One culturewasselected

from each of the groupsorspeciesoforganismslisted

inTable 1.Ail possible combinations oftwocultures

wereused(e.g.,cultures A andB, A andC,AandD,

Band C, B andD, etc.), and each combinationwas

tested in fourdifferent ratios of the two strains

in-volved. Forexample,P.aeruginosa and E. coliwere

tested in ratios of

107:10i,

106:104, 105:107,and 104:106,

respectively. The identification procedures via the

AMS and manual methodswerethe same as

previ-ously described. Yeasts were not employed in this

phase of thestudy.

Datahandling. Alldata,includingtapesand

hand-writtenreports fromtheAMS, manual, and routine

methods,wereforwardedtothe AMS computer

facil-ity of the McDonnell-Douglas Astronautics, East

Corp.Bothindividuallaboratoryandsummary results

werecomplied and returnedtothecollaborating

lab-oratories. The various authors also compiled some

data. In thisstudy,sensitivitywasdefinedasthe ratio

oftruepositive reportstothesumofthetruepositive

andfalsenegative reports,i.e.,theabiityofthe AMS

todetectaparticular organismwhen itwasshownto

be presentbyconventional methods. Specificitywas

definedasthe ratio ofthetruenegativereportstothe

sumof the truenegativeand thefalsepositive reports,

i.e.,theabiityof theAMSnot todetectanorganism when it was notdetected by conventional methods.

Reliabiitywasdefinedasthe ratioofthe truepositive

reports to the sum of the true positive and false

positive reports, i.e., the confidence that could be

placedinanAMSreportthat anorganismwaspresent.

RESULTS

Pureculturestudy.Thesensitivity,

specific-ity, andreliability of the AMS indetectingand

identifyingnine groups oforganisms commonly

found in urineareshown in Table1. The basis for this tabulation wasthe results obtained by

the manualmethod. Thesensitivityof theAMS ranged from 83.3% for Serratia to 98.9% for E.

coli, with a mean of 92.8% and a median of 95.1%. Both the positive control and enumera-tion chambers of the Identi-Pak were affected by testsinwhichvery small populations of or-ganisms (ca. 103/ml) at levels well below the thresholds set for detection and identification wereused.

Specificity values ranged from 98.1% for E. coli to 100% for group D streptococci, with a mean of 99.4% and a median of 99.7%. The positive control specificity is not a significant

figure because of the very small number of "true"negativeresults. The enumeration speci-ficityvalue (83.4%) indicates atendency ofthe AMS to report higher counts on about 17% of thespecimenswith bacterialpopulations evalu-atedmanuallyatlevelsbelow70,000/ml.

Thepercentreliability calculations reflectthe

degree ofconfidence to be placed in apositive AMSresult.Reliabilitypercentages ranged from a low of 71% for E. coli to 100% for group D

streptococci,with a mean of 92.1% and a median of95.4%. The low value for E.coli resulted from anappreciablenumber of Identi-Pak chambers

permitting other organisms tested, not E. coli, to attain detectable cell densities. This same

tendencywasnotedlater in the second phase of the study, in which mixed cultures were used. In addition to testing the AMS with pure cultures of the nine groups or species listed in

(cocci)

Catalase

~1

Bileesculin,

6.5% NaCI broth, CAMPtests

t

Coaugulase

VOL. 8,1978 659

on February 7, 2020 by guest

http://jcm.asm.org/

(4)

Table1,which thesystemwasdesignedto iden-tify, 224challengeswere madewithother

orga-nisms whichmay occurinurine (although

infre-quently), but which thesystemwasnotdesigned

toidentify. Thesewereusedtotestthelimits of differential capabiity ofthe AMS, and the

re-sults ofthese challengesare shown in Table 2.

With the exception ofcertain species of

Pseu-domonas, particularlyP.stutzeri, P.

maltophi-lia, P.diminuta,and P.

fluorescens,

mostof the strains tested grew in the Identi-Pak positive

controlchamber andwere not identified.

How-ever, 45 of the 167 strains which grew were

named,althoughincorrectly. Mostofthe named

iresconsistedofgroupBstreptococciwhich

e incorrectly identifiedas group D

strepto-cocci. Somewhat

disturbing, although

not

un-expected,

wasthemisidentification of10strains ofSalmonellaasE. coli.

Theagreement ofidentification

by

theAMS

was measured on both an

inter-laboratory

and

intra-laboratory

basis.

Inter-laboratory

agree-mentisindicated in Table3,whichshows agree-ment ofresults from all sixcollaborating

labo-ratories,

and offive of the six

laboratories,

with 152cultures. The manual method results

agreed

with thereferenceresults inall sixlaboratories

in89.5% of thetestsand infive of the six labo-ratories in 97.5% of thetests.Interestingly,AMS results

agreed slightly

better with thereference

results than with the manual method

results,

indicating

some faults in the latter. The AMS

TABLE 1. Summaryofpureculturestudyresults:AMSsensitivity, specificityandreliability, basedon

manual methodresults

No.of AMSresults Sensitiv- Specific-

Reliabil-Organism chal- ity ity ity

lenges T+ F- F+ T- (%) (%) (%)

P.aeruginosa 125 119 6 22 1,819 95.2 98.8 84.4

Proteusspecies 116 104 12 3 1,886 89.7 99.8 97.2

C.freundii 95 82 13 3 1,904 86.3 99.8 96.5

Serratia 90 75 15 7 1,902 83.3 99.6 91.5

E. coli 90 88 1 36 1,811 98.9 98.1 71.0

Klebsiella/Enterobacter 110 103 7 5 1,880 93.6 99.7 95.4

Yeasts 348 331 17 4 1,597 95.1 99.8 98.8

GroupDStreptococcus 140 137 3 0 162 97.9 100.0 100.0

S. aureus 144 137 7 9 728 95.1 98.8 93.8

Positive control 1,540 1,406 134 0 13 91.3 100.0 100.0

Enumeration 1,540 1,121 419 81 407 72.8 83.4 93.3

aSensitivity= [(T+) x 100]/[(T+) + (F-)];specificity = [(T-) x

100]/[(T-)

+ (F+)];

reliability

=[(T+)

x 100]/[(T+)+ (F+)].

bT+=AMS +,

manual

method+; F-=AMS-,

manual

method+; F+ =AMS+,

manual

method

-;

T-=AMS-,manualmethod -.

TABLE 2. AMS responsetochallengewithmiscellaneousorganisms

CorrectAMSresponse Incorrect AMS response

Organism No. Growthin Correct

tested positive Not iden- Incorrect genus, No

control

tified

genus wrong growth

species

Pseudomonads, other than P. 79 36 30 4 2a 43

aeruginosa

Aeromonas 12 12 il 1 0 0

GroupBstreptococci 29 26 4 0 22b 3

a-Streptococci,notgroup D 30 25 24 1 0 5

S. epidermidis 17 14 12 0 2c 3

Salmonella 12 12 2 lod 0 0

Flavobacterium 12 12 10 2 0 0

Acinetobacter 22 19 18 1 0 3

Providencia il il il 0 0 0

Total 224 167 122 19 26 57

aBothidentified as P.aeruginosa.

b

Ail

identifiedasenterococci. C BothidentifiedasS. aureus.

dAll

identified

as E.coli.

on February 7, 2020 by guest

http://jcm.asm.org/

(5)

andmanual methods produced thesameresults, however, in83% of the participating laboratories (five of six) in93.4% of the challenges made and

in all six laboratories in 81.6% of these chal-lenges. Intralaboratory agreement of results, with 136challengesin eachlaboratory,is shown

inTable4.Obviously,the data were inexcellent

agreement inthevariouslaboratories, and there

is nosignificantvariation in theseresultsamong thelaboratories.

Table 5 presents the sensitivity, specificity,

andreliability data for each laboratory and for

each groupoforganismstested in the pure

cul-turestudy. Thegreatestvariation insensitivity

occurred withC.freundii and Serratia cultures, and itappearsthatrelativelypoorperformance

in two laboratories reduced the overall values

obtained (see Table1).Specificitywasextremely high and uniforminall laboratories. Reliability

wasthe mostvariable oftheseparameters,

par-ticularly

withP. aeruginosa and E. coli. Two laboratories obtained low values forP.

aerugi-nosa,thusreducing the overallreliability from

over94%to84.4%(Table1).Similarly,two very low values were obtained for E. coli, with a

resultant low overall reliability (Table1). How-ever,reliabilityof apositiveE. coliresultwas a

common problem among all six laboratories, withnoneof themachieving higher than 83.3%. Therewas noindication fromthe sensitivity or

reliability data thatany onelaboratory

consist-ently performed less well than did other labo-ratories.

The three Identi-Pakswhich wereinoculated witheveryfifthspecimen were used todefmie,to someextent, the lower limits of detection and identification capabilities of the AMS. Some

indicationof this isshown inTable6,where the

AMS results are tabulated in relation to cell

population and

organism.

Certain

organisms,

such as P. aeruginosa, S. auréus, and yeast, were often notdetected at lowcell

concentra-tions

(103/ml), although others,

suchasE. coli andgroup D

streptococci,

werenot

only

detected butwerealso

identified

atsuch

populations.

At

cell populations generally considered to be

in-TABLE 3. Interlaboratory agreement of identification (152 challenges)

Agreement byall Agreement by 5 Comparison 6laboratories of 6 laboratories

No. % No. %

Manualvsrefer- 136 89.5 148 97.4

ence

AMS vs refer- 127 83.6 145 95.4

ence

AMSvsmanual 124 81.6 142 93.4

TABLE 4. Intralaboratory reproducibilityofAMS

results (136 challenges)

Labora-

_Labora-

AS AMS No. No.

vs

AMS

Positive false false tory refer- manal growth posi-

nega-ence

tive

1 124 124 132 5 4

2 127 127 131 2 3

3 124 123 131 6 5

4 128 127 128 5 2

5 128 129 134 0 6

6 122 121 125 1 3

dicative of

probable urinary

tract

infection,

i.e., between 104 and

105/ml,

the first three

orga-nisms mentioned abovewerestillnotaccurately identifiedin most tests. The variationobserved

with yeasts was partly due to the variety of

genera andspecies tested, some of which grew

faster than others. In 13 tests with yeast in

populations of 104 to 9 x 104/ml, no growth occurredinthepositive controlchamberofthe

Identi-Pak. In contrast, normally fast-growing

organisms,

such as E. coli, were occasionally detected and identified at populations of less than

103/ml

(data notshown).

Mixed culture

study.

Theunknowns distrib-uted for this

phase

of the studywere mixedin

eachcollaborating laboratory accordingto a

pro-tocol which resultedinall culturesbeing tested inpairs and eachpair of cultures being testedin

fourdifferent ratios oforganisms. The detection and identification

capabilities

ofthe AMS for these mixtures are shown in Table 7, which is

comparable to Table 1. The lowest values for both sensitivity and

specificity

were observed for E. coli; the highest values were observed withC.freundii.The meansensitivity for these

organisms,

evenin the presence ofother

orga-nisms, was96.3%, with a median of 98.3%.The

specificity

ranged

from 96.6% for E.colito100% forC.

freundii

and theKlebsiella/Enterobacter species, witha mean of 99.3% andamedian of 99.8%.

Another point of investigation in this mixed

culturestudywastheeffect, ifany, ofthe pres-enceofone

particular

organismonthe

ability

of the AMStodetect anotherorganismwhen pres-entsimultaneously; Tables8 to 10contain these

data. In Table 8, the false positive and false

negativeASM results are tabulated withrespect tobothlaboratory and

organism.

It showsthat E.coli was detected in 22 instances when it was notpresent (false positive),and in 12 instances it was not detected when it was present (false negative),asdeterminedbythe manual method. Thirteen of the22false positiveE. coliresults werefrom onelaboratory-a variation of about

VOL. 8,1978

on February 7, 2020 by guest

http://jcm.asm.org/

(6)

TABLE 5. Pureculturestudy: sensitivity, specificity, and reliability results by organism andlaboratorya

P. aeru- Pro- C. Serra- E. Klebsiella Y Group D S. Determination ginosa teus freundii tia

coli

Enterobacter east Strepto- aureus

coccus Sensitivity

Laboratory1 2 3 4 5 6 Specificity

Laboratory 1 2 3 4 5 6 Reliabiity

Laboratory1 2 3 4 5 6

82.6 94.7 83.3 87.5 100 100

100 89.4 93.7 87.5 100 100

95.2 90.0 84.2 88.2 100 94.7

100 88.2 100 84.6 94.1 89.4

100 80.9 83.3 66.6 94.1 100

94.7 95.0 75.0 90.0 100 83.3

99.3 100 99.7 99.7 97.0 100

99.6 100 100 99.3 99.0 100

97.2 100 100 99.7 98.1 100

97.0 99.6 100 99.6 98.0 100

99.3 99.3 100 99.6 98.7 98.7

100 100 99.3 99.6 97.3 99.6

90.4 100 90.9 93.3 54.5 100

95.0 100 100 87.5 83.3 100

68.9 100 100 93.7 71.4 100

70.3 93.7 100 91.6 76.1 100

92.3 89.4 100 92.3 80.0 80.0

100 100 85.7 90.0 63.6 95.2

96.4 100 96.9

93.3 100 88.8

93.3 94.1 84.8

96.6 100 100

96.5 96.2 100

94.3 100 100

99.6 100

100 100

100

99.5 100 99.6 99.6 100

98.2 100

100 100

98.3 100 98.2 100 98.0 100

100 96.9 98.4 99.2 99.1 100

94.1 72.7 93.3 96.1 96.7 100

aDefinitions forsensitivity,specificity, andreliabilityaregiven in thetextandthe footnotesforTable1.

TABLE 6. AMSresponse todifferent bacterialpopulations

Rangeofcellpopulations

103/ml 104/ml

Organism

No Positive

Positive

No Positive

Positive

growth control cation growth control cation

P.aeruginosa il 13 2 1 15 15

Proteusspecies 0 8 5 0 5 8

C.freundii 0 8 6 0 7 9

E.coli 0 0 il 0 0 7

Klebsiella/Enterobacter 0 8 6 0 2 14

Serratia 0 14 1 0 8 9

Yeast 15 8 2 13 15 21

GroupDStreptococcus 1 5 18 0 0 24

S.aureus il 15 0 3 il 12

TABLE 7. Summaryof mixedculturestudyresults:AMSsensitivity,specificity,andreliability, basedon

manualmethodresults

AMS

resuita

Sensitivity

Specificity

Reliability

Organism Challenges F- F+ T- ()(%%

T+ F- F+ T-(%()

P.aeruginosa 119 111 8 1 639 93.2 99.8 99.1

Proteusspecies 129 127 2 2 648 98.4 99.6 98.4

C.freundii 122 122 0 0 655 100.0 100.0 100.0

Serratia 132 125 7 9 646 94.6 98.6 93.2

E.coli 117 105 12 22 629 89.7 96.6 82.6

Klebsiella/Enterobacter 221 215 6 0 500 97.2 100.0 100.0

GroupDStreptococcus 115 114 1 1 649 99.1 99.8 99.1

S.aureus 119 117 2 2 643 98.3 99.6 98.3

Positive control 841 840 1 0 0 99.8 100.0

Enumeration 841 841 0 1 1 100.0 50.0 99.8

aSee

footnote

b,Table1.

on February 7, 2020 by guest

http://jcm.asm.org/

(7)

EVALUATION OF AUTOMICROBIC SYSTEM 663

seventimes the averageoccurrence of such

re-ports in the other laboratories. There was no

unusual distribution of false negativereports of

E.coliamongthe sixparticipating laboratories. Table-9presentsinmoredetail the conditions

under which the falsepositive AMS resultswere

obtained. It isapparentthateveryfalsereport

of E. coli occurred when Serratiawereactually

present.Similarly, six of the nine falsereportsof

Serratia occurred whenagroupDstreptococcus

waspresent.Theprecisereasonsfor these

asso-ciationsarenotknown, butone reasonmight be

that certain combinations oforganisms act

syn-ergistically to overcome the specific inhibitors

presentin the E. coliorSerratiagrowth

cham-bers of the Identi-Pak.

The

félse

negative reports observed in the mixed culture study are listed in Table 10. E. coliwasmissedmostoften(12 times), andmost

of these misses occurred when therewasgrowth

in the Klebsiella/Enterobacterchamber of the

Identi-Pak. C. freundii was most frequently

presentwhen the AMSfailedtodetect another

organism, and most ofthese associations were

with falsenegative Serratia and E. coli results. This couldperhaps be attributed toan

antago-nistic relationship between certain combinations oforganisms, but there isnoadditional evidence

tosupportthis. Even if thisistrue, areciprocal

relationship between C.freundii and Serratiaor

E. coli was not observed, because C. freundii

wasnotmissed during thisphase of the study.

DISCUSSION

Theconcept ofanautomated and

computer-ized system for enumeration, detection, and

identification oforganisms in urine, in the pres-enceof otherorganisms, isadramaticdeparture

from theconceptsinvolved in ingrained,

conven-tional pureculturetechniques. The AMS relies ontheprinciple of utilizing specific compounds

that selectively favor the growth and

metabo-TABLE 8. Mixed culturestudy: AMSfalsepositive andfaIsenegative results

Falsepositive in laboratory no.: Falsenegativeinlaboratory no.: Organism

1 2 3 4 5 6 Total 1 2 3 4 5 6 Total

P.aeruginosa 0 0 0 1 0 0 1 1 5 0 0 0 2 8

Proteusspecies 0 0 0 1 1 0 2 0 0 0 0 0 2 2

C.freundii O O O O O O O O O O O O O O

Serratia 0 4 1 1 2 1 9 1 1 1 1 2 1 7

E.coli 2 0 13 4 1 2 22 3 3 0 0 3 3 12

Klebsiella/Enterobacter O O O O O O 0 3 2 0 0 1 0 6

Group DStreptococcus 0 0 0 1 0 0 1 0 0 0 0 0 1 1

S.aureus 0 0 0 1 0 1 2 2 0 0 0 0 0 2

TABLE 9. Mixed culturestudy: AMSfalsepositive results

Pseudo- Pro- C. Serra- E. Kleb- Entero- Group D S.

Organims present mnonas teus freundii tia

coli

siella bacter co aureus

C.freundii/Serratia la

S.aureus/Serratia 8

Klebsiella/Serratia 1

Enterobacter/Serratia 3

Proteus/Serratia 2

P.aeruginosa/Serratia 3

Group D streptococci/Ser- 4

ratia

P. aeruginosa/Group D 4

streptococci

Klebsiella/group D strepto- 2

cocci

C. freundii/group D strep- 2

tococci

P.aeruginosa/Klebsiella 1 i

Proteus/Enterobacter 1 1

E.coli/S.aureus 1 i

GroupDstreptococci/S.au- 1

reus

aFigures indicate number ofoccurrencesof a

false

positive report when the indicated pairs of organisms were

actuallypresent.

VOL. 8,1978

on February 7, 2020 by guest

http://jcm.asm.org/

(8)

TABLE 10. Mixed culture study:AMSfalse negative results Organism missed Organiam

alsopresent Pseudo- Pro- C. Serra- E. Kleb- Entero- Group D S. monas teus freundii tia coli siella bacter co aureus

Pseudomonas 1 1

Proteus 1 2

C.freundii 1 5 4 2

Serratia 2 1

E.coli

Klebsiella 2 1 4

Enterobacter 1 1 4

Group D streptococci 1

S. aureus 1

lism ofspecificgroups orspecies ofbacteria, and

combines this withautomatedand computerized

instrumentation.

Thesystemisdesignedtobe usedfor screen-ing urine specimensofthetypes mostcommonly encountered, in which infection isusually

asso-ciated with bacterialcounts of105 or more

or-ganismsperml. Specimens collected by cathe-terization, suprapubic aspiration, or from pa-tients undergoing cystoscopy, in which low

countsmay beclinically significant, should not

be testedintheAMS, butshouldreceivespecial handling.

Publisheddevelopmental studiesonthis sys-tem

(2)

wereencouraginginthat morethan 92%

correlation

was obtained with conventional manual results on about 3,400simulated urine

specimens, and, with theexception ofP.

aerugi-nosa, more than 93% correlation was obtained with 1,500 clinical urine

specimens.

Problems

wereencountered, however, with both false

pos-itive andfalsenegativeresults andwith various

technological

difficulties.

Alaterstudyof theAMS,inwhich a prepro-totype instrumentand both simulatedand

clin-icalspecimenswereused(1),showedhigh levels of correlation

(>90%)

for most

organisms,

but

also revealed weaknesses in the identification capabilities oftheAMS forP.

aeruginosa,

group D

streptococci,

andS. aureusin clinical

speci-mens.

Reproducibility

experimentsinthis same

study, with twomachines,suggested thatmore

variation was encountered in conventional

man-ual methodsthan in the AMS

procedure.

The multi-laboratory collaborative study re-portedhere hasevaluatedthesensitivity, speci-ficity,andreproducibilityof the AMS. The tech-niques employed,

i.e.,

use of

lyophiized

pure

cultures, conventionalmediafromasingle

sup-plier, anda manual

method,

were designed to

minimize variation in

experimental

conditions.

Nevertheless, some variations did occur. Urine Identi-Paks from the same batch could not be

provided to allparticipants forthe duration of

the study; thus, several production batches of

Identi-Paks were employed. Furthermore,

dur-ing the studysomeof thespecificchamber media

used in the Identi-Paks werefound tobe defi-cient.They werelater modified, thus invalidat-ingresults previously obtained. Additionally, it wasrecognized that"humanerror" could not be

totally eliminated from the manual manipula-tionsinvolvedinsettingupthe varioustests, as

partially

showninTable8.Finally,asmentioned

previously, the manual method used in all six

laboratories was neither an exhaustive nor an

extremely abbreviated identification system; it

was acompromise between these extremes. The method was reasonably accurate and reflected thelevel of recognition possessed by the

instru-ment.Itstill permittedsomesubjective conclu-sions, andthus somevariations occurred.These variationsundoubtedly hadsomeadverse

influ-ence onthe

experimental

datapresented. Con-versely, theymore closely approximated

condi-tions which might be encountered ifthe AMS

wereused in a number ofwidely scattered lab-oratories.

We consider the results of this study to be

satisfactorydespite themanyvariablesinherent in it. With pure

cultures,

five of the nine

group/speciestested showed

sensitivity

ofover 95%, compared with the manual method; one

was93.6%, two were between 85 and 90%, and

only one (Serratia) was between 80 and 85%. Thespecificityof theAMSwasextremely high

forall ofthe

organisms tested, being

atleast 98% orbetter. The

reliabiity

of the AMS showeda

wider range than did either the

sensitivity

or specificity. The data

(Table

1) show that an

AMS result of E. coli wasrecorded 124

times,

yet E. coliwere

actually

present

only

88times. This was one of the most disturbing

problems

encountered,

because of both the

frequency

and the

significance

of E.coli in

urinary

tract infec-tions. The

reliabiity

value observed for E. coli

on February 7, 2020 by guest

http://jcm.asm.org/

(9)

was, however, not representative of values ob-tainedfor other organisms.Onlyoneothervalue, that for P. aeruginosa (84.4%),wasbelow 90%.

The limitsof theAMS fordetectionand

iden-tification wereseverelytestedbyincluding var-iousorganisms which are seen only rarely,but

still may occur, in urine specimens (Table 2).

The slow-growing pseudomonads were not de-tected for the most part, whereas most of the

otherorganisms testedweredetected butwere notidentified. The misidentifications ofgroup B

streptococciasgroup Dstreptococci and of Sal-monella as E. coliillustrate limitations in the capabilities of the AMS Identi-Pakto differen-tiateselectivelyamongrelatedorganisms. These limitations (misidentifications) could havesome serious therapeutic consequences, but they should not be overemphasized. Probably the

most commonly encountered problem here would be misidentification ofgroup B

strepto-cocci, becausetheyare not uncommoninurine from females. The reason for this error is not

clear, because

hydrolysis

of esculin andtolerance

tohighsaltconcentrationsarerequired for AMS identification of group D streptococci. Most group Bstreptococci will tolerate6.5%salt,but

they do not hydrolyze esculin. Possibly there was sufficient change in turbidity, because of growth of group B streptococci to register a

positive result,but thisdoesnotaccountforthe hydrolysis of esculin. Certainlyconditions exis-tentin a microchamber aredifferent fromthose inconventionaltesttubes,and this mayalso be

a factor. The significance of these data lies in

therecognition that such discrepanciesmay oc-cur, andso

microbiologists

andclinicians must

view themintheproperperspective.

Theproblems mentionedinthetwopreceding

paragraphs

regarding specificity, sensitivity, and reliabilityare influencednotonly by the selec-tive compounds employed inthe specific iden-tification chambers, but also by the threshold levels set for each chamber. These thresholds

are

adjustable,

andvaryfrom chamberto cham-ber, so that only the desired

organisms

be de-tected. It is appropriate to mention that this

investigationwas

designed

toaid infme-tuning

thesystem before its use in the clinicalsetting and that future adjustment of both

selective

compounds and identification chamber thresh-oldlevelsmay modify, if not alleviate, some of theproblemsencountered.

Agreement of identification of AMS results was considered tobe very good, both among the sixcollaboratinglaboratories and within each of theselaboratories. Table 3 shows that the man-ual and reference results agreed among all six laboratories in 136of 152 challenges and in five

of sixlaboratoriesin 148of152

challenges.

The

figuresused for the "AMSversusmanual" com-parison were 124/152

(for

all six

laboratories)

and 142/152 (forfiveofsixlaboratories). How-ever, a morerealisticmeasuremight betheuse ofthe "manualversusreference"

figures

asthe

denominators in these

calculations,

thus

chang-ing the percents to 91.2 and 95.9, respectively. Thesemoreaccurately reflect theagreementof theAMS and the manual methodresults. Some of thevariationinsensitivity notedamong indi-vidual laboratories (Table5) was probably the resultof batch-to-batch variationinurine Identi-Paks.

Thisstudyconfirmed, in six laboratories, the

report by Sonnenwirth (1) that the AMS was

capable ofdetectingandof sometimes identify-ingspecific organismsatlevels of

103

to9 x

103

CFU/ml.Thiswas

particularly

notedwith rela-tively

fast-growing

organisms. Only with slower growingorganisms, suchasStaphylococcus au-reus, yeasts, and

pseudomonads,

did

significant

numbers of tests result in no growth in the Identi-Pakatsuchlow cell concentrations.

Atconcentrations of104to 9 x

104 CFU/m1,

theAMSnearly alwaysdetected the presenceof

organisms

andusuallycorrectly identified them. Onlyyeast culturesstillwere notdetectedin a

significant

numberoftestswhenpresent at this

concentration. Thus, theabilityof the AMS to

recognizethepresence of

microorganisms

in

sim-ulated urine specimens at levels below those

considered indicative ofurinary tract infection

wasestablished.

Itshouldbe reemphasizedthat in this study the enumeration threshold was set at 70,000

CFU/mi.

It

might

beargued that the detection and identification ofsmall cell populations(<104 CFU/ml) bythe AMS isproblematical from a

clinical standpoint,because suchpopulationsare

usually regarded as

insignificant.

Thus, detec-tion of low populations by the AMS might be

falsely interpreted as representing significant

bacteriuria. This

possibility

is not considered

very likely, however, because theAMSwill re-port whether counts are above or below70,000

CFU/mi.

Anotherpointofconsiderationis that

patients withurinarytractinfection oftenhave

extremely

high-count bacteriuria

(107

to

109

CFU/ml). Inthe present study, populations of

cells in seeded urines rarely exceeded

107 CFU/ml,

so the question might arise as to whether the

specificity

of the AMS would re-main as high with

significantly

higher count urines. Thiscannotbe answeredonthe basis of results of thisstudy; the answer

will

depend on results ofclinical studies with the AMS.

We wish toemphasizethat the AMS

enumer-VOL. 8,1978

on February 7, 2020 by guest

http://jcm.asm.org/

(10)

ation and identificationreports on mixed speci-mens do not reflect the proportions in which various organisms are present, and thus give

equal weight to all ofthem, whereas only one mayactuallybepresent in significantnumbers.

Some information may begainedfrom

compar-ing thetimeperiods required for identifyingtwo

organisms,but this is nottotally reliablebecause of the varying growth rates of different orga-nisms in their respective identification cham-bers.

The mixed-culture phase of this study was

intended to explore the effect ofoneorganism

onthe AMS'sabilitytodetect anotherorganism, when bothwerepresentsimultaneously. Onlya

lowlevel of falsereports wereencountered:4.4% false positive and 4.5% false negative. These values compare to 4.4%and4.0%, respectively, in thepureculturestudy. The onlyassociations thatcouldbemaderelatingtothe occurrenceof false results were as follows: (i) a correlation

between false positive reports of E. coli when

Serratiawerealsopresent (Table 9); (ii)a ten-dency forSerratiatobe

reported

when group D

enterococci were present

(Table

9); and (iii) a

possible, but not clear-cut failure to identify SerratiawhenC.freundiiwasalsopresent

(Ta-ble 10). Thefirstoftheseassociationsseemsthe

clearest, but, because of the small amount of data available, confirmationby testing of fresh urinespecimensis

highly

desirable.

Thepresentstudywasdesignedtotestvarious capabilities of the AMS, by usingpure cultures

and simulated mixed cultures of

organisms

found in urinary tract infections. No attempt wasmade to simulate the distributionof

orga-nisms which occur most frequently in urine spec-imens ofpatients with urinary tract infections. Thus,thisstudy is weighted towardsorganisms

lesscommonlyencountered in the usualclinical

practice.Nevertheless, such a study is necessary toproperly definesomeofthelimitsof perform-anceofany device intended for clinical applica-tion.Stockcultures ofsomeEnterobacteriaceae

may notbeasmetabolicallyactive as fresh clin-ical isolates, but even the largest clinical labo-ratories do not encounter sufficient numbers of some organisms to permit a complete study of thistaxonomic family. Thus, this study has

at-tempted to provide a rigid test of the AMS,

undercontrolled conditions,and has shown that

the AMS appears to have potential for use in

clinical microbiology. Clinical studies now in progress shouldfocusonthispotentialand fur-ther define the AMS's capabilities and

limita-tionsinrelationtospecimens frompatients

sus-pectedofhavingurinarytractinfections. In

ad-dition, ourearlyexperiences withdefectivelots

of Identi-Paks show that the manufacturer is

obligated to maintain a high level of quality controlinthelarge-volume production of these products, and the user is obligated to perform his own quality control tests with known

cul-tures atappropriate intervals.

LITERATURE CITED

1. Aldridge,C.,P. W.Jones,S.Gibson,J.Lanham,M. Meyer, R. Vannest, and R. Charles.1977.Automated microbiologicaldetection/identification system. J. Clin. Microbiol. 6:406-413.

2. Sonnenwirth, A. C.1977.Preprototypeof anautomated microbial detectionandidentification system:a devel-opmental investigation. J. Clin.Microbiol. 6:400-405.