Accuracy and precision of the autobac system for rapid identification of Gram negative bacilli: a collaborative evaluation

JOURNAL OFCLINICAL MICROBIOLOGY,June 1982,p.1111-1119

0095-1137/82/061111-09$02.00/0 Vol.15,No.6

Accuracy and Precision of the Autobac System for Rapid

Identification

of

Gram-Negative

Bacilli: a

Collaborative

Evaluation

A. L.

BARRY,"*

T. L.

GAVAN,2

P. B.

SMITH,3

J.

M.

MATSEN,4

J. A.

MORELLO,5

AND B. H.

SIELAFF6t

UniversityofCalifornia-DavisMedical Center, Sacramento, California 958171;TheCleveland Clinic Foundation, Cleveland, Ohio

441062;

Centersfor Disease Control, Atlanta, Georgia303333; University of

UtahSchoolof Medicine, Salt Lake City, Utah 841324;Universityof Chicago Medical Center, Chicago, Illinois606375;andPfizer Inc., Groton, Connecticut063406

Received 23 November 1981/Accepted 28 January 1982

Gram-negative

bacilli were

identified within 3 to 6 h by determining

susceptibil-ity to 18 different antibacterial agents in the Autobac I system and by applying a

two-stage quadratic discriminant analysis to the susceptibility

patterns.

The

Autobac system was compared with standard reference methods for identifying

glucose nonfermenters

and glucose

fermenters. Intralaboratory and

interlabora-tory

precision

of the Autobac system was comparable to that of the reference

methods. Sensitivity (accuracy) and specificity of the two systems were also

comparable, although there were some differences with certain species. Autobac

responses were considered to be equivocal (needing additional tests) if the relative

probability

of an accurate

identification was <0.70. Only 5% of 2,889 strains

produced such equivocal results; a similar number of strains gave low probability

levels with the

reference methods. When the two systems disagreed, an

indepen-dent

reference

laboratory

arbitrated,

confirming 49% of the Autobac responses

and

36%

of the reference

identifications.

With

equivocal

responses excluded, the

overall

accuracy of the

Autobac

system was

95.3%

compared

with

95.9%

for the

reference method. The respective accuracy estimates would be 93.8% and 93.1%

if

all

first-choice identifications

were

evaluated.

In

general, previous

attempts to automate or

mechanize

identification

of

microorganisms

have been

accomplished by

simply adapting

traditional methods

for

use

in a mechanized test

system. In

1973, Friedman and

MacLowry

(5)

reported

a

computer-assisted

system

for

identifi-cation of bacteria based

on an

analysis

of the

pattern

of

susceptibility

to common

antimicrobi-al agents.

Sielaff

et

al.

(8) mechanized this

approach

by

using

susceptibility

test

results

ob-tained after 3 to 5 h with the Autobac I system

(Pfizer

Diagnostics, Groton, Conn. [now

the

property

of

General

Diagnostics,

Morris

Plains,

NJ

07950]).

By

applying

a

quadratic

discriminant

function

technique for

data

analysis,

they

ob-tained

a

97% correlation with conventional

iden-tification

procedures.

Matsen et al.

(personal

communication)

have

examined

the

possibility

of

utilizing

a

wide

variety

of

antibacterial agents,

other

than the common

chemotherapeutic

agents, for

the

purpose of bacterial

identifica-tion. After

extensive

screening

of

potentially

useful

agents, 18 different antibacterial agents

t Present address: Minnesota Mining and Manufacturing Co.,St.Paul,MN 55144.

were

selected because of their

discriminatory

capabilities. Sielaff

et

al.

(9) describe

a system

for

rapid (3-

to

6-h) identification

of

gram-nega-tive

bacilli, based on

analyzing the patterns of

susceptibility to

18

different

antibacterial agents.

This

system used the Autobac

I

system fitted

with a programmed computer which will

per-form

a

two-stage

quadratic

discriminant

analysis

of the

susceptibility

patterns.

The

present

report documents the

accuracy

and

precision

of the

proposed

Autobac system

for

rapid identification of

gram-negative

bacilli.

For this

evaluation,

standard

reference methods

were used to

confirm the identification of each

isolate. The

accuracy of

the

reference methods

selected for this

collaborative

study

was also

estimated

by

further

testing

selected

strains

at

the Centers for Disease

Control,

Atlanta,

Ga.

The

precision of each identification system

was

evaluated

by

testing

92

isolates in

triplicate

in

each of five

independent

laboratories.

MATERIALSANDMETHODS

Thestudyconsisted oftwophases. The firstphase wasdesignedtodocumentprecision (reproducibility) of the reference methodsand of the Autobacsystems. 1111

on February 7, 2020 by guest

http://jcm.asm.org/

1112 ET AL.

The second phase consisted oftesting fresh clinical isolates and stock culturestodocument thespecificity andsensitivity of the Autobac system, in comparison withreference methods.

Bacterial isolates.For the firstphaseof thisstudy,92 stock cultureswereselectedatthe Centers forDisease Control. The identification of each isolatewas recon-firmed, and each strainwassuspended in rabbit blood, dispensed into 15 different Durhamtubes, and frozen at -70°C. The cell suspensions were identified only with randomly selected code numbers. Each of the fiveparticipating laboratories received 276 coded cell suspensions representing three tubes of each stock culture. Personnel performing these tests were not awareof the initialidentificationsorof which suspen-sions wereduplications. The frozensuspensions sent toeach testing laboratory were thawed and subcul-turedjust before being identified with the reference methods and with the Autobac system. Subsequently, bothinterlaboratoryandintralaboratory reproducibili-tyof bothidentification systemswasevaluated. This culture collection included 3Escherichiacoli,2 Kleb-siella pneumoniae, 3 Klebsiella ozaenae, 3 Proteus mirabilis, 3 Proteus vulgaris,4Morganella morganii, 3Providenciastuartii, 2 Providencia rettgeri,4 Salmo-nellaenteritidis, 2 Arizonahinshawii, 3Shigella son-nei, 3Serratiamarcescens,4Enterobacter aerogenes, 3Enterobactercloacae,3Enterobacteragglomerans, 3 Hafnia alvei, 3 Citrobacterfreundii, 3 Citrobacter diversus, 3Edwardsiellatarda, 3 Yersinia enterocoli-tica, 3 Yersiniapseudotuberculosis, 3Acinetobacter calcoaceticus subsp. anitratus, 3 Aeromonas hydro-phila, 2 Moraxellalacunata,2Flavobacterium menin-gosepticum, 2Alcaligenes odorans, 3 Pseudomonas aeruginosa, 3 Pseudomonasmaltophilia,3 Pseudomo-nas cepacia, 2 Pseudomonas putidalfluorescens group,3Pseudomonasputrefaciens,and 3 Pseudomo-nasstutzeristrains. Thelaboratories failed to report all 1,380pairs ofidentifications: 1,253 reference identifi-cations and 1,240 Autobac identifiidentifi-cations were avail-able foranalysis. The unreportedidentifications were randomly distributed, not likely to introduce signifi-cantbias to the data.

The secondphase of this study involved tests with recent clinical isolates and a few stock cultures, in-cludedasrepresentatives of the less common species. Thefour clinicallaboratoriesparticipating in this study tested 2,889isolates with standard reference methods and withthe Autobac system. For each isolate, both testsystems wereinitiated at the same time, and the resultsofall testswere stored in a central computer facility. Subcultures of 653 strains were submitted to the Centers for Disease Control forarbitration: 327 strains were identified asbelonging to the same spe-cies with theAutobac and reference test systems and 326strains haddiscrepant identifications with the two independent systems. The former 327 strains were submitted forarbitration because the initial program gave discrepant results, but they were found to be in agreement when the modified program that is de-scribed in this report was applied to the data. The first experimental program is not described because it provedtobeinaccurate.

Ten quality control strains were tested by both identification systems at approximately biweekly in-tervals during the second phase of this study. The control strains included E. coli, K.pneumoniae, P.

vulgaris,

M.

morganii,

S.marcescens, C.diversus,P. aeruginosa, P. putrefaciens, M. lacunata, and A. calcoaceticus

subsp.

antitratus. These strains were

selectedto

provide

atleastonepositiveresponseand

one

negative

response toeach of the individualtests included in bothidentification systems.

Autobacidentificationsystem. Thetestsystem evalu-ated in this study has been described in detail

by

Sielaffetal. (9). Briefly,eachisolatewassubcultured toblood agar and MacConkey agarplates. The next day, the following information was recorded: (i) growthonMacConkey agar, (ii)lactosefermentation

on

MacConkey

agar,(iii)presence ofprecipitatedbile aroundcoloniesonMacConkeyagar,(iv)spot oxidase test

(6) results, (v)

spot indoletest(10) results,and(vi) swarming growthonblood agar. These six pieces of informationwereentered into thecomputerized identi-fication system before the Autobac cuvettes were

read. To inoculate a cuvette, one or more isolated coloniesweresuspendedinphosphate-bufferedsaline, and the suspension was then adjusted to a standard turbidity (ca. 107 colony-forming units per ml),

by

usingtheAutobacphotometer. For each 13-chamber cuvette, 2 ml ofinoculumsuspensionwasaddedto18 ml ofEugonbroth(BBL Microbiology Systems, Cock-eysville, Md.).In the present study,two13-chamber cuvettes were inoculated, and 24 different elution disksweretested.Afterourdatawereanalyzed, the 18 elution disks described by Sielaff et al. (9) were

selected, and the identifications presented here are basedonlyontheresults oftestswiththose 18disks. A19-chambercuvettewillsoonbe available fortesting the 18elutiondisks, providingonecontrolchamber.

Onceinoculated,thecuvettes wereallowedto incu-bate at 35°C in an Autobac incubator-shaker. The cuvettes were all read after 3 h of incubation. If sufficientgrowthwas notinitiated in the control cham-ber,thecuvettes werereincubated and readat

hourly

intervals for thenext3h. Ifsufficient growth hadnot been obtained after 6 h ofincubation, the test was aborted, and the strain was retested the next day. Repeated failure togrow within 6 h was a very rare occurrenceand when this didhappen, the strainwas removed from our study. As soon as satisfactory growthwasobtained in the controlchamber,turbidity in each test chamber wasmeasured, and the results were automatically entered into the computeralong with thepreliminarytestdata notedearlier. The pro-gramthen providedafirst- andsecond-choice identifi-cation along with the relative probability (R.P.) that expresses theconfidence with which one may accept each reported identification. A lowR.P. indicates that additional tests are needed toconfirm the identifica-tion; it mightrepresent a species that is not in the data base.

Referencemethods.Conventional tubed media were usedtoobtainareferenceidentification. These media (ScottLaboratories, Fiskeville, R.I.) were essentially identical to those used at the Centers for Disease Control for identification ofEnterobacteriaceae (3) or byG. L.Gilardi (personalcommunications) for identi-fication of theglucosenonfermenters. The media used toidentify glucosefermenters andnonfermenters are described more completely below. Most tests were read after 24 h and, ifnegative, they were read again after 48 h ofincubation.

o-Nitrophenyl-P-D-galacto-pyranoside tests were readafter 20min and 1, 2, 4, and J.CLIN. MICROBIOL.

on February 7, 2020 by guest

http://jcm.asm.org/

AUTOBAC ID SYSTEM 1113 48 h or until a positive reaction was seen.

Voges-Proskauer, phenylalanine deaminase, and indole tests were performed after 48 h of incubation only, since externalreagents had to be added. The results of all tests were entered into the computer, which was programmed with percentage figures obtained from the appropriate sources (primarily from the Centers for DiseaseControl,EntericBacteriology Section, and from G. L. Gilardi). The latter computer program was developed to identify only the genera or species in-cluded in the Autobac data base, using rather tradi-tional methods. When the two systems disagreed, many more discriminatory tests were performed, as needed, for final identification of each isolate. This arbitration work was performed exclusively at the Centers for Disease Control by P. B. Smith, D. L. Rhoden, and A. 0.Esaias.Arbitration was performed with 653 isolates for which discrepant identifications were initially obtained by Autobac and reference methods. The Autobac program wasrevised at the end ofthis study, and when the revised program was used, 327of the 653 Autobac identifications were in agree-mentwith the initial referenceidentifications. Only the revised program is evaluated in this report.

Statistical analysis. Forthe purpose of this report, the term "accuracy"indicates agreement with a refer-ence identification (reference methods orarbitration tests when available). "Precision" is used inter-changeablywith"reproducibility"toindicate repeat-ability ofa particular test response. Interlaboratory and intralaboratory reproducibilities are each ex-pressed as "reproducibility index," rather than per-centageof responses in agreement with modesorwith theexpected responses. Thisallowed comparisonof allpossible pairsof data and expresses theproportion of datapairsthatwerein agreement(4,7).Some data did notpermitthe selection ofamodalresponsetobe used asanindex of agreement; thussomedata could not be analyzed in the traditional manner. Further-more,thereproducibilityindex couldeasily accommo-datemissing data,i.e., ifonlytwoof three responses were reported, one pair of data was available for analysis. "Sensitivity" ofan identification system is defined as the percentage of strains within a given speciesthatwereaccurately identified.

"Specificity,"

on the otherhand, examines the number of timesa

given species identification wasreported bythe test systemand expresses the percentage of times those reportswerein agreement with the reference identifi-cation(accurate).

C)C)Cx

. _C.

-0

C)

n 5

C) .

--o. °.n

o_.F _

,-C U)C

C

0X

to) -I

0X

-C)oC

C) 0.

- U) RESULTS

Precision of referencetests.Tenqualitycontrol strainswereeach tested 29 and 41 differenttimes against the appropriate battery of reference tests,thus providinganopportunity toestimate theprecisionof each of the individualtest

proce-dures. False-positiveorfalse-negativereactions

wererecordedwith about 2to3% ofmost tests (Tables 1 and2). Overall precision variedfrom 93.5% forM.morganiito99.2%for A. calcoace-ticus. Additional estimates of precision were

obtained from replicate tests that were per-formed with the larger collection of isolates

_,

II

w+ A¢,+ " w

0%

-.

00 s.

t-4

00 -+ + + ( + + + +

-+ +- + I l- +

I + + I -l

+ + + w+ +I.-&4+ 1-- t-i+ + + + I -+

+ + ) +t

+ ,_"+

'IO A I-10% 4 t-'O w

o- + w -"

+ WJI

I + +..

+- + +

+

" I +-I

o-I +I.+ Iw-+Wv+o,

+w I I+ + +

w$ 00

.. .1 PO

-en~ >A ao w xo

VOL. 15, 1982

,0n

C)^ C2 ze z r' C) c c <i _ c ° o), c (1c

ti

r C) . 0 C) 0

C)

C_ co C) 00 CD-U) -9 C) _.

C)

_.

0z

₀

0O

C) 0

C)

3-9 _. C) 3 -9

a'

0-_. c D A r) D B B D 3 rv i2 CY. w 2 D. 3 D ?I 10 A 2 D 10 a. (c l< --I, o O I=F

C),.

00 o x1

a"-'

0.

C)

-CD, 2?. a' C) C) I CD,

A' o

10 w ~-+ +s

on February 7, 2020 by guest

http://jcm.asm.org/

TABLE 2. Precisionof reference methods used for identification ofglucose-nonfermenting gram-negative bacilli'

Most commonresultband no. of errors recorded with: Overall Control strain

(no.of timestested) Mo- Oxi- Argi- H2S 10% 1% NO3 Growth precision

tility dase nine (KIA) Lactose Glucose togas at 42°C (%)

P.aeruginosa(40) + + + - - + + +

1 2 98.8

P.putrefaciens (33) + + - + - - +

4 1 3 1 3 94.5

A.calcoaceticus (32) - - - - + + - +

1 1 99.2

M.lacunata(14)d - + - - -

-2 98.2

Precision%C 95.8 100 99.2 100 100 96.6 96.6 95.0 97.9

aReplicatetestswithfour control strains tested in four

laboratories;

seeTable 3 for

description

oftests.

bPositive(+)ornegative (-), "errors"areeither

false-positive

or

false-negative

results.

cPrecisionexpressedaspercentageoftestswith

expected

results.

dThecontrol strain ofM.

lacunata

astested

repeatedly

in

only

twolaboratories.

included in the first

phase of this

study.

Table

3

describes the

individual

tests

performed and lists

the

precision estimated

on the basis of 15

repli-cate tests

with each of 92 strains. In that series

of tests, false-positive or false-negative results

with

one or more

reference

tests

resulted in

a

misidentification

(13.8%)

of the

1,380

identifica-tions

recorded

or

in

an

"unable

to

identify"

response

(3%);

83.2% of

the

1,380

identifications were

considered

correct. The relative

probabili-ty

of

an accurate

result

(R.P. value)

was

low

(<0.70) with 28% of incorrect

responses but

with only

3% of the

correct

identifications.

When low R.P.

values

are

obtained,

erroneous

reference

testsmay

be

suspected, and additional

testing is needed before

a

final

identification

can

be made with confidence.

Intralaboratory and

interlaboratory

reproduc-ibility. The

precision of Autobac identifications

was

established and compared with

that

of

the

reference methods by

examining

results of the

first phase

of this study. Both interlaboratory

and

intralaboratory precisions

were expressed as

reproducibility

indexes rather than

percent-ages

of

responses in agreement with the modal response (as done in Tables 1 through 3). With this approach, all possible pairs of responses are

compared

(4,

7).

For example, when three re-sponses are

being

compared, there will be three

pairs, i.e., first

and second, first and third, and second and

third. If

the three identifications were E. coli, E. coli, and E. cloacae, the

repro-ducibility index

would be 0.33 because only one

of

the

three pairs (33.3%)

was in agreement. In most

studies

to date, such reports would have been

considered 66.7%

reproducible (two of three correct when compared with the mode).

The

reproducibility

ratio is statistically more

valid

because

all

randomly occurring situations are

considered.

It

permits inclusion

of all data,

evenwhen some data are

missing

or when the

results

are

such that

a

modal

response cannot

be

established

for

estimating precision. However,

the

reproducibility

index

provides figures

which

appear

much

lower than

percent agreement

fig-ures

that

are

customarily

quoted. Because the

Autobac and

reference results

were

analyzed in

the

same way,

the reader

is urged

to compare

reproducibility

indexes

recorded for the

two

methods and

not to

be

concerned with their

absolute

magnitudes. Each reproducibility index

was

calculated by

dividing the number of pairs in

agreement

by the total number of pairs being

compared (Table 4).

Intralaboratory precision ranged from

0.67 to

0.84

for the Autobac

system

compared

with

0.58

and

0.88

for the reference method.

When data

from all laboratories

were

combined,

intralabor-atory

reproducibility

indexes

were

nearly

com-parable. The

chi-square

test

demonstrated

no

significant differences between the

reproducibil-ities

of the

two

methods

(X2

=

1.62,

P >

0.20).

Interlaboratory reproducibility indexes

were

0.82

for both methods. Combined

ratios,

calcu-lated

by comparing

all

possible pairs

of respons-es, were 0.74

and

0.76

for the Autobac and

reference

methods,

respectively.

Because over

8,800

pairs of

responses were

compared in

this

analysis,

the

difference between

the two

indexes

was

statistically significant

(P < 0.05). Howev-er, the

difference is probably

too

small

to

be of

clinical

importance.

Accuracy of identifications. When the

identifi-cation obtained

with the Autobac system

dis-agreed with

that obtained with the reference

method, independent

arbitration was used to

determine

whether either of the methods was correct.

Since

entirely different

approaches are used in

the

two systems to achieve an

identifica-tion, both

systems are not likely to be in error at 1114 BARRY ET AL.

on February 7, 2020 by guest

http://jcm.asm.org/

AUTOBAC ID SYSTEM 1115 TABLE 3. Description and estimated precision of conventional tests utilized to obtain reference

identifications

Test Precision

(%)M

Test Precision(%)O

Glucosenonfermenters

Ornithine

decarboxylase, Moeller

H2S, Kligler

iron agar(KIA)slants 98.1 basalmedium .97.5

Oxidasespottest, 1%

tetramethyl-p-

DNase,toluidine blue medium 97.2 phenylenediamine ... 97.5

ONPG'

test for

P-D-galactosidase

Arginine dihydrolase, Moeller basal

activity

.96.4

broth

.96.2

Sucrosefermentation, peptone broth

Nitrate reduction togas, indole withAndrade indicator .96.1 nitrate broth .95.6

Xylose fermentation,

peptone broth

10%o Lactose oxidation, purple agar withAndrade indicator.96.1 slants ... 95.4 Adonitol fermentation, peptone

Motility, hanging droppreparation. 93.7 broth withAndrade indicator .... 95.9

Glucose, HughandLeifson

OFb

Arabinose fermentation, peptone

basalmedium... 88.5 broth withAndrade indicator .... 95.4

Growthat42°C, tryptic soy agar Lactosefermentation, peptone broth slants... ... 74.9 withAndrade indicator... 94.6

Flagellum stains, selected strains Voges-Proskauer (VP) test, MR-VPe

only

...-c broth and Barrittreagent... 94.3

Phenylalanine deaminase (PD),

Glucose fermenters

phenylalanine

agarslants .94.2

Oxidase spot test, 1%tetramethyl-p- Salicinfermentation, peptone broth

phenylenediamine

... 99.4 with Andradeindicator... 92.5

Lysinedecarboxylase, Moellerbasal Motility test medium with

triphenyl-medium... 98.3 tetrazolium chloride... 89.8

Indoleproduction, peptonebroth Malonateutilization, malonate agar

and Kovacreagent... 98.1 slants... 86.9

H2S,triple sugariron(TSI) agar Ureaseactivity, Christensen urea

slants... 97.7 agarslants... 86.3

a Precision of each conventional testexpressedas thepercentageof correct reactions noted whentriplicate tests were performed with 92 strains (66 glucose fermenters and 26 nonfermenters) in each offive separate laboratories.Fifteentest results were obtained with eachstrain,and the mostcommonresponsewasacceptedas thecorrect result forestimating precision.

bOF, Oxidation-fermentation.

c, Insufficient datato estimateprecision.

dONPG,

o-Nitrophenyl-p-D-galactopyranoside

test.

'MR-VP, Methylred-VogesProskauer.

the

same

time,

andif they are, they are unlikely to

provide the

same erroneous

identification.

For that

reason, we

assumed that the reference

method

wascorrect

when it

confirmed the

Auto-bac

response,

thus

allowing

us to

evaluate the

accuracy

of the reference method without

arbi-tration of

an

excessive

number

of strains.

The data in

Table

5

confirm the

validity

of

this

assumption, i.e.,

when

both

systems

agreed,

323

of

327

identifications

were

found

to

be

accurate.

When the

two tests

disagreed,

arbitration

con-firmed 49o of the Autobac

responses

and

36% of the reference identifications.

With both

testsystems,the computer

printout

provided

an R.P.

value which indicates the

confidence that

may

be

attached to the

first-choice

identification. The

response maybe

con-sidered

equivocal if

a

low

R.P. value

is

listed;

i.e.,

additional

tests are

needed before

the final

identification

can be

made with confidence.

When

estimating

the accuracy

of

atest system,

equivocal

responses

should

be excluded since

they

can

be

neither

accurate nor

inaccurate.

Withother commercial systems, about 5 to 10% of the strains tested have been found to be equivocal (1, 2). The practical utility of a test system is diminished if more than 10% of re-sponsesindicateaneedfor additional confirma-tory tests.

The data in Table 6 were accumulated to determine how the accuracy of both systems was affected by excluding strains with various R.P.values. With bothtestsystems, elimination ofidentificationswithlowR.P. values excluded more erroneousresponses than correct identifi-cations, thus increasing the overall accuracy. Both systemswere

approximately

97%accurate ifallresponseswith R.P. <0.95 wereexcluded, but 11 and19% of the strains would be consid-eredequivocal. Withthe Autobac system, only 8.5% ofthe strains had R.P. values of <0.80, and 96.2% of the

remaining

identifications were considered accurate. However, a significant number

(about

25%)

of Enterobacterspp. would beexcludedbecause the R.P.was <0.80. Fewer Enterobacter spp. would be excluded if only VOL. 15,1982

on February 7, 2020 by guest

http://jcm.asm.org/

TABLE 4.

Intralaboratory

and

interlaboratory reproducibility'

ofAutobac identification

compared

with that ofidentificationsobtained with reference methods

Pairs inagreement/totalno.ofpairs ReproducibilityindeXb Comparison

Autobac Reference Autobac Reference

Intralaboratory

Lab1 211/257 220/255 0.82 0.86

Lab 2 211/251 213/257 0.84 0.83

Lab 3 162/211 125/214 0.77 0.58

Lab 4 211/258 219/263 0.82 0.83

Lab 5 177/263 232/264 0.67 0.88

All laboratories 972/1240 1,009/1,253 0.78 0.81

Interlaboratoryc

675/826 699/857 0.82 0.82

Combination'

6,516/8,809 6,733/8,823 0.74 0.76

aFive laboratories tested 92 strains, each in

triplicate,

yielding

1,380 responses for each method; fewer responseswereavailableforanalysisbecause somelaboratoriesfailedtoreport alltest results.

bNumber ofpairs inagreement divided

by

number ofpossiblepairs.

Comparing

the most

frequent

identification reported by each of five laboratories; when one

laboratory

reportedthree differentresponses,one wasrandomly selected forcomparison.

dOverall

combination of

intralaboratory

and

interlaboratory precision;

eachidentificationwascomparedwith all otheridentifications. For eachstrain, 105

possible

pairsofresponseswerecompared.

R.P. values of <0.70 were considered to be

equivocal. Thiswould excludeonly5%of allof

the responses and would result in an overall

accuracy of 95.3%. Consequently, we

recom-mend that first-choice Autobac responses be

accepted only if the R.P. value is .-0.70. When wedidthat,theoverallaccuracyof theAutobac

system wascomparabletothat of the reference

tests.

Sensitivityandspecificity.Thesensitivityofan

identification system may be defined as the

percentage of strains within a given species or

group ofspecies that was accurately identified

(agreedwith the referenceidentification).Onthe

otherhand, specificitydesignatestheconfidence

that may begiventoaparticular species

identifi-cationreportedbythe systemunderevaluation.

For example, 94 strains of C. freundii were

tested with the reference tests, but only 55%

were accuratelyidentified (sensitivity),whereas

94% of the C.freundii identifications obtained

TABLE 5. Resultofarbitrationtests comparedwith

first-choice identificationsinitially obtained with reference and Autobacsystemsin theparticipating

laboratories

Initial Autobac andreference

Arbitrationtests identification:

confirmed

Disagreed Agreed

Referencetests 118a _b

Autobactests

159-Neither 49 4

Both -323

Total 326 327

aDatapresentedas number of strains in each

cate-gory.

b-,Nonepossible.

with

the referencetestswereaccurate

(specific-ity).

Further,

the reference

methods

were

97%

sensitive

but

only

88%

specific

for A. calcoaceti-cus: allbut 3 of the 101 A.

calcoaceticus

strains were

correctly identified,

but other

species

were also

being

misidentified

asA.

calcoaceticus.

The

calculated

sensitivity

and

specificity

for

each of

the

microbial

species

included

in the second

phase

of this

study

are

presented

in Table 7. All

first-choice identifications

obtained wth the ref-erence method may be contrasted with those obtained with the

Autobac

system.

In

addition,

the

percentages of strains

within each

species

that would be

excluded because

the R.P. values

TABLE 6. Estimatedaccuracyof reference and Autobacidentifications: effect of

excluding equivocal

identifications,

basedonthe R.P. that each response wasaccurate

%Excludedateach %Accurateaafter R.P. R.P. level excludingtests excluded

Referenceb Autobac Referenceb Autobac

Nonec -_c - 93.1 93.8

<0.60 4.1 3.0 95.4 95.2

<0.70 5.2 5.0 95.9 95.3

<0.75 6.3 6.5 96.2 95.6

<0.80 7.1 8.5 96.4 96.2

<0.85 7.8 11.1 96.6 96.5

<0.90 9.2 15.0 97.0 96.9

<0.95 11.2 19.3 97.4 97.4

aAccuracywasjudgedasagreementwith the refer-ence identifications or arbitration tests (when

avail-able).

"Eighteen

strains that could notbeidentifiedwith the initial referencetestswereexcludedwhen

calculat-ing

accuracy.

cEvaluation of allfirst-choice

identification,

regard-less ofthe R.P. valuegiven;-,none.

J. CLIN. MICROBIOL.

on February 7, 2020 by guest

http://jcm.asm.org/

AUTOBAC ID SYSTEM 1117

TABLE 7. Sensitivity andspecificity of the reference methods and of Autobac identifications (before and after excluding equivocal results with R.P. below 0.70)

% Sensitivity (accuracy) of: %Specificityc of:

No.of Autobac A

Referenceidentification strains responses Reference Autobac responses Reference Autobac responses tested excluded testb All Excluding testb All Excluding R.P. <0.70 methods tests R.P. < 0.70 methods tests R.P. < 0.70

A. calcoaceticus 101 0.0 97.0 97.0 97.0 88.2 97.0 98.0

Aeromonas sp. 38 0.0 97.4 100.0 100.0 92.5 100.0 100.0

Alcaligenes sp. 17 17.6 64.7 88.2 92.9 78.6 83.3 86.7

C.diversus 56 3.6 98.2 96.4 98.1 91.7 96.4 98.1

C.freundii 94 21.1 55.3 80.0 91.9 94.5 72.4 75.6

E. tarda 7 0.0 85.7 100.0 100.0 100.0 58.3 70.0

E. cloacae 229 14.8 89.4 83.0 83.9 96.7 90.9 94.7

E. aerogenes 173 6.4 98.8 91.3 93.8 96.6 95.2 96.2

E. agglomerans 33 15.2 87.5 75.8 75.0 43.8 64.1 70.0

E. coli 414 4.3 97.6 95.4 97.5 97.6 97.8 99.2

Flavobacterium sp. 13 0.0 38.5 100.0 100.0 83.3 86.7 86.7

H.alvei 39 5.1 87.2 92.3 94.6 87.2 85.7 89.7

K.pneumoniae 312 4.2 97.8 96.5 97.6 98.0 94.1 96.7

Klebsiella (other species) 9 22.2 77.8 77.8 100.0 63.6 70.0 70.0

Moraxella sp. 4 0.0 100.0 100.0 100.0 25.0 66.7 66.7

M.morganii 100 3.0 96.9 97.0 97.9 90.5 94.2 94.9

P. mirabilis 304 1.6 98.3 99.3 100.0 100.0 98.7 99.3

P. vulgaris 64 1.6 96.8 89.1 88.9 90.9 100.0 100.0

ProvidencialP. rettgeri 91 4.4 63.3 96.7 98.8 100.0 89.8 94.4

P. cepacia 9 0.0 77.8 77.8 77.8 100.0 70.0 77.8

P. maltophilia 57 3.5 71.9 94.7 98.2 95.3 93.1 93.1

P.putidalfluorescens 30 6.7 96.6 93.3 92.9 84.8 96.6 96.3

P. stutzeri 6 0.0 100.0 83.3 83.3 66.7 100.0 100.0

P. aeruginosa 308 1.0 98.7 97.4 98.7 99.3 98.7 99.0

Pseudomonas(otherspecies) 10 0.0 50.0 66.7 60.0 41.7 50.0 75.0

SalmonellalArizona 94 4.3 95.7 92.6 96.6 92.8 93.5 94.5

Serratia sp. 185 1.6 98.9 97.3 97.8 95.8 100.0 100.0

Shigella sp. 71 4.2 97.2 93.0 94.1 72.6 95.7 95.5

Y.enterocolitica 18 16.7 94.1 88.9 93.3 72.7 69.6 77.8

Y.pseudotuberculosis 3 33.3 33.3 33.3 50.0 33.3 100.0 100.0

Total 2,889 5.0 93.1 93.8 95.3

aAutobac responses were evaluated with and without excluding equivocal results, R.P. < 0.70. The

percentage of strains within eachspecieswithequivocalresponses islisted.

bExcluding 18 strains that couldnotbeidentified by the initial reference tests, includingallotherfirst-choice

identifications.

cSpecificityindicates theconfidence thatcanbeplaceduponagivenspecies identification, i.e.,thenumberof correctresults dividedbythe numberof times eachspecies wasreported.

for Autobac

responses were

<0.70

are noted in

Table

7.

By

excluding

such

equivocal

identifica-tions,

the

sensitivity

and

specificity

of

the

Auto-bac

systemwere

somewhat

improved.

In

addition

to the clinical isolates

reported

here,

tests were

performed

with 10

isolates

belonging

to

species

thatare notincluded in the Autobac program

(6

Achromobacter

xylosoxi-dans,

3

Pasteurella

multocida,

and 1

Bordetella

bronchiseptica

isolate).

TheAchromobactersp.

isolates

were all misidentified as Pseudomonas sp., as was the Bordetella sp. The Pasteurella sp. isolatesweremisidentifiedas Moraxella sp., E.

agglomerans,

and Edwardsiella sp. All of thosemisidentificationswere

reported

with R.P. values

of

.0.70

and thus would not be consid-ered

equivocal

identifications.

DISCUSSION

The Autobac

system for

rapid

identification of

gram-negative

bacilli

represents

a

unique

ap-proach

to

bacterial taxonomy based

onthe

pat-terns

of

susceptibility

to

various antibacterial

agents.

Continuing

efforts to find

non-chemo-therapeutic

agents

that can be used in such a

system

should

improve

the

reliability

of

this

approach.

Resistant variants

might

appear in

some

environments,

leading

to

atypical

suscepti-bility

patterns

whch

might

leadto

misidentifica-tions. Because there

should

be no

selective

pressure for variants that become

resistant

to the

non-chemotherapeutic

agents,

efforts

should be made to

replace

the

therapeutic

agents

that

are

currently

included in the

system.

VOL. 15, 1982

on February 7, 2020 by guest

http://jcm.asm.org/

The taxonomic groups that were used to de-velop the Autobac

identification

system were limited to 30

species

orgroupsof species that are likely to be found in clinical

specimens.

No attempt was made to

identify species

of Aetro-monas, Alcaligenes, Fl/a

tobacteriuln,

Morax-ella, Serri-atia, orSlhigella. In most clinical

situa-tions, identifications

tothe

generic

level will be

sufficient. Serological

confirmation

of Sailinioniel-la spp. and

Shigella

spp. would be anecessary supplement to this system.

Sailmonella

sp. and

Ar-izona

sp. are not

distinguished,

but many hold that Arizona sp. should be

placed

in the genus Sallmonella. P.

rettugeri

is not

distinguished

from P.

stiuartii

(urea

positive

and urea

negative).

Failure to make that distinction

currently

pre-sents no

important

clinical

problem.

K.

pnieia-mnoniae

and K. oxytoca are not

separated,

but that could be

accomplished by

simply referring

to the spot indole test

(10).

K.

o.xvtoca being

indole

positive.

Furthermore, the Autobac com-puterprogram

does

notyet

identify

three recent-ly recognized species (Enterobacter sakaz-akii, E.

gergo'ii,

andCitrobacter

amnalonaticis).

Our

limited

experience

with

testing

a few species that are not

included

in the current program underscores the

possibility

that other

misidenti-fications

can occur with

uncommonly

encoun-tered species.

In spite of these

relatively minor limitations.

the Autobac systemcanrapidly

identify

the vast majority of gram-negative bacilli found in clini-cal material. The

sensitivity.

specificity.

and

precision

of the Autobac systemare comparable tothe

corresponding

characteristics of the refer-encetestsystem. However, theAutobac system is much morerapid (3to 6hversus 48

h).

and the

mechanization

and computer-assisted

interpre-tation minimize

the

technologist

time required to obtain reliable results.

In most studies of this nature, a new test system is normally compared directly with a

reference

system, and any

discrepancies

are assumed to represent errors on the part of the new system. The present study provided a unique opportunity to evaluate the reference method, as well as the Autobac system. When the two

disagreed,

the Autobac system was in error about half of the time, and the reference methods were in error a little more than half of the time (Table 5). Although every

effort

was madetostandardize and to control the reference methods,

false-positive

and false-negative reac-tions were obtained with about 2 to

3%

ofthe

individual

tests. Occasionally, the erroneous testresults were important enough to result in a

misidentification

and, subsequently, a disagree-ment with the Autobac system. For the same reason,

false-positive

or false-negative results with the Autobac system could result in

errone-ousidentifications which

disagree

with the refer-ence methods. For that reason, it is easy to understand

why

correlation between two

inde-pendent

identification systems

rarely

exceeds 90 to

95%, depending

upon theprecision of thetwo systems. An even greater number of

discrepan-cies

might

be expected if conventional tests rather than standard reference methods had been used for evaluating the Autobac system. For example, conventional tests may represent standard tubed media read afterovernight

incu-bation,

accepting some loss of precision and accuracy for the sake of convenience. Such procedures are not appropriate reference meth-ods forevaluating new identification systems.

By excluding

testswithalowprobability ofan accurate response, a significant proportion of misidentifications were eliminated. The overall accuracy of both systems was as great as 97% when all responses with R.P. levels of <0.95 were excluded. But that would have required supplementary tests with 11 or 19% of the iso-latesincluded inthe secondphase of this

study.

We concluded that Autobac identifications with R.P. values of .0.70 could be accepted and those of <0.70 should be confirmed with supple-mentarytests. Only 5% of the isolates included in this study would require such confirmation, and the overall accuracy of both the Autobac and the reference methods was 95 to 96%. Sensitivity and specificity of the two methods varied somewhat with different species, but, in general, the Autobac system was as sensitive and specific asthe reference method.

In summary, the Autobac system

provides

a unique approach to the rapid identification of gram-negative bacilli. The overall results indi-catethat the system is just assensitive, specific, and precise as the standard reference methods. Themechanized system, with computer-assisted

interpretation,

requires a

minimum

amount of technologist time and provides reliable results within 3 to6h.

ACKNOWLEDGMENTS

We express oursinceregratitude to the following microbiol-ogists whoperformed thestudies described in this report: R. Aaron. C. William Bacon. Robert E. Badal. Ann 0. Esaials, B. B. Gardner. Michael H. Graves. C. Knapp, Dwane L.

Rhoden,and M.Telenson. We alsoacknowledgethe invalu-able assistance provided by microbiologists in the Enteric Bacteriology Section of the Centers for Disease Control and by G. L.Gilardi,whoprovided certainunpublisheddata that wereneeded forestablishingthedata base used for

interpreti-tionof thestandardreference tests.

LITERATURE CITED

1. Barry,A.L., and R. E. Badal. 1979.Rapid identification ofEnterobocterioaoea with the Micro-ID system versus API 20E and conventional media. J. Clin. Microbiol. 10:293-298.

2. Barry, A. L., R. E. Badal, and L. J. Effinger. 1979.

on February 7, 2020 by guest

http://jcm.asm.org/

AUTOBAC ID SYSTEM 1119

fication of Enterobocteriaceae in frozen microdilution

traysprepared by Micro-Media Systems. J.Clin. Micro-biol. 10:492-496.

3. Ewing, W. H.,andB. R. Davis. 1975.Media andtestsfor

differentiationofEnterobocterioceae. U.S. Department of

Health, Education and Welfare. National Communicable

DiseaseCenter. Atlanta. Ga.

4. Fleiss, J. L. 1971. Measuring nominal scale agreement

among manyraters. Psychol. Bull. 76:378-382. 5. Friedman, R., and J. MacLowry. 1973.Computer

identifi-cation of bacteriaonthe basis of their antibiotic

suscepti-bilitypatterns. AppI. Microbiol. 26:314-317.

6. Kovacs,N. 1956.IdentificationofPseilonionaos

ae(lruinl-osaby the oxidase reaction. Nature(London)178:703.

7. Light, R.J. 1971. Measures ofresponse agreement for qualitative data: some generalizations and alternatives.

Psychol. Bull.76:365-377.

8. Sielaff, B. H., E. A. Johnson, and J. M. Matsen. 1976.

Computer-assistedbacterial identification utilizing

antimi-crobial susceptibility profiles generatedby Autobac 1. J. Clin. Microbiol. 3:105-109.

9. Sielaff, B. H., J. M. Matsen, and J. E. McKie. Novel approachtobacterial identification thatusesthe Autobac

system.J. Clin. Microbiol. 15:1103-1110.

10. Vracko, R., and J. C. Sherris. 1963. Indole-spot test in

bacteriology. Am.J.Clin.Pathol. 39:429-432.

VOL. 15 1982

on February 7, 2020 by guest

http://jcm.asm.org/