The ngram chief complaint classifier: A novel method of automatically creating chief complaint classifiers based on international classification of diseases groupings

The

n

gram chief complaint classifier: A novel method of automatically creating

chief complaint classifiers based on international classification of diseases groupings

Philip Brown

_{, Sylvia Halász}

_{, Colin Goodall}

_{, Dennis G. Cochrane}

_{, Peter Milano}

_{, John R. Allegra}

a

AT&T Labs – Research, Florham Park, NJ, USA

b

Emergency Medical Associates of New Jersey, Livingston, NJ, USA

c

Morristown Memorial Hospital Residency in Emergency Medicine, Morristown, NJ, USA

a r t i c l e

i n f o

Article history: Received 13 May 2009 Available online 27 August 2009 Keywords: Disease outbreaks Epidemiology Public health Surveillance

a b s t r a c t

Introduction:Thengram classifier is created by using text fragments to measure associations between chief complaints (CC) and a syndromic grouping of ICD-9-CM codes.Objectives: For gastrointestinal (GI) syndrome to determine: (1)ngram CC classifier sensitivity/specificity. (2) Daily volumes forngram CC and ICD-9-CM classifiers.Methods:Design: Retrospective cohort. Setting: 19 Emergency Departments. Participants: Consecutive visits (1/1/2000–12/31/2005). Protocol: (1) Used an existing ICD-9-CM filter for ‘‘lower GI” to create thengram CC classifier from a training set and then measured sensitivity/specificity in a test set using an ICD-9-CM classifier as criterion. (2) Compare daily volumes based on ICD-9-CM with that predicted by thengram classifier.Results:For a specificity of 0.96, sensitivity was 0.70. The daily vol-ume correlation forngram vs. ICD-9-CM wasR= 0.92.Conclusion:Thengram CC classifier performed sim-ilarly to manually developed CC classifiers and has advantages of rapid automated creation and updating, and may be used independent of language or dialect.

Ó2009 Elsevier Inc. All rights reserved.

1. Introduction

Early detection of disease outbreaks, whether from bioterrorism or from natural or accidental causes, is important for effective treatment, containment, and minimizing morbidity and mortality [1]. Once detected, the size, spread and tempo of outbreaks can also be monitored[2].

Although many types of routinely collected clinical, administra-tive, pharmacy and laboratory data have been explored as possible data sets for this monitoring and detection[3], emergency depart-ment (ED) databases containing patient chief complaint (CC) and/ or International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM)[4]codes have proved especially promis-ing[5–9]. These data sets are valuable resources for epidemiolo-gists and public health officials as they are widely available and can be monitored to alert public health officials if the current inci-dence of disease exceeds a threshold established from historical data. A CC is usually a one-line statement about why the patient came to the emergency department. It is frequently written in the patient’s own words, and is generally entered by a triage nurse or registration clerk.

The ICD-9-CM codes are based on the ED physicians’ diagnoses. Frequently, these CCs and ICD-9-CM codes are grouped into

syn-dromes for the purposes of surveillance. ‘‘Synsyn-dromes” are chosen to identify disease processes. Typical categories of syndromes in-clude: respiratory, gastrointestinal, rash, fever, and neurological. This is done because the definitive diagnosis may rely on delayed confirmatory laboratory studies and may not be available at the time of the disease outbreak[10]. An upward trend in the volume of visits within a syndrome may give an early warning of an out-break, and may also be used to identify an outbreak when specific diagnostic testing is not available. The assignment of an ED visit to a syndrome is based on CC and ICD-9-CM code, both of which are available from the ED record. These are not affected by laboratory culture results, possible inpatient evaluation, discharge informa-tion or follow-up data.

This syndrome classification is currently done through several methods. In some, such as the Tally Sheet System used by the Santa Clara County Public Health Department[11], data are manually collected. In others the data collection is automated. These include the ICD-9-CM based Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE)[12]which downloads ICD-9-CM code data from U.S. Department of Defense health care facilities around the world and performs daily analyses, and the natural-language CC based New York City Department of Health and Mental Hygiene system which codes complaints into syndromes on the basis of matching keywords[13]. Although auto-mated systems have the advantage of being rapidly deployed to new data sets, natural-language systems[3]are appropriate only for the language and dialect in which they were developed. Also, 1532-0464/$ - see front matterÓ2009 Elsevier Inc. All rights reserved.

doi:10.1016/j.jbi.2009.08.015

*Corresponding author. Address: 7 Valley View Dr., Montville, NJ 07045, USA. Fax: +1 973 290 7209.

E-mail address:[email protected](J.R. Allegra).

Contents lists available atScienceDirect

Journal of Biomedical Informatics

their development is often a long and labor-intensive process [3,14,15].

It would be useful if rapidly available CC data sets could also be used across the world, in ‘‘event-based” or ‘‘drop-in” surveillance [11]. Current systems of this nature are of limited utility as existing manual and natural-language based syndrome classifiers cannot be used across languages or dialects.

It has been shown that whenever possible, syndromic assign-ments should be made based on a combination of CC and ICD-9-CM codes [1]. However, while CCs are often available in ‘‘real-time”, ICD-9-CM codes may not be available in real-time. In some systems such as ESSENCE, the ICD-9-CM code is assigned at the time of the patient visit, but in many systems the ICD-9-CM codes are used for billing, and are not assigned until days after the visit. Thus, there is motivation to develop syndromic assignment meth-ods that use CCs exclusively as an input, for the times that CCs are the only data available in a timely fashion.

It has been suggested that integrated surveillance approaches be developed which incorporate different data sources as they become available[1]. In surveillance systems where both the CC and the ICD-9-CM code are collected for each visit but the ICD-ICD-9-CM code is col-lected later, the ICD-9-CM code may nevertheless be useful. For example, the ICD-9-CM code groupings could be used by the com-puter algorithm generating the CC classifier. Using that method, the CC classifier might be created and updated more rapidly and with less labor than a manual method based on the CC alone[3].

ICD-9-CM also has an advantage over CC of being independent of the spoken language, dialect or local idiosyncrasies of CC usage. Existing natural-language[3,15–17]or manual[18–20]CC classifi-ers are language dependent and require considerable labor to de-velop and maintain. CC classifiers dede-veloped automatically from the ICD-9-CM code could be more easily created in multiple spo-ken languages and dialects. This would be particularly useful for rapid development and deployment of a CC classifier in a ‘‘drop-in” surveillance situation.

Previously we have described in an abstract the use of an auto-mated method for producing a CC classifier based on ICD-9-CM code groupings, called the ‘‘nGram Classifier”[34]. This classifier is trained on a set of ED visits for which both the ICD-9-CM diag-nosis code and CC are available by measuring the associations of text fragments within the CC with a syndromic group of ICD-9-CM codes (e.g. 4 characters for a ‘‘4-gram”: ‘‘iarh”, ‘‘diah” or ‘‘rrea”, parts of common misspellings for diarrhea). The choice of the auto-mated method was based on its speed and effectiveness. The

ngram classifier creation is a heuristic approach most closely lated to stepwise regression variable selection. The method re-quires three passes through the training set (one to generate absolute predictive values and two ‘‘pruning” passes to reduce the overallngram count and to create a decision tree of relative predictive values). For the training sets used in this study (one year of ED visits—roughly half a million visits), anngram classifier can usually be built in less than half an hour. Applying the classifier to a test set of equivalent size typically takes less than 10 min.

Our objectives in this study were twofold: (1) to characterize the sensitivity and specificity of anngram CC classifier for a gastrointes-tinal (GI) syndrome through designating each CC as ‘‘in” or ‘‘out” of the syndrome by setting various thresholds to the probability that a givenngram is associated with the ICD-9-CM classifier for the GI syndrome, and (2) to determine how closely the daily volume esti-mates of anngram CC classifier matched the ICD-9-CM classifier for a GI syndrome. We make the daily volume comparison omitting the ICD-9-CM code for undifferentiated abdominal pain to demon-strate the seasonal peaks of gastroenteritis which would otherwise be obscured by the large number of visits for undifferentiated abdominal pain. This code is assigned when the source or cause of the abdominal pain is unknown at the time of ED disposition.

2. Methods

The method developed for the assignment of patient CC to syn-dromes is based on an ‘‘ngram” text processing program adapted from business research technology (AT&T Labs). The method ap-plies the ICD-9-CM classifier to a training set of ED visits for which both the CC and ICD-9-CM code are known. A computerized meth-od is used to automatically generate a collection of CC substrings with associated probabilities, and then generate a CC classifier pro-gram. The method includes specialized selection techniques and model pruning to automatically create a compact and efficient classifier.

For each objective, we used a computerized database of consec-utive visits seen by ED physicians in 19 emergency departments in New Jersey and New York from 1/1/2000 to 12/31/2005 (approxi-mately 3.5 million visits).

For the first objective, to characterize the sensitivity and speci-ficity of anngram CC classifier for a GI syndrome, we used an exist-ing ESSENCE syndromic groupexist-ing of CM codes as our ICD-9-CM classifier[12]. Prevalence in the test population was 13.7%.

The ICD-9-CM classifier was applied to a training set (approximately half a million visits for the year 2004) to create the ngram based CC classifier. This generated classifier, limited to 4-grams that appeared in at least 100 CCs, contained 83

ngrams (or CC substrings) and their associated probabilities. We found that classifiers based on 4-grams performed the best. We then used thengram CC and ICD-9-CM classifiers to catego-rize individual visits from the test set (approximately half a million visits for the year 2005). We were able to characterize each visit as ‘‘in” or ‘‘out” of the syndrome by setting different cutoffs, or thresholds, to the probability of the ngram being associated with the ICD-9-CM classifier. We then determined the sensitivity and specificity for each threshold using the ICD-9-CM classifier as the criterion standard, creating a recei-ver-operating characteristics curve for the method based on these different thresholds.

For the second objective, we used a modified version of the ES-SENCE ICD-9-CM classifier for ‘‘lower GI”. To highlight seasonal variations, we removed the ICD-9-CM code corresponding to ‘‘undifferentiated abdominal pain” from the ‘‘lower GI” syndrome and extended the test set over a five-year period. Prevalence in the test population was 3.4%.

The ICD-9-CM classifier was applied to a training set (approx-imately half a million visits for the year 2000) to create a new

ngram based CC classifier. We chose the year 2000 as the train-ing set for the second objective, because it was the first year of the dataset, and we wished to examine seasonal variations in subsequent years. This generated classifier contained 17 CC sub-strings and their associated probabilities. We then used this

ngram CC and ICD-9-CM classifier to categorize visits from the test set (approximately three million visits for the years 2001 through 2005). We obtained the daily predicted volumes for

ngram CC classifier by calculating the probability that each visit belonged to the syndromic group. We then added up all the probabilities for all visits for the day to determine the predicted volume. The daily volumes for the ICD-9-CM classifier were ob-tained simply by categorizing each visit as ‘‘in” or ‘‘out” of the syndrome. We generated a time series graph of the daily visit volume estimates for each of the two methods. We then ana-lyzed the agreement between the ngram CC classifier and the ICD-9-CM classifier using a correlation coefficient. We compared daily visit volumes for these classifiers rather than visit-by-visit agreement, as this is what would be more useful to epidemiolo-gists looking for disease outbreaks.

3. Results

For the first objective, thengram approach was used to make ‘‘in” or ‘‘out” determinations using various thresholds on the prob-ability that thengram is associated with the ‘‘lower GI” ICD-9-CM classifier with undifferentiated abdominal pain included. The re-ceiver-operator curve based on these threshold variations is given inFig. 1. The sensitivity and specificity of thengram CC classifier at four arbitrarily chosen threshold probabilities of high specificity are given inTable 1. Threshold probabilities were chosen for this table that had high specificities so that the CC classifier would not generate a large number of false positives.

For the second objective,Fig. 2shows a time series graph of dai-ly visit volumes by thengram and ICD-9-CM classifiers for ‘‘lower GI” syndrome without undifferentiated abdominal pain. Visual inspection of this time series graph demonstrates that thengram CC classifier without undifferentiated abdominal pain identified the same seasonal gastroenteritis peaks found by the ICD-9-CM classifier. The scatter plot ofngram vs. ICD-9-CM classifiers for dai-ly visit volumes for ‘‘lower GI” syndrome without undifferentiated abdominal pain is shown in Fig. 3with a correlation coefficient,

R= 0.92, indicating a strong agreement between these two methods.

4. Discussion

When ‘‘cut-offs” were set at various thresholds (Table 1), we found that for 96% specificity, thengram method had a sensitivity of 70%. These results are similar to those of the natural-language CC classifier Complaint Coder (CoCo), which when tested on a set of 527,228 ED visit CCs, had a sensitivity of 69.0% and a specificity of 95.6% for a GI syndrome[21]. On the other hand, another natu-ral-language respiratory syndrome CC classifier applied to a pedi-atric population had a sensitivity of 47% and specificity of 97%[22]. Visual inspection ofFig. 2demonstrates that thengram CC clas-sifier identified the same seasonal gastroenteritis peaks found by the ICD-9-CM classifier. The correlation coefficient for daily vol-umes forngram classifier vs. ICD-9-CM code classifier shows a high degree of correlation.

Thengram method differs from most other classifiers in that it is able to assign a probability that each visit falls within a syn-drome. One then obtains an estimate of daily volumes by summing up these probabilities. CoCo, a Bayesian CC classifier, is also

prob-abilistic[16], however, we found no other studies that used a prob-abilistic approach to compare expected daily volumes to a criterion standard with a correlation coefficient.

Natural-language CC classifying systems have had several suc-cesses, albeit within their language barrier. Examples are the appli-cation of the Bayesian CC natural-language classifier to the real-time ‘‘drop-in” surveillance of free-text CC from 10 EDs and 20 walk-in clinics in Salt Lake City during the high profile 2002 Winter Olympics [16], and similar deployments in Pennsylvania, Utah, Atlantic City and Ohio[17]. Notably, the surveillance was able to detect a cluster of carbon monoxide exposures within 4 h of the presentation of the first case to an ED[17]. When outbreaks are recognized early, they can be mitigated more effectively.

CCs are difficult to use, as the use of free-text in syndromic sur-veillance requires managing the substantial word variation that re-sults from use of synonyms, abbreviations, acronyms, truncations, concatenations, misspellings, and typographic errors. For example, in a study of 10,000 ED visits there were 852 abdominal pain visits, with 440 different free-text CCs. Of the 440 free-text CCs, 397 were used only once [23]. There are even hospital idiosyncrasies in recording chief complaints[25,33]. Failure to detect these varia-tions results in missed cases, and traditional methods for capturing these variations require ongoing, labor-intensive maintenance[24] and are restricted to the language and dialect in which they are written. Natural-language processing systems which clean ED text, such as the Emergency Medical Text Processor (EMT-P)[3], have been developed to reduce the variation in ED text for natural-lan-guage system CC classifiers. Thengram system has the advantage of being indifferent to language and dialect, as well as format (i.e. free-text CC or computer pick list CC triage systems). To our knowledge, there exists no CC classifier other thanngram which can be deployed across languages, dialects, and different health-care providers with their specific usages in recording chief

com-0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1 - Specificity

Sensitivity

CC classifier by Beitel et al. [22]

Fig. 1.The receiver-operating characteristics curve forngram CC classifier generated by varying the probability threshold.

Table 1

The sensitivity and specificity of thengram CC classifier at four threshold probabilities of thengrams being associated with the ICD classifier.

Threshold (%) Sensitivity Specificity

40 0.70 0.96

45 0.62 0.97

50 0.61 0.98

plaint text. We have also successfully applied thengram method to ICD-10 codes[26].

We used ICD-9-CM codes as the criterion standard as others have done in the past[21]. One could argue that we should use a Kappa statistic to compare thengram system to ICD-9-CM codes instead of sensitivity and specificity as the ICD-9-CM code system may not be the best criterion standard. We chose to compare the two systems as we did since the potential users may be interested in the false negative and false positive rates which are reflected in the sensitivity and specificity. The Kappa statistic gives only the overall agreement above chance.

In addition to sensitivity and specificity, a criterion that the Center for Disease Control (CDC) stresses in their guidelines for evaluating public health surveillance systems is acceptability, which they define as ‘‘reflected by the willingness of participants and stakeholders to contribute to the data collection, analysis and use”[27]. Many syndromic surveillance systems currently in

place require extra effort from physicians, nurses and other per-sonnel. The Tally Sheet System requires triage nurses to record on paper tally sheets whether a patient has any of the syndromes of interest[18]. The Rapid Syndrome Validation Project (RSVP) re-quires clinicians to enter into a touch screen clinical and demo-graphic data on patients that fall into six syndromes [19]. The Light-weight Epidemiology Advanced Detection and Emergency Response System (LEADERS) system requires administrative staff to complete additional Web-based forms [20]. The ngram uses existing clinical data exclusively and does not require additional effort by medical personnel.

Authors have pointed out that while CC data are timely and widely available, they can be limited by field size and are intended to capture only the primary reason for a visit[28,29]. Triage notes, on the other hand, may give more complete descriptions of the pa-tient’s symptoms and can include multiple complaints[30]. One study showed that that inclusion of triage notes in the syndrome

0 20 40 60 80 100 120 140 160 1/1/2001 5/1/2001 9/1/2001 1/1/2002 5/1/2002 9/1/2002 1/1/2003 5/1/2003 9/1/2003 1/1/2004 5/1/2004 9/1/2004 1/1/2005 5/1/2005 9/1/2005

Daily Counts

Fig. 2.Time series of daily counts forngram CC classifier and ICD-9-CM classifier.

0 20 40 60 80 100 120 0 20 40 60 80 100 120 140 160

Daily Counts by ICD-9-CM

Daily Counts by

n

gram

R = 0.92

queries improved the sensitivity of the syndromic surveillance queries[30]. Thengram method of classification may also be ex-panded to classify triage notes to syndromes.

5. Limitations

We used one year of CC and ICD-9-CM data as the training set for thengram system as many diseases, especially infectious dis-eases, show marked seasonal variations [31]. We chose to use ‘‘lower GI” as a syndrome, which is generally not used. This was chosen because we could easily remove undifferentiated abdomi-nal pain, for which visits are so prevalent that they tend to obscure other diagnoses.

Another limitation of our study was that ESSENCE ICD-9-CM codes were used as the criterion standard for these data. A better cri-terion standard would have been a manual chart review. However, a chart review of one year of visits would require a prohibitive amount of labor. On the other hand, ESSENCE has been found to perform well for the GI syndrome with a high sensitivity (89.0%) and specificity (96.0%)[32]. In addition, large training sets such as those used in this study, must be available to apply this method. Finding large comput-erized training sets may be a difficult task.

This study was performed for one region of the United States. Further work needs to be carried out in other regions. In addition, we present here one syndrome in one language. Further work needs to be done with other syndromes, such as the respiratory syndrome and in other languages and alphabets.

6. Conclusions

Thengram CC classifier performed similarly to the ICD-9-CM classifier. In addition, with the appropriate threshold, thengram CC classifier can be adapted to dichotomize visits as ‘‘in” or ‘‘out” of a syndrome with similar sensitivity and specificity as a manually developed CC classifier. Manually developed classifiers require more labor-intensive development and updating. Thengram meth-od applied in this study has promise in that it may offer a comple-mentary method to using manual and natural-language processing techniques to create CC classifiers. It has the advantages that it al-lows the rapid automated creation and updating of CC classifiers based on ICD-9-CM groupings and may be independent of the spo-ken language or dialect.

Acknowledgment

Jonathan Rothman of Emergency Medical Research Associates Research Foundation for providing these data.

References

[1] Reis BY, Mandl KD. Syndromic surveillance: the effects of syndrome grouping on model accuracy and outbreak detection. Ann Emerg Med 2004;44(3):235–41.

[2] Henning KJ. What is syndromic surveillance? MMWR Morb Mortal Wkly Rep 2004;53(Suppl):5–11.

[3] Travers DA, Haas SW. Evaluation of emergency medical text processor, a system for cleaning chief complaint text data. Acad Emerg Med 2004;11(11):1170–6.

[4] American Medical Association. ICD-9-CM 2002: international classification of diseases, 9th revision. Chicago, IL: American Medical Association; 2002. [5] Beitel AJ, Olson KL, Reis BY, Mandl KD. Use of emergency department chief

complaint and diagnostic codes for identifying respiratory illness in a pediatric population. Pediatr Emerg Care 2004;20:355–60.

[6] Espino JU, Wagner MM. Accuracy of ICD-9-coded chief complaints and diagnoses for the detection of acute respiratory illness. Proc AMIA Symp 2001:164–8.

[7] Ivanov O, Wagner MM, Chapman WW, Olszewski RT. Accuracy of three classifiers of acute gastrointestinal syndrome for syndromic surveillance. Proc AMIA Symp 2002:345–9.

[8] Mocny M, Cochrane DG, Allegra JR, Nguyen T, Heffernan R. A comparison of two methods for biosurveillance of respiratory disease in the emergency department: chief complaint vs ICD-9 diagnosis code. Acad Emerg Med 2003;10:513.

[9] Tsui FC, Wagner MM, Dato V, Chang CC. Value of ICD-9 coded chief complaints for detection of epidemics. Proc AMIA Symp 2001:711–5.

[10] Begier EM, Sockwell D, Branch LM, et al. The national capitol region’s emergency department syndromic surveillance system: do chief complaint and discharge diagnosis yield difference results? Emerg Infect Dis 2003;9:393–6.

[11] Bravata DM, McDonald KM, Smith WM, et al. Systematic review: surveillance systems for early detection of bioterrorism-related diseases. Ann Int Med 2004;140(11):910–22. [Review].

[12] Pavlin JA, Kelley PW. ESSENCE: electronic surveillance system for early notification of community-based epidemics. Silver Spring, MD: U.S. Department of Defense, Global Emerging Infections Surveillance and Response System; 4 October 2001.

[13] Mikosz CA, Silva J, Black S, Gibbs G, Cardenas I. Comparison of two major emergency department-based free-text chief-complaint coding systems. MMWR Morb Mortal Wkly Rep 2004;53(Suppl):101–5.

[14] Chapman WW, Christensen LM, Wagner MM, et al. Classifying free-text triage chief complaints into syndromic categories with natural language processing. Artif Intell Med 2005;33(1):31–40.

[15] Travers DA, Haas SW. Using nurses’ natural language entries to build a concept-oriented terminology for patients’ chief complaints in the emergency department. J Biomed Inform 2003;36(4–5):260–70.

[16] Tsui FC, Espino JU, Wagner MM, et al. Data, network, and application: technical description of the Utah RODS Winter Olympic Biosurveillance System. Proc AMIA Symp 2002:815–9.

[17] Wagner MM, Espino J, Tsui FC, et al. Syndrome and outbreak detection using chief-complaint data—experience of the real-time outbreak and disease surveillance project. MMWR Morb Mortal Wkly Rep 2004;53(Suppl):28–31.

[18] Bravata DM, Rahman MM, Luong N, Divan HA, Cody SH. A comparison of syndromic incidence data collected by triage nurses in Santa Clara County with regional infectious disease data. J Urban Health 2003;80:122. [19] Zelicoff A, Brillman J, Forslund DW, et al. The rapid syndrome validation

project (RSVP). Albuquerque, NM: Sandia National Laboratories; 2001. [20] Idaho Technology products. Salt Lake City, UT: Idaho Technology, Inc. Accessed

from:www.idahotech.comon 29 October 2001.

[21] Chapman WW, Dowling JN, Wagner MM. Classification of emergency department chief complaints into 7 syndromes: a retrospective analysis of 527, 228 patients. Ann Emerg Med 2005;46(5):445–55.

[22] Beitel AJ, Olson KL, Reis BY, Mandl KD. Use of emergency department chief complaint and diagnostic codes for identifying respiratory illness in a pediatric population. Pediatr Emerg Care 2004;20(6):355–60.

[23] Talotta, Cochrane D, Allegra J. Chief complaint based quality indicators: the challenge for automation posed by chief complaint variability (abstract). Ann Emerg Med 2005;46:S40.

[24] Shapiro AR. Taming variability in free-text: application to health surveillance. MMWR Morb Mortal Wkly Rep 2004;53(Suppl):95–100.

[25] Terry W, Ostrowsky B, Huang A. Should we be worried? Investigation of signals generated by an electronic syndromic surveillance system– Westchester County, New York. MMWR Morb Mortal Wkly Rep. 2004 Sep 24;53 Suppl:190-5.

[26] Halász S, Brown P, Oktay C, et al. Talking Turkish: usingn-grams for syndromic surveillance in a Turkish emergency department without the need for English translation. (abstract) Acad Emerg Med 2006:13:S115.

[27] Guidelines for evaluating surveillance systems. MMWR Morb Mortal Wkly Rep 1988;37(Suppl 5):1–18.

[28] Travers DA, Waller A, Haas SW, Lober WB, Beard C. Emergency department data for bioterrorism surveillance: electronic data availability, timeliness, sources and standards. AMIA Annu Symp Proc 2003:664–8.

[29] Lober WB, Karras BT, Wagner MM, Overhage JM, Davidson AJ, Fraser H. Roundtable on bioterrorism detection: information system-based surveillance. J Am Med Inform Assoc 2002;9(2):105–15.

[30] Ising AI, Travers DA, MacFarquhar J, Kipp A, Waller AE. Triage note in emergency department-based syndromic surveillance. Adv Dis Surv 2006;1:34.

[31] Stone L, Olinky R, Huppert A. Seasonal dynamics of recurrent epidemics. Nature 2007;446(7135):533–6.

[32] Betancourt JA, Hakre S, Polyak CS, Pavlin JA. Evaluation of ICD-9 codes for syndromic surveillance in the electronic surveillance system for the early notification of community-based epidemics. Mil Med 2007;172(4):346–52. [33] Levy RS, Cochrane DG, Allegra JR. Analysis of interhospital variability in the

sensitivity of a gastrointestinal chief complaint syndromic surveillance method. Acad Emerg Med 2005;12:26.

[34] Halász S, Brown P, Goodall C, Cochrane DG, Allegra J. Thengram classifier: a novel method of automatically creating cc classifiers based on ICD9 groupings (abstract). Adv Dis Sur 2006:1–30.