Evaluating the Accuracy of Transcribed Computer-Stored Immunization Data

(1)

Evaluating

the

Accuracy

of Transcribed

Computer-Stored

Immunization

Data

Richard Wilton, MD* and Alfred

J.

Pennisi, MD

ABSTRACT. Objective. To evaluate the accuracy of

immunization records transcribed into a computer-based

immunization tracking system and to assess factors that

contribute to inaccurate or incomplete immunization

record keeping.

Design. Computer-stored immunization records were

analyzed for 2098 children up to 2 years of age at the time

of their most recent well-child visit to the UCLA

Chil-dren’s Health Center over a 12-month period. For

chil-dren whose immunizations were not up to date, the

com-puter-stored records were analyzed for sources of

inaccuracy by comparison with the handwritten records

from which the computer-stored data were transcribed.

Results. An underimmunization rate of 22.5% (472 of

2098) was observed based on analysis of the

computer-stored records. Comparison of the computer-stored and

handwritten records revealed an overall transcription

er-ror rate of at least 10.2%. In addition, 38.4% of these

apparently underimmunized children had received

unre-corded immunizations from providers outside UCLA.

When transcription errors were corrected and other

avail-able sources of immunization data were taken into

account, the estimated rate of underimmunization

decreased from 22.5% to 10.9%.

Conclusion. Unavoidable inaccuracies can diminish

the utility of the data recorded in an immunization

track-ing system. Some inaccuracies are related to the process

of transcription, but failures to record and communicate

immunization data consistently also contribute to the

inaccuracy of computer-stored immunization records.

Pediatrics 199494:902-906; computer-based immunization

tracking system.

ABBREVIATIONS. UCLA, University of California Los Angeles;

DVT, diphtheria, pertussis, tetanus; TVOP, trivalent oral polio; MMR, measles, mumps, rubella.

A well-publicized goal of the United States Public

Health Service is to ensure that at least 90% of all

children have completed a basic set of

immuniza-tions by their second birthday.’ The data that

docu-ment achievement of that goal necessarily will be

obtained from comprehensive records of childhood

immunizations. The formidable problem of ensuring

the accuracy and completeness of each child’s

immu-nization record will probably be addressed by the

From the Division of General Pediatrics, University of California Los Angeles Department of Pediatrics, Los Angeles, California.

Received for publication Nov 16, 1993; accepted Apr 14, 1994.

Reprint requests to (RW.) UCLA Department of Pediatrics, 12-321 MDCC, Los Angeles, CA 90024-1752.

widespread adoption of computer-based

immuniza-lion tracking systems.

Computer systems can be used for a variety of

purposes in clinical practice: for appointment

sched-uling, for accounting or billing, and for recording

clinical information about individual patients in the

form of an electronic medical record. When clinical

records for a number of patients are stored in a

computer database, it becomes possible to evaluate

clinical data for an entire population. A number of

assessments of the immunization status of pediatric

populations have been based on the analysis of

corn-puter-stored immunization data.25

The use of computer-stored clinical data to

deter-mine immunization status implies that the data are

sufficiently accurate and complete to draw reliable

conclusions about the population of patients

repre-sented in the aggregated data. In the process of

eval-uating immunization coverage in the pediatric

out-patient clinics at the University of California Los

Angeles (UCLA), we carried out a limited analysis of

transcription accuracy of a sample of our own

corn-puter-stored immunization records. Of the children

who were underimmunized according to

computer-stored records, 50% had inaccurately transcribed

records in the computer system.6

Based on this preliminary experience, we designed

a more comprehensive retrospective study in which

we analyzed the accuracy of immunization records

for a larger cohort of children who received

continu-ing care in the UCLA Children’s Health Center. The

analysis was designed specifically to describe the

sources of inaccuracy in transcribed,

computer-stored immunization data and to determine the

ex-tent to which inaccurate and incomplete data in

corn-puter-stored immunization records might affect our

ability to identify underimmunized children in our

clinic population.

METHODS

We analyzed the computer-stored clinical records of 8908 en-counters with 4040 patients seen in the outpatient clinics in the UCLA Children’s Health Center in the 12-month period from July 1992 through June 1993. The records covered patients seen in the UCLA housestaff continuity clinic as well as patients seen in the

general pediatric faculty clinics. The clinical records were cross-checked against computer-stored appointment scheduling and registration data (which record all outpatient encounters in the UCLA Children’s Health Center) to ensure that all visits with all patients in these clinics during the study period were included in the analysis.

at Viet Nam:AAP Sponsored on September 1, 2020

www.aappublications.org/news

(2)

Data Collection

Data-entry personnel transcribed immunization data after each clinic visit from two locations in the handwritten chart.

Immuni-zations given on the day of the visit were transcribed from a

structured encounter form on which the physician or pediatric nurse practitioner wrote the patient’s medical-management plan.

Transcribers also reviewed each chart’s immunization history form, which lists the dates of all of a child’s immunizations and which is updated after each visit by clinic nursing staff. Transcnb-em were nonmedical personnel who were instructed to transcribe only the date (month, day, and year) and type of each

immumza-tion into the computer system. Incomplete immunization data,

including immunizations without dates, and comments such as

“up to date,” were not transcribed. During the study period, seven different transcribers were responsible for entering immunization data into the computer system.

To improve data-entry speed and accuracy, all immunization

data were transcribed using software that displayed a simple

check-off menu and that allowed data entry using the computer keyboard or mouse. The software also allowed both menu-based and free-text transcription of other significant portions of the outpatient medical record, including problem lists, current medi-cations, and the salient diagnostic content of each clinic encoun-ter.7 The computer-stored records included the date of birth, race, sex, and insurance coverage of each patient, the name of each patient’s primary continuity physician, and dated records of each immunization given during that period.

Routine childhood immunizations were recorded as follows:

DPi’ (diphtheria, pertussis, tetanus), TVOP (trivalent oral polio),

and MMR (measles, mumps, rubella). Combinations of antigens

that were used infrequently, such as dT (diphtheria-tetanus) and IPV (inactivated polio vaccine), were also recorded. For purposes of analysis and for comparison to previous studies of immuniza-tion coverage, records of immunizations against other pathogens, including Haemophilus influenzae type b, influenza, and Pneumococ-cus, were not included in our study. Because the recommendation

for universal immunization against hepatitis B was adopted

dur-ing the study period, the data for these immunizations also were excluded from our analysis.

Data Analysis

We analyzed the immunization records in two phases. First, the computer-stored records were reviewed to identify children whose immunizations were delayed. Then the computer-stored

records were compared with the corresponding handwritten records in the outpatient chart.

Analysis of Computer-Stored Records

We first reconciled all computer-stored immunization records with computer-stored appointment scheduling records, to ensure

that immunization records were reviewed for all children who

had arrived for a clinic visit during the 12-month period studied. We then identified the subset of these children whose age was 732 days (2 years) or less at the time of their most recent clinic visit,

and who had fewer than the recommended number of

age-appro-pnate immunizations according to the computer-stored

immuni-zation records.

To categorize immunization status, we used the schedule of

age-appropriate immunizations that was in effect in the UCLA

Children’s Health Center during the study period. This schedule is

based on the recommendations of the American Academy of

Pediatrics8 and the Center for Disease Control’s Immunization

Practices Advisory Committee.9 The schedule recommended DPi’

immunization at 2, 4, 6, and 15 months; TVOP at 2, 4, and 15 months; and MMR at 12 months. We categorized each child’s immunization status as being up to date if the child had received DPi’, TVOP, and MMR immunizations according to the criteria listed in Table I.These criteria for immunization timeliness were

derived from the routine immunization schedule by adding a

30-day grace period to the age for each scheduled immunization. We determined each child’s primary provider of well-child care by using the appointment scheduling records to count the number

of well-child visits to UCLA Children’s Health Center. A UCLA

clinic was assumed to be the child’s primary provider if the child had had at least as many well-child visits in the clinic as would

have been necessary to administer all recommended

immuniza-TABLE 1.

Age Ranges Used to Determine Timeliness of Im-munizations*

Age Numbe r of Immunizations

DVF P/OP MMR

1-91d(0-3mo) 0 0 0

92-151 d (3-5mo) I I 0

152-213 d (5-7mo) 2 2 0

214-395 d (7-13 mo) 3 2 0

396-487 d (13-16 mo) 3 2 1

488-732 d (16-24 mo) 4 3 1

* A child in one of these age ranges was considered up to date if he or she had received at least the corresponding number of each immunization. Abbreviations: DPi’, diphtheria, pertussis, tetanus;

p/op,

trivalent oral polio; MMR, measles, mumps, rubella.

tions, according to the criteria in Table 1. We categorized a child’s primary-care provider as “unknown” if the child had not been seen in a UCLA clinic often enough to receive all recommended

immunizations.

Chart Review

For each child identified as being underimmunized based on computer-stored records, we reviewed the handwritten clinic chart for the following information: 1) transcription errors, i.e., if

the computer-stored list of dates and immunizations did not

match those in the handwritten immunization history in the chart; 2) immunization information written in the chart in a way that it could not have been transcribed, e.g., immunizations recorded without dates or recorded only as “up to date”; 3) records of immunizations received from providers other than the UCLA Children’s Health Center; 4) immunization data written in por-tions of the chart that were not transcribed routinely into the computer system, including inpatient discharge summaries, phy-sicians’ orders, and photocopies of medical records from non-UCLA health care providers; and 5) explicit indication that the child’s immunizations were delayed for any reason.

If a transcription error was noted or if additional immunization data were available in part of the chart that was not transcribed routinely, the child’s immunization status was reevaluated using the information written in the chart.

RESULTS

Analysis of Computer-Stored Records

Of the 4040 patients whose immunization records

were analyzed, 2098 were 2 years of age (732 days) or

younger at the time of their most recent well-child

visit. Analysis of the computer-stored immunization

records of these 2098 children revealed 472 (22.5%)

whose immunization status was not up-to-date for

age, according to the criteria in Table I. The

hand-written UCLA Medical Center charts were reviewed

for 458 of the 472 underimmunized patients. (The 14

charts not reviewed were unavailable from the

out-patient medical records department during the

4-week period during which the immunization

records were verified.)

The UCLA clinics were considered to be the

pri-mary provider of well-child care for 1869 (89.0%) of

the 2098 children less than 2 years of age, as

deter-mined by analysis of appointment scheduling

records (Table 2). Children for whom the UCLA

clinics were the primary provider of well-child care

had an underimmunization rate of 16.2% (303 of

1869); children whose primary provider was

un-known (that is, children not seen frequently enough

at UCLA to have received all recommended

immu-nizations) had an underimmunization rate of 70.7%

(3)

TABLE 2.

Immunization Status Related to Primary Provider of Well-Child Care

Continuity Provider

Immunizations Up to Date

(n = 1626)

Immunizations

Not Up to

Date (n = 472)

Total (n = 2098)

UCLA clinics 1566 303 1869

Unknown* 60 169 229

* Children whose continuity provider is unknown were those who had too few visits to UCLA clinics to have received all recom-mended immunizations.

Chart Review

Of the 458 charts we reviewed for patients who

apparently were underimmunized, 244 were

tran-scribed accurately and completely from the

hand-written record into the computer system. In 214

records, at least one transcription error was noted.

The observed transcription error rate was thus at

least 10.2% (214 of 2098).

Immunizations Received From Non-UCLA Providers

Of the 458 records reviewed, 176 (38.4%) indicated

that the patient had received at least one

immuniza-tion from a provider other than the UCLA faculty or

residents’ continuity clinics.

Nontranscribed Immunization Data

Of the 458 charts reviewed, 72 (15.7%) contained

additional immunization information in locations

that were not transcribed routinely, including

en-counter notes, inpatient discharge summaries, and

photocopies of immunization records from

non-UCLA health care providers. An additional 87 charts

contained immunization data that could not be

tran-scribed: 37 (8.1%) noted that a child’s immunizations

were “up to date” but did not provide explicit dates

that could be recorded in the computer system, and

50 (10.9%) contained an explicit statement that

a patient was underimmunized. In 299 (65.3%) of

the charts, no additional immunization data were

recorded.

Reevaluation of Immunization Status

Immunization status was reevaluated using the

458 handwritten charts we reviewed, including the

214 charts revealing a transcription error in the

cor-responding computer-stored record and the 58

accu-rately transcribed charts that contained additional

458 not UTD in computer.stored records

‘I

244 transcribed accurately

43 UTD based on additional

immunization data

I

214 not transcribed accurately from

handwiitten chart into computer system

I

201 UTD based on handwritten chart

I

244 (43+201) UTD on paper but not UTD in computer system

Figure. Reevaluation of immunization status based on review of handwritten charts. UTD, up to date.

nontranscribed immunization data (Figure). Of these

458 charts, 244 were reevaluated as being up-to-date

based on data in the handwritten chart. When these

244 patients are recategorized, the perceived

under-immunization rate is 10.9% [(472 - 244)/2098] rather

than 22.5% as originally estimated.

DISCUSSION

In our experience, a computer-based

immuniza-tion tracking system is a practical repository for the

immunization records of individual patients, as well

as a ready resource for the aggregation of

inimuni-zation data for a clinic population. Nevertheless, the

accuracy of the immunization data is influenced by a

number of factors, including not only transcription

errors but also inconsistent or incomplete

immuni-zation record keeping on the part of doctors, nurses,

and parents.

Accuracy of Transcribed Immunization Records

We observed an underimmunization rate of 22.5%

(472 of 2098) in the cohort of children whose records

we analyzed during the 12-month study period. This

rate is comparable to underimmunization rates

ob-served previously in other urban populations.’#{176}

However, any interpretation of this

underimmuniza-tion rate must be qualified by analyzing the validity

of the underlying immunization data.

The observation that at least 10.2% of the

comput-er-stored immunization records were transcribed

in-accurately suggests caution in using the

computer-stored records to determine immunization rates.

Similar transcription error rates have been observed

in at least one other computer-based immunization

tracking system,” as well as in a pediatric

emergen-cy-department record-keeping system.12 In the case

of immunization records, such errors are

predomi-nantly errors of omission, so the true rate of complete

immunization is actually higher than the

computer-stored data indicate. As noted above, our estimated

underimmunization rate fell from 22.5% to 10.9%

when we took such errors into account.

Sources of Transcription Errors

In the present study, immunization data were

transcribed after each clinic visit from two

comple-mentary sources, the encounter form (a record of an

individual visit) and the immunization history form

(a cumulative record in the outpatient chart).

Al-though the present study did not differentiate

be-tween errors in transcribing the encounter forms

ver-sus the immunization history forms, both forms were

transcribed at the same time, so it is unlikely that

there would be a significant difference in the

tran-scription error rate between the two. Regardless of

the origin of the handwritten data, any

immuniza-tion tracking system that requires transcription of

written records wifi be susceptible to transcription-related errors.

This study was not designed to detect the false

entry of immunization data into the computer

sys-tern, that is, the recording of immunizations that

were not in fact given. However, the data-entry

soft-ware was designed so that immunization data could

(4)

be entered only by date (month, day, and year) and

by immunization type (selected from a check-off

menu on the computer screen). This software design

does not exclude errors stemming from the incorrect

entry of a date or of an immunization, but it

mini-mizes the possibility of inadvertently entering

im-munization data that are not available on paper. In

fact, of the 214 charts that contained transcription

errors, only two contained transposed immunization

data (e.g., a DPT immunization was recorded instead

of an MMR). All of the remaining errors were

errors of omission, whereby immunizations

writ-ten on paper were transcribed incompletely or not

at all.

Completeness of Computer-Stored Immunization

Records

Several factors influence the completeness of

corn-puter-stored immunization records. For example, we

observed that children with more frequent visits to

UCLA clinics had a much lower rate of

underimmu-nization than children who were seen less frequently

at UCLA (Table 2). However, these data are

con-founded by the fact that non-UCLA immunization

records were not transcribed routinely into the

corn-puter system.

The prevalence of such incomplete immunization

records decreases the utility of the data stored in the

computer system. In particular, it has been suggested

that computer-generated reminder letters to patients

can improve the rate at which parents bring their

children to a clinic to receive needed

immuniza-lions.’3 However, if such reminders are based on

incomplete or inaccurate immunization data, some

of the reminders will be sent to parents whose

chil-dren may actually be up to date. Movement of

pa-tients into and out of a primary-care practice also

may impair the effectiveness of any computer-based

reminder system.14 Because their computer-stored

immunization records are incomplete, children who

receive immunizations from multiple providers may

be mistakenly identified as underimmunized.

Interpreting Computer-Stored Immunization Data

We anticipate that the use of computer-stored

im-munization data will make it possible to estimate

immunization coverage in communities where such

information is difficult to obtain from paper records.

However, it is not easy to determine how well

corn-puter-stored data estimate true immunization rates.

As noted above, there are inherent inaccuracies in the

data resulting from the process of transcription.

Computer-stored immunization records also must be

interpreted cautiously because immunizations

pro-vided outside the scope of the immunization

track-ing system may not be recorded in the system, and

because it is difficult to ensure that immunization

records are transcribed completely from a

handwrit-ten medical chart. These problems complicate any

estimations of community-wide immunization

coy-erage based on computer-stored data.

Representativeness of Data

Because the UCLA Children’s Health Center is

only one of many pediatric health care providers in

the community, it is obvious that UCLA’s recorded

immunization data represent only a sample of the

community’s pediatric population. If we assume that

this sample is random, we compute that the

oh-served 22.5% underimmunization rate has a

stan-dard error of 3.8%. We thus estimate, based on our

sample of the community’s pediatric population, that

the underimmunization rate in the community has a

95% confidence interval from 18.7% to 26.3%.

However, our data almost certainly do not

repre-sent a truly random sample of the pediatric

popula-tion in the community served by UCLA. This is not

because there is an unusually high proportion of

high-risk or subspecialty patients in the UCLA

resi-dent and faculty clinic population.’5 Instead, our

computer-stored data may be unrepresentative

be-cause it is impossible to use the computer-stored

immunization records to determine the true

immu-nization status of children who are not always seen

at UCLA. Children with incomplete immunization

records at UCLA may in fact be fully immUnized

(when immunizations received from other providers

are recorded), underimmi.mized, or entirely

unim-mtinized. Other sources of immunization data, such

as interviews with parents or review of the written

records of immunization providers, must be

accumu-lated to sample the overall immunization coverage in

a community. Such data have been gathered in other

studies in the United States. However, the kinds of

inaccuracies we observed in our transcribed

immu-rnzation data also may exist in other providers’

im-munization data. Because of the scarcity of

cornput-er-stored immunization records in the United States,

published assessments of immunization rates

typi-cally have relied on sampling of handwritten records

in selected populations.’6”7 Although transcription

errors and other factors affecting the accuracy of

sampled immunization data have not been analyzed

systematically, there is no reason that inaccuracies

such as those observed in our computer-stored data

should not also exist in sampled handwritten data.

For example, school records remain an important

source of the data used to estimate immunization

coverage in preschool children,18 yet studies that rely

on school immunization records’#{176}”9 may be

particu-larly susceptible to transcription errors and other

inaccuracies, because school immunization records

are handwritten transcriptions of immunization data

recorded elsewhere.#{176}

Methodologic Considerations

Another barrier to consistent interpretation of

computer-stored immunization data is that there is

no consensus on how to define whether a child’s

immunizations are up to date or delayed.2’ A

num-ber of previous studies of immunization

cover-age’#{176}’17’19have counted a child’s immunizations as

being current if the child receives four DPT, three

plop,

and one MMR by age 2 years. However, this

(5)

useful for evaluating immunization coverage in a

population of children of different ages. For example,

an unimmunized child who is 1 day short of his

second birthday would be considered to be up to

date using the “4:3:1” criterion.

We used a 30-day grace period after the

recom-mended age for each immunization to establish

im-munization coverage for each child at the age they

were last seen in the clinic. Although this method of

evaluating immunization timeliness makes it

practi-cal for assessing the immunization status of a child at

any age, the size of the grace period affects the

perceived immunization status of a significant

number of children. Had we used instead a 60-day

grace period, we would have identified only 408

rather than 472 children as being potentially behind

in immunizations.

Conclusions

There are unavoidable inaccuracies in

computer-stored immunization data. These inaccuracies stem

from several sources. Some result from errors

di-rectly related to the process of transcribing data

from a handwritten record into a computer system.

However, inaccuracies in computer-stored

immu-nization records also may arise from operational

factors unrelated to the mechanical process of

tran-scription.

ACKNOWLEDGMENTS

We thank George Chavez, Simona De La Torre, Ten

Hoff-man, Jennifer Moreda, Mishelle Sharp, Adam Singer, and Scott Welford for their diligent data-entry efforts; Mary Ellen Curry and the outpatient medical records staff in the UCLA Medical Plaza; and Peter Christenson, PhD, for statistical review of the data.

REFERENCES

1. Public Health Service. Healthy People 2000: NationalHealth Promotion and Disease Prevention Objectives. DHHS publication no. (PHS)91-50213. Washington, DC: US Department of Health and Human Services, Public Health Service; 1991

2. Li J,Taylor B. Factors affecting uptake of measles, mumps, and rubella immunization. BMJ. 1993307:168-171

3. Scrivens E. Management information in the National Health Service: the

use of the child-health computer system. Community Medicine. 1984;6:

299-305

4. Alberman E, Watson E,Mitchell P, Day S. The development of perfor-mance and cost indicators for preschool immunization. Arch Dis Child.

1986;61:251-256

5. Hodes Di’, Timms SL, Gill ON. A successful measles campaign as a model for achieving high uptakes of MMR. PublicHealth. 1990;104:27-31

6. Wilton R, Pennisi AJ. Evaluating the accuracy of transcribed dinical data. In: Proceedings of the 17th Annual Symposium on Computer Applica-lions in Medical Care. New Yoric McGraw Hill; 199&279-283

7. Wilton R, McCoy JM. An outpatient clinic information system based on distributed database technology. In: Proceedings of the 13th Annual Sym-posium on Computer Applications in Medical Care. Washington, DC: IEEE Computer Society; 1989:372-376

8. American Academy of Pediatrics. Report of the Committee on Infections Diseases. Elk Grove Village, IL: American Academy ofPediatrics; 1991

9. Centers for Disease ControL Recommendations of the Immunization Practices Advisory Commiftee: general recommendations on immuni-ration. MMWR. 198938:205-228

10. Afford D, Kelly J, Nickey L, Crider R, Arrandondo J,Simpson DM, Kimbler A. Retrospective assessment of vaccination coverage among school-aged children-selected U.S. cities, 1991. MMWR. 1992;41:

103-107

11. Payne 1, Kanvik S, Seward R, Beeman D, et aL Development and

validation of an immunization tracking system in a large health main-tenance organization. Am IPrey Med. 1993996-100

12. Chan IS, Schonfeld N. How much information is lost during process-ing? A case study of pediatric emergency department records. Comput BiOmed Res. 199326:582-591

13. Kemper KJ, Goldberg H. Do computer-generated reminder letters Ira-prove the rate of influenza immunization in an urban pediatric clinic? Am JDis Child. 1993;147:718-719

14. Moran WP, Nelson K, WoffordJL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial. I Gen Intern Med.

1992;7:535-537

15. Wilton R, Pennisi AJ. Insurance coverage and residents’ experience in a pediatric teaching clinic. Am IDis Child. 1993;147:284-289

16. HymelTJ, ShermanJ, Pope SK, Kelleher KJ. Inadequate immunizations: identification using dinic charts. Clin Pediatr. 199332:156-160

17. Bobo JK, Gale JL, Purushoftam BT, Wassilak SGF. Risk factors for delayed immunization in a random sample of 1163 children from Oregon and Washington. Pediatrics. 199391:308-314

18. Orenstein WA. A comprehensive immunization registry. [Transcript of presentation at the 1993 Annual Meeting of the American Academy of Pediatrics.J The SCOT Repert. December 1993;16-18 (Newsletter) 19. Behrens E, Libby J,Borden G, Safranek T. Early ChildhOOd vaccination

in two rural counties-Nebraska, 1991-1992. MMWR. 1992;41:688-691

20. Herold AH, Roetzheim RG, Woodard U, Pamies RJ, Van Durme DJ,

Manning R. Validity of immunization documentation presented to a student health program. IFam Pract. 199336:158-IW

21. Varrasso DA, Redlener LDefining delayed immunizations Eletterl.

Pc-diatr Infect Dis J.1992;11:897

(6)

1994;94;902

Pediatrics

Richard Wilton and Alfred J. Pennisi