AND
PROBLEMS
Barbara Starfield, M.D., M.P.H., Henry Seidel, M.D., Gertrude Carter, B.S.,
William Garvin, M.D., and Johanna Seddon, B.S.
From the Department of Medical Care and Hospitals of The Johns Hopkins University School of Hy-giene & Public Health, the Department of Pediatrics, Johns Hopkins University School of
Medicine, Baltimore, and Clinical Program, Columbia Medical Plan, Columbia, Maryland
ABSTRACT. All 96 physicians in pediatric practice in the Baltimore area were invited to join in a study assessing the quality of health management. The 15 practitioners who participated helped to set the criteria which were modifications of the standards set by the American Academy of Pedi-atrics. Medical records of 5-year-olds were audited for recording of preventive procedures, throat cul-tures for suspected pharyngitis, and urine cultures for suspected urinary tract infection. Some pro-cedures were uniformly well performed; the per-formance of others varied widely. None of the
practices had high rates of recording for all
pro-cedures. Existing records proved inadequate to
assess the sequence of performance of activities and the relationships between problems, diagnostic procedures, responses, and follow-up. Data from questionnaires sent to all 96 physicians indicated that the performance of the 15 participants was
probably higher than that which would be
ob-tained if all the practitioners in the area had par-ticipated. This study has implications for the way in which peer review activities are organized and carried out. Pediatrics, 52:344, 1973, PEDIATRIC PRACTICE, QUALITY, MEDICAL RECORDS, AUDIT, PEER REvIEW.
I
N Apiur. 1971, The Bulletin of PediatricPractice of the American Academy of Pediatrics stated that, “. . . quality
assur-ance will certainly be demanded through-out the country, and it will be preferable to have those who deliver the care set the guidelines rather than those who do not fully understand the problems involved.”
Quality of care was defined as follows:
“.. . a scientffic approach to health
super-vision; the establishment of a diagnosis of deviation from optimum health; institu-tion of appropriate therapy, and manage-ment designed to satisfy the overall needs
of the patient. It should be readily avail-able, efficiently rendered and properly documented (emphasis ours). Preventive
care should be utilized to assure optimal physical, intellectual and emotional growth and development.”1 The Academy has em-barked upon a project to develop a way to measure the quality of health care deliv-ered to children.’ Starting with a few dis-eases, some aspects of health supervision, and some office facilities and equipment, criteria are being developed and tested. Pediatricians and family practitioners are being asked to apply the criteria in their
(Received January 11, revision accepted for publication March 15, 1973.)
Dr. Starfield is a recipient of a Career Scientist Development Award (K02HS46225) from the National
Center for Health Services Research and Development, U.S. Department of Health Education and
Welfare.
G. C., W. G., and J. S. were medical student participants in an apprenticeship-traineeship in Health
Services Research.
This study was supported by grants 5-R01-HS-00110 and 5-T01-HS-00012 from the
Nation-al Center for Health Services Research and Development, grant 5-D04-AH-00076 from the
Na-tional Institutes of Health, U.S. Department of Health, Education, and Welfare, grant 5-A07-AH--00204 from the National Institutes of Health, U.S. Department of Health, Education, and Welfare and contract HSM-110--69-297 from the National Center for Health Services Research and Development.
ADDRESS FOR REPRINTS: (B.S.) Johns Hopkins University School of Hygiene & Public Health,
Practice
1 2 3 4 5 6 7 8 9 10 11
(50) (22) (48) (49) (50) (46) (46) (50) (50) (50) (31) Mean
DPT (489) 88 81 69 80 85 85 78 80 66 90 84 80
Polio (488) 82 81 69 80 88 83 85 94 71 92 87 83
Measles (491) 90 86 58 78 98 91 94 98 47 98 87 84
Mumps (472) 56 10 53 69 91 84 66 67 35 40 40 58
Rubella (481) 34 62 56 70 87 78 68 88 49 68 13 63
Smallpox (492) 94 76 60 87 98 85 91 98 69 96 87 86
Hemoglobin-Hematocrit (492) 28 0 31 10 0 20 9 0 4 6 16 12
Hearing (492) 28 0 0 2 13 0 87 0 2 74 0 20
Tuberculin Blood pressure
(492) (492)
74
26 86
0 69
4 88
27 94
90 94
89 85
83 98
0 66
14 96 26
61 29
83 37
Urinalysis (492) 36 0 19 90 88 20 83 0 34 82 19 46
ARTICLES
TABLE I
CRITERIA FOR COMPLETENESS OF RECORDS
DPT immunization (3 plus booster) Hearing screening: Any type
Polio immunization (2 plus booster) Blood pressure determination
Vaccination for smallpox Rubella vaccine
Hematocrit or hemoglobin determination Measles vaccine
Tine test or other screening procedure for tuberculosis Mumps vaccine Urinalysis: Either pH, sugar and protein (usually by “dipstick”) or microscopic
examination.*
Throat culture: All children with recorded suspicion or diagnosis of pharyngitis (or tonsil-litis, tonsillopharyngitis, sore throat or quinsy).
Urine culture: All children with recorded suspicion or diagnosis of urinary tract infection (or cystitis, bladder infection, urethritis, pyelonephritis).
345
* Even though complete urinalysis should contain all these procedures, the AAP standards2 did not specify the components of a urinalysis. Because of this, the criterion was intentionally made lenient.
own practices (emphasis in the original) by auditing their records.
At the time these recommendations were made, the reported study was already un-derway.
It sought to assess systematically the quality of health care management and to explore the extent to which recommended procedures were recorded in the records of children in pediatric practices.
METHODS
In an attempt to avoid selecting a biased sample of practitioners, all pediatricians
listed by the Baltimore City Health Depart-ment as being in practice at least 50% of their professional time were contacted by mail and invited to participate. Thirty-two
of 96 pediatricians responded; of these, 16 originally agreed to participate but one subsequently declined. Of the 15, three
were associates in one practice and two in another. At a meeting, attended by nine
of the invited 16 participants and the
senior investigators, health maintenance
procedures recommended for 2-year-old
children by the American Academy of
Pediatrics were discussed.2 It was the
con-TABLE II
Polio .tleasles Vaccination Tuberculin ti umps Rubella
test
or Hearing Blood I Trina1sis Test Pressure
.55 .86 -.50 .12 .08 .13
.75 .51 .15 .77 .39
-.47 .00 .21 .18
* Spearlnan Rank Correlation Coefficients (rU)’:All calculations were carried out to five decimal places, only the first two of which are given in the table.
t §: p value less than .05, .01, and .02, respectively. The significance of the obtained r, was tested by
sensus of the practitioners that, although the procedures were appropriate for pedi-atric practice, it would be more realistic to expect their attainment by the age of
5 years. In addition to exploring the
ex-tent to which these procedures were per-formed, the participants also sought to de-termine the frequency of use of mumps, rubella, and smallpox vaccines and the ex-tent to which bacterial cultures had been recorded as done when the diagnosis of
pharyngitis or urinary tract infection was
made.
A data sheet to be used for each record was developed. Revision followed a pre-test on records of ten children who were too young for inclusion in the study. Ar-rangements were then made to examine the first 50 records#{176} of 5-year-olds in each of the 12 practices. Medical students col-lected the data.
Criteria for completeness of records are given in Table I.
#{176}Records were in alphabetic order by last name,
but the order of the alphabet was rearranged using a table of random numbers.
Excluded from the analyses are (1) the one practice containing only seven 5-year-olds, (2) records of children for whom the pediatrician was a consultant only, and (3) records of children who had not been seen in the most recent 18 months and/or which were classified as “inactive.”
Two practices contained fewer than 50 eligible 5-year-olds. The final sample in-cluded a total of 492 children in 11 prac-tices involving 14 practitioners. For some of the immunization categories, the total num-ber of charts is less than 492 because some children had either acquired natural im-munity to one of the diseases or had a medical contraindication to performance of the procedure.
RESULTS
Recording of “Routine” Procedures
Table II indicates that the practices differed in the extent to which any specific procedure was recorded as done and shows the wide variability in completeness of per-formance of different procedures within individual practices.
There appeared to be two principal
pat-TABLE III
CORRELATION BETWEEN PREVENTIVE PROCEDURES RANKED FOR EXTENT OF COMPLETENESS IN 11 PEDIATRIC PRACTICES*
Hematocrit
DPT
Polio Measles Vaccination Tuberculin
Mumps Rubella Hematocrit or
Hemoglobin Hearing Test Blood Pressure
Test
.59 .64t .56 .43 .93 .85 .63t
.87 .76
.69
Hemoglobin
.21 .07 .01
.31 .45 -.43
.45 .58 - .35 .51 .54 -.40
.30 .37 .18
.20 .25 .06
.33 .34 .13
.45 .22 .34
-.06 .14 .05
.37 .76
All 5-lear-Old Children Currently in Practice
Practice No. of Children No. of Cultures
With Suspected Recorded New (IT!
5-Tear-Old Children In Practice Since Birth
No. of Children No. of Children No. of Children
With Cultures With Positive Cultures Unknown
2
1 1 1 28 1
2 0 - 15 0
3 1 1 5 0
4 5 4 11 2
5 0 - 20
-6 1 1 25
-7 1 1 2
-8 1 1 37 1
9 0 - 14
-10 2 2 0
0
11 0 - 0 -
-Total 12 11 157 4 2
All 5-Tear-Old Children C’urrently 5-lear-Old Children in Practice Since Birth
Children With Pharyngitis in Practice
No. of New Practice Diagnoses of
Pharyngitis
52
2 11
3 76
4 35
5 43
6 58
7 30
8 54
9 59
10 36
11 7
Total 461
Diagnoses of Pharyngitis
No. per
Cultures Done No. of Children
No. %
42 81 28
2 18 15
17 22 5
35 100 11
41 95 20
57 98 25
22 73 2
1 2 37
8 14 14
35 97 0
7 100 0
267 58 157
No. % No. child
19 68 25 .89
3 20 3 .20
2 40 5 1.00
6 55 10 .91
16 80 24 1.20
20 80 36 1.44
0 - 0 .00
25 68 49 1.32
9 64 18 1.29
100 64 170 1.08
TABLE IV
CULTURES FOR SUSPECTED NEW URINARY TRACT INFECTIONs (UT!):
11 PEDIATRIC PRACTICES
347
terns of performance of procedures: those with uniformly high rates of performance and those with generally low but quite variable rates.
Spearman’s rank order correlation coeffi-cients were calculated for each pair of pro-cedures (Table III ).3 Diphtheria,
pertus-sis, tetanus, polio, measles, and smallpox immunization and tuberculin testing tended
to be correlated with each other more often than the other six procedures were corre-lated with each other and considerably more often than each of them were cor-related with any of the other six proce-dures. But even for these uniformly well-recorded procedures (all of them generally well accepted in pediatric care) there was variability among the practices; the extent
TABLE V
to which all five were recorded as having been done varied from 37% to 84% of records.
It was, therefore, of interest to determine whether the priority given to the recording of the different procedures was the same for all the practices, and whether some practices would be found to have gener-ally higher recording rates than others. When the procedures were ranked accord-ing to frequency of recording in each prac-tice, an analysis of variance of ranks showed that the practices did not differ
significantly (p < .3) in their rankings of the five major procedures, but that they did vary significantly in their rankings of the other six procedures (p < .001) and all 11 procedures (p < .001). After rank-ing the practices according to the frequency of recording of each procedure, the same statistical procedure4 showed that there were some practices which ranked
consis-tently and significantly higher in their
re-cording of the five major procedures (p < .001) and all 11 procedures (p < .02) than other practices, but that there was only borderline signfficance (p < .10) with regard to the other six procedures.
Diagnostic Cultures for Urinary Tract
Infection and Pharyngitis
Table IV shows the extent to which cul-tures were recorded as having been done
where a new urinary tract infection had
been suspected. Rather unexpectedly, su-spected urinary tract infection was a rela-tively rare occurrence in these practices, occurring about once every 150 person-years of care of children during the first five years of life. The Table also shows that urine cultures were almost always recorded in instances of suspected urinary tract in-fection. In the one child who did not have one (practice 4), a culture was recorded as performed on a follow-up visit after
therapy was begun.
Table V gives the same information for diagnoses of pharyngitis. Suspected pharyn-gitis occurred once per five years of child
care. For the 157 children in the practices since birth, an average of about two-thirds had a diagnosis of pharyngitis. The percent of instances in which cultures were re-corded varied from 2 to 100, but five of the practices did cultures more than 95% of the time.
Although it had been our intention to examine the chronologic sequence of diag-nostic and therapeutic events for these diagnoses, the recapture of this informa-tion from the written records was impossi-ble. It was also not feasible to study the
process of reculture. Culture results were
not always recorded and, even when they were, it was often impossible to determine if recultures were done on a routine basis or because of a recurrence or persistence of symptoms.
Characteristics of Participating
Physicians
In order to determine if physicians who participated in the project differed in some way from those who did not, all pedi-atricians in the Baltimore area were asked to complete a questionnaire. Study partici-pants received their questionnaire in per-son at the time the student visited their offices; others received theirs by mail.
Twelve of the original 15 (73%) partici-pants returned their questionnaire. Of the other 81 physicians, 47 (58%) who com-pleted the questionnaire are designated as
non participants; the remaining 34 are the
nonrespondents.
resi-TABLE VI
CHARACTERISTICS OF BALTIMORE SMSA OFFICE-BASED PEDIATRICIANS
Participants Nonparticipants Nonrespondents
(l4)* (47)*
28.3 68.1
(34)*
16.7 50.0 50.0
57.1
63.6
45.5t
57.1 27.3
80.Ot 71.4
70.5 ND
15.6t ND
56.5 87.2
44.4t 71.7
35.3 ND
ND 62.9
* In most instances, there were a few individuals in each category for whom information concerning the
particular question was unavailable. Thus, the denominators differ, usually by less than 5 from one calculation to another.
fDifferences significant at p <.05 (Chi-Square test), ND = no data.
Differences significant at p <.01 (Chi-Square test). Age, % under 40
Academy fellows (%)
Years of postgraduate pediatric training
% less than 3 Year of graduation
% later than 1960
% work office location in tracts with income >$l3,000
% believing AAP guidelines realistic
% with lab facilities for hematocrit determinations
on staff of a university hospital
dency were taken in university hospitals
(40% of practitioners had their internships and 45% their residencies in a university hospital); current hospital affiliation; and the percent having facilities for hemo-globin determination (81.8 versus 77.7), for hearing determination (50.0 versus 52.3), and for urinalyses without micro-scopic examination (100% for both). How-ever, nonparticipants were more likely (p < .01 in most instances and p < .02 in all instances) to respond that they them-selves (rather than other office personnel) perform procedures such as DPT, polio immunizations, measles, mumps, and ru-bella and smallpox vaccinations, tuberculin tests, and hearing tests. The differences be-tween participants and nonparticipants in the proportion who perform their own hemoglobin or hematocrit determination
and urinalysis were not significant.
DISCUSSION
Even if it were possible to establish widely acceptable standards for physician performance, efforts to examine the effec-tiveness of such performance would be complicated by the important role played
by patients. As long as patients continue to shop for care and fail to comply with recommendations, individual practitioners can hardly be judged unilaterally at fault for not providing comprehensive and con-tinuous care. In this study, the criteria for inclusion of patients were sufficiently rigid to ensure that those whose charts were examined achieved a high degree of regu-larity of care from the indicated physician. Still, there was no assurance that even these patients did not go elsewhere for some
care.
Another factor hampering efforts to as-sess the quality of care is the current in-adequacy of measuring instruments. The limitations of medical records as a source of information about patient-physician in-teractions are well known. The record,
American Academy of Pediatrics accepts the record as an important aspect of
“quality” of practice.1
In this study, the consistent recording of performance of well-accepted, routine pro-cedures and cultures for suspected urinary tract infections demonstrated that the rec-ord can give evidence of procedures done. The records, however, were a poor source of information about the sequence of per-formance of activities and about the phy-sicians’ thought processes. This study con-firmed the finding of others1’3 that most records do not reveal the relationships of problems, diagnoses, diagnostic tests, re-sponses, and follow-ups. Perhaps the adop-tion of the problem-oriented record14 will improve the situation.
To what extent can the performance levels found in these few practices be con-sidered representative of the private prac-tice of pediatrics as a whole? After all, the participants volunteered to take part in the study, and the criteria for assessment were set by those who participated. Our evi-dence indicates that the performance found in these practices was about as high as would be found anywhere in private prac-tice, certainly in Baltimore. Increasing age has been shown to be associated with “poorer” levels of care13’ and the partici-pants, as compared with nonparticipants had a greater proportion of younger, more recent graduates. Participants tended to have more laboratory facilities for perform-ance of at least one of the procedures and at least as many facilities for performance of the others;+ participants have more help from supporting office personnel in the per-formance of almost all procedures. Thus, there was no indication that either the non-participants or the nonrespondents differed
from the participants in any way which
+In a recent (1970) mail survey of a nation-wide sample of pediatric office practitioners, 79% provided hemoglobin or hematocrit determinations, 60% conducted audiologic screening and 88% per-formed urinalysis.16 The participants resembled this national sample with 80%, 52%, and 100%,
would indicate the likelihood of better
per-formance on their part. Actually, although many more of the nonparticipants regis-tered an acceptance of the guidelines set
by
the
Academy, there was no evidence that they were better prepared than the participants to implement them.The attempt to assess quality can be hazardous. There are many efforts to per-fect measuring instruments.’72’ We have suggested that the medical record, as it is now used in the offices of pediatricians, may not be a sufficient source of informa-tion. In any event, no one indicator can establish “quality” of practice. The first step is to set general goals; in this study
this was accomplished by consensus of the participants. When goals are determined, levels of satisfactory achievement should be established. Unfortunately, in this study this was not done,
and
there was no similarprior effort
against
which
performancecould be judged. Morehead and col-leagues,22 using other standards, criteria, and scoring techniques, showed that hemo-globin determinations and urinalyses were the most neglected procedures in an evalu-ation of various types of clinics and prac-tices, but the procedures for assessment and the definition of the
population
were so different that comparisons with per-formance in this study cannot be made. Given the data, each practitioner had to decide individually whether or not improve-ment was warranted.What can be inferred from the incom-pleteness of the records in all of these private practices? The fact that not all practices had facilities for performing all procedures might be a contributory factor. However, lack of facilities could not ex-plain the relatively low rates of adminis-tration of some of the vaccines. Many of the children could have received these immunizations elsewhere: in well-baby clinics, physician’s offices, schools, or dur-ing community drives. The uneveness of performance among the practices and the
351
tients were on Medicaid) makes this un-likely. A more probable explanation is that judgments about the advisability of some of the procedures
varied
among the prac-titioners. This appeared to be an individual rather than a group phenomenon. None of the practitioners rejected or accepted all of the procedures. There was no procedure which had either uniformly high or uni-formly low rates of performance, either ab-solutely or relatively. Even those practices which had overall high rates of recording were found to have quite low rates of re-cording for at least one procedure. If aprocedure was not done, was there a valid reason? A low rate of performance may be the result of rational decision, even if it be at variance with common practice. It could be an indication of excellence.
The poor response of pediatric prac-titioners to the original request to partici-pate was perhaps the most disappointing aspect of the study; less than one in six
physicians volunteered. It seems appropri-ate that those who may one day have to conform to an as yet undetermined set of standards should seek opportunities to help in their development.
REFERENCES
1. The American Academy of Pediatrics. Bulletin of Pediatric Practice, vol. 5, No. 2, 1971, p. 1.
2. Standards of Child Health Care. A Report of the Council on Pediatric Practice of the
American Academy of Pediatrics on the
Delivery of Health Care to Children.
EvanS-ton, Illinois: American Academy of Pedi-atrics, 1967.
3. Siegal, S.: Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill, 1956, pp. 202-213.
4. Siegal, S.: Op cit., pp. 184-193.
5. Physician’s Referral Listing. Medical Care Services Section. Baltimore: Baltimore City
Health Department, 1968 and 1971. 6. American Pediatric Directory, ed. 14. Mrs.
Joe T. Smith, editor and publisher,
Knox-ville, Tennessee, 1969.
7. Directory of Medical Specialists, ed. 15, vol.
2. Chicago: Marquis Who’s Who Inc., 1971. 8. Lembcke, P.: Medical auditing by scientific
methods. JAMA, 162:646, 1956.
9. Fessel, W., and VanBrunt, E. : Assessing qua!-ity care from the medical record. N. Eng.
J. Med., 286:134, 1972.
10. Martin, D.: The Disposition of Patients from
a Consultant Medical Clinic. In White,
K. L., ed : Medical Care Research. Oxford: Pergamon Press, 1965, pp. 113-121.
1 1. Tufo, H., and Speidel, J.: Problems with
medical records. Med. Care, 9:509, 1971. 12. Starfield, B., and Scheff, D.: Effectiveness of
pediatric care : The relationship between processes and outcome. PsnirmCs, 49:547,
1972.
13. Peterson, 0. L., Andrews, L., Spain, R., and
Greenberg, B.: An analytical study of North Carolina general practice. J. Med. Educ., 31 (part 2):1, 1956.
14. Weed, L.: Quality control and the medical record. Arch. Intern. Med., 127:101, 1971. 15. Stolley, P., Becker, M., Lasagna, L., McEvilla,
J., and Sloane, L.: The relationship between physician characteristics and prescribing ap-propriateness. Med. Care, 10:17, 1972. 16. Lengthening Shadows. A Report of the
Coun-cil on Pediatric Practice of the American Academy of Pediatrics on the Delivery of Health Care to Children. Evanston, Illinois: American Academy of Pediatrics, 1971. 17. Hoekelman, R., and Peters, E.: A health
supervision index to measure standards of child care. Health Sew. Rep., 87:537, 1972. 18. Richardson, F.: Peer review of medical care.
Med. Care, 10:29, 1972.
19. Williamson, J.: Evaluating quality of patient
care: A strategy relating outcome and
process assessment. JAMA, 218:564, 1971. 20. Dreyfus, E., Minson, R., Sbarbaro, J., and
Cowen, D.: Internal chart audits in a neigh-borhood health program: A problem-or!-ented approach. Med. Care, 9:449, 1971. 21. Horrigan, E. C. ed.: Peer Review Manual.
Chicago: American Medical Association, 1971.
22. Morehead, M., Donaldson, R., and Seravalli, M.: Comparisons between OEO neighbor-hood health centers and other health care providers of ratings of the quality of health care. Amer. J. Public Health, 61:1294, 1971.
ACKNOWLEDGMENT
