Examining High-Risk Children for Learning Problems in the Health Care Setting

(1)

Examining

High-Risk

Children

for Learning

Problems

in the Health

Care

Setting

James

A. Blackman,

MD, MPH,

and Julie

Bretthauer

From the Department of Pediatrics, University of Iowa, Iowa City

ABSTRACT. The validity of the Pediatric Evaluation of

Educational Readiness (PEER) in evaluating high-risk 5-year-old children who passed developmental screenings through age 30 months was assessed by comparing it with a battery of standardized psychoeducational tests.

High-risk children who “failed” the PEER scored significantly

below those who “passed” the PEER on tests of verbal, perceptual-motor, and preacademic skills. Furthermore,

scores on the PEER of the high-risk group were

signifi-cantly below those of a normal comparison group. When the standardized test battery was used as the true

mdi-cator of developmental concerns, the sensitivity of the

PEER averaged 0.60; specificity averaged 0.88. The over-all hit rate was 78%. False-positive rate was 27%, the false-negative rate, 20%. Observations of behavior, in-cluding attention and activity, correlated at the .63 level

(P

<

.001) with those made independently by a

psychom-etrist. The correlation of these observations to ratings of behavior by parents on the Child Behavior Checklist was .32 (P < .001). It is concluded that the PEER

distin-guishes between groups of children at risk and not at risk

for learning problems; however, in individual cases, the PEER and the standardized test agreed that a child had problems only 60% of the time. Thus two out of five

children who may have problems would be missed by the

PEER. The observations ofbehavior feature ofthe PEER seemed to be a reliable measure and to have some rela-tionship to concerns indicated by parents. With the health history and physical examination, the PEER can

assist in the developmental surveillance of children

known to be at risk for learning problems. However,

based on this study, it cannot be recommended for

screen-ing of general populations. Pediatrics 1990;86:398-404;

learning problems, screening, high-risk children, behavior

problems.

ABBREVIATIONS. PEER, Pediatric Examination of Educational

Readiness; CBCL, Child Behavior Checklist.

Received for publication Jun 14, 1989; accepted Sep 25, 1989.

Presented, in part, at the American Public Health Association

annual meeting in Boston, MA, November 1988.

Reprint requests to (J.A.B.) Kluge Children’s Rehabilitation

Center, University of Virginia, 2270 Ivy Rd, Charlottesville, VA

22901.

American Academy of Pediatrics.

Public Law 99-457, Education for the Handi-capped Amendments of 1986, places renewed em-phasis on early identification and assessment of children with or at risk for developmental disabili-ties during infancy and the preschool years.’ Title I (Part H) of this Act specifies components of services to be implemented with children through age 2; Title II focuses on the processes for identifi-cation and interventions for 3- to 5-year-olds.2 Al-though general developmental screening tools are available for infants, a different approach is needed for preschool children. Screening of infants is in-tended to detect high-severity problems such as mental retardation and cerebral palsy, whereas screening of preschoolers must be aimed at detec-tion of more subtle disorders of behavior and learn-ing.

Changes in neonatal intensive care have reduced significantly mortality rates for low birth weight infants. Whereas more than 90% of all infants weighing less than 1000 g died in 1960, the neonatal mortality rate for this group is now approximately 50%. While early severe neurologic morbidity as-sociated with low birth weight and other neonatal complications has been reported at overall rates of approximately 10% to 20% among survivors in the modern neonatal intensive care era,4 the incidence of learning problems is only recently becoming ap-preciated as long-term studies of survivors of mod-em neonatal intensive care are appearing in the

(2)

* Criteria are not mutually exclusive.

uou

r

-______

_____

L?!ofl.uo0

C,IC.,tcP

. 455Q!4 .5>

I:

3iock Coou

z FrQIr Oppo

Objict

U, Word So.,

BELOW LEVEL LEVEL l.(VEt. REFUSED

LEVELS ONE Two ThROD

013 fJ.5 012 03-4 0

n.-2 n. o

U E’r..24

_13Up$.. 0ups 0

cU3I.p.u 0Zipppp,0

D!3$ou.. 0ZUu.s C’ 0 O.45’ C 064.sc. 0

01,2 0 34 0

01,3 04.7 Ii

__!InIcc 0 AokoWd [ C’jA OS 0

LilA OS U

C±lkscc. UAolrowd

US 1)5 0

0005u#{149}dl

CaI.gSoz.

( &u 012 13-rU Li were at or above 100; nevertheless, 33% were in

special education classes or required individualized educational programming for learning problems.

There is a need, therefore, for some method of evaluation of children who do not have cerebral

palsy and are not mentally retarded but remain at risk for difficulties in school because of specific

learning disabilities. Ideally the evaluation would

occur at the time of or before formal school entry,

usually kindergarten; would be relatively inexpen-sive; could be performed in a familiar setting (such

as the pediatrician’s office or a clinic); and would

include integration of health, social, and develop-mental information.

The Pediatric Examination of Educational

Read-mess (PEER) is a combined neurodevelopmental, behavioral, and health assessment’2 designed for evaluation of children between the ages of 4 and 6

years. It is intended to generate a profile of a child’s developmental and behavioral strengths and

weak-nesses that may be helpful in planning integrated health, educational, and developmental services.

Original norms for PEER were derived from test-ing 386 Boston area children. Interobserver relia-bility ranged from 84% to 95% on various sections. Concurrent validity comparing certain dimensions

of the PEER with the McCarthy Scales of

Chil-dren’s Abilities showed statistically significant

agreement.’3 Predictive validity was established by monitoring the children’s performance in kinder-garten after administration of the PEER. Children with multiple areas of concern on the PEER were found to have lower teacher rating scores at the end of kindergarten than those with isolated or no concerns.’4 Several modifications ofthe PEER have been made since the initial standardization but no reports of validation have appeared since the orig-inal cohort was tested.

The purpose of this study was to determine the concurrent validity of the PEER in identifying potential learning and behavior problems among a high-risk population. Although it was intended originally to provide descriptive information about a child rather than a “score,” quantitative results were used in this study to compare findings from the PEER with those from a battery of standardized psychoeducational tests.

METHODS

Subjects were 270 children enrolled at birth in a statewide high-risk infant follow-up program be-cause of biologic risk factors (Table 1). The PEER (Fig 1) was administered by a pediatrician or pedi-atric nurse practitioner and an extensive psycho-educational battery oftests (Table 2) by a psychom-etrist. Both tests were given the same day;

approx-TABLE

1.

Proportion of Pediatric Examination ucational Readiness Examinees Meeting Each High-Risk Follow-up Program Entry Criteria*

of of

Ed-the

Criteria %

Birth wt <1500 g 27

Respiratory distress syndrome 41

Bacterial meningitis 2

5 mm Apgar score s6 29

Hypoglycemia 6

Neonatal seizures 7

Ventilatory assistance 2 h 50

Hypotonia 2

Other 15

4;i-F--Th

1(3 13-4 [1 >4

1ro3 Cowt4Jd_ _iij:o 21!SJ9

L_:’ ““s IistrrwoUL8Co

Fig 1. Developmental Attainment section of the Pedi-atric Examination of Educational Readiness.

imately half the subjects received the PEER first, whereas the order was reversed for the other half. These children had previously passed develop-mental screenings with the Denver Developmental Screening Test’5 through age 30 months. The ex-aminers were blind, as far as possible, to whether a child was in the high-risk or the comparison group (described below). Health histories were not avail-able until the testing was complete. The PEER testers were trained by the author (J.A.B.) and observed until administration instructions were fol-lowed precisely and there was agreement on scoring. Mean age (SD) of the sample was 60.2 months

(1.7), 54% were male, mean maternal educational

(3)

Vocabulary (WPPSI)24*

Preacademic learning

Letter identification’#{176}

Segmenting Test3’

Number concepts’#{176}

Number questions32 ___________________________________________________

* WPPSI, Wechsler Preschool and Primary Scale of Intelligence.

TABLE 2.

Subtests Used in Psychoeducational Battery

Test Instruments

Verbal ability

Boston Naming Test22

Token Test23

Sentences (WPPSI)24*

Auditory association25

Perceptuomotor ability

Block patterns

Developmental Test of

Visual-Motor Integration27

Recognition-diecrimination

Corsi cubes

Functions Assessed

Expressive language: verbal labeling and word retrieval in naming

drawings of common objects

Receptive language: understanding and execution of verbal direc-tions

Expressive language: oral vocabu-lary and description of word meaning

Immediate verbal memory for

meaningful information

Understanding and explaining

ver-bal analogies

Visuospatial perception and

visuo-constructive ability in three-di-mensional block building Visuospatial perception and

gra-phomotor skill in copying

geo-metric designs

Visual scanning, visual perception, and shape matching (without motor demands)

Spatial memory span in recalling positions of randomly arranged

blocks

Recognition and naming of

upper-case and lowercase letters Ability to divide words into

compo-nent phonemes

Counting specified numbers of ob-jects

Solving oral arithmetic problems

Inasmuch as the purpose of this study was to examine the PEER’s utility as a valid assessment instrument, it was necessary to develop pass/fail criteria. Local norms for the PEER and the psy-choeducational battery were obtained from a group of 70 five-year-olds without a history of neonatal complications and matched with the high-risk group by age in months and parental education. Scores for the PEER were derived by multiplying the number of “below level” checks by 0, “level 1” by 1, “level 2” by 2, and “level 3” by 3 in each developmental area (eg, orientation) and for the entire Developmental Attainment section (Fig 1). Failure on the PEER was defined arbitrarily as a total score of greater than 1 SD below the mean (ie, below the 17th percentile).

Because the tests in the psychoeducational bat-tery (Table 2) were developed using different sam-ples and each has its own norms, a new scale was constructed by transforming the comparison group’s raw scores for all tests into a normal distri-bution with the mean set at 50 and 1 SD at 10.

Performance of less than 40 (which was greater

than 1 SD below the mean) was considered a failure on these tests.

The PEER contains an “Associated Observa-tions” section (Fig 2), which is divided into three

parts: selective attention/activity, processing

(4)

chil-A B

I

TABLE 4.

Test Properties of the Developmental At-tainment Section of the Pediatric Examination of Edu-cational Readiness

oE

---.----. SCRIPT 1

ACTIVITY LEVEL

DISTRACTISILITY

P

.5rLD5.L TisEs ..j... ...L...

Oo.r-#{149}-,rOor 0 i SolNy TOSS. 0 0 ___.._i FATIGABILJTY ‘ ‘ ow .5!> 5

‘r. Mpp r AlllosEs

PS5DL TuMo __.__ -s ...j.. .2_.____ ...L...

.Uraor-! #{149}‘--.S$ES ____ 2____.

TASK PERSISTENCE N..d.d ProddIpp on QolorTwo

N.sdsd Proddlpp or,Mors Thoo Two TapEs Aliost.d SOsQosle TlmposNI Tasks

REFLECTIVE (SoS (S.. -0.... f&&ks T U..5

BEHAVIOR

Appropr$st.(NoCsncsmScsr. SO 510

ooQEs Hs R

2

0 Ioo5Es

TASK EXECUTION

PVOOPOS4E

--Tos Tasks orionI.

SQRspsk

INSTRUCTION AND Rsps4 10’S

DEMONSTRATION Fs#{231},ir US

SorB 0( s..,,.

Of Ds,.as-r-#{149}s’ on ORETask I I

or#{149}ppppp..&5fl 5fl)On. Task

---Appropilsis (NoCoocsrn)Scsrs 44 4.5

i i’_

:ol

a3

-:#{176}__

RAPPORT WITH EXAMINER aND ::

INVOLVEMENT WITH EXAMINATION

RNsy IITSSER Q,ap5y

p AnyTasks

2__ 2 ‘?____ 1____ o 2____ ToEs

Apprspds*s (NB Casc.m)Scsrs 3.4 34

Fig 2. Associated Observations section of the Pediatric

Examination of Educational Readiness.

then with high externalizing scores tend to “act out” feelings and are likely to be aggressive, hostile, or hyperactive. The higher the score on the CBCL, the worse the behavior.

RESULTS

Mean scores on the verbal, perceptual-motor, and preacademic skill areas of the psychoeducational battery were significantly lower for high-risk chil-then who failed the Developmental Attainment sec-tion of the PEER than for those who passed (Table 3). However, while such group comparisons provide a sense of the overall ability of the PEER to detect children with potential learning problems, an ex-amination of the sensitivity and specificity of the PEER is necessary to determine how accurate the PEER is likely to be in individual cases.

The PEER indicated that 29% of high-risk chil-then were having problems in verbal, perceptual-motor, or preacademic skills, whereas the psycho-educational battery showed problems in 35%. Table 4 summarizes the test properties of the PEER in relation to the psychoeducational battery. When

passes and failures on the entire Developmental Attainment section of the PEER were compared with passes and failures on the three skill areas of the psychoeducational battery, sensitivity ranged from 0.55 to 0.62 and specificity from 0.86 to 0.90. Thus, the PEER correctly identified children who had problems approximately 60% of the time; it correctly identified children who did not have prob-lems 90% of the time. The two evaluations agreed that there were problems in approximately 21% of

cases and that there were no problems in

approxi-S

0

LATENCY OP RESPONSES ks orTos Tssks

5o

Thss Too TapEs

TABLE 3.

Psychoeducational Scores of Children Who “Passed” vs Children Who “Failed” the Developmental

Attainment Section of the Pediatric Examination of

Ed-ucational Readiness (PEER)

PEER Results Mean (SD) Scores On Psychoeducational

Battery

Verbal Perceptual- Preacademic

Motor

Pass 48.0 (10.8) 46.6 (7,5)* 48.5 (9.1)*

Fail 34.9 (10.4) 35.7 (7.2) 33.3 (9.7)

*P< .001.

Feature Verbal

Perceptual-Motor

Preaca-demic

Sensitivity 0.55 0.63 0.62

Specificity 0.86 0.88 0.90

Total hit rate (true posi- 0.75 0.79 0.80 tive

+

true negative +

by total)

False-positives 0.30 0.28 0.22

False-negatives 0.23 0.17 0.19

Predictive test positive 0.70 0.72 0.78

Predictive test negative 0.77 0.83 0.81

TABLE 5.

Correlation Between the PEER

mental Attainment Section and the Correspon

Areas of the Psychoeducational Batteryk

Develop-ding Skill PEER Psychoeducational Battery r Linguistic Verbal

Visual-fine motor Perceptual-motor Preacademic learning Preacademic

Total PEER score Verbal

Total PEER score Perceptual-motor

Total PEER score Preacademic

.566 .649 .649 .604 .649 .670

* PEER, Pediatric Examination of Educational

Readi-ness. All correlations significant at the P < .001 level.

mately 58% of cases, with an average overall hit rate of 79%. The false-positive rate averaged 27%;

false-negative rate was approximately 20%. When

the PEER result was positive, it was correct ap-proximately 75% of the time (predictive test posi-tive); when the result was negative, it was correct approximately 80% of the time (predictive test neg-ative).

(5)

of the PEER (eg, orientation, gross motor, sequen-tial tasks) that did not have corresponding areas on the psychoeducational battery. These skill areas may add evaluative dimensions not captured on the psychoeducational battery, but they did not appear to add to or detract from the positive correlations of the total PEER score with the psychoeducational battery. The high-risk group scored significantly lower on PEER visual-fine motor, sequential, lin-guistic, and preacademic tasks than the comparison group (P = .01), but there were no differences on

orientation or gross motor tasks.

The Associated Observations section of the PEER was completed by both the pediatrician and the psychometrist. Their total scores correlated at the .63 level (P

<

.001), providing one measure of interobserver reliability for this section. The pedia-trician or the pediatric nurse practitioner com-pleted two sets of observations, one in the middle and one at the completion of the Developmental Attainment section. These two observation points on the same child correlated at the .78 level (P <

.001), reflecting some stability of the child’s behav-ior as well as the observer’s ratings.

The high-risk group did not differ from the corn-parison group on total associated observation scores or total CBCL scores. Correlations between the pediatrician’s observations of behavior and the par-ent-completed CBCL are summarized in Table 6. The highest correlations were between selective attention/activity and the externalizing score of the CBCL and between the total associated observation score and the internalizing, externalizing, as well as total CBCL score.

DISCUSSION

The purpose of this study was to assess the validity of the PEER as an evaluation instrument. Validity, an essential attribute of a test, has four elements: content, construct, concurrent, and pre-dictive.’7 Content validity is usually established by asking experts whether the items are a representa-tive sample of the skills and traits that are to be measured. According to its manual, the design of

the PEER was based on the expertise of a large number of professionals collaborating for an 8-year period.’2

Although predictive validity may be the most desired quality of a screening instrument, it is the most elusive. There is growing consensus that the early signs of learning problems may become man-ifest during the preschool years and can be accu-rately identified.’8 However, there are many con-ceptual pitfalls to this process. The child’s devel-opmental status may change over time; abnormality can be confused with atypicality or developmental variation; and the precursors oflearning disabilities may be more difficult to identify than the learning disabilities themselves. It should not be assumed that developmental screening tests will identify all individuals who at a later time will manifest devel-opmental problems.’#{176} This study did not attempt to address the predictive validity of the PEER.

Construct and concurrent validity are somewhat

related. It has been shown elsewhere that, as a group, high-risk infants have a higher than ex-pected rate of learning problems even though they generally display intelligence in the normal range.7

11 In this study, the high-risk group scored

signifi-cantly below the comparison group on individual skill areas as well as on the total Developmental

Attainment section of the PEER. Thus, it appears that the PEER has a certain construct validity in that it successfully identified the group most likely to have learning problems.

The problem with determining the concurrent validity of an evaluation instrument is choosing an appropriate standard against which the test can be judged. For this study published tests or subtests

appropriate for 5-year-olds were grouped into three major developmental skill areas and administered to a group of matched comparison subjects with normal birth histories. This re-normed psychoedu-cational battery became the standard of comparison or the criterion test for the PEER. Similar to the original report of concurrent validity of the PEER,’3 children in our study who failed the PEER scored significantly lower on the psychoeducational battery than those who passed the PEER. However,

TABLE 6. Correlations Between PEER Examiners’ Associated Observations and the

Child Behavior Checklist*

PEER Associated

Observa-tions

C hild Behavior Checklist

Internalizing Externalizing Total

Selective attention/activity .287 .308 .297

Processing efficiency .279 .212 .283

Adaptation .208 .178 .196

Total .324 .308 .329

* PEER, Pediatric Examination of Educational Readiness. All correlations are negative

(6)

if the PEER is to be used as an evaluation instru-ment, it must have acceptable sensitivity and spec-ificity. The probability that the PEER would iden-tify a child who performed poorly on the verbal, perceptual-motor, or preacademic skill areas of the psychoeducational battery was 55% to 60%. When a child had no problems, according to the psycho-educational battery, the PEER agreed approxi-mately 90% of the time. Specificity is usually high in low-incidence conditions. In other words, if one “guessed” that an individual did not have a rare condition, he would be right most ofthe time. False-positive and false-negative rates were 20% to 25%. Interpreting these results depends on expectations for any test attempting to detect developmental dysfunction in preschoolers adequately and for standardized tests to serve as the gold standard of comparison. Although the test properties of the PEER might be graded as only fair, perhaps they are as accurate a sampling of developmental skills among preschoolers as could be expected. Even though the PEER examines certain dimensions of development not typically assessed in other avail-able tests (ie, orientation, motor skills, and se-quencing abilities), these dimensions did not en-hance the test properties of the PEER over its linguistic, visual-fine motor, and preacademic learning sections. As seen in Table 5, the total PEER score correlated approximately as well as individual skill areas to the components of the psychoeducational battery. A dimension such as sequencing may prove to be a useful area to screen in identifying more specific areas of learning diffi-culty.

A unique feature of the PEER is the Associated Observations section. There was reasonably good correlation between the observations of the pedia-trician and the psychologist made independently and at different times, though on the same day.

This suggests that this scale is a reliable method of

recording behavioral style. Relationships between these observations and parent report of behavior were less strong. The strongest correlation was found between the selective attention/activity rat-ings made at the end of the PEER and the Exter-nalizing subscale of the CBCL (which includes di-mensions of hyperactivity) and between the total associated observation score and both the subscales and total scores of the CBCL. These findings sup-port the value of direct observations of behavior under stressed conditions. However, conclusions must be based on serial observations and on obser-vations made in a variety of settings such as home, school, and play.

Because no other currently available develop-mental test for preschoolers incorporates health

and behavioral dimensions as well, there is appeal to using the PEER in a variety of ways, including screening, especially of children thought to be at risk. Based on our results, the PEER cannot be recommended for screening general populations (the limited sensitivity would not justify the ex-penditure of resources); but, with reservations, it may be of use for children at risk or about whom concerns have been raised. Rather than being used for community-wide screening of children, the PEER can provide further information about chil-dren who seem to be at risk during the preschool years. The high proportion of children with prob-lems on both the PEER and psychoeducational battery suggests that many high-risk children who have passed early developmental screening may fit into this category. Cutoff scores or norms must be used with caution and the recognition that if a child performs poorly on the PEER, concerns are war-ranted however, if a child performs acceptably on the PEER, there is no guarantee that the child will succeed in school. The PEER may be strongest in allowing direct, qualitative observations of devel-opment and behavior. It offers an alternative to the piecemeal approach: the child goes sequentially to a physician for a health history and physical ex-amination, to a psychometrist for educational and cognitive testing, and to a counselor for behavioral evaluation. Health care providers or a family might not have ready access to all these professionals or such a complete evaluation might not seem war-ranted without a first book On the other hand, the PEER requires training and experience to admin-ister and interpret properly and takes up to 2 hours to complete with provision of feedback to parents. Often reimbursement for such services is limited.

While the PEER takes a bit longer to administer than screening tests to which pediatricians are ac-customed, low-severity dysfunctions necessitate a wide range of developmental, preacademic, and be-havioral observation. For better or for worse, such a wide band of functional observation requires more time than is allowed on the usual rapid screening

(7)

reduce frustration and possible disruptive behavior in a child with fine motor weakness by limiting work with scissors. The child with dysfluent speech or auditory processing problems would be referred to a speech pathologist for further evaluation and recommendations. The strengths of a child who is shy or who has low self-esteem could be emphasized to parents and teachers. Developmental screening must be periodic and should be integrated with the opinions and concerns of parents, day-care pro-viders, preschool teachers, and others familiar with the chi1d.#{176}Parent and teacher questionnaires are available to use in conjunction with the PEER.2’

Conceptually the PEER is consistent with rec-ommended approaches to providing comprehensive services, including developmental assessment, to children. Further work is needed to refine its con-tent and to validate it in a variety of settings with diverse populations, as well as against other meas-ures of developmental and behavioral function.

ACKNOWLEDGMENTS

This study was supported, in part, by a grant from The Robert Wood Johnson Foundation (Princeton, NJ).

We thank Muriel Metz, Kathy Breese, and Brenda

Koonce for typing assistance.

REFERENCES

1. DeGraw C, Edell D, Ellers B, et al. Public Law 99-457: new

opportunities to serve young children with special needs. J

Pediatr. 1988;113:971-974

2. Education of the handicapped amendments of 1986. Federal

Register. November 18, 1987;1145-1177

3. Neonatal Intensive Care for Low Birth Weight Infants: Costs and Effectiveness. Washington, DC: Office of Technology

Assessment; 1987:4-5. Health technology case study 38,

publication OTA-HCS-38

4. McCormick MC. Long-term follow-up ofinfants discharged

from neonatal intensive care units. JAMA.

1989;261:1767-1772

5. Blackman JA, Hem HA. Iowa’s system for screening and

tracking high risk infants. Am J Dis Child.

1985;139:826-831

6. Bennett FC. The effectiveness of early intervention for

infants at increased biologic risk. In: Guralnick MJ, Bennett

FC, eds. The Effectiveness of Early Intervention for At-risk and Handicapped Children. Orlando, FL: Academic Press

Inc;1987:79-112

7. Blackman JA, Lindgren SD, Hem HA, Harper DC.

Long-term surveillance of high risk children. Am J Dis Child.

1987;141:1293-1299

8. Sell EJ, Gaines JA, Gluckman C, Williams MS. Early

iden-tification of learning problems in neonatal intensive care

graduates. Am J Dis Child. 1985;139:460-463

9. Klein N, Hack M, Gallagher J, Fanaroff AA. Preschool

performance of children with normal intelligence who were

very low-birth-weight infants. Pediatrics. 1985;75:531-537

10. Siegel LS. Reproductive, perinatal, and environmental

van-ables as predictors ofdevelopment ofpretenm (<1501 grams)

and full term children at 5 years. Semin PerinatoL

1982;6:274-279

11. Vohr BR, Garcia Coil CT. Neurodeveiopmental and school

performance of very low-birth-weight infants: a seven-year

longitudinal study. Pediatrics. 1985;76:345-350

12. Levine MD, Schneider EA. Pediatric Examination of

Edu-cational Readiness. Cambridge, MA: Educators Publishing

Service, mc; 1982:iii

13. Levine MD, Obenklaid F, Ferb TE, Hanson MA, Palfrey JS,

Aufseeser CL. The pediatric examination of educational

readiness: validation of an extended observation procedure.

Pediatries. 1980;66:341-349

14. Meltzer U, Levine MD, Palfrey JS, Aufseeser CL, Obenklaid

F. Evaluation of a multidimensional assessment procedure

for preschool children. J Dev Behav Pediatr. 1981;2:67-73

15. Frankenburg WK, Goldstein AD, Camp BW. The revised

Denver Developmental Screening Test: its accuracy as a

screening instrument. J Pediatr. 1971;79:988-995

16. Achenbach TM, Edelbnick C. Manualfor the Child Behavior

Checklist and Revised Child Behavior Profile. Burlington,

VT: University ofVermont, Department of Psychiatry; 1983

17. Kosecoff JB. Evaluation Basics. Beverly Hills, CA: Sage

Publications, mc; 1982:123

18. Adelman HS. Identifying learning problems at an early age:

activated appraisal. J Clin Child PsychoL 1982;11:255-261

19. Frankenburg WK, Chen J, Thornton SM. Common pitfalls

in the evaluation of developmental screening tests. J

Pe-diatr. 1988;113:111O-1113

20. Dworkin PH. Developmental screening expecting the

im-possible? Pediatrics. 1989;83:619-622

21. Levine MD. ANSER System. Cambridge, MA: Educator’s

Publishing Service; 1980

22. Kaplan E, Goodglass H, Weintraub S. Boston Naming Test.

Philadelphia, PA: Lea & Febiger; 1983

23. DeRenzi E, Faglioni P. Normative data and screening power

of a shortened version of the Token Test. Cortex.

1978;14:41-49

24. Wechsler D. Wechsler Preschool and Primary Scale of

Intel-ligence. New York, NY: Psychological Corp; 1967

25. Kirk SA, McCarthy JJ, Kirk WD. Illinois Test of

Psycholin-guistic Ability. Champaign, IL: University of Illinois; 1968

26. Hiskey MS. Hiskey-Nebraska Test of Learning Aptitude.

Lincoln, NE: University of Nebraska; 1966

27. Beery KE. Developmental Test of Visual-Motor Integration.

Cleveland, OH: Modern Curriculum Press; 1983

28. Satz P, Fletcher J. Florida Kindergarten Screening Battery.

Odessa, FL: Psychological Assessment Resources mc; 1982

29. Milner B. Interhemispheric differences in the localization of

psychological processes in man. Br Med BulL

1971;27:272-277

30. Brigance A. Brigance Diagnostic Inventory of Basic Skills.

Woburn, MA: Curriculum Associates mc; 1977

31. Fox B, Routh DK. Analyzing spoken language into words,

syllables, and phonemes: a developmental study. J

Psycho-linguist Res. 1975;4:331-342

32. McCarthy D. McCarthy Scales of Children’s Abilities. New

(8)

1990;86;398

Pediatrics

James A. Blackman and Julie Bretthauer