Evaluating Intensive Family Preservation Programs: A Methodological Review

(1)

Evaluating

Intensive

Family

Preservation

Programs:

A Methodological

Review

Amy M. Heneghan, MD*; Sarah M. Horwitz, PhD; and John M. Leventhal, MD

ABSTRACT. Objectives. To determine the adequacy of evaluations of family preservation services UPS), which are designed to support families and prevent out-of-home placements of children at risk of abuse or ne. glect, and to assess the effectiveness of FPS at reducing out-of-home placements of children.

Data Sources. References published from 1977 to

1993 were identified from a computerized search of da-tabases for English-language publications using the key phrases “family preservation,” “child abuse,” and “fam-ily-based services.” Unpublished references were identi-fled by mail or phone from a listing of more than 200 programs in a national directory.

Selection of Studies. Of 802 references initially

identified, 46 program evaluations were reviewed. Ten studies met the following inclusion criteria: (1) evalu-ated an intensive family preservation program, (2)

in-cluded outcome data in the report, and (3) used a

comparison group. Five were randomized trials, and 5 were quasi-experimental studies (nonrandomized).

Data Extraction. Descriptive information about the

programs and evaluations was collected. To determine methodological quality, two independent raters used a 15-item questionnaire to examine the assignment of families to treatment groups, the interventions provided, and the out-comes assessed. A composite score of 11 or greater repre-sented an acceptable study, 6 to 10 represented an adequate study, and 5 or less represented an unacceptable study.

Results. Only two studies were rated acceptable, four

were adequate, and four were unacceptable.

Methodologi-cal shortcomings induded poorly defined assessment of

risk, inadequate descriptions of the interventions provided, and nonblinded determination of the outcomes. Rates of out-of-home placements were 21% to 59% among families who received FPS and 20% to 59% among comparison fam-flies. The relative risk of placement was significantly re-duced by H’S in only two studies (one randomized trial and one quasi-experimental study).

Conclusions. Despite current widespread use of FPS

to prevent out-of-home placements of children, evalua-tions of Fl’S are methodologically difficult and show no benefit in redudng rates of out-of-home placements of children at risk of abuse or neglect in 8 of 10 studies. Consistent, methodologically rigorous evaluations are needed to determine the effectiveness of FF5 and to

From the Wood Johnson Clinical Scholars Program and

Depart-ments of Epidemiology and Public Health and §Pediatrics, Yale University

School of Medicine, New Haven, Connecticut.

This work was presented in part at the 34th Annual Meeting of the

Ambu-latory Pediatric Association, Seattle, WA, May 1994.

Received for publication Apr 17, 1995; accepted Oct 24, 1995.

Reprint requests to (A.M.H.) Assistant Professor of Pediatrics, Albert Fin-stein College of Medicine, NR 75 19, 1300 Morris Park Ave. Bronx, NY

10461.

Acad-emy of Pediatrics.

guide social policy for high-risk children and their fam-ilies. Pediatrics 199697:535-542; family preservation,

child abuse, family-based services, methodological review.

ABBREVIATIONS. FPS, family preservation services; RR, relative risk; CI, confidence interval; HEN, Hennepin County, Minnesota;

RAM, Ramsey County, Minnesota.

During the past two decades, programs to

main-tam and improve family functioning have become an

important approach to treatment when children are

referred to protective service agencies for abuse or

neglect.13 Such family preservation programs

pro-vide short-term, home-based services to help

stabi-lize families and to reduce the risk of out-of-home

placement of children. Family preservation in the

context of child protection grew throughout the

1980s and 1990s as rates of crack cocaine use and

childhood poverty rose and reports of child abuse

and neglect steadily increased from 669 000 cases in

1976 to 2.99 million cases in 1993. During the same

decade, estimates of children in out-of-home

place-ments exceeded 500 #{216}#{216}tJ6

Family preservation is based on the principle that

families are more responsive to change during times

of crisis,7’8 and that by modifying family behaviors,

many out-of-home placements can be avoided. Child

welfare systems and policymakers have supported

the use of family preservation programs strongly,

because these programs maintain the child within

the family and are less costly than out-of-home care (eg, foster care). Their widespread use was facilitated

by the 1980 Adoption Assistance and Child Welfare

Act (Public Law 96-272) that requires states to make “reasonable efforts, prior to the placement of a child into foster care, to prevent or eliminate the need for

removal of the child from his home.”9 In addition to

the legislative mandate, family preservation

pro-grams and other family-focused services have been

promoted both by the Children’s Bureau of the US

Department of Health and Human Services and a

number of private organizations and foundations,

such as the Edna McConnell Foundation.3 As a

re-sult of the current political and professional support

for family preservation, there has been a marked

increase in the number of programs, with more than

30 states integrating family preservation services

(H’S) into their state-wide child welfare systems.10’11

Likewise, fiscal support has continued to grow

with increased funding for these services assured by

modifications to Title N-B of the Social Security Act

at Viet Nam:AAP Sponsored on September 1, 2020

www.aappublications.org/news

(2)

by the Omnibus Budget Reconciliation Act of 1993.

This legislation provides entitlement funding to

states’ welfare agencies “for the purpose of encour-aging and enabling each state to develop, establish,

expand, and operate a program of family

preserva-lion services.”2 Funds of $60 million are already

available to state child protection agencies to

imple-ment plans to provide family support services and

FPS. Yearly increases are allocated to a maximum of

$255 million dollars in 1998, with a total budgetary commitment to FPS close to $1 billion dollars during the next 4 years. In addition, states have used fund-ing from other legislation to support FF5, including Title IV-E and the Child Abuse Prevention and Treat-ment Act.’3

Despite the current widespread use of family

pres-ervation programs and the recent surge of federal

dollars allocated for further implementation, it is

unclear how effectively such programs increase

fam-ily stability and functioning, prevent child

maltreat-ment, and improve the child’s well-being. Although

there have been several evaluations of H’S, these

evaluations, which have focused primarily on the

prevention of out-of-home placements, have

pro-duced conificting results.Ll4lS Therefore, we

con-ducted a methodological review to determine: (1)

whether evaluations of FF5 were conducted in a

scientifically rigorous manner; and (2) whether the

evaluation data currently available support the

con-clusion that FPS reduce the rates of out-of-home

placements for children compared with customary

social service interventions. We performed a

meth-odological analysis rather than a meta-analysis that

used pooled data, because we wished to describe,

both qualitatively and quantitatively, the scientific validity of the evaluations examined.

METHODS

Selection Criteria for Inclusion of Evaluations

Evaluations of FF5 were included in the methodological review if they met the following eligibility criteria.

Assessment of an Intensive Family Preservation Program

Because of the heterogeneity of programs providing FPS, we

confined our review to intensive family preservation programs,

which are based on the Homebuilders’ model,7”6”7 or are like 1-lomebuilders and share several distinct characteristics.’”4 In these programs, family workers maintain intensive contact with two to four families at a time and provide home-based, short-term

services, including case management, family counseling,

improve-ment of communication skills, and provision of other concrete

services, such as financial assistance and transportation. The goals of intensive FPS are: (1) to resolve the immediate crisis that puts

children at risk of removal from the home; (2) to improve family

functioning so that children can be maintained at home safely and

adequately; and (3) to prevent out-of-home placements.#{176}4.’7.’8 &J..

though many other models of family preservation and family

support services have been developed, we focused only on pro-grams that provided intensive services. Therefore, references de-scribing programs that provided primary preventive services to

families or other types of family support programs were excluded from the analysis.

Outcome Data Available in the Report

Review articles of family-based services or references providing

descriptions of programs were excluded.

A Comparison Group

To determine whether intensive FPS were more effective

than customary social services, a comparison group was judged important.

Sources of Data

Both published and unpublished evaluations of intensive fam-ily preservation programs were included. References published

between 1977 (when the Homebuilders’ program first published their findings)7 and 1993 were identified through a computerized

search of databases for all English-language publications listed in

MEDLINE, Education Resources Information Center (ERIC),

Psy-chlit, Sociofile, and Health Plan. Key phrases such as “family

preservation,” “family-based services,” and “child abuse,” were

used to identify potential references. Additional references were

identified from published bibliographies and from three

nation-ally available bibliographies (the Center for the Study of Soda!

Policy, the National Resource Center for Family Based Services, and the Child Welfare League). Unpublished references were

obtained by direct contact (mail or phone). In addition, the

Anno-tated Directory of Family-based Services, a national directory of

family-based services’0 maintained by the National Resource

Cen-ter for Family-based Services, provided information on 391

pro-grams. Letters were sent to 130 programs that described them-selves as family preservation programs and stated that evaluation

data were available. When results from a program were reported

in more than one document, data from the most recent document were used.

Data Collection

Data collected about each program included the program’s

location, model and goals, funding sources, ages of the children

served, and services provided to families (type, intensity, and

duration). Descriptive information about the evaluations included

sample size, design of the evaluation, use and selection of

com-parison groups, eligibility and exclusion criteria, outcomes

exam-ined, and timing of the assessment of the outcomes after

termina-lion of services.

To determine the methodological quality of the evaluations reviewed, we adapted criteria from Chalmers et al’9 and used a 15-item questionnaire to examine three critical methodological components like those used in a clinical trial. Five questions examined assignment of families to the treatment groups: (1) whether eligibifity criteria for participation by families in the

family preservation program were described; (2) whether a

stan-dardized assessment of imminent risk of placement was used; (3)

whether exclusions were specified; (4) whether appropriate tech-mques of assembling the treatment groups were applied; and (5) whether families were maintained within groups during the study

period (ie, no crossover). Six questions assessed the adequacy of the description of the intervention: whether the types, intensity, and duration of services received by the treatment and compari-son groups were documented. Finally, four questions assessed

outcomes: (1) whether specific criteria for placement were used;

(2) whether the need for placement was determined by observers

blinded to the group assignment; (3) whether all families were

accounted for in the analysis; and (4) whether outcomes other than

placement were measured. Each question in the scale was an-swered by either “yes” or “no,” resulting in a maximum additive score of 15 points. Individual questions were not weighted.

To allow comparisons across studies, an attempt was made to

stratify studies by total score. A total score of at least 11 (73%) of

15 was considered an acceptable study, 6 to 10 (40% to 67%) of 15

represented an adequate study, and 0 to 5 (<40%) of 15 repre-sented an unacceptable study. Each evaluation was scored inde-pendently by two of the authors, who were aware of the scoring

criteria for quality. Interrater reliabifity for this instrument had 91% agreement and a ic of 0.83, which showed excellent agree-ment.#{176}2’ When disagreements occurred over a response to a

spe-dfic question, a consensus was reached by the two raters. The scores presented are based on the consensus scores.

The characteristics of each program and program evaluation were described. Rates of out-of-home placement, the outcome most consistently reported, were compared. The relative risk (RR) of placement with 95% confidence intervals (CIs) was calculated

from the data presented in each study to determine whether there

(3)

were statistically significant differences between the treatment and comparison groups.

RESULTS

Of the 802 references initially identified from both

published and unpublished sources, 756 were

ex-cluded because they did not meet the first two eligi-biity criteria: 50% did not describe family

preserva-lion programs; 40% were review articles; and 10%

presented no outcome data in the reports. Of the 130

queries to programs listed in the Annotated

Direc-tory of Family-based Services, responses were

re-ceived from 59 programs (45%). Twenty-nine letters

(22%) were returned unopened, because the

pro-grams were no longer in existence or had relocated,

and 42 programs (32%) did not respond to our

re-quests for evaluative materials.

We identified 46 evaluations of programs, 36 of

which were excluded because they contained no

comparison groups. Of the 10 eligible studies, 5 were

randomized trials, and 5 were quasi-experimental in

design, using nonrandom assignment to groups.

Characteristics of the Programs

The 10 intensive family preservation programs3’ (Table 1) shared the primary goal of preventing

out-of-home placements of children who were abused or

ne-glected and who had been referred to states’ child

protection agencies. Six of the 10 programs used

the Homebuilders model (New York, Michigan,

Connecticut, California, New Jersey, and illinois); two

used home-based, family-centered models not

sped-fled as that of Homebuilder’s (Hennepin [HEN] and

Ramsey [RAM] Counties, Minnesota); one used a sodal

learning theory (Oregon); and one provided no

docu-mentation of the program model (Texas). Two

pro-grams (New York and Texas) were supported by both

public and private funds; five (Michigan, Connecticut, California, New Jersey, and illinois) stated that public

funding was used; and three (RAM, Oregon, and HEN)

did not report their sources of funding.

Eight programs provided services to children of all

ages; two targeted specific populations, such as

school-aged children (New York) or adolescents

(HEN). Programs provided a variety of services,

in-cluding family-focused therapy (100%), concrete

ser-vices (90%), behavior modification (60%), parenting

support (60%), and life skills (60%). The durations of

services ranged from 5 to 420 days but was not

described in 5 of 10 evaluations (Connecticut, HEN,

California, Iffinois, and Oregon). The intensity of

services was measured by the number of contacts

with a family during the treatment period (ranging

from 9 to 11 h/wk) and the number of families

assigned per worker (ranging from 2 to 10). Intensity (defined by either of these measures) was provided

in 8 of 10 studies. Two of the randomized trials

(illinois and Oregon) provided no information about

either the duration or intensity of services.

Characteristics of Evaluations

All 10 studies were conducted to evaluate program

effectiveness and examined outcomes such as

reduc-ing out-of-home placements of children into foster

care (Table 2). The sample sizes for the treatment

groups ranged greatly, from 24 to 817, and for the

comparison groups ranged from 12 to 396. Among

quasi-experimental studies, the comparison groups

were assembled from a geographically different area,

where FPS were not available (Texas) or from

over-flow child protection cases (New York, Connecticut,

TABLE 1. Characteristics of Programs Location and Reference Age of

Children, y

Types of Services Duration of

Services, d

Intensity of Services

No. of No. of

Contacts, Families/

h/wk Worker,

Range or Average

Quasi-experimental studies

Texas-Dennis-Small and Washburn

0-18 Family therapy, parenting, concrete services

90-420 ... 7-10

Bronx, New York-Mitchell et

aP-6-11 Family therapy, parenting, life skills

5-110 9.0 2

Michigan-Berquist et al24 Connecticut-Wheeler et aP’

0-5

0-18

Family therapy, parenting, housing, concrete services

Family therapy, life skills,

concrete services

30

...

.. .

9.1

2 2-5

Hennepin County,

Minnesota-Schwartz et al’6

12-18 Family therapy ... .. . 2

Randomized trials California-Yuan et al”

New Jersey-Feldman

Illinois-Schuerman et al’s

Oregon-Szykula and Fleischman’#{176}

0-18

Mean, 7

0-18

Mean, 13 0-12

0-18

Family therapy, life skills,

concrete services

Family therapy, life skills,

concrete services, advocacy Family therapy, parenting,

advocacy, concrete services Family therapy, concrete services

...

7-63

.. .

. ..

...

11.0

...

2-4

...

Ramsey County, Minnesota-Lyle and Nelson’1

0-18 Mean, 6

Family therapy, concrete services, parenting

313 ... 8-10

*Information was not available from printed reports.

(4)

TABLE 2. Characteristics of Evaluations

Location and Reference Sample Eligibility Criteria Timing of Outcomes Assessed

Outcome

Assessment

(mo) After Treatment, Comparison,

n n Source _Services

End

Quasi-experimental studies

Texas-Dennis-Small and 87 85 Risk of placement, parent . .. Placement, costs, rates of

Washburn” Geographically willing, no mental illness recurrent abuse

Bronx, New York-Mitchell 43 12 Risk of placement 12 Placement, family

et al” Overflow child

protection cases

functioning, satisfaction, service use

Michigan-Berquist et al’4 225 225

Foster care

Risk of placement 12 Placement, costs, family satisfaction, service use

Connecticut-Wheeler et al’s

Hennepin County, Minnesota-Schwartz et al’6

195

55

97

Overflow child protection cases

58

Overflow child

protection cases

Risk of placement

Risk of placement, age >12 y, alternative available

12

16

Placement, family

functioning

Placement, service use

Randomized trials

California-Yuan et al” 143 150

Child protection cases

Risk of placement, no

mental illness, no sexual

abuse

6 Placement, costs, family

functioning, service use

New Jersey-Feldman 117 97

Child protection

cases

Risk of placement, no prior placement, parent

willing, no homelessness, other services tried

12 Placement, family

functioning, satisfaction,

service use

Illinois-Schuerman et al’s 817 396

Child protection cases

Risk of placement 18 Placement, family

functioning, satisfaction,

rates of recurrent abuse

Oregon-Szykula and 24 24 Risk of placement, age . .. Placement

Fleischman’#{176} Child protection

cases

between 3-12 y

Ramsey County, Minnesota- 34 40 Risk of placement, parent .. . Placement, costs, family

Lyle and Nelson” Child protection

cases

willing, no mental

retardation, no longterm

placement

functioning

*Information was not available from printed reports.

and HEN). In one study (Michigan), children already in foster care represented the comparison group. In

all randomized studies, comparison groups were

as-sembled from other child protection cases.

Calcula-lions of sample sizes for power determinations were

not noted in any study.

Eligibility for families to receive FF5 varied across programs. All stated that the risk of out-of’home place-ment was a prerequisite to receiving services. Several programs cited other criteria. Three programs (Texas,

New Jersey, and RAM) required parental willingness

to partidpate in the program. Two (HEN and Oregon)

limited partidpation to children of certain ages.

Exdu-sions induded mental illness or mental retardation of

either child or parent (Texas, California, and RAM), homelessness (New Jersey), the likelihood of long-term placement (RAM), and if no other services were avail-able or effective (FIEN and New Jersey).

Outcomes were assessed at different times among

studies, with 12 months after the termination of

ser-vices the most frequently documented (New York,

Michigan, Connecticut, and New Jersey). The

short-est interval between termination of services and

as-sessment of the outcome was 6 months (California);

the longest was 18 months (Illinois). Three

evalua-lions did not report the timing of the assessments of

the outcomes (Texas, Oregon, and RAM).

The frequencies of out-of-home placements were

reported in all evaluations. Family functioning (6 of

10), family satisfaction (4 of 10), specific social

ser-vices used by families, such as homemaker services

or substance abuse counseling (5 of 10), and rates of

recurrent abuse or neglect (2 of 10) were other

out-comes measured. Cost analysis and reported cost

savings were also documented in 4 of 10 studies

(Texas, Michigan, California, and RAM). Only one

study used out-of-home placements as the single

measured outcome (Oregon).

Methodological Review

Based on the 15-point methodological

question-naire, only two evaluations (New Jersey and RAM)

scored at least 11 (73%) of 15 for methodological quality (Table 3). Four studies (New York, Michigan,

HEN, and California) were determined to be

ade-quate, with scores between 6 and 10 (40% to 67%) of

15. Four studies (Texas, Connecticut, Illinois, and

Oregon) scored 0 to 5 (<40%) of 15 and were judged

unacceptable methodologically. These included two

of the randomized trials (illinois and Oregon).

When examining the first methodological domain,

how children were assigned to the treatment groups, we found that in seven studies, the actual criteria for

eligibility were described (Texas, Michigan, HEN,

California, New Jersey, Oregon, and RAM). A

stan-dardized assessment, using a rating scale or

(5)

TABLE 3. Methodological Criteria

Criteria Quasi-ex penmen tal Studies Ran domized Trials

TX NY MI CF HEN* CA NJ IL OR RAMS

Assignment to groups

Eligibility criteria described Standardized assessment of

+ -+ -+ -+ -+ + -+ -+ -imminent risk of placement

Exclusions described + - - - - + + - - +

Assignment to groups:

Similarity of comparison demonstrated

- - - -

-Randomization blinded - + + - +

Groups maintained (no crossover)

Intervention

- - - + - - +

Family preservation services Types Intensity Duration + -+ + + + + + + + + + -+ + + + + + + + -+ -+ + + +

Customary social services

Types Intensity Duration -+ -+ + + -+ + + Outcomes

Criteria for placement defined Placement determined by

observers blinded to treatment

-+ -+ -+ -+ --

-All families accounted for in - + + - + - + + - +

analysis

Outcomes other than placement Total + 5 + 6 + 6 + 5 + 6 + 10 + 11 + 3 -3 + 12

*HEN, Hennepin County, Minnesota; RAM, Ramsey County, Minnesota.

+, present; -, absent.

tested questionnaire that would allow a uniform

ap-proach to referring cases for services, however, was

used in only one randomized trial (New Jersey).

Exdusion criteria, which provide additional

infor-mation about the population under investigation,

were described in one of five quasi-experimental studies (Texas) and in three of the randomized stud-ies (California, New Jersey, and RAM).

The five quasi-experimental studies provided no

documentation of the similarity of the two groups

with regard to sododemographic characteristics or

severity of family problems, although the population

from which the comparison group was assembled

was known. Among the studies characterized as

ran-domized trials, the process of randomization was

done blindly in only three evaluations (New Jersey,

illinois, and RAM). Two studies (Illinois and RAM)

assigned families to groups by using a

computer-generated process, presumably randomly, and the

other (New Jersey) used an outside clinical team to assign families to one group or the other, although no clear assignment procedure was described in the

evaluation. There was no assurance in any of

the quasi-experimental studies and in only two of the

randomized trials (New Jersey and RAM) that the

integrity of the groups was maintained throughout the study period (ie, no crossover occurred).

The second methodological domain concerned the

description of the interventions. Each evaluation

compared FPS with usual or customary child welfare

services to determine whether there were differences in specific outcomes between the groups. Because the

groups were receiving different services (family

preservation or customary services), both qualitative

and quantitative descriptions of the services were

important.

Descriptions of FPS were provided in four of five

quasi-experimental studies and four of five

random-ized trials, but customary child weifare services were

described in only three studies (HEN, California, and

RAM). For the group receiving FPS, four of five

quasi-experimental studies (New York, Michigan,

Connecticut, and HEN) and three randomized trials

(California, New Jersey, and RAM) described the

intensity of services as the number of contacts (hours

per week; 30%) or the number of families assigned to

each worker (70%). Durations of services in days

were described in all quasi-experimental studies and

in four of five randomized trials. Customary sodal

services were less well defined. No

quasi-experimen-tal study documented the intensity or duration of

services, and only two randomized trials (California

and RAM) documented both the intensity and

dura-tion of customary social services. Only one study

(California) described the type, intensity, and dura-tion of the services received by both the treatment and comparison groups.

The third methodological domain focused on the

ascertainment of outcomes. Prevention of

out-of-home placements was consistently used to determine the effectiveness of each program. Criteria for defin-ing placement, however, was different in each study.

For example, placement with other family members

or friends was considered differently from one study

to another. Some investigators defined placement as

any out-of-home placement of any length of time

(Connecticut, California, and New Jersey), whereas

(6)

others used temporal criteria (eg, at least two weeks of placement [New York]).

Our review showed that no specified criteria were

stated in any study for determining when placement of a child was needed. For children receiving FPS, the

decision to place a child in an out-of-home setting

was made by the family preservation case workers,

who not only were aware of the child’s group

assign-ment, but who, even in the same program, likely had

different thresholds for the need to place a child.

Studies did not routinely account for losses to

follow-up as a component of the research design. In

four studies (Texas, Connecticut, California, and

Or-egon), it was impossible to determine whether the

same families constituting the initial study

popula-tion were those who were followed throughout the

study period. Three studies (Michigan, Illinois, and

RAM) reported the number of families included in

the evaluation from the potential pool of families

referred for services.

Outcomes other than placement were used in all

but one study. Family functioning was assessed in

six studies, most often with the Family and Child

Well-being Scales,32’ which assess levels of family

functioning in domains such as parental problem

solving, parental support systems, stability of hous-ing, nurturing of the children, and use of parental

discipline. In addition, 40% of the evaluations

exam-med family satisfaction with FPS, and 40% assessed

cost savings. Despite the use of these additional

out-comes, there was limited attention paid to the

chil-dren who were being served. Repeated episodes of

abuse or neglect were assessed in only two studies,

and no study directly evaluated whether there were

changes in the child’s functioning (eg, development or school functioning).

Success or failure of the intervention was

deter-mined in each study by testing whether there was a

difference between treatment and comparison

groups in rates of out-of-home placements (Table 4).

In the five quasi-experimental studies, rates of

place-ment ranged from 21% to 56% in the treatment

groups and 25% to 59% in the comparison groups.

The RR of placement was statistically significantly different in favor of the family preservation interven-tion in only one quasi-experimental study (RR, 0.67;

95% CI, 0.50-0.90), which used children already

placed in foster care as the control group (Michigan).

The five randomized trials also showed a wide range

of placement rates from 24% to 43% in the treatment

groups and 20% to 57% in the comparison groups.

The RR of placement was again statistically

signifi-cant in favor of family preservation in only one study

(RR, 0.75; 95% CI, 0.57-0.99). The rates of placement

in this study, however, were the highest reported,

43% and 57% in the treatment and comparison

groups, respectively.

DISCUSSION

Our review of the evaluations of FPS provides

clear evidence of the methodological shortcomings of

these studies and the conflicting results with regard

to rates of out-of-home placements. Only 2 of the 10

studies reviewed were rated methodologically

ac-ceptable; 4 were rated adequate; and 4 were rated

unacceptable; of those rated as methodologically

Un-acceptable, 2 were randomized trials. FPS provided no benefit with regard to placement in 8 of 10 stud-ies, whereas in 2 studies, the rates of out-of-home

placements were reduced after families received FPS.

When only the 2 methodologically acceptable studies

were considered, one (New Jersey) showed

signifi-cant differences in rates of out-of-home placements

between the treatment and comparison groups, but

overall rates of placement were the highest reported for both groups.

Our methodological analysis of the 10 studies

il-lustrates the need to use objective criteria to assess

the quality and clinical usefulness of study results. We realize the difficulty in evaluating human service

interventions; however, the importance of

well-constructed studies cannot be overstated. Both

Chalmers et al’9 and Gerbarg and Horwitz have

underscored the use of strict criteria in

methodolog-ical review and meta-analyses. In using the 15-point scale to assess methodological quality, we set criteria that are minimally necessary for scientific validity,

namely, proper assignment to groups, a

standard-ized intervention, and appropriate selection and

measurement of outcomes.

Methodological inadequacies were observed in

each of the three domains reviewed (assignment to

groups, description of the interventions, and

ascer-tainment of outcomes). Assignment of families to

treatment groups was inconsistently done across the

10 studies reviewed. Assessment of the risk of

out-of-home placement was done using a standardized

risk assessment tool in only I study. The lack of a

standardized assessment of risk is a particularly im-portant problem, because it can introduce selection

TABLE 4. Rates of Placement

Program Location Placement Rate, (%) Relative Risk (95% Confidence Interval)

Treatment Comparison

Texas 18/87 (21) 23/85 (27) 0.76 (0.45-1.31)

New York 11/43 (25) 3/12 (25) 1.02 (0.34-3.09)

Michigan 53/225 (24) 79/225 (35) 0.67 (0.50-0.90)

Connecticut 60/195 (31) 24/97 (25) 1.24 (0.83-1.87)

Hennepin County, Minnesota 31 /55 (56) 34/58 (59) 0.96 (0.70-1.32)

California 36/143 (25) 30/150 (20) 1.26 (0.82-1.93)

New Jersey 50/117 (43) 55/97 (57) 0.75 (0.57-0.99)

Illinois 237/817 (29) 103/396 (26) 1.12 (0.92-1.36)

Oregon 8/24 (33) 10/24 (42) 0.80 (0.38-1.67)

Ramsey County, Minnesota 8/34 (24) 18/40 (45) 0.52(026-1.05)

(7)

bias before the initiation of either FPS or customary

services, and it may affect conclusions about the

effectiveness of H’S compared with customary social

services markedly. Providing clear definitions of el-igibifity to receive H’S and predicting who will

ben-efit most from their use are issues that have been

raised by other investigators1’5 but clearly have not been addressed in previous evaluations.

In the studies reviewed, the predse nature of both H’S and customary sodal services was difficult either to describe or to quantify, because limited descriptions of these services were provided. In addition, the

heter-ogeneous and individualized nature of the services

delivered makes comparisons across studies difficult.

As an outcome, placement has been described by

other researchers as “program-based,” one that may

not be an adequate surrogate for determining the

positive effects of the intervention of family

preser-vation.’ Using placement as an outcome has other

problems as well. First, determining when placement

is necessary, because the threshold to place a child is

variable among social service workers, and no

mea-surable standards for dedding when to place a child

were provided in any study. Second, because

work-ers are unblinded and are readily aware of the

treat-ment being offered to children and their famifies,

placement may be discouraged for children receiving FPS. Finally, the very definition of placement varies across studies, making comparisons difficult.

Despite the methodological limitations and the

ab-sence of benefit with regard to placement noted in the majority of studies, FPS are considered by many child welfare agendes and family advocates to be a success-ful approach to intervening in families in which chil-dren have been abused or neglected. As has been

sug-gested by other investigators, measuring other

outcomes, such as child development, maternal-child interactions, and repeated episodes of maltreatment or unintentional injury, might demonstrate more condu-sively that children and families benefit more from FPS

than from foster care. Given increasing mandates

for states to provide FPS to families, while ensuring the safety of children, the challenge becomes identifying who will benefit from specific interventions.

Our findings are consistent with those of other

in-vestigators who have documented the limitations of

studies done to date.’27#{176} Our findings are also con-sistent with a 1995 General Accounting Office report

that summarizes state and federal family preservation

efforts before the Omnibus Budget Reconciliation Act of 1993.’ Based on a survey mailed to family

preser-vation and family support programs, the General

Ac-counting Office reported that most states relied on in-ternal program reviews to assess program effectiveness and that less than half of the programs used measur-able outcomes in their evaluations.

Our review raises concerns about research efforts that have been designed to answer administrative and programmatic questions rather than hypotheses

about how such interventions affect families and

their children. To advise public policy makers best,

program evaluations must be held to the standards

of scientific inquiry used in determining the effec-tiveness of any treatment regimen.4’ Evaluations that

lack a sound theoretical base and methodological design are likely to produce conflicting results that

become subject to discretionary interpretation. As

Gelles42 stated, given the lack of research evidence, it is unclear whether FPS are “penicillin” or “poison” for families receiving them.

In future evaluations of FPS, three methodological areas will need specific attention: (1) a standardized

assessment of imminent risk; (2) improved

descrip-tions of the interventions offered, not only to families receiving FPS, but also to those receiving customary sodal services; and (3) clear criteria for placement

and blinded determination of the need for

place-ment. In addition, it will be important to expand the

types of outcomes that are assessed. Although

out-of-home placements certainly should be considered, more attention should be directed toward

determin-ing whether the child’s overall functioning has

im-proved because of the services received. Has abuse

or neglect reoccurred? Have the child’s growth and

development been optimized? Has the child’s

cogni-tive and social development shown changes for the

better? These and other outcomes will need to be

addressed to obtain a clearer understanding of the

benefits and limitations of family preservation.

Implications

Family preservation is not a panacea, yet

policymak-ers have encouraged its widespread adoption. The

movement to preserve families has received bipartisan support because it saves money and values the family unit. As many supporters of family preservation point out, it is difficult to argue against family preservation.

Conservative sodal theorists support it because it is

consistent with supporting the nudear family and

lim-its state interference in the lives of families. Liberals support it because it supplies disadvantaged families

with needed resources. Any policy, however, when it

becomes categorically applied, is bound to harm some

families and children, because reliance on a single

strat-egy cannot address all of the needs of children who

have been maltreated.

The child weifare crisis in the United States must be addressed by multiple strategies to develop

opti-mal, comprehensive services for children, because

there are certainly limitations to every strategy used to assist children and families. Alternatives to family

preservation, such as permanency planning

(adop-tion) and foster care, also must be reexamined in the context of child safety and child well-being.

The ethos that children are best cared for by their

parents, and that a breakdown in parental caring

oc-curs when sodal stressors become too great, is central

to the acceptance of FPS by child weifare agendes.

Maltreating parents are viewed as needy and under

stress, rather than as evil; therefore, therapy is needed

more that punishment. Yet, when children have been

abused or neglected, out-of-home placement may be in

the child’s best interest. As FPS become even more

dosely incorporated into child protection systems, the

net effect may be that placement of children, even

when necessary, is discouraged. Applying family

pres-ervation to every family, as a matter of policy, may actually be placing children at risk.2

(8)

During a senate hearing for child welfare and

pre-ventive services in June 1992, Senator Moynihan

questioned the “information base and conceptual

base” that are used to approach child welfare policy and FPS, stating, “We want data.” Unfortunately, policy in this area has evolved more on faith than on fact. As specific federal efforts begin to determine

whether FF5 are effective at reducing placement of

children and helping families function better,’2”3

studies should include attention to the methodolog-ical issues addressed in this review and to outcomes that assess how FPS affect the lives of children.

ACKNOWLEDGMENTS

Dr Heneghan was supported by the Robert Wood Johnson Clinical Scholars Program. Dr Leventhal was supported in part by

behavioral pediatrics training grant 9087 from the Maternal and

Child Health Bureau. Dr Horwitz was supported in part by grant

R01-MH48456 from the National Institute of Mental Health.

REFERENCES

I. Rossi P. Evaluating Family Preservation Programs: A Report to the Edna McConnell Clark Foundation. Amherst, MA: Social and Demographics

Research Institute; 1991:4-78

2. Fraser MW, Pecora PJ, Haapala DA. Families in Crisis: The Impact of Intensive Family Preservation Services. Hawthorne, NY: Aldine DeGruyter; 1991:1-16

3. Nelson KE, Landsman MJ. Alternative Models of Family Preservation:

Family-based Services in Context. Springfield, IL: Charles C Thomas;

1992:8-20

4. American Humane Association. Highlights of Official Child Neglect and Abuse Reporting: 1986. Denver, CO: American Humane Association;

1986:7

5. McCurdy K, Daro D. Current Trends in Child Abuse Reporting and

Fatalities: The Results of the 1993 Annual Fifty State Survey. Chicago:

National Committee to Prevent Child Abuse; 1994:1-21

6. Select Committee on Children, Youth and Families, US House of Rep-resentatives. No Place to Call Home: Discarded Children in America. Wash-ington, DC: Government Printing Office; 1990. Publication 25-744

7. KinneyJM, Madsen B, Flemming T, Haapala D. Homebuilders: keeping

families together. I Consult Clin Psychol. 1977;45:667-673

8. Edna McConnell Clark Foundation. Keeping Families Together: The Case for Family Preservation. New York: Edna McConnell Clark Foundation;

1985

9. Pub L No. 96-272, Title I : Foster Care and Adoption Assistance, Part E, 471, 15; 1980:501-503

10. University of Iowa School of Social Work. Annotated Directory of Selected Family Based Services Programs. 7th ed. Iowa City: University of Iowa

School of Social Work, National Resource Center on Family Based

Services; 1991;1-391

11. Center for the Study of Social Policy. State Family Preservation Services Program Information Sheet. Washington, DC: Center for the Study of Social Policy; 1992:1-4

12. Pub L No. 103-66, Part IB: Child Welfare Services, Foster Care, and

Adoption Assistance, Subpart 2, Family Preservation and Support

Ser-vices, §430-435; 1993:649-655

13. General Accounting Office. Opportunities to Further Enhance Family Pres-ervation and Support Activities. Washington, DC: General Accounting Office; 1995:1-34. Publication GAO/HEHS-95-112

14. Nelson K, Emlen A, Landsman M, et al. Factors Contributing to Success and Failure in Family Based Welfare Services. Iowa City: University of Iowa

School of Social Work, National Resource Center on Family Based

Services; 1988:1-146

15. Pecora PJ, Fraser MW, Haapala DA. Client outcomes and issues for program design. In: Wells K, Biegel D, eds. Family Preservation Services: Research and Evaluation. Newbury Park, CA: Sage Publications; 1991:3-5

16. Haapala DA, Kinney JM. Homebuilders’ approach to the training of in-home therapists. In: Maybanks 5, Bryce M, eds. Home-based Services for Children and Families. Springfield, IL: Charles C Thomas; 1979:

248-259

17. Haapala DA, Kinney JM. Avoiding out-of-home placement of high-risk

status offenders through the use of intensive home-based family

pres-ervation services. Criminal Justice Behav. 1988;15:334-348

18. WhittakerJK, Tracy FM. Family preservation services and education for

social work practice: stimulus and response. In: Whittaker JK, Kinney J,

Tracy EM, Booth C, eds. Improving Practice Technology for Work with High-risk Families: Lessons from the Homebuilders’ Social Work Education Project. Seattle, WA: University of Washington School of Social Work,

Center for Social Welfare Research; 1988:9-18

19. Chalmers TC, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Controlled Clin Trials. 1981;2:31-49 20. Cohen 1. A coefficient of agreement for nominal scales. Educ Psychol

Measures. 1960;20:37-46

21. Fleiss JL. StatisticalMethods for Rates and Proportions. 2nd ad. New York: John Wiley & Sons, mc; 1981:217-225

22. Dennis-Small L, Washburn K. Family-Centered Home-based Intervention

Project for Protective Services Clients, Final Report. Austin, TX: Texas

Department of Human Services; 1986; Part 1:1-16, Part 2:1-8 23. Mitchell C, Tovar P. Knitzer J. The Bronx Homebuilders Program: An

Evaluation of the First 45 Families. New York: Bank Street College of

Education; 1989:2-20

24. Berquist C, Szwejda D, Pope G. Evaluation of Michigan’s Families First Program: Summary Report. Lansing, MI: Michigan Department of Social

Services, University Associates; 1993:1-21

25. Wheeler CE, Reuter G, Struckman-Johnson D, Yuan YY. Evaluation of State of Connecticut Intensive Family Preservation Services, Phase V Annual Report. Sacramento, CA: Walter R. McDonald & Associates, mc; 1992:

chaps 1-4

26. Schwartz IM, Au Claire P, Harris L. Family Preservation as an Alternative to the Out-of-Home Placement of Seriously Emotionally Disturbed Adolescents: The Hennepin County Experience. Ann Arbor, MI: Center for the Study of Youth Policy, University of Michigan School of Social Work; 1990:1-29

27. Yuan YY, McDonald WR, Wheeler CE, Struckman-Johnson D, Rivest M.

Evaluation ofAB 1562 In-home Care Demonstration Projects. Volume I: Final Report. Sacramento, CA: Walter R. McDonald & Associates, mc; 1990;I:

chaps 1-7

28. Feldman LH. Assessing the Effectiveness of Family Preservation Services in New Jersey Within an Ecological Context. Trenton, NJ: Department of Human Services, New Jersey Division of Youth Services; 1991:1-140

29. Schuerman JR, Rzepnicki U, Littell JH. Evaluation of the Illinois Family First Placement Prevention Program: Progress Report, June 1992. Chicago:

Chapin Hall for Children, University of Chicago; 1992:1-111

30. Szykula SA, Fleischman MJ. Reducing out-of-home placements of

abused children: two controlled field studies. Child Abuse Negl. 19859:

277-283

31. Lyle CG, Nelson, J. Home-based vs. Traditional Child Protection Services: A Study of the Home-based Services Demonstration Project in the Ramsey County Community Human Services Department. Minneapolis: Hubert H Humphrey Institute, University of Minnesota; 1983:1-18

32. Magura 5,Moses BS. Outcome Measures for Child Welfare Services: Theory and Applications. Washington DC: Child Welfare League of America, Inc; 1986:83-154

33. Magura 5, Moses BS,Jones MA. Assessing Risk and Measuring Changes in Families: The Family Risk Scales. Washington, DC: Child Welfare League of America, mc; 1987:15-43

34. Gerbarg ZB, Horwitz RI. Resolving conflicting clinical trials: guidelines

for meta-analysis. I Clin Epidemiol. 1988;41:503-509

35. Kirk RS, Reed K, Lin A. Evaluability Assessment ofNorth Carolina’s Family Preservation Services Program, Final Report. Chapel Hill, NC: Human Services Research and Design Laboratory, School of Social Work, Uni-versity of North Carolina; 1993:7-55

36. Weiss HB, Jacobs FH. Evaluating Family Programs. Hawthorne, NY: Aldine de Gruyter; 1988:73-94

37. WaId MS. CarlsmithJM, Leiderman PH. Protecting Abused and Neglected Children. Stanford, CA: Stanford University Press; 1988:181-200

38. Frankel H. Family-centered home-based services in child protection: a review of the literature. Soc Service Rev. 1988;62:137-157

39. Magura S. Are services to prevent foster care effective? Child Youth

Services Rev. 19813:193-212

40. Stein TJ. Projects to prevent out-of-home placement. Child Youth Services Rev. 1985;7:109-121

41. Campbell DT. Problems for the experimenting society in the interface

between evaluation and service providers. In: Kagan SL, Powell DR,

Weissbourd B, Zigler EF, eds. America’s Family Support Programs. New

Haven, CT: Yale University Press; 1987:345-351

42. Gelles RJ. Family reunification/family preservation: are children really being protected? JInterpersonal Violence. 1993;8:556-562

43. Reid WJ, Kagan RM, Schlosberg S. Prevention of placement: critical factors in program success. Child Welfare. 1988;67:25-36

44. Child Welfare and Preventive Services, Senate Committee on Finance

Hearing Y4.F49:S.Hrg. 102-1002; 1992 at Viet Nam:AAP Sponsored on September 1, 2020

(9)

1996;97;535

Pediatrics

Amy M. Heneghan, Sarah M. Horwitz and John M. Leventhal