Validation and Predictive Ability of the
North Carolina Family Assessment
Scale for the Intensive In-Home
Visitation Program in Kentucky
RAMONA F. STONE
1University of Kentucky Lexington, KY, USA
GERARD M. BARBER
University of LouisvilleLouisville, KY, USA
SARAH HENDRIX
Union CollegeBarbourville, KY, USA
Abstract
The Community Collaboration for Children (CCC) program is a complex initiative implemented in Kentucky by a network of state and non-governmental agencies that provide intensive in-home services to families at risk for child abuse and neglect. The primary focus of this program is to maintain children who are at risk of being removed from their family, in their own homes while supporting and building family strengths in areas such as safety, stability, and interaction skills. The program’s evaluation research design is longitudinal, data was collected quarterly by the services providers, social and child workers. This paper focuses on the validation of the North Carolina Family Assessment Scale (NCFAS), using the intake data collected from 1959 families who participated in the CCC intensive in-home services during July 1, 2006 through December 31, 2009. NCFAS is a practice tool utilized by service providers to assess families on five domains: environment, child wellbeing, family interaction, family safety and family capability as related to child wellbeing. The factors extracted using two approaches - general factor analysis and a congeneric, single-factor analysis- were used to test the predictive ability of each subscale using logic regression analyses, while controlling for the intensity of in-home visitation services. Both factor analysis approaches yielded valid and reliable results. Of the five NCFAS domains, the family interaction was the strongest predictor of case outcome, assuming that families are provided with 11 to 20 hours of services.
Keywords: Family functioning; Family safety; Child wellbeing; Family preservation; Family permanency; In-Home visitation; Scale validation; Reliability.
Introduction
In the United States, state government agencies have the mission to deliver quality services that enhance the health, safety, and wellbeing of their children. To serve this mission, family preservation programs are implemented in all states and include primary, secondary and tertiary prevention programs funded with federal, state and local support. Family preservation programs are based on the Homebuilders model developed in Tacoma, Washington during the 1970s.
1 Postal Address: Department of Health Behavior, College of Public Health, University of Kentucky, Bowman Hall
In Kentucky, the first family preservation program was implemented in 1985 as a pilot, with private foundation funding; in 1990 the state legislature passed the Family Preservation Act (FPA) and allocated funding for 47 out of the 120 counties to establish family preservation programs. Services offered under the FPA were intended to be a short-term, intensive, and acute intervention services to prevent removal of children from their homes (Westat, 2002). Over time the family preservation program services grew, as other sources of funding became available. One such program is the federally funded community-based initiative to prevent and reduce the child abuse and neglect in Kentucky, called the Community Collaboration for Children (CCC). CCC is primarily focused on secondary prevention services that were provided by contracted agencies across the state. Secondary prevention programs are designed for the families identified to be at risk for child abuse and neglect, but not yet substantiated, while primary preventions focus on the population at large and tertiary ones on the families with substantiated abuse and neglect.
This article reports on data collected for the evaluation of the CCC’s intensive In-Home Services herein referred as IHS, a secondary prevention program, utilized when children are at risk to be removed from their homes and placed with relatives, in kinship care or in out-of-home care. Thus, the permanency goal for the families enrolled in this program is to maintain the children with their parents. The IHS program is open to families who self-refer or families may be referred by a variety of community agencies including churches; however, about 80% were referred by the Kentucky Department of Community Based Services (DCBS) within the Cabinet for Health and Family Services (CHFS). Families referred by the DCBS are expected to participate in order to avoid removal of their children from the home. Once a referral was made, regardless of the source, a home visit is scheduled within five working days. The focus is on a comprehensive treatment plan intended to address the family’s practical and material difficulties, together with their behavioral problems or mental health needs. The IHS program itself entails a series of home visits during which the family and the local service providers work together toward identifying and addressing the family needs and reach the goal to maintain the children in the home. The timing of these home visits is based upon the needs of the families and at the discretion of the program provider.
The IHS services seek to develop, support, and empower the family by teaching positive child development practices and problem solving skills, as well as by assisting parents and coordinating available community resources. Services combine skill-based intervention with maximum flexibility so that they were available to families according to their unique needs.
Service providers teach the families how to live together safely, while addressing their immediate basic needs, refer families for community resources and counseling programs as needed, and may also include parent education intervention among their services. Parent education, when available, follows an established, nationally recognized, research-based curriculum. The qualifications of the service providers include 30 or more college hours in a human services area or a completed high school education with one year of experience providing similar services. They are expected to advocate for the best interest of the child, to develop specific goals with the family, and to guide them in locating more specialized resources as needed.
The primary focus of home visits is to provide individualized parent education through the use of mentoring and coaching techniques. Topics include child development, age appropriate behavior, communication skills building, mutual trust, and increasing self-esteem and confidence in parenting. In working with the family over time, the service provider has the opportunity to observe parents’ and children’s behavior in the home, to understand the underlying issues and to help them to successfully maintain the children in the home.
Background
by the Title II of the Child Abuse Prevention and Treatment Act Amendments of 1996 and reauthorized in 2010. The CCC program in Kentucky went through four phases: a startup period (Phase I) when decisions regarding the types of programs to be funded with CCC funds were made by the local Area Development Districts (ADD). The program evaluation design during this phase was qualitative (Stone and Barber, 2008). After the Adoption and Safe Families Act (ASFA) was signed into law in 1997, the CHFS stipulated what types of programs or services would be provided to families to help the state achieve the federally mandated outcomes required by the new policy (Phase II) and to ensure that the agency made reasonable efforts to prevent unnecessary out of home child placement. As a result, although the programs funded within the CCC network during Phase II continued to vary greatly across the state, the better defined outcomes and the more rigorous program expectations lead to the redesign of the CCC program evaluation. Prior to 2004, all CCC programs reported client information in an aggregate format and the service definitions varied greatly from one agency to another. These variations made it difficult to compare programs across the state and to measure changes in individual family outcomes (Stone & Barber, 2008).
During the contract years of 2004-2006 (Phase III), the CHFS and the Service Regional Administrators (SRA) made even stronger recommendations about the services needed to meet the ASFA mandatory goals. More emphasis was placed on family team meetings, intensive in-home services (secondary prevention), and supervised visitations (tertiary prevention). The intent was to work with fewer families and bring about specific changes that were consistent with CHFS’ outcome priorities as mandated by ASFA. By funding a limited, but more focused number of services, and by stressing the importance of comparing program performance measures and of child and family outcomes, the CCC evaluation developed comparative performance measures and implemented systematic digital data collection processes. The evaluation continued to rely on qualitative data, but it added a quantitative component. Standardized reporting measures for monitoring program outputs, such as service provision (e.g., number of home visits, number of hours of home visits completed), family and child outcomes (e.g., safety, wellbeing, environment, parental capabilities, family interactions), and children’s placement or permanency status were also collected.
The data collection instruments were further improved during Phase IV for the 2006-2010 contracts. Specifically, the data collection burden was reduced by eliminating fields that were confusing or yielded a large number of cases with missing information. During Phase IV, more focused and more accurate information on CCC program performance and child and family outcomes were collected. Identifying information was collected so that the CCC data could be matched with the child welfare administrative records from The Worker Information SysTem (TWIST); new confidentiality and security procedures for data storage and reporting were set and approved by the Institutional Review Boards (IRB) at the CHFS and at the University of Louisville.
The purpose of the IHS research evaluation was to determine whether the services were provided to the intended population, in the intended quantity (number of home visits) and with the intended intensity (number of hours per visit), and to evaluate the outcomes of the program. The evaluation plan was also developed in conjunction with the CHFS technical assistance and with direct feedback from service providers.
This paper stems from the quantitative research evaluation of the intensive in-home services provided in Kentucky by a network of state and private non-profit agencies. It focuses primarily on the validation of the North Carolina Family Assessment Scale (NCFAS) as a family functioning assessment practice tool
Methods and data
Methods
providers submitted a baseline assessment for each family completed at the time of intake into the program and a closure assessment for all cases that were closed by the end of the study period. A case was noted as closed when the case worker’s assessment indicated that the child or children were safe of abuse and neglect (successful closure), or when the assessment indicated that the child needed to be removed and placed in out-of-home care (unsuccessful closure). Each service provider collected data with a standardized assessment of family functioning for program evaluation. The standardized instrument utilized is known as the North Carolina Family Assessment Scale (NCFAS).
The NCFAS validation was tested with the intake data using two factor analysis methods: the general factor model and, its special case, the single-factor model known as the congeneric factor analysis. The factors extracted with the two approaches were saved as regression variables and were used in the bivariate and multivariate analyses.
Data collection
The data collection forms included identifiers, such as names and social security numbers, necessary to link the CCC data overtime and to match it with the TWIST CHFS administrative child welfare data. However, once the data was matched and unduplicated, the identifiers were deleted from the dataset that was used for the validation and predictive analyses. The data was collected over 14 quarters from the service providers who completed the forms for each family. The intake form was completed immediately after the first home visit. At the end of each quarter, for as long as the family was active in the program, the service provider continued to submit an assessment form, until the case was closed and thus, a closure form was submitted instead. In rare cases, families were referred to the program for more services. On average, the time span between case intake and case closure was about six-weeks. The service providers reported that the time spent on completing a quarterly reporting family assessment form was about 30 minutes.
Data Items
The assessment forms had three main sections: 1) Socio-demographic characteristics; 2) Services provided, including the number of visits and number of hours provided while the family was active in the program, and 3) Family functioning assessment measured by the North Carolina Family Assessment Scales (NCFAS). The demographic section collected data about each family member, and included questions about physical and mental disability. The services provided to families over multiple quarters were summated with a focus on changes in the family functioning assessment measures.
The outcome of primary interest was the family’s permanency status at case closure which was measured with a dichotomous variable. Code “1” was assigned to closed cases successful in maintaining their children in their home and a “0” code was assigned to cases that were either closed due to the children being removed from the family, or due to the inability to provide services (e.g. parents’ lack of cooperation, lost contact, moved out of the area) due to the family dropping out of the program. In some cases, the children had different permanency statuses; for instance, children ages 16 or older could have requested consideration for emancipation, which does not assume forceful removal from the family. Thus, if any of the children were maintained in the family, the case was coded as successful.
One of the reasons this scale was chosen, besides its proven validity and reliability with this population, is that it does not require that the person who administers it has a minimum degree or license. The original NCFAS has 39 items; four items, irrelevant to IHS or impossible to collect, were eliminated. Each of the sub-scales has 5 to 10 items, measured on a six-point ordinal scale ranging from -3 (serious problem) to the 0 point (baseline or adequate) to +2 (clear strength). Note that the “overall” item of each sub-scale is not an average of the domain items, but rather an overall worker assessment of the focus of the respective domain (e.g., overall environment, overall family interactions, etc.); workers were asked to assess the overall score after they assessed all other items on the scale.
This article utilizes the NCFAS data collected at the time of program enrollment (intake), the case status at closure successful (when children maintained with parents) vs. non-successful (when children were placed with relatives, kinship care, or in out-of-home care), and the total number of hours and home visits provided during the time the family was active in the program (summated hours or visits recorded in the quarterly forms available for each family).
Data Analyses
Data analyses plan included descriptive statistics of socio-demographic and service measures for the entire caseload and for the cases that completed the program successfully. The intent was to identify any variables that could help set apart the successful cases within the caseload from the unsuccessful ones, at the time of intake. Then, NCFAS scale validation was conducted using a general factor model and a congeneric factor model; the extracted factors were saved as regression variables. Next, t-tests were used to identify statistically significant differences in the regression factors between the successful and the unsuccessful cases. The factors that were significantly different between the successful and the unsuccessful cases were used as predictors of the case outcome using logistic regression models. Finally, a Pearson’s correlation analysis of the extracted factors provided insights regarding the scale’s concurrent validity. The data management and the data analyses were conducted with IBM SPSS 22.
Results
Descriptive Analyses
During the 42-month study period 1959 families were enrolled in and provided with CCC intensive in-home visitation services and 1741 of the 1959 cases, or 88.9% of all cases were closed during the study period. As shown in Table 1, the 1959 CCC families included a total of 7220 clients of which 3006 were adults and 4214 were children. The majority of the children (77.5%) were age 12 or younger with 38.9% age 5 or younger, 38.6% were ages 6 to 12 and 22.4% were teenagers. Thirty five and a half percent (35.5%) of the families had three or more children.
Table 1:Caseload Characteristics
Intake /All Caseload Successful Cases
N % N %
Cases 1959 100.0 1202 61.4
Closed 1741 88.9 1202 100.0
Demographics
Adults 3006 100.0 1869 100.0
Children 4214 100.0 2568 100.0
Ages 0 to 1 480 11.4 286 11.1
Ages 2 to 5 1159 27.5 705 27.5
Ages 6 to 12 1627 38.6 1009 39.3
Ages 13 to 18 948 22.5 568 22.1
Households
Single adult 971 49.6 564 46.9
Minority 383 19.6 219 18.2
Female HOH 1618 82.6 975 81.1
Age HOH (𝑥̅, σ) 34 14.4 34 10.8
Adult with Disability 478 24.4 293 24.4
Child with Disability 469 23.9 309 25.7
Anyone with Disability 762 38.9 479 39.9
With 1-2 children 1263 64.5 783 65.1
3-4 children 590 30.1 359 29.9
5+ children 106 5.4 60 5.0
Services
Home visits scheduled 21103 100.0 14285 100.0 Home visits completed 17710 83.9 12474 87.3
Hours completed 29518 100.0 21161 100.0
<1 hour 68 3.5 11 0.9
1-10 hours 750 38.3 325 27.0
11-20 hours 724 37.0 556 46.3
21-30 hours 239 12.2 183 15.2
31+ hours 178 9.1 127 10.6
Services (𝑥̅, σ)
Home visits completed 9.19 7.51 10.43 7.73
Hours completed 15.46 13.54 17.75 13.38
Family Assessment (adequate or better)
Environment 943 48.1 625 52.0
Parental Capabilities 780 39.8 522 43.4
Family Interactions 1072 54.7 704 58.6
Family Safety 505 25.8 324 27.0
Child Well-Being 724 37.0 485 40.3
Note: 𝑥̅ = mean, σ=standard deviation; †%successfully closed of all cases; ‡ % successfully closed of all closed cases
During the study period, the IHS programs provided 29518 hours or about 15 hours/case (standard deviation of 13.5 hours) and 17710 in-home visits to 1959 cases, or an average of about 9 visits per family (standard deviation of 7.5 visits). There were 68 families or 2.6% who received less than one hour of services and were referred for other community services; 38.3% or 750 cases received 1 to 10 hours of in-home services; 37% (724 cases) received 11 to 20 hours; and, 21.3% (417 cases) received 21+ hours of in-home services.
family functioning on a specific item. It was noted that family safety items were especially difficult to assess at intake any other way than with a “0”. Majority of families were assessed to have adequate (0) or better (1, 2) score on the family interaction domain, and 48.1% on the environment domain. These percentages were comparable with NCFAS literature (Kirk, Kim & Griffith, 2005) for the environment and child wellbeing domains, but the IHS program had a higher percentage of adequate/better scores on parental capability and family interactions, and a lower percentage on the family safety domain.
Further, at right, Table 1 shows the patterns of service provision for the successfully closed cases, the families that successfully maintained their children in the home. Of the 1741 closed cases, 1202 or 69% were successful in maintaining their children in the home; this means that 61.4% of the original IHS caseload maintained the children with parents. A majority of the CCC children that removed from their home were placed with relatives or in kinship care and of the 531 closed but unsuccessful cases, only 25 cases were cases of children removed from the family and placed in out-of-home or foster care.
Successfully closed cases were very similar in their demographics and family structure with the overall caseload, but there were differences in the family functioning intake scores and in service provision. The proportion of families with adequate or better family functioning scores at intake were slightly higher for successfully closed cases than for the overall caseload. These differences in percentage points seem small, but a series of χ2 tests show that all but family safety sub-scale (p=.134), were statistically significant (p<.001). The proportion of completed visits of all scheduled visits was significantly higher for the successful cases than for the unsuccessful cases. Families with multiple and complex problems are very hard to serve because they have a greater tendency to reschedule, cancel or miss appointments, which makes it even more difficult to provide them with social services. Overall, about 13% of the scheduled visits were missed or cancelled by parents in the unsuccessful cases, 5% of the visits were rescheduled, and 20% of the scheduled and/or rescheduled visits were attempted by the CCC staff could not be completed due to parents missing the appointment. The two measures of service provision, number of home visits and number of hours of visits completed, were highly correlated with each other (R=.850, p<.001); but, the total hours of visitation completed was more strongly associated with the outcome (χ2 (2)=206.5, p<.001) and thus, a recoded version of it was used in the logistic regression analyses.
NCFAS Validation
The NCFAS scale validation with the Kentucky’s CCC IHS population was conducted using both a general factor analysis (Reed-Ashcraft, Kirk & Frasier, 2001; Hattie, 1985; DeVellis, 2003) and a congeneric factor analysis (Hattie, 1985; DeVellis, 2003). Principal Axis Factoring (PFA) extraction and Varimax rotation (Reed-Ashcraft, Kirk & Frasier, 2001; Kirk & Reed, 2000; Kirk & Reed-Ashcraft, 2004; Kirk & Griffith, 2007; Kirk, 2012; Kirk, Kim & Griffith, 2005) seeks to explain the common variance in the set of items (Hair et al., 1998; Velicer, Eaton & Fava, 2000; Tabachnick & Fidell, 2007; Ledesma & Valero-Mora, 2007; DeVellis, 2003) measuring family functioning. Varimax rotation maximizes moderate and high correlations between items, and minimizes the low correlations (Tabachnick & Fidell, 2007), to achieve the best orthogonal solution.
The average Corrected Item-Total Correlation (CITC), a coefficient used to estimate a scale’s convergent validity, varied between α=.577 for the environment sub-scale to α=.776 for family safety sub-scale; it is desired to be 0.6 or above. The overall proportion of variance explained by the general factor analysis model was 69.1%. The largest proportion of variance was explained by the environment sub-scale (40.5%), followed by the family interaction (15.8%), and by child wellbeing (6.5%) and family safety (6.3%).
Table 2:Comparisons of Scale Statistics
Model Domain N Items α % Variance Explained
Published models
Overall n/a n/a n/a 61.31
Environment 1188 10 .922 20.13
Child Well-Being 979 8 .801 16.48
Family Interactions 881 5 .771 14.24
Family Safety 1036 6 .767 4.70
Parental Capabilities 1187 7 .814 n/l
General Factor Analysis
Overall 1616 18 .911 69.089
Environment 1616 8 .899 40.471
Family Interaction 1616 4 .880 15.780
Child Wellbeing 1616 3 .807 6.550
Family Safety 1616 3 .884 6.288
Parental Capability n/l n/l n/l n/l
Congeneric Factor Analysis
Environment 1877 9 .903 57.098
Family Interaction 1374 5 .871 66.862
Parental Capability 1881 5 .847 62.374
Family Safety 1430 6 .859 59.046
Child Wellbeing 1311 6 .903 67.391
(n/a= not available; n/l = no loadings; α from Kirk and Griffith, 2007; %Variance explained from Reed-Ashcraft, Kirk, and Frasier, 2001, with N=288)
General Factor Analysis: PAF with Varimax rotation was conducted with the 35 items collected at the time of IHS intake, using a cutoff point of |0.40| (Lance, Butts & Michaels, 2006; Petkov, Harvey & Battersby, 2010). The validation process included an iteration of factor and reliability analyses until the best solution for the data was found. The overall reliability coefficient was .911 (Table 2); the sub-scale reliability coefficients varied between .807 for the child wellbeing sub-scale and .899 for the environment scale, indicating “very good” levels of reliability (DeVellis, 2003).
Table 3: Factor Loadings by Model
Published Model† General Model Congeneric Model
Item E CW FI FS E FI CW FS E FI CW FS PC
Environment
Overall .854 .830 .831
Housing
Habitability .774 .770 .761
Housing Stability .774 .768 .766
Financial
Management .732 .646 .679
Learning .707 .641 .707
Hygiene .780 .640 .693
Nutrition .758 .635 .709
Community Safety .718 .626 .675
Transportation .700 <.40 .642
Income -- <.40 <.40
Family Interaction
Overall .799 .829 .910
Expectations of
Children .593 .727 .788
Bonding .685 .669 .758
Mutual Support .670 .638 .733
Caregivers’
Relationship -- <.40 .632
Child Wellbeing
Overall .727 .778 .849
Child Behavior .838 .818 .850
Child Mental
Health .775 .781 .812
Caregiver
Relationship .805 <.40 .744
Siblings
Relationships .592 <.40 .732
Peer Relationships .706 <.40 .689
Cooperation/motiva
tion .614 <.40 <.40
Family Safety
Overall -- .657 .739
Child Psychological Abuse
.683 .677 .740
Child Neglect -- .650 .711
Child Physical
Abuse -- <.40 .740
Child Sexual
Abuse .712 <.40 .701
Domestic Violence .438 <.40 .606
Parental Capability
Overall .841
Discipline .749
Enrichment
Opportunities .716
Supervision .697
Parent Mental
Health .634
Parental Physical
Health <.40
Parent Substance
Abuse <.40
The items loading on the environment and family interaction factors, extracted with the general factor analysis, were similar to the published model results, but the amount of variance explained was different. The environment factor in the general model explained 40.5% of the variance as compared to 20.1%; the family interaction factor explained 15.8% as compared to 14.2%. Significant departure of the general model from the literature was also noted on the child wellbeing and family safety factors. Specifically, in the published model, the child wellbeing subscale explained the second largest proportion of variance (16.5%) and included all of the original seven sub-scale items. In the general model, the wellbeing factor was the third extracted factor, had three item loadings (overall, behavior, and mental health) and explained only 6.6% of the variance. The items with loadings below the cutoff point were measures of relationships with siblings, peers, caregivers, and child cooperation.
Family safety was the fourth extracted factor in both models, each with three items loadings; they had similar proportion of variance explained but their item composition was different. The psychological abuse item loaded in both models; the other two items were child neglect and the overall family safety (general model) and respectively, sexual abuse and domestic violence (published model) items.
Congeneric Factor Analysis: The second approach was to conduct PAF with a Varimax rotation with a cutoff point of |0.40| separately for each domain or sub-scale; the extracted factors (Table 3) were again saved as regression scores. As with the general factor model, reliability analyses (Table 2) were conducted for each domain. The environment sub-scale was computed using 1877 valid cases, included 9 out of 10 items, explained 57.1% of the variance, and had an excellent Cronbach’s reliability coefficient (α=.903). The child wellbeing sub-scale had 6 out of the 7 items with loadings above the cutoff point, with an excellent reliability coefficient (α=0.903), and explained a large proportion of variance (67.4%). Family interaction and family safety retained all of their items, had very good reliability coefficients (α>.85), and explained 66.9% and respectively 59% of the variance. The congeneric approach allowed for the examination of the parental capability subscale, which was not extracted with the general model, or with the published model; 5 of its 7 items had loadings of |.40| or above, had very good reliability (α=.847) and explained 62.4% of the variance in the data.
Overall, with the congeneric approach, 31 of the 35 items had factor loadings greater than |.40| as compared to the general model where only 18 of the 35 items were retained in the factor analysis. The four items with loadings below the cutoff point were: income (E domain), child cooperation/motivation to maintain the family (CW domain), and parental physical health and substance abuse (PC domain).
Differences in NCFAS Means Successful Vs. Unsuccessful Cases
All factors extracted with the two methods, general and congeneric factor analyses were saved as regression scores in the dataset. The congeneric model yielded five factors and the general model yielded four factors, for a total of nine new variables in the dataset.
The differences in factor scores between the two outcome groups (successful and not successful in maintaining the children in the home) were tested with a series of independent t-tests (Table 4). The left side of the table displays the results of the Levene’s test of equality of variances, indicating significant differences between the two groups in the dispersion of several factor scores (environment, family interaction, child wellbeing, and parental capability); the standard deviations showed that the unsuccessful group was a more heterogeneous distribution of factor scores than the successful group.
Table 4:Successful Vs. Unsuccessful Cases on Key Indicators
Levene's Test T-test
F Sig. T Df Sig.
General Environment 13.116 .000*** -3.467 1017.53 .001** Family Interaction 4.206 .040* -3.364 1114.18 .001**
Family Safety 2.500 .114 1.951 1614 .051†
Child Well-Being 5.782 .016* -.178 1095.09 .859 Congeneric Environment 11.621 .001** -3.801 1310.65 .000***
Family Interaction 4.472 .035* -3.338 971.23 .001** Parental Capabilities 5.645 .018* -2.792 1401.54 .005** Family Safety 3.464 .063† .884 1428 .377 Child Well-Being 9.011 .003** -1.698 853.15 .090† IHS Home visits completed 1.245 .265 -9.508 1926 .000***
Hours of home visits .102 .749 -9.757 1907 .000*** *** p<.001, ** p<.01, * p<.05, †p<.10
The congeneric model approach yielded similar results, although in this case there were no significant differences in family safety (p=.377) and there were marginally significant differences in child wellbeing (p=.090). The remaining three congeneric factors were significantly different between the groups: environment (p<.001), family interaction (p=.001), and parental capability (p=.005). Note that the family safety factor extracted with the general model indicated that unsuccessful cases scored better at intake than their counterparts, but the congeneric model results showed no statistical significant differences between the two groups. The groups were also significantly different in the amount of services received; the successful cases received on average an additional 3.27 home visits or 6.1 hours of visitation as compared to their counterparts.
Figure 1:NCFAS Intake Factor Scores by Case Outcome
Logistic Regression
To determine the validity of the NCFAS scale in relation to the program’s goal to prevent children’s removal from the home, the extracted intake factor scores were tested as predictors of the outcome at closure (Kirk, Kim & Griffith, 2005) with logistic regression. While cross-validation analyses using randomly split files were conducted during the preliminary analyses, the analysis presented here used the entire dataset.
NCFAS is a practice tool intended to measure family functioning on multiple domains, and it should logically be able to detect changes over time, if they occur. Therefore, closure scores should be strongly related to the case outcome, to likelihood that the children are maintained in or removed from their home; moreover, they should be correlated to the number of hours of services provided during the program. This paper attempts to identify whether the intake scores are related to the outcome, just as the closure scores are. This is important because NCFAS intake assessments assist practitioners in their decisions regarding the type and the intensity of services the family needs in order to successfully maintain their children in the home.
The t-tests (Table 4) identified five intake sub-scale scores that were statistically different (at a p<.05 level) between successful cases as compared to the unsuccessful cases, and two that were marginally significant (p<.10). These seven variables were further used as predictors of case success (1=successful, 0=unsuccessful) using logistic regression. NCFAS are standardized variables, with a mean of 0 and a standard deviation of 1, thus one unit on this scale equals one standard deviation. The unadjusted odds ratios showed that all seven sub-scales were significantly related to the case closure outcome; the minimum increase in the likelihood of success was 11.4% (child wellbeing) and the maximum was 24% (environment) for every 1-standard unit increase in the factor score. The unadjusted model for the safety factor seems to indicate that the likelihood to maintain the children in the home decreases with family safety improvement; certainly, this does not make any sense. At intake, child abuse and neglect or domestic violence are generally not substantiated, and thus providers used the scale’s “0” point, which unfortunately had a double meaning of baseline or adequate. This is a limitation of the scale that should and could be addressed by adding an “unknown” or “unable to assess” category. Nevertheless, family safety factor became non-significant (p=.965) when adjusting for the number of hours of services provided. All other sub-scales continued to explain a significant amount of variation in the data.
Next, logistic analyses were adjusted for the number of hours of intensive in-home visitation received, to account for the impact of service intensity on the likelihood of success. The number of hours was recoded into three categories: up to 10 hours of home visitations, 11 to 20 hours, and 21 or more hours of intensive home visits; the last category (21 or more hours) was used as a reference group. Table 5 displays the logistic regression coefficients, the odds ratios and the p-values for all seven adjusted logistic models.
The adjusted odds-ratios showed that for every 1-standard unit increase on the environment and family interaction scores, the likelihood that the family will maintain the children in the home increased by about 30%. Note that the odds ratios for environment and family interaction were similar (1.305 vs. 1.29, and respectively 1.305 vs. 1.330) in the general and congeneric models, indicating that no matter the extraction method the results were valid and reliable. Thus, families who successfully maintained their children in the home were significantly better off at intake on their environment and family interaction domains.
Table 5:Unadjusted and Adjusted Logistic Regression Models
Unadjusted Adjusted
General Factors N B S.E. Sig. OR B S.E. Sig. OR R2
Environment 1580 .215 .057 .000 1.240 .266 .060 .000 1.305 .117
Hours:21+ .000 .161
Hours: 1_10 -1.404 .152 .000 .246
Hours:11_20 .149 .160 .354 1.160
Constant .664 .053 .000 1.943 1.212 .127 .000 3.359
Family Interaction 1580 .192 .058 .001 1.211 .255 .063 .000 1.290 .115
Hours:21+ .000 .160
Hours: 1_10 -1.431 .153 .000 .239
Hours:11_20 .111 .161 .491 1.117
Constant .663 .053 .000 1.941 1.237 .127 .000 3.445
Family Safety 1580 -.114 .063 .069 .892 -.036 .066 .589 .965 .106
Hours:21+ .000 .147
Hours: 1_10 -1.348 .150 .000 .260
Hours:11_20 .169 .160 .291 1.184
Constant .660 .053 .000 1.934 1.174 .126 .000 3.236
Congeneric Factors
Environment 1826 .202 .052 .000 1.224 .266 .054 .000 1.305 .110
Hours:21+ .000 .151
Hours: 1_10 -1.372 .139 .000 .254
Hours:11_20 .096 .145 .507 1.101
Constant .567 .049 .000 1.762 1.119 .115 .000 3.061
Family Interaction 1343 .200 .061 .001 1.222 .285 .065 .000 1.330 .123
Hours:21+ .000 .170
Hours: 1_10 -1.592 .171 .000 .204
Hours:11_20 -.044 .176 .802 .957
Constant .661 .058 .000 1.937 1.344 .143 .000 3.833
Child Wellbeing 1283 .108 .061 .078 1.114 .139 .064 .031 1.149 .109
Hours:21+ .000 .151
Hours: 1_10 -1.387 .164 .000 .250
Hours:11_20 .146 .174 .401 1.158
Constant .652 .059 .000 1.920 1.184 .135 .000 3.267
Parental Capability 1831 .149 .052 .004 1.161 .241 .056 .000 1.273 .109
Hours:21+ .000 .149
Hours: 1_10 -1.408 .139 .000 .245
Hours:11_20 .072 .146 .623 1.074
Constant .571 .049 .000 1.771 1.153 .115 .000 3.167
Note: The top R2 is Cox and Snell, the second one is Nagelkerke’s R2 estimate.
The adjusted logistic regression models had similar coefficients of determination (R2); the Nagelkerke R2 showed varied between 14.7% and 17%, while Cox& Snell varied between 10.6% and 12.3%. The goodness-of-fit, Hosmer-Lemeshow χ2 test was marginally significant for the environment congeneric factor (χ2 (8) =14.654, p=.066), and for the environment general factor (χ2 (8) =13.436, p=.098); all other χ2 tests had non-significant p-values (p>.25) indicating a good model fit for the data.
sub-scales appeared to explain more of the variation in the data while adjusting for the number of hours of home visits provided, have better sensitivity and specificity, and fit the data the best.
Convergent Validity
Finally, the NCFAS scale was tested for its convergent validity using Pearson’s correlation (Table 6). While all factors extracted with the congeneric model were significantly correlated with each other (bottom quadrant to the right), the general model factors were not. Specifically, the environment factor was not correlated with the family interaction of with child wellbeing; also, even when correlations were significantly different from zero, they were weak (R<.30). In contrast, the congeneric factors were all significant and many were moderately (.30<R<.70) or strongly (R>.70) correlated with each other.
Table 6:Pearson’s R Correlation Coefficients for Concurrent and Predictive Validity
General Model Congeneric Model
E FI CW FS E FI CW FS
FI .042 1
CW .008 .118** 1
FS .079** .074** .055* 1
E .969** .187** .094** .194** 1
FI .231** .932** .321** .261** .407** 1
CW .127** .401** .905** .153** .261** .603** 1
FS .326** .218** .200** .872** .454** .468** .351** 1 PC .386** .520** .406** .334** .519** .692** .598** .489**
*p<.05, **p<.01
The family interaction scales, identified with logistic regressions as the strongest predictors of outcome, were highly correlated with each other (R=.932); in addition, they were significantly correlated with child wellbeing and family safety, and with parental capability, suggesting that improvement on one was associated with improvement on the others. Taken all together, the family interaction factor, regardless of the extraction method used, appears to be the best indicator to predict whether the children will be maintained in the family or not, assuming that the family receives between 11 and 20 hours of home visitation.
Discussion
The primary focus of the CCC program was to provide supports to parents and families allowing them to avoid removal and subsequent placement of their children in out of home care. Families were provided with intensive in-home services on an individualized basis and also referred to outside supportive services as needed. There were commonalities across a statewide service area in terms of program objectives, but there were also significant differences in the actual program implementation and processes. Agencies varied greatly in the number of home visits and the number of hours of visits provided per family; some agencies had fewer referrals and hence served a smaller number of families, but they provided a larger number of home and hours of visits. Other agencies had a much larger number of referrals, and thus provided fewer one-hour home visits to a larger number of families. Individual agencies also differed in the proportion of successful cases, although all met their goal of 60% or more of the successfully closed cases.
Furthermore, it has been suggested before (Tungate, 2006) that the intake scores were more reliable and were better predictors of permanency outcomes than the closure scores.
NCFAS is a family assessment practice tool widely used by the family prevention programs across the United States; it is a valid measure of family functioning, and has five domains (Kirk, Kim & Griffith, 2005; Kirk, 2012; Reed-Ashcraft, Kirk & Frasier, 2001). To validate the tool with the CCC population, two approaches were employed. First, the general validation model was fit. This model is commonly used because it allows for multiple latent concepts to underlie the scale items, closer to how the real-world data is (DeVellis, 2003). With this method all items belonging to all sub-scales were included in the same factor analyses and through an iterative process a smaller number of reliable and valid factors were extracted. Second, the congeneric model presumes that factor and reliability analyses are conducted by scale domain. The factors extracted with both methods were saved as regression scores and used as predictors of program outcome in subsequent analyses; the main difference between the two sets of factors, was the parental capability factor which was not extracted with the general factor analysis.
Descriptive statistics were conducted to identify differences in the intake scores, between the successful and the unsuccessful cases, with the intent to use them as predictors of case outcome. The tests of differences in factor means between successful and unsuccessful cases yielded a list of seven potential outcome predictors. The t-tests showed that the cases with children who were maintained in the home had significantly better start-up scores on environment, family interaction, and parental capability. Improvements on the family interaction domain were strongly correlated with improvements in child wellbeing, parenting capabilities, family safety and environment. The regression findings suggest a tipping point in the number of hours of visitation (11 to 20 hours), critical for the success of the intervention. This was also a point of diminishing returns, because the family gains were minimal after 20 hours of services and the need for continued family support should be reassessed when reaching this point. The data support the recommendation that at intake a special attention should be given to the family interaction domain, as this factor was the best predictor of case success while adjusting for the number of hours of home visitation.
Conclusions
References
1. DeVellis, R.F. (2003). Scale Development: Theory and Applications. Thousand Oaks, CA: Sage Publications.
2. Hair, J. E., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis. Upper Saddle River, USA: Prentice Hall.
3. Hattie, J. (1985). Methodology review: assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(2), pp. 139-164.
http://dx.doi.org/10.1177/014662168500900204
4. Kirk, R.S., & Reed, K.B. (2000). Designing and developing a new assessment and evaluation tool for family preservation service programs. F. Jacobs, P. Hrusu Williams, J. Kapuscik, & E. Kates (Eds.). Evaluating family preservation services: A guide for state administrators. Medford, MA: Family Preservation Services Project, Tufts University.
5. Kirk, R.S., & Reed-Ashcraft, K.B. (2004). NCFAS Research Report. Guide submitted to the National Family Preservation Network. Retrieved September 5, 2014,
http://nfpn.org/Portals/0/Documents/ncfas_scale_development.pdf.
6. Kirk, R.S., Kim, M.M., & Griffith, D.P. (2005). Advances in the reliability and validity of the North Carolina Family Assessment Scale. Journal of Human Behavior in the Social Environment, 11(3/4), pp. 157-176. http://dx.doi.org/10.1300/J137v11n03_08
7. Kirk, R.S., & Griffith, D.P. (2007). An Examination of Intensive Family Preservation Services. In cooperation with the National Family Preservation Network. Technical report submitted to the Annie E. Casey Foundation, Retrieved October 1, 2014 from
http://tinyurl.com/khc9muo.
8. Kirk, R.S. (2012) Development, Intent, and Use of the North Carolina Family Assessment Scales and their Relation to Reliability and Validity of Scales. In cooperation with the National Family Preservation Network. Retrieved September 5, 2014 http://www.nfpn.org/Portals/0/Documents/ncfas_scale_development.pdf.
9. Lance, C. E., Butts, M. M., & Michaels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9(2), pp. 202–220.
http://dx.doi.org/10.1177/1094428105284919
10. Ledesma, D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: an easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research and Evaluation, 12(2), pp. 1-11.
11. Petkov, J., Harvey, P., & Battersby, M. (2010). The internal consistency and construct validity of the partners in health scale: validation of a patient rated chronic condition self-management care. Quality of Life Research, 19(7), pp. 1079-1085.
http://dx.doi.org/10.1007/s11136-010-9661-1
12. Reed-Ashcraft, K.B., Kirk, R. & Fraser M. (2001). The reliability and validity of the North Carolina Family Assessment Scale. Research on Social Work Practice, 11(4), pp. 503-520.
13. Stone, R.F., & Barber, G.M. (2008). Final Evaluation Report: Community Collaboration for Children Programs 2006-2008. Submitted to the Cabinet for Health and Family Services, Frankfort, Kentucky.
14. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Fifth Edition. Boston: Allyn & Bacon.
15. Tungate, P. N. (2006). Legacy Children: Whose Legacy Are They? PhD Dissertation, University of Louisville.
16. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. G. D. Goffin and E. Helmes (Eds.), Problems and Solutions in Human Assessment. (pp 44–71), Boston: Springer.
http://dx.doi.org/10.1007/978-1-4615-4397-8_3
17. Westat Chapin Hall Center for Children James Bell Associates (2002). Evaluation of Family Preservation and Reunification Programs: Final Report. Department of Health and Human Services Assistant Secretary for Planning and Evaluation, Washington, DC. Retrieved October 13, 2014 from
