3 These SMART goals are understood

by family and community members who indicate by signing off on the goals.

1=Not Met; 2=Partially Met; 3=Fully Met 1 2 3

2 _{SMART stands for:}

3. Each ECMHC Project Will Assess the Quality of Their Implementation and Use These Data to Guide Service Planning, and to Improve ECE Provider and Family Engagement.

Rationale: Systematic and regular data collection is a powerful tool for program monitoring, quality improvement and, ultimately, sustainability (Duran et al, 2009, p.61). This outcome evaluation process should be ongoing and will assist in fully examining all elements when implementing ECMHC to determine if the model is working or not and how it can be improved to heighten the program’s positive impact.

Another dimension of ECMHC program success is family engagement (Duran et al, 2009, p. 83). To do that requires time and effort to build trust, communicate

effectively, and understand cultural and family dynamics. One of the primary roles in consultation is to initiate and sustain communication that promotes optimal family engagement.

Guidance: The most important first step in assessing quality is to identify both common and unique outcome indicators of quality performance. Ideally all stakeholders should be involved in the identification of these outcomes. To do this may involve time in conversation to study and select the outcomes that are most meaningful to that setting, using evidence-based practices in that selection process.

As programs begin a formal program evaluation process, there are key questions to consider. The key questions below in Figure 4 are from the EMCHC Toolkit

(Hepburn et al, 2007, p36). These should be discussed and answered with program funders, administrators, staff and the community, including families who have used the services. The answers help frame the program evaluation process and tease out the indicators.

Figure 4. Key Questions for Formal Observation

Which process measures will truly capture whether or not our consultation program has been delivering the services we intended?

Have the established process measures we are adopting been successfully used to track consultation services delivered in similar settings?

Do we need to design and test some of our own measures, due to unique features of our program or consultant services?

Are the tools we are considering user-friendly; that is will the staff who are providing process data be able to easily understand the measures and complete them?

Is the package of process measures realistic in terms of staff time needed to fill them out?

Do the tools actually address the outcomes we intend to produce through the consultation process? Which outcome measures will successfully gauge the short- and long-term?

consequences of the program?

Is there evidence that the tools have been successfully used in other evaluation studies in early childhood settings?

Have the reliability and validity of these measures been established?

Are the measures we are considering sensitive enough to detect changes given the duration and intensity of our consultation program?

Do we need to adopt measures that have norms for populations similar to ours, so that we can compare our outcomes to the scores of other groups?

Are the tools culturally appropriate for the staff, children, and families in our program?

Is the package of outcome measures realistic in terms of the staff members’ and family members’ time required to complete it?

Do the tools actually address the outcomes we intend to produce through the consultation process?

Examples of common quality indicators can be organized into four outcome areas based on stakeholders: program staff (including the MHC), ECE providers,

families and children. Each stakeholder expects something different from ECMHC and is impacted differently; thus the outcomes vary.

Common quality indicators are found in best practices nationally and regionally. Unique indicators are those that are specific to one program that are tied to each

community and those that reflect specific characteristics of the families served (military, itinerant, non-native English speakers, other, etc). These may include specific

numbers of families served based on the characteristics above or other factors that one community considers critical to measure (e.g. score changes using a particular tool, such as the Preschool Mental Health Climate Scale or number of partners used in the delivery of services to show a continuum of care). Table 10 lists examples of outcomes by stakeholder group.

Table 10. Best Practice Outcomes and Unique Outcomes by Stakeholder Stakeholder Common Outcomes

(Evidence Based)

Sample of Unique Outcomes

Program Increased quality

indicators

Increased number of community program linkages

Decreased job turnover

Increased quality indicators for special populations (military, non- English speaking, children with special needs)

ECE Provider

Increased skills for service success

Increased skills for early intervention

Improved

communication with families/caregivers

Increased representativeness (service providers match

characteristics such as language, race, ethnicity, gender)

Families/ Caregivers

Increased follow-through with referrals and

services

Decreased stigma toward mental health

Increased utilization of specific services recommended

services

Decreased stress in family

Child Increased school

readiness Increased pro-social behavior Increased appropriate placements Decreased problem behaviors Decreased expulsion rates

Decreased frequency of expulsions for children of minority cultures

Source: Adapted from Duran et al, 2009, pp 89-90.

By selecting one outcome measure per stakeholder at a time, it is possible to build an incremental program evaluation process that is grounded in best practices, that measures nationally recognized common indicators, and that uses the local community to identify and measure unique indicators. Once an outcome indicator has been selected, it is important to look at tools to assess each of those measures. There are a variety of measures, which each collect specific information. A roster of instruments to measure outcomes can be found in Early Childhood Mental Health Consultation, An Evaluation Toolkit, Appendix C pp. 75-84. This document can be downloaded at: http://gucchd.georgetown.edu/products/ECMHCToolkit.pdf

../AppData/Local/Microsoft/Windows/Temporary Internet

Files/Low/AppData/Local/Microsoft/Windows/Mary/AppData/Local/Microsoft/Windows/Temporary Internet Files/Low/Content.IE5/SMX4RQS2/gucchd.georgetown.edu/products/72402.html

A sample satisfaction measure is presented in this same document in Appendix B, pp 71-72.

Data collection is not a single event. To measure the impact of an ECMHC program, data should be collected before (baseline) and after (follow-up) the consultation has been provided. Baseline and follow up data need to be collected systematically, in that the same tools administered by the same person should be used at the beginning of consultation and at the conclusion.

Strategies to engage family members revolve about building relationships. With sensitivity to culture, it is possible to identify which relationships are most powerful. Persons of authority vary from culture to culture. Sometimes it is a physician who can communicate the need for interventions with a young child (Duran et al, 2009, p.95), a respected elder in a family or community, or even a staff person who can broker among cultures due to the depth of relationships forged, often because of similarities of

background (Duran et al, 2009, p.165). Materials to support family engagement strategies are included in a Best Practice Tutorial on www.echmc.org.

Families who are engaged tend to follow through with data collection and

referrals to other providers; they report less stress in the family (anecdotally or through satisfaction surveys). Overall their understanding of mental health is improved so that the stigma is reduced (Cohen & Kaufmann, 2005).

Self assessment indicators for Standard F3. 1. The ECMHC program will

demonstrate annual assessment of the quality of the consultation using both process (satisfaction) and outcome measures (impact on children and their families).

1=Not Met; 2=Partially Met; 3=Fully Met 1 2 3

2. Families and other community stakeholders are involved actively in identifying indicators of quality and key performance.

1=Not Met; 2=Partially Met; 3=Fully Met 1 2 3