Findings from the BOND process study will be shared with SSA throughout the demonstration project. In addition to individual site progress summaries, process study findings will appear in eight BOND
evaluation reports including four rounds of interim reports, an early assessment report and policy brief, and two synthesis reports—the final process study report and final report on participation, impacts, and benefits and costs (see Exhibit 4-8). For each of these reports, the qualitative process study data will be combined with statistics based on data from BODS, SSA administrative data, and baseline and follow up surveys. Chapter Nine describes each of the Stage 1, Stage 2, synthesis, and special topics reports. Below is a description of how process study data will be incorporated into these reports.
Process study site reports: After each round of site visits, the process study team will compile brief summary reports for each site that describe the basic implementation of BOND. The reports will be used to monitor the progress of each of the sites and to prepare for the next round of visits.
All summaries will be submitted to SSA within two weeks of the last site visit for that round.
Stage 1 interim participation, process, and impact reports: The BOND demonstration includes two rounds of Participation, Process, and Impact Reports for Stage 1 demonstration activities. Process study data from round 6 and 7 visits will be used for the first and second reports, respectively. The first report will be submitted to SSA in December 2015. The second report will be delivered two years later, in December 2017.
Stage 2 interim participation, process, and impact reports: The BOND evaluation team will deliver a second series of interim synthesis reports. The first of these reports will be submitted in March 2014 and will include process study findings from round 5 site visits. The second report, submitted to in June 2016, will incorporate findings from round 7 site visits.
Stage 2 early assessment report and policy brief: The early assessment report, completed March 2013, will include findings from round 3 process study site visits. These findings will also be used to generate a policy brief based on the early assessment report.
Final process study synthesis report: In June 2014, the process study team will submit a comprehensive report from Stage 1 and Stage 2 summarizing the implementation findings across the first five rounds of site visits. This report will provide a detailed look at the context of the demonstration and early findings on the fidelity of implementation to the BOND model.
Final synthesis report on participation, impacts, and benefits and costs. In the final report for this project, the evaluation team will pull from process study findings across the seven rounds of visits. The report will include the full assessment of fidelity of BOND to its design. The findings will be used to help interpret impact study findings and provide recommendations for how SSA might revise the BOND model to improve program outcomes under national implementation.
Exhibit 4-8 Gant Chart for the Process Evaluation
Year
2009 2010 2011 2012 2013 2014 2015 2016 2017 Implementation Milestones
Operate local BOND site offices
Pilot solicitation and random
assignment (Stage 2)
Stage 1 random assignment
Stage 1 outreach
Stage 2 main solicitation
Stage 2 random assignment
Offset, WIC, and EWIC in place Data Collection
Site Visits
Stage 2 Baseline Survey
Stage 2 12 Month Survey
Stage 2 36 Month Survey
Stage 1 36 Month Survey
Administrative Data*
Reports
Process study site reports
Stage 1
Interim participation, process, and
impact reports (2)
Stage 2
Early assessment report
Policy brief
Interim participation, process, and
impact reports (2)
Synthesis Reports
Final process study report
2009 2010 2011 2012 2013 2014 2015 2016 2017
*Administrative data used for the process study includes TRF and BODS
Abt Associates Inc. Evaluation Analysis Plan 92
Chapter Five. Participation Analysis
The participation analysis will document the engagement of prospective BOND subjects in work activity from before entry into the demonstration through the end of the demonstration. It is expected that the use of BOND demonstration services by both treatment groups, including their interest in volunteering to participate in these services in Stage 2, will vary across beneficiary subgroups (e.g., younger versus older workers). The participation analysis will summarize patterns of participation by subgroups and, more broadly examine whether the BOND interventions influenced all types of work activity.
The first component of the participation analysis will focus on the recruitment of volunteers for Stage 2.
In Stage 1, recruitment is automatic in that subjects are assigned to the T1, C1, or Stage 2 solicitation pool directly from administrative records. In Stage 2, however, subjects targeted for recruitment must
volunteer if they are to participate in the demonstration. Hence, Stage 2 recruitment provides an important opportunity to examine beneficiary interest prior to the delivery of intervention services. This information is important in understanding the characteristics of treatment and control subjects recruited in Stage 2 and, more broadly, the characteristics of beneficiaries who could be potentially targeted in future SSA
demonstrations for return-to-work interventions based on their interest in BOND. This recruitment analysis will address the following questions:
1. To what extent do subjects targeted for recruitment for Stage 2 volunteer?
2. What characteristics distinguish volunteers from non-volunteers?
The second component of the participation analysis will include a summary of beneficiary engagement in work and use of demonstration services in both stages. The findings will provide information on the engagement of prospective BOND subjects in work activities and, for the treatment group, use of demonstration services, across both stages. The analysis for the second component will address the following questions:
3. To what extent do subjects in each treatment group work or use employment services and benefits counseling?
4. Who works, uses counseling services and other work incentives, and eventually uses the offset?
5. How did the demonstration affect the use of work-incentive counseling and the services delivered by counselors?
6. What characteristics distinguish offset users from others?
7. How do work and use of work incentives vary across demonstration groups?
8. How do work and use of work incentives change with time?
Information on the eight research questions will be updated for various reports throughout the
demonstration as the data become available (see Chapter Nine for details). The findings are expected to inform interpretation of the impact analysis, support policymaker and administrator decisions concerning
the design and implementation of a national program, and provide basic knowledge about beneficiary behavior.