BODS will be used to support and track delivery of demonstration services to all BOND subjects.
Although it is central to the implementation of the BOND treatments, it also will play a vital role in evaluation. BODS will support the following activities:
Random Assignment: BODS will include data from SSA secured administrative extracts on prospective BOND subjects. The implementation team will use these data to randomly assign cases in Stage 1 and—for those who volunteer—Stage 2.
50 A key component of ensuring comparability in data collection is that Abt and Mathematica will collaborate on interviewer training so interviewers in each organization have the same preparation about the purpose and importance of the study, refusal avoidance techniques, data security protocols, documenting call attempts, and administering the interview. They will have the same training on assessing respondents’ ability to complete the interview and use of proxies. Further, instructions for questions that are the same across the Stage 1 and Stage 2 instruments will be identical. The only difference in the interviewer training across the organizations will be related to differences in the Stage 1 and Stage 2 instruments.
Abt Associates Inc. Evaluation Analysis Plan 58
Outreach and Recruitment: BOND site offices will use BODS to track contacts with eligible beneficiaries, including outreach to Stage 1 treatment subjects and outreach and recruitment for those in the Stage 2 solicitation pool.
Earnings Reporting: The BOND implementation team will use BODS to compile earnings information from BOND beneficiaries and transmit it to SSA to support adjudication of TWP completion, SGA determinations, and benefit adjustments.
Enhanced Work Incentives Counseling: All counselors providing EWIC services to
demonstration subjects will use BODS to record their contacts with beneficiaries and the services delivered.
Work Incentives Counseling: Counselors providing WIC who operate in WIPA programs will use regular WIPA data collection procedures to track their contact with BOND subjects.
Specifically, WIC providers that are WIPA agencies will enter data into an Effort to Outcomes (ETO) database, the contents of which will be transferred to BODS at appropriate points. All EWIC providers will enter their information directly into BODS. The data items on WIC in BODS will match the variables included in ETO so that direct comparisons can be made of EWIC and WIC services. WIC counselors who do not operate in WIPA programs will track their
contacts with beneficiaries in BODS.51
Support Call Center: The call center will record BODS information on its interactions with beneficiaries.
Support Data Collection: The BOND team will use BODS to obtain contact information to support survey data collection activities and to conduct matches to other administrative data systems.
The BODS system includes program group identifiers for all prospective BOND subjects (i.e., for T1, T21, T22, C1, C2, and non-volunteers). The evaluation will use these variables to identify the analytic samples for all analyses. The BOND team will link this information to the administrative and survey files described above to obtain analysis files for the right samples. For example, to analyze SSDI program status of T1 subjects, BODS data indicating T1 group membership will be linked to SSA administrative data on SSDI program status.
3.3.1 Data on WIC and EWIC Services
BODS will include information on WIC and EWIC services. WIPA projects will provide benefits counseling to the C1 and C2 beneficiary groups based on existing SSDI program rules. WIPA projects that have been designated as BOND WIC providers will provide WIC services to T1 and T21 subjects based on the BOND rules and EWIC providers will provide WIC to T22 subjects based on the BOND rules.52 As is described in more detail in Chapters One and Two, the EWIC services are an enhanced
51 As noted, data on WIC services collected via ETO will be added to the BODS counseling data for evaluation purposes. For simplicity, then, we consider these data to be part of the “BODS data” throughout the remainder of this document.
52 As will be described in more detail in Chapter Five, the BOND team will distinguish between the benefits counseling provided to treatment and control group subjects given that treatment subjects receive specific information on the BOND work incentive.
version of WIC services, including proactive outreach to beneficiaries, additional beneficiary assessments, job development and referrals.
As shown in Exhibit 3-3, BODS has been designed to collect information on WIC services provided to all BOND subjects. BODS will include data elements for the T1, T21, C1 and C2 groups, entered by
counselors into the data system currently used by all WIPA projects. It will also include comparable information on the enhanced EWIC services, entered by EWIC counselors. 53
Exhibit 3-3. Counseling Variables for the Evaluation from the BODS
Information about Counseling Provided to All Treatment and Control Subjects Random Assignment Identifier
Identifier for T1, T21, T22, C1, and C2 groups Contact Details
Services provided (outreach, problem-solving and advocacy, work incentives analysis, and long-term support)
Amount of time providing services Counseling services terminated Beneficiary Assessment
Employment (employed, type of work, hours of work)
Beneficiary goals (plans to pursue education, stop receiving benefits, strategies for employment outcomes)
Other benefits (private health insurance, public health insurance, other cash supports [e.g., Unemployment Insurance, Temporary Assistance for Needy Families])
Knowledge and use of SSA work incentives (e.g., assigned Ticket, Trial Work Period, Extended Period of Eligibility, etc)
Work related training services (use of Vocational Rehabilitation, One Stop Career Centers) Outcome of Counseling Contact
Working beneficiary outcomes (quit working, increased work hours, reduced work hours) Non-working beneficiary outcomes (job search, job acceptance)
New or updated work incentives plan prepared Percent for whom follow-up contact recommended
Additional Information on Enhanced Work Incentive Counseling (T22 group only) EWIC Contact Details
Type of EWIC contact (telephone, e-mail, in person, in office, mail or somewhere else) Amount of time providing services
EWIC Beneficiary Assessment
Barrier domains identified from barriers and needs assessment (health, employment, financial, othera) Vocational assessment
53 WIPA providers enter service documentation into their ETO data collection system. In the same manner, the WIC providers who are WIPAs will document their service provision in ETO. EWIC service providers will enter this information into the Beneficiary Tracking System (BTS), a subsystem of BODS. All WIC service providers will follow the same ETO protocols for entering information that WIPA agencies currently follow.
EWIC service providers will document the benefits counseling portion of EWIC services in BTS, using the same data elements tracked in ETO.
Abt Associates Inc. Evaluation Analysis Plan 60 Job Development and Referrals
Services planned as part of employment services plan (ESP) development (education, pre-employment skills training, unpaid work experience, placement in paid employment, other related work supportb) Service referral
Pre-employment skills development Employer partnership development Referral to job placement organization Job placement follow-up
Job retention follow-up
a Other barrier domains include social, family, personal capacity, and other.
b Other related work support includes job coaching, accommodation, job retention/post placement support, transportation, and other.
The variables common to counseling services provided to all BOND treatment and control subjects include contact details, beneficiary assessments, and outcomes of the WIC contact. BODS will also include additional information about EWIC services: type of contact (e.g., in person versus telephone), additional assessments (e.g., barriers to employment), and career exploration and referrals.54 Because the latter information is not being collected from WIC counselors, it can only be used for purposes of describing the nature of the services provided by EWIC counselors, not how they differ from those provided by WIC counselors.
3.3.2 Data Development and Quality
The BOND team will continuously monitor BODS data throughout the demonstration and internal checks will be made to test data quality. The BOND team will test each of the variables in BODS to verify that its functionality matches the documented specifications (Bell, Sokol, Hoover and Thurston, 2010). These methods include manually testing elements and inspecting the underlying data to ensure that it has not been modified unexpectedly. The BOND team will also use data elements from BODS in management reports to periodically assess the reliability of the data items and, as necessary, provide feedback to sites on potential improvements.