Although administrative records typically provide higher data quality at lower cost than surveys, they do not capture all of the information essential to evaluate BOND; as a result, the evaluation will administer surveys to study subjects to obtain data not contained in administrative records. For example, follow-up surveys of members of the Stage 1 and Stage 2 samples will collect information on barriers to
employment and work activities, recent work, customer satisfaction, and individual characteristics, such as health status.
Stage 1 has a single survey (36 months after random assignment), whereas Stage 2 has a baseline survey and two follow-up surveys (12 and 36 months after random assignment).45 Each survey is designed to take less than one hour to complete. Both stages have a 36-month follow-up survey to track long-term outcomes of BOND subjects. Stage 2 also has a baseline survey to describe the volunteer population at random assignment and a 12-month follow-up survey to track short-term outcomes and customer satisfaction with demonstration services, especially EWIC as compared to WIC. An 80 percent or higher response rate is expected for all follow-up surveys (and 100 percent for the Stage 2 baseline survey).46 All Stage 2 baseline surveys will be conducted in-person using Computer Assisted Personal Interviewing (CAPI). All of the follow-up surveys for both stages will be conducted using a mixed mode approach—
Computer Assisted Telephone Interviewing (CATI) with in-person CAPI as follow-up. The latter will be used for those who cannot or will not be interviewed by telephone. Follow-up interviewing will begin with a contact letter half a month before the relevant month and continue for four months. For example, for the 36-month survey, the effort to conduct an interview will begin with a contact letter in month 35.5 after random assignment, with the objective of completing the interview by month 40. Additional details for each survey are provided in Exhibit 3-2, including interview duration and mode, sample size and composition, and interview content.
The balance of this section provides more information on the surveys. Even more detail, including the survey instruments, appears in McInnis, Bell, and Ciemnecki (2010). First, we discuss the single Stage 1 survey at 36 months. Then, we discuss the three Stage 2 surveys—baseline, 12-month, and 36-month.
This more intensive surveying of the Stage 2 sample is consistent with gathering more detailed
information on samples that will, by design, have a much higher concentration of working beneficiaries than the Stage 1 samples.
Exhibit 3-2. Questionnaire Content, Target Population, Interview Mode and Length for Stage 1 and Stage 2 Surveys
Type of interview CATI/CAPI CAPI CATI/CAPI CATI/CAPI
Groups Interviewed
Target sample size 10,000 12,600 12,600 12,600
45 Since Stage 1 random assignment will occur without contacting beneficiaries prior to random assignment, there will be no baseline survey of these subjects.
46 Only individuals who have completed the baseline will be randomly assigned at Stage 2.
Abt Associates Inc. Evaluation Analysis Plan 54
Continuous employment history Past 36 months
Volunteer work Past 36
months
Past 12 months
Current or last Current or last Employment and earnings of
other family members
Barriers to employment
Cost of working
Perceived ability to return to
work; attitudes towards work
Employer attitudes; willingness
to accommodate
Training Programs
Vocational rehabilitation
One Stop
Other
Health and Functioning Physical and mental health
status
Activities of daily living and instrumental activities of daily living
Use of medical services and
facilities
Current health insurance
coverage
Total Income and Sources of Income
Total income
Availability of transportation
Household income
Financial hardship
Household composition
Stage 1 Stage 2 36 month
(45 minutes) Baseline
(55 minutes) 12 month
(26 minutes) 36 month (30 minutes) Stage 2 Surveys Only
Understanding of Work Incentives Understanding of and attitudes
towards SSDI work incentives
Receipt and Satisfaction of Stage 2 Services Receipt of BOND support from
counselors
Satisfaction with services
3.2.1 Stage 1 Follow-Up Survey
The Stage 1 36-month survey will collect long-term outcome information on a randomly selected
subsample of the Stage 1 subjects. (We discuss sample sizes immediately below.) The survey will include information on several outcomes that are important to evaluating impacts and measuring the benefits and costs of BOND, including employment and related activities (e.g., recent employment history, type of work, and barriers to work), participation in training programs (e.g., SVRA and One Stop), and total income. Additionally, the survey will include contextual information that is not included or measured well in the administrative data, such as education, race, and family composition.
The sample for the 36 month survey will include 5,000 beneficiaries randomly drawn from the 80,000 subjects in T1 and 5,000 randomly drawn from the much larger group of subjects in C1—expected to exceed 500,000. The random selection methodology for the two groups will be somewhat different because, by design, the two groups differ in two important respects: the percentage of current
beneficiaries and the percentage of short-duration beneficiaries. The different methodologies will ensure that the two samples are comparable. The methodology for the C1 sample will be a modified version of the methodology for the T1 sample. The methodology to be applied to T1 is described first, followed by the modifications that will be used for C1.
The T1 sampling methodology will oversample T1 subjects most likely to benefit from the offset—
namely those most likely to work. An econometric model developed from C1 data will be used to predict the likelihood of employment under current law for all T1 subjects. 47 The probability that a T1 subject is selected will be highest for those with the highest predicted probability of employment. This is likely to result in a sample in which the percentage of concurrent beneficiaries and the percentage of short-duration beneficiaries are each different than the corresponding percentages in the entire T1 sample.
47 The predicted probabilities will be based on a logit model estimated with C1 data for 2011. Employment will be defined as annual earnings in excess of $1,000, based on MEF data. The exact specification for the explanatory variables has not yet been developed, but the following variables are expected to be important predictors of employment: duration on the rolls, employment in the previous year for those on the rolls in the previous year, prior completion of the TWP, age, primary insurance amount (PIA), primary impairment, and sex. Although demonstration site or local area characteristics are likely to be predictive of employment, they will not be included in the model as doing so might lead to a survey sample with a cross-site distribution that is quite different from the cross-site distributions of T1 and C1 subjects. Differences in the distributions of individual characteristics across sites are likely to lead to some differences between the cross-site distribution of the survey sample and the cross-site distributions of T1 and C1 subjects.
Abt Associates Inc. Evaluation Analysis Plan 56
The methodology for the C1 survey sample will use the same econometric model to predict the probability of employment for each C1 subject, but sampling will be stratified by SSI status and short- versus long-duration. The number sampled from each of the four strata defined by these two beneficiary characteristics will be the same as for the T1 sample. Within each stratum, the probability of selection will be highest for those with the highest predicted probability of employment.
Survey weights that make the weighted samples representative of the national population will be used to produce weighted means. Differences in weighted mean survey outcomes for the T1 and C1 respondents will estimate the impacts of the benefit offset on the means of those outcomes for the national beneficiary population.
3.2.2 Stage 2 Surveys
All beneficiaries who volunteer and are randomly assigned to any of the three Stage 2 demonstration cells will become part of the Stage 2 survey sample—12,600 individuals across all 10 sites; any reduction in the number of those surveyed will be a result of attrition.
Baseline Survey
The Stage 2 baseline survey will be used to collect information on background characteristics of
volunteers that are not measured well in the administrative records. The baseline survey variables include recent employment history, current income, contextual information on demographic and family status, and an assessment of the subject’s current understanding of SSDI work incentives and attitudes toward the demonstration. These detailed data will be used to describe the sample of volunteers, form subgroups for separate analysis, provide covariates in the impact analyses,48 and help to adjust for non-response on the follow-up surveys.
Prior to random assignment, all volunteers must complete a baseline survey. Most of the volunteers will be invited to come to the local BOND office for a comprehensive intake and enrollment session with a specialist from the BOND team. After the BOND specialist explains the demonstration to a volunteer, the specialist will seek consent to participate in BOND. Those who volunteer will be interviewed in-person by a separate individual in the local BOND office. Immediately upon completion of the baseline
interview, the volunteer will return to the BOND specialist for random assignment. It is expected that 65 percent of the Stage 2 baseline interviews will be completed this way. For those subjects who cannot easily come to the site office, the survey will be conducted in the subject’s home or at another location that is more convenient for the beneficiary.49
48 For example, variables such as employed at baseline and use of other program services, which are likely correlated with key demonstration outcomes (especially employment) are likely to be used as explanatory variables in regression-adjusted analyses of impacts. As the characteristics for the treatment and control groups are expected to be very similar because of random assignment, the primary reason for using these explanatory variables is to increase the precision of the impact estimates.
49 A designated “circuit rider” will travel to remote locations to conduct intake. These circuit riders may meet with prospective subjects at a location that is convenient for the beneficiary. The intake and baseline survey
administration procedures will be similar to those conducted in the BOND office.
12-Month Survey
The 12-month survey will collect information on short-term outcomes and participation with service providers, especially WIC and EWIC providers. The 12-month survey is designed to be brief (26 minutes) and collect information on employment outcomes, understanding of SSA work incentives, and receipt of, and satisfaction with, Stage 2 services (e.g., EWIC services).
36-Month Survey
The goal of the Stage 2 36-month survey is to collect long-term outcome information on Stage 2 volunteers. The survey questionnaire will include many of the same outcomes as the Stage 1 36-month questionnaire (e.g., employment and training), for use in the impact and benefit cost analyses.
Additionally, the Stage 2 36-month survey will collect information on receipt of BOND services (e.g., WIC and EWIC) and understanding of work incentives that will be used in the process and participation analyses.
3.2.3 Data Development and Quality
As outlined in McInnis et al. (2010), there are three key components to collecting high quality BOND survey data. First, the use of computer-assisted telephone interviewing (CATI) and computer-assisted personal interviewing (CAPI) reduces errors in survey administration because skip patterns are automated so the interviewers do not need to follow complex skip patterns manually. Automated interview
technology also prevents entry of invalid answers and prompts interviewers to validate or confirm critical data items such as income and earnings amounts. A second component is the use of experienced
interviewers that are or will be trained specifically in interviewing persons with disabilities. Finally, the third component is the development of common materials for training all interviewers and other survey staff, regardless of the BOND survey on which they are working.50