The BOND analyses will include administrative data from SSA program administrative records, SSA earnings records, and administrative records from CMS and the RSA (Exhibit 3-1). We discuss each source in turn.
38 The BOND Operations Data System is a “Moderate” impact system under National Institute of Standards and Technologies (NIST) guidance. BODS will implement applicable controls of a "Moderate" impact system as outlined in NIST Special Publication 800-53 Revision 2, Recommended Security Controls for Federal Information Systems.
Abt Associates Inc. Evaluation Analysis Plan 50 3.1.1 SSA Program Administrative Records from the Ticket Research File (TRF)
The BOND team will use SSA program administrative records from the Ticket Research File (TRF) to examine the characteristics and SSA program outcomes of all BOND subjects. Originally constructed to support the research needs of the Ticket to Work evaluation, the TRF is a longitudinal database that currently includes records for all SSDI and SSI beneficiaries age 10 or older who have received an SSDI or SSI disability benefit in any month since January 1996.39 This group totals more than 22 million individuals. The TRF combines data from the SSDI and SSI programs over many years into a single record for each individual on beneficiary characteristics, such as age, gender, race/ethnicity, state of residence, and impairment type, and on program outcomes such as program history and benefit
payments.40 Special updates of the TRF will be made available during the BOND demonstration so that the latest program information can be incorporated into the project’s evaluation reports (see Chapter Nine). We anticipate that these records will be available with a two month lag. For example, a TRF extract in March 2012 would include full program records on BOND subjects through January 2012.
3.1.2 SSA Master Earnings File
The evaluation will draw on earnings information from SSA’s Master Earnings File (MEF). The MEF contains longitudinal information on the total wages and self-employment income reported to the Internal Revenue Service (IRS). The data are derived from IRS Form W-2, quarterly earnings records, and annual income tax forms (Olsen and Hudson 2009). The major advantage of the MEF is that it contains
comprehensive historical information on earnings sources of all prospective BOND subjects that can be used to construct annual employment and earnings estimates. The BOND team will work with SSA staff throughout the demonstration to track impacts on employment and earnings. SSA staff have direct access to MEF data, but contractors do not, because the data are collected by the IRS and are therefore subject to IRS access rules. Consequently, SSA staff will access the data, submit programs developed by the BOND team to measure impacts, review output in collaboration with SSA researchers to ensure that it complies with privacy requirements, and then summarize the findings for the BOND team. The MEF earnings data are updated annually, with more than 90 percent of the records updated by August of the following calendar year. They are complete by the following February. Hence, the lag in obtaining earnings information is approximately 9 to 14 months.41
39 For more details on the data elements in the TRF, see Page et al. (2009).
40 There are two SSDI benefit variables in the TRF, both of which are of interest to the evaluation because of the way that BOND will affect benefit payments. The first benefit variable for each month is the “amount paid,”
which represents the benefit actually sent to the beneficiary. The second is the “amount due,” which represents the amount that SSA is scheduled to pay the beneficiary based on the individual’s current status. The benefit amount paid and due can differ if there are changes in the beneficiary’s status. For example, if SSA
retroactively finds that a beneficiary had engaged in SGA during a past month and had already completed the TWP and grace period, the amount due will be zero, but the amount paid will reflect the benefit actually paid.
Collection of overpayments in later months can result in the amount paid being less than the amount due. The BOND team plans to use the “amount paid” variable as the primary way of measuring the benefit cost to SSA.
The BOND team will also test the sensitivity of the results to using the amount due variable and report whether any differentials exist. As discussed in Chapter Two, the BOND payment system seems likely to reduce overpayments; if so, the impact on benefits paid will likely differ from the impact on benefits due.
41 The BOND team will report impacts to SSA in a series of internal and external reports (see Chapter Eight for more details). To obtain timely information, the evaluation team will use the nearly complete data that are available from the MEF in August to provide SSA with preliminary findings. The evaluation team will validate
3.1.3 Other Administrative Sources
Finally, the BOND team will incorporate information from other administrative sources on State Vocational Rehabilitation Agency (SVRA) services, Medicare, and Medicaid participation. The RSA maintains administrative records on SVRA applicants and types of services provided to participants. The CMS maintains databases on the use of Medicare and Medicaid, including eligibility and claims
information for both programs. The CMS Medicare data are available with a short lag (approximately six months), but the lags in obtaining the state VR and Medicaid data are much longer, 27 and 39 months, respectively.42 This long lag means that information on most SVRA service and Medicaid outcomes will only be reported in later evaluation reports.43
3.1.4 Data Development and Quality
The BOND team will rely on variables from the administrative data sources above that have been used in previous reports for SSA and other agencies using linked TRF, CMS and/or RSA data (e.g., Thornton et al. 2007, Gimm et al. 2009, and Stapleton et al. 2010b). Based on previous experience, the data quality for the variables included in the analytic tables in subsequent chapters is very high, as most fields have relatively limited missing data and provide information that is important for operational purposes. The surveys described in the next section focus on information that is either not available in administrative records, or of inadequate quality.44
these findings against the complete data that are available in February of the following calendar year to ensure that no bias was introduced by using the earlier data.
42 RSA administrative records have approximately a 27 month lag in data availability. RSA records on VR participation only become available after a case is closed, and case closure may not occur until several years after service enrollment. For example, GAO found that individuals who receive services spend on average two to three years receiving services, and many are involved for longer; about 90 percent of individuals who begin SVRA services finish services within five years (GAO 2007). Medicare utilization and claims data are rolled up from various carriers and fiscal intermediaries and, by June of each year, 99 percent of the annual claims for the prior year are compiled. Thus, for instance, data from calendar year 2010 are likely to be essentially complete and available no later than July 2011. The Medicaid data also include eligibility, enrollment, and claims information, though there is a longer analytic lag of more than two years because these data are only updated after all state Medicaid agencies submit their files to CMS.
43 Assignment of Tickets to SVRA and Ticket payments to SVRA will be observed more quickly, via SSA data. A beneficiary can, however, obtain SVRA services without assigning his or her Ticket. In the future, SSA expects to receive more timely reports of entry into SVRA services even if the beneficiary’s Ticket is not assigned to the SVRA as a result of regulations for the TTW program that were implemented in 2008. The RSA data will, however, still be the only source of information on services delivered and SVRA costs.
44 Education is an example of a low quality variable in administrative records; it is often missing because it does not pay an important role in most disability determinations or other administrative processes.
Abt Associates Inc. Evaluation Analysis Plan 52 Exhibit 3-1. Summary of Selected Variables from Administrative Files
Potential Variables SSA Administrative Program Files (Ticket Research File) SSA Earnings (Master Earnings File) RSA and CMS Administrative Data Beneficiary Characteristics
Demographic characteristics (age, race, gender)
Diagnosis, impairment status
Historical program information
Historical Earnings Information
Program Participation and Earnings
DI eligibility, SSI eligibility, benefit amounts, program exits
Use of work incentives (IRWE, Ticket to Work)
Earnings above SGA, TWP, and EPE
Annual earnings
Use of Any Vocational Rehabilitation Service *
Type of Vocational Rehabilitation Service
Medicare eligibility
Annual Medicare expenditures and service utilization
Medicaid eligibility
Annual Medicaid expenditures and service utilization
Use of Medicaid Buy-In
Analytic Lag in Data Availability (for data through December 31) 2 Months 9 – 14
Months Varies Notes: The other administrative data include information from RSA, Medicare, and Medicaid administrative records that might be used in later reports. All lags are estimates based on previous work. The actual duration of the lags may be shorter, especially if improvements are made in data collection methodology, or longer, if RSA, Medicare, and Medicaid records are available with an estimated 6, 21, and 39 month lag, respectively.
* In the future, SVRAs may report service entry for all beneficiaries directly to SSA in a routine and timely manner.