Improving Research Designs using Observational Data

Recent meta-analyses show that prison-based postsecondary education programs do reduce recidivism rates, so it is not clear why education programs did not have any long-term effects on participants’ ability to maintain employment or reduce criminal involvement (Davis et al., 2013; Duwe & Clark, 2014). Null findings may reveal ineffective programs, but they may also reflect low rates of participation among men receiving education and employment services. Future research should examine program dosage, in terms of program intensity, duration, content, and delivery method (Davis et al., 2013; Steurer et al., 2001). Existing studies have not consistently differentiated education programs by type, so it is not clear whether some types of correctional education yield more benefits than others (Steurer et al., 2001).

Regression Discontinuity

This study used propensity score weighting to balance participants and nonparticipants, but this method only reduces observed heterogeneity. Regression discontinuity designs can diminish

138

observed and unobserved heterogeneity, thereby increasing the methodological rigor of future program evaluations. To implement a regression discontinuity design, evaluators must have access to a continuous measure, such as Tests of Basic Adult Education (TABE), that reliably measures individuals’ latent ability. Evaluators then use a cut-point to assign respondents to the treatment and comparison groups; scores that fall above the cut-point are included in the

treatment group and scores that fall above the cut-point are assigned to the comparison group. If there is a linear association between latent ability (as measured by the continuous scale) and the observed outcome, then the difference in outcomes for sample members above and below the cut-point should reflect the true effect of the program (Davis et al., 2013).

Successful regression discontinuity designs may entail more planning during the research design stage than do some propensity score methods, which can be implemented successfully at the analysis stage. It can be difficult to find an appropriate continuous variable in administrative records, so program evaluators may need to modify the sample selection process to include testing. The internal validity of this method is reduced when program implementers violate the assignment rule, so it is critical that practitioners faithfully apply the rule when assigning participation status. Furthermore, poorly implemented or designed programs may not meet the linearity assumption (Davis et al., 2013).

Use of Smartphones to Collect Data

Recent studies have experimented with the use of text messaging to collect data from

respondents on a more frequent basis (Gaggioli et al., 2013; Sugie, 2014). This technique has been successfully implemented in a study following parolees during the first 3 months of release from prison (Sugie, 2014). Compared to men who were assigned to the traditional interview condition, men assigned to the smartphone condition were more likely to agree to participate in

139

the study (89% vs. 68%) and to remain involved in the project throughout the 3-month study period (Sugie, 2014).1 The completion rate for men who entered the study (82%) is higher than has been observed in other longitudinal studies following prisoners after release, including the SVORI evaluation (58% completion rate for the 3-month follow-up interview).

Furthermore, the use of smartphones enabled the study investigators to collect data on participants’ job search activities, work activity, and current mood each day. Participants received two text messages daily, each of which took only a few minutes to complete. The first was sent randomly between 9am and 6pm, and it included questions about participants’ current activity and mood. The second text was sent at 7pm and the questions addressed participants’ activities and mood for the whole day. To encourage smartphone respondents to complete each survey within an hour of receiving the text, participants received a $15 bonus for completing at least 75% of the interviews each week. As a result, men answered approximately 78% of all texts that they received, with two-thirds of the men reaching the 75% target completion rate. In general, smartphone respondents preferred the use of text messaging to weekly interviews and they enjoyed having access to a smartphone (Sugie, 2014).2

Most importantly, the smartphone interviews captured fine-grained data on changes in mood and job search activity during the early reentry period (Sugie, 2014). The variables for labor force participation, job search difficulties, and psychological distress were among the weakest variables in the current study, due to the lag in measurement periods for employment, the weak

1 _{These percentages reflect participation and completion rates for the full sample of eligible individuals. Attrition}

from the study was higher among smartphone participants than among interview participants (70% complete rate among smartphone users who completed the initial interview, compared to 86% for those completing interviews (Sugie, 2014).

2 _{Given the high use of no-contract cell phones among former prisoners, and the importance of maintaining a stable}

contact number during the job search, the subsidized smartphone provided a form of reentry intervention that may improve men’s likelihood of gaining employment (Sugie, 2014).

140

measure of labor force participation (e.g., worked at least once within a given month), and the narrow 7-day reference period for psychological distress measures. Using smartphones to capture men’s current status in non-research settings would provide far more reliable measures of job search activities and mood than by use of retrospective interviews (Gaggioli et al., 2013; Sugie, 2014). For instance, men reported higher levels of anger, sadness, and stress at the first interview than were reported during smartphone interviews, and men’s daily happiness levels showed substantial variability that was not captured by weekly interview responses (Sugie, 2014).

6.2.2 Testing Theoretical Concepts

Future research should examine the association between employment and crime using shorter observation periods, to assess whether the findings of this study hold with changes in lagged periods (Aaltonen et al., 2013; Horney et al., 1995; Skardhamar & Telle, 2012). Research that used monthly observation periods would be better equipped to assess whether and how

employment reduces crime or crime reduces employment (van der Geest et al., 2011). Detailed information on employment status, including the number of weeks worked, average number of hours, type of job, and reasons for labor force exit, would provide insight into the associations between labor force participation, job quality, and criminal activity (Sugie, 2014; van der Geest et al., 2011).

Future research should examine the temporal association between financial need/strain, criminal activity, and criminal justice involvement. The results of this study provide cautious support for the negative effects of ongoing criminal activity on men’s financial status, psychological

wellbeing, and employment status. Despite this, the results provide limited support to suggest that financial strain contributes to later criminal activity. The results presented also cannot

141

determine whether recent criminal activity was actually associated with current financial needs and psychological distress, and not vice versa, as strain theories would predict. Future research using shorter observation periods would provide stronger evidence in support of or against existing theories of work, crime, and financial need.

Future research with prisoner populations should include validated financial strain measures in surveys to be sure financial need/strain is being measured accurately. Research is also needed to assess whether financial strain is a relevant concept among young former prisoners, who may perceive limited need for goods and services commonly included in questions (e.g., inability to pay bills on time, inability to obtain medical assistance). It may be necessary to modify financial strain measures to reflect the social context faced by young and marginalized former prisoners.