Random Assignment and Sample Development

The cornerstone of the evaluation of Project GATE is random assignment. Those who attended a GATE orientation, submitted a GATE application, and were found eligible for Project GATE were randomly assigned to either the program group or the control group. Members of the program group were offered Project GATE services; members of the control group were not offered Project GATE services.

The use of random assignment ensures that the applicants assigned to the program group will have, on average, the same observable and unobservable characteristics as applicants assigned to the control group. As a result, any differences in outcomes between the program and control groups can be directly attributed to Project GATE with a known degree of statistical precision. Without random assignment, there is always a concern that any differences in outcomes between the program and control group members are a result of differences in their underlying characteristics rather than program participation.

2.2.1 The Counterfactual

The goal of any program evaluation is to provide an estimate of the effects of a program by

comparing outcomes of program participants to what would have happened to them in the absence of the program. Since it is impossible to do this, an experimental evaluation uses the control group

as a counterfactual that credibly represents what would have happened to program participants had they not been offered GATE services.

Control group members could not participate in Project GATE—they could neither receive a GATE assessment nor be referred to a GATE provider for free business counseling and/or

classroom training. However, control group members were not prevented from receiving any other self-employment services offered in the community. Hence, they could receive services from providers that were not chosen for Project GATE or did not want to participate in Project GATE. They could even receive services from the GATE providers. However, control group members needed to find these providers on their own and may have had to pay for the services. The names of the GATE providers were provided only to GATE program group members after random assignment; control group members were never given the names of any self-employment service providers.

The services that the control group members receive—the counterfactual—determine the research questions that the evaluation addresses. Hence, it is important to note that this evaluation does not address the impact of Project GATE compared to receiving no self-employment services. Instead, it addresses the more policy-relevant question: What is the effect of adding Project GATE to the array of self-employment services already offered in the communities?

2.2.2 Random Assignment Procedures

Individuals interested in participating in Project GATE were asked to register for the program and attend an orientation. At the orientation, they were shown a video that described the challenges of self-employment, the services provided by Project GATE, and the evaluation, including random assignment. Those still interested in the program after the orientations were asked to complete a nine-page application package. This package asked for information to determine eligibility for Project GATE, including information on the characteristics of the applicant and his or her business idea, detailed contact information, and the signature of the applicant confirming willingness to participate in the evaluation. The package was mailed to the evaluation contractor, which checked that the applicant was eligible, had completed most of the application package, had consented to participate in the study, and had not applied to Project GATE previously. Once the applicant had passed these checks, he or she was randomly assigned to the program group or the control group,

each with a probability of 50 percent. The evaluation contractor then notified the applicant by mail of their assignment to either the program group or the control group and sent a list of new program group members to the service providers. After the orientation, applicants took an average of 2.0 weeks to complete an application and mail it to the evaluation contractor. On average, they were randomly assigned less than one week later.

2.2.3 Sample Enrollment

Nearly all GATE applicants were randomly assigned. Only three applicants were found ineligible, because their business idea was inappropriate for federal funding. In total, 4,198 applicants were randomly assigned between September 2003 and July 2005 (see Table II.1); just under half were assigned to the program group and just over half were assigned to the control group.

Table II.1: Number of GATE Applicants by Site

Number of Applicants

Site Total Program Group Control Group

Philadelphia 1,179 601 578 Pittsburgh 595 288 307 Minneapolis/St. Paul 1,654 834 820 Northeast Minnesota 203 97 106 Maine 567 275 292 Total 4,198 2,095 2,103 More than two-thirds of all applicants were in two sites—Philadelphia and Minneapolis/St. Paul.

Less than one-fifth of the applicants were from the rural sites—northeast Minnesota and the three sites in Maine. This variation may be explained by various factors, including differences in density of population, business culture, economic environment, and demographic characteristics of the local population.

As expected, random assignment produced program and control groups whose members had similar background characteristics at baseline (Appendix Table A.1). Of the 121 characteristics examined,

program and control group members differed in 8 characteristics at the 5 percent level of statistical significance, which is close to what would be expected by chance. Of relevance, program group members were slightly younger, had received fewer weeks of unemployment insurance (UI) benefits in the past year, and were slightly less likely to have ever been self-employed. These differences were accounted for in the impact analyses.

2.2.4 Business Partners

Examination of the application data revealed that some applicants applied to Project GATE with their business partners. Of the 4,198 applicants, 245 applicants (about 6 percent of all applicants) reported on their application packages a plan for a business that was identical to that of another applicant. All but about 2 percent of these applicants lived with the person who had the same plan. While about 91 percent of these applicants had one other partner who applied, 9 percent were in partnerships with 2 to 4 other applicants. The 245 applicants represented 118 business partnerships.

The participation of people who work together on a business raises a concern about contamination in the evaluation. A control group member who had a business partner in the program group could benefit from any services or information received by his or her partner. Some GATE providers would even allow business partners to accompany the GATE participant to classes and business counseling sessions.

To avoid the problem of control group contamination, all members of the business partnership were deleted from the sample if: (a) at least one member of the business partnership was in the control group, (b) at least one other member was in the program group, and (c) these two individuals lived at the same address or submitted the same business idea on their GATE Application Form. Of the 118 business partnerships, 56 (47 percent) had at least one program group member and one control group member and were removed from the sample. These 56 business partnerships consisted of 120 applicants. The remaining sample of 4,078 people contained only business partners who were either all in the program group or all in the control group and hence were not at risk of

contamination. The remaining sample members in business partnerships were representative of the sample members who were removed from the sample and were reweighted such that the weighted sum of business partnerships was the same as it was in the original sample.