In summer 2000 a survey of all the New Deal for Disabled People pilot areas was
conducted. This included Employment Service areas and Contract areas. The main aims of the pilot area surveys (in both types of pilot areas) were to:
• establish the differences between those who participated in the New Deal for Disabled People Personal Adviser Service and those who did not;
• identify people’s responses to their interviews and dealings with the Personal Adviser Service and the help offered to them; and
• consider the range of activities people had carried out over a two year observation period, including any events since their participation in the programme.
The original intention had been to select a large number cases to allow only small effects of NDDP to be detected. However, constraints on resources available for the study at the time the contract was awarded led to a reduction in the sample sizes for the ES and Contract areas studies to be reduced to 1050 participants and 300 non-participants. Because an early survey had already been conducted in the Employment Service pilot areas, the research and sampling strategy for the survey in these areas differed from the one in the Contract Areas.
In the Employment Service areas the intention had been to conduct follow ups with individuals interviewed in 1999. In practice, there were insufficient cases identified for interview in the early survey because of the low levels of participation in the programme. As a result, the sample for this study comprised follow-up interviews and interviews with a boost sample.
• follow-up interviews with participants and non-participants who responded to the early survey (Sample 1). These people had been invited to take part in the Personal Adviser Service between mid-January and mid-May 1999 or had had an interview with an Adviser between March and July 1999. In effect, the sample was drawn across a five- month period, beginning six months after the launch of the programme; and
• interviews with a booster sample of New Deal for Disabled People participants in the six Employment Service areas (Sample 4). These people had had an interview with an Adviser between August or September 1999.
The sample drawn from the contract areas (Sample 2) involved interviews with both participants and non-participants. The sampling window was between September 1999 and January 2000 for participants (and six weeks earlier for non-participants to allow for the lag between invitation and participation).
In the following sections we provide some more detail about the sampling strategy in the Employment Service areas and the Contract areas in turn.
3.3.1 Selecting the follow-up interviews in ES pilot areas (Sample 1)
The aim of selecting Sample 1 was to re-contact participants and non-participants already interviewed in the Employment Service areas to understand what had happened since their first interview and identify any labour market outcomes.
Administrative records suggested that in the early survey we had interviewed 369 non- participants against a target of 375 (which was intended to ensure a yield of 300 interviews at this follow up stage, based on an 80 per cent response rate). However, the early survey actually identified a much larger number, 554 individuals, who said they had not
participated in the programme. This was because many people who were identified as participants on the database actually appeared to be wrongly classified and did not have direct experience of the programme. Alternatively they may not have been aware that someone they had spoken to was a New Deal adviser. To avoid interviewing too many non-participants in the main stage survey, we randomly sub-sampled 300 of the 554 non- participants and issued these for interview as part of Sample 1.
A more important result of the discrepancy between administrative and survey data was that we consequently had a shortfall of participants who were interviewed at the first stage of interviews. Of 893 individuals who we believed to be participants, only 818 individuals identified themselves as such in the survey. Clearly, all 818 were issued as members of Sample 1. Even more significant, however, was the fact that the 893 achieved sample fell substantially short of the original target of 1300 participants. This was largely because insufficient sample was provided because of low levels of participation in the programme. The target that had been agreed in July 1999 at the start of the project in light of the prevailing uptake was 1300 since we believed this would generate a second stage achieved sample of approximately 1040 participants assuming an 80 per cent response rate for the second stage pilot area surveys. Because of the shortfall we elected to select a booster sample of participants and decided to administer to this sample, a hybrid version of the full NDDP survey.
It is important to remember the original sampling window from which those individuals who were interviewed in the early survey and carried forward to the main pilot area surveys were drawn. Sample 1 included participants and non-participants who had been sent invitation letters (but had not participated for a minimum period of six weeks) between mid- January and mid-May. The sample also included those who had a NDDP interview
between March and July 1999. In effect, the sample was drawn across a five-month period, beginning six months after the launch of the programme in September 1998.
3.3.2 Selecting the participant booster sample in ES areas (Sample 4)
As explained above, the target sample for the survey in the Employment Service pilot areas was 1050 achieved interviews with participants and 300 achieved interviews with non- participants. The aim of selecting sample type 4 was to boost the achieved sample of participants in the ES pilots to the level agreed in July 1999.
In effect, 723 additional participants were selected to allow for a 7 per cent opt-out rate and a 68 per cent field response rate. We also assumed that 17 per cent of people defined by the administrative database as participants would actually turn out to be non-participants. All of the individuals selected for the participant booster sample were recorded as having had an interview with an Adviser between August or September 1999.
3.3.3 Selecting the sample in Contractor led pilot areas (Sample 2)
The aim of selecting Sample 2 was to interview participants and non-participants in the Contract areas to provide a basis for comparison with participants and non-participants in the Employment Service pilot areas. The target sample for the Contract areas was 1350; split 1050 participants and 300 non-participants9.
And the sampling window was fixed between September 1999 and January 2000, a five-month window that mimicked the Employment Service pilot area window. As before, non-participation was defined when, six week after the issue of the invitation letter, a client had not contacted the Personal Adviser Service.
We selected 2,164 individuals from the Contract area databases to allow for a 9 per cent opt out and a 68 per cent field response rate.
3.3.4 The pilot area samples are not stratified by stock and flow
When carrying out the surveys in the Employment Service pilot areas, we had intended to sample participants and non-participants from the stock of claimants already receiving eligible benefits and from the flow of claimants passing the 28 week eligibility threshold in the ratio of 55:45. Shortage of numbers meant that no such stratification took place. Indeed the administrative data failed to record whether non-participants belonged to the stock or flow. In practice, the survey data from the Employment Service pilot areas
suggests that about 75 per cent of the total sample was comprised of stock, 24 per cent of flow and the remaining one per cent of retention cases or those that were known.
Because it was not possible to stratify the sample by stock and flow for the Employment Service pilot areas, a decision was made to abandon this in the Contract areas.