Impact analysis methodology

The impact of NDDP is assessed by comparing the experiences of participants with a matched group of incapacity-related benefit recipients who did not register for the programme. The impact analysis uses a non-experimental comparison, or matched estimation methodology, that was identified following secondary analysis of a similar US programme evaluated using random assignment (for further details see Orr et al. (2004)). The analysis is based on the following three cohorts of claimants using benefit and employment administrative data (Orr et al., 2007:6).50

Table 5.1 Cohorts of claimants using benefit and employment administrative data

Number of months over which impacts assessed Cohort Recipients’ registration period (‘follow-up period’)

Early 1 July 2001 to 31 December 2002 24 Maximum Follow-up 1 July 2001 to 31 December 2001 36 Late 1 January 2004 to 30 June 2004 6

The Early Cohort is designed to provide estimates of NDDP as the programme was originally devised, whilst the Late Cohort provides an initial assessment of the programme’s effectiveness following the policy changes introduced in 2003 (see Section 1.3.2). The Maximum Follow-up Cohort is a sub-set of the Early Cohort that allows examination of impacts over the longest period of time since registration (36 months).

50 _{The benefit data (for incapacity-related benefits and Jobseeker’s Allowance) was} provided by the Department for Work and Pensions and the employment data by HM Revenue and Customs (HMRC).

Each cohort comprises two groups: more recent claimants and longer-term claimants. Except for the Late Cohort, more recent claimants made a claim for benefit during the relevant registration period, whilst longer-term claimants had done so at the beginning of the registration period. For the Late Cohort the dividing line between recent and longer-term claimants falls one year prior to the beginning of the registration period, that is, 1 January 2003.

The non-participant comparison group is drawn from the eligible population and is also divided into more recent and longer-term claimants. Impacts on participants are dated from the month of registration on NDDP. By definition non-participants do not have a month of registration. Accordingly, a ‘start date’ has been randomly assigned for each non-participant based on the distribution of registration months for participants who entered benefit at the same time. This procedure ensured that non- participants had a start date comparable to the registration date for participants. (Further details of the method used are given in Orr et al., (2007:8).)

Participants and non-participants were matched on the basis of personal characteristics and benefit history (Orr et al., 2007:9).51 _{For each cohort the matching exercise} produced the following number of cases for analysis:

51 _{The participants were matched to non-participants on the basis of:} • registration/start month;

• years of prior benefit receipt (including all spells);

• type of incapacity benefit received at registration/starting month; • age at registration/starting month;

• sex;

• type of disability;

• DWP administrative region;

• (for new claimants) month in which benefit receipt began; and • (for new claimants) whether there was a work-focused interview.

Up to 10 non-participants were matched to each participant, and each non- participant was matched with only one participant. The non-participant data were also re-weighted to ensure that the comparison group mimics the demographic characteristics of the participant sample. Carrying these weights into the impact analysis assures that the outcomes of the two groups are as similar as available data can make them apart from the effects of the intervention, thus reducing the potential for ‘selection bias’ to distort the research findings when differences in outcomes between the two samples are interpreted as effects of the programme.

Table 5.2 Matching exercise for each cohort

Participants Non-participants Longer-term More recent Longer-term More recent

recipients claimants recipients claimants

Early Cohort 23,696 5,585 211,782 49,354 Maximum follow-up Cohort 5,635 295 48,607 3,164 Late Cohort 12,116 6,722 109,823 55,867

To further facilitate the comparison, multivariate analysis was used to control for other baseline differences between the two samples.

Average net impacts on more recent and longer-term claimants in each cohort were estimated for the following six outcomes for each month of the relevant follow-up period:

• receipt of Incapacity Benefit, Income Support with disability premium or Severe Disablement Allowance (that is, incapacity-related benefits);

• monthly amount of combined Incapacity Benefit, Income Support with disability premium and Severe Disablement Allowance;

• receipt of Jobseeker’s Allowance;52

• monthly amount of Jobseeker’s Allowance; • rate of employment; and

• proportion of the follow-up period employed.53

The Early Cohort was also used to analyse the impact of NDDP on various sub-groups (defined by participants’ age, type of disability or health condition, length of benefit spell, nearness to labour market and Job Broker and local community characteristics). The analysis looked at the impact of NDDP on the amount and receipt of incapacity- related benefits and employment rates. The analysis identifies which sub-groups of participants benefited more or less from NDDP.

52 _{People cannot receive Jobseeker’s Allowance and incapacity-related benefits at} the same time.

53 _{The employment data used for the last two of these outcomes may omit or} inexactly record some people’s spells of employment. The administrative data used do not include jobs providing earnings below the tax and National Insurance thresholds, and in some cases contain unspecified start dates for jobs that are coded to the start of the tax year and unspecified end dates coded to the end of the tax year. It is also possible for errors in identifying information to occasionally result in mismatches of jobs to individuals and to result in the omission of those jobs from the data. Comparisons with the Survey of Registrants and Job Broker records suggest the data are fairly complete.

As part of the sub-group analysis, Orr et al., (2007:56) developed a measure of ‘distance from the labour market’ for participants and eligible non-participants. A regression model was used to predict the likelihood of employment at any point during the Early Cohort’s two-year follow-up period in the absence of NDDP. The model was based on a random 50 per cent sub-sample of non-participants, which suggested what patterns of past benefit receipt and employment, as well as what demographic characteristics were associated with a higher or lower probability of getting a job in the absence of the assistance of a Job Broker. The model was used to predict which of the remaining non-participants, and which participants, are among the most or least likely to gain employment on their own. The top third of the distribution of predicted probability of employment for participants is considered ‘nearest to the labour market’ within the population served by NDDP, and the bottom third of that distribution classified as ‘furthest from the labour market’, leaving a middle tercile as well. Non-participants were then selected for comparison to each of these three subgroups by focusing on individuals with predicted probabilities in the same numeric range.

The benefit recipients found to be nearest the labour market were those who at baseline:

• received Severe Disablement Allowance, rather than Incapacity Benefit or Income Support with a Disability Premium;

• had been on benefits for a shorter duration, and for fewer months over their lifetimes;

• had received benefits intermittently rather than continuously, and did so in relatively recent months and years rather than at a younger age;

• had participated in a Work Focused Interview (WFI); • were aged under age 50;

• were male; and

• suffered from a circulatory/cardiac, musculo-skeletal, or injury/poison-related disability (mental and behavioural disabilities fell into the middle range, with nervous system and other types of disabilities furthest from the labour market). Initial benefit amount and geographic region also related significantly to estimated labour market closeness but in complex ways.

Although the estimation methods produce the best estimates possible with available data, it should be noted that an analysis of data from the Survey of Registrants for a small subset of this population suggests that the inability to control for individual characteristics and circumstances not included in the administrative data, may have caused a bias away from zero accounting for about one-third of the measured effect in the estimates presented here (Orr et al., 2007:16 and 20-21). For this reason, the estimates presented here should be viewed with some caution, although the conclusion remains that NDDP did have a significant net impact.