Taking research to practice: Program design and evaluation LESSONS FROM THE DOORWAY PILOT

(1)

LESSONS FROM THE DOORWAY PILOT

Taking research to practice: Program

design and evaluation

(2)

Overview of Doorway

• Designed to enhance the capacity of individuals with a

serious mental illness who were homeless or at risk of

homelessness to lead independent, healthy and meaningful

lives in stable housing within communities of their choice

• Pilot was implemented by MI Fellowship with funding from

the Victorian Department of Health (now DHHS)

• Pilot housed and supported 59 individuals over a three year

period between 2011 and 2014 in two metro and one

(5)

Demand for the program

Chose not to continue Left Doorway 77 105 59 50 18 9 Referrals

received Intake Housed

Housed (11/13)

?

Queries

(6)

Core elements of the model

• Serious mental illness and requiring service from an AMHS • Homeless or at risk of homelessness

• Eligible for segment 1 of the public housing waiting list (but may not be currently on the list) • Currently case-managed by AMHS

• Want to live in the designated area • Willing to accept support

• Willing to contribute 30% of household income to rent

Choice Social Inclusion Sustainability Eligibility

criteria

Participant

• Employment Consultant • AOD worker

• Physical health professionals • Cultural and spiritual supports • Family members, friends and community

members

• Case manager from AMHS • Housing & Recovery Worker

Flexible elements Core elements

Integrated team

(7)

Key enhancements to Housing First

Adaptations that reflect Doorway’s local operating context

• Integrated teams

• Emphasis on psychosocial support

Adaptations intended to improve non–housing outcomes

• Focus on natural support networks • Specialist employment assistance

Adaptations intended to enhance sustainability

• Open rental market

• Participants hold the lease

(8)

• The average time in bed-based clinical mental health services per participant per year decreased by half in the 12 months pre- and post-housing

• The total Emergency Department presentations across all

participants in the 12 months pre- and post-housing decreased by one third

• The mental health of one-third of participants improved to the point of their being able to be discharged from their Area

Mental Health Service

• Participants largely attributed their improved mental health outcomes to having stable accommodation and an integrated support team – two firsts for many participants.

(9)

Economic outcomes

• The percentage of participants engaged in paid

and unpaid work increased from 16% to 27% at

the time of the evaluation

• The proportions of Doorway participants

accessing education and vocational training

opportunities and receiving qualifications whilst

in the program increased

(10)

Housing outcomes

• Most participants reported feeling more

independent, having greater levels of self-respect

and pride and finding greater meaning in their lives

as a direct result of having more stable and secure

accommodation

• The number of tenancy related incidents was

relatively low, with six lease breaks by participants,

ten breach of duty notices and no evictions

• Participants developed tenancy related skills,

including money management

(11)

Comparison of housing costs

Figure 1: Comparison of housing costs (2010-2011 data where available)

^ All social housing options include the cost of capital to Government – with the exception of community housing given that data is not available. $10,136 $26,802 $9,417 $16,060 $28,105 $21,900 Doorway housing (average p/a)

Public housing^ Community housing Crisis accomm -Hostel style^ Crisis accomm/transitional housing^ Other supported accomm^

(12)

(13)

(14)

(15)

nousgroup.com 15

Our work on the Doorway pilot program taught us four

important lessons as evaluators

1. The importance of commencing evaluation work before a program is implemented

2. The benefits of genuine partnership between the evaluator and implementing agency

3. The value of being able to draw on multiple data sources to evaluate progress

(16)

nousgroup.com 16

• Program logic framework completed before first participant joined • Evaluation tools purpose built to align with indicators

• Data collection practices embedded into service delivery • Able to see the evolution of the story from day one

Lesson 1: The importance of commencing evaluation work

(17)

nousgroup.com 17

• Composition of teams remained unchanged • Presented evaluation findings on monthly basis • Interaction across all levels of MI Fellowship • Genuine value in reflective practice

• Able to evolve and enhance model as new evidence emerged

Lesson 2: The benefits of genuine partnership between

(18)

nousgroup.com 18

• Diverse array of data sources

• Reduced dependence on single data sources

• Multiple avenues for exploring and validating outcomes

Lesson 3: The value of being able to draw on multiple data

sources to evaluate progress

Quantitative data Qualitative data

• Six monthly data collection by staff • Outcomes measurement tools

• Department of Health (Vic) datasets (CMI-ODS, VAED, VEMD)

• Six monthly data collection by staff • Participant and carer focus groups • Key stakeholder interviews

(19)

nousgroup.com 19

• MI Fellowship as the primary client

• DHHS’s role as independent but very interested funders • Strong levels of buy-in by Doorway participants

Taking research to practice: Program design and evaluation LESSONS FROM THE DOORWAY PILOT

LESSONS FROM THE DOORWAY PILOT