Policy Implementation and Improvement

Dean Fixsen, Karen Blase,

Allison Metz, Melissa Van Dyke

National Implementation Research Network

FPG Child Development Institute

University of North Carolina at Chapel Hill

Policy

Implementation

and Improvement

The history of implementation began with

concerns about Policy Implementation in

the 1930s

(Saetren, 2005; Norway)

•

Legislation often failed to produce public

benefits (Pressman & Wildavsky, 1973)

•

Good intentions often evoked unintended

reactions and outcomes (Rittel & Webber,

1973)

Legislation

Administration

Implementation

Services

Policy Implementation

Pu

b

lic

B

en

efi

t

Effective

Innovations

Effective

Implementation

Enabling

Contexts

Socially

Significant

Outcomes

•

Letting it happen

–

Recipients are accountable

•

Helping it happen

–

Recipients are accountable

•

Making it happen

–

Purposeful use of implementation practice

and science

–

Implementation Teams are accountable

Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou

(2004); Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)

!

Usable Innovations

!

Implementation Stages

!

Implementation Drivers

!

Improvement Cycles

!

Implementation Teams

!

Enabling Change

Active Implementation

Frameworks

©

Coaching

Training

Selection

Integrated &

Compensatory

Systems

Intervention

Facilitative

Administration

Decision

Support

Data

System

Adaptive

Technical

Leadership Drivers

Performance Assessment

(fidelity)

Drivers

Reliable Benefits

Consistent uses of

Innovations by Practitioners

OUTCOMES

(% of Participants who Demonstrate Knowledge, Demonstrate

new Skills in a Training Setting,

and Use new Skills in the Classroom)

TRAINING

COMPONENTS

Knowledge

Skill

Demonstration

Use in the

Classroom

Theory and

Discussion

10%

5%

0%

..+Demonstration

in Training

30%

20%

0%

…+ Practice &

Feedback in

Training

60%

60%

5%

…+ Coaching in

Classroom

95%

95%

95%

Joyce and Showers, 2002

Competency Drivers

Competency Drivers: DBT

Drivers

Competency Drivers

and Practitioner Turnover

N=153

Aarons et al., 2009

Implementa)on  

Drivers  

T1  

T2  

T3  

Selec)on  

1.44  

2.00*  

1.89*  

Training  

1.33  

1.50*  

1.10  

Coaching  

1.27  

1.73*  

1.83*  

Perf.  Assessment  

0.78  

1.34  

2.00*  

DSDS  

0.18  

1.36  

2.00*  

Fac.  Administra)on  

1.38  

2.00*  

2.00*  

Systems  Interven)on  

1.29  

1.86*  

2.00*  

Average  Composite  

Score  

1.1  

1.68*  

1.83*  

Fidelity  (%  of  cases)  

18%  

83%  

83%  

Metz et al. (2014)

Implementation Teams

Teams

0

10

20

30

40

50

60

70

80

90

100

Practitioner Development

Organization Development

O

p

er

ati

n

g

6+

Yr

s.

N = 84

N = 219

Fixsen, Blase, Timbers, & Wolf (2001)

17%

84%

Implementation Teams

Expert Impl. Team

NO Impl. Team

Effective

Effective use of

Implementation

Science & Practice

IMPLEMENTATION

IN

T

ER

VEN

T

IO

N

80%, 3 Yrs

14%, 17 Yrs

Balas & Boren, 2000

Green, 2008

Fixsen, Blase,

Timbers, & Wolf, 2001

Saldana &

Chamberlain, 2012

Letting it Happen

Helping it Happen

Implementation Teams

Teams

Effective

Innovations

Effective

Implementation

Enabling

Contexts

Socially

Significant

Outcomes

"  

Innovations are not used as intended

and with good outcomes

"  

Innovations are not sustained for a

useful period of time

"  

Innovations are not used on a scale

sufficient to impact social problems

Best Data Show These Methods, When Used

Alone, Are Necessary but Not Sufficient:

•

Diffusion/ Dissemination of information

•

Training

•

Passing laws/ mandates/ regulations

•

Providing funding/ incentives

•

Organization change/ reorganization

About 5% to 20% Realize Intended Benefits

Evidence-based

Actual Supports

Years 1-3

Outcomes

Years 4-5

Every Teacher

Trained

Fewer than 50% of

the teachers

received some

training

Fewer than 10% of

the schools used the

CSR as intended

Every Teacher

Continually

Supported

Fewer than 25% of

those teachers

received support

Vast majority of

students did

not benefit

Vernez, Karam, Mariano, & DeMartini, 2006

Longitudinal Studies of a Variety of Comprehensive School Reforms

Approach

Usage

Success

Months

Intervention/

Facilitation

8%

87%

14

Participation/

Internal Team

18%

73%

16

Persuasion

37%

47%

21

Edict

37%

35%

15

Nutt (2001)

Implementation Science

Approaches to implementation of innovations in 376

organizations (public, private, profit, non-profit)

“Public managers have to learn a range of

often new and detailed techniques in

order to implement what are often

ambiguous policy directives.”

Schofield (2004, p 283; UK)

(Sundell, Soydan, Tengvald, & Anttila, 2009; Sweden)

"All organizations [and systems] are

designed, intentionally or unwittingly,

to achieve precisely the results they

get.”

…R. Spencer Darling

Human Services are not improving as

fast as problems are changing.

National Assessment of Education Progress (NAEP)

Churning Around a Mediocre Mean

•

Innovative practices do not fare well in

existing organizational structures and

systems

•

Organizational and system changes are

essential to successful use of innovations

–

Expect it

– 

Plan for it

Existing System

Effective Innovations

Are Changed to

Fit The System

Existing System Is

Changed To Support

The Effectiveness Of

The Innovation

Effective Innovation

Enabling Contexts

Compliance  and  Crises,

Urgent,  Time  Sensi)ve!!  

•     

Services  not  mee)ng  Standards  

•

 Deal  with  urgent  and  high  profile  issues  

Best  Prac)ces  

Implemented  Fully    

With  Good  Outcomes  

In  a  Transforma)on  Zone

System  Supports  &  Stability

•   Regulatory  roles  

•   Basic  Data  Systems  

•   Financing  and  Fiscal  Accountability  

•   Accredita)on/  Licensing  Standards  

•   HR  rules  and  regula)ons  

•   Safety  Standards  

•   Work  with  Legislature  

•   Inclusion  of  Stakeholders  

System  Supports    

&  Stability  

Mandates,    

Opera)ng  Units,  

Founda)onal  Polices  &  Regula)ons  

Implementation

Team

Executive

Management

Team

Practitioners

Innovations

Beneficiaries

Sy

ste

m

Change

Adaptive Challenges

•

Duplication

•

Fragmentation

•

Hiring criteria

•

Salaries

•

Credentialing

•

Licensing

•

Time/ scheduling

•

Union contracts

•

RFP methods

•

Federal/ State laws

“Ex

te

rn

al

”

Sy

ste

m

C

h

an

g

e

Su

p

p

o

rt

Pr

ac

ti

ce

Po

lic

y

C

o

m

m

u

n

ic

ati

o

n

L

o

o

p

Po

lic

y En

ab

le

d

Pr

ac

ti

ce

“The fault

cannot lie in

the part

responsible

for the

repair.”

Ashby

(1956)

System Reinvention

Local Agency

LIT

Ministry

Management

Team

Practitioners

Innovations

Beneficiaries

“Ex

te

rn

al

”

Sy

ste

m

C

h

an

g

e

Su

p

p

o

rt

Pr

ac

ti

ce

-Po

lic

y

C

o

m

m

u

n

ic

ati

o

n

L

o

o

p

Po

lic

y En

ab

le

d

Pr

ac

ti

ce

Implementation Infrastructure

“The fault

cannot lie in

the part

responsible

for the

repair.”

Ashby

(1956)

Municipality

MIT

Regional Entity

RIT

State Implementation and Scaling up of

Evidence-Based Programs Center

• 

The purpose is to help establish

implementation and scaling capacity in

state, regional, and district educational

systems

• 

The goal is to maximize academic and

behavioral outcomes for all students

Implementation Infrastructure

State

ESDs

Districts

Schools

Linking Implementation Teams

1 State

9 Regions 295 Districts

2,275 Schools

100,000 Teachers and Staff

1 million students

State

Management

Team

Regional

Implementation

Team

“District” Teams

N

≈

100 Schools

Regional

Implementation

Team

“District” Teams

N

≈

100 Schools

Regional

Implementation

Team

“District” Teams

N

≈

100 Schools

Regional

Implementation

Team

“District” Teams

N

≈

100 Schools

SI

SEP

Su

p

p

o

rt

Transformation

State

Specialists

State

Design

Team

State

Management Team

Regional

Implementation

Team

State Transformation Specialists

State

Design

Team

Repurpose

Roles,

Functions, &

Structures to

Maximize

Outcomes

SI

SEP

a

n

d

ST

S

Su

p

p

o

rt

State  and  Regional  Capacity  

Assessment  

State Capacity Assessment: Fixsen, Duda, Blase and Horner, 2009

SMT = State

Management Team

RIT = Regional

Implementation Team

District Capacity Assessment

District Capacity Assessment: Duda, et al. 2012

District #5

District Capacity Assessment

District Capacity Assessment: Duda, et al. 2012

District #6

District Capacity Assessment

District Capacity Assessment: Duda, et al. 2012

District #7

OTISS Data (n=18 teachers)

Clear Instruction Demostrate Interactions Prompt Feedback Adjusts Response Practice Opportunities Student Engagement

School 1 School 2 School 3 School 4 School 5

National Assessment of Education Progress (NAEP)

Leverage Visible Improvements!

Original and Revised Assessment Formats

Educationally Significant Outcomes Effective Innovations Effective Implementation Enabling Contexts

•

Stop wasting time and money on things

that don’t work (and never have!)

–

Sugai’s Law: For every new initiative, stop

two current ones that have poor outcomes

–

De-scale; Avoid layering and fragmentation

•

Set aside 15% of funds for implementation

•

Require quarterly reports of fidelity data

Effective

Innovations

Effective

Implementation

Enabling

Contexts

Socially

Significant

Outcomes

!

Usable Innovations

!

Implementation Stages

!

Implementation Drivers

!

Improvement Cycles

!

Implementation Teams

!

Enabling Change

Active Implementation

Frameworks

The European Implementation

Collaborative (EIC)

•

Interested in the EIC?

• 

Contact Bianca Albers

GIC May 26-28, 2015

Dublin, Ireland

Allison Metz

–

[email protected]

Melissa Van Dyke

–

[email protected]

Frank Porter Graham Child Development Institute

University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/

www.scalingup.org

HTTP://NIRN.FPG.UNC.EDU

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).

Implementation Research: A Synthesis of the Literature.

Tampa, FL: University of

South Florida, Louis de la Parte Florida Mental Health Institute, The National

Implementation Research Network (FMHI Publication #231).

Implementation

Research:

A Synthesis of

the Literature

Get Connected!

www.scalingup.org

SISEP

@SISEPcenter

For more on Implementation Science

http://nirn.fpg.unc.edu

• Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of

evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77(2), 270-280. doi: 10.1037/a0013223

• Ashby, W. R. (1956). An introduction to cybernetics. London: Chapman & Hall.

• Balas, E. A., & Boren, S. A. (2000). Managing clinical knowledge for health care improvement. In J. Bemmel & A.

T. McCray (Eds.), Yearbook of Medical Informatics 2000: Patient-Centered Systems. (pp. 65-70). Stuttgart, Germany: Schattauer Verlagsgesellschaft.

• Deming, W. E. (1986). Out of the crisis. Cambridge, MA: MIT Press.

• Elmore, R. (2002). Bridging the gap between standards and achievement: The imperative for professional

development in education. Washington, DC: The Albert Shanker Institute.

• Fixsen, D. L., Blase, K. A., Timbers, G. D., & Wolf, M. M. (2001). In search of program implementation: 792

replications of the Teaching-Family Model. In G. A. Bernfeld, D. P. Farrington & A. W. Leschied (Eds.), Offender rehabilitation in practice: Implementing and evaluating effective programs (pp. 149-166). London: Wiley.

• Fixsen, D. L., Blase, K., Duda, M., Naoom, S., & Van Dyke, M. (2010). Implementation of evidence-based

treatments for children and adolescents: Research findings and their implications for the future. In J. Weisz & A.

Kazdin (Eds.), Implementation and dissemination: Extending treatments to new populations and new settings (2nd

ed., pp. 435-450). New York: Guilford Press.

• Green, L. W. (2008). Making research relevant: if it is an evidence-based practice, where’s the practice-based

evidence? Family Practice, 25, 20-24.

• Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service

• Grigg, W. S., Daane, M. C., Jin, Y., & Campbell, J. R. (2003). The nation's report card: Reading 2002.

Washington, DC: U.S. Department of Education, Institute of Education Sciences.

• Higgins, M., Weiner, J., & Young, L. (2012). Implementation teams: A new lever for organizational change.

Journal of Organizational Behavior. doi:10.1002/job.1773

• Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.

• Kutner, M., Greenberg, E., Jin, Y., Boyle, B., Hsu, Y., & Dunleavy, E. (2007). Literacy in everyday life: Results

from the 2003 National Assessment of Adult Literacy (NCES 2007–480). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

• Linehan, M. M., Dimeff, L. A., Reynolds, S. K., Comtois, K., Welch, S. S., Heagerty, P., & Kivlahan, D. R. (2002).

Dialectical Behavior Therapy Versus Comprehensive Validation Therapy Plus 12-Step for the Treatment of Opioid Dependent Women Meeting Criteria for Borderline Personality Disorder. Drug and Alcohol Dependence, 67(1). doi: 10.1016/S0376-8716(02)00011-X

• Kivlahan, D. R. (2002). Dialectical Behavior Therapy Versus Comprehensive Validation Therapy Plus 12-Step for

the Treatment of Opioid Dependent Women Meeting Criteria for Borderline Personality Disorder. Drug and Alcohol

Dependence, 67(1). doi: 10.1016/S0376-8716(02)00011-X

• Marzano, R., Waters, T., & McNulty, B. (2005). School leadership that works: From research to results.

Alexandria, VA: Association for Supervision and Curriculum Development (ASCD).

• Metz, A., Bartley, L., Ball, H., Wilson, D., Naoom, S., & Redmond, P. (2014). Active Implementation Frameworks

(AIF) for successful service delivery: Catawba County child wellbeing project. Research on Social Work Practice, 1-8. doi: 10.1177/1049731514543667

• Morgan, G., & Ramirez, R. (1983). Action learning: A holographic metaphor for guiding social change. Human

Relations, 37(1), 1-28.

• Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved April 22, 2007, from http://www.useit.com/ alertbox/20000319.html.

• Nutt, P. (2002). Why decisions fail: Avoiding the blunders and traps that lead to debacles. San Francisco: Berrett-Koehler Publishers Inc.

• Nutt, P. C. (2001). Contingency approaches applied to the implementation of strategic decisions. International

Journal of Business, 6(1), 53-84.

• Pressman, J. L., & Wildavsky, A. (1973). Implementation: How Great Expectations in Washington are Dashed in

Oakland. Berkeley, CA: University of California Press.

• Rhim, L. M., Kowal, J. M., Hassel, B. C., & Hassel, E. A. (2007). School turnarounds: A review of the cross-sector

evidence on dramatic organizational improvement. Lincoln, IL: Public Impact, Academic Development Institute. • Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155-169.

• Saetren, H. (2005). Facts and myths about research on public policy implementation: Out-of-fashion, allegedly dead, but still very much alive and relevant. The Policy Studies Journal, 33(4), 559-582.

• Saldana, L., Chamberlain, P., Wang, W., & Brown, H. C. (2012). Predicting program start-up using the stages of

implementation measure. Administration and Policy in Mental Health, 39(6), 419-425. doi: 10.1007/ s10488-011-0363-y

• Schofield, J. (2004). A model of learned implementation. Public Administration, 82(2), 283-308.

• Stacey, R. D. (1996). Complexity and creativity in organizations. San Francisco: Berrett-Koehler Publishers.

• Sundell, K., Soydan, H., Tengvald, K., & Anttila, S. (2009). From opinion-based to evidence-based social work:

The Swedish case. Research on Social Work Practice. doi: 10.1177/1049731509347887

• Vernez, G., Karam, R., Mariano, L. T., & DeMartini, C. (2006). Evaluating comprehensive school reform models

©Copyright Dean Fixsen and Karen Blase

This content is licensed under Creative Commons license CC

BY-NC-ND, Attribution-NonCommercial-NoDerivs. You are free to

share, copy, distribute and transmit the work under the following

conditions: Attribution — You must attribute the work in the

manner specified by the author or licensor (but not in any way that

suggests that they endorse you or your use of the work);

Noncommercial — You may not use this work for commercial

purposes; No Derivative Works — You may not alter or transform

this work. Any of the above conditions can be waived if you get

permission from the copyright holder.