Performance Measures Primer WIOA Performance. What We (now) Know What We (still) Don t Know What s Keeping Us Up at Night

(1)

Performance Measures Primer

WIOA Performance

What We (now) Know

What We (still) Don’t Know

(2)

Introduction



WIOA brings Massive Change – Even for Texas

Performance Measurement & Accountability

   

Replace Existing Common Measures Changes in State & Local Negotiations Bigger Consequences for Failure

Significant Reporting Requirements

 Now includes Eligible Training Provider (ETP) Reporting

Opportunities Exist but Opportunities bring

Complications

(3)

New Common Measures

 

NonYouth

      Employed 2nd _{Qtr Post-Exit}

Median Earnings 2nd _{Qtr Post-Exit}

Employed 4th _{Qtr Post-Exit}

Credential within 1 Year Measureable Skills Gain

“Indicators of Effectiveness of Service To Employers”

Youth

     

Employed or In Education 2nd _{Qtr Post-Exit}

Median Earnings 2nd _{Qtr Post-Exit}

Employed or In Education 4th _{Qtr Post-Exit}

Credential Rate

Measureable Skills Gain

(4)

Origin of New Common Measures



 

Integrated Performance Initiative (IPI)

  

DOL funded grant from 2004 Led by Washington State

Included a number of states & National Governors’ Assn

Goal to Develop Measures Acceptable to WF, K-12, &

Post-Secondary Education

Championed by NGA in all WIA Reauthorization drafts

Applicable to all Core Programs

  

Title I Adult, DW, Youth, Wagner-Peyser Adult Education & Literacy

Vocational Rehabilitation

(5)

Employed 2

nd

_{Qtr Post-Exit}

_(NonYouth)

    

Essentially Replaces “Entered Employment”

Includes ALL Participants, not just Unemployed

Likely called “Employment Rate”

Good

 Picks up instances where we help an employed person get

a 2nd _Job

Bad

 



Adds 1 Qtr of Lag to the Measurement Period

Counts employed people looking for new/2nd _{jobs even if}

they stay employed at old job

Dept of Ed seems to want their own definition for AEL but the NPRM doesn’t provide one

 

Would only include those “in the Labor Force”

(6)

Employed or In Education

2

nd

_{Qtr Post-Exit}

_(Youth)

    

Replaces “Placed in Employment/Education”

Includes ALL Participants, not just

Unemployed/Out-of-School

New measure counts ANY training/education – not

just post-secondary, counts those still in HS

Good

 Picks up instances where we help an employed person get

a 2nd _{Job or help an in-school person continue their}

education

Bad

  

Adds 1 Qtr of Lag to the Measurement Period Counts people who stay employed at old job

Similarly counts those already in school who remain in school even if we did little/nothing to support that

(7)

Median Wages 2

nd

_{Qtr Post-Exit}

_(All)

 



Essentially Replaces “Average Earnings”

Good

  

Shorter Lag than Average Earnings (2nd _{Qtr vs. 2}nd _{& 3}rd₎

Adds wage focused measure for Youth “Median” reduces influence of outliers

 Better for smaller Boards

Bad

 

Harder to Statistically Model

Smaller Numbers are less meaningful to Stakeholders

Unknown

 Will it include all Exiters or just those Employed?  Probably just those Employed

(8)

Employed 4

th

_{Qtr Post-Exit}

_(NonYouth)



IPI Measure & Statutory Language focus Exclusively

on 4

th

_{Qtr Post-Exit}

 Not a “retention” measure

Feds want input on value of reporting Retention too

 

Subset of those Employed 2nd _{Qtr Post-Exit}

If so, likely NOT an Accountability Measure

Good

 Reporting on those not employed until later

Bad

 

Counts people who stay employed at old job

Dept of Ed seems to want their own definition for AEL but the NPRM doesn’t provide one

 

Would only include those “in the Labor Force” Might try to get it in via guidance letter

(9)

Employed or In Education

4

th

Qtr Post-Exit (

Youth)

     

No analog to Existing Youth measures

IPI Measure & Statutory Language focus Exclusively

on 4

th

_{Qtr Post-Exit}

New measure counts ANY training/education – not

just post-secondary, counts those still in HS

NPRM doesn’t mention interest in a Retention

measure for Youth

Good

 Reporting on those employed/in education later

Bad

 Counts people who stay in school/employed at old job

(10)

Credential Within 1 Year

(All)

    

Will Replace Youth Attainment of Degree/Certificate

& could Replace Educational Achievement

Likely called “Credential Rate”

Good

 Having a NonYouth Credential measure

Bad

 

Lots of Lag

Extra Lag for Diploma/GED

 Only counts if also Employed or in Post-Secondary within 1 year of Exit – assumedly using wage records which means more lag

Unknown



How the Lag will work – Does “Within 1 year after Exit” mean 365 days or the 4th _quarter?

Any Occupational Training & Education or that which “leads to a recognized postsecondary credential or employment?”10

(11)

Measureable Skills Gain

(All)



 



Most Similar to Literacy/Numeracy Gains & AEL

Educational Gain

Likely called “Measureable Skills Gain Rate”

Good



Having an “in progress” measure for those in

training/education “for a specified reporting period”

Having a “shorter term” measure as a leading indicator for the longer term Credential Measure (little lag here)

Unknown

 

What is the “specified reporting period?”

What does “documented academic, technical, occupational or other forms of progress, towards such a credential or

employment” mean?

How to measure for non-academic training?

(12)

Effectiveness Serving Employers

(All)

     

Undefined in Statute

 Secretaries required to consult with States, Locals,

Employers, Stakeholders to develop

Measure(s) to be Developed by 6/30/16

Good



Having Natl Measures of Effectiveness Serving Employers

Possibly 1 measure/set rather than 1 per program

Bad

 

Starting from Scratch (for Texas)

Possibility that we get something like “Employers Served”

Unknown

 MANY Unknowns

NPRM Asked for input on Employment Connection

(13)

Employment Connection System



ECS is a system that uses UI Wage Records to identify

new Employment Connections between JS & ER

 EC assumed if wages reported in Current Q but not Prior Q

Indicator of Effectiveness Serving Employers

 

If Employer’s need is qualified workers then the EC is 1st _step

But turnover is a killer for Employers

“Maintaining Employment Connection” could be

Indicator of Effectiveness

 Employment Connection lasting at least 2 Qtrs after EC

MEC run for ALL Employment Connections in Texas



 

Not Just WF System Customers

Performance Standards based on REAL WORLD PERFORMANCE Can do analysis by industry!

 Do our connections last longer than the general populace?

(14)

State Negotiations



Negotiating Targets for 2 Years

 Assume 2nd year will focus on Improvement

Negotiations Must Consider

  

Government Performance & Results Act National Targets Targets of Other States

Continuous Improvement & ROI

 ROI? How will that be measured? NPRM doesn’t say!  Statistical Adjustment Model

 Economic & Demographic Factors 



Automatic Target Adjustments at End of Year

 Must reflect actual Economic Conditions & Participant

Characteristics

NPRM asks whether Negotiations should encompass

more than the DOL/Dept of Ed statistical models

₁₄

(15)

Statistical Adjustment Models

     

HUGE Challenges

Existing DOL Models are limited

 

WIA Only

Unemployment Rate is only Economic Variable

DOL has limited Wagner-Peyser Customer Data

Dept of Ed has NO AEL & Limited VR Customer Data

One Model for all States/Bds?

  

ESL has a different impact in El Paso & Detroit

Movie Industry Matters a lot in CA / not much in Montana Universal Coenrollment/Low % Training vs. “Classic” Model

Models need to be Regularly Updated



Impact of Oil/Gas in South Dakota 10 years ago vs. Today

NPRM requests input on how to Update

(16)

Local Negotiations



 

Negotiations Must Consider

 

State Targets

Local Economic & Participant Characteristics via Statistical Adjustment Model

Where’s Continuous Improvement & ROI?

 Not listed but essentially included by proxy via

consideration of State Targets (which include such consideration)

Nothing about Negotiating Targets for 2 Years

Automatic Target Adjustments at End of Year

 

Both down and up

Must reflect actual Economic Conditions & Participant Characteristics

(17)

Consequences of Failure

  

State

 

TA & Performance Improvement Plan for 1st _{Year Failure}

Reduction in Governor’s Reserve for 2nd _{Year Failure}

 

Doesn’t matter which of 6 WIOA programs fails

Failure to Report & Failure to Perform are cumulative 10% cut

Local

 

TA & Performance Improvement Plan

3rd _{Year Failure requires Reorg Plan including:}

 



Appointment/Certification of new Local Board

Prohibition of use of eligible providers and one-stop partners with poor performance

Other significant action as the Governor determines appropriate

 Upshot is requirement to track contractor performance

Definition of Failure is KEY

(18)

Definition of Failure is Key



NPRM creates Definitions for State Level

 

Failure to achieve an Avg of 90% of Target on All Measures on any single WIOA Program

Failure to achieve an Avg of 90% of Target on a single measure across all WIOA Programs

 Ex: Failure to achieve an Avg of 90% of Target on Median Earnings

 Failure to achieve 50% of Target on any single Measure

NPRM allows states to define Definition of Failure for

Boards

NPRM asks for feedback on proposals so these could

change

THIS PART OF NPRM MAKES IT CLEAR THAT THEY ARE

NOT CURRENTLY THINKING ABOUT INTEGRATED

REPORTING – WE NEED TO COMMENT ON THAT

(19)

More Reporting



Annual Report



 

Includes 4 Years of Data

Data broken out by all kinds of demographic groupings Average Cost

 

By level of Service?

Reported at a Customer Level? 



Administrative Cost Data – Including at Local Level

Section for Pay-for-Performance Contractors including EVALUATION of the design of such programs/strategies



ETP Report



 



Much the same as above but only includes 3 Employment Outcome & Credential Rate Measures

Reported for each Program of Study

Includes all students, not just those funded by WF system

(20)

Areas of Opportunity



Core & Intensive Services now just Career Services?

 Except the old Intensive Services are still segregated in the

Career Services section & only used “if determined to be appropriate in order . . . to obtain or retain employment”

Really there are 3 Types of Services

(not a Fed concept)



 

Informational Services (LMI & Availability of/Eligibility for Services)

Employment Connection Services (Job Search Assistance) Employability Improvement Services (Education/Training)



Definition of Participant

 

“Informational Service Only” proposed to be Excluded NPRM also excludes Self-Serve-Only customers

 

States have invested too much in Self-Service tools to do that Also, Self-Serve-Only customers have slightly better performance

(21)

Areas of Opportunity



Separating the Committed from the Curious in

Defining Participant

Ideally a customer would be a Participant if Received:

  

Employability Improvement Services (Education/Training); Staff-Assisted Employment Connection Services;

Self-Serve Employment Connection Services over an extended period



Extended Period?



Maybe self-serve in 2 of 3 or 3 of 4 consecutive weeks Key is to separate the Committed from the Curious



Definition of Exit – Change from 90 days?

 

NPRM proposes 90 days without Staff-Assisted Service NPRM proposes ignoring Self-Service entirely when calculating Exit but seeks input

(22)

Opportunities Bring Complications



LBB/TWIC Definitions aligned with DOL Definitions

 

Who is a Participant?

When does a Participant Exit?

Changes to Federal definitions Require

 

Changes to LBB/TWIC definitions OR

A 2nd _{Common Measures Reporting system}  One for LBB/TWIC; One for Federal

Problem: No idea what final definitions will be

  

Proposed Regs due in January 2015 but were issued DATE Final Regs due in January 2016 - will they be timely?

Legislative Session ends May 2015; Next ends May 2017

(23)

Speaking of Complications



WIOA continues to require a standard reporting

mechanism for all Core Programs

 Don’t know what the elements are or if we currently capture

them – Proposed “WISRD” won’t be out until late spring

Common Fiscal & Management Accountability System



“Establish and operate a fiscal and management accountability information system based on guidelines established by the

Secretary of Labor and the Secretary of Education . . .”

WIA required this as well but DOL/DOE seem far more interested in enforcing this provision, largely ignored under WIA

Feds seem to envision eventually having ONE common system for all core programs covering both financial & customer data

 Remember Core includes Adult, DW, Youth, WP, AEL & VR

 Of course there’s no money to pay for this . . .

(24)

To Be Continued

 



WIOA brings Massive Change – Even for Texas

Performance Measurement & Accountability

   

Replace Existing Common Measures Changes in State & Local Negotiations Bigger Consequences for Failure

Significant Reporting Requirements

 Now includes ETP Reporting

Opportunities Exist but Opportunities bring

Complications

Next Up: Reading & Replying to 2,658 Pages of

Regulations in ~60 Days

(25)

Additional Resources



Info on Existing Common Measures:

https://intra.twc.state.tx.us/intranet/plan/html/ta.html

Info on Employment Connection System:

https://intra.twc.state.tx.us/intranet/plan/html/development.html

Narrated Version of this Presentation:

https://intra.twc.state.tx.us/intranet/plan/docs/dev_wioa_performance.zip