Formative Assessment for Evaluation: Collaborative, community-based design of evaluation tools and systems

(1)

Formative Assessment for Evaluation:

Collaborative, community-based design of evaluation tools and systems

Dr Amy Gullickson Hadeel Al-Nawab

Centre for Program Evaluation

September 2015

(2)

• Context

• Plan

• Components (Webinars and workshop)

(3)

Advanced

Technological

Education

(4)

Funding Levels

Centers

(5)

177 Projects

38

Centers

(6)

Program Development and Improvement Curriculum Development

Professional Development Targeted Research

Leadership Capacity Building Teacher Preparation

Business and Entrepreneurial Skills Development for Students

(7)

(8)

Principal Investigator (PI) Co-Principal Investigator (Co-PI) Staff Evaluator

Roles

(9)

Principal Investigator (PI) Co-Principal Investigator (Co-PI) Staff Evaluator

Levels of Experience

(10)

(11)

Streams of Activity

(12)

(13)

Discipline areas

(14)

(15)

Discipline areas

(16)

(17)

(18)

OF PROJECT EVALUATION TOOLS / SYSTEMS

FORMATIVE ASSESSMENT SYSTEMS FOR ATE

(19)

VISION

A system of tools that ATE PIs can adapt

and use to identify and collect real-time

data for day to day learning and decision

making on their projects.

COMMUNITY BASED DESIGN OF PROJECT EVALUATION TOOLS / SYSTEMS

FORMATIVE ASSESSMENT FOR ATE

(20)

(21)

Streams of Activity

(22)

Logic Models (LM) Backwards Design Workshop Recruiting LM for Workshop Webinar 1 Webinar 1 Webinar 2 Webinar 2 Workshop Workshop Benefits for Participants

Benefits for the Project

Activity

(23)

Recipe for Success

• MOA – for participants

(24)

Webinars

Individual Project Logic Model Evaluative Activities

Workshop

Across Projects

High Level Logic Models

High Level Evaluation Data/Tool Needs

Synthesis

Blogs

Strategy and Prioritising

System Tools, Activities, and Priorities

(25)

Arlen Gullickson Amy Gullickson Lori Wingate Gordon Snyder Dee Al‐Nawab

Executive

Group

Advisory

Committee

Elaine Craft Mike Lesiecki Joyce Malyn‐ Smith

Helen Sullivan

Workshop

Keynote

Speakers

Janet Clinton John Hattie

Leadership Team, Advisory Committee & Keynote Speakers

ATE PIs & Senior

(26)

Webinars

Workshop

Across Projects High Level Logic

Models

Synthesis

Blogs

Strategy and Prioritising System Tools, Activities,

and Priorities

(27)

Webinars Overview

(28)

Webinars Planning & Implementation

Content

Design

Logistics

Blackboard

Online vs PDF versions

(29)

Ba

Blackboard

Chat

(30)

Logic Models (LM) Backwards Design Workshop Recruiting LM for Workshop Webinar 1 Webinar 1 Webinar 2 Webinar 2 Workshop Workshop Benefits for Participants

Activity

(31)

Recipe for Success

Logic Model (LM)

Backwards Design

Workshop Recruiting

Logic models and early identification of formative needs Webinar 1 Webinar 1 Webinar 2 Webinar 2 Workshop Workshop Benefits for Participants

(32)

Webinars

Workshop

Across Projects

Synthesis

Blogs

Strategy and Prioritising System Tools, Activities,

and Priorities

(33)

WORKSHOP

Moving toward Real Time Data for ATE

Projects

(34)

Principal Investigator (PI) 11 Co-Principal Investigator (Co-PI) 6 Staff 4 Evaluator 10

Workshop Attendees

Facilitator 5

(35)

(36)

(37)

Activity Structure INPUTS Facilitator Techniques Participant Expertise Post -its ACTIVITIES

 Each person/team gets a big post it and a pack of small post its.

 Use the big post it to create a logic model of your project or center in the activity stream you are assigned

OUTPUTS

Logic Model for each participant / team

15 Minutes

Task 1 Logic Models

Refining Individual Logic Models, Creating High Level Logic Models

(38)

Draft logic model

Backwards design: identify data collection

needs

Refined logic model & data mapped onto management needs and project success

throughout

Workshop Recruiting

Logic models and early identification of formative needs Webinar 1 Webinar 1 Webinar 2 Webinar 2 Workshop Workshop Benefits for Participants

(39)

Recipe for Success

Draft logic model

Backwards design: identify data collection

needs

Refined logic model Data mapped onto management needs and project success

throughout

Workshop Recruiting

Logic models and early identification of

formative needs

Logic models for activity streams, Prioritised needs for

tools and systems, Collaborators & next

steps Webinar 1 Webinar 1 Webinar 2 Webinar 2 Workshop Workshop Benefits for Participants

(40)

Webinars

Workshop

Across Projects

Synthesis

Blogs

Survey

Strategy, System Tools, Activities, and Priorities

(41)

(42)

(43)

SURVEY Results Highlights

Value of Ideas

Presented

Outcome of

Participation

Subsequent

Action

(44)

Continued collaboration

• Dashboard

development

• Common items for

surveys

• Evaluation checklist for

PD projects

(45)

(46)

Thank you