Quality Assurance

In this section of the survey respondents were asked about the evaluation and assessment of professional development services they provided. The questions asked were as follows:

A. How are the professional development opportunities evaluated (check all that apply)?

B. Do you link your professional development offerings to measureable outcomes (e.g., Environmental Rating Scores)?

C. If you do link your professional development offerings to measureable outcomes, what outcomes are measured?

D. Are faculty and/or trainers evaluated at least annually?

E. Do faculty and/or trainers receive any type of supervision?

F. For those able to articulate prior knowledge and skills to higher education, is there a measure(s) that is used to determine competency?

G. If there is a measure(s) to determine prior knowledge/skills, what is that measure(s)?

H. To what degree is/are: (1=not involvement/expectations/action and 5=high involvement/expectations/action)?¹⁵

I. Other comments related to quality assurance.

Evaluation: Most often, survey respondents report professional development opportunities are evaluated through assessments at the completion of the course or training; this was true for 79% of respondents. 71% say they use tests or quizzes of competencies and/or learning objectives. 59% use projects to evaluate training, 57% use demonstrations of competencies met and 55% use satisfaction questionnaires. A smaller percentage, less than 50% of respondents, use surveys (48%); papers (47%); course or training follow-up after a certain period of time (35%); interviews with participants (26%);

and focus groups with participants (17%).

15 See appendix for complete list of indicators for this question

55% 57% 59% 71% 79%

satisfaction

questionnaires demonstration of

competencies met projects tests/quizzes of competencies and/or

learning objectives

course/training assessment at the

completion of a course and/or

training

Type of PD Evaluation

Full Aggregate Report Page 57 of 208 Measureable outcomes: 36 respondents (62%) reported they link their professional development offerings to measureable outcomes (for example: the Environment Rating Scores). 11 respondents (19%) do not link their professional development to measureable scores and another 19% are not sure. Of those who said they do link PD to measurable outcomes, 28 responded with more detail. In general most respondents referred to program testing and evaluation (including: children, teachers, family and community). As well, evaluation and testing of the students accessing professional development was frequently mentioned. Some respondents indicated multiple outcomes; therefore the number of measureable outcomes reported is greater than the number of respondents.

Many of the respondents indicated the outcomes measured are associated with accreditation, certification and/or competencies (16). These include:

NAEYC Accreditation Standards (9) NAFCC Accreditation Standards (2) Oklahoma State Competencies (8)

o General reference to state competency testing (1)

o Oklahoma State Competencies for Licensure/Certification (1) o Oklahoma Commission for Teacher Preparation State Standards (1) o Oklahoma General Competencies for Teacher Certification (1) o Oklahoma Core Competencies for Early Childhood Practitioners (2) o Oklahoma Elementary and Early Childhood Competencies for Reading

Instruction (1)

o Oklahoma Christian University Competencies and Conceptual Framework (1) National Council for Accreditation of Teacher Education (CAEP) (3)

General knowledge of standards and competencies in ECE (1)

A few of the respondents mentioned the following classroom assessments (3):

ERS (3) CLASS (3)

Teaching Strategies Classroom Ratings (1) Arnett Caregiver Interaction Scale (1) ELLCO (1)

PAS (1) BAS (1)

More general responses included:

Child progress/performance (3) Course competency and testing (4) Oklahoma Early Learning Guidelines (1)

Teacher qualifications and receipt of credentials (2)

Full Aggregate Report Page 58 of 208 Health and safety (facility and equipment) (2)

Adequate equipment to provide individualized education (1) Staff and parent surveys (1)

Disability services (1) Family needs (1)

Application of knowledge gained through professional development (examples:

designing effective workshops, programs and/or lessons plans; observation and evaluation of program and children; demonstrate professionalism and communicate effectively lessons learned) (3)

Dependant on training received (1)

Faculty/Trainers Administering Professional Development Opportunities:

Respondents were asked if the faculty or trainers of the professional development opportunities they offer are evaluated annually. 90% said yes, 7% said no and 3% were not sure. When asked if the faculty/trainers receive supervision, 89% said yes, 7% said no and another 7% were not sure.

Measurement of prior knowledge/skills: In response to the question, “For those that are able to articulate prior knowledge and skills to higher education, is there a measure(s) that is used to determine competency?,” 34% said yes, 22% said no and 44% were not sure. Of those who said yes, 18 provided more detail. A few of the respondents indicated multiple measures for assessing knowledge/skills; therefore the number of measures is greater than the number of respondents. The most frequently reported measure in general was the use of testing (9). Specific comments regarding testing as a measure were:

90% 87%

7% 3% 7% 7%

Get evaluated annually? Receive Supervision?

Do Faculy/Trainers of PD...

Yes No Not Sure

Full Aggregate Report Page 59 of 208 7 assessments required by NAEYC/NCATE, including three certification tests, post program questionnaire of administrators, teachers and alumni (1) Adjuncts utilize a pre/post test (1)

Competency certification tests (1) Examinations/testing (2)

ODCTE State Competency Exam (1) TABE and SAGE tests (1)

WorkKeys test (1)

Oklahoma Core Competencies (1)

The second most often reported measure was the use of education related references, transcripts and/or certificates (including professional development) (7). A few

respondents referenced the use of portfolios/projects as a measurement to determine prior knowledge (4). It is interesting to note that 1 respondent reported that prior knowledge is “overwhelmingly difficult to measure”.

Involvement, expectations, action: Using a scale of 1 to 5, where 1=no

involvement/expectations/action and 5=high involvement/expectations/action,

respondents were asked to rate to what degree a number of statements were true. Those who answered “not sure” or “n/a” were excluded from data analysis. The highest rated statement, with an average rating of 3.9 was:

Oklahoma's early childhood program standards (e.g., pre-K, QRIS, Early Childhood, home visiting, licensing) encompasses professional development expectations

Also rated highly, with average ratings of 3.8, respondents believe to a large degree that accountability and monitoring are a part of the early childhood professional development system and that Oklahoma's core competencies for early Childhood practitioners are encompassed in professional development opportunities. The lowest rated statement, although still rated at a high degree with an average of 3.3, respondents said that stakeholder and public input (including families) are sought regarding the professional development of early childhood professionals.

Full Aggregate Report Page 60 of 208 To what degree is/are: (1=no involvement/expectations/action and 5=high

involvement/expectations/action)

Average Rating a. Oklahoma's early childhood program standards (e.g., pre-K, QRIS,

Early Childhood, home visiting, licensing) encompassing professional development expectations

3.9

b. accountability and monitoring a part of the early childhood

professional development system 3.8

c. Oklahoma's core competencies for early Childhood practitioners

encompassed in professional development opportunities 3.8 d. Oklahoma's early learning guidelines encompassed in professional

development opportunities 3.7

e. stakeholder and public input (including families) sought regarding

the professional development of early childhood professionals 3.3

A final comment related to quality assurance stated that, “There is a verbal commitment to quality standards but there is such high variability in the interpretation of the standards and what they look like in actual practice that I would say there is little consistency.”

Full Aggregate Report Page 61 of 208