YSU Program Student Learning Assessment Report Due Date: Thursday, October 31, 2013

(1)

A Robust Assessment Process— “Closing the Loop”

YSU Program Student Learning Assessment Report

Due Date: Thursday, October 31, 2013

As part of our ongoing accreditation by the Higher Learning Commission (HLC), all academic degree programs at YSU participate in assessment processes and practices. This process is facilitated with help from the YSU Office of Assessment (OOA).

The purpose of the student learning assessment process is to provide ongoing opportunities for faculty to discuss student learning, review student artifacts, and improve educational programs through departmental discussions. The assessment planning and reporting process is about program self-‐evaluation and collective reflection for improvement of student learning within a program; it is not an evaluation of individual students, courses, faculty, or departments. Assessment is an ongoing activity to improve student learning and student success.

When is the report due? The deadline for all academic program assessment reports is Thursday, October 31, 2013.

What is crucial to include in these reports? Assessment reports should focus on data and explanations of how data are used from the previous academic year (2012-‐13); strengths and challenges of students’ learning; and realistic action steps for continuous improvement. It is the vitality of the program’s assessment processes, not results, the Office of Assessment will evaluate in its review. Reports should also update the status of action steps

resulting from previous years’ assessment processes. We encourage you to describe your assessment activities and findings in ways that are useful to you. As explained in the instructions, we will accept reports in other formats that include all necessary components.

Do I have to complete an assessment plan and an assessment report? NO! Each program may choose to complete either an assessment plan or an assessment report. Do not complete both. If a program wishes to realign or redesign assessment activities, they may choose to submit an assessment plan (in lieu of a report) using the plan template found on our website. Note: programs should not complete a plan two years in a row.

Where do I send the report? Your report should be submitted through our online reporting form at

www.jotformpro.com/ysuassessment/2013acadreport

(2)

What’s new this year?

Combined Workshops: As part of the streamlining process the OOA has combined the plan and report workshops into one. Workshops will be held in late September. If you would like to sign up for a workshop, or would like to access the workshop materials, please visit

web.ysu.edu/assessment/templates.

Report Preparation Open Forums: In addition to workshops open forums for academic departments will be held in October. These drop-‐in forums at the OOA office will allow departments to ask specific questions or brainstorm ideas. The forums are for those who do not need to attend a training

workshop but would like to discuss their reports. See the OOA calendar at http://web.ysu.edu/assessment for forum dates.

Online Reporting Process: In 2012 the Office of Assessment moved to an online reporting process. This allowed the office to streamline the reporting process and will allow for more powerful analysis of plans and reports. A walkthrough of the form will be provided at the workshops and open forums, or you may contact our office for assistance. If you are experiencing extreme difficulties with this system we will allow email submissions.

How will the reports be reviewed? A team from the YSU Assessment Council will review the reports and provide feedback based on a rubric (a set of evaluative criteria). Feedback will be returned separately for each degree program, even if a combined report is submitted. To ensure that our focus is on the vitality of program self-‐appraisal and improvement of student learning outcomes (and not the form itself), note that questions on the rubric are weighted differently. You can view the rubric and specific question values on the OOA website.

The Office of Assessment also uses volunteers to assist the Council in report reviews; this is a great, short-‐term university service opportunity. If you would like to volunteer please contact our office.

Do I need to keep a record of this report? Faculty need to maintain their own department’s student learning archive for 10 years. The archive can be electronic and/or paper and must include: all summaries of data on student learning, representative examples of student work at various levels of performance, and rubrics and/or criteria used to evaluate student learning. The archive can also include newsletters, minutes of meetings on

assessment, representative examples of the tools used to measure student learning, assessment plans and reports (2008-‐2018), and any other relevant materials.

For accreditation purposes the OOA maintains a record of all plans and reports. Assessment plans and reports will be posted on a secure site, currently the OOA Word Press site, to provide programs a wider distribution of their results and recommendations. The OOA also seeks to provide information back the campus community in useful ways; for example, the OOA “Use of Data” Chart shares how other programs are collecting and using student learning data for improvement purposes. Contact our office if you would like to access the report archive.

How do I know what you are looking for in the report?

• For definitions of important terms, as well as tips on filling out the report, please see the “Assessment Report Tips and Hints” document on the OOA website at web.ysu.edu/assessment/templates.

(3)

What should happen if our program’s Student Learning Outcomes (SLOs) have changed? If your department has made changes to the learning outcomes, since the 2012 SLO review, please send a separate copy of all the SLOs to both the Office of Assessment ([email protected]) and to either Deborah Lowe for undergraduate learning outcomes ([email protected]), or Angie Urmson-‐Jeffries for graduate learning outcomes

([email protected]), by October 31, 2013. SLO revisions are incorporated into future editions of the Undergraduate and Graduate Bulletins.

Report Instructions

To Complete the Report:

Please answer all questions regarding your department’s assessment activities for the prior academic year, 2012-‐13. Include data on only two Student Learning Outcomes (SLOs) per program. Each SLO will, in turn, include two methods (for a total of two SLOs and four methods in a report). For programs with well-‐developed assessment processes, or those programs with low numbers of majors, it is allowable (but does not mandate the need) to supply multiple years’ data and evaluation for a particular SLO.

Departments with more than one degree program: Separate reports may be submitted for each degree program or certificate program with 18 or more semester hours. In some cases it is possible to combine multiple programs into one report. There are several considerations when combining reports:

• Multiple programs—if SLOs are the same or are substantially similar for multiple programs in your department (e.g., a BA and BS in

Mathematics), you may include related programs in one report. However, if you choose this option, you must disaggregate data on assessment outcomes for each program.

• Multiple degree levels—similar programs at different levels (e.g. an AA and BA program) may be combined into one report; however, you should break out data for each degree level.

• Multiple Tracks—when a program has multiple tracks, but these all result in the same degree (e.g., teacher education has multiple tracks, including elementary and secondary education), the department may rotate through the tracks on a department-‐developed timeline, but should cover all SLO’s within a 3-‐4 year cycle Note: tracks are possible areas of concentration in a degree program; tracks are not equivalent to a minor. Your report could, depending on the programs in your department, include a combination of different programs, tracks, or various degree levels of the program. As long as you rotate and assess the SLO’s in all programs over the course of your cycle, the way they are completed in the report is a

departmental decision.

SECTION 3 (Learning outcome data) INSTRUCTIONS:

PAY CAREFUL ATTENTION TO SECTION 3: While you have the option to turn in just one report for your department, you should report on two Student Learning Outcomes for each program on this template. (e.g. the Foreign Languages Department would submit TWO SLOs for EACH of their programs, French, Italian, and Spanish —a total of six SLOs). If multiple programs share the same SLOs, you could complete just one Section 3, as long as you show data for each group (so, if all Foreign Languages programs share 2 SLOs, just in different languages, they could then complete one Section 3 with data for each program. Also, if you are submitting a combined report and need more than the table provided in section three, just copy and paste. Note that you will have this option in the online form as well.

Alternate to Completing the Form:

To create a meaningful process, the Program Student Learning Assessment Report Form is designed to be flexible. If you prefer to write a narrative to describe your assessment process or if you have produced an accreditation report/self-‐study within the past year (or produce accreditation assessment documents yearly), you can include such document(s) as attachment(s) to the Program Student Learning Assessment Report and then provide page

(4)

numbers from your attachment on the report form. Mapping the page numbers on the report form enables reviewers to readily identify how your narrative addresses each question as it is asked on the Report form. Additionally, highlighting that information on the corresponding page within the narrative or accreditation document will help the Assessment Council quickly locate responses for which they are looking. Please also read the form and form instructions before completing an alternate report so that all necessary components may be included.

Assessment process image credit, page one: http://www.westminster.edu/acad/oaac/cycle.cfm.

(5)

Section 1 – Identifying and Contact Information

Note: Save time and don’t fill out this section until you are completing the online form!

Department: Counseling, Special Education, and School Psychology

Name: Matthew J. Paylo, Ph.D.

Email Address: [email protected]

Campus Phone: 330-‐941-‐3264

Degree Program(s) Covered by This Report: Counseling Program Level (associate, bachelor, undergraduate or graduate

certificate, master, doctorate): Master of Science Degree in Education (M.S. Ed.)

List of All Program Tracks in Program (if applicable):

Clinical Mental Health Counseling, School Counseling,

Addictions Counseling, and

Student Affairs and College Counseling

Specific Program Tracks Included in Report: (if applicable):

Student Affairs and College Counseling Number of attachments included with this report (not

including this report) 0

Section 2 – Outside Accreditation

1. Does any program covered by this report undergo program-‐specific accreditation? If yes, please list the

program(s). If no, proceed to Section 3.

Yes

All Program Tracks:

Student Affairs and College Counseling 2. Please provide the name of your accrediting agency. If

multiple programs with various accrediting agencies are covered in this report, please reference the name of each

agency to the appropriate program.

Council for Accreditation of Counseling and Related Educational Programs (CACREP) 3. How often does your accreditation occur, and when is

your next self-‐study, site visit, and/or interim report

due? Include all dates as necessary for each program

Accreditation visits occur every 8 years; the next accreditation visit will occur in Fall 2014 and the self-‐study is due January 2014.

(6)

included in this report.

4. How often do you submit interim reports to your

primary accreditation agency or agencies? Interim Reports are submitted every four _years

(7)

Section 3 – Assessment and Evaluation of Student Learning Outcomes

Note: This form only contains a table for one program/two learning outcomes. If you are combining multiple programs into one report, space will be provided in the online form for additional program and learning outcome information. For programs drafting in this template, just cut and paste the table as needed.

Student Learning Outcomes for

Counseling Program (program

name)

Learning Outcome 1

Learning Outcome 2

1.

State the student learning

outcome assessed:

Students will demonstrate a professional identity, including professionalism, and fitness to practice as a professional counselor

Students will demonstrate knowledge and skills specific to their program areas (i.e., Addiction, Clinical Mental Health, School, & Student Affairs and College Counseling)

2. How did you assess student

performance?

Please include:

• Method of collection-‐please provide

no more than two methods for each learning outcome. One method must be a direct measure.

• Where data was gathered

• Number of students assessed

• Performance criteria

• Data summary-‐What patterns

emerged as you studied the data with regard to this particular student

learning outcome.

1. (Direct Measure). The Counselor Preparation Comprehensive Examination (CPCE): Section 8

(Professional Orientation and Ethics section). The CPCE is our program’s comprehensive examination and our students are required to pass it before they are approved to start Practicum and Internship. The CPCE is sent A total of 23 students took the CPCE examination in the last academic year (2012-‐2013). Of these, 91.3% (21) passed the examination and 8.7% (2) did not pass the examination. Passing the CPCE requires scores within ones standard deviation of the mean; this is a general practice for counseling programs throughout the country. Subsequently, both students passed on their second attempt. Within that examination, there are eight specific sections that align with CACREP’s (Council for

Accreditation of Counseling and Related Educational Programs) eight core areas. Section 8 of the examination addresses the Professional Orientation and Ethics

standards. Our students exceeded the national average (M=11.14) on this section and performed higher (M

1. (Direct Measure). The Counselor Preparation Comprehensive Examination (CPCE): Section 8 (Professional Orientation and Ethics section); Praxis (School Guidance and Counseling exam; School counselor Licensure Examination); National Counselor Examination (NCE; 1st_{Tier
Counselor
Licensure}

Examination); and National Clinical Mental Health Counselor examination (NCMHCE; 2nd_{Tier
Counselor}

Licensure Examination).

A total of 58 students took the CPCE (23), Praxis (12), NCE (8), and the NCMHCE (15). These examinations are utilized to measure this standard because the knowledge and skill specific domains within each on of our tracks (Addiction, Clinical Mental Health, School, & Student Affairs and College Counseling) are evaluated by one of these examinations. Our performance criteria for the counseling program is an 85% first time pass rate. The 2012-‐2013 academic year data on these four examinations:

(8)

=11.47) in this knowledge area of Professionalism and Ethical behaviors, than the other sections.

CPCE = 91.3% Passing rate (21 pass; 2 fail) PRAXIS = 100% Passing Rate (12 pass) NCE= 87.5 Passing Rate (7 pass; 1 fail)

NCMHCE = 86.7 Passing Rate (13 pass; 2 fail)

The data reviewed that on the CPCE (general core knowledge and skills) our students performed consistently well (at or above national mean averages). The school-‐counseling track seems to prepare students extremely well for the praxis examination. The NCE and NCMHCE were lower pass rates (yet still within our performance criteria) and mirrored state and national examination results. 2. (Direct Measure). Site Supervisor’s Final Semester

Evaluation.

All 87 Site Supervisor’s Final Semester Evaluations (2012-‐ 2013) were reviewed. Additionally, these evaluations have qualitative questions, which include a short statements of their judgment of the intern as a prospective counselor, their perception of the interns fitness to practice, the intern’s professionalism, and the intern’s overall performance). These forms are kept and maintained in the student’s clinical files behind a lock door (3rd_{floor
Beeghly).

After
reviewing
all
of
the
evaluations,}

no complaints or issues were made specifically to the competency, professionalism and fitness to practice as a counselor, and overall performance was high (i.e., average of 92.08 of a potential 100).

2. (Direct Measure). Site Supervisor’s Final Semester Evaluation.

All 87 Site Supervisor’s Final Semester Evaluations were reviewed (Questions including intern’s specific track knowledge, skills, competencies, and the intern’s overall performance). After reviewing all of the evaluations, interns scored extremely high throughout and their overall performance was extremely high (i.e., average of 92.08 of a potential 100).

(9)

3. What successes and

challenges do you see in

students’ learning as a result

of these assessments?

NOTE: This question specifically looks for you to assess the students’ strengths and

weaknesses based on the data. It does not ask

for an assessment of pedagogy or curriculum.

Strengths: The vast majority of our students have developed adequate knowledge around professional practice, ethics, and professionalism. In addition, all of the site-‐supervisors reported interns not only possessed professional knowledge, but displayed professional skills and ethical behaviors throughout their internship experience.

Weakness: A small number of our students are lacking in the general knowledge to pass the CPCE examination on the first attempt; subsequently, passing on the second examination.

Strengths: The vast majority of our students have developed specific knowledge and within their track of student (i.e., Addiction, Clinical Mental Health, School, & Student Affairs and College Counseling). These students not only meet the academic rigor of the program’s coursework, but a majority of the students display their specific knowledge and skills by pass their professional examination on their first attempt. In addition, all of the site-‐supervisors reported that these interns possessed specific knowledge and skills within their track and rated them overall extremely high. Weakness: A small number of our students are lacking in the general knowledge and skills to pass the CPCE, NCE, and NCMHCE. While we are consistent with state and national averages, some of our students are lacking in the specific knowledge and skills required to be licensed and successful in their chosen profession.

4. How did you use the data: e.g.

what recommendations and

action steps to the program

have resulted from reviewing

this data and where is the

department in this process?

Note: Data may not indicate the need for change; this is fine. In this case, just explain your department’s conclusion resulting from

the data.

During this review of our comprehensive exam (CPCE) data, the faculty determined that we needed to change textbooks in COUN 6972 Career Counseling and COUN 6976 Social and Cultural Issues in Counseling to provide students with more of the content knowledge necessary to increase their performance on the CPCE.

During the review of our SLO 6 data, faculty decided to increase the assessment process of the specific

knowledge and skills within each track. Therefore, faculty reviewed and established new critical tasks with assessment rubrics for every course within our entire program, which highlights the core CACREP standards and specific CACREP track (i.e., addiction, clinical mental health, school, and student affairs and college counseling) standards that are being

addressed. This will be done through the use of TaskStream (i.e., www.taskstream.com), which provides faculty a means to manage all of this additional data.

(10)

10 Section 4 – Use of Data

Complete this section only once, regardless of the number of programs included in the report. NOTE: It is of critical importance to the Higher Learning Commission that programs make strong connections between the data they are collecting on Student Learning Outcomes (Section 3) and using that data to: reflect upon the current state of student learning; make changes to the curriculum (when necessary); share that data with stakeholders (both internal and external).

5. How is your department working to engage all faculty in

the assessment process?

For example, it is recommended that all departments meet once a year to discuss assessment processes—the highlights of such a meeting could be

included here.

During faculty meetings (which convene once per month), curriculum committee meetings (which convene twice per year), and community advisory board meetings (which convene each spring), faculty evaluate the counseling program (all tracks) in light of the program’s mission statement, objectives, and student learning

outcomes. Additionally, The Counseling Program faculty have rigorously reviewed SLOs, survey data, Comprehensive Examinations (CPCE), National Counselors Examinations (NCE), Praxis examinations, critical tasks in coursework and consulted with administrators, alumni, employers, site supervisors, and students. All faculty full time faculty members are present for these meetings aimed at program assessment.

6. How are you sharing the results of the data discussed in

section three with your students, your college, and other

stakeholders?

Stakeholders should include groups internal (e.g., students) & external (e.g., Career Services, college-‐wide assessment committees) to your

department.

Systematic program evaluations are made available to all faculty members, to

students, to the college, and to the community/alumni. The Department

Annual Report, which includes the discussion of progress on our student

learning outcomes, is available for student and faculty review in the

departmental office. Summary reports of assessment data are uploaded on the

Counseling Program website. An annual announcement that these reports are

available for student review is posted to the Counseling Program listserv.

Faculty, students, and the external college and community are encouraged to

review this information.

Internal/Faculty: Program faculty review the data from program

evaluations to determine areas of strength and needs for improvement;

faculty make revisions to course content and/or curricula -as needed-

via the counseling program’s curriculum committee. Faculty convene

two meetings per year (one in the spring and one in fall) to specifically

(11)

11 program development/ planning/ improvement as well as to evaluate

prior action steps.

Internal/Students: All student learning outcomes are available to all

graduate students on the departmental website

http://web.ysu.edu/gen/coe/Evaluation_m38.html

External/College/ Community: The outcome of our program

evaluations, SLOs, alumni, student, and site-supervisor surveys are

now available on the departmental website

http://web.ysu.edu/gen/coe/Evaluation_m38.html

Therefore, other college members, students, alumni and community

members may review this information. Annually, an email summary

of our program evaluation is sent out to all counseling stakeholders.

7. In the past several years (e.g., 2008-‐2012) you have analyzed data and identified action steps for learning outcomes. Considering action steps from previous years, what has been an impact on student learning as a result of (one of) those action steps?

Only one example is needed; you can reference past reports (2008-‐2011) for this information.

The Counseling Program maintains a culture of continuous improvement.

The following are a few examples of the evidence of modifications at the

overall counseling program level, counseling track level, and course level

during the last three years.

Program Level: The counseling faculty wished to enhance and strengthen

the counselor identity of students in all tracks, while concurrently ensuring

that students have a basic level of mastery of the eight core CACREP

standards. As such, faculty decided to move towards using the

Comprehensive Examination (i.e., Counselor Preparation Comprehensive

Exams) for all tracks (addictions, clinical mental health, school, student

affairs and college counseling). This became a program policy in fall of

2011. Over the past two years, our Student Affairs students have performed

well on the CPCE (89 % passing rate) slightly exceeding our expectation of

an 85% passing rate.

(12)

YSU Program Student Learning Assessment Report Due Date: Thursday, October 31, 2013